MQTT + UDP: Bringing AI Down to Edge Devices

When we brought our AI assistant onto hardware devices (ESP32-S3), the biggest challenge wasn't the AI model — it was the transport protocol. This post explains why we chose MQTT + UDP over plain WebSocket, and how we designed the MQTT Gateway.

The Problem with WebSocket for IoT#

WebSocket is the natural choice when you already have an API over HTTP/WebSocket. But on embedded devices, it exposes several serious limitations:

• High RAM consumption — Maintaining a TLS session plus HTTP upgrade headers costs ~45KB of RAM on the ESP32 (320KB total)
• Complex reconnection — Losing WiFi requires handling reconnect + re-auth manually
• No QoS — No delivery guarantees when the network is unstable
• Unsuitable for audio — TCP overhead (~50ms) is too high for realtime voice streaming

MQTT for Device Control#

MQTT (Message Queuing Telemetry Transport) was designed specifically for IoT back in the 1990s at IBM. It's a lightweight pub/sub protocol with a minimal 2-byte header.

# Topic structure in Assistant Core
TOPIC_CMD    = "assistant/{device_id}/command"
TOPIC_RESP   = "assistant/{device_id}/response"
TOPIC_STATUS = "assistant/{device_id}/status"
TOPIC_AUDIO  = "assistant/{device_id}/audio"

# QoS levels in use
QOS_STATUS = 1   # At least once — device state
QOS_CMD    = 2   # Exactly once — critical commands
QOS_AUDIO  = 0   # At most once — audio (latency > reliability)

MQTT's advantages for IoT:

• Last Will Testament (LWT) — The broker automatically publishes when a device goes offline, with no manual heartbeat needed
• Retained messages — A device that just reconnected immediately receives the latest state
• RAM savings — The ESP-MQTT client library uses only ~12KB of RAM
• Offline buffering — The broker holds messages while a device is temporarily offline (QoS 1/2)

UDP for Audio Transport#

With voice streaming, every millisecond of latency matters. TCP's retry mechanism (WebSocket/MQTT) introduces jitter that's unacceptable for realtime audio. UDP is the right choice because:

• No connection overhead, no retries
• Low, stable latency (~30–50ms on LAN)
• 1–2% packet loss has no meaningful impact with the Opus codec

# Audio pipeline on the device
Microphone (16kHz, 16-bit, mono)
  → VAD (Voice Activity Detection)
  → Opus encoder (8kbps)
  → UDP packet (20ms frames = 160 bytes)
  → Gateway → ASR → LLM → TTS
  → UDP packet (Opus audio)
  → Opus decoder on device
  → Speaker

MQTT Gateway Architecture#

We built mqtt-gateway/ — a Node.js service that bridges WebSocket and MQTT. This lets us:

• Connect the web dashboard to devices over standard WebSocket
• Connect devices over lightweight MQTT
• Have the gateway translate messages between the two protocols in real time
• Do server-side audio processing without exposing an API directly

// mqtt-gateway: bridge WebSocket ↔ MQTT
mqttClient.on('message', (topic, payload) => {
  const deviceId = parseDeviceId(topic)
  const wsClient = wsClients.get(deviceId)

  if (wsClient?.readyState === WebSocket.OPEN) {
    wsClient.send(payload)  // forward to web client
  }
})

wsServer.on('connection', (ws, req) => {
  const deviceId = parseDeviceId(req.url)
  wsClients.set(deviceId, ws)

  ws.on('message', (data) => {
    // forward command to device via MQTT
    mqttClient.publish(
      `assistant/${deviceId}/command`,
      data, { qos: 2 }
    )
  })
})

Real-World Measurements#

After switching from plain WebSocket to MQTT + UDP:

When Do We Still Use WebSocket?#

MQTT + UDP isn't always the right answer. We still use WebSocket for:

• The web chat interface (browsers have native WebSocket)
• Real-time monitoring on the admin dashboard
• Cases where deploying a separate MQTT broker isn't worth it

The simple rule: if the device is a browser → WebSocket. If the device is hardware with limited RAM that needs voice → MQTT + UDP.