The ESP32-S3 is a remarkable chip for a few dollars: Wi-Fi, enough compute for a wake word, I2S audio in and out, and a small display if you want one. What it cannot do is understand speech, reason about a question, remember the user, or call an API. A voice assistant on this class of hardware is therefore an exercise in deciding what stays on the board and what moves to the cloud.
This article describes the split we use in production: the firmware keeps the microphone, speaker, display, and sensors, and a cloud runtime does speech recognition, the language model, memory, knowledge, tools, and speech synthesis. It is the same runtime that answers on the web and on messaging channels, so a device is one more channel of an assistant, not a separate product.
The split
| On the ESP32-S3 | In the cloud runtime |
|---|---|
| Wake word, audio capture, playback | Speech-to-text, chat model, text-to-speech, or a realtime speech-to-speech model |
| Wi-Fi setup, reconnection, status display | Memory, knowledge base, tools, and MCP |
| Encrypted audio over MQTT + UDP | Session routing to the right assistant and user |
| Firmware updates (OTA) | Activation codes, device ownership, firmware catalog |
| Hardware tools the firmware exposes (volume, LEDs, sensors) | Deciding when to call them, as part of the conversation |
The firmware is built on the open-source xiaozhi-esp32 project and ships for ESP32 and ESP32-S3 boards. Because the heavy work runs in the cloud, a device gets every model upgrade, every new tool, and every knowledge base change without a new firmware build.
From a blank board to a conversation
- Flash from the browser. The documentation site flashes firmware over Web Serial; no toolchain is needed. The firmware already points at the hosted platform's OTA URL, so the only time you change it is when you build your own firmware.
- Join Wi-Fi. The device opens a setup hotspot, and you enter the network credentials from a phone.
- Read the activation code. On its first connection the device shows and speaks a six-digit code.
- Register it. In the admin panel or the web app, enter the code and choose who owns the device. From then on the device is bound to one assistant and one user, and it talks with that user's memory and the assistant's knowledge and tools.
- Talk. Say the wake word and ask. The spoken reply comes back to the speaker, and the conversation appears in the user's history like any other.
Every time the device connects to Wi-Fi, it calls the OTA endpoint first. The answer carries the server time, the activation code if the device is not linked yet, the connection details for the voice session, and a firmware update when one is available and auto-update is on. Devices stay current without ever being re-flashed.
Why MQTT + UDP, not WebSocket
Our first firmware streamed audio over WebSocket. Moving to MQTT for control messages and UDP for audio cut round-trip latency on the device from 380 ms to 165 ms, reduced RAM use from 45 KB to 12 KB, and extended battery life by about 40%, with audio encrypted with AES-128 in transit. The full measurements and the protocol design are in MQTT + UDP: bringing AI down to edge devices. A gateway terminates these connections and hands each session to the runtime, which routes it to the right assistant.
Device tools: the assistant controls the hardware
The interesting part of a voice device is that the conversation can act on the device itself. The firmware exposes hardware functions as tools: volume, screen brightness, LEDs, a sensor reading, or a motor on a custom board. During a voice session, those tools sit next to the assistant's other tools, and the model calls them when the conversation needs them. "Turn it down a bit" becomes a tool call; so does "what's the temperature in here" if the board has a sensor.
Device tools only exist during a session with that device and are declared by the firmware, so there is nothing to configure in the admin panel beyond the assistant's normal tool settings. Custom assets, such as a different theme on the display, can be pushed from the web app to an online device.
Realtime or Standard on a device
A screenless device is where latency matters most, and the runtime's two voice modes behave differently there. Standard (speech-to-text, chat model, text-to-speech) reaches first audio about 1.8 seconds after the user stops talking; Realtime (one speech-to-speech model) reaches it in about 0.7 seconds and can be interrupted naturally. The trade-offs on accuracy and cost are the same as in the browser, and we compare them in speech-to-speech vs ASR → LLM → TTS. Whether barge-in works at all depends on echo cancellation on the board, which we cover in AEC for AI voice assistants.
What this is good for today
This stack is built for smart speakers, desk companions, and voice front-ends for internal tools. Robots, toys, and in-car devices are on the roadmap: the device layer supports them, but there is no packaged product for them yet, and we prefer to say so.
Each plan allows a number of devices per assistant, and a voice session on a device uses credits like any other voice session. To try it, flash a supported board from the device documentation and register it against a free assistant.