Sebastian: bidirectional voice on my own hardware

I've had a gadget sitting on my desk for a few weeks that already talks: Sebastian, a conversational voice-to-voice speaker built on a Seeed ReSpeaker XVF3800 board with a XIAO ESP32-S3. The lab question that justifies it: what do you learn if, instead of buying a ready-made speaker, you control the complete voice path, from the microphone silicon to the agent that answers? You speak, the device captures and cleans your voice in hardware, publishes it over WebRTC into a LiveKit room, and a Python agent answers you through the speaker. The entire loop is voice.

And it works: bidirectional conversation validated on real hardware, with an LED ring pointing at whoever is speaking (direction of arrival), a mute button and, as far as I know, the first LiveKit client on ESP32 written in Zig. It isn't a product: it's my lab. The repo is public, and here I'll tell you how it's built, what broke along the way and what's still missing.

The audio path

Loading diagram...

The piece in charge is the XVF3800, an XMOS voice DSP: the four-microphone array connects to it, and the dirty work —beamforming, echo cancellation, noise suppression— happens in its silicon. The ESP32-S3 doesn't process signal: it moves PCM between the I2S bus and the network.

The details that gave me the most trouble:

The XVF is the I2S master (48 kHz, 32-bit, stereo) and the ESP32 the slave, with two separate I2S ports for mic and speaker: a single duplex channel corrupted the DMA and crashed the board.
Capture runs at the consumer's pace, with no intermediate ring buffer. With a free-running buffer, the drift between the XVF's clock and the consumer's sounded like a helicopter.
There is a single noise-suppression pass in the chain. Two in series —the chip's and the agent's— left a metallic, tinny artifact. At first I used the array's raw beam and delegated that pass to LiveKit's BVC; when I self-hosted I discovered BVC is a Cloud service, so the single pass moved to the chip's processed channel. Measured with the dual-channel probe: 38.5 dB of SNR versus the raw beam's 30.7 dB. As a bonus, that enabled full-duplex with tracking: you can interrupt it while it's speaking and the beam follows you.
The mic is published as Opus at 48 kHz; the reply comes down over WebRTC, is rendered at 32 bits —at 16 it sounded gritty— and goes out through an AIC3104 codec to a 5 W speaker.

The echo canceller deserves a separate confession. It wouldn't converge, and the first diagnosis blamed a factory parameter that shipped at zero: I published the “fix” —a four-byte I2C write— and called it solved. It was a placebo. The real fix was in the reference gain and enabling the far-end DSP, and the code comment tells it without anesthesia:

/// FAR_EXTGAIN is dB = external gain past the reference tap; our tap is after the
/// ESP sw-vol and the AIC3104 is 0 dB → 0.0 (factory) is correct. The 1.0 we
/// shipped earlier was a placebo from the FAR_EXTGAIN misdiagnosis.
pub fn applyConfig() bool {
    if (!writeF32Verified(RESID_AUDIO_MGR, MGR_REF_GAIN, 1.0, "REF_GAIN")) return false;
    if (!writeF32Verified(RESID_AEC, AEC_FAR_EXTGAIN, 0.0, "FAR_EXTGAIN")) return false;
    ...
    if (!writeF32Verified(RESID_AUDIO_MGR, MGR_MIC_GAIN, 90.0, "MIC_GAIN")) return false;
    ...
    if (!writeU8Verified(RESID_AUDIO_MGR, MGR_FAR_END_DSP_ENABLE, 1, "FAR_END_DSP_ENABLE")) return false;

The barge-in has its subtleties: the model generates audio faster than real time, so on interruption there were seconds of reply queued in the device's FIFO —“it won't stop”—. Canceling the generation isn't enough: the agent publishes interrupted and the firmware drains the render FIFO. A clean interruption.

The word that wakes it

Listening is local: a microWakeWord model —a 62 KB streaming CNN— decides on the device itself when to open a session. Nothing leaves the gadget until it hears its word, and the LiveKit room (with the paid model behind it) only opens on demand: zero cost at rest. A 12-second pre-roll in PSRAM keeps what you said before the room is ready, so “turn on the living-room light” said in one breath arrives whole.

Here comes the part of lab work that hurts to tell. I trained my own Spanish wake word —“Sebastián”, of course— on my M4 Pro: 18,220 positive samples (TTS with nine Piper voices plus 120 real recordings made with the XVF itself), about 20 GB of negatives and a 99.29% recall on validation. In real use, however, it let through too many false positives, so today the device ships with the stock English “Okay Nabu” model with the threshold raised. Getting the Spanish wake word back is still on the roadmap.

The integration was a five-bug war: the same model scored 0.996 in Python and 0% on the device. The worst one was invisible: the CNN's internal state wasn't reset between armings, and after closing a session the model fired at 99% with no sound at all —an infinite loop of detection, session and close—. The most expensive to find was the aliasing when decimating from 48 to 16 kHz without a filter: recordings and TTS kept being detected at 97–99% because they were already band-limited to 8 kHz, but live voice dropped to zero. A 19-tap low-pass FIR before decimation fixed it.

Recall and precision pull the same threshold in opposite directions, so they live in different layers: the board keeps the threshold lax so no activation is missed, and the agent re-verifies each trigger by transcribing the pre-roll —if it wasn't the wake word, it aborts silently—. Every rejection saves a clip: the hard-negatives dataset builds itself with use.

Firmware in Zig, with a trick

The firmware's application layer is written in Zig on ESP-IDF, with Espressif's Xtensa fork; the WebRTC core stays in C (the LiveKit SDK is not rewritten). Zig's translate-c chokes on ESP-IDF's newlib headers, so the bindings are written by hand, declaring only what is used. The obvious risk —layouts drifting out of sync— is paid for with a file of _Static_assert checks that verify them on every build:

#define CHECK_SIZE(type, bytes) _Static_assert(sizeof(type) == (bytes), #type " size drifted — sync csdk.zig")
#define CHECK_OFF(type, field, bytes) _Static_assert(offsetof(type, field) == (bytes), #type "." #field " offset drifted — sync csdk.zig")
...
CHECK_SIZE(livekit_pub_options_t, 36);
CHECK_OFF(livekit_pub_options_t, kind, 0);
CHECK_OFF(livekit_pub_options_t, video_encode, 4);
CHECK_OFF(livekit_pub_options_t, audio_encode, 20);
CHECK_OFF(livekit_pub_options_t, capturer, 32);

My favorite trick: the Zig code itself flashes the XVF3800 over DFU via I2C on first boot, installing the firmware that turns it into the I2S master. The binary —868 KB— is embedded in the executable with @embedFile; the factory image stays intact as a safety net, so there's no way to brick it, and subsequent boots detect the version and skip it. No ESPHome, no external tools.

The S3 is extremely tight on internal RAM, and that dictates the house rule: the firmware is frozen and new features go to the server. Adding a microphone to another room means flashing another unit against the same server, with zero firmware changes.

The other end of the cable

The agent is Python on LiveKit Agents, with a native speech-to-speech model: Gemini with native audio by default and gpt-realtime-mini as backup, switchable by environment variable. A local VAD (Silero) detects when you talk over the agent to cut it off instantly, and the house lights are controlled via Home Assistant's MCP. Session ending is decided by the model's intent, not a keyword list: “turn the light off in the living room” must not hang up the call.

Before choosing a model I ran the numbers, and the cost matrix in the repo leads to a counterintuitive conclusion: the classic STT + LLM + TTS pipeline is not the cheap route —TTS dominates the bill— and native speech-to-speech models set the price floor: in moderate use, about $10 a month versus the pipeline's $15–26, and with lower voice-to-voice latency. That analysis backs the Gemini default.

The control plane

Behind the gadget there's a service written in Go, contract-first: the openapi.yaml is the source, and types and router are generated from it. Its main endpoint, POST /v1/sessions, authenticates the device, creates the explicit dispatch of the agent in LiveKit and returns a short-lived credential; the firmware carries no secrets and no static JWTs. And every session gets a fresh room: the bug of two agents sharing a room and answering each other is impossible by construction.

PostgreSQL is the source of truth, and every change that must be published is recorded in the same transaction as an outbox event; a worker delivers them as durable CloudEvents on NATS JetStream to the React admin panel, where the recordings catalog lives. LiveKit's SFU runs self-hosted on my Talos cluster, deployed via GitOps: audio local to the LAN and zero euros per room-minute.

The whole system is observed from the same place. The agent exports metrics and the transcription of every turn, with its latencies, to a Grafana stack via OpenTelemetry. The board has two paths: on the bench, a bridge reads the serial port and turns its logs into metrics; in production, the firmware mirrors those same logs over syslog UDP onto the LAN, fire-and-forget, and the vitals —heap, mic level, wake probabilities— reach Grafana with no cable. The two halves of the conversation, side by side.

Up above is the production dashboard, with the serial port unplugged: 146 detections, 146 sessions, zero panics and zero restarts in ten days. This telemetry was what uncovered, among other things, a storm of SCTP retransmissions —hundreds of INITs per session— that ended up dead with a one-line patch and 7 KB of heap recovered per session.

The door between the bench and the living room

The repo has the door written down that separates “it works on the bench” from “it lives in the living room”: the living-room-ready milestone. It isn't a feature list; they're exit criteria, each with its test. The big ones:

Authentication and rate-limiting on the exposed HTTP surface: today the legacy token endpoint answers any anonymous request and fires up a paid model.
OTA with rollback and flash encryption: today a deployed device can only be updated and recovered over USB.
Opt-in session recording, with clear retention and consent — today it's on by default, and I'm saying it as it is.
The TV hijacks my sessions: with the mic open in duplex, its audio gets transcribed as user turns and the adaptive beam points at it. The fix comes in layers: lock the beam to the wake direction, filter turns by direction and per-turn speaker verification.

There was a fifth criterion —cable-free telemetry, because the firmware's vitals only came out through the serial port— that has just been crossed off with the syslog mirror: it's the dashboard above. For the rest, the definition of done: a week unattended in the living room, with every failure reconstructable from Grafana without touching the gadget.

Sebastian is the spearhead of the same line of work as the Zetesis agents portal: agents configured as data, with budgets and a model gateway… and now a speaker that talks. The experiment is still open, and so is the repo: github.com/Zetesis-Labs/Sebastian. When it crosses the living-room door, I'll tell the story here.

Zetesis-Labs/Sebastian

Bidirectional voice speaker on ReSpeaker XVF3800 and XIAO ESP32-S3, with Zig firmware and a Python agent.