Skip to content

Voice agents & barge-in

A conversational agent is: speech-to-text → LLM → this API → the caller's speaker, with the human allowed to interrupt. The TTS leg should add tens of milliseconds, not hundreds, and must stop the instant the caller starts talking.

Shape of the loop

  1. Open one WS /v1/tts/stream per call when the call starts (or when the first reply is about to be spoken). Keep it open.
  2. As the LLM streams tokens, cut at sentence boundaries (., ?, !, ।) and send each sentence as its own message with a request_id like turn-12-s0, turn-12-s1.
  3. Forward the binary audio to the telephony leg as it arrives (resample 24 kHz → 8 kHz if needed). Start playing after ~100 ms of buffer.
  4. On first_audio_ms, record TTFA for your dashboards; on end, record rtf and audio_s.
  5. On barge-in (your STT's speech-start event), close the socket, flush your playout buffer, discard any audio tagged with the interrupted turn, and open a new socket for the next turn.

Latency budget

Stage Typical
Engine, warm (cached prefill) 18 ms
Engine, cold chunk 55–80 ms
Server + Mumbai network +10–30 ms
Your playout buffer 100–200 ms (your choice)

Repeated system lines (greetings, "please hold", disclaimers) hit the prefill cache after the first play and start in under 20 ms of engine time. Warm them once at deploy time by synthesising each line.

Capacity planning

One worker per vCPU, one stream per worker, RTF ≈ 0.14–0.19 per stream. A 4-vCPU node sustains four concurrent streams with headroom (12.4 audio-seconds per wall-second at four streams). Because an agent speaks perhaps 40 % of a call, four streams serve ~10 concurrent calls comfortably. Capacity grows by adding nodes behind the load balancer; when every worker is busy you get 503 / error status 503 immediately rather than a queue that grows your latency (details).

Telephony notes

  • 8 kHz μ-law/A-law: resample the pcm16 on your media server (ffmpeg -f s16le -ar 24000 -ac 1 -i - -ar 8000 -acodec pcm_mulaw), or use opus where supported (WebRTC, SIP with Opus).
  • WebRTC/browser: play pcm16 through Web Audio (quickstart snippet) or request opus and decode with the platform decoder.
  • Pipecat / LiveKit style pipelines: implement a TTS service that opens our WebSocket once per session and yields audio frames; the end event marks the utterance boundary your turn-taking logic needs.

Correlating logs

Give every message a request_id; it comes back on every event and is stored in your usage log (visible under Usage → Recent in the portal, with ttfa_ms, rtf, queue_ms). Quote it when you talk to us.