Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI
A low-latency voice harness can pair fast provisional work with slower corrections. The design speculates on transcripts and responses, runs tools in parallel, and reuses cached speech prefixes.
EliseAI’s cascaded voice stack separates transcription, language-model planning, and speech output. A fast streaming transcript is corrected by **Scribe V2**, background agents inject tool results into the main context, and a **prefix cache** can start familiar speech before fresh audio arrives.
For latency-sensitive agents, speculate only where work can be canceled or replaced safely. Start generation on partial input, parallelize predictable tool calls, and preserve full text for the speech provider so cached and newly generated audio retain consistent prosody.
EliseAI’s cascaded voice stack separates transcription, language-model planning, and speech output. A fast streaming transcript is corrected by **Scribe V2**, background agents inject tool results into the main context, and a **prefix cache** can start familiar speech before fresh audio arrives. For latency-sensitive agents, speculate only where work can be canceled or replaced safely. Start generation on partial input, parallelize predictable tool calls, and preserve full text for the speech provider so cached and newly generated audio retain consistent prosody. The demo shows one appointment-booking call but provides no latency, error-rate, or cache-hit measurements. Prefix splicing may introduce a small audible discontinuity, and premature generations add compute even when users never hear them.
This makes latency reduction an orchestration problem: stream and correct transcripts, start cancelable work early, parallelize predictable calls, and cache speech prefixes without discarding full-text prosody. It extends voice reliability guidance with concrete speculative techniques, but the single demo and absent measurements leave their net latency, error, compute, and audible-splice tradeoffs unquantified.