# Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

Source: [AI Engineer](https://www.youtube.com/watch?v=MBHOH1NmDqc)  
Feed7 permalink: https://feed7.dev/p/realtime-voice-agents-with-frontier-intelligence-bohan-li-eliseai-0u0q1mh  
Published: 2026-09-15T16:00:02.000Z  
Trust: Source Linked (source_linked)

## Why Included

A low-latency voice harness can pair fast provisional work with slower corrections. The design speculates on transcripts and responses, runs tools in parallel, and reuses cached speech prefixes.

## Source Summary

EliseAI’s cascaded voice stack separates transcription, language-model planning, and speech output. A fast streaming transcript is corrected by **Scribe V2**, background agents inject tool results into the main context, and a **prefix cache** can start familiar speech before fresh audio arrives.

## Practical Implication

For latency-sensitive agents, speculate only where work can be canceled or replaced safely. Start generation on partial input, parallelize predictable tool calls, and preserve full text for the speech provider so cached and newly generated audio retain consistent prosody.

## Agent-Ready Context

EliseAI’s cascaded voice stack separates transcription, language-model planning, and speech output. A fast streaming transcript is corrected by **Scribe V2**, background agents inject tool results into the main context, and a **prefix cache** can start familiar speech before fresh audio arrives.

For latency-sensitive agents, speculate only where work can be canceled or replaced safely. Start generation on partial input, parallelize predictable tool calls, and preserve full text for the speech provider so cached and newly generated audio retain consistent prosody.

The demo shows one appointment-booking call but provides no latency, error-rate, or cache-hit measurements. Prefix splicing may introduce a small audible discontinuity, and premature generations add compute even when users never hear them.

## Connected Context

Feed7 judgment across 807 accumulated Signals:

This makes latency reduction an orchestration problem: stream and correct transcripts, start cancelable work early, parallelize predictable calls, and cache speech prefixes without discarding full-text prosody. It extends voice reliability guidance with concrete speculative techniques, but the single demo and absent measurements leave their net latency, error, compute, and audible-splice tradeoffs unquantified.

- [5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo](https://feed7.dev/p/5-voice-agent-failure-modes-you-ll-hit-in-week-one-venky-b-plivo-0xcwuby) — The speculative pipeline addresses the same reasoning-latency pressure, while Plivo’s validation and pronunciation guidance covers accuracy and output-quality risks that early generation can amplify.
- [AI Agents Are Just Distributed Systems Now — Salman Munaf, TikTok](https://feed7.dev/p/ai-agents-are-just-distributed-systems-now-salman-munaf-tiktok-1v4yc47) — Speculative tool calls introduce timeout, cancellation, and unknown-outcome hazards; idempotency, bounded retries, durable traces, and reconciliation are therefore prerequisites when early work can affect external state.
- [An Empirical Study of Harness Design for Coding Agents](https://feed7.dev/p/2609-20804v1-17ecr62) — Its finding that harness benefits vary by model and budget reinforces the need to measure prefix caching, parallel calls, and speculative generation rather than assume each added mechanism pays off.
- [In Code They Act, In Proof We Trust — Erik Meijer, Leibniz Labs](https://feed7.dev/p/in-code-they-act-in-proof-we-trust-erik-meijer-leibniz-labs-1lyno2y) — The proof-gated execution model sharpens the Signal’s safety boundary: speculative planning or generation may start early, but consequential side effects need inspectable validation before execution.

## Context Map

- Layer: agent
- Domains: audio
- Topics: harness-engineering, tool-use, context-caching

## Uncertainty

- The demo shows one appointment-booking call but provides no latency, error-rate, or cache-hit measurements. Prefix splicing may introduce a small audible discontinuity, and premature generations add compute even when users never hear them.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.