How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit
Voice agents need to be evaluated as full conversations, including interruptions, noisy audio, and backend work. Builders should measure the caller’s wait for audible speech and turn simulations into regression cases.
The talk argues that model leaderboards score components, while voice agents ship conversations. The response window from a caller’s last syllable to the agent’s first audible reply is about **1.5 seconds**; **500–600 ms** feels more natural. A TTS provider’s first byte can arrive well before audio begins.
Test a complete agent with simulated callers who interrupt, change goals, mumble, and encounter background noise. Measure time to first audio and first sentence, confirm critical dates and amounts, and keep listening while slow tools run. Turn observed failures into repeatable regression scenarios.
The talk argues that model leaderboards score components, while voice agents ship conversations. The response window from a caller’s last syllable to the agent’s first audible reply is about **1.5 seconds**; **500–600 ms** feels more natural. A TTS provider’s first byte can arrive well before audio begins. Test a complete agent with simulated callers who interrupt, change goals, mumble, and encounter background noise. Measure time to first audio and first sentence, confirm critical dates and amounts, and keep listening while slow tools run. Turn observed failures into repeatable regression scenarios. The reported Gemma 4 31B result applies to the speaker’s hotel agent; the same model ran slower through another provider. The talk’s open-source simulations can help compare stacks for a specific workflow, but the material provides no universal winning model or benchmark score.
This talk shifts voice-agent evaluation from component scores to complete conversations, emphasizing response-to-audio latency, interruptions, changing goals, and verified details. Its simulations offer workflow-specific regression scenarios, while the supplied evaluation candidates show that simulated tests can be useful but do not replace evidence about transfer to other environments or deployed stacks.