# How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit

Source: [AI Engineer](https://www.youtube.com/watch?v=OmyqelTwpOo)  
Feed7 permalink: https://feed7.dev/p/how-to-build-voice-agents-that-don-t-fall-apart-when-you-interrupt-jesse-0kyz1xt  
Published: 2026-10-26T07:00:26.000Z  
Trust: Source Linked (source_linked)

## Why Included

Voice agents need to be evaluated as full conversations, including interruptions, noisy audio, and backend work. Builders should measure the caller’s wait for audible speech and turn simulations into regression cases.

## Source Summary

The talk argues that model leaderboards score components, while voice agents ship conversations. The response window from a caller’s last syllable to the agent’s first audible reply is about **1.5 seconds**; **500–600 ms** feels more natural. A TTS provider’s first byte can arrive well before audio begins.

## Practical Implication

Test a complete agent with simulated callers who interrupt, change goals, mumble, and encounter background noise. Measure time to first audio and first sentence, confirm critical dates and amounts, and keep listening while slow tools run. Turn observed failures into repeatable regression scenarios.

## Agent-Ready Context

The talk argues that model leaderboards score components, while voice agents ship conversations. The response window from a caller’s last syllable to the agent’s first audible reply is about **1.5 seconds**; **500–600 ms** feels more natural. A TTS provider’s first byte can arrive well before audio begins.

Test a complete agent with simulated callers who interrupt, change goals, mumble, and encounter background noise. Measure time to first audio and first sentence, confirm critical dates and amounts, and keep listening while slow tools run. Turn observed failures into repeatable regression scenarios.

The reported Gemma 4 31B result applies to the speaker’s hotel agent; the same model ran slower through another provider. The talk’s open-source simulations can help compare stacks for a specific workflow, but the material provides no universal winning model or benchmark score.

## Connected Context

Feed7 judgment across 928 accumulated Signals:

This talk shifts voice-agent evaluation from component scores to complete conversations, emphasizing response-to-audio latency, interruptions, changing goals, and verified details. Its simulations offer workflow-specific regression scenarios, while the supplied evaluation candidates show that simulated tests can be useful but do not replace evidence about transfer to other environments or deployed stacks.

- [Simulated Control Evals](https://feed7.dev/p/2610-31593v1-1lanp1p) — Both use simulated interactions to evaluate agent behavior; Simulated Control Evals adds evidence that such testing can forecast deployment-specific safety, with transfer still unresolved.
- [5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo](https://feed7.dev/p/5-voice-agent-failure-modes-you-ll-hit-in-week-one-venky-b-plivo-0xcwuby) — Plivo’s failure modes add typed input validation and speech normalization to the target’s end-to-end checks for interruptions, latency, and critical values.
- [Spoken Function Calling: A New Perspective on Spoken Language Understanding for Large Audio Language Models](https://feed7.dev/p/2608-05126v1-0gpey5d) — Spoken Function Calling focuses on extracting structured tool calls from speech, while the target evaluates the surrounding conversation and timing needed for a complete voice agent.
- [EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence](https://feed7.dev/p/2610-31650v1-1wmxt69) — EpiSEM-Bench separates evidence seeking from safe action; the target’s checks on dates and amounts can expose consequential details that a voice agent must confirm.

## Context Map

- Layer: benchmark
- Domains: audio
- Topics: agent-evals, agent-reliability, tool-use

## Uncertainty

- The reported Gemma 4 31B result applies to the speaker’s hotel agent; the same model ran slower through another provider. The talk’s open-source simulations can help compare stacks for a specific workflow, but the material provides no universal winning model or benchmark score.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.