# Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer

Source: [AI Engineer](https://www.youtube.com/watch?v=btxG75rNJC4)  
Feed7 permalink: https://feed7.dev/p/two-bugs-that-hid-in-plain-sight-a-vllm-debugging-detective-story-asaf-g-1s5yaqd  
Published: 2026-09-19T18:30:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

Two rare vLLM failures produced confident gibberish without errors; cross-engine log-prob comparisons, constrained memory, and request IDs exposed scheduler and cache-index bugs.

## Source Summary

AI21 saw gibberish roughly **once per thousand prompts**, only in vLLM and under load. One bug scheduled decode before prefill for a fresh Mamba request, reusing stale state; another used a **32-bit cache index** that wrapped instead of raising an error.

## Practical Implication

Build a deterministic log-prob comparison against a simpler inference engine, then vary memory pressure and rollout count to change the failure’s timing or location. Carry request identity into the model forward path so a bad output can be tied to scheduler and kernel metadata.

## Agent-Ready Context

AI21 saw gibberish roughly **once per thousand prompts**, only in vLLM and under load. One bug scheduled decode before prefill for a fresh Mamba request, reusing stale state; another used a **32-bit cache index** that wrapped instead of raising an error.

Build a deterministic log-prob comparison against a simpler inference engine, then vary memory pressure and rollout count to change the failure’s timing or location. Carry request identity into the model forward path so a bad output can be tied to scheduler and kernel metadata.

Neither defect crashed the process, and ordinary quality checks could therefore misclassify corrupted inference as model behavior. The fixes were specific—a prefill guard and changing the index to size_t—but the broader debugging method matters most for stateful runtimes.

## Connected Context

Feed7 judgment across 823 accumulated Signals:

This adds inference-runtime corruption as a prerequisite check for trustworthy agent evaluation. Rare, load-dependent scheduler and integer-overflow bugs can emit plausible failures without crashes, so benchmark scores may measure serving defects as model behavior. Differential log-prob tests against a simpler engine, load variation, and request-linked runtime metadata complement final-state and constraint grading by first establishing that the evaluated model execution is sound.

- [Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models](https://feed7.dev/p/2607-12962v1-0q4i26c) — The placebo-controlled study separates genuine error-conditioned repair from retry effects; this signal adds a prior control—verify that rare runtime corruption did not create or alter the failures being repaired.
- [SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents](https://feed7.dev/p/2609-04167v1-1vvofgw) — SWE-Gate shows that passing functional tests can hide constraint violations, while this signal shows ordinary quality checks can also hide corrupted inference; both require independent acceptance signals beyond headline success.
- [When LLM Decompilers Recompile More and Preserve Less](https://feed7.dev/p/2609-05370v1-03wuqdv) — Both advocate differential comparison to a reference implementation because outputs can pass conventional checks while silently diverging: original binaries for decompilation and a simpler inference engine for serving.
- [When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI](https://feed7.dev/p/when-will-the-benchmaxxing-plague-end-nick-heiner-surge-ai-178gqcg) — The benchmark critique warns that opaque test conditions weaken rankings; load-sensitive serving bugs make runtime, scheduler, and kernel conditions another necessary part of an inspectable evaluation setup.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, agent-reliability, benchmark-integrity

## Uncertainty

- Neither defect crashed the process, and ordinary quality checks could therefore misclassify corrupted inference as model behavior. The fixes were specific—a prefill guard and changing the index to size_t—but the broader debugging method matters most for stateful runtimes.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.