AI EngineerVideoSource Linked

Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer

Two rare vLLM failures produced confident gibberish without errors; cross-engine log-prob comparisons, constrained memory, and request IDs exposed scheduler and cache-index bugs.

AI Engineer · Sep 19, 2026

Open Source Open Markdown Open JSON

Source Summary

AI21 saw gibberish roughly **once per thousand prompts**, only in vLLM and under load. One bug scheduled decode before prefill for a fresh Mamba request, reusing stale state; another used a **32-bit cache index** that wrapped instead of raising an error.

Practical Implication

Build a deterministic log-prob comparison against a simpler inference engine, then vary memory pressure and rollout count to change the failure’s timing or location. Carry request identity into the model forward path so a bad output can be tied to scheduler and kernel metadata.

Agent-Ready Context

AI21 saw gibberish roughly **once per thousand prompts**, only in vLLM and under load. One bug scheduled decode before prefill for a fresh Mamba request, reusing stale state; another used a **32-bit cache index** that wrapped instead of raising an error.

Build a deterministic log-prob comparison against a simpler inference engine, then vary memory pressure and rollout count to change the failure’s timing or location. Carry request identity into the model forward path so a bad output can be tied to scheduler and kernel metadata.

Neither defect crashed the process, and ordinary quality checks could therefore misclassify corrupted inference as model behavior. The fixes were specific—a prefill guard and changing the index to size_t—but the broader debugging method matters most for stateful runtimes.

Connected Context · Feed7 Judgment

This adds inference-runtime corruption as a prerequisite check for trustworthy agent evaluation. Rare, load-dependent scheduler and integer-overflow bugs can emit plausible failures without crashes, so benchmark scores may measure serving defects as model behavior. Differential log-prob tests against a simpler engine, load variation, and request-linked runtime metadata complement final-state and constraint grading by first establishing that the evaluated model execution is sound.

Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code ModelsThe placebo-controlled study separates genuine error-conditioned repair from retry effects; this signal adds a prior control—verify that rare runtime corruption did not create or alter the failures being repaired.SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering AgentsSWE-Gate shows that passing functional tests can hide constraint violations, while this signal shows ordinary quality checks can also hide corrupted inference; both require independent acceptance signals beyond headline success.When LLM Decompilers Recompile More and Preserve LessBoth advocate differential comparison to a reference implementation because outputs can pass conventional checks while silently diverging: original binaries for decompilation and a simpler inference engine for serving.When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AIThe benchmark critique warns that opaque test conditions weaken rankings; load-sensitive serving bugs make runtime, scheduler, and kernel conditions another necessary part of an inspectable evaluation setup.

Context Map

benchmarkcoding#agent-evals#agent-reliability#benchmark-integrity

Uncertainty

Neither defect crashed the process, and ordinary quality checks could therefore misclassify corrupted inference as model behavior. The fixes were specific—a prefill guard and changing the index to size_t—but the broader debugging method matters most for stateful runtimes.