Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer
Two rare vLLM failures produced confident gibberish without errors; cross-engine log-prob comparisons, constrained memory, and request IDs exposed scheduler and cache-index bugs.
AI21 saw gibberish roughly **once per thousand prompts**, only in vLLM and under load. One bug scheduled decode before prefill for a fresh Mamba request, reusing stale state; another used a **32-bit cache index** that wrapped instead of raising an error.
Build a deterministic log-prob comparison against a simpler inference engine, then vary memory pressure and rollout count to change the failure’s timing or location. Carry request identity into the model forward path so a bad output can be tied to scheduler and kernel metadata.
AI21 saw gibberish roughly **once per thousand prompts**, only in vLLM and under load. One bug scheduled decode before prefill for a fresh Mamba request, reusing stale state; another used a **32-bit cache index** that wrapped instead of raising an error. Build a deterministic log-prob comparison against a simpler inference engine, then vary memory pressure and rollout count to change the failure’s timing or location. Carry request identity into the model forward path so a bad output can be tied to scheduler and kernel metadata. Neither defect crashed the process, and ordinary quality checks could therefore misclassify corrupted inference as model behavior. The fixes were specific—a prefill guard and changing the index to size_t—but the broader debugging method matters most for stateful runtimes.
The debugging case shows that an evaluation stack can silently corrupt model output: rare scheduler-state and integer-width failures may look like ordinary model variance while passing crash-based health checks. It therefore adds differential log-prob testing, load-sensitive reproduction, and request-level runtime tracing as prerequisites for trustworthy agent benchmarks and production evaluations.