Open Brain
arXivPaperNeeds Review

Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models

Spool removes expired video frames from hybrid VLM recurrent state without replaying the window through the backbone. Across 20 model-benchmark tests, it cut p95 time-to-first-token by a geometric mean of 6.6× versus the faster vLLM baseline.

arXiv · Nov 17, 2026
Open Source Open MarkdownOpen JSON
Source Summary

Spool caches per-frame state transformations and composes them as a video window advances, avoiding a full replay when old frames expire. In **20 tests** across five open-source hybrid VLMs and four video reasoning benchmarks, it reduced p95 time-to-first-token by a geometric mean of **6.6×** versus the faster of two recent vLLM baselines; average accuracy also rose **2.3 percentage points**.

Practical Implication

Builders working on streaming video agents could consider this approach when they need bounded frame history and faster answers. The method is training-free and uses a dedicated data structure and GPU kernels.

Agent-Ready Context
Spool caches per-frame state transformations and composes them as a video window advances, avoiding a full replay when old frames expire. In **20 tests** across five open-source hybrid VLMs and four video reasoning benchmarks, it reduced p95 time-to-first-token by a geometric mean of **6.6×** versus the faster of two recent vLLM baselines; average accuracy also rose **2.3 percentage points**.

Builders working on streaming video agents could consider this approach when they need bounded frame history and faster answers. The method is training-free and uses a dedicated data structure and GPU kernels.

Frame removal is exact within each recurrent layer, but expired frames can still influence later layers through cached inputs. The reported results cover the tested models and benchmarks; the material does not establish performance for other workloads.
Connected Context · Feed7 Judgment

Spool adds a concrete streaming-inference result to the efficiency discussion: caching frame-state transformations can cut p95 time to first token while slightly improving average accuracy on the tested hybrid VLMs. Its exactness is limited to removal within each recurrent layer, so the result supports workload-specific measurement rather than a general claim that cached state is equivalent to full replay.

Context Map
infravideoresearch#agent-reliability
Uncertainty
Frame removal is exact within each recurrent layer, but expired frames can still influence later layers through cached inputs. The reported results cover the tested models and benchmarks; the material does not establish performance for other workloads.