Context Caching

Agentic inference makes KV-cache locality and prefill pressure first-class scheduling concerns. Red Hat's case study improved latency and capacity, but P/D separation needs suitable traffic and fast networking.

Prefix Sliding drops stale reasoning tokens while retaining instructions and recent work, capping memory during long inference and reportedly making existing models 3× faster without training.

LFU was the strongest simple semantic-cache default, but answer validity mattered far more than eviction policy. Raw hit rates above 50% fell near 2% after quality adjustment.

In this AI-tutor evaluation, keeping full history beat compaction on recall and cost because cached tokens were cheap. Measure your constraint before shortening agent context.

Blast Radius predicts which code and context a prompt will reach, then reversibly archives unused history. Across seven OpenAI models, it cut token consumption by 17–26%.

Model routing can cut agent costs only when delegation, escalation, and context transfer are designed together; duplicating context across models can erase the savings.

TokTier targets a hidden agent-serving bottleneck: repeatedly tokenizing long transcripts after small tool results. Its stateful repair cut median vLLM time to first token by 16–34% in reported tests.

ReToken adds one learned retrieval embedding to select relevant visual tokens from a cached context, improving long-image and video benchmarks while fitting training and inference on one H100.

Two API settings—reasoning retention and compaction—reportedly tripled GPT-5.6’s ARC-AGI-3 score. Agent evals should treat runtime configuration as part of the tested system.

A delayed KV-eviction policy helps when memory reuse is sharp and observable, but loses its edge on natural-text benchmarks—a useful warning for agent-memory claims.

Caveman compresses agent inputs and outputs, with local tooling for noisy commands, browsing, memory, and skills. Its own caveat matters: terse workloads can cost more once the skill prompt is included.

Reasonix is a configurable terminal coding agent built to preserve DeepSeek prefix-cache reuse across long sessions, with optional planner/executor models and subprocess-based plugins.

oMLX serves local models to coding agents on Apple Silicon, combining continuous batching, persistent SSD-backed KV caching, and menu-bar control behind compatible APIs.

OmniRoute puts many model providers behind one OpenAI-compatible endpoint, with routing, quota failover, cost telemetry, compression, and coding-agent setup. Its breadth raises operational and trust questions.