Context Caching

Cursor cut agent token cost by shrinking static instructions, loading tools on demand, improving cache boundaries, and using subagents more selectively.

GPT-6 adds cache diagnostics, explicit breakpoints, and controls aimed at improving prompt-cache hit rates, giving agent builders more levers over repeated-context latency and cost.

Flash-dLLM combines an I/O-aware KV-cache kernel with self-drafting parallel decoding, reporting large inference speedups without retraining or an auxiliary model.

CliffCompaction cuts long-horizon agent costs by dropping or truncating original context instead of rewriting summaries, aiming to prevent drift across repeated compactions.

Agent-serving systems should optimize whole-task latency, preserving shared prefixes across tool calls and routing later steps toward the GPU that already holds their KV cache.

Inference gains increasingly depend on training-aware techniques: weight quantization, KV-cache management, and speculative decoding can matter more than swapping serving code.

Agent requests often repeat most of their input, making KV-cache-aware routing a direct latency and cost lever for hosted coding-agent workloads.

A low-latency voice harness can pair fast provisional work with slower corrections. The design speculates on transcripts and responses, runs tools in parallel, and reuses cached speech prefixes.

Long contexts can exhaust GPU memory before model weights do. Capacity planning for agent workloads must include KV cache, concurrency, and serving-engine behavior.

Agentic inference makes KV-cache locality and prefill pressure first-class scheduling concerns. Red Hat's case study improved latency and capacity, but P/D separation needs suitable traffic and fast networking.

Prefix Sliding drops stale reasoning tokens while retaining instructions and recent work, capping memory during long inference and reportedly making existing models 3× faster without training.

LFU was the strongest simple semantic-cache default, but answer validity mattered far more than eviction policy. Raw hit rates above 50% fell near 2% after quality adjustment.

In this AI-tutor evaluation, keeping full history beat compaction on recall and cost because cached tokens were cheap. Measure your constraint before shortening agent context.

Blast Radius predicts which code and context a prompt will reach, then reversibly archives unused history. Across seven OpenAI models, it cut token consumption by 17–26%.

Model routing can cut agent costs only when delegation, escalation, and context transfer are designed together; duplicating context across models can erase the savings.

TokTier targets a hidden agent-serving bottleneck: repeatedly tokenizing long transcripts after small tool results. Its stateful repair cut median vLLM time to first token by 16–34% in reported tests.

ReToken adds one learned retrieval embedding to select relevant visual tokens from a cached context, improving long-image and video benchmarks while fitting training and inference on one H100.

Two API settings—reasoning retention and compaction—reportedly tripled GPT-5.6’s ARC-AGI-3 score. Agent evals should treat runtime configuration as part of the tested system.

A delayed KV-eviction policy helps when memory reuse is sharp and observable, but loses its edge on natural-text benchmarks—a useful warning for agent-memory claims.

OmniRoute puts many model providers behind one OpenAI-compatible endpoint, with routing, quota failover, cost telemetry, compression, and coding-agent setup. Its breadth raises operational and trust questions.

Reasonix is a configurable terminal coding agent built to preserve DeepSeek prefix-cache reuse across long sessions, with optional planner/executor models and subprocess-based plugins.

Caveman compresses agent inputs and outputs, with local tooling for noisy commands, browsing, memory, and skills. Its own caveat matters: terse workloads can cost more once the skill prompt is included.

oMLX serves local models to coding agents on Apple Silicon, combining continuous batching, persistent SSD-backed KV caching, and menu-bar control behind compatible APIs.

Context Mode keeps bulky tool output outside an agent’s prompt and restores relevant session state after compaction. Its MCP tools and hooks support 17 clients, though enforcement varies by platform.