# Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders

Source: [arXiv](https://arxiv.org/abs/2608.20280v1)  
Feed7 permalink: https://feed7.dev/p/2608-20280v1-1ygnhyz  
Published: 2026-08-20T17:14:19.000Z  
Trust: Needs Review (needs_review)

## Why Included

LFU was the strongest simple semantic-cache default, but answer validity mattered far more than eviction policy. Raw hit rates above 50% fell near 2% after quality adjustment.

## Source Summary

Seven eviction policies were tested across three query corpora, three capacities, and two encoders. None beat LFU by more than **0.041 percentage points** in any of the 18 settings, while FIFO and streaming SISO trailed it by up to 8.67 and 8.55 points.

## Practical Implication

For an agent response cache, validate whether a matched answer can actually substitute for a new response before tuning replacement logic. Use LFU as the simple baseline, exact search for comparisons, and recalibrate thresholds for each encoder.

## Agent-Ready Context

Seven eviction policies were tested across three query corpora, three capacities, and two encoders. None beat LFU by more than **0.041 percentage points** in any of the 18 settings, while FIFO and streaming SISO trailed it by up to 8.67 and 8.55 points.

For an agent response cache, validate whether a matched answer can actually substitute for a new response before tuning replacement logic. Use LFU as the simple baseline, exact search for comparisons, and recalibrate thresholds for each encoder.

At MiniLM’s median threshold, only **2.1–3.9%** of sampled hits were answer-substitutable. Raw hit rates of **51–60%** became quality-adjusted rates of **1.1–2.2%**, and thresholds did not transfer between embedding models.

## Connected Context

Feed7 judgment across 545 accumulated Signals:

This narrows the cache-engineering agenda established by the prior candidates: for response caches, semantic substitutability and encoder-specific threshold calibration matter far more than sophisticated eviction, with LFU an adequate baseline. It also distinguishes response reuse from prompt or token caching, where preserving history or incremental tokenization can improve serving without asserting that an old answer is valid for a new query.

- [Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI](https://feed7.dev/p/context-engineering-in-2026-louis-francois-bouchard-omar-solano-samridhi-1dnlyr0) — Both argue for measuring the actual workload before optimizing context, but this study concerns reuse of prior answers and shows that nominal cache hits may rarely be valid substitutes, whereas the tutor result supports retaining fully cached history.
- [TokTier: Exact Stateful Tokenization for Agentic LLM Serving](https://feed7.dev/p/2607-29678v1-1fuv1qw) — TokTier shows that prompt-cache hits do not remove tokenization cost; this study adds that response-cache hits do not guarantee answer reuse. Together they require separate telemetry for serving efficiency and semantic validity.
- [Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo](https://feed7.dev/p/inside-847-production-clinical-ai-notes-sebastian-fox-composo-0lrc8td) — The clinical-note evidence reinforces the study’s warning that plausible similarity is not enough: consequential omissions can escape generic judgment, making answer-substitutability validation especially important in high-cost domains.

## Context Map

- Layer: infra
- Domains: data
- Topics: context-caching, retrieval

## Uncertainty

- At MiniLM’s median threshold, only **2.1–3.9%** of sampled hits were answer-substitutable. Raw hit rates of **51–60%** became quality-adjusted rates of **1.1–2.2%**, and thresholds did not transfer between embedding models.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.