{
  "schema_version": "1.1",
  "id": "archive:https://arxiv.org/abs/2608.20280v1",
  "slug": "2608-20280v1-1ygnhyz",
  "url": "https://feed7.dev/p/2608-20280v1-1ygnhyz",
  "title": "Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders",
  "why_included": "LFU was the strongest simple semantic-cache default, but answer validity mattered far more than eviction policy. Raw hit rates above 50% fell near 2% after quality adjustment.",
  "summary": "Seven eviction policies were tested across three query corpora, three capacities, and two encoders. None beat LFU by more than **0.041 percentage points** in any of the 18 settings, while FIFO and streaming SISO trailed it by up to 8.67 and 8.55 points.",
  "practical_implication": "For an agent response cache, validate whether a matched answer can actually substitute for a new response before tuning replacement logic. Use LFU as the simple baseline, exact search for comparisons, and recalibrate thresholds for each encoder.",
  "agent_context": "Seven eviction policies were tested across three query corpora, three capacities, and two encoders. None beat LFU by more than **0.041 percentage points** in any of the 18 settings, while FIFO and streaming SISO trailed it by up to 8.67 and 8.55 points.\n\nFor an agent response cache, validate whether a matched answer can actually substitute for a new response before tuning replacement logic. Use LFU as the simple baseline, exact search for comparisons, and recalibrate thresholds for each encoder.\n\nAt MiniLM’s median threshold, only **2.1–3.9%** of sampled hits were answer-substitutable. Raw hit rates of **51–60%** became quality-adjusted rates of **1.1–2.2%**, and thresholds did not transfer between embedding models.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2608.20280v1",
    "published_at": "2026-08-20T17:14:19.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "infra",
  "domains": [
    "data"
  ],
  "topics": [
    "context-caching",
    "retrieval"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "At MiniLM’s median threshold, only **2.1–3.9%** of sampled hits were answer-substitutable. Raw hit rates of **51–60%** became quality-adjusted rates of **1.1–2.2%**, and thresholds did not transfer between embedding models."
  ],
  "connected_context": {
    "meaning": "This narrows the cache-engineering agenda established by the prior candidates: for response caches, semantic substitutability and encoder-specific threshold calibration matter far more than sophisticated eviction, with LFU an adequate baseline. It also distinguishes response reuse from prompt or token caching, where preserving history or incremental tokenization can improve serving without asserting that an old answer is valid for a new query.",
    "corpus_size": 545,
    "generated_at": "2026-08-23T18:05:12.902Z",
    "connections": [
      {
        "title": "Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=WP3hjUXd918",
        "feed7_url": "https://feed7.dev/p/context-engineering-in-2026-louis-francois-bouchard-omar-solano-samridhi-1dnlyr0",
        "reason": "Both argue for measuring the actual workload before optimizing context, but this study concerns reuse of prior answers and shows that nominal cache hits may rarely be valid substitutes, whereas the tutor result supports retaining fully cached history."
      },
      {
        "title": "TokTier: Exact Stateful Tokenization for Agentic LLM Serving",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2607.29678v1",
        "feed7_url": "https://feed7.dev/p/2607-29678v1-1fuv1qw",
        "reason": "TokTier shows that prompt-cache hits do not remove tokenization cost; this study adds that response-cache hits do not guarantee answer reuse. Together they require separate telemetry for serving efficiency and semantic validity."
      },
      {
        "title": "Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=yqF6XhzbWBk",
        "feed7_url": "https://feed7.dev/p/inside-847-production-clinical-ai-notes-sebastian-fox-composo-0lrc8td",
        "reason": "The clinical-note evidence reinforces the study’s warning that plausible similarity is not enough: consequential omissions can escape generic judgment, making answer-substitutability validation especially important in high-cost domains."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-08-20T17:14:19.000Z",
  "modified_at": "2026-08-20T17:14:19.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2608-20280v1-1ygnhyz",
    "json": "https://feed7.dev/p/2608-20280v1-1ygnhyz.json",
    "markdown": "https://feed7.dev/p/2608-20280v1-1ygnhyz.md"
  }
}