# TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

Source: [arXiv](https://arxiv.org/abs/2611.26619v1)  
Feed7 permalink: https://feed7.dev/p/2611-26619v1-1cp2t0h  
Published: 2026-11-19T18:58:31.000Z  
Trust: Needs Review (needs_review)

## Why Included

TideCache streams attention-cache data between host and GPU memory to free capacity for concurrent agent requests. Its reported gains are promising, but they come from specific SGLang and superchip configurations.

## Source Summary

TideCache combines a shared GPU cache pool, layer-wise prefetching, and request-aware scheduling. In tests on **GB300 and GH200** across **seven configurations**, it handled up to **2.04× as many agent turns** and cut median per-token latency by **20–59%** versus SGLang with HiCache.

## Practical Implication

If you operate hybrid-model serving, consider whether streaming attention-cache prefixes could trade GPU residency for more concurrent requests while overlapping transfers with compute. The implementation is integrated with SGLang and keeps generated KV resident.

## Agent-Ready Context

TideCache combines a shared GPU cache pool, layer-wise prefetching, and request-aware scheduling. In tests on **GB300 and GH200** across **seven configurations**, it handled up to **2.04× as many agent turns** and cut median per-token latency by **20–59%** versus SGLang with HiCache.

If you operate hybrid-model serving, consider whether streaming attention-cache prefixes could trade GPU residency for more concurrent requests while overlapping transfers with compute. The implementation is integrated with SGLang and keeps generated KV resident.

The reported gains are specific to the tested superchips and configurations. The material does not establish how the design performs on other hardware or serving stacks.

## Connected Context

Feed7 judgment across 1135 accumulated Signals:

TideCache reports higher agent-turn capacity and lower token latency from shared GPU cache management and layer-wise streaming on GB300 and GH200. It adds an implementation-level option for the cache-locality and GPU-capacity concerns in the supplied serving talks, but its results do not establish gains on other hardware or serving stacks.

- [KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat](https://feed7.dev/p/kv-cache-aware-routing-and-p-d-disaggregation-on-kubernetes-yuchen-fama-0u2jkju) — Both address KV-cache pressure in agent serving; Red Hat focuses on routing and prefill/decode separation, while TideCache changes cache pooling, prefetching, and scheduling within a serving stack.
- [The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI](https://feed7.dev/p/the-frontier-ai-inference-cloud-for-agents-byung-gon-gon-chun-friendliai-0b7vgoo) — FriendliAI argues that preserving shared prefixes and routing toward resident KV can improve whole-task latency; TideCache offers a distinct serving mechanism for managing cache residency and transfers, with results bounded to its tested hardware.
- [Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher](https://feed7.dev/p/deep-dive-on-llm-inference-at-scale-harshul-jain-audible-tanmay-sah-inde-04utjkh) — The inference deep dive identifies KV cache as a potential GPU capacity constraint for long contexts and concurrency; TideCache measures a way to increase capacity under its specific tested configurations.

## Context Map

- Layer: infra
- Domains: coding
- Topics: cloud-agents, context-caching

## Uncertainty

- The reported gains are specific to the tested superchips and configurations. The material does not establish how the design performs on other hardware or serving stacks.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.
