Open Brain
arXivPaperNeeds Review

TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

TideCache streams attention-cache data between host and GPU memory to free capacity for concurrent agent requests. Its reported gains are promising, but they come from specific SGLang and superchip configurations.

arXiv · Nov 19, 2026
Open Source Open MarkdownOpen JSON
Source Summary

TideCache combines a shared GPU cache pool, layer-wise prefetching, and request-aware scheduling. In tests on **GB300 and GH200** across **seven configurations**, it handled up to **2.04× as many agent turns** and cut median per-token latency by **20–59%** versus SGLang with HiCache.

Practical Implication

If you operate hybrid-model serving, consider whether streaming attention-cache prefixes could trade GPU residency for more concurrent requests while overlapping transfers with compute. The implementation is integrated with SGLang and keeps generated KV resident.

Agent-Ready Context
TideCache combines a shared GPU cache pool, layer-wise prefetching, and request-aware scheduling. In tests on **GB300 and GH200** across **seven configurations**, it handled up to **2.04× as many agent turns** and cut median per-token latency by **20–59%** versus SGLang with HiCache.

If you operate hybrid-model serving, consider whether streaming attention-cache prefixes could trade GPU residency for more concurrent requests while overlapping transfers with compute. The implementation is integrated with SGLang and keeps generated KV resident.

The reported gains are specific to the tested superchips and configurations. The material does not establish how the design performs on other hardware or serving stacks.
Connected Context · Feed7 Judgment

TideCache reports higher agent-turn capacity and lower token latency from shared GPU cache management and layer-wise streaming on GB300 and GH200. It adds an implementation-level option for the cache-locality and GPU-capacity concerns in the supplied serving talks, but its results do not establish gains on other hardware or serving stacks.

KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red HatBoth address KV-cache pressure in agent serving; Red Hat focuses on routing and prefill/decode separation, while TideCache changes cache pooling, prefetching, and scheduling within a serving stack.The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAIFriendliAI argues that preserving shared prefixes and routing toward resident KV can improve whole-task latency; TideCache offers a distinct serving mechanism for managing cache residency and transfers, with results bounded to its tested hardware.Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI ResearcherThe inference deep dive identifies KV cache as a potential GPU capacity constraint for long contexts and concurrency; TideCache measures a way to increase capacity under its specific tested configurations.
Context Map
infracoding#cloud-agents#context-caching
Uncertainty
The reported gains are specific to the tested superchips and configurations. The material does not establish how the design performs on other hardware or serving stacks.