TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming
TideCache streams attention-cache data between host and GPU memory to free capacity for concurrent agent requests. Its reported gains are promising, but they come from specific SGLang and superchip configurations.
TideCache combines a shared GPU cache pool, layer-wise prefetching, and request-aware scheduling. In tests on **GB300 and GH200** across **seven configurations**, it handled up to **2.04× as many agent turns** and cut median per-token latency by **20–59%** versus SGLang with HiCache.
If you operate hybrid-model serving, consider whether streaming attention-cache prefixes could trade GPU residency for more concurrent requests while overlapping transfers with compute. The implementation is integrated with SGLang and keeps generated KV resident.
TideCache combines a shared GPU cache pool, layer-wise prefetching, and request-aware scheduling. In tests on **GB300 and GH200** across **seven configurations**, it handled up to **2.04× as many agent turns** and cut median per-token latency by **20–59%** versus SGLang with HiCache. If you operate hybrid-model serving, consider whether streaming attention-cache prefixes could trade GPU residency for more concurrent requests while overlapping transfers with compute. The implementation is integrated with SGLang and keeps generated KV resident. The reported gains are specific to the tested superchips and configurations. The material does not establish how the design performs on other hardware or serving stacks.
TideCache reports higher agent-turn capacity and lower token latency from shared GPU cache management and layer-wise streaming on GB300 and GH200. It adds an implementation-level option for the cache-locality and GPU-capacity concerns in the supplied serving talks, but its results do not establish gains on other hardware or serving stacks.