Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher
Long contexts can exhaust GPU memory before model weights do. Capacity planning for agent workloads must include KV cache, concurrency, and serving-engine behavior.
For Mistral 7B, the workshop calculates **131 KB per cached token**. At **16,000 context tokens** and **80 concurrent users**, KV cache alone needs about **42 GB**, exceeding a 24 GB GPU before other memory costs are counted.
When self-hosting models for agents, size capacity from context length and concurrency, not weights alone. Evaluate paged attention, continuous batching, prefix caching, model quantization, and KV-cache quantization in a serving engine against your own latency and throughput targets.
For Mistral 7B, the workshop calculates **131 KB per cached token**. At **16,000 context tokens** and **80 concurrent users**, KV cache alone needs about **42 GB**, exceeding a 24 GB GPU before other memory costs are counted. When self-hosting models for agents, size capacity from context length and concurrency, not weights alone. Evaluate paged attention, continuous batching, prefix caching, model quantization, and KV-cache quantization in a serving engine against your own latency and throughput targets. Quantization assumes an acceptable quality trade-off and needs external evaluation. The workshop is beginner-to-intermediate, and its examples do not establish one engine or optimization as best for every workload.
This quantifies why model weights are an incomplete capacity estimate for long-running agents: context length and concurrency can make KV cache the binding GPU constraint. It supplies the memory premise behind cache-aware serving techniques, while narrowing optimization claims to workload-specific measurements and quality evaluation rather than a universally best engine.