# Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

Source: [AI Engineer](https://www.youtube.com/watch?v=y2W4FNAuPEA)  
Feed7 permalink: https://feed7.dev/p/deep-dive-on-llm-inference-at-scale-harshul-jain-audible-tanmay-sah-inde-04utjkh  
Published: 2026-09-08T15:00:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

Long contexts can exhaust GPU memory before model weights do. Capacity planning for agent workloads must include KV cache, concurrency, and serving-engine behavior.

## Source Summary

For Mistral 7B, the workshop calculates **131 KB per cached token**. At **16,000 context tokens** and **80 concurrent users**, KV cache alone needs about **42 GB**, exceeding a 24 GB GPU before other memory costs are counted.

## Practical Implication

When self-hosting models for agents, size capacity from context length and concurrency, not weights alone. Evaluate paged attention, continuous batching, prefix caching, model quantization, and KV-cache quantization in a serving engine against your own latency and throughput targets.

## Agent-Ready Context

For Mistral 7B, the workshop calculates **131 KB per cached token**. At **16,000 context tokens** and **80 concurrent users**, KV cache alone needs about **42 GB**, exceeding a 24 GB GPU before other memory costs are counted.

When self-hosting models for agents, size capacity from context length and concurrency, not weights alone. Evaluate paged attention, continuous batching, prefix caching, model quantization, and KV-cache quantization in a serving engine against your own latency and throughput targets.

Quantization assumes an acceptable quality trade-off and needs external evaluation. The workshop is beginner-to-intermediate, and its examples do not establish one engine or optimization as best for every workload.

## Connected Context

Feed7 judgment across 732 accumulated Signals:

This quantifies why model weights are an incomplete capacity estimate for long-running agents: context length and concurrency can make KV cache the binding GPU constraint. It supplies the memory premise behind cache-aware serving techniques, while narrowing optimization claims to workload-specific measurements and quality evaluation rather than a universally best engine.

- [KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat](https://feed7.dev/p/kv-cache-aware-routing-and-p-d-disaggregation-on-kubernetes-yuchen-fama-0u2jkju) — The calculated KV-cache pressure explains when cache-aware routing and split prefill/decode may become valuable; both require validation against actual concurrency, prompt shape, latency, and network costs.
- [TokTier: Exact Stateful Tokenization for Agentic LLM Serving](https://feed7.dev/p/2607-29678v1-1fuv1qw) — TokTier identifies tokenization as an additional long-session bottleneck, showing that solving GPU KV-cache capacity alone does not guarantee low time to first token.
- [jundot/omlx](https://feed7.dev/p/omlx-04mmjt8) — oMLX implements two techniques highlighted here—continuous batching and persistent KV caching—for local concurrent agents, but its limited hardware evidence reinforces the need for workload-specific evaluation.

## Context Map

- Layer: infra
- Domains: coding
- Topics: context-caching, observability

## Uncertainty

- Quantization assumes an acceptable quality trade-off and needs external evaluation. The workshop is beginner-to-intermediate, and its examples do not establish one engine or optimization as best for every workload.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.