Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism
Meta details the tensor, context, and expert parallelism behind its LLM serving: sub-350ms first token, sub-25ms per token, and 1M-token contexts processed in under a minute on one H100 host.
**The gist** Meta laid out the three parallelism schemes behind its LLM inference: tensor parallelism with **DDA** allreduce (10–50% faster decode on **MI300X**), context parallelism with Pass-KV and Pass-Q ring attention that processes **1M tokens in under a minute** on a single H100 host (10M across hosts), and expert parallelism using two-shot all-to-all for MoE routing.
**Why it matters** The stated targets — **TTFT under 350ms**, **TTIT under 25ms** — are a usable yardstick for judging your own serving stack or provider. The framing also explains agent-relevant behavior: prefill is compute-bound and decode memory-bound, which is why long contexts hurt first-token latency, and communication alone can eat **10–30%** of end-to-end time at scale.
**The gist** Meta laid out the three parallelism schemes behind its LLM inference: tensor parallelism with **DDA** allreduce (10–50% faster decode on **MI300X**), context parallelism with Pass-KV and Pass-Q ring attention that processes **1M tokens in under a minute** on a single H100 host (10M across hosts), and expert parallelism using two-shot all-to-all for MoE routing. **Why it matters** The stated targets — **TTFT under 350ms**, **TTIT under 25ms** — are a usable yardstick for judging your own serving stack or provider. The framing also explains agent-relevant behavior: prefill is compute-bound and decode memory-bound, which is why long contexts hurt first-token latency, and communication alone can eat **10–30%** of end-to-end time at scale. **Watch out** This is Meta-scale infrastructure tuned on fleets of **H100s and MI300Xs**; the techniques apply directly only if you run your own inference. API users feel them secondhand, and this post attaches no open-source release.