# Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism

Source: [Meta AI](https://engineering.fb.com/2025/10/17/ai-research/scaling-llm-inference-innovations-tensor-parallelism-context-parallelism-expert-parallelism/)  
Feed7 permalink: https://feed7.dev/p/scaling-llm-inference-innovations-tensor-parallelism-context-parallelism-0wubti2  
Published: Unknown  
Trust: Official Source (official_source)

## Why Included

Meta details the tensor, context, and expert parallelism behind its LLM serving: sub-350ms first token, sub-25ms per token, and 1M-token contexts processed in under a minute on one H100 host.

## Source Summary

**The gist** Meta laid out the three parallelism schemes behind its LLM inference: tensor parallelism with **DDA** allreduce (10–50% faster decode on **MI300X**), context parallelism with Pass-KV and Pass-Q ring attention that processes **1M tokens in under a minute** on a single H100 host (10M across hosts), and expert parallelism using two-shot all-to-all for MoE routing.

## Practical Implication

**Why it matters** The stated targets — **TTFT under 350ms**, **TTIT under 25ms** — are a usable yardstick for judging your own serving stack or provider. The framing also explains agent-relevant behavior: prefill is compute-bound and decode memory-bound, which is why long contexts hurt first-token latency, and communication alone can eat **10–30%** of end-to-end time at scale.

## Agent-Ready Context

**The gist** Meta laid out the three parallelism schemes behind its LLM inference: tensor parallelism with **DDA** allreduce (10–50% faster decode on **MI300X**), context parallelism with Pass-KV and Pass-Q ring attention that processes **1M tokens in under a minute** on a single H100 host (10M across hosts), and expert parallelism using two-shot all-to-all for MoE routing.

**Why it matters** The stated targets — **TTFT under 350ms**, **TTIT under 25ms** — are a usable yardstick for judging your own serving stack or provider. The framing also explains agent-relevant behavior: prefill is compute-bound and decode memory-bound, which is why long contexts hurt first-token latency, and communication alone can eat **10–30%** of end-to-end time at scale.

**Watch out** This is Meta-scale infrastructure tuned on fleets of **H100s and MI300Xs**; the techniques apply directly only if you run your own inference. API users feel them secondhand, and this post attaches no open-source release.

## Context Map

- Layer: infra
- Domains: None
- Topics: None

## Uncertainty

- This is Meta-scale infrastructure tuned on fleets of **H100s and MI300Xs**; the techniques apply directly only if you run your own inference. API users feel them secondhand, and this post attaches no open-source release.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.