{
  "schema_version": "1.0",
  "id": "archive:https://engineering.fb.com/2025/10/17/ai-research/scaling-llm-inference-innovations-tensor-parallelism-context-parallelism-expert-parallelism/",
  "slug": "scaling-llm-inference-innovations-tensor-parallelism-context-parallelism-0wubti2",
  "url": "https://feed7.dev/p/scaling-llm-inference-innovations-tensor-parallelism-context-parallelism-0wubti2",
  "title": "Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelism",
  "why_included": "Meta details the tensor, context, and expert parallelism behind its LLM serving: sub-350ms first token, sub-25ms per token, and 1M-token contexts processed in under a minute on one H100 host.",
  "summary": "**The gist** Meta laid out the three parallelism schemes behind its LLM inference: tensor parallelism with **DDA** allreduce (10–50% faster decode on **MI300X**), context parallelism with Pass-KV and Pass-Q ring attention that processes **1M tokens in under a minute** on a single H100 host (10M across hosts), and expert parallelism using two-shot all-to-all for MoE routing.",
  "practical_implication": "**Why it matters** The stated targets — **TTFT under 350ms**, **TTIT under 25ms** — are a usable yardstick for judging your own serving stack or provider. The framing also explains agent-relevant behavior: prefill is compute-bound and decode memory-bound, which is why long contexts hurt first-token latency, and communication alone can eat **10–30%** of end-to-end time at scale.",
  "agent_context": "**The gist** Meta laid out the three parallelism schemes behind its LLM inference: tensor parallelism with **DDA** allreduce (10–50% faster decode on **MI300X**), context parallelism with Pass-KV and Pass-Q ring attention that processes **1M tokens in under a minute** on a single H100 host (10M across hosts), and expert parallelism using two-shot all-to-all for MoE routing.\n\n**Why it matters** The stated targets — **TTFT under 350ms**, **TTIT under 25ms** — are a usable yardstick for judging your own serving stack or provider. The framing also explains agent-relevant behavior: prefill is compute-bound and decode memory-bound, which is why long contexts hurt first-token latency, and communication alone can eat **10–30%** of end-to-end time at scale.\n\n**Watch out** This is Meta-scale infrastructure tuned on fleets of **H100s and MI300Xs**; the techniques apply directly only if you run your own inference. API users feel them secondhand, and this post attaches no open-source release.",
  "source": {
    "name": "Meta AI",
    "url": "https://engineering.fb.com/2025/10/17/ai-research/scaling-llm-inference-innovations-tensor-parallelism-context-parallelism-expert-parallelism/",
    "published_at": null
  },
  "source_class": "blog_post",
  "content_type": "Engineering Post",
  "layer": "infra",
  "domains": [],
  "topics": [],
  "verification": {
    "status": "official_source",
    "label": "Official Source",
    "method": "source_feed",
    "verified_at": null
  },
  "uncertainty": [
    "This is Meta-scale infrastructure tuned on fleets of **H100s and MI300Xs**; the techniques apply directly only if you run your own inference. API users feel them secondhand, and this post attaches no open-source release."
  ],
  "lifecycle": "Current",
  "published_at": null,
  "modified_at": null,
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/scaling-llm-inference-innovations-tensor-parallelism-context-parallelism-0wubti2",
    "json": "https://feed7.dev/p/scaling-llm-inference-innovations-tensor-parallelism-context-parallelism-0wubti2.json",
    "markdown": "https://feed7.dev/p/scaling-llm-inference-innovations-tensor-parallelism-context-parallelism-0wubti2.md"
  }
}