{
  "schema_version": "1.0",
  "id": "archive:https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel",
  "slug": "accelerating-fine-tuning-nvidia-nemo-automodel-0dj2ywc",
  "url": "https://feed7.dev/p/accelerating-fine-tuning-nvidia-nemo-automodel-0dj2ywc",
  "title": "Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel",
  "why_included": "NVIDIA's NeMo AutoModel makes MoE fine-tuning ~3.4-3.7x faster on Transformers v5 while keeping the AutoModelForCausalLM API — relevant if you tune open models like Qwen3-30B-A3B for your own agents.",
  "summary": "**The gist** NVIDIA released **NeMo AutoModel**, an open-source library on **Transformers v5** that subclasses AutoModelForCausalLM and accelerates MoE fine-tuning: **3.69x** throughput on **Qwen3-30B-A3B** (11,340 vs 3,075 tokens/sec per GPU, 29% less memory) and 3.36x on Nemotron 3 Nano, both on 8x H100; a 550B Nemotron reached 815 tokens/sec per GPU across **128 GPUs**. Published **June 24, 2026**, supporting 20+ architectures including Mixtral and DeepSeek V2/V3.",
  "practical_implication": "**Why it matters** Fine-tuning an open MoE model for a specialized agent gets materially cheaper without changing APIs — existing HF Transformers training code keeps working while **Expert Parallelism**, **DeepEP** fused dispatch, and **TransformerEngine** kernels do the work underneath.",
  "agent_context": "**The gist** NVIDIA released **NeMo AutoModel**, an open-source library on **Transformers v5** that subclasses AutoModelForCausalLM and accelerates MoE fine-tuning: **3.69x** throughput on **Qwen3-30B-A3B** (11,340 vs 3,075 tokens/sec per GPU, 29% less memory) and 3.36x on Nemotron 3 Nano, both on 8x H100; a 550B Nemotron reached 815 tokens/sec per GPU across **128 GPUs**. Published **June 24, 2026**, supporting 20+ architectures including Mixtral and DeepSeek V2/V3.\n\n**Why it matters** Fine-tuning an open MoE model for a specialized agent gets materially cheaper without changing APIs — existing HF Transformers training code keeps working while **Expert Parallelism**, **DeepEP** fused dispatch, and **TransformerEngine** kernels do the work underneath.\n\n**Watch out** The gains lean on **Transformers v5** — v4 outright **deadlocks** on Qwen3-30B-A3B under FSDP, so this is also a forced-upgrade story — and the benchmarks are NVIDIA's own, all on **H100** hardware.",
  "source": {
    "name": "huggingface.co",
    "url": "https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel",
    "published_at": null
  },
  "source_class": "blog_post",
  "content_type": "Source",
  "layer": "infra",
  "domains": [],
  "topics": [
    "open-models"
  ],
  "verification": {
    "status": "source_linked",
    "label": "Source Linked",
    "method": "source_feed",
    "verified_at": null
  },
  "uncertainty": [
    "The gains lean on **Transformers v5** — v4 outright **deadlocks** on Qwen3-30B-A3B under FSDP, so this is also a forced-upgrade story — and the benchmarks are NVIDIA's own, all on **H100** hardware."
  ],
  "lifecycle": "Current",
  "published_at": null,
  "modified_at": null,
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/accelerating-fine-tuning-nvidia-nemo-automodel-0dj2ywc",
    "json": "https://feed7.dev/p/accelerating-fine-tuning-nvidia-nemo-automodel-0dj2ywc.json",
    "markdown": "https://feed7.dev/p/accelerating-fine-tuning-nvidia-nemo-automodel-0dj2ywc.md"
  }
}