Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel
NVIDIA's NeMo AutoModel makes MoE fine-tuning ~3.4-3.7x faster on Transformers v5 while keeping the AutoModelForCausalLM API — relevant if you tune open models like Qwen3-30B-A3B for your own agents.
**The gist** NVIDIA released **NeMo AutoModel**, an open-source library on **Transformers v5** that subclasses AutoModelForCausalLM and accelerates MoE fine-tuning: **3.69x** throughput on **Qwen3-30B-A3B** (11,340 vs 3,075 tokens/sec per GPU, 29% less memory) and 3.36x on Nemotron 3 Nano, both on 8x H100; a 550B Nemotron reached 815 tokens/sec per GPU across **128 GPUs**. Published **June 24, 2026**, supporting 20+ architectures including Mixtral and DeepSeek V2/V3.
**Why it matters** Fine-tuning an open MoE model for a specialized agent gets materially cheaper without changing APIs — existing HF Transformers training code keeps working while **Expert Parallelism**, **DeepEP** fused dispatch, and **TransformerEngine** kernels do the work underneath.
**The gist** NVIDIA released **NeMo AutoModel**, an open-source library on **Transformers v5** that subclasses AutoModelForCausalLM and accelerates MoE fine-tuning: **3.69x** throughput on **Qwen3-30B-A3B** (11,340 vs 3,075 tokens/sec per GPU, 29% less memory) and 3.36x on Nemotron 3 Nano, both on 8x H100; a 550B Nemotron reached 815 tokens/sec per GPU across **128 GPUs**. Published **June 24, 2026**, supporting 20+ architectures including Mixtral and DeepSeek V2/V3. **Why it matters** Fine-tuning an open MoE model for a specialized agent gets materially cheaper without changing APIs — existing HF Transformers training code keeps working while **Expert Parallelism**, **DeepEP** fused dispatch, and **TransformerEngine** kernels do the work underneath. **Watch out** The gains lean on **Transformers v5** — v4 outright **deadlocks** on Qwen3-30B-A3B under FSDP, so this is also a forced-upgrade story — and the benchmarks are NVIDIA's own, all on **H100** hardware.