# Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel

Source: [huggingface.co](https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel)  
Feed7 permalink: https://feed7.dev/p/accelerating-fine-tuning-nvidia-nemo-automodel-0dj2ywc  
Published: Unknown  
Trust: Source Linked (source_linked)

## Why Included

NVIDIA's NeMo AutoModel makes MoE fine-tuning ~3.4-3.7x faster on Transformers v5 while keeping the AutoModelForCausalLM API — relevant if you tune open models like Qwen3-30B-A3B for your own agents.

## Source Summary

**The gist** NVIDIA released **NeMo AutoModel**, an open-source library on **Transformers v5** that subclasses AutoModelForCausalLM and accelerates MoE fine-tuning: **3.69x** throughput on **Qwen3-30B-A3B** (11,340 vs 3,075 tokens/sec per GPU, 29% less memory) and 3.36x on Nemotron 3 Nano, both on 8x H100; a 550B Nemotron reached 815 tokens/sec per GPU across **128 GPUs**. Published **June 24, 2026**, supporting 20+ architectures including Mixtral and DeepSeek V2/V3.

## Practical Implication

**Why it matters** Fine-tuning an open MoE model for a specialized agent gets materially cheaper without changing APIs — existing HF Transformers training code keeps working while **Expert Parallelism**, **DeepEP** fused dispatch, and **TransformerEngine** kernels do the work underneath.

## Agent-Ready Context

**The gist** NVIDIA released **NeMo AutoModel**, an open-source library on **Transformers v5** that subclasses AutoModelForCausalLM and accelerates MoE fine-tuning: **3.69x** throughput on **Qwen3-30B-A3B** (11,340 vs 3,075 tokens/sec per GPU, 29% less memory) and 3.36x on Nemotron 3 Nano, both on 8x H100; a 550B Nemotron reached 815 tokens/sec per GPU across **128 GPUs**. Published **June 24, 2026**, supporting 20+ architectures including Mixtral and DeepSeek V2/V3.

**Why it matters** Fine-tuning an open MoE model for a specialized agent gets materially cheaper without changing APIs — existing HF Transformers training code keeps working while **Expert Parallelism**, **DeepEP** fused dispatch, and **TransformerEngine** kernels do the work underneath.

**Watch out** The gains lean on **Transformers v5** — v4 outright **deadlocks** on Qwen3-30B-A3B under FSDP, so this is also a forced-upgrade story — and the benchmarks are NVIDIA's own, all on **H100** hardware.

## Context Map

- Layer: infra
- Domains: None
- Topics: open-models

## Uncertainty

- The gains lean on **Transformers v5** — v4 outright **deadlocks** on Qwen3-30B-A3B under FSDP, so this is also a forced-upgrade story — and the benchmarks are NVIDIA's own, all on **H100** hardware.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.