GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Meta doubled GEM’s training efficiency by co-designing recommendation-specific kernels, precision, parallelism, networking, and memory. The takeaway is to profile workload shape before reusing an LLM training stack.
Meta reports GEM reached **20–25% end-to-end MFU** while training FLOPs grew **4× in 12 months** across several thousand GPUs. Custom kernels, MXFP8 training, topology-aware **5D parallelism**, and communication changes address jagged sequences and sparse-dense parameters.
Builders operating specialized model workloads should separate local compute efficiency from distributed scaling efficiency. Profile sequence shapes, padding, numerical sensitivity, communication overlap, memory pressure, and rank imbalance before assuming standard LLM kernels or parallelism recipes transfer.
Meta reports GEM reached **20–25% end-to-end MFU** while training FLOPs grew **4× in 12 months** across several thousand GPUs. Custom kernels, MXFP8 training, topology-aware **5D parallelism**, and communication changes address jagged sequences and sparse-dense parameters. Builders operating specialized model workloads should separate local compute efficiency from distributed scaling efficiency. Profile sequence shapes, padding, numerical sensitivity, communication overlap, memory pressure, and rank imbalance before assuming standard LLM kernels or parallelism recipes transfer. These techniques target Meta’s ads model with trillions of sparse and billions of dense parameters. The material does not establish that the same kernels, precision choices, or topology will benefit smaller models or different recommendation workloads.
This extends observability from serving and agent traces into distributed training efficiency. It separates kernel-level utilization from scaling losses caused by communication, topology, memory pressure, and rank imbalance, making shape- and stage-specific profiling a prerequisite for optimization. Unlike the candidates’ broadly reusable operational controls, its reported techniques remain tied to Meta’s unusually large sparse-dense ads workload and should not be assumed to transfer.