UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
A UE5M3 block-scaling recipe trained an 8B model in FP4 without Hadamard transforms or BF16 final layers, while reporting better losses and downstream estimates than the compared recipe.
The authors paired E2M1 payloads with **UE5M3 block scales**, used selective stochastic rounding, removed the randomized Hadamard transform, and applied FP4 to eligible internal linears. They pretrained an **8B model on nearly 190B tokens**.
For low-precision training work, test whether wider block scales can replace auxiliary transforms and higher-precision exemptions. The simpler recipe may reduce work outside FP4 matrix multiplications.
The authors paired E2M1 payloads with **UE5M3 block scales**, used selective stochastic rounding, removed the randomized Hadamard transform, and applied FP4 to eligible internal linears. They pretrained an **8B model on nearly 190B tokens**. For low-precision training work, test whether wider block scales can replace auxiliary transforms and higher-precision exemptions. The simpler recipe may reduce work outside FP4 matrix multiplications. Results come from software-emulated training and recipe-specific inference policies. The reported **21.2% throughput increase** is a native execution ablation that jointly removed two components, not a direct end-to-end UE5M3 hardware result.
This adds numerical format and scaling design to model-selection judgment: a wider FP4 block scale may simplify low-precision pretraining by removing transforms and exemptions. The evidence remains recipe- and execution-specific, so the throughput figure should motivate native hardware validation rather than be treated as an end-to-end deployment gain.