# UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

Source: [arXiv](https://arxiv.org/abs/2609.02846v1)  
Feed7 permalink: https://feed7.dev/p/2609-02846v1-014naa0  
Published: 2026-09-02T17:32:07.000Z  
Trust: Needs Review (needs_review)

## Why Included

A UE5M3 block-scaling recipe trained an 8B model in FP4 without Hadamard transforms or BF16 final layers, while reporting better losses and downstream estimates than the compared recipe.

## Source Summary

The authors paired E2M1 payloads with **UE5M3 block scales**, used selective stochastic rounding, removed the randomized Hadamard transform, and applied FP4 to eligible internal linears. They pretrained an **8B model on nearly 190B tokens**.

## Practical Implication

For low-precision training work, test whether wider block scales can replace auxiliary transforms and higher-precision exemptions. The simpler recipe may reduce work outside FP4 matrix multiplications.

## Agent-Ready Context

The authors paired E2M1 payloads with **UE5M3 block scales**, used selective stochastic rounding, removed the randomized Hadamard transform, and applied FP4 to eligible internal linears. They pretrained an **8B model on nearly 190B tokens**.

For low-precision training work, test whether wider block scales can replace auxiliary transforms and higher-precision exemptions. The simpler recipe may reduce work outside FP4 matrix multiplications.

Results come from software-emulated training and recipe-specific inference policies. The reported **21.2% throughput increase** is a native execution ablation that jointly removed two components, not a direct end-to-end UE5M3 hardware result.

## Connected Context

Feed7 judgment across 669 accumulated Signals:

This adds numerical format and scaling design to model-selection judgment: a wider FP4 block scale may simplify low-precision pretraining by removing transforms and exemptions. The evidence remains recipe- and execution-specific, so the throughput figure should motivate native hardware validation rather than be treated as an end-to-end deployment gain.

- [Adaption Labs: Gradient-Free Continual Learning — Sara Hooker, Adaption](https://feed7.dev/p/adaption-labs-gradient-free-continual-learning-sara-hooker-adaption-1pozjjc) — The FP4 configuration is a concrete example of the interacting architecture and training-recipe choices that broader automated adaptation systems would need to search and validate together.
- [When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning](https://feed7.dev/p/2608-31118v1-1abyqwd) — The ontology study’s task-dependent scaling results reinforce the same selection discipline: neither parameter count nor a precision-format throughput ablation substitutes for matched quality and deployment measurements.

## Context Map

- Layer: model
- Domains: None
- Topics: model-selection

## Uncertainty

- Results come from software-emulated training and recipe-specific inference policies. The reported **21.2% throughput increase** is a native execution ablation that jointly removed two components, not a direct end-to-end UE5M3 hardware result.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.