Where Rounding Happens Matters: Rounding-Location Alignment Stabilizes LLM Reinforcement Learning
A study across four Qwen-family models finds that matching selected training-side BF16 rounding points to inference can prevent RL collapse, without changing the inference engine or learning objective.
The paper aligns training-side rounding with inference in **RoPE**, SwiGLU, and residual RMSNorm by keeping selected intermediates in FP32 until BF16 outputs. Across **four Qwen-family models**, aligning all three regions prevented collapse in the studied runs; RoPE-path alignment alone also sufficed.
For teams training LLMs with separate inference and training stacks, inspect where intermediate values are rounded. Matching selected rounding locations may stabilize BF16 reinforcement learning without replacing kernels or changing the learning objective.
The paper aligns training-side rounding with inference in **RoPE**, SwiGLU, and residual RMSNorm by keeping selected intermediates in FP32 until BF16 outputs. Across **four Qwen-family models**, aligning all three regions prevented collapse in the studied runs; RoPE-path alignment alone also sufficed. For teams training LLMs with separate inference and training stacks, inspect where intermediate values are rounded. Matching selected rounding locations may stabilize BF16 reinforcement learning without replacing kernels or changing the learning objective. The result is limited to the models and runs studied. RoPE-only alignment worked despite remaining numerical differences, but the abstract does not establish whether that holds across other stacks or workloads.
This points to a specific systems-level cause of instability in the studied BF16 RL runs: training and inference can round intermediate values at different points. Aligning rounding locations, especially on the RoPE path, is a targeted stabilization option alongside score-drift correction, though neither result establishes a universal fix across stacks.