arXivPaperNeeds Review

Score Centering Stabilizes Off-policy Reinforcement Learning

Training–inference mismatch may destabilize LLM reinforcement learning through accumulating score drift. An additive centering correction matched or beat importance sampling in the reported tests.

arXiv · Sep 17, 2026

Open Source Open Markdown Open JSON

Source Summary

The paper attributes instability under training–inference mismatch to persistent score drift that accumulates across steps. Its additive **score-centering** correction cancels that drift across models from **0.6B to 30B parameters**.

Practical Implication

If you train models with rollout and training engines that cannot be made identical, test score centering before paying the efficiency cost of eliminating the mismatch. The correction can also be combined with **importance sampling** rather than replacing it.

Agent-Ready Context

The paper attributes instability under training–inference mismatch to persistent score drift that accumulates across steps. Its additive **score-centering** correction cancels that drift across models from **0.6B to 30B parameters**.

If you train models with rollout and training engines that cannot be made identical, test score centering before paying the efficiency cost of eliminating the mismatch. The correction can also be combined with **importance sampling** rather than replacing it.

The abstract reports matching or better results under quantization and stronger combined results in staleness experiments, but gives no absolute scores or broader workload evidence. This is primarily relevant to model-training infrastructure, not day-to-day agent use.

Connected Context · Feed7 Judgment

This adds a targeted systems correction for off-policy RL instability when rollout and training engines differ: cancel accumulated score drift instead of assuming the mismatch must be eliminated. It complements importance sampling and appears compatible with quantized execution, but the missing absolute results and workload breadth leave its practical advantage over engine alignment unquantified.

UE5M3 FP4 Block Scaling for Stable Language Model PretrainingBoth address numerical stability under low precision, but at different stages: UE5M3 changes the FP4 pretraining recipe, while score centering corrects drift caused by mismatched rollout and training execution.The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent GloballyThe quantization-damage work concerns how reduced precision degrades model weights broadly; this Signal addresses a distinct consequence of quantized or stale execution—persistent score drift during off-policy training.RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward ConstructionRRC constructs a better reward from relative judgments, whereas score centering stabilizes how an off-policy training signal behaves across mismatched engines; the mechanisms address complementary parts of the RL pipeline.

Context Map

model

Uncertainty

The abstract reports matching or better results under quantization and stronger combined results in staleness experiments, but gives no absolute scores or broader workload evidence. This is primarily relevant to model-training infrastructure, not day-to-day agent use.