# Score Centering Stabilizes Off-policy Reinforcement Learning

Source: [arXiv](https://arxiv.org/abs/2609.20807v1)  
Feed7 permalink: https://feed7.dev/p/2609-20807v1-1tql3tf  
Published: 2026-09-17T17:58:17.000Z  
Trust: Needs Review (needs_review)

## Why Included

Training–inference mismatch may destabilize LLM reinforcement learning through accumulating score drift. An additive centering correction matched or beat importance sampling in the reported tests.

## Source Summary

The paper attributes instability under training–inference mismatch to persistent score drift that accumulates across steps. Its additive **score-centering** correction cancels that drift across models from **0.6B to 30B parameters**.

## Practical Implication

If you train models with rollout and training engines that cannot be made identical, test score centering before paying the efficiency cost of eliminating the mismatch. The correction can also be combined with **importance sampling** rather than replacing it.

## Agent-Ready Context

The paper attributes instability under training–inference mismatch to persistent score drift that accumulates across steps. Its additive **score-centering** correction cancels that drift across models from **0.6B to 30B parameters**.

If you train models with rollout and training engines that cannot be made identical, test score centering before paying the efficiency cost of eliminating the mismatch. The correction can also be combined with **importance sampling** rather than replacing it.

The abstract reports matching or better results under quantization and stronger combined results in staleness experiments, but gives no absolute scores or broader workload evidence. This is primarily relevant to model-training infrastructure, not day-to-day agent use.

## Connected Context

Feed7 judgment across 807 accumulated Signals:

This adds a targeted systems correction for off-policy RL instability when rollout and training engines differ: cancel accumulated score drift instead of assuming the mismatch must be eliminated. It complements importance sampling and appears compatible with quantized execution, but the missing absolute results and workload breadth leave its practical advantage over engine alignment unquantified.

- [UE5M3 FP4 Block Scaling for Stable Language Model Pretraining](https://feed7.dev/p/2609-02846v1-014naa0) — Both address numerical stability under low precision, but at different stages: UE5M3 changes the FP4 pretraining recipe, while score centering corrects drift caused by mismatched rollout and training execution.
- [The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally](https://feed7.dev/p/2609-01587v1-1kd5xo7) — The quantization-damage work concerns how reduced precision degrades model weights broadly; this Signal addresses a distinct consequence of quantized or stale execution—persistent score drift during off-policy training.
- [RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction](https://feed7.dev/p/2608-06310v1-1e0x375) — RRC constructs a better reward from relative judgments, whereas score centering stabilizes how an off-policy training signal behaves across mismatched engines; the mechanisms address complementary parts of the RL pipeline.

## Context Map

- Layer: model
- Domains: None
- Topics: None

## Uncertainty

- The abstract reports matching or better results under quantization and stronger combined results in staleness experiments, but gives no absolute scores or broader workload evidence. This is primarily relevant to model-training infrastructure, not day-to-day agent use.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.