Open Brain
arXivPaperNeeds Review

A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets

A policy-optimization theory paper describes how REINFORCE-style training can lose access to good responses even as reward rises. It analyzes reference resets as a way to mitigate that failure under stated assumptions.

arXiv · Nov 10, 2026
Open Source Open MarkdownOpen JSON
Source Summary

The paper identifies **coverage collapse**: REINFORCE-style methods can make optimal responses increasingly unlikely even while reward improves. It proposes Approximate Policy Mirror Descent with repeated KL-reference resets to mitigate the problem.

Practical Implication

For builders fine-tuning models, the work highlights a potential gap between rising reward and retaining access to strong responses. When evaluating post-training, consider tracking response coverage as well as reward.

Agent-Ready Context
The paper identifies **coverage collapse**: REINFORCE-style methods can make optimal responses increasingly unlikely even while reward improves. It proposes Approximate Policy Mirror Descent with repeated KL-reference resets to mitigate the problem.

For builders fine-tuning models, the work highlights a potential gap between rising reward and retaining access to strong responses. When evaluating post-training, consider tracking response coverage as well as reward.

The convergence guarantees apply to softmax linear policies over pretrained features and specified algorithm variants. The paper is a theoretical analysis; the provided material does not establish that reference resets improve outcomes in arbitrary production training runs.
Connected Context · Feed7 Judgment

The theory warns that reward gains can coexist with losing access to optimal responses, so reward alone may hide a training failure. Coverage tracking is a useful evaluation implication, while the proposed mitigation remains bounded by the paper’s policy assumptions and is not established for production runs.

Context Map
modelresearch#reasoning#open-models
Uncertainty
The convergence guarantees apply to softmax linear policies over pretrained features and specified algorithm variants. The paper is a theoretical analysis; the provided material does not establish that reference resets improve outcomes in arbitrary production training runs.