A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets
A policy-optimization theory paper describes how REINFORCE-style training can lose access to good responses even as reward rises. It analyzes reference resets as a way to mitigate that failure under stated assumptions.
The paper identifies **coverage collapse**: REINFORCE-style methods can make optimal responses increasingly unlikely even while reward improves. It proposes Approximate Policy Mirror Descent with repeated KL-reference resets to mitigate the problem.
For builders fine-tuning models, the work highlights a potential gap between rising reward and retaining access to strong responses. When evaluating post-training, consider tracking response coverage as well as reward.
The paper identifies **coverage collapse**: REINFORCE-style methods can make optimal responses increasingly unlikely even while reward improves. It proposes Approximate Policy Mirror Descent with repeated KL-reference resets to mitigate the problem. For builders fine-tuning models, the work highlights a potential gap between rising reward and retaining access to strong responses. When evaluating post-training, consider tracking response coverage as well as reward. The convergence guarantees apply to softmax linear policies over pretrained features and specified algorithm variants. The paper is a theoretical analysis; the provided material does not establish that reference resets improve outcomes in arbitrary production training runs.
The theory warns that reward gains can coexist with losing access to optimal responses, so reward alone may hide a training failure. Coverage tracking is a useful evaluation implication, while the proposed mitigation remains bounded by the paper’s policy assumptions and is not established for production runs.