$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
β-OPSD exposes self-distillation’s fixed regularization as a tunable parameter, then approximates policy optimization through logit mixing. It targets more stable reasoning training without direct RL.
The paper identifies vanilla on-policy self-distillation as the **β=1** case of a broader objective. **β-OPSD** uses β to balance a reference policy against a privileged teacher, with the optimum expressed as a geometric interpolation.
For builders training reasoning models, tune this reference-to-teacher tradeoff instead of accepting OPSD’s implicit setting. The method mixes token logits to form a distillation target and adds return-to-go credit assignment, avoiding direct high-variance RL optimization.
The paper identifies vanilla on-policy self-distillation as the **β=1** case of a broader objective. **β-OPSD** uses β to balance a reference policy against a privileged teacher, with the optimum expressed as a geometric interpolation. For builders training reasoning models, tune this reference-to-teacher tradeoff instead of accepting OPSD’s implicit setting. The method mixes token logits to form a distillation target and adds return-to-go credit assignment, avoiding direct high-variance RL optimization. The method reportedly improves stability and reasoning performance over vanilla OPSD on mathematical benchmarks. The supplied material provides no model sizes, absolute scores, compute costs, or evidence for coding-agent tasks.
This turns OPSD’s fixed reference–teacher balance into an explicit training control and pairs it with lower-variance credit assignment. Against prior distillation work, it adds an objective-level tradeoff rather than token-level gating or weak-to-strong reward transfer. The evidence currently narrows its relevance to mathematical reasoning; it does not yet justify claims about coding agents, multimodal models, or deployment efficiency.