$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
β-OPSD exposes self-distillation’s fixed regularization as a tunable parameter, then approximates policy optimization through logit mixing. It targets more stable reasoning training without direct RL.
The paper identifies vanilla on-policy self-distillation as the **β=1** case of a broader objective. **β-OPSD** uses β to balance a reference policy against a privileged teacher, with the optimum expressed as a geometric interpolation.
For builders training reasoning models, tune this reference-to-teacher tradeoff instead of accepting OPSD’s implicit setting. The method mixes token logits to form a distillation target and adds return-to-go credit assignment, avoiding direct high-variance RL optimization.
The paper identifies vanilla on-policy self-distillation as the **β=1** case of a broader objective. **β-OPSD** uses β to balance a reference policy against a privileged teacher, with the optimum expressed as a geometric interpolation. For builders training reasoning models, tune this reference-to-teacher tradeoff instead of accepting OPSD’s implicit setting. The method mixes token logits to form a distillation target and adds return-to-go credit assignment, avoiding direct high-variance RL optimization. The method reportedly improves stability and reasoning performance over vanilla OPSD on mathematical benchmarks. The supplied material provides no model sizes, absolute scores, compute costs, or evidence for coding-agent tasks.
This replaces vanilla OPSD’s fixed teacher–reference balance with an explicit tuning parameter and adds token-level targets plus return-to-go credit assignment. It therefore broadens the self-distillation design space, but the supplied evidence only supports mathematical reasoning claims; compute efficiency, coding transfer, and advantages over other disagreement- or reward-based distillation methods remain open.