DemoPSD: Disagreement-Modulated Policy Self-Distillation
DemoPSD gates self-distillation per token by teacher–student disagreement, cutting the answer-leakage shortcuts that hurt generalization; beats GRPO and SDPO on science QA in and out of domain.
**The gist** DemoPSD tweaks on-policy self-distillation, where one LLM acts as both teacher (with privileged information like the answer) and student. Instead of matching the teacher everywhere, the student is pulled toward a **reverse-KL barycenter** — a teacher–student blend weighted per token by how much the two disagree. On **SciKnowEval** (four scientific fields) and out-of-distribution **GPQA**, it beats **GRPO and SDPO** while keeping training entropy higher.
**Why it matters** On-policy distillation is gaining ground as a cheaper alternative to RL for post-training reasoning models, and this paper names its core failure mode: dense teacher supervision leaks **answer-dependent shortcuts** that vanish at test time and suppresses **exploration**. If you distill or fine-tune your own models, gate teacher guidance selectively; if you only consume models, it explains why distilled ones can ace in-domain evals and stumble elsewhere.
**The gist** DemoPSD tweaks on-policy self-distillation, where one LLM acts as both teacher (with privileged information like the answer) and student. Instead of matching the teacher everywhere, the student is pulled toward a **reverse-KL barycenter** — a teacher–student blend weighted per token by how much the two disagree. On **SciKnowEval** (four scientific fields) and out-of-distribution **GPQA**, it beats **GRPO and SDPO** while keeping training entropy higher. **Why it matters** On-policy distillation is gaining ground as a cheaper alternative to RL for post-training reasoning models, and this paper names its core failure mode: dense teacher supervision leaks **answer-dependent shortcuts** that vanish at test time and suppresses **exploration**. If you distill or fine-tune your own models, gate teacher guidance selectively; if you only consume models, it explains why distilled ones can ace in-domain evals and stumble elsewhere. **Watch out** The abstract reports **no concrete numbers, base models, or scale**, results cover scientific QA rather than coding or agent tasks, and this is a **v1 preprint** without peer review.