# DemoPSD: Disagreement-Modulated Policy Self-Distillation

Source: [arXiv](https://arxiv.org/abs/2607.02502v1)  
Feed7 permalink: https://feed7.dev/p/2607-02502v1-0wngknx  
Published: Unknown  
Trust: Needs Review (needs_review)

## Why Included

DemoPSD gates self-distillation per token by teacher–student disagreement, cutting the answer-leakage shortcuts that hurt generalization; beats GRPO and SDPO on science QA in and out of domain.

## Source Summary

**The gist** DemoPSD tweaks on-policy self-distillation, where one LLM acts as both teacher (with privileged information like the answer) and student. Instead of matching the teacher everywhere, the student is pulled toward a **reverse-KL barycenter** — a teacher–student blend weighted per token by how much the two disagree. On **SciKnowEval** (four scientific fields) and out-of-distribution **GPQA**, it beats **GRPO and SDPO** while keeping training entropy higher.

## Practical Implication

**Why it matters** On-policy distillation is gaining ground as a cheaper alternative to RL for post-training reasoning models, and this paper names its core failure mode: dense teacher supervision leaks **answer-dependent shortcuts** that vanish at test time and suppresses **exploration**. If you distill or fine-tune your own models, gate teacher guidance selectively; if you only consume models, it explains why distilled ones can ace in-domain evals and stumble elsewhere.

## Agent-Ready Context

**The gist** DemoPSD tweaks on-policy self-distillation, where one LLM acts as both teacher (with privileged information like the answer) and student. Instead of matching the teacher everywhere, the student is pulled toward a **reverse-KL barycenter** — a teacher–student blend weighted per token by how much the two disagree. On **SciKnowEval** (four scientific fields) and out-of-distribution **GPQA**, it beats **GRPO and SDPO** while keeping training entropy higher.

**Why it matters** On-policy distillation is gaining ground as a cheaper alternative to RL for post-training reasoning models, and this paper names its core failure mode: dense teacher supervision leaks **answer-dependent shortcuts** that vanish at test time and suppresses **exploration**. If you distill or fine-tune your own models, gate teacher guidance selectively; if you only consume models, it explains why distilled ones can ace in-domain evals and stumble elsewhere.

**Watch out** The abstract reports **no concrete numbers, base models, or scale**, results cover scientific QA rather than coding or agent tasks, and this is a **v1 preprint** without peer review.

## Context Map

- Layer: model
- Domains: research
- Topics: reasoning

## Uncertainty

- The abstract reports **no concrete numbers, base models, or scale**, results cover scientific QA rather than coding or agent tasks, and this is a **v1 preprint** without peer review.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.