{
  "schema_version": "1.0",
  "id": "archive:https://arxiv.org/abs/2607.05394v1",
  "slug": "2607-05394v1-0vq2jpn",
  "url": "https://feed7.dev/p/2607-05394v1-0vq2jpn",
  "title": "Weak-to-Strong Generalization via Direct On-Policy Distillation",
  "why_included": "Direct-OPD reuses a small model's RL run to improve a bigger one: the pre/post-RL log-ratio becomes a dense reward for the stronger student, lifting Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in 4 hours on 8 A100s.",
  "summary": "**The gist** **Direct On-Policy Distillation (Direct-OPD)** runs RL with verifiable rewards on a small, cheap model, then treats the log-ratio between the post-RL teacher and its pre-RL reference as a dense implicit reward applied to the stronger student's own rollouts. It lifts **Qwen3-1.7B from 48.3% to 62.4% on AIME 2024** in **4 hours on 8 A100s** and beats step-matched direct RL.",
  "practical_implication": "**Why it matters** Rollout-heavy post-training is becoming the bottleneck as models scale; this shows RL gains can be reused across model sizes as a reward signal instead of repeated per model. Cheaper reasoning post-training feeds the pace at which better coding-agent models arrive, and the policy shifts **compose sequentially**.",
  "agent_context": "**The gist** **Direct On-Policy Distillation (Direct-OPD)** runs RL with verifiable rewards on a small, cheap model, then treats the log-ratio between the post-RL teacher and its pre-RL reference as a dense implicit reward applied to the stronger student's own rollouts. It lifts **Qwen3-1.7B from 48.3% to 62.4% on AIME 2024** in **4 hours on 8 A100s** and beats step-matched direct RL.\n\n**Why it matters** Rollout-heavy post-training is becoming the bottleneck as models scale; this shows RL gains can be reused across model sizes as a reward signal instead of repeated per model. Cheaper reasoning post-training feeds the pace at which better coding-agent models arrive, and the policy shifts **compose sequentially**.\n\n**Watch out** Evidence sits on **small open models** and **math benchmarks** like AIME; whether the transfer holds at frontier scale or on long-horizon agentic tasks is untested, and a weak teacher's blind spots may still leak through the distilled signal.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2607.05394v1",
    "published_at": null
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "model",
  "domains": [
    "research"
  ],
  "topics": [
    "reasoning"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "Evidence sits on **small open models** and **math benchmarks** like AIME; whether the transfer holds at frontier scale or on long-horizon agentic tasks is untested, and a weak teacher's blind spots may still leak through the distilled signal."
  ],
  "lifecycle": "Current",
  "published_at": null,
  "modified_at": null,
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2607-05394v1-0vq2jpn",
    "json": "https://feed7.dev/p/2607-05394v1-0vq2jpn.json",
    "markdown": "https://feed7.dev/p/2607-05394v1-0vq2jpn.md"
  }
}