# Weak-to-Strong Generalization via Direct On-Policy Distillation

Source: [arXiv](https://arxiv.org/abs/2607.05394v1)  
Feed7 permalink: https://feed7.dev/p/2607-05394v1-0vq2jpn  
Published: Unknown  
Trust: Needs Review (needs_review)

## Why Included

Direct-OPD reuses a small model's RL run to improve a bigger one: the pre/post-RL log-ratio becomes a dense reward for the stronger student, lifting Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in 4 hours on 8 A100s.

## Source Summary

**The gist** **Direct On-Policy Distillation (Direct-OPD)** runs RL with verifiable rewards on a small, cheap model, then treats the log-ratio between the post-RL teacher and its pre-RL reference as a dense implicit reward applied to the stronger student's own rollouts. It lifts **Qwen3-1.7B from 48.3% to 62.4% on AIME 2024** in **4 hours on 8 A100s** and beats step-matched direct RL.

## Practical Implication

**Why it matters** Rollout-heavy post-training is becoming the bottleneck as models scale; this shows RL gains can be reused across model sizes as a reward signal instead of repeated per model. Cheaper reasoning post-training feeds the pace at which better coding-agent models arrive, and the policy shifts **compose sequentially**.

## Agent-Ready Context

**The gist** **Direct On-Policy Distillation (Direct-OPD)** runs RL with verifiable rewards on a small, cheap model, then treats the log-ratio between the post-RL teacher and its pre-RL reference as a dense implicit reward applied to the stronger student's own rollouts. It lifts **Qwen3-1.7B from 48.3% to 62.4% on AIME 2024** in **4 hours on 8 A100s** and beats step-matched direct RL.

**Why it matters** Rollout-heavy post-training is becoming the bottleneck as models scale; this shows RL gains can be reused across model sizes as a reward signal instead of repeated per model. Cheaper reasoning post-training feeds the pace at which better coding-agent models arrive, and the policy shifts **compose sequentially**.

**Watch out** Evidence sits on **small open models** and **math benchmarks** like AIME; whether the transfer holds at frontier scale or on long-horizon agentic tasks is untested, and a weak teacher's blind spots may still leak through the distilled signal.

## Context Map

- Layer: model
- Domains: research
- Topics: reasoning

## Uncertainty

- Evidence sits on **small open models** and **math benchmarks** like AIME; whether the transfer holds at frontier scale or on long-horizon agentic tasks is untested, and a weak teacher's blind spots may still leak through the distilled signal.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.