# Rethinking On-Policy Distillation of Large Language Models II: One Training Example

Source: [arXiv](https://arxiv.org/abs/2609.04172v1)  
Feed7 permalink: https://feed7.dev/p/2609-04172v1-0kofdqa  
Published: 2026-09-03T17:54:38.000Z  
Trust: Needs Review (needs_review)

## Why Included

On-policy distillation recovered most full-data gains from one query; diverse rollouts mattered more than dataset size, while slow student alignment remained the bottleneck.

## Source Summary

A single-query OPD run reached **71.5% state coverage** and recovered most of full-data OPD’s gain across tested tasks and model families. With **16 queries**, coverage rose to **98.9%** and matched full-data training.

## Practical Implication

If you distill smaller models for agent workloads, optimize prompts for diverse visited states before collecting a large task dataset. The experiments suggest rollout coverage, not query count or topical content alone, is the useful data signal.

## Agent-Ready Context

A single-query OPD run reached **71.5% state coverage** and recovered most of full-data OPD’s gain across tested tasks and model families. With **16 queries**, coverage rose to **98.9%** and matched full-data training.

If you distill smaller models for agent workloads, optimize prompts for diverse visited states before collecting a large task dataset. The experiments suggest rollout coverage, not query count or topical content alone, is the useful data signal.

This does not make training cheap: student-teacher alignment still took **hundreds of steps**, even on fixed states. The claims concern OPD experiments, so deployment quality and transfer to a specific coding-agent workload still need validation.

## Connected Context

Feed7 judgment across 691 accumulated Signals:

This sharply narrows data-efficiency claims for distillation: topical dataset breadth may be unnecessary when a few prompts generate broad state coverage, but optimization remains costly because alignment still requires hundreds of steps. For agent adaptation, prompt selection should be judged by visited-state diversity, not query count alone.

- [Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI](https://feed7.dev/p/data-quality-is-the-compute-multiplier-ari-morcos-datologyai-0x7k2ve) — Makes the general data-curation principle operational for OPD by identifying rollout-state coverage as the relevant information-value signal.
- [OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling](https://feed7.dev/p/2608-05141v1-0kai3a6) — Both favor structurally informative training exposure over raw volume: dependency-linked contexts in mid-training and diverse visited states in distillation.
- [Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms](https://feed7.dev/p/2608-27409v1-1bivffd) — Adds a useful metric contrast: OPD gains track visited-state coverage, while RLVR fusion did not broaden solution coverage despite similar average performance.
- [Adaption Labs: Gradient-Free Continual Learning — Sara Hooker, Adaption](https://feed7.dev/p/adaption-labs-gradient-free-continual-learning-sara-hooker-adaption-1pozjjc) — Supplies a concrete constraint for automated adaptation systems: discovering a tiny effective prompt set would not remove the substantial student-alignment compute.

## Context Map

- Layer: model
- Domains: None
- Topics: reasoning, model-selection

## Uncertainty

- This does not make training cheap: student-teacher alignment still took **hundreds of steps**, even on fixed states. The claims concern OPD experiments, so deployment quality and transfer to a specific coding-agent workload still need validation.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.