Rethinking On-Policy Distillation of Large Language Models II: One Training Example
On-policy distillation recovered most full-data gains from one query; diverse rollouts mattered more than dataset size, while slow student alignment remained the bottleneck.
A single-query OPD run reached **71.5% state coverage** and recovered most of full-data OPD’s gain across tested tasks and model families. With **16 queries**, coverage rose to **98.9%** and matched full-data training.
If you distill smaller models for agent workloads, optimize prompts for diverse visited states before collecting a large task dataset. The experiments suggest rollout coverage, not query count or topical content alone, is the useful data signal.
A single-query OPD run reached **71.5% state coverage** and recovered most of full-data OPD’s gain across tested tasks and model families. With **16 queries**, coverage rose to **98.9%** and matched full-data training. If you distill smaller models for agent workloads, optimize prompts for diverse visited states before collecting a large task dataset. The experiments suggest rollout coverage, not query count or topical content alone, is the useful data signal. This does not make training cheap: student-teacher alignment still took **hundreds of steps**, even on fixed states. The claims concern OPD experiments, so deployment quality and transfer to a specific coding-agent workload still need validation.
This sharply narrows data-efficiency claims for distillation: topical dataset breadth may be unnecessary when a few prompts generate broad state coverage, but optimization remains costly because alignment still requires hundreds of steps. For agent adaptation, prompt selection should be judged by visited-state diversity, not query count alone.