AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Language Models
AutoScaffold uses a frozen teacher to turn training failures into adaptive strategies and task hints; students performed better after those supports were removed at evaluation.
A frozen teacher reviews training experience and creates shared strategies or hints for difficult tasks, changing the support as the student improves. Across **four domains**, evaluation without scaffolds beat the corresponding RL baselines; gains included **7.2 points** in ALFWorld success and **4.0 points** in QA exact match.
For builders training task agents, the useful idea is to diagnose recurring and task-specific failures during training, then reduce guidance as the model learns. The paper also reports a higher share of faster-than-reference correct GPU kernels, by up to **5.0 points**.
A frozen teacher reviews training experience and creates shared strategies or hints for difficult tasks, changing the support as the student improves. Across **four domains**, evaluation without scaffolds beat the corresponding RL baselines; gains included **7.2 points** in ALFWorld success and **4.0 points** in QA exact match. For builders training task agents, the useful idea is to diagnose recurring and task-specific failures during training, then reduce guidance as the model learns. The paper also reports a higher share of faster-than-reference correct GPU kernels, by up to **5.0 points**. The results cover ALFWorld, search-based QA, mathematical reasoning, and GPU kernel generation. They do not establish that the approach transfers to other tasks or improves deployed coding agents directly.
AutoScaffold shifts the intervention to training: a frozen teacher turns observed failures into temporary hints and strategies, then evaluates the student without them. The reported gains across four domains support this scaffold-and-fade approach within those settings, while leaving transfer to deployed coding agents unestablished.