arXivPaperNeeds Review
Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models
Placebo-controlled tests found no evidence that small frozen code models repaired failures because of the error content itself. Retry scaffolds and mismatched feedback performed as well or better.
arXiv · Jul 14, 2026
Source Summary
PoPE tested frozen **0.5–1.5B code models** with live errors and placebos through prompts and adapter training. In prompting, a form-only placebo unlocked **12 units versus 10** for live error patterns on a resistant 40-unit band.
Practical Implication
When measuring self-repair, preserve the retry scaffold but remove or mismatch task-relevant feedback. If the placebo performs similarly, do not attribute gains to the compiler or test error; the retry format itself may be doing the work.
Agent-Ready Context
PoPE tested frozen **0.5–1.5B code models** with live errors and placebos through prompts and adapter training. In prompting, a form-only placebo unlocked **12 units versus 10** for live error patterns on a resistant 40-unit band. When measuring self-repair, preserve the retry scaffold but remove or mismatch task-relevant feedback. If the placebo performs similarly, do not attribute gains to the compiler or test error; the retry format itself may be doing the work. In adapter training, error content tied the no-intervention baseline at **8–8**, while a deranged-error placebo reached **10 unlocks**. These are public-tier screening results, not equivalence evidence; hidden-tier confirmation was deferred.
Context Map
benchmarkcoding#agent-evals#benchmark-integrity#agent-reliabilityUncertainty
In adapter training, error content tied the no-intervention baseline at **8–8**, while a deranged-error placebo reached **10 unlocks**. These are public-tier screening results, not equivalence evidence; hidden-tier confirmation was deferred.