arXivPaperNeeds Review
Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models
Placebo-controlled tests found no evidence that small frozen code models repaired failures because of the error content itself. Retry scaffolds and mismatched feedback performed as well or better.
arXiv
Source Summary
PoPE tested frozen **0.5–1.5B code models** with live errors and placebos through prompts and adapter training. In prompting, a form-only placebo unlocked **12 units versus 10** for live error patterns on a resistant 40-unit band.
Practical Implication
When measuring self-repair, preserve the retry scaffold but remove or mismatch task-relevant feedback. If the placebo performs similarly, do not attribute gains to the compiler or test error; the retry format itself may be doing the work.
Agent-Ready Context
PoPE tested frozen **0.5–1.5B code models** with live errors and placebos through prompts and adapter training. In prompting, a form-only placebo unlocked **12 units versus 10** for live error patterns on a resistant 40-unit band. When measuring self-repair, preserve the retry scaffold but remove or mismatch task-relevant feedback. If the placebo performs similarly, do not attribute gains to the compiler or test error; the retry format itself may be doing the work. In adapter training, error content tied the no-intervention baseline at **8–8**, while a deranged-error placebo reached **10 unlocks**. These are public-tier screening results, not equivalence evidence; hidden-tier confirmation was deferred.
Context Map
benchmarkcoding#agent-evals#benchmark-integrity#agent-reliabilityUncertainty
In adapter training, error content tied the no-intervention baseline at **8–8**, while a deranged-error placebo reached **10 unlocks**. These are public-tier screening results, not equivalence evidence; hidden-tier confirmation was deferred.