Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B
At equal generated-token cost, repeated sampling matched or beat seven reflection, critique, selection, and debate methods. Agent evals should budget every generated token, not compare against one-shot baselines.
Researchers tested **seven methods** on 1.5B, 3B, and 7B open models across two math benchmarks, counting critique, reflection, debate, and checking tokens. Across **36 paired comparisons**, none reliably beat repeated sampling at equal cost.
For agent workflows, benchmark reflection loops against repeated independent attempts using the same measured token budget. Self-inspection deserves particular scrutiny: **all 18 comparisons were negative**, and ten results were reliably worse.
Researchers tested **seven methods** on 1.5B, 3B, and 7B open models across two math benchmarks, counting critique, reflection, debate, and checking tokens. Across **36 paired comparisons**, none reliably beat repeated sampling at equal cost. For agent workflows, benchmark reflection loops against repeated independent attempts using the same measured token budget. Self-inspection deserves particular scrutiny: **all 18 comparisons were negative**, and ten results were reliably worse. The evidence covers two math benchmarks with 150 questions each and models only up to 7B parameters, so it may not transfer to larger models or coding tasks. At 7B, Self-Refine and forced Reflexion remained **3.6–10.1 points below** baseline.
This establishes repeated independent sampling as the necessary token-matched baseline for claims that reflection improves small-model reasoning. It strengthens skepticism about self-inspection beyond isolated scaffold failures, but narrows the conclusion to two math benchmarks and models up to 7B; it does not establish that reflection is inferior for larger models, coding agents, or long-horizon environments.