Open Brain
arXivPaperNeeds Review

Higher-Scoring Coding Models Did Not Get More Work Done

A natural experiment and five model-upgrade comparisons found no statistically significant gain in merged pull requests, despite benchmark score differences. The study argues benchmark improvements may not predict team output.

arXiv · Nov 17, 2026
Open Source Open MarkdownOpen JSON
Source Summary

During a **19-day** suspension of Claude Fable 5, researchers tracked **297** affected users and compared them with 8,908 developers who stayed on Opus 4.8. Access to Fable 5 corresponded to a 1.4 percentage-point change in seven-day merge rate, with an interval spanning zero; the five upgrade comparisons also found no significant gains.

Practical Implication

When choosing a coding model, measure outcomes in your own workflow, such as accepted changes or time to merge, alongside benchmark scores. Treat a model upgrade as a hypothesis to test against a stable baseline.

Agent-Ready Context
During a **19-day** suspension of Claude Fable 5, researchers tracked **297** affected users and compared them with 8,908 developers who stayed on Opus 4.8. Access to Fable 5 corresponded to a 1.4 percentage-point change in seven-day merge rate, with an interval spanning zero; the five upgrade comparisons also found no significant gains.

When choosing a coding model, measure outcomes in your own workflow, such as accepted changes or time to merge, alongside benchmark scores. Treat a model upgrade as a hypothesis to test against a stable baseline.

The study is observational, based on GitHub pull requests, and reports intervals that include no effect. It does not show that models never help individual developers or tasks; the reported comparisons found no detectable productivity increase at the measured scale.
Connected Context · Feed7 Judgment

This observational study finds no detectable productivity gain from the tested higher-scoring coding-model upgrades at the measured scale: the merge-rate interval includes zero, as did five upgrade comparisons. It makes workflow outcomes such as accepted changes and time to merge necessary evidence for model selection, without implying that upgrades cannot help particular developers or tasks.

Reward hacking is swamping model intelligence gainsThe Signal questions whether higher benchmark scores translate into delivered work; Cursor identifies benchmark-score inflation from retrieving known fixes, further weakening benchmark rank as a proxy for workflow value.Separating signal from noise in coding evaluationsBoth constrain model-selection claims based on coding evaluations: the Signal finds no detectable productivity gain in its measured comparisons, while OpenAI reports reliability and accuracy problems in a benchmark used for ranking.Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOceanBoth support evaluating model choices against local coding work: the Signal measures merge outcomes across model access, while the routing talk reports a session-level cost comparison and says workload-specific evaluation is still needed.
Context Map
benchmarkcodingresearch#model-selection#agent-evals
Uncertainty
The study is observational, based on GitHub pull requests, and reports intervals that include no effect. It does not show that models never help individual developers or tasks; the reported comparisons found no detectable productivity increase at the measured scale.