Higher-Scoring Coding Models Did Not Get More Work Done
A natural experiment and five model-upgrade comparisons found no statistically significant gain in merged pull requests, despite benchmark score differences. The study argues benchmark improvements may not predict team output.
During a **19-day** suspension of Claude Fable 5, researchers tracked **297** affected users and compared them with 8,908 developers who stayed on Opus 4.8. Access to Fable 5 corresponded to a 1.4 percentage-point change in seven-day merge rate, with an interval spanning zero; the five upgrade comparisons also found no significant gains.
When choosing a coding model, measure outcomes in your own workflow, such as accepted changes or time to merge, alongside benchmark scores. Treat a model upgrade as a hypothesis to test against a stable baseline.
During a **19-day** suspension of Claude Fable 5, researchers tracked **297** affected users and compared them with 8,908 developers who stayed on Opus 4.8. Access to Fable 5 corresponded to a 1.4 percentage-point change in seven-day merge rate, with an interval spanning zero; the five upgrade comparisons also found no significant gains. When choosing a coding model, measure outcomes in your own workflow, such as accepted changes or time to merge, alongside benchmark scores. Treat a model upgrade as a hypothesis to test against a stable baseline. The study is observational, based on GitHub pull requests, and reports intervals that include no effect. It does not show that models never help individual developers or tasks; the reported comparisons found no detectable productivity increase at the measured scale.
This observational study finds no detectable productivity gain from the tested higher-scoring coding-model upgrades at the measured scale: the merge-rate interval includes zero, as did five upgrade comparisons. It makes workflow outcomes such as accepted changes and time to merge necessary evidence for model selection, without implying that upgrades cannot help particular developers or tasks.