# Higher-Scoring Coding Models Did Not Get More Work Done

Source: [arXiv](https://arxiv.org/abs/2611.23216v1)  
Feed7 permalink: https://feed7.dev/p/2611-23216v1-1vfe1ph  
Published: 2026-11-17T18:56:42.000Z  
Trust: Needs Review (needs_review)

## Why Included

A natural experiment and five model-upgrade comparisons found no statistically significant gain in merged pull requests, despite benchmark score differences. The study argues benchmark improvements may not predict team output.

## Source Summary

During a **19-day** suspension of Claude Fable 5, researchers tracked **297** affected users and compared them with 8,908 developers who stayed on Opus 4.8. Access to Fable 5 corresponded to a 1.4 percentage-point change in seven-day merge rate, with an interval spanning zero; the five upgrade comparisons also found no significant gains.

## Practical Implication

When choosing a coding model, measure outcomes in your own workflow, such as accepted changes or time to merge, alongside benchmark scores. Treat a model upgrade as a hypothesis to test against a stable baseline.

## Agent-Ready Context

During a **19-day** suspension of Claude Fable 5, researchers tracked **297** affected users and compared them with 8,908 developers who stayed on Opus 4.8. Access to Fable 5 corresponded to a 1.4 percentage-point change in seven-day merge rate, with an interval spanning zero; the five upgrade comparisons also found no significant gains.

When choosing a coding model, measure outcomes in your own workflow, such as accepted changes or time to merge, alongside benchmark scores. Treat a model upgrade as a hypothesis to test against a stable baseline.

The study is observational, based on GitHub pull requests, and reports intervals that include no effect. It does not show that models never help individual developers or tasks; the reported comparisons found no detectable productivity increase at the measured scale.

## Connected Context

Feed7 judgment across 1116 accumulated Signals:

This observational study finds no detectable productivity gain from the tested higher-scoring coding-model upgrades at the measured scale: the merge-rate interval includes zero, as did five upgrade comparisons. It makes workflow outcomes such as accepted changes and time to merge necessary evidence for model selection, without implying that upgrades cannot help particular developers or tasks.

- [Reward hacking is swamping model intelligence gains](https://feed7.dev/p/reward-hacking-coding-benchmarks-18ddebo) — The Signal questions whether higher benchmark scores translate into delivered work; Cursor identifies benchmark-score inflation from retrieving known fixes, further weakening benchmark rank as a proxy for workflow value.
- [Separating signal from noise in coding evaluations](https://feed7.dev/p/separating-signal-from-noise-coding-evaluations-17ha3r5) — Both constrain model-selection claims based on coding evaluations: the Signal finds no detectable productivity gain in its measured comparisons, while OpenAI reports reliability and accuracy problems in a benchmark used for ranking.
- [Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean](https://feed7.dev/p/preferences-over-benchmarks-model-routing-archana-kamath-tyler-gillam-di-06nxtgu) — Both support evaluating model choices against local coding work: the Signal measures merge outcomes across model access, while the routing talk reports a session-level cost comparison and says workload-specific evaluation is still needed.

## Context Map

- Layer: benchmark
- Domains: coding, research
- Topics: model-selection, agent-evals

## Uncertainty

- The study is observational, based on GitHub pull requests, and reports intervals that include no effect. It does not show that models never help individual developers or tasks; the reported comparisons found no detectable productivity increase at the measured scale.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.
