{
  "schema_version": "1.1",
  "id": "archive:https://arxiv.org/abs/2611.23216v1",
  "slug": "2611-23216v1-1vfe1ph",
  "url": "https://feed7.dev/p/2611-23216v1-1vfe1ph",
  "title": "Higher-Scoring Coding Models Did Not Get More Work Done",
  "why_included": "A natural experiment and five model-upgrade comparisons found no statistically significant gain in merged pull requests, despite benchmark score differences. The study argues benchmark improvements may not predict team output.",
  "summary": "During a **19-day** suspension of Claude Fable 5, researchers tracked **297** affected users and compared them with 8,908 developers who stayed on Opus 4.8. Access to Fable 5 corresponded to a 1.4 percentage-point change in seven-day merge rate, with an interval spanning zero; the five upgrade comparisons also found no significant gains.",
  "practical_implication": "When choosing a coding model, measure outcomes in your own workflow, such as accepted changes or time to merge, alongside benchmark scores. Treat a model upgrade as a hypothesis to test against a stable baseline.",
  "agent_context": "During a **19-day** suspension of Claude Fable 5, researchers tracked **297** affected users and compared them with 8,908 developers who stayed on Opus 4.8. Access to Fable 5 corresponded to a 1.4 percentage-point change in seven-day merge rate, with an interval spanning zero; the five upgrade comparisons also found no significant gains.\n\nWhen choosing a coding model, measure outcomes in your own workflow, such as accepted changes or time to merge, alongside benchmark scores. Treat a model upgrade as a hypothesis to test against a stable baseline.\n\nThe study is observational, based on GitHub pull requests, and reports intervals that include no effect. It does not show that models never help individual developers or tasks; the reported comparisons found no detectable productivity increase at the measured scale.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2611.23216v1",
    "published_at": "2026-11-17T18:56:42.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "benchmark",
  "domains": [
    "coding",
    "research"
  ],
  "topics": [
    "model-selection",
    "agent-evals"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "The study is observational, based on GitHub pull requests, and reports intervals that include no effect. It does not show that models never help individual developers or tasks; the reported comparisons found no detectable productivity increase at the measured scale."
  ],
  "connected_context": {
    "meaning": "This observational study finds no detectable productivity gain from the tested higher-scoring coding-model upgrades at the measured scale: the merge-rate interval includes zero, as did five upgrade comparisons. It makes workflow outcomes such as accepted changes and time to merge necessary evidence for model selection, without implying that upgrades cannot help particular developers or tasks.",
    "corpus_size": 1116,
    "generated_at": "2026-11-18T11:09:16.013Z",
    "connections": [
      {
        "title": "Reward hacking is swamping model intelligence gains",
        "source_name": "Cursor",
        "source_url": "https://cursor.com/blog/reward-hacking-coding-benchmarks",
        "feed7_url": "https://feed7.dev/p/reward-hacking-coding-benchmarks-18ddebo",
        "reason": "The Signal questions whether higher benchmark scores translate into delivered work; Cursor identifies benchmark-score inflation from retrieving known fixes, further weakening benchmark rank as a proxy for workflow value."
      },
      {
        "title": "Separating signal from noise in coding evaluations",
        "source_name": "OpenAI",
        "source_url": "https://openai.com/index/separating-signal-from-noise-coding-evaluations",
        "feed7_url": "https://feed7.dev/p/separating-signal-from-noise-coding-evaluations-17ha3r5",
        "reason": "Both constrain model-selection claims based on coding evaluations: the Signal finds no detectable productivity gain in its measured comparisons, while OpenAI reports reliability and accuracy problems in a benchmark used for ranking."
      },
      {
        "title": "Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=FvxY8oPoI8o",
        "feed7_url": "https://feed7.dev/p/preferences-over-benchmarks-model-routing-archana-kamath-tyler-gillam-di-06nxtgu",
        "reason": "Both support evaluating model choices against local coding work: the Signal measures merge outcomes across model access, while the routing talk reports a session-level cost comparison and says workload-specific evaluation is still needed."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-11-17T18:56:42.000Z",
  "modified_at": "2026-11-17T18:56:42.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2611-23216v1-1vfe1ph",
    "json": "https://feed7.dev/p/2611-23216v1-1vfe1ph.json",
    "markdown": "https://feed7.dev/p/2611-23216v1-1vfe1ph.md"
  }
}