# Separating signal from noise in coding evaluations

Source: [OpenAI](https://openai.com/index/separating-signal-from-noise-coding-evaluations)  
Feed7 permalink: https://feed7.dev/p/separating-signal-from-noise-coding-evaluations-17ha3r5  
Published: Unknown  
Trust: Official Source (official_source)

## Why Included

OpenAI reports reliability and accuracy problems in SWE-Bench Pro, making its coding-model rankings unsafe to treat as settled evidence.

## Source Summary

OpenAI says its analysis found **reliability and accuracy problems** in **SWE-Bench Pro**. That puts the benchmark’s model rankings in doubt.

## Practical Implication

Builders comparing coding agents should avoid making model or routing decisions from this leaderboard alone until the evaluation issues are understood.

## Agent-Ready Context

OpenAI says its analysis found **reliability and accuracy problems** in **SWE-Bench Pro**. That puts the benchmark’s model rankings in doubt.

Builders comparing coding agents should avoid making model or routing decisions from this leaderboard alone until the evaluation issues are understood.

The supplied material does not describe the defects, their size, or corrected results, so it cannot show which rankings change or by how much.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, benchmark-integrity, model-selection

## Uncertainty

- The supplied material does not describe the defects, their size, or corrected results, so it cannot show which rankings change or by how much.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.