Separating signal from noise in coding evaluations

OpenAI says its analysis found **reliability and accuracy problems** in **SWE-Bench Pro**. That puts the benchmark’s model rankings in doubt.

Builders comparing coding agents should avoid making model or routing decisions from this leaderboard alone until the evaluation issues are understood.

The supplied material does not describe the defects, their size, or corrected results, so it cannot show which rankings change or by how much.