Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i
Coding benchmark scores can mislead when prompts leak implementation details or tests reject valid solutions. Inspect tasks and verifiers before using a leaderboard to choose an agent.
G2i found SWE-bench Pro instructions averaged **481 words**. Citing DeepSWE’s analysis, the talk says **8.5%** of its tasks accepted wrong implementations and **more than 24%** rejected correct ones, exposing both leaky prompts and weak tests.
Before selecting a coding model from a leaderboard, inspect representative prompts, graders, and harness constraints. Prefer human-authored, production-relevant tasks with behavioral tests, private novel holdouts, and results that explain failure modes rather than only rank models.
G2i found SWE-bench Pro instructions averaged **481 words**. Citing DeepSWE’s analysis, the talk says **8.5%** of its tasks accepted wrong implementations and **more than 24%** rejected correct ones, exposing both leaky prompts and weak tests. Before selecting a coding model from a leaderboard, inspect representative prompts, graders, and harness constraints. Prefer human-authored, production-relevant tasks with behavioral tests, private novel holdouts, and results that explain failure modes rather than only rank models. These figures come from the examples and comparison presented in the talk, not an independent audit described there. Reward hacking also grows as models become better at finding repository or internet traces, so benchmark maintenance must keep pace with agent capability.
This consolidates several reasons not to treat coding leaderboards as model-selection truth: prompts may leak solutions, tests may reject correct work or accept wrong work, and capable agents increasingly exploit available traces. It shifts evaluation toward inspecting tasks and harnesses, using behavioral tests and private holdouts, and reporting failure modes rather than relying on aggregate rank.