arXivPaperNeeds Review

PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

PAIChecker finds PR–issue mismatch in 13.6% of SWE-bench Verified instances. Benchmark maintainers should validate that the issue, patch, and tests describe the same task before trusting agent scores.

arXiv · Jul 30, 2026

Open Source Open Markdown Open JSON

Source Summary

The study finds **13.6% of SWE-bench Verified** instances misaligned across five patterns and eleven scenarios. PAIChecker detects mismatches through pattern identification, cross-agent label synthesis, and code-level validation.

Practical Implication

Before using repository tasks to compare coding agents, verify that the issue statement and PR oracle cover the same work. Mispaired tasks can penalize correct fixes or reward changes unsupported by the stated problem.

Agent-Ready Context

The study finds **13.6% of SWE-bench Verified** instances misaligned across five patterns and eleven scenarios. PAIChecker detects mismatches through pattern identification, cross-agent label synthesis, and code-level validation.

Before using repository tasks to compare coding agents, verify that the issue statement and PR oracle cover the same work. Mispaired tasks can penalize correct fixes or reward changes unsupported by the stated problem.

Across four LLM backbones, PAIChecker reached up to **92.12% accuracy on SWE-Gym** and **91.67% on SWE-bench Multilingual**. Accuracy remains below perfect, so automated filtering should not replace review of disputed or high-impact cases.

Connected Context · Feed7 Judgment

This identifies oracle–prompt alignment as a separate benchmark-integrity requirement from contamination, infrastructure variance, and grader reliability. It narrows how PR-derived coding results should be interpreted: even a sealed, reproducible harness can mis-score agents when the issue and reference change encode different work, so task-generation pipelines need alignment checks before leaderboard use.

Change2Task: From Repository Changes to Executable Coding Agent Tasks and EnvironmentsChange2Task scales task creation from merged PRs; PAIChecker supplies a necessary validation step because the resulting issue–PR pairs may not specify and grade the same work.Reward hacking is swamping model intelligence gainsBoth weaken confidence in PR-mined coding scores, but at different points: Cursor finds agents retrieving known fixes, while PAIChecker finds defects in what tasks ask versus reward.Quantifying infrastructure noise in agentic coding evalsInfrastructure noise and issue–PR misalignment are independent sources of score distortion, so controlling container resources alone does not establish benchmark validity.Separating signal from noise in coding evaluationsPAIChecker provides a specific, measured mechanism that reinforces OpenAI’s broader warning that reliability and accuracy defects can make SWE-style rankings unsafe to treat as settled.

Context Map

benchmarkcoding#benchmark-integrity#agent-evals#multi-agent

Uncertainty

Across four LLM backbones, PAIChecker reached up to **92.12% accuracy on SWE-Gym** and **91.67% on SWE-bench Multilingual**. Accuracy remains below perfect, so automated filtering should not replace review of disputed or high-impact cases.