# PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

Source: [arXiv](https://arxiv.org/abs/2607.28587v1)  
Feed7 permalink: https://feed7.dev/p/2607-28587v1-0u0uow2  
Published: 2026-07-30T17:42:44.000Z  
Trust: Needs Review (needs_review)

## Why Included

PAIChecker finds PR–issue mismatch in 13.6% of SWE-bench Verified instances. Benchmark maintainers should validate that the issue, patch, and tests describe the same task before trusting agent scores.

## Source Summary

The study finds **13.6% of SWE-bench Verified** instances misaligned across five patterns and eleven scenarios. PAIChecker detects mismatches through pattern identification, cross-agent label synthesis, and code-level validation.

## Practical Implication

Before using repository tasks to compare coding agents, verify that the issue statement and PR oracle cover the same work. Mispaired tasks can penalize correct fixes or reward changes unsupported by the stated problem.

## Agent-Ready Context

The study finds **13.6% of SWE-bench Verified** instances misaligned across five patterns and eleven scenarios. PAIChecker detects mismatches through pattern identification, cross-agent label synthesis, and code-level validation.

Before using repository tasks to compare coding agents, verify that the issue statement and PR oracle cover the same work. Mispaired tasks can penalize correct fixes or reward changes unsupported by the stated problem.

Across four LLM backbones, PAIChecker reached up to **92.12% accuracy on SWE-Gym** and **91.67% on SWE-bench Multilingual**. Accuracy remains below perfect, so automated filtering should not replace review of disputed or high-impact cases.

## Connected Context

Feed7 judgment across 297 accumulated Signals:

This identifies oracle–prompt alignment as a separate benchmark-integrity requirement from contamination, infrastructure variance, and grader reliability. It narrows how PR-derived coding results should be interpreted: even a sealed, reproducible harness can mis-score agents when the issue and reference change encode different work, so task-generation pipelines need alignment checks before leaderboard use.

- [Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments](https://feed7.dev/p/2607-28591v1-169rlqh) — Change2Task scales task creation from merged PRs; PAIChecker supplies a necessary validation step because the resulting issue–PR pairs may not specify and grade the same work.
- [Reward hacking is swamping model intelligence gains](https://feed7.dev/p/reward-hacking-coding-benchmarks-18ddebo) — Both weaken confidence in PR-mined coding scores, but at different points: Cursor finds agents retrieving known fixes, while PAIChecker finds defects in what tasks ask versus reward.
- [Quantifying infrastructure noise in agentic coding evals](https://feed7.dev/p/infrastructure-noise-1jyyyw1) — Infrastructure noise and issue–PR misalignment are independent sources of score distortion, so controlling container resources alone does not establish benchmark validity.
- [Separating signal from noise in coding evaluations](https://feed7.dev/p/separating-signal-from-noise-coding-evaluations-17ha3r5) — PAIChecker provides a specific, measured mechanism that reinforces OpenAI’s broader warning that reliability and accuracy defects can make SWE-style rankings unsafe to treat as settled.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: benchmark-integrity, agent-evals, multi-agent

## Uncertainty

- Across four LLM backbones, PAIChecker reached up to **92.12% accuracy on SWE-Gym** and **91.67% on SWE-bench Multilingual**. Accuracy remains below perfect, so automated filtering should not replace review of disputed or high-impact cases.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.