# SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents

Source: [arXiv](https://arxiv.org/abs/2609.04167v1)  
Feed7 permalink: https://feed7.dev/p/2609-04167v1-1vvofgw  
Published: 2026-09-03T17:53:34.000Z  
Trust: Needs Review (needs_review)

## Why Included

SWE-Gate shows why green tests are an incomplete agent-eval signal: 221 of 644 functionally passing repairs still violated constraints derived from code review.

## Source Summary

SWE-Gate contains **303 repair instances** from **75 Python repositories**, with separate tests for functionality and constraints derived from real pull-request reviews. Across four model backends, 644 repairs passed functional tests, but **221** still violated review constraints.

## Practical Implication

Treat green tests as one gate, not final acceptance, for coding-agent patches. Encode review expectations as executable checks where possible, and evaluate issue resolution separately from compliance with repository-specific requirements.

## Agent-Ready Context

SWE-Gate contains **303 repair instances** from **75 Python repositories**, with separate tests for functionality and constraints derived from real pull-request reviews. Across four model backends, 644 repairs passed functional tests, but **221** still violated review constraints.

Treat green tests as one gate, not final acceptance, for coding-agent patches. Encode review expectations as executable checks where possible, and evaluate issue resolution separately from compliance with repository-specific requirements.

The benchmark uses synthesized repair instances and a common agent scaffold, so its failure rates may not transfer directly to your repos. It nevertheless exposes a concrete blind spot in functional-only coding-agent evaluations.

## Connected Context

Feed7 judgment across 691 accumulated Signals:

This converts a general warning about coding-agent benchmarks into a specific acceptance gap: functional success can coexist with violations of repository review constraints. It argues for separately executable functionality and compliance gates, while synthesized tasks and a shared scaffold limit direct extrapolation to production repositories.

- [DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve](https://feed7.dev/p/deepswe-a-contamination-resistant-coding-benchmark-james-shi-datacurve-08p61c0) — Complements DeepSWE’s contamination and long-horizon focus by measuring a different hidden failure mode: patches that work but disregard review-derived constraints.
- [Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software](https://feed7.dev/p/rethinking-environments-for-long-horizon-work-rayan-garg-theta-software-11r7wbx) — Implements the call for final-state inspection by splitting acceptance into functional state and repository-specific constraint compliance.
- [Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models](https://feed7.dev/p/2607-12962v1-0q4i26c) — Together they weaken simplistic repair evidence: retries may not use error content, and passing functional tests may still leave the requested constraints unresolved.
- [When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI](https://feed7.dev/p/when-will-the-benchmaxxing-plague-end-nick-heiner-surge-ai-178gqcg) — Provides concrete support for the claim that verifier design can inflate coding-agent success when tests omit requirements that human reviewers enforce.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, benchmark-integrity, agent-reliability

## Uncertainty

- The benchmark uses synthesized repair instances and a common agent scaffold, so its failure rates may not transfer directly to your repos. It nevertheless exposes a concrete blind spot in functional-only coding-agent evaluations.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.