# SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents

Source: [arXiv](https://arxiv.org/abs/2609.04167v1)  
Feed7 permalink: https://feed7.dev/p/swe-gate-passing-functional-tests-is-not-enough-for-soft-4951efd983  
Published: 2026-09-03T00:00:00.000Z  
Trust: Needs Review (needs_review)

## Why Included

Green tests missed review constraints in 221 of 644 passing repairs, so encode repository expectations as separate executable checks.

## Source Summary

SWE-Gate shows why green tests are an incomplete agent-eval signal: 221 of 644 functionally passing repairs still violated constraints derived from code review.

## Practical Implication

Treat green tests as one gate, not final acceptance, for coding-agent patches. Encode review expectations as executable checks where possible, and evaluate issue resolution separately from compliance with repository-specific requirements.

## Agent-Ready Context

SWE-Gate contains **303 repair instances** from **75 Python repositories**, with separate tests for functionality and constraints derived from real pull-request reviews. Across four model backends, 644 repairs passed functional tests, but **221** still violated review constraints.

Treat green tests as one gate, not final acceptance, for coding-agent patches. Encode review expectations as executable checks where possible, and evaluate issue resolution separately from compliance with repository-specific requirements.

The benchmark uses synthesized repair instances and a common agent scaffold, so its failure rates may not transfer directly to your repos. It nevertheless exposes a concrete blind spot in functional-only coding-agent evaluations.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, benchmark-integrity, agent-reliability

## Uncertainty

- Automatically selected from source material; feed7 has not independently tested the claim.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.