# SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving

Source: [arXiv](https://arxiv.org/abs/2609.26777v1)  
Feed7 permalink: https://feed7.dev/p/2609-26777v1-1ylj2uv  
Published: 2026-09-22T17:54:59.000Z  
Trust: Needs Review (needs_review)

## Why Included

SWE-Serve tests whether coding agents can ship repository-scale inference features, showing that patches passing local checks still fail production-style end-to-end tests.

## Source Summary

SWE-Serve contains **53 SGLang tasks** across six inference-engineering families, each run on CPU or one H100. Across **11 models and 31 configurations**, the best setup reaches 75% mean pass@1.

## Practical Implication

Builders should treat end-to-end serving checks as a distinct acceptance gate for agent patches. Hidden functional and regression tests alone missed failures involving coordinated changes across model support, runtime execution, and public APIs.

## Agent-Ready Context

SWE-Serve contains **53 SGLang tasks** across six inference-engineering families, each run on CPU or one H100. Across **11 models and 31 configurations**, the best setup reaches 75% mean pass@1.

Builders should treat end-to-end serving checks as a distinct acceptance gate for agent patches. Hidden functional and regression tests alone missed failures involving coordinated changes across model support, runtime execution, and public APIs.

On 19 tasks with end-to-end coverage, pass rate falls from **69.4% without E2E scoring** to **45.9% under the full verifier**. The benchmark is narrowly grounded in SGLang and production inference work, so it does not establish the same gap for every repository.

## Connected Context

Feed7 judgment across 856 accumulated Signals:

This adds domain-specific evidence that production inference patches need a full-stack acceptance gate: functional and regression checks can pass while coordinated model-support, runtime, and public-API behavior still fails. It reinforces multi-gate evaluation with a large measured drop under end-to-end verification, but narrows the conclusion to SGLang-style serving work rather than all software repositories.

- [SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents](https://feed7.dev/p/2609-04167v1-1vvofgw) — Both separate functional test success from deployable correctness: SWE-Gate checks review-derived constraints, while SWE-Serve exposes failures through end-to-end serving execution.
- [Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen](https://feed7.dev/p/2609-26749v1-1fj6jga) — The vulnerability-repair study similarly shows that a convenient proxy can reward broken patches, reinforcing SWE-Serve’s requirement for execution-grounded acceptance rather than isolated checks.
- [Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer](https://feed7.dev/p/two-bugs-that-hid-in-plain-sight-a-vllm-debugging-detective-story-asaf-g-1s5yaqd) — The vLLM failures show why serving evaluation must include runtime behavior: scheduler and cache defects can yield invalid outputs without ordinary process errors, matching SWE-Serve’s concern with coordinated engine-level correctness.
- [When LLM Decompilers Recompile More and Preserve Less](https://feed7.dev/p/2609-05370v1-03wuqdv) — Both demonstrate that passing shipped or functional tests does not establish behavioral fidelity, though decompilation can use differential execution against an original binary while serving patches require end-to-end system checks.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, agent-reliability, benchmark-integrity

## Uncertainty

- On 19 tasks with end-to-end coverage, pass rate falls from **69.4% without E2E scoring** to **45.9% under the full verifier**. The benchmark is narrowly grounded in SGLang and production inference work, so it does not establish the same gap for every repository.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.