arXivPaperNeeds Review

SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving

SWE-Serve tests whether coding agents can ship repository-scale inference features, showing that patches passing local checks still fail production-style end-to-end tests.

arXiv · Sep 22, 2026

Open Source Open Markdown Open JSON

Source Summary

SWE-Serve contains **53 SGLang tasks** across six inference-engineering families, each run on CPU or one H100. Across **11 models and 31 configurations**, the best setup reaches 75% mean pass@1.

Practical Implication

Builders should treat end-to-end serving checks as a distinct acceptance gate for agent patches. Hidden functional and regression tests alone missed failures involving coordinated changes across model support, runtime execution, and public APIs.

Agent-Ready Context

SWE-Serve contains **53 SGLang tasks** across six inference-engineering families, each run on CPU or one H100. Across **11 models and 31 configurations**, the best setup reaches 75% mean pass@1.

Builders should treat end-to-end serving checks as a distinct acceptance gate for agent patches. Hidden functional and regression tests alone missed failures involving coordinated changes across model support, runtime execution, and public APIs.

On 19 tasks with end-to-end coverage, pass rate falls from **69.4% without E2E scoring** to **45.9% under the full verifier**. The benchmark is narrowly grounded in SGLang and production inference work, so it does not establish the same gap for every repository.

Connected Context · Feed7 Judgment

This adds domain-specific evidence that production inference patches need a full-stack acceptance gate: functional and regression checks can pass while coordinated model-support, runtime, and public-API behavior still fails. It reinforces multi-gate evaluation with a large measured drop under end-to-end verification, but narrows the conclusion to SGLang-style serving work rather than all software repositories.

SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering AgentsBoth separate functional test success from deployable correctness: SWE-Gate checks review-derived constraints, while SWE-Serve exposes failures through end-to-end serving execution.Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware ScreenThe vulnerability-repair study similarly shows that a convenient proxy can reward broken patches, reinforcing SWE-Serve’s requirement for execution-grounded acceptance rather than isolated checks.Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval BelferThe vLLM failures show why serving evaluation must include runtime behavior: scheduler and cache defects can yield invalid outputs without ordinary process errors, matching SWE-Serve’s concern with coordinated engine-level correctness.When LLM Decompilers Recompile More and Preserve LessBoth demonstrate that passing shipped or functional tests does not establish behavioral fidelity, though decompilation can use differential execution against an original binary while serving patches require end-to-end system checks.

Context Map

benchmarkcoding#agent-evals#agent-reliability#benchmark-integrity

Uncertainty

On 19 tasks with end-to-end coverage, pass rate falls from **69.4% without E2E scoring** to **45.9% under the full verifier**. The benchmark is narrowly grounded in SGLang and production inference work, so it does not establish the same gap for every repository.