{
  "schema_version": "1.1",
  "id": "s13:https://arxiv.org/abs/2609.26777v1",
  "slug": "2609-26777v1-1ylj2uv",
  "url": "https://feed7.dev/p/2609-26777v1-1ylj2uv",
  "title": "SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving",
  "why_included": "SWE-Serve tests whether coding agents can ship repository-scale inference features, showing that patches passing local checks still fail production-style end-to-end tests.",
  "summary": "SWE-Serve contains **53 SGLang tasks** across six inference-engineering families, each run on CPU or one H100. Across **11 models and 31 configurations**, the best setup reaches 75% mean pass@1.",
  "practical_implication": "Builders should treat end-to-end serving checks as a distinct acceptance gate for agent patches. Hidden functional and regression tests alone missed failures involving coordinated changes across model support, runtime execution, and public APIs.",
  "agent_context": "SWE-Serve contains **53 SGLang tasks** across six inference-engineering families, each run on CPU or one H100. Across **11 models and 31 configurations**, the best setup reaches 75% mean pass@1.\n\nBuilders should treat end-to-end serving checks as a distinct acceptance gate for agent patches. Hidden functional and regression tests alone missed failures involving coordinated changes across model support, runtime execution, and public APIs.\n\nOn 19 tasks with end-to-end coverage, pass rate falls from **69.4% without E2E scoring** to **45.9% under the full verifier**. The benchmark is narrowly grounded in SGLang and production inference work, so it does not establish the same gap for every repository.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2609.26777v1",
    "published_at": "2026-09-22T17:54:59.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "benchmark",
  "domains": [
    "coding"
  ],
  "topics": [
    "agent-evals",
    "agent-reliability",
    "benchmark-integrity"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "On 19 tasks with end-to-end coverage, pass rate falls from **69.4% without E2E scoring** to **45.9% under the full verifier**. The benchmark is narrowly grounded in SGLang and production inference work, so it does not establish the same gap for every repository."
  ],
  "connected_context": {
    "meaning": "This adds domain-specific evidence that production inference patches need a full-stack acceptance gate: functional and regression checks can pass while coordinated model-support, runtime, and public-API behavior still fails. It reinforces multi-gate evaluation with a large measured drop under end-to-end verification, but narrows the conclusion to SGLang-style serving work rather than all software repositories.",
    "corpus_size": 856,
    "generated_at": "2026-09-23T09:06:29.105Z",
    "connections": [
      {
        "title": "SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.04167v1",
        "feed7_url": "https://feed7.dev/p/2609-04167v1-1vvofgw",
        "reason": "Both separate functional test success from deployable correctness: SWE-Gate checks review-derived constraints, while SWE-Serve exposes failures through end-to-end serving execution."
      },
      {
        "title": "Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.26749v1",
        "feed7_url": "https://feed7.dev/p/2609-26749v1-1fj6jga",
        "reason": "The vulnerability-repair study similarly shows that a convenient proxy can reward broken patches, reinforcing SWE-Serve’s requirement for execution-grounded acceptance rather than isolated checks."
      },
      {
        "title": "Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=btxG75rNJC4",
        "feed7_url": "https://feed7.dev/p/two-bugs-that-hid-in-plain-sight-a-vllm-debugging-detective-story-asaf-g-1s5yaqd",
        "reason": "The vLLM failures show why serving evaluation must include runtime behavior: scheduler and cache defects can yield invalid outputs without ordinary process errors, matching SWE-Serve’s concern with coordinated engine-level correctness."
      },
      {
        "title": "When LLM Decompilers Recompile More and Preserve Less",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.05370v1",
        "feed7_url": "https://feed7.dev/p/2609-05370v1-03wuqdv",
        "reason": "Both demonstrate that passing shipped or functional tests does not establish behavioral fidelity, though decompilation can use differential execution against an original binary while serving patches require end-to-end system checks."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-09-22T17:54:59.000Z",
  "modified_at": "2026-09-22T17:54:59.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2609-26777v1-1ylj2uv",
    "json": "https://feed7.dev/p/2609-26777v1-1ylj2uv.json",
    "markdown": "https://feed7.dev/p/2609-26777v1-1ylj2uv.md"
  }
}