# An Empirical Study of Harness Design for Coding Agents

Source: [arXiv](https://arxiv.org/abs/2609.20804v1)  
Feed7 permalink: https://feed7.dev/p/2609-20804v1-17ecr62  
Published: 2026-09-17T17:58:07.000Z  
Trust: Needs Review (needs_review)

## Why Included

Harness components pay off differently by model and budget. Elide before summarizing, use planning selectively, and avoid elaborate tools when the model is already strong with bash.

## Source Summary

Researchers tested **176 matched settings** across **four models**, varying planning, action space, and context management on SWE-Bench Verified and Terminal-Bench 2.1. Context handling mattered more as budgets tightened, chiefly by preventing overflow.

## Practical Implication

Stage rule-based elision before LLM summarization. Use planning as an accuracy scaffold for weaker models and a cost control for stronger ones; offer predefined tools when bash skill is weak, but consider bash-only operation for capable models on CLI-heavy work.

## Agent-Ready Context

Researchers tested **176 matched settings** across **four models**, varying planning, action space, and context management on SWE-Bench Verified and Terminal-Bench 2.1. Context handling mattered more as budgets tightened, chiefly by preventing overflow.

Stage rule-based elision before LLM summarization. Use planning as an accuracy scaffold for weaker models and a cost control for stronger ones; offer predefined tools when bash skill is weak, but consider bash-only operation for capable models on CLI-heavy work.

Recoverable elision added machinery without an accuracy gain because models rarely used recovery. The evidence spans **five context strategies** and **four window budgets**, but the abstract provides no effect sizes and covers only two benchmarks.

## Connected Context

Feed7 judgment across 807 accumulated Signals:

This supplies comparative evidence for choices that prior harness guidance treated mainly as design principles. It narrows the highest-leverage coding-agent controls to budget-aware context management, model-dependent planning, and action spaces matched to tool competence, while warning that extra recovery machinery is not automatically useful. Its two-benchmark scope does not establish the broader production controls needed for stateful or irreversible work.

- [Harness Engineering: Building the Production Cage for Powerful Domain Agents — Mike Chambers, AWS](https://feed7.dev/p/harness-engineering-building-the-production-cage-for-powerful-domain-age-0la85zn) — The production-harness framework defines the surrounding system broadly; this study adds matched empirical evidence about which planning, context, and tool-interface choices matter within that boundary.
- [Agents Without Code: Skills, YAML, and Filesystems Replaced Python — Philipp Schmid, Google DeepMind](https://feed7.dev/p/agents-without-code-skills-yaml-and-filesystems-replaced-python-philipp-0t7a4s8) — The reduced-orchestration approach favors general tools and instructions, while these results qualify that choice by showing that predefined tools can still help models with weaker bash competence.
- [AI Agents Are Just Distributed Systems Now — Salman Munaf, TikTok](https://feed7.dev/p/ai-agents-are-just-distributed-systems-now-salman-munaf-tiktok-1v4yc47) — The study measures accuracy and context behavior on coding benchmarks, whereas this candidate identifies external-state semantics and recovery controls that remain necessary beyond benchmark task completion.

## Context Map

- Layer: agent
- Domains: coding
- Topics: harness-engineering, context-engineering, tool-use

## Uncertainty

- Recoverable elision added machinery without an accuracy gain because models rarely used recovery. The evidence spans **five context strategies** and **four window budgets**, but the abstract provides no effect sizes and covers only two benchmarks.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.