Back
arXivPaperNeeds ReviewagentNew
An Empirical Study of Harness Design for Coding Agents
Harness tests favor rule-based elision before summarization, selective planning, and bash-only tools for models already strong at CLI work.
arXivSep 17, 20262 min
Source Summary
Harness components pay off differently by model and budget. Elide before summarizing, use planning selectively, and avoid elaborate tools when the model is already strong with bash.
Practical Implication
Stage rule-based elision before LLM summarization. Use planning as an accuracy scaffold for weaker models and a cost control for stronger ones; offer predefined tools when bash skill is weak, but consider bash-only operation for capable models on CLI-heavy work.
Agent-Ready Context
Researchers tested **176 matched settings** across **four models**, varying planning, action space, and context management on SWE-Bench Verified and Terminal-Bench 2.1. Context handling mattered more as budgets tightened, chiefly by preventing overflow. Stage rule-based elision before LLM summarization. Use planning as an accuracy scaffold for weaker models and a cost control for stronger ones; offer predefined tools when bash skill is weak, but consider bash-only operation for capable models on CLI-heavy work. Recoverable elision added machinery without an accuracy gain because models rarely used recovery. The evidence spans **five context strategies** and **four window budgets**, but the abstract provides no effect sizes and covers only two benchmarks.
Context Map
Uncertainty
Automatically selected from source material; feed7 has not independently tested the claim.
Rate This Item
Personal Note
No note yet. Notes are included in exported bundles.
Related — Every Edge Explained
No approved edges yet for this post.