Harness Engineering
Current Answer
No editorial synthesis yet — the evidence below is collected automatically from source labels. A current answer lands here once an editor approves one.
Evidence
CPU-ARC turns prior reasoning traces into reusable CPU-executable procedures, reporting frontier-level performance on math and science tasks at 16–291× lower invocation cost. It suggests a way to avoid rerunning a model for every similar instance.
SwarmEvolve uses latent communication between agent swarms to search for better programs. Its reported speed suggests a different way to structure expensive, iterative coding-agent experiments.
ReproBridge finds that agent-ready ML artifacts improve reproduction rates and cut API spending across four evaluated models. The practical lesson is to preserve setup and execution decisions alongside code.
Ramp describes three ways applied research can improve production agents: a finance specialist model, prebuilt compressed context, and large parallel security sweeps. The gains target latency, cost, and coverage.
Strategist uses a reusable library of executable reasoning blueprints that agents retrieve, test, and revise across tasks. The paper reports similar average accuracy to its strongest baseline at lower inference cost.
Meta generates tests that expose behavioral changes in risky diffs, then combines deterministic rules, an LLM ensemble, and aggregate human review to filter likely false positives before code lands.
GEPA-Curator filters noisy agent failures and retrieves relevant successful runs before prompt optimization, improving results over two baselines across four environments.
The AI SDK's fx adapter moves the harness and model requests onto the host while keeping built-in shell and filesystem tools sandboxed, so model credentials stay off the sandbox.
The workshop shows a loop that uses production traces to find recurring agent failures, propose code fixes, and turn failures into evals and regression data. Builders should keep human review and cost checks in that loop.
An EBench study finds that a robot-control agent can complement post-trained action policies: a harness that invokes them matched the strongest compared policies while using fewer tokens. Invocation and handoff still limit performance.
FOCUS uses selective branching to improve value estimates for long-horizon language-model agents. Its results suggest a way to assign credit across segments without spending the same rollout budget on every state.
A budgeted agent turned corpus anomalies into reusable tests, but static sampling found more valid selectors for less money. The useful pattern is agent discovery followed by deterministic checks.
Vercel’s open-source durable-harness logs model and tool steps so cloud coding agents can resume after crashes without rerunning completed calls.
Local coding agents could delete their own execution traces without guardrail alerts. Treat agent-controlled logs as untrusted and capture them through an independent mechanism.
RAPID turns one visual demonstration into an executable, testable robot program, showing how agent loops can derive their own specifications, tools, and verification environment.
Paris keynotes frame agent productivity as a systems problem: consolidate connectors and skills, support long-running sandboxed work, and keep permissions and company data under control.
Cursor cut agent token cost by shrinking static instructions, loading tools on demand, improving cache boundaries, and using subagents more selectively.
A learned harness moved recurring control into reusable code, cutting model calls and inference cost while keeping web-agent performance stable across 4B–120B deployment models.
Reliable agent skills need more than prompts: use hooks as automatic guardrails, test across models, and compile harness-specific variants when instruction-following differs.
onPanda turns the first bad token into an annotation boundary, regenerating from each correction to create on-policy supervision with less manual rewriting.
RRSI regularizes automated harness editing to curb benchmark overfitting, improving held-out tasks while using fewer policy tokens than unregularized evolution.
Large agent workloads turn inference into an orchestration problem: scheduling must account for workflow state, KV cache, hardware, retries, and cost per completed task.
CodeMidas derives verified coding-agent RL tasks directly from working source, widening training data beyond issues and commits while improving repair, construction, and terminal benchmarks.
Treat memory, context control, concurrency, and model routing as harness concerns. The workshop shows how these layers can improve agent behavior without changing model weights.
A safety prompt did not stop robot coding agents from colliding with obstacles. SafeHarness instead makes route verification, replanning, and contact constraints explicit in the execution loop.
Harness components pay off differently by model and budget. Elide before summarizing, use planning selectively, and avoid elaborate tools when the model is already strong with bash.
Delegating retrieval to a trained search subagent can keep noisy results out of the main agent’s context while cutting the token, latency, and cost burden of search-heavy workflows.
Knowledge agents need intent decomposition, specialist researchers, and multiple retrieval primitives; copying a coding-agent loop fails when evidence is ambiguous and context-dependent.
Jev gives agent loops typed decisions and probabilities without generating prose, offering a smaller control primitive for routing tools, retries, reviews, and guardrails.
A low-latency voice harness can pair fast provisional work with slower corrections. The design speculates on transcripts and responses, runs tools in parallel, and reuses cached speech prefixes.
Voice agents trade reasoning time for latency, then fail on transcription, data capture, and pronunciation. Treat inputs as typed fields, validate them, and normalize speech output yourself.
Stellar Colosseum coordinates parallel strategy search, falsification, decomposition, and verifier feedback for long research tasks. Its harness patterns may transfer to agents handling interdependent coding work.
HypoEvolve turns multi-agent scientific work into explicit population updates governed by a genetic algorithm. The result suggests agent collaboration is easier to test when selection and revision are formalized.
A GitHub review agent is rebuilt with progressively less orchestration code, ending as instructions and setup files around general tools. Evals still remain the builder’s job.
Vercel’s data agent improved after replacing prescriptive specialist pipelines with a filesystem, Bash and company-specific context. Eve packages that pattern as conventions.
PostHog treats every context source as part of an agent’s supply chain, scanning at build and use time while reserving blocking decisions for deterministic controls.
A production agent's harness is everything around the model: tools, memory, runtime, isolation, and deployment controls. The practical lesson is to standardize that layer across agents.
Dioxus found that generating more Rust was easy but merging it was not. Agents paid off on bounded research and maintenance work when architecture, real-device tests, and line-by-line review stayed human-owned.
AI SDK now runs GitHub Copilot behind the same HarnessAgent interface as nine other coding harnesses, using an official adapter and ACP connection.
Token counts do not show whether an agent was worth running. Define outcomes and cheap acceptance checks first, then reserve agents for work that is uncertain to execute but relatively easy to verify.