Harness Engineering

SafeEvolve turns completed agent trajectories into reversible harness updates and policy training, linking runtime safeguards with learned multi-step behavior.

Agent authorization should scale with stakes and reversibility: logs may cover coding changes, while open-ended payments need scoped, time-bound, independently verifiable mandates.

CordisBench finds that models lose reliability as harness lifecycle interactions grow, while deterministic execution semantics can replace costly reasoning on controlled cases.

This survey separates verbal feedback by when it acts and what it changes, helping agent builders distinguish task specification, test-time guidance, and parameter updates.

For fuzzy requests, an agent should identify the missing constraint with the most decision value, elicit it in the right modality, then choose a response format suited to the task.

Agent checkout needs more than payment tools: enforce user-set limits outside the agent, expose structured merchant data, honor cancellation, and log every decision.

A two-stage ensemble reconciles outputs from string, graph-embedding, and RAG-based ontology aligners. Composition matters: mixed paradigms favor precision, while LLM-only groups more often favor F1.

Vercel found that prose alone produced inconsistent agent-made pages, then paired design.md with fixed CSS primitives and repeatable evals to encode brand judgment.

An AWS demo treats a robot as an agent tool: an LLM selects existing hardware policies while separate agents handle thinking, communication, and voice. The live failures expose the safety gap.

Maersk’s production agents depend less on a clever loop than on executable SOPs, bounded tools, replayable traces, and a correction system shared by experts and engineers.

Navan treats production agents as stateful services needing persistence, composable skills, traces, authorization gates, and trajectory-based tests before multi-agent complexity.

Once agents mutate external state, timeouts mean unknown outcomes. Builders need idempotent tools, bounded retries, scoped credentials, durable traces, and explicit recovery paths.

Agent architectures are expiring quickly. Keep model, search, and orchestration choices replaceable, and evaluate platforms by how well they handle repeated change.

Shared agent skills can turn team know-how into portable workflows, but unmanaged catalogs create duplication and policy risk. Treat skills as versioned, evaluated dependencies.

Figma’s cautious agent adopters expose missing safeguards, not resistance. Use their feedback to improve verification, plans, review size, and the labeling of AI-generated work.

Amazon’s pilot suggests coding-agent gains come from redesigning the workflow: prepare agent-readable context, fast local feedback, scoped specs, and parallel work instead of constant chat.

A study of 8,351 Claude Code plugins finds instruction files and scripts often require coordinated maintenance, treating skills as evolving software rather than static prompts.

Persona-Execution Separation isolates mutable agent behavior from audited, stateful work through a governed bridge. It is an architecture option for multi-user agents under compliance constraints.

Mike Krieger describes shifting from stepwise prompting to goal-level delegation, backed by verification, observability, feature flags, and frequent decisions to continue or stop experiments.

Vercel’s AI SDK can now run Cursor behind the same HarnessAgent interface as seven other coding agents, reducing application changes when swapping harnesses.

TraceML exposes why coding agents stall on long ML work: they repeat narrow optimization loops instead of alternating work types, revisiting discarded ideas, and pivoting like experts.

Cloudflare’s GTM workflow shows a reusable agent pattern: preprocess business data, encode domain knowledge in skills, then separate drafting, verification, and tone.

Exa’s internal agent clone combines writing examples, decision evals, and permission-scoped tools—a concrete pattern for encoding judgment without sharing full access.

Snowflake’s rollout favors narrow, high-accuracy coverage, staged adoption, and log-driven iteration over connecting every data source before launch.

Ramp’s GTM agents share a customer-data layer, durable execution, tools, and skills. The reusable pattern is to automate one narrow workflow, then extend the same substrate across teams.

Clay frames GTM agents as a data and orchestration problem: resolve entities, evaluate providers, isolate agent-written state, and account for delayed syncs before automating decisions.

Reliable background agents need runtime primitives more than elaborate framework abstractions: durable events, queues, versioned inputs, causal traces, diffs, and replay.

Agent adoption becomes a team-systems problem: improve shared context and harnesses, assign platform ownership, and measure fewer human interventions instead of individual prompt speed.

Run-level policies cut benchmarked agent spend while completing more runs, suggesting builders should steer loops, context, and tool output before enforcing hard caps.

Production agents need bounded write authority, infrastructure-stamped identity, and human-only overrides; a broad token and tool list cannot control blast radius.

Self-refinement pipelines need not use equally capable models: invest capacity in generation and revision, while a small critic may preserve gains at lower compute cost.

Re³Cap uses multimodal retrieval to find caption omissions and hallucinations before refinement, offering a concrete retrieval-and-review pattern for vision agents.

Hugging Face automated research-artifact outreach with a CLI, one skill, and a sandbox. The case shows when an agent can replace custom workflow code, but undisclosed automated outreach raises trust questions.

Compound engineering treats each agent mistake and product judgment as reusable context. The useful discipline is to invest in memory and review so the next feature becomes easier, not merely faster.

Agent instructions are not security boundaries. Put identity, scoped capabilities, policy gates, audit receipts, and revocation outside the model before granting production access.

A manager can turn fragmented time into agent steering: prepare context, run bounded work overnight, then personally test and review small PRs before involving the team.

When an agent moves from fixed chains to model-led orchestration, memory, tools, and sandboxes, its old answer-quality evals cover only part of the system. Evals must follow the architecture.

Maven’s agent-heavy workflow shortens planning, caps reviewable PR size, and tests nondeterministic features repeatedly. The bottleneck shifts from implementation to decisions and verification.

Vertical agents need narrow jobs, proprietary data, observability, and expert judgment. Generic models and self-grading cannot establish whether domain-specific output is actually useful.

Healthcare agents can use X12 transactions as a typed contract across calls, portals, and claims. The standard constrains actions and validation, but payer data can still be stale or wrong.