AI EngineerVideoSource Linked

Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust

When an agent moves from fixed chains to model-led orchestration, memory, tools, and sandboxes, its old answer-quality evals cover only part of the system. Evals must follow the architecture.

AI Engineer · Aug 20, 2026

Open Source Open Markdown Open JSON

Source Summary

Evaluation scope expanded from final-answer checks in a **single model call**, to parser, retrieval, node, branch, and retry checks in orchestration graphs. Newer agent systems add model-led planning, long-horizon tool use, memory, sandboxes, MCP, and skills, creating further failure surfaces.

Practical Implication

Map evals to the architecture actually in production. Measure trajectories and repeated-run reliability, preserve regression cases from older capabilities, and mine production traces for failure categories that existing datasets do not cover whenever the system is replatformed.

Agent-Ready Context

Evaluation scope expanded from final-answer checks in a **single model call**, to parser, retrieval, node, branch, and retry checks in orchestration graphs. Newer agent systems add model-led planning, long-horizon tool use, memory, sandboxes, MCP, and skills, creating further failure surfaces.

Map evals to the architecture actually in production. Measure trajectories and repeated-run reliability, preserve regression cases from older capabilities, and mine production traces for failure categories that existing datasets do not cover whenever the system is replatformed.

Production clustering can suggest missing cases, but it only sees observed traffic and depends on useful traces and scoring definitions. Architectural congruence improves coverage; it does not prove that the dataset represents rare or safety-critical failures.

Connected Context · Feed7 Judgment

This reframes eval maintenance as an architectural migration obligation: when agents gain planning, memory, tools, retries, or sandboxes, final-answer datasets no longer cover the system being shipped. It confirms production traces as a source of regression cases and strengthens repeated-run and trajectory evaluation, while warning that observed traffic and chosen scorers still cannot establish coverage of rare or safety-critical failures.

From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AIReplayable trace environments directly implement the call to evaluate trajectories and retries under fixed conditions after architectural changes.Designing Agents (The Floor Is the Frontier) — Ben Hylak, RaindropBoth warn that harness- or model-coupled evals decay after upgrades; this Signal adds the requirement to remap checks to every new architectural failure surface while retaining old regressions.SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)Simulated multi-turn traces can extend coverage beyond observed production traffic, but both accounts preserve production comparison and human review as necessary checks on representativeness.Reinforcement Learning without Verifiable Rewards — Will Brown, Prime IntellectBoth mine traces for missing tasks and failure classes; the candidate clarifies that simulators and judge-derived rubrics introduce proxy and reward-hacking risks that architectural congruence alone does not solve.

Context Map

benchmarkcoding#agent-evals#agent-reliability#harness-engineering

Uncertainty

Production clustering can suggest missing cases, but it only sees observed traffic and depends on useful traces and scoring definitions. Architectural congruence improves coverage; it does not prove that the dataset represents rare or safety-critical failures.