# Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust

Source: [AI Engineer](https://www.youtube.com/watch?v=nxokqOq1imY)  
Feed7 permalink: https://feed7.dev/p/your-agent-evolved-your-evals-didn-t-ameya-bhatawdekar-braintrust-1loaqv2  
Published: 2026-08-20T13:00:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

When an agent moves from fixed chains to model-led orchestration, memory, tools, and sandboxes, its old answer-quality evals cover only part of the system. Evals must follow the architecture.

## Source Summary

Evaluation scope expanded from final-answer checks in a **single model call**, to parser, retrieval, node, branch, and retry checks in orchestration graphs. Newer agent systems add model-led planning, long-horizon tool use, memory, sandboxes, MCP, and skills, creating further failure surfaces.

## Practical Implication

Map evals to the architecture actually in production. Measure trajectories and repeated-run reliability, preserve regression cases from older capabilities, and mine production traces for failure categories that existing datasets do not cover whenever the system is replatformed.

## Agent-Ready Context

Evaluation scope expanded from final-answer checks in a **single model call**, to parser, retrieval, node, branch, and retry checks in orchestration graphs. Newer agent systems add model-led planning, long-horizon tool use, memory, sandboxes, MCP, and skills, creating further failure surfaces.

Map evals to the architecture actually in production. Measure trajectories and repeated-run reliability, preserve regression cases from older capabilities, and mine production traces for failure categories that existing datasets do not cover whenever the system is replatformed.

Production clustering can suggest missing cases, but it only sees observed traffic and depends on useful traces and scoring definitions. Architectural congruence improves coverage; it does not prove that the dataset represents rare or safety-critical failures.

## Connected Context

Feed7 judgment across 525 accumulated Signals:

This reframes eval maintenance as an architectural migration obligation: when agents gain planning, memory, tools, retries, or sandboxes, final-answer datasets no longer cover the system being shipped. It confirms production traces as a source of regression cases and strengthens repeated-run and trajectory evaluation, while warning that observed traffic and chosen scorers still cannot establish coverage of rare or safety-critical failures.

- [From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI](https://feed7.dev/p/from-agent-traces-to-agent-simulations-rustem-feyzkhanov-snorkel-ai-0zwlzjq) — Replayable trace environments directly implement the call to evaluate trajectories and retries under fixed conditions after architectural changes.
- [Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop](https://feed7.dev/p/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-0uoems4) — Both warn that harness- or model-coupled evals decay after upgrades; this Signal adds the requirement to remap checks to every new architectural failure surface while retaining old regressions.
- [SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)](https://feed7.dev/p/simulationmaxxing-how-we-ship-agents-20-faster-aman-gupta-nubank-shreya-0r3nm6u) — Simulated multi-turn traces can extend coverage beyond observed production traffic, but both accounts preserve production comparison and human review as necessary checks on representativeness.
- [Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect](https://feed7.dev/p/reinforcement-learning-without-verifiable-rewards-will-brown-prime-intel-0mauwh9) — Both mine traces for missing tasks and failure classes; the candidate clarifies that simulators and judge-derived rubrics introduce proxy and reward-hacking risks that architectural congruence alone does not solve.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, agent-reliability, harness-engineering

## Uncertainty

- Production clustering can suggest missing cases, but it only sees observed traffic and depends on useful traces and scoring definitions. Architectural congruence improves coverage; it does not prove that the dataset represents rare or safety-critical failures.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.