{
  "schema_version": "1.1",
  "id": "archive:https://www.youtube.com/watch?v=nxokqOq1imY",
  "slug": "your-agent-evolved-your-evals-didn-t-ameya-bhatawdekar-braintrust-1loaqv2",
  "url": "https://feed7.dev/p/your-agent-evolved-your-evals-didn-t-ameya-bhatawdekar-braintrust-1loaqv2",
  "title": "Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust",
  "why_included": "When an agent moves from fixed chains to model-led orchestration, memory, tools, and sandboxes, its old answer-quality evals cover only part of the system. Evals must follow the architecture.",
  "summary": "Evaluation scope expanded from final-answer checks in a **single model call**, to parser, retrieval, node, branch, and retry checks in orchestration graphs. Newer agent systems add model-led planning, long-horizon tool use, memory, sandboxes, MCP, and skills, creating further failure surfaces.",
  "practical_implication": "Map evals to the architecture actually in production. Measure trajectories and repeated-run reliability, preserve regression cases from older capabilities, and mine production traces for failure categories that existing datasets do not cover whenever the system is replatformed.",
  "agent_context": "Evaluation scope expanded from final-answer checks in a **single model call**, to parser, retrieval, node, branch, and retry checks in orchestration graphs. Newer agent systems add model-led planning, long-horizon tool use, memory, sandboxes, MCP, and skills, creating further failure surfaces.\n\nMap evals to the architecture actually in production. Measure trajectories and repeated-run reliability, preserve regression cases from older capabilities, and mine production traces for failure categories that existing datasets do not cover whenever the system is replatformed.\n\nProduction clustering can suggest missing cases, but it only sees observed traffic and depends on useful traces and scoring definitions. Architectural congruence improves coverage; it does not prove that the dataset represents rare or safety-critical failures.",
  "source": {
    "name": "AI Engineer",
    "url": "https://www.youtube.com/watch?v=nxokqOq1imY",
    "published_at": "2026-08-20T13:00:06.000Z"
  },
  "source_class": "video",
  "content_type": "Video",
  "layer": "benchmark",
  "domains": [
    "coding"
  ],
  "topics": [
    "agent-evals",
    "agent-reliability",
    "harness-engineering"
  ],
  "verification": {
    "status": "source_linked",
    "label": "Source Linked",
    "method": "source_feed",
    "verified_at": null
  },
  "uncertainty": [
    "Production clustering can suggest missing cases, but it only sees observed traffic and depends on useful traces and scoring definitions. Architectural congruence improves coverage; it does not prove that the dataset represents rare or safety-critical failures."
  ],
  "connected_context": {
    "meaning": "This reframes eval maintenance as an architectural migration obligation: when agents gain planning, memory, tools, retries, or sandboxes, final-answer datasets no longer cover the system being shipped. It confirms production traces as a source of regression cases and strengthens repeated-run and trajectory evaluation, while warning that observed traffic and chosen scorers still cannot establish coverage of rare or safety-critical failures.",
    "corpus_size": 525,
    "generated_at": "2026-08-21T10:08:41.370Z",
    "connections": [
      {
        "title": "From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=Ib5t2RLtxvM",
        "feed7_url": "https://feed7.dev/p/from-agent-traces-to-agent-simulations-rustem-feyzkhanov-snorkel-ai-0zwlzjq",
        "reason": "Replayable trace environments directly implement the call to evaluate trajectories and retries under fixed conditions after architectural changes."
      },
      {
        "title": "Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=jHMiYtjoJfA",
        "feed7_url": "https://feed7.dev/p/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-0uoems4",
        "reason": "Both warn that harness- or model-coupled evals decay after upgrades; this Signal adds the requirement to remap checks to every new architectural failure surface while retaining old regressions."
      },
      {
        "title": "SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=KMR_RBoCa4M",
        "feed7_url": "https://feed7.dev/p/simulationmaxxing-how-we-ship-agents-20-faster-aman-gupta-nubank-shreya-0r3nm6u",
        "reason": "Simulated multi-turn traces can extend coverage beyond observed production traffic, but both accounts preserve production comparison and human review as necessary checks on representativeness."
      },
      {
        "title": "Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=AQv3qRCG6Gw",
        "feed7_url": "https://feed7.dev/p/reinforcement-learning-without-verifiable-rewards-will-brown-prime-intel-0mauwh9",
        "reason": "Both mine traces for missing tasks and failure classes; the candidate clarifies that simulators and judge-derived rubrics introduce proxy and reward-hacking risks that architectural congruence alone does not solve."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-08-20T13:00:06.000Z",
  "modified_at": "2026-08-20T13:00:06.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/your-agent-evolved-your-evals-didn-t-ameya-bhatawdekar-braintrust-1loaqv2",
    "json": "https://feed7.dev/p/your-agent-evolved-your-evals-didn-t-ameya-bhatawdekar-braintrust-1loaqv2.json",
    "markdown": "https://feed7.dev/p/your-agent-evolved-your-evals-didn-t-ameya-bhatawdekar-braintrust-1loaqv2.md"
  }
}