# From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

Source: [AI Engineer](https://www.youtube.com/watch?v=Ib5t2RLtxvM)  
Feed7 permalink: https://feed7.dev/p/from-agent-traces-to-agent-simulations-rustem-feyzkhanov-snorkel-ai-0zwlzjq  
Published: 2026-07-25T01:00:02.000Z  
Trust: Source Linked (source_linked)

## Why Included

Reconstructing production traces as replayable environments lets teams compare agent stacks under fixed conditions and gate releases on task outcomes, cost, latency, and retries.

## Source Summary

The proposed benchmark turns **production traces** into **offline simulations** by rebuilding the database state, tools, APIs, files, and policies an agent encountered. Verifiers inspect the final state, trace, and artifacts, while an Oracle confirms each task is solvable.

## Practical Implication

Treat this benchmark like integration-test software: pin dependencies, run it in CI, and use it as a **release gate**. Compare whole agent configurations—not only models—and place fixes in the harness, skills, prompts, or output schema according to the failure.

## Agent-Ready Context

The proposed benchmark turns **production traces** into **offline simulations** by rebuilding the database state, tools, APIs, files, and policies an agent encountered. Verifiers inspect the final state, trace, and artifacts, while an Oracle confirms each task is solvable.

Treat this benchmark like integration-test software: pin dependencies, run it in CI, and use it as a **release gate**. Compare whole agent configurations—not only models—and place fixes in the harness, skills, prompts, or output schema according to the failure.

Faithful reconstruction is substantial engineering work, and simulated users or mocked services may diverge from production. Keep a held-out set, cover routine and edge cases, and route verifier disagreements to subject-matter experts.

## Context Map

- Layer: benchmark
- Domains: coding
- Topics: agent-evals, agent-reliability, harness-engineering

## Uncertainty

- Faithful reconstruction is substantial engineering work, and simulated users or mocked services may diverge from production. Keep a held-out set, cover routine and edge cases, and route verifier disagreements to subject-matter experts.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.