# SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)

Source: [AI Engineer](https://www.youtube.com/watch?v=KMR_RBoCa4M)  
Feed7 permalink: https://feed7.dev/p/simulationmaxxing-how-we-ship-agents-20-faster-aman-gupta-nubank-shreya-0r3nm6u  
Published: 2026-07-29T19:00:06.000Z  
Trust: Source Linked (source_linked)

## Why Included

Nubank uses simulated multi-turn traces to evaluate agent changes before production, shortening release cycles while checking simulation results against real data and human review.

## Source Summary

Nubank reports using simulation across **five production agents** serving a bank with **135 million customers**. Simulated personas exercise real agents with mocked tools and consistent synthetic state, producing multi-turn traces that feed the existing evaluation pipeline.

## Practical Implication

Generate eval cases before production when real traces are slow or risky to collect. Nubank says this shortened its release cycle by about **20×**, enabled more experiments, caught regressions, and helped one agent double its customer-satisfaction measure.

## Agent-Ready Context

Nubank reports using simulation across **five production agents** serving a bank with **135 million customers**. Simulated personas exercise real agents with mocked tools and consistent synthetic state, producing multi-turn traces that feed the existing evaluation pipeline.

Generate eval cases before production when real traces are slow or risky to collect. Nubank says this shortened its release cycle by about **20×**, enabled more experiments, caught regressions, and helped one agent double its customer-satisfaction measure.

Synthetic traces are useful only when the sim-to-real gap is measured. Domain experts judged **80%** of reviewed simulations usable, but teams still need online comparisons, human review, aligned metrics, and real production data.

## Connected Context

Feed7 judgment across 297 accumulated Signals:

This provides production-scale evidence that synthetic, state-consistent multi-turn traces can move evaluation earlier and accelerate agent iteration, even when real traces are scarce or risky. It also narrows the claim: simulation is an eval-data multiplier, not a substitute for production feedback, because usefulness and sim-to-real alignment still require expert review, online comparison, and real outcomes.

- [From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI](https://feed7.dev/p/from-agent-traces-to-agent-simulations-rustem-feyzkhanov-snorkel-ai-0zwlzjq) — The approaches are complementary: Nubank generates cases before sufficient production data exists, while trace reconstruction turns observed production behavior into fixed replay environments.
- [Vending-Bench: Long-Horizon Agent Evals — Lukas Petersson, Andon Labs](https://feed7.dev/p/vending-bench-long-horizon-agent-evals-lukas-petersson-andon-labs-0fu78nz) — Vending-Bench reinforces the stated limitation that repeatable simulations must be paired with real-world tests because agent behavior can diverge outside the eval.
- [Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber](https://feed7.dev/p/building-closed-loop-evals-for-a-multimodal-agent-at-scale-soumya-gupta-1cqjbe2) — Uber’s closed production-feedback loop supplies the downstream mechanism needed to measure and correct the sim-to-real gap.
- [State of Data — Sean Cai, Independent / State of Data](https://feed7.dev/p/state-of-data-sean-cai-independent-state-of-data-0v9fy69) — The candidate’s preference for live workflow trajectories qualifies Nubank’s synthetic-first method: generated traces speed early testing, but real trajectories remain necessary evidence.

## Context Map

- Layer: benchmark
- Domains: None
- Topics: agent-evals, agent-reliability, harness-engineering

## Uncertainty

- Synthetic traces are useful only when the sim-to-real gap is measured. Domain experts judged **80%** of reviewed simulations usable, but teams still need online comparisons, human review, aligned metrics, and real production data.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.