SimulationMaxxing: How we ship agents 20× faster — Aman Gupta (Nubank) + Shreya Rajpal (Snowglobe)
Nubank uses simulated multi-turn traces to evaluate agent changes before production, shortening release cycles while checking simulation results against real data and human review.
Nubank reports using simulation across **five production agents** serving a bank with **135 million customers**. Simulated personas exercise real agents with mocked tools and consistent synthetic state, producing multi-turn traces that feed the existing evaluation pipeline.
Generate eval cases before production when real traces are slow or risky to collect. Nubank says this shortened its release cycle by about **20×**, enabled more experiments, caught regressions, and helped one agent double its customer-satisfaction measure.
Nubank reports using simulation across **five production agents** serving a bank with **135 million customers**. Simulated personas exercise real agents with mocked tools and consistent synthetic state, producing multi-turn traces that feed the existing evaluation pipeline. Generate eval cases before production when real traces are slow or risky to collect. Nubank says this shortened its release cycle by about **20×**, enabled more experiments, caught regressions, and helped one agent double its customer-satisfaction measure. Synthetic traces are useful only when the sim-to-real gap is measured. Domain experts judged **80%** of reviewed simulations usable, but teams still need online comparisons, human review, aligned metrics, and real production data.
This provides production-scale evidence that synthetic, state-consistent multi-turn traces can move evaluation earlier and accelerate agent iteration, even when real traces are scarce or risky. It also narrows the claim: simulation is an eval-data multiplier, not a substitute for production feedback, because usefulness and sim-to-real alignment still require expert review, online comparison, and real outcomes.