Agents That Self Improve While You Sleep - Fuad Ali, Arize AI
The workshop shows a loop that uses production traces to find recurring agent failures, propose code fixes, and turn failures into evals and regression data. Builders should keep human review and cost checks in that loop.
Arize’s Signal reads agent traces and groups recurring issues. In the demo, it found redundant product-search retries, traced them to a query casing mismatch, and proposed a repository fix. The presenter says the agent worked for **6 minutes** to find the cause.
For coding-agent systems, capture traces, preserve failure cases in a dataset, and turn useful findings into evals. Run experiments to check that a fix addresses the failure without breaking other behavior, and route uncertain results to a human reviewer.
Arize’s Signal reads agent traces and groups recurring issues. In the demo, it found redundant product-search retries, traced them to a query casing mismatch, and proposed a repository fix. The presenter says the agent worked for **6 minutes** to find the cause. For coding-agent systems, capture traces, preserve failure cases in a dataset, and turn useful findings into evals. Run experiments to check that a fix addresses the failure without breaking other behavior, and route uncertain results to a human reviewer. The loop adds cost through context, eval runs, and the Signal agent itself. The right sampling rate and configuration depend on the harness and use case; the workshop gives no general budget target, and proposed fixes still need review.
This describes a concrete trace-to-fix loop: cluster recurring failures, diagnose a specific cause, propose a repository change, then validate it through experiments and evals with human review. It reinforces production traces as improvement material while leaving sampling, cost, and general effectiveness dependent on the harness and task.