# Agents That Self Improve While You Sleep - Fuad Ali, Arize AI

Source: [AI Engineer](https://www.youtube.com/watch?v=UGuNt2_Oz8s)  
Feed7 permalink: https://feed7.dev/p/agents-that-self-improve-while-you-sleep-fuad-ali-arize-ai-19hmq9b  
Published: 2026-10-26T23:00:39.000Z  
Trust: Source Linked (source_linked)

## Why Included

The workshop shows a loop that uses production traces to find recurring agent failures, propose code fixes, and turn failures into evals and regression data. Builders should keep human review and cost checks in that loop.

## Source Summary

Arize’s Signal reads agent traces and groups recurring issues. In the demo, it found redundant product-search retries, traced them to a query casing mismatch, and proposed a repository fix. The presenter says the agent worked for **6 minutes** to find the cause.

## Practical Implication

For coding-agent systems, capture traces, preserve failure cases in a dataset, and turn useful findings into evals. Run experiments to check that a fix addresses the failure without breaking other behavior, and route uncertain results to a human reviewer.

## Agent-Ready Context

Arize’s Signal reads agent traces and groups recurring issues. In the demo, it found redundant product-search retries, traced them to a query casing mismatch, and proposed a repository fix. The presenter says the agent worked for **6 minutes** to find the cause.

For coding-agent systems, capture traces, preserve failure cases in a dataset, and turn useful findings into evals. Run experiments to check that a fix addresses the failure without breaking other behavior, and route uncertain results to a human reviewer.

The loop adds cost through context, eval runs, and the Signal agent itself. The right sampling rate and configuration depend on the harness and use case; the workshop gives no general budget target, and proposed fixes still need review.

## Connected Context

Feed7 judgment across 955 accumulated Signals:

This describes a concrete trace-to-fix loop: cluster recurring failures, diagnose a specific cause, propose a repository change, then validate it through experiments and evals with human review. It reinforces production traces as improvement material while leaving sampling, cost, and general effectiveness dependent on the harness and task.

- [Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop](https://feed7.dev/p/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-0uoems4) — Both use production failures to guide evaluation and improvement; the candidate emphasizes prioritizing anomalies by onset and reach and preserving regression tests, which can make Signal’s clustering findings more actionable.
- [From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI](https://feed7.dev/p/from-agent-traces-to-agent-simulations-rustem-feyzkhanov-snorkel-ai-0zwlzjq) — Signal turns traces into proposed fixes and eval cases, while the candidate adds replayable environments for comparing agent stacks under fixed conditions; replay could help test whether a proposed change resolves a failure.
- [Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust](https://feed7.dev/p/your-agent-evolved-your-evals-didn-t-ameya-bhatawdekar-braintrust-1loaqv2) — Both argue that evals should track evolving agent behavior; Signal provides recurring trace-derived cases, while the candidate warns that architecture changes require broader trajectory and repeated-run evaluation.

## Context Map

- Layer: agent
- Domains: coding
- Topics: agent-reliability, agent-evals, harness-engineering

## Uncertainty

- The loop adds cost through context, eval runs, and the Signal agent itself. The right sampling rate and configuration depend on the harness and use case; the workshop gives no general budget target, and proposed fixes still need review.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.