Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain
Agent quality problems often live in traces, not source code. Mine production runs for recurring failures, compaction drift, and counterfactual model choices, then validate changes with replayable evals.
Agents emit tool calls, messages, API activity, and CLI actions that reveal behavior invisible in source inspection. The proposed loop is **ship, collect traces, mine them, then run experiments** against the failures and patterns found.
Turn tracing on early and use agents to search large trace collections for user friction, post-compaction degradation, model substitutions, and candidate eval cases. Use those findings to update prompts, tools, orchestration, memory, or narrowly fine-tuned models.
Agents emit tool calls, messages, API activity, and CLI actions that reveal behavior invisible in source inspection. The proposed loop is **ship, collect traces, mine them, then run experiments** against the failures and patterns found. Turn tracing on early and use agents to search large trace collections for user friction, post-compaction degradation, model substitutions, and candidate eval cases. Use those findings to update prompts, tools, orchestration, memory, or narrowly fine-tuned models. Trace mining becomes expensive when runs contain millions of tokens or collections reach gigabytes or terabytes. Automated reviewers can also optimize the wrong signal, so dense feedback and independent checks remain necessary.
This turns observability from postmortem tooling into the data source for agent improvement: production traces reveal failure modes, seed evals, and guide changes to prompts, tools, memory, orchestration, or models. It confirms live workflow data as valuable evidence while narrowing automation claims through trace-scale costs and the risk that reviewers optimize misleading proxies.