# Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain

Source: [AI Engineer](https://www.youtube.com/watch?v=CvRngaQZQ3Y)  
Feed7 permalink: https://feed7.dev/p/improving-agents-is-a-data-mining-problem-vivek-trivedy-langchain-1cgibr8  
Published: 2026-08-12T19:00:01.000Z  
Trust: Source Linked (source_linked)

## Why Included

Agent quality problems often live in traces, not source code. Mine production runs for recurring failures, compaction drift, and counterfactual model choices, then validate changes with replayable evals.

## Source Summary

Agents emit tool calls, messages, API activity, and CLI actions that reveal behavior invisible in source inspection. The proposed loop is **ship, collect traces, mine them, then run experiments** against the failures and patterns found.

## Practical Implication

Turn tracing on early and use agents to search large trace collections for user friction, post-compaction degradation, model substitutions, and candidate eval cases. Use those findings to update prompts, tools, orchestration, memory, or narrowly fine-tuned models.

## Agent-Ready Context

Agents emit tool calls, messages, API activity, and CLI actions that reveal behavior invisible in source inspection. The proposed loop is **ship, collect traces, mine them, then run experiments** against the failures and patterns found.

Turn tracing on early and use agents to search large trace collections for user friction, post-compaction degradation, model substitutions, and candidate eval cases. Use those findings to update prompts, tools, orchestration, memory, or narrowly fine-tuned models.

Trace mining becomes expensive when runs contain millions of tokens or collections reach gigabytes or terabytes. Automated reviewers can also optimize the wrong signal, so dense feedback and independent checks remain necessary.

## Connected Context

Feed7 judgment across 462 accumulated Signals:

This turns observability from postmortem tooling into the data source for agent improvement: production traces reveal failure modes, seed evals, and guide changes to prompts, tools, memory, orchestration, or models. It confirms live workflow data as valuable evidence while narrowing automation claims through trace-scale costs and the risk that reviewers optimize misleading proxies.

- [From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize](https://feed7.dev/p/from-signal-to-pr-anatomy-of-a-self-improving-agent-jason-lopatecki-ariz-1i6f0ba) — Extends the trace-mining loop from diagnosis into investigated issues and draft fixes, while retaining stronger human review as the resulting changes grow.
- [State of Data — Sean Cai, Independent / State of Data](https://feed7.dev/p/state-of-data-sean-cai-independent-state-of-data-0v9fy69) — Reinforces production trajectories as richer evidence than manufactured tasks and adds that conclusions should be tested across scaffolds because harness changes can shift results.
- [Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop](https://feed7.dev/p/designing-agents-the-floor-is-the-frontier-ben-hylak-raindrop-0uoems4) — Narrows trace mining toward failures with measurable onset and reach, then anchors resulting checks in code so they survive model or harness upgrades.
- [Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect](https://feed7.dev/p/reinforcement-learning-without-verifiable-rewards-will-brown-prime-intel-0mauwh9) — Carries mined failures into simulation and training, but confirms the warning that automated judges and proxy rewards can optimize the wrong behavior.

## Context Map

- Layer: agent
- Domains: coding, data
- Topics: observability, agent-evals, harness-engineering

## Uncertainty

- Trace mining becomes expensive when runs contain millions of tokens or collections reach gigabytes or terabytes. Automated reviewers can also optimize the wrong signal, so dense feedback and independent checks remain necessary.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.