arXivPaperNeeds Review
TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development
TraceML exposes why coding agents stall on long ML work: they repeat narrow optimization loops instead of alternating work types, revisiting discarded ideas, and pivoting like experts.
arXiv
Source Summary
TraceML records **4,465 human Kaggle trajectories across 134 competitions**. Seven competitions also include **430 paired human and 207 agent trajectories**, with each code version labeled by action, intent, edit size, timing, score, and score effect.
Practical Implication
For long-running ML agents, plan explicit shifts among data, validation, modeling, and ensembling, and preserve discarded approaches for later review. Codex concentrated on ensemble weighting and submission tuning, while MLEvolve repeatedly mutated its current model.
Agent-Ready Context
TraceML records **4,465 human Kaggle trajectories across 134 competitions**. Seven competitions also include **430 paired human and 207 agent trajectories**, with each code version labeled by action, intent, edit size, timing, score, and score effect. For long-running ML agents, plan explicit shifts among data, validation, modeling, and ensembling, and preserve discarded approaches for later review. Codex concentrated on ensemble weighting and submission tuning, while MLEvolve repeatedly mutated its current model. A short planning prompt moved named behaviors toward the human profile and improved scores, but the agents' overall effort patterns remained distinct. The paired analysis covers only seven competitions and two scaffolds.
Context Map
benchmarkcodingdata#agent-evals#agent-reliability#harness-engineeringUncertainty
A short planning prompt moved named behaviors toward the human profile and improved scores, but the agents' overall effort patterns remained distinct. The paired analysis covers only seven competitions and two scaffolds.