TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development
TraceML exposes why coding agents stall on long ML work: they repeat narrow optimization loops instead of alternating work types, revisiting discarded ideas, and pivoting like experts.
TraceML records **4,465 human Kaggle trajectories across 134 competitions**. Seven competitions also include **430 paired human and 207 agent trajectories**, with each code version labeled by action, intent, edit size, timing, score, and score effect.
For long-running ML agents, plan explicit shifts among data, validation, modeling, and ensembling, and preserve discarded approaches for later review. Codex concentrated on ensemble weighting and submission tuning, while MLEvolve repeatedly mutated its current model.
TraceML records **4,465 human Kaggle trajectories across 134 competitions**. Seven competitions also include **430 paired human and 207 agent trajectories**, with each code version labeled by action, intent, edit size, timing, score, and score effect. For long-running ML agents, plan explicit shifts among data, validation, modeling, and ensembling, and preserve discarded approaches for later review. Codex concentrated on ensemble weighting and submission tuning, while MLEvolve repeatedly mutated its current model. A short planning prompt moved named behaviors toward the human profile and improved scores, but the agents' overall effort patterns remained distinct. The paired analysis covers only seven competitions and two scaffolds.
TraceML supplies a concrete human baseline for evaluating how ML agents allocate effort, revise plans, and preserve alternatives, rather than judging only final scores. The planning prompt shows that scaffolding can shift named behaviors and improve outcomes, but persistent differences from human effort patterns and coverage of only two scaffolds across seven paired competitions limit generalization.