Source-linked AI summary
TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development
Jiarui Yan, Weiwei Sun, Sijie Li, Wenhan Li, Yiming Yang
TL;DR
Autonomous machine-learning development remains weaker than strong human work, while outcome-based benchmarks provide little evidence about the process behind that gap. TraceML compares human and agent trajectories through a shared version-level schema and finds that agents follow narrower development patterns; planning prompts shift some behaviors and scores, but effort remains agent-shaped.
Problem
Outcome-based benchmarks show agents underperforming strong humans on autonomous machine-learning development but provide limited evidence about the underlying development process.
Method
TraceML represents human and agent runs on the same competitions as ordered, version-level trajectories with scores, timestamps, actions, intents, edit sizes, and score effects.
Results
Agents develop through narrower patterns than humans, while a planning harness narrows some behavioral differences and lifts scores on some competitions without changing the agent-shaped effort profile.
Takeaways & Limitations
TraceML makes agent development comparable to human practice and shows that instruction shifts checklist-like behaviors while leaving deeper planning differences for agent design.
Takeaways & Limitations
Agents lack searchable access to their own earlier states, so they do not revisit work they have set aside.
Abstract
from arXiv · showhide
Large language models write correct code for isolated problems but remain far weaker at autonomous machine-learning development, where an agent must revise data pipelines, models, and validation over hours of feedback, and on most competitions still finishes below strong human competitors. Outcome-based benchmarks record this gap but not its cause, because they grade the final submission and discard the development process behind it. We introduce TraceML, which pairs human and agent work on the same competitions under one version-level schema: 4,465 human Kaggle trajectories across 134 competitions, seven of which are also worked by two agent scaffolds, giving 430 paired human and 207 agent trajectories. Every code version carries its score, its timestamp, and labels for the action taken, its intent, the edit size, and the score effect. Read this way, the gap becomes concrete. Experts alternate data work, validation, model changes, and ensembling, and return to approaches they had set aside. Each agent scaffold instead collapses into a narrow loop: Codex spends its steps re-weighting ensembles and tuning submissions, MLEvolve mutates its model in place, and neither pivots at the human rate nor reopens abandoned work. A short planning prompt distilled from human practice moves the behaviors it names toward the human profile and lifts scores, but the effort profile stays agent-shaped: instruction closes only the part of the gap that reduces to instructions. We release the corpus, the schema, the labelers, and the extraction pipeline at https://huggingface.co/datasets/jerryyan/TraceML.
1 Introduction
TraceML exposes why autonomous machine-learning agents lag strong human competitors by representing development as comparable, version-level processes rather than final outcomes alone. Its trajectories reveal distinct exploration and optimization behaviors, while a human-derived planning harness narrows some behavioral differences and improves scores.
- Motivation: Outcome-based benchmarks cannot explain process differences because they grade final submissions while discarding the edits and decisions that produced them.Runs with identical scores can reflect careful experimentation or blind tuning.
- TraceML framework: TraceML represents human and agent development as ordered code-version trajectories with scores, timestamps, action and intent labels, edit sizes, and score effects.The unified schema covers Kaggle notebooks, CLI commits, and tree-search journals.
- Process differences: Humans alternate exploration, diagnosis, validation, model changes, and ensembling, whereas Codex emphasizes submission bookkeeping and MLEvolve emphasizes local model and training mutations.Both agent scaffolds spend more budget than humans for each unit of newly covered ground.
- Planning intervention: A planning harness derived from human workflow observations narrows some behavioral differences and lifts scores on part of the competitions.The intervention improves named behaviors without eliminating the broader process gap.
2 Related Work
Prior ML-agent benchmarks measure outcomes, search operators, or abstract plans, but generally do not preserve how humans develop toward comparable scores. TraceML addresses this gap by retaining scored intermediate versions, their code, and task-matched human development across 134 competitions.
- ML-agent development: ML agents interleave reasoning, tool use, environment feedback, data inspection, coding, experiments, and self-improvement, while Codex and MLEvolve differ in their search scaffolds.Codex uses a single edit-run-observe loop; the supplied passage introduces this distinction but truncates the remainder of Codex’s description.
- Existing benchmarks: MLAgentBench, MLE-bench, and AIRA evaluate bounded workflows, historical Kaggle competitions, or search operators and validation feedback, without recording how people reached comparable scores.These benchmarks compare agents with other agents or leaderboard positions, rather than preserving human development trajectories.
- TraceML’s distinction: TraceML is the only listed entry preserving scored intermediate versions, their code, and task-matched human development across 134 competitions.RE-Bench comes closest by pairing time-budgeted human and agent attempts, but uses seven bespoke environments with partial code; TraceML keeps full code for every scored version.
- Planning mechanisms: Tree of Thoughts and Reflexion seek more deliberate multi-step behavior, but evaluations mostly test abstract plans in classical planning domains rather than hours of real ML development.Human comparison provides a reference point, with RE-Bench showing that time-bounded human and agent performance can be compared.
3 TraceML
TraceML reconstructs human and agent development as comparable version-level trajectories, pairing 430 human trajectories with agent runs across seven competitions. Its competition-agnostic schema labels pipeline state and transition behavior while accounting for scoring, observability, and licensing constraints.
- Trajectory reconstruction: TraceML converts notebook histories, Git commits, and search journals into ordered trajectories of scored, timestamped code versions.The representation aligns human saves with Codex commits and MLEvolve search nodes, collapsing adjacent identical states and preserving branch structure.
- Trajectory reconstruction: 430 human trajectories across seven competitions form the paired subset for comparisons with 11 baseline Codex, 7 prompted Codex, and 13 MLEvolve runs.The paired subset spans all human author tiers and uses a twelve-hour agent budget.
- Measurement and limitations: Both sources use competition metrics, with behavior-outcome analyses reported in within-competition percentiles because metrics differ across competitions.Two retention checks show headline gaps survive, although off-platform human work remains unobserved and the corpus is therefore a reference distribution of public practice.
- Behavioral schema: The schema describes versions with eight coarse pipeline stages and 136 fine tags, while transitions receive action, intent, magnitude, and score-effect labels.Labels are derived from diffs, surrounding code, and the grader, making the schema applicable across competitions and other compatible scaffolds.
- Labeling and release: Open-weight labelers replace infeasible hand-labeling of 151,088 versions, using a teacher and two Qwen3-1.7B students evaluated for compliance, agreement, and behavioral fidelity.Agreement is lowest on intent.
- Labeling and release: TraceML releases annotations, schemas, code, and labeler weights under stated licenses while excluding Kaggle competition data and redistributing only permissively licensed notebook source.Human source licensing is verified per kernel, and retained usernames and tiers come from public Meta Kaggle metadata.
4 Empirical Findings
Human and agent development trajectories differ most in fine-grained actions, directional pivots, revisiting abandoned work, and ensemble construction rather than in their coarse action mix alone. Codex rarely changes direction or returns to prior approaches, while MLEvolve pivots frequently without corresponding gains.
- Action profiles: Fine-grained action usage separates agents from humans: humans alternate data, validation, modeling, checkpoints, and ensembling, while each scaffold settles into a narrow behavioral band.Codex disproportionately re-weights ensembles, stacks models, adds members, and tweaks post-processing.
- Directional pivots: Humans pivot on 25% of transitions, versus 9% for Codex and 58% for MLEvolve; matched-state comparisons still show Codex being out-pivoted three to one.Human post-pivot steps average +0.089, compared with −0.008 for MLEvolve, indicating that frequent pivots do not necessarily improve outcomes.
- Search memory: Top humans return to earlier work on 9% of eligible versions, with 78% of those returns ending above the revisited version, while Codex returns once and MLEvolve never.The agents can recover from setbacks by tuning forward, but they do not reopen abandoned approaches.
- Ensembling: Among human ensemble edits, adding or changing a member raises the chance that the next version improves by 6.4 points, whereas re-weighting alone lowers it by 5.8 points.Codex performs 78% of its ensemble edits by re-weighting a member set it never grows; MLEvolve mostly averages seeds, while top humans more often add members.
5 From Human-Agent Gaps to a Planning Harness
A compact planning prompt shifts several agent behaviors toward the human profile and improves scores on most competitions, but it cannot reach behaviors already present or replace prohibitions with destinations. The intervention therefore probes which parts of the human-agent gap reduce to instructions and which require changing the agent itself.
- Prompt design: A roughly one-thousand-token skill combines anti-loop constraints with human-prior practices such as early K-fold validation, ensembling, cached out-of-fold predictions, and multi-seed or multi-model blending.The constraints target single-holdout validation, repeated tuning, and large first-version rewrites before a working baseline exists.
- Experiment setup: Across all 7 paired competitions at a fixed 12-hour budget, the study varies only the prompt while measuring trajectory behavior and best valid held-out score against human percentile bands.The backend, tools, extraction pipeline, and grader are held fixed.
- Behavioral results: Three behaviors move onto the human value: re-weighting falls roughly fivefold, attention shifts toward ensembling, and small early edits rise from near zero to above the top human rate.The agent stops re-weighting an ensemble when it should add a member and begins making small early edits it had almost entirely skipped.
- Limitations: The prompt overshoots by driving plain hold-out use to zero, while K-fold averaging and persisted out-of-fold predictions do not move because Codex already met or exceeded human rates.A prohibition supplies direction without a destination, and prescribing behavior already present cannot shift it further.
- Score results: Five of the seven competitions improve, two are within noise, and none regress; removing planning content while preserving injection timing eliminates the gains.The results indicate that improvement comes from the skill’s content rather than repetition frequency.
- Interpretation: Instruction transfers only when it names an unmet behavioral level, making the prompt a probe of what instructions can reach versus what requires changing the agent itself.Directions without destinations and practices already beyond the human value do not transfer.
6 Discussion and Future Work
TraceML identifies searchable history and state-aware control as design problems behind agents’ weaker development behavior. Final scores and practice checklists cannot distinguish these process failures.
- Memory: Agents do not revisit abandoned work, indicating a searchable history mechanism is needed rather than a different recovery capability.Codex recovers from setbacks as well as top humans and benefits when it rarely changes direction, but lacks searchable access to its earlier run states.
- Control: Both scaffolds miss the human pivot rate from opposite sides, while only human pivots pay, motivating a controller that reads the run’s current state.Each scaffold follows one policy throughout, suggesting control—not merely a stronger base model—as the target for intervention.
- Evaluation: Final scores cannot separate memory and control failures, and practice checklists cannot distinguish the three cohorts.The discussion argues that process-level evidence is necessary because outcome scores and binary practice indicators conceal how development unfolds.
7 Conclusion … B.3 Unit Alignment Statistics
TraceML compares human and agent development through version-level trajectories, revealing distinct mismatches between agent behavior and the human profile. Its appendix audits retention, reuse, alignment, and extraction choices underlying these comparisons.
- 7 Conclusion: TraceML pairs human and agent development on the same competitions under one version-level schema, exposing different development patterns rather than one clean gap.The two scaffolds miss the human profile from opposite sides: one tunes without changing direction, while the other changes direction without consolidating.
- B Dataset Construction and Alignment: The appendix documents retention, reuse, unit alignment, run inclusion, extraction tooling, and a worked pipeline example supporting the main analyses.These materials correspond to Sections B.1 through B.6 and support Sections 3.1–3.3.
- B.1 Human Retention Filter: Audit of Removed Kernels: The human retention filter requires in-window versions, at least 5 versions spanning at least 3 days with at least 1 score, and no near-static resubmission.The filter does not test phase variety, action diversity, or intent.
- B.1 Human Retention Filter: Audit of Removed Kernels: 100% of runs in both agent cohorts pass the one-score condition, while 9 of 11 Codex runs and 111 of 189 MLEvolve branches pass the five-version condition.The three-day condition does not apply to runs measured in hours; restricting to filter-passers preserves every Section 4 gap and increases Codex’s action-space distance from 0.077 to 0.095 bits.
- B.1 Human Retention Filter: Audit of Removed Kernels: The retention rule does not favor the human side.The removed set is described as dominated by post-deadline write-ups rather than weak development trajectories, with a medal rate comparable to the retained set.
- B.2 Fork and Reuse Rates: Fork lineage and copied public baselines can make Kaggle trajectory units statistically dependent and limit how much recorded human work is original.Table 4 measures visible reuse in both the retained corpus and the paired human-agent sample; near-duplicate code is defined as similarity ≥0.9, with observed links confined within competitions.
- B.2 Fork and Reuse Rates: Uncertainty estimates cluster by competition, human fork-lineage group, and agent run, with MLEvolve branches sharing tree nodes resampled together.This accounts for dependence induced by fork lineage and shared baselines.
- B.3 Unit Alignment Statistics: 1,290 graded commits reduce to 16 versions in one representative Codex run after adjacent identical-code commits are collapsed.The version unit treats submission-producing agent commits as the analogue of deliberate Kaggle save-versions rather than raw log entries.
B.4 Analysis Scope and Cohort Composition … C Annotation Schema and Reliability
TraceML separates corpus-wide, paired, and twelve-hour scopes, then standardizes extraction, worked examples, and annotation-reliability reporting across human and agent trajectories. The analysis emphasizes run-level uncertainty, concrete development decisions, and a reusable cross-scaffold pipeline.
- B.4 Analysis Scope and Cohort Composition: TraceML uses three nested scopes: the full corpus, the seven-competition paired subset, and a twelve-hour scope covering the same seven competitions and 430 human trajectories.Behavioral comparisons in §4 are computed within the twelve-hour scope, while the paired subset fixes the human reference for agent comparisons.
- B.4 Analysis Scope and Cohort Composition: Human trajectories are split into a top cohort and the rest by leaderboard rank, while MLEvolve’s shared tree nodes make run count—not branch count—the effective sample size.The cohort split and effective-sample-size rule are specified in Table 6.
- Per-run composition of the Codex cohort: Pooling transitions would overweight short runs, so §4 intervals are clustered by run and matched-state analysis reports a run-clustered confidence interval.This design prevents a seven-transition run from contributing as much per observation as a thousand-transition run.
- B.5 Cross-Scaffold Extraction Toolkit: A single command extracts, grades, labels, and reports TraceML trajectories for Codex CLI, MLEvolve, AIDE, Claude Code, and Gemini CLI.The toolkit processes any CLI-agent run directory and produces reports against released human cohorts in minutes on one A6000 GPU.
- B.6 Worked Example of the Pipeline: A retained Grandmaster trajectory contains 20 saved versions and 19 transitions, with Table 8 representing four consecutive middle transitions around its best score.Each transition records actions, intents, magnitude, and leaderboard score effect.
- B.6 Worked Example of the Pipeline: The worked trajectory alternates model optimization with debugging and retains a regressing step, a development pattern largely absent from agents in §4.Its four decisions switch to roberta-large, inject target-noise sampling, correct validation-label extraction, and train longer on the base checkpoint.
- C Annotation Schema and Reliability: Annotation reliability covers teacher stability, cross-model agreement, and teacher–student transfer using Cohen’s κ and multi-label Jaccard on held-out state and action annotations.Teacher–student scores compare released Qwen3-1.7B labelers with the gpt-5.4-mini teacher, and Figure 6 illustrates schema output for one transition.
C.1 Intent Label Audit by Cohort · D Additional Statistics for the Behavioral Findings
The intent-label audit finds substantial annotator and human agreement, with conclusions unchanged under alternative class groupings and a label-free check. Additional statistics examine fairness treatments, action distributions, and two forms of returning to earlier work.
- C.1 Intent Label Audit by Cohort: Table 10 reports the human audit of released intent labels stratified across three cohorts.The table provides the cohort-level audit underlying the intent-label assessment.
- C.1 Intent Label Audit by Cohort: κ=0.611 measures cross-model agreement on 499 held-out transitions when unparseable outputs count as disagreements.Among items with two valid labels, agreement rises to κ=0.724.
- C.1 Intent Label Audit by Cohort: 81% human-audit agreement with released labels is observed across 100 audited items, with κ=0.68.The audit compares human judgments against the released intent labels.
- C.1 Intent Label Audit by Cohort: Six-to-three intent-class collapses under two independent groupings change no finding.The label-free test also supports the claim that agents rarely diagnose.
- C.1 Intent Label Audit by Cohort: Agreement on agent transitions is not lower than agreement on human transitions, so behavioral gaps are not attributed to weaker agent-label reliability.The 75–85% spread lies within binomial uncertainty, roughly ±12 points at n=40.
- D Additional Statistics for the Behavioral Findings: The additional statistics test whether headline gaps survive every fairness treatment.This analysis is designated D.1 in the appendix.
- D Additional Statistics for the Behavioral Findings: The appendix reports the action distributions from which the behavioral divergences are computed.These distributions are designated D.2.
- D Additional Statistics for the Behavioral Findings: The appendix distinguishes two senses in which a run can return to earlier work.These analyses are designated D.3.
D.1 Headline Gaps Under Every Fairness Treatment … F Harness Experiment Details
Across fairness checks, action frequencies, return-to-earlier-work measures, feature analysis, and harness details, TraceML shows persistent human–agent process gaps. Human practice is broader and more adaptive, while agent behavior concentrates in narrower action loops that a targeted prompt only partly addresses.
- D.1 Headline Gaps Under Every Fairness Treatment: Every headline human-minus-agent gap survives the fairness treatments, with every interval excluding zero; filtering agents widens Codex’s gaps.The comparisons include scored-only humans, agent filtering, and matched-budget agent runs, with uncertainty estimated by a two-stage cluster bootstrap.
- D.1 Headline Gaps Under Every Fairness Treatment: Debugging intent is lower for agents than humans, with cohort coefficients of −1.82 for Codex and −1.89 for MLEvolve.The transition-level generalized estimating equation reports SE 0.52, z=−3.5 for Codex and SE 0.29, z=−6.6 for MLEvolve.
- D.2 Full Coarse-Action Frequency Table: Humans devote 6.3% of action mass to ensembling versus Codex’s 19.7%, while model-plus-training accounts for 21.9% for humans versus MLEvolve’s 38.4%.These produce gaps of −13.4 and −16.5 percentage points, respectively; action shares are trajectory-equal and may overlap because transitions can carry multiple actions.
- E Feature Reference: The feature reference defines 19 trajectory-level features, while six carry almost all of the association between process and final rank.Feature correlations are computed separately on paired and disjoint humans-only splits; final rank percentage is smaller-is-better, with gold-medal kernels averaging 4.5% and no-medal kernels 6.8%.
- E.1 The validation-and-ensembling discipline index: The six features form a validation-and-ensembling discipline index spanning fold averaging, out-of-fold persistence, early ensembling, ensemble-and-inference effort, and real-member blending.The harness prompt operationalizes these clauses as a description of human practice rather than a quantity to maximize arbitrarily.
- D.3 Returning to Earlier Work: Agents rarely return to earlier approaches: Codex has one solution revisit in 658 eligible versions and MLEvolve has zero in 344.These counts contrast with 60 and 31 occurrences predicted by the top human rate; setback recovery is a separate score-based measure that can occur without revisiting earlier work.
- F Harness Experiment Details: The harness appendix supplies the scores behind the percentile view, the intervention prompts, and the behaviors the prompt does not reach.These materials constitute the support for Section 5’s harness experiment analysis.
F.1 Matched-Budget Scores and Ablations · F.2 Agent-Run Prompts · F.3 Behaviors the Prompt Does Not Reach
Under matched twelve-hour conditions, the planning skill improves scores on five competitions without regressions, while ablations show that planning content—not injection cadence alone—drives the gains. The prompt specifies human-inspired validation, ensembling, model cycling, and self-checks, but the harness still switches working modes far less often than humans and retains agent-like effort allocation.
- F.1 Matched-Budget Scores and Ablations: Five competitions improve over the matched baseline, two remain within noise, and none regress.Repeated same-condition runs differ by 0.012 RMSE on commonlit and 0.009 C-index on equity; differences below those scales are treated as noise.
- F.1 Matched-Budget Scores and Ablations: Abl-B lands at or below baseline everywhere, with hms reaching 1.377 against a 1.050 baseline.Abl-B preserves the 30-minute re-injection cadence but removes planning content, showing cadence alone contributes nothing.
- F.1 Matched-Budget Scores and Ablations: Abl-A recovers much of the gain on aes2 and hms but none on gquest or ranzcr.Abl-A supplies a single planning-content block rather than the full skill, separating content effects from the complete schedule.
- F.2 Agent-Run Prompts: The harness differs from baseline by expanding the planning skill block and re-injecting its reminder every 30 minutes.Both conditions share the same task prompt; per-run variables are filled by the runner before dispatch.
- F.2 Agent-Run Prompts: The planning block directs agents to use K-fold validation, cycle validation, features, models, and ensembles, and cache OOF predictions for revisitable models.It also recommends early ensembling, multiple model families, small early changes, metric-aligned validation, and periodic self-checks.
- F.2 Agent-Run Prompts: The harness reminder forces recurring checks of validation, metric alignment, weighting or custom loss, ensembling, and OOF persistence before further iteration.The runner re-injects the reminder every 30 minutes after the first ensemble window, and instructs the agent to continue improving submission.csv rather than write a long report.
- F.3 Behaviors the Prompt Does Not Reach: The harness switches dominant working modes on 2.2% of steps, versus 13.2–14.1% for every human cohort and 7.1% for prior Codex.Thus, the prompt moves diagnosis rhythm and timing but remains far below human switching rates.
G Limitations
TraceML’s measurements are bounded by incomplete public histories, task-aligned but uncontrolled human–agent settings, inferred annotations, and dependent trajectory units. Broader workflows, richer logs, larger gold annotation sets, and matched-budget reruns would reduce these limitations.
- Data coverage: Human trajectories capture saved Kaggle versions rather than all work, excluding private experiments, reused notebooks, and unpublished or cleaned-up versions.The histories may omit substantial off-platform activity and later edits before publication.
- Experimental control: Human and agent settings differ in calendar horizons, tooling, compute, and collaboration, so human behavior represents public practice rather than optimal planning.The settings are task-aligned but not fully controlled.
- Measurement validity: Intent labels are inferred from code changes rather than observed thoughts, so they are interpreted coarsely alongside actions, timing, and score changes.State and action labels are grounded in code structure and diffs, whereas intent is inferred from the change.
- Statistical dependence: Trajectory units are dependent because notebooks share fork lineage and code, while MLEvolve branches share tree nodes; intervals are therefore clustered by component.This dependence affects the paper’s interval-level analysis.
- Future mitigation: More harnesses, newer competitions, non-Kaggle workflows, richer logs, larger gold annotation sets, and matched-budget reruns would reduce platform, labeling, and budget caveats.These extensions would also address platform-specific bias and tighten the label and budget limitations.