PaperlayerView exact revision on arXiv

Source-linked AI summary

MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipeline

Rushi Qiang, Yuchen Zhuang, Anikait Singh, Percy Liang, Chao Zhang, Sherry Yang, Bo Dai

arXiv:2510.07307v1cs.LGcs.AI

TL;DR

MLE benchmarks rely on static, manually curated tasks, limiting their scalability and requiring extensive effort to produce. MLE-Smith automates generation and verification of competition-style MLE tasks from raw datasets. Its generated tasks strongly correlate with human-designed benchmarks, and the system produces a diverse suite from real-world datasets.

  • Problem

    Existing MLE benchmarks rely on static, manually curated tasks that require extensive time and effort to produce.

    Source: §1 · p. 1

  • Method

    MLE-Smith uses a multi-agent generate–verify–execute pipeline to transform raw datasets into competition-style MLE tasks and validate their structure, semantics, and empirical solvability.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    MLE-Smith generated tasks strongly correlate with human-designed benchmarks, and its application to real-world datasets produced a diverse suite of tasks.

    Source: §6 · pp. 9–10

  • Takeaways & Limitations

    The results support using automatically generated MLE tasks to scale task creation while retaining benchmark alignment and task quality.

    Source: §6 · p. 10

Abstract

from arXiv · show

While Language Models (LMs) have made significant progress in automating machine learning engineering (MLE), the acquisition of high-quality MLE training data is significantly constrained. Current MLE benchmarks suffer from low scalability and limited applicability because they rely on static, manually curated tasks, demanding extensive time and manual effort to produce. We introduce MLE-Smith, a fully automated multi-agent pipeline, to transform raw datasets into competition-style MLE challenges through an efficient generate-verify-execute paradigm for scaling MLE tasks with verifiable quality, real-world usability, and rich diversity. The proposed multi-agent pipeline in MLE-Smith drives structured task design and standardized refactoring, coupled with a hybrid verification mechanism that enforces strict structural rules and high-level semantic soundness. It further validates empirical solvability and real-world fidelity through interactive execution. We apply MLE-Smith to 224 of real-world datasets and generate 606 tasks spanning multiple categories, objectives, and modalities, demonstrating that MLE-Smith can work effectively across a wide range of real-world datasets. Evaluation on the generated tasks shows that the performance of eight mainstream and cutting-edge LLMs on MLE-Smith tasks is strongly correlated with their performance on carefully human-designed tasks, highlighting the effectiveness of the MLE-Smith to scaling up MLE tasks, while maintaining task quality.

1 INTRODUCTION

MLE benchmarks are valuable for evaluating MLE agents, but static, manually curated tasks require substantial expert and engineering effort and constrain scalability, diversity, and realism. MLE-Smith automates task generation and verification to produce challenges designed to be structurally sound, semantically meaningful, and empirically solvable.

1 INTRODUCTION

  • Existing MLE benchmarks rely on static, manually curated tasks, whose creation and standardization require extensive human and engineering effort and limit scalability, diversity, and realism.The work calls for an automated framework that can continuously generate and verify tasks at scale.Source: §1 · pp. 1–2
  • Generated tasks must meet three intertwined criteria: structural integrity, semantic soundness, and empirical solvability.The criteria require end-to-end execution, a coherent objective grounded in the dataset, and non-trivial but tractable performance.Source: §1 · p. 2
  • MLE-Smith transforms raw datasets into competition-style MLE tasks through a fully automated generate–verify–execute pipeline.The pipeline combines multi-agent generation, hybrid verification, and interactive execution-based validation.Source: §1 · p. 2, Figure 1 · p. 3
  • The multi-agent workflow and hybrid verification are designed to make generated challenges standardized, executable, and practically meaningful without human intervention.The verification mechanism combines static format checks, semantic alignment, and execution-based tests.Source: §1 · p. 2
  • Applied to 224 real-world datasets, MLE-Smith generated 606 tasks, and evaluation on 50 tasks found strong correlation between eight LLMs’ rankings and their rankings on human-curated benchmarks.The task suite spans modalities, objectives, and domains.Source: §1 · p. 2

2 RELATED WORKS

MLE-agent benchmarks and interactive environments have expanded across domains, including end-to-end MLE workflows, but their finite task sets remain largely manually curated. MLE-Smith addresses this gap with automated generation of MLE tasks.

2 RELATED WORKS

  • Existing MLE testbeds cover end-to-end workflows, but their task collections remain finite and manually curated.The cited examples range from curated MLE tasks and standardized competitions to interactive environments with over 200 executable tasks.Source: §2 · p. 3
  • MLE-Smith is presented as an automated MLE task-generation framework intended to support continual generation of novel tasks.The related-work section distinguishes it from automated task-generation systems for software engineering, coding, and other agent tasks.Source: §2 · p. 3

3 METHODS

MLE-Smith transforms raw datasets into competition-style MLE tasks through sequential multi-agent generation, standardization, hybrid verification, and interactive execution. Its verification layers check structure, semantic alignment, and empirical solvability before retaining tasks.

3 METHODS

  • MLE-Smith couples structured multi-agent generation, hybrid verification, and execution-based validation to turn raw datasets into competition-style MLE tasks.The sequential design balances diverse task proposals with structural correctness and downstream usability.Source: §3 · p. 4

3.1 MULTI-AGENT GENERATION WORKFLOW

  • The Brainstormer proposes multiple dataset-grounded task formulations, while the Designer builds each selected formulation into a reproducible, end-to-end task package.The Designer supplies deterministic data splits, input/output schemas, evaluation protocols, and preparation, submission, evaluation, and testing artifacts.Source: §3.1 · p. 4
  • Refactor rewrites candidate tasks into a shared schema that standardizes interfaces, file structure, metrics, and feedback while preserving task-specific logic.Verification routines check conformance and support downstream end-to-end testing.Source: §3.1 · pp. 4–5

3.2 HYBRID VERIFICATION MECHANISM

  • Deterministic Assertions check required files, directory structure, schemas, interfaces, and execution scripts so downstream modules receive automation-ready artifacts.Assertions operate before and after Refactor to check both Designer outputs and conformance to the unified task format.Source: §3.2 · p. 5
  • LLM-mediated Reviews assess task-description clarity, metric appropriateness, and whether the setup encourages meaningful behavior rather than shortcuts.They can flag omissions or ground-truth leakage that structural checks would not detect.Source: §3.2 · p. 5
  • Interactive execution tests whether the full task pipeline runs without assistance and whether agents achieve non-trivial performance with a metric sensitive to method quality.Failures are logged and routed to targeted refinement or re-execution; only tasks passing all three verification layers are retained.Source: §3.2 · p. 5

4 AUTOMATED TASK GENERATION

The task-generation study samples Kaggle datasets and evaluates generated tasks against MLE-Dojo, using GPT-5 agents with specified action and retry budgets. MLE-Smith produces tasks across varied modalities, objectives, domains, and metrics, with reported generation costs and agent-wise workflow statistics.

4 AUTOMATED TASK GENERATION

  • The study sampled 300 high-usability Kaggle datasets, generated 807 tasks, and reserved 50 tasks to evaluate alignment with the MLE-Dojo leaderboard.GPT-5 served as the agents’ backbone; the Brainstormer could propose up to three formulations per dataset, and the validation agent had up to 10 steps.Source: §4 · p. 6, §4.1 · p. 6

4.2 STATISTICS OF GENERATED TASKS

  • MLE-Smith produced 606 verified tasks from 224 datasets, averaging 2.71 tasks per dataset, 419.98 seconds of preparation per task, and $0.78 per task.The preparation-time figure excludes execution-based verification, whose runtime varies by task and hardware; execution time is typically below 600 seconds.Source: §4.2 · p. 6
  • Generated tasks span tabular, image, video, audio, natural-language, and time-series modalities, with classification and regression alongside ranking, multi-label, structured-prediction, and generation objectives.Tabular and natural-language tasks occur more frequently, reflecting the source datasets; the reported task distributions are shown in Figure 2.Source: §4.2 · pp. 6–7
  • F1, precision, and recall account for 24.7% of task metrics; AUC/ROC accounts for 18.3%, RMSE/MAE/MSE for 17.3%, and custom domain-specific metrics for 16.2%.Ranking-based measures such as NDCG and MAP account for 3.6%.Source: §4.2 · p. 7
  • The Designer succeeds on its first attempt in over 99% of cases, while Refactor needs a second attempt for around 6% and a third for about 1%.The Designer uses no more than 15 steps in approximately 92% of cases; Refactor consistently uses more than 13 steps, with most tasks requiring 15–22 steps.Source: §4.2 · p. 7

5 EXPERIMENTS: TASK EVALUATION

The evaluation compares eight LLMs across real MLE-Dojo tasks and generated MLE-Smith tasks, finding consistent model rankings across the two sets. Step-wise score trajectories and agreement statistics further assess whether generated tasks are learnable and reflect human-designed benchmarks.

5.1 EXPERIMENT SETUPS

  • The evaluation uses 50 real Dojo tasks, 50 generated Smith tasks, and their 100-task Combined set to compare eight LLMs.The models are evaluated in an MLE-Dojo environment; the setup uses up to 15 action steps and 12 hours per task run.Source: §5 · p. 7, §5.1 · pp. 7–8
  • Elo ratings provide the primary cross-model comparison, estimated from sample-weighted pairwise outcomes with ties treated as symmetric half-wins.The Elo model uses a Bradley–Terry-style logistic fit with base-10 log odds, scale 400, and offset 1000.Source: §5.1 · p. 8

5.2 MAIN RESULTS

  • Gemini-2.5-Pro ranks at the top across nearly every modality, while DeepSeek-V3.1-Reasoner and o4-mini form a competitive second tier and GPT-4o models lag, especially on vision.Model rankings remain consistent across the Dojo, Smith, and Combined sets.Source: §5.2 · p. 8

5.3 STEP-WISE PERFORMANCE DYNAMICS

  • Across categories, models’ normalized best-so-far scores rise with interaction steps, supporting the authors’ assessment that generated tasks are learnable and differentiate modeling approaches.Figure 4 averages category-level trajectories across tasks and excludes information-requesting steps.Source: §5.3 · pp. 8–9, Figure 4 · p. 9

5.4 REALISM AND QUALITY OF GENERATED TASKS

  • Dojo–Smith Elo scores correlate at r = 0.982, and the three sets yield α = 0.993 and ICC(2, 1) = 0.981.The reported rank agreement is Spearman ρ = {0.952, 0.976, 0.976} and Kendall τb = {0.857, 0.929, 0.929} across the three pairs.Source: §5.4 · p. 9, Table 2 · p. 9

6 CONCLUSION

The conclusion presents MLE-Smith as an automated pipeline for turning raw datasets into competition-style MLE tasks. Its generated tasks strongly correlate with human-designed benchmarks, supporting their realism and discriminative power.

6 CONCLUSION

  • MLE-Smith transforms raw datasets into competition-style MLE tasks through a fully automated generate–verify–execute pipeline.The pipeline is described as ensuring structural integrity, semantic soundness, and empirical solvability.Source: §6 · p. 9
  • The generated task suite strongly correlates with human-designed benchmarks, matching real competitions in realism and discriminative power.The conclusion describes the suite as large and diverse.Source: §6 · pp. 9–10

A.1 FULL LIST OF EVALUATION TASKS

The appendix presents an inventory of competition datasets, including their names, sizes, and tags. The listed entries span varied domains and data types.

A.1 FULL LIST OF EVALUATION TASKS

  • The inventory records dataset names, sizes, and tags, with dataset size and tags used to show coverage across domains, modalities, and formulations.The passage describes the listed dataset sizes as relatively large.Source: §A.1 · p. 13
  • The inventory also includes text-oriented entries, such as datasets tagged for music text generation and news summarization.Examples include devdope/900k-spotify and gowrishankarp/newspaper-textsummarization-cnn-dailymail.Source: §A.1 · p. 14
  • Listed examples include a sports dataset tagged for running, data visualization, and tabular data, alongside entries tagged for video and audio.The inventory includes kgmuchiri/world-athletics-alltime-dataset and media datasets such as rohanmallick/kinetics-train-5per.Source: §A.1 · pp. 14–15

A.2 UNIFIED TASK STRUCTURE

The unified task structure standardizes generated tasks into a shared directory and interface format. Assertions check the presence of essential artifacts and conformance of preparation and metric implementations.

A.2 UNIFIED TASK STRUCTURE

  • The required structure includes files such as prepare.py, metric.py, description.txt, and sample submission.csv, together with raw, public, and private directories.Figure 5 depicts the unified directory structure that Refactor should deliver.Source: §A.2 · p. 15, Figure 5 · p. 15
  • Assertions verify that essential files and directories exist and that prepare.py and metric.py follow the required formats.prepare.py must implement a prepare function that accepts the raw, public, and private directories; metric.py must implement the designated Metric class and methods.Source: §A.2 · p. 15
  • The shared structure standardizes preparation interfaces, input/output specifications, metric implementation, file layout, and feedback reporting while preserving task-specific logic.The conventions support structural validation and end-to-end execution checks.Source: §A.2 · p. 15

A.3 NORMALIZATION DETAILS

The normalization procedure accounts for metric direction and produces best-so-far trajectories that can be averaged across tasks. It also specifies a special case for constant observed scores.

A.3 NORMALIZATION DETAILS

  • Metric direction is +1 when higher raw values are better and −1 when lower values are better.Source: §A.3 · p. 15
  • When a task-model’s observed scores are constant, observed entries are set to 1 and missing entries to 0.Source: §A.3 · p. 16
  • Missing indices are forward-filled, then a prefix maximum produces a nondecreasing best-so-far trajectory of length 10 per task-model pair.Source: §A.3 · p. 16
  • These trajectories are averaged pointwise across tasks to obtain category-level and overall trajectories.Source: §A.3 · p. 16

A.4 PROMPTS FOR MLE-SMITH AGENTS

The agent prompts guide competition design from dataset exploration and diverse task brainstorming through task creation, refactoring, and testing. They emphasize meaningful, solvable challenges and consistent, participant-facing files.

A.4 PROMPTS FOR MLE-SMITH AGENTS

  • The Brainstormer explores dataset insights and proposes diverse, dataset-supported competition designs, with the number of ideas adapted to the data’s intrinsic properties.Designs cover the problem, data use, processing, metric, rationale, ignored data, difficulty, and tags.Source: §A.4 · p. 16
  • Prompt principles call for train/test splits and precise, reliable labels, while allowing the number of brainstorming outputs to reflect the dataset’s natural variety.Source: §A.4 · p. 16
  • The Designer is instructed to produce a challenging, meaningful, ML-solvable competition with a fair, precise metric and a concise, accurate description.The prompt also specifies deterministic splits, sample submissions, separated test answers, and tests for preparation and metric code.Source: §A.4 · pp. 17–18
  • The Refactor prompt standardizes preparation and metric implementations, requires deterministic preparation and comprehensive tests, and separates participant-visible public files from private test answers.Source: §A.4 · pp. 18–19
  • The prompts require a participant-facing description that matches the public directory, avoids revealing answers, and remains concise and informative.Source: §A.4 · pp. 18–19

A.5 DETAILS OF STATISTICAL MEASURES FOR ELO SET AGREEMENT

The appendix presents complementary agreement measures for comparing Elo sets, covering score association, model ranking, numerical agreement, and reliability. Its measures address different comparison needs, including rank ties and score-scale discrepancies.

A.5.1 PEARSON LINEAR CORRELATION (r)

  • Pearson correlation measures linear association between Elo scores and whether score magnitudes are proportionally similar.Source: §A.5.1 · p. 20

A.5.2 COEFFICIENT OF DETERMINATION (R2)

  • R2 represents the proportion of variance in one Elo set explained by another; for simple correlation, R2 equals r2.Source: §A.5.2 · p. 20

A.5.3 SPEARMAN RANK CORRELATION (ρ)

  • Spearman correlation assesses whether model ordering is preserved independently of absolute score scales, including under monotonic nonlinear relationships.Source: §A.5.3 · p. 20

A.5.4 KENDALL RANK CORRELATION (τb)

  • Kendall τb measures pairwise ranking agreement while accounting for ties, which are common in Elo ratings.Its interpretation is the probability of concordance minus the probability of discordance.Source: §A.5.4 · pp. 20–21

A.5.5 TOP-k OVERLAP

  • Top-k overlap measures how consistently the leading models coincide, focusing on agreement in the most competitive leaderboard region.Source: §A.5.5 · p. 21

A.5.6 LIN’S CONCORDANCE CORRELATION COEFFICIENT (CCC)

  • Lin’s CCC assesses both correlation-based precision and accuracy relative to the 45° identity line, supporting checks of numerical interchangeability.Source: §A.5.6 · p. 21

A.5.7 BLAND–ALTMAN ANALYSIS

  • Bland–Altman analysis plots paired-score differences against their means to reveal systematic bias and visualize discrepancies even when correlation is high.The limits of agreement use the mean difference and its standard deviation.Source: §A.5.7 · p. 21

A.5.8 CRONBACH’S α

  • Cronbach’s α estimates consistency across multiple raters or measurement methods; values above 0.9 indicate excellent reliability in this comparison.Source: §A.5.8 · p. 21

Related papers

Loading 2510.07307v1…