Source-linked AI summary
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, Aleksander Mądry
TL;DR
Autonomous end-to-end ML engineering lacks holistic benchmarks that reflect difficult real-world tasks and permit comparison with human performance. MLE-bench addresses this gap with 75 challenging Kaggle competitions and evaluates frontier language models using open-source agent scaffolds. The best setup, o1-preview with AIDE, earns a medal in 16.9% of competitions, while the benchmark also examines resource scaling and contamination.
Problem
Few benchmarks holistically measure autonomous end-to-end ML engineering, despite advances in coding, ML tasks, programming tools, and agent scaffolding.
Method
MLE-bench manually curates 75 Kaggle competitions into an offline environment with challenging tasks and human-comparable leaderboard evaluation, then evaluates models with agent scaffolds.
Results
16.9% of competitions received a medal from the best-performing setup, o1-preview with AIDE.
Takeaways & Limitations
MLE-bench facilitates research into agents’ autonomous ML engineering capabilities and supports deeper understanding needed for safe deployment of more powerful models.
Takeaways & Limitations
MLE-bench does not cover the full spectrum of AI R&D because its competitions have clear problem statements, clean documented datasets, and clear optimization metrics.
Abstract
from arXiv · showhide
We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, we curate 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments. We establish human baselines for each competition using Kaggle's publicly available leaderboards. We use open-source agent scaffolds to evaluate several frontier language models on our benchmark, finding that the best-performing setup--OpenAI's o1-preview with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in 16.9% of competitions. In addition to our main results, we investigate various forms of resource scaling for AI agents and the impact of contamination from pre-training. We open-source our benchmark code (github.com/openai/mle-bench/) to facilitate future research in understanding the ML engineering capabilities of AI agents.
1 INTRODUCTION
MLE-bench addresses the lack of holistic benchmarks for autonomous end-to-end ML engineering by providing challenging Kaggle-based tasks with human-comparable evaluation. Experiments show meaningful agent performance, with results improving through resource scaling and revealing difficulties with debugging and recovery.
- Few existing benchmarks holistically measure autonomous end-to-end ML engineering despite progress in coding, ML tasks, programming tools, and agent scaffolding.
- MLE-bench is an offline Kaggle competition environment designed around challenging, representative tasks and comparison with human-level performance.
- 75 diverse Kaggle competitions span domains including natural language processing, computer vision, and signal processing, with $1,948,016 in total prizes awarded.
- The benchmark supports research on autonomous ML engineering, resource scaling, contamination, and monitoring tools for plagiarism and cheating.
- 16.9% of competitions earned a medal for the best setup, o1-preview with AIDE, while pass@8 increased o1-preview’s score to 34.1%.GPT-4o’s score also rose from 8.7% with 24 hours to 11.8% with 100 hours.
- Agents perform well on competitions solvable with known approaches but struggle to debug issues and recover from missteps.
2 MLE-BENCH
MLE-bench curates 75 manually selected Kaggle competitions into locally gradable tasks that preserve comparison with human leaderboard performance. Its metrics use Private-leaderboard medal thresholds, while rules and reporting guidance address validity, cheating, and reproducibility.
- Dataset curation: MLE-bench contains 75 manually sourced Kaggle competitions representing day-to-day ML engineering skills in frontier labs.
- Dataset curation: Each competition includes a description, dataset, local grading code, and a leaderboard snapshot for ranking submissions against humans.
- Dataset curation: The benchmark starts from 5673 completed Kaggle competitions, manually screens 586 for modern ML engineering relevance, and excludes unreproducible grading or unreasonable splits.
- Dataset curation: 75 competitions comprise 22 low-complexity, 38 medium-complexity, and 15 high-complexity tasks, plus 7 development competitions.
- Dataset curation: Reconstructed test sets use publicly available training data when original test sets are unavailable, with distribution checks against the original competition setup.
- Metrics: MLE-bench applies Kaggle’s Private-leaderboard medal thresholds, which vary with participant count, and reports the percentage of attempts receiving any medal.
- Reporting: Evaluations should report models, scaffolding, internet access, hardware, runtime, prompts containing solutions, and deviations from the experimental setup.
- Rules: Agents may use local validity checks but cannot write predictions directly, view online solutions, or submit code highly similar to top Kaggle notebooks.
3 EXPERIMENTS AND RESULTS
Experiments evaluate how scaffolds, models, and resource scaling affect MLE-bench performance. Results show strong effects from scaffold and model choice, more attempts and time, while compute hardware and competition familiarity show limited benefits.
- Varying scaffolding: GPT-4o (AIDE) achieves 8.7% medals, exceeding MLAB at 0.8% and OpenHands at 4.4%.The experiments use open-source scaffolds and report mean scores with standard errors in Table 2.
- Varying models: o1-preview with AIDE achieves medals in 16.9% of competitions, almost twice the next-best model’s average.Four models are evaluated with the AIDE scaffold.
- Varying amount of compute available: GPT-4o (AIDE) performs similarly with CPU-only, one-A10, and two-A10 configurations.Logs did not identify attempts to use the second GPU in the extra-GPU experiment.
- Competition familiarity: GPT-4o’s familiarity with a competition has no positive correlation with its performance on that competition.Performance is measured using scores normalized between the sample-submission and gold-medal scores.
4 CONTAMINATION & PLAGIARISM
The paper examines whether contamination from public Kaggle materials inflates MLE-bench results through memorized solutions or recognizable competition descriptions. Across familiarity and obfuscation experiments, it finds no systematic inflation, while noting subtler contamination effects cannot be ruled out.
- Motivation: Contamination could produce non-generalizing performance if models adopt high-level strategies from public winning solutions.The paper distinguishes subtler strategy reuse from detectable plagiarism.
- Familiarity with top solutions: GPT-4o’s competition familiarity was measured using mean token probabilities on each competition page and five popular discussion posts.The discussion posts typically describe competition winners’ approaches.
- Familiarity with top solutions: No correlation was found between GPT-4o’s familiarity with a competition and its performance on that competition.This result does not support systematic score inflation from familiarity.
- Obfuscating competition descriptions: Obfuscating competition descriptions produced no significant difference in GPT-4o’s performance.The descriptions retained key information while removing Kaggle references, competition names, and unnecessary text.
- Conclusion: The experiments suggest contamination effects are minimal, but they do not rule out subtler contamination effects.The conclusion concerns GPT-4o and the experiments reported here.
5 RELATED WORK
MLE-bench extends prior evaluations of ML engineering and agent capabilities by using more numerous, complex, and open-ended Kaggle competitions. It differs from related benchmarks in task scope, codebase interaction, and competition selection criteria.
- Evaluating ML Engineering Capabilities: MLAgentBench evaluates 13 Kaggle and bespoke ML tasks by measuring whether agents improve at least 10% over simple baseline solutions.MLE-bench instead provides significantly more and more complex tasks.
- Evaluating ML Engineering Capabilities: ML-Bench tests code generation and command execution for interacting with ML repositories, rather than developing solutions to open-ended ML problems.The distinction is between applying pre-existing codebases and creating ML solutions.
- Evaluating ML Engineering Capabilities: State-of-the-art models at AIDE’s announcement surpassed the MLE-bench median score about 10% of the time, far below a reported claim of beating over 50% of human competitors.The authors interpret this gap as evidence that their competition selection is more difficult.
- Evaluating ML Engineering Capabilities: DSBench focuses on data science tasks and excludes competitions whose datasets do not fit a simple template for automated task creation.MLE-bench manually processes competitions and retains non-standard formats.
- Evaluating AI Agents: AgentBench, GAIA, and related benchmarks evaluate multi-step agent interaction across operating systems, real-world questions, and other environments.MLE-bench applies the multi-step agent evaluation paradigm specifically to ML engineering competitions.
6 LIMITATIONS
MLE-bench has important scope, comparability, contamination, and resource constraints. Its tasks cover core ML engineering competencies but do not represent the full range of real-world AI R&D workflows.
- Contamination & plagiarism: Public Kaggle materials may contaminate benchmark results through memorized competition details, datasets, solutions, or high-level strategies.The authors find no systematic contamination effect for GPT-4o but make no guarantees about future models.
- Coverage of AI R&D capabilities: MLE-bench does not cover the full spectrum of AI R&D because its competitions have clear problem statements, clean documented datasets, and defined optimization metrics.Real-world AI R&D may require determining the problem, dataset, and metrics itself.
- Differences to real competitions: Different train-test splits and reimplemented grading code raise concerns about comparability with original Kaggle human leaderboards.The authors report preserving similar distributions and checking sample and gold submissions against leaderboard-consistent results.
- Accessibility: A main experiment run requires 1,800 GPU hours across 75 competition attempts, and o1-preview with AIDE used 127.5M input and 15.0M output tokens per seed.These requirements make the benchmark particularly resource-intensive to run.
7 CONCLUSION
The paper introduces MLE-bench to evaluate autonomous ML engineering with challenging Kaggle competitions and compare agents with human competitors. Its results show measurable but limited agent performance, while motivating further evaluation for safe deployment and broader AI research capabilities.
- Conclusion: MLE-bench evaluates AI agents on challenging Kaggle-based ML engineering tasks and supports direct comparison with human competitors.The benchmark is open-sourced to facilitate further research.
- Conclusion: 16.9% of competitions earned a medal from the best-performing setup, o1-preview with AIDE scaffolding.This is the principal reported benchmark result.
- Ethics and safety: Agents capable of open-ended ML research could improve frontier-model capabilities faster than human researchers.The paper links this possibility to risks if understanding, security, alignment, and control do not advance in parallel.
- Future evaluation: The authors encourage evaluations of automated ML research beyond MLE-bench, especially workflows involving researchers training large language models.They believe solving a large fraction of MLE-bench likely indicates capability on many open-ended ML tasks.
- Reproducibility and data: The benchmark uses public Kaggle competitions, without sensitive data, and provides code for reproducing datasets in compliance with relevant licenses.The authors also provide details and code for reproducing the benchmark and experiments.
A.1 DATASET CURATION CRITERIA
MLE-bench competitions were manually screened by at least two ML engineers against criteria for relevance, solvability, local evaluation, and reproducible execution.
- Each candidate competition was manually filtered and screened by at least two ML engineers at leading AI companies.
- Competitions had to require modern ML engineering capabilities to achieve a medal.
- Descriptions had to be sufficiently specified and solvable without crucial missing information or reliance on external materials.
- Evaluation metrics had to be computable locally, and competitions had to be finished and unlikely to change.
- The dataset, split, submission, and access requirements constrained inclusion to competitions with usable data and reproducible CSV-based evaluation.The criteria excluded extensively reused datasets, required train and test data from the same distribution, required CSV submissions, and prohibited non-Kaggle downloads.
A.2 DISTRIBUTION OF COMPETITIONS
MLE-bench’s 75 competitions are manually characterized by problem category and complexity level to show how its tasks are distributed.
- Figure 6 provides a high-level overview of MLE-bench’s problem-category and complexity-level distributions.Both dimensions were manually labeled.
- The distribution analysis covers the benchmark’s competition problem categories.
- The analysis also covers the complexity levels of the competitions.
A.3 RUNS ANALYSIS
The runs-analysis tool examines agent logs and code outputs for rule violations, but human review found all detected violations to be false positives.
- The provided analysis tool processes agent logs and code outputs using a rubric of questions.
- The tool was run on all medal-winning submissions.
- All non-zero detected violations were judged to be false positives after human review.The tool flagged behaviors in o1-preview AIDE and GPT-4o OpenHands runs, but reviewers found no actual violations.
A.4 PLAGIARISM DETECTION TOOL
MLE-bench checks medal-winning submissions for plagiarism against associated Kaggle notebooks and uses controlled execution and dataset-splitting procedures across its competition suite.
- Plagiarism detection: Dolos compares submission files with top associated Kaggle notebooks using token fingerprints and a 23-token subsequence setting.
- Plagiarism detection: No detected plagiarism cases were found after submissions exceeding 60% similarity were surfaced for human review.The similarity analysis also found no winning submission above the 60% threshold.
- Execution setup: Agents were evaluated in Docker-based environments with specified compute, storage, datasets, time limits, and scaffold configurations.The setup included an Azure A10 virtual machine, Docker containers, three tested scaffolds, and prompts containing available time, execution limits, and hardware.
- Scaffold modifications: Scaffold modifications addressed robustness issues including submission validation, retries, error handling, truncation, and preventing premature termination.
- Dataset construction: The benchmark dataset documents reconstructed test splits made from publicly available training data when original test sets were unavailable.Splits were generally created at a 10% test ratio, with competition-specific exceptions recorded in the dataset table.