Source-linked AI summary
ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments
Ziyang Cai, Amir Saeidi, Harkirat Behl
TL;DR
ML-AutoResearch tackles the shortage of scalable process-level supervision for agents performing long-horizon ML research. It automatically synthesizes executable, dataset-grounded research environments and teacher trajectories, then uses them for supervised fine-tuning. Across three benchmarks, two model families, and three model sizes, ML-AR training produces consistent capability gains, including up to 9.1% mean MLGym AUP improvement.
Problem
Autonomous ML research requires massive, diverse process-level supervision, but static benchmarks omit debugging and incremental reasoning while manual data collection is prohibitively expensive.
Method
ML-AutoResearch automatically generates executable synthetic ML research tasks grounded in real-world datasets, verifies them through self-debugging, and collects teacher trajectories for supervised fine-tuning.
Results
5.7%, 9.1%, and 5.9% mean MLGym AUP gains occur at 4B, 8B, and 14B, respectively, with improvements also transferring to MLAgentBench and AutoResearch.
Takeaways & Limitations
ML-AR provides scalable process-level training data whose learned experimentation, debugging, and feedback-driven refinement transfer beyond the primary evaluation environment.
Takeaways & Limitations
The supervised-imitation objective teaches feedback-driven execution and refinement rather than explicitly optimizing exploration or scientific novelty, and task generation remains mediated by GPT-5.
Abstract
from arXiv · showhide
With the advent of AI agents, automated scientific discovery is becoming an increasingly plausible goal. However, training agents to autonomously execute the engineering-heavy labor of machine learning (ML) research requires massive, process-level supervision. Existing static benchmarks omit critical intermediate steps such as debugging and incremental reasoning, and manual data collection is prohibitively expensive. To overcome this data bottleneck, we introduce ML-AutoResearch (ML-AR), a scalable pipeline for automatically generating synthetic, end-to-end ML research tasks. Each task defines a complete research cycle, including problem specification, dataset selection, baseline implementation, and iterative improvement. To ensure realism and executability, tasks are grounded in real-world datasets and refined via an automated self-debugging procedure without requiring human supervision. We construct a large-scale dataset of teacher trajectories on these synthetic tasks to train student agents via supervised fine-tuning. We evaluate the resulting agents across 3 diverse ML research benchmarks. Our comprehensive experiments across 2 distinct model families and 3 model sizes demonstrate that training on ML-AR trajectories yields consistent and significant capability gains. Fine-tuning improves the Aggregation Under the Performance (AUP) by up to 9\% and substantially boosts overall pass rates, highlighting robust out-of-domain generalization.
1 Introduction
ML-AutoResearch addresses the lack of scalable, process-level supervision for autonomous ML research by generating executable synthetic tasks and rich interaction trajectories. Training on these trajectories improves agents across diverse benchmarks and model settings.
- Motivation: Static papers, repositories, and benchmarks omit failed experiments, debugging cycles, and incremental reasoning needed for multi-step ML problem solving.Manual collection is prohibitively expensive, while existing human-authored benchmarks provide limited scale and diversity.
- Approach: ML-AutoResearch generates synthetic end-to-end ML research tasks with problem formulation, dataset selection, baseline implementation, and iterative improvement.Tasks are grounded in real-world datasets and refined through automated self-debugging, with minimal manual effort.
- Dataset Construction: 555 verified environments and approximately 34K teacher trajectories, filtered to 17K high-quality examples, form the training data for supervised fine-tuning.The environments were generated from 1,000 sampled topics, with verification removing candidates that could not become executable environments.
- Results: 5.7%, 9.1%, and 5.9% mean MLGym AUP gains occur at 4B, 8B, and 14B model sizes, respectively, across three evaluation seeds.Additional mixed GPT-5/GPT-5.4 trajectories raise the 14B model’s AUP by 7.0% over its baseline.
- Framework: The framework samples topics, verifies real-world dataset matches, synthesizes starter code and evaluation procedures, then executes tasks to collect and filter trajectories.Its task environments are represented by problem specifications, dataset descriptors, baseline code, and evaluation procedures.
- Results: ML-AR improves performance across MLGym, MLAgentBench, and AutoResearch, including when Qwen3-32B replaces GPT-5 as the trajectory-generation teacher.The reported gains span two model families and three diverse ML benchmarks.
4 Experiments
ML-AutoResearch improves ML-agent performance across model sizes, model families, and distinct benchmarks, while ablations identify data curation and trajectory integrity as important design choices. The evaluation also examines scaling, alternative teachers, transfer, and behavioral changes in failure modes.
- Main Results: Mean AUP improves by 5.7%, 9.1%, and 5.9% at 4B, 8B, and 14B, respectively, across three evaluation seeds.Independent training-seed experiments also yield mean AUP gains of 4.4% at 8B and 4.0% at 14B.
- Main Results: Seed-42 best-run AUP reaches 1.1297 for ML-AR-14B-Mixed, improving 7.0% over Qwen3-14B and 5.2% over original ML-AR-14B.The mixed model adds GPT-5.4 trajectories to the original GPT-5 trajectory dataset; these best-run values are separate from three-seed means.
- Main Results: ML-AutoResearch improves MLAgentBench normalized performance by 0.101, 0.171, and 0.294 at 4B, 8B, and 14B, respectively.The results span codebases, objectives, and evaluators different from the synthetic training environments.
- Main Results: Tuned models improve AutoResearch val_bpb by 5% over baselines and consistently propose superior hyperparameter configurations on their first attempt.AutoResearch requires agents to analyze training code, observations, configurations, and resource constraints.
- Ablation Studies: Successful experience selection performs best, complete trajectories outperform truncated rollouts, and z = 0.1 gives the best empirical filtering result.The ablations favor strict, performance-aligned curation and preserving full temporal and causal coherence over increasing dataset size.
5 Conclusion
ML-AutoResearch generates executable ML research tasks and process-level training trajectories, then evaluates them across model sizes, benchmarks, and teacher settings. The framework reports consistent AUP gains, while acknowledging limitations in supervision objectives, teacher dependence, and computational cost.
- ML-AutoResearch generates executable ML research tasks and process-level training trajectories for supervised fine-tuning.
- 5.7%, 9.1%, and 5.9% mean AUP gains are reported at 4B, 8B, and 14B, respectively, in three-seed MLGym evaluation.
- 7.0% best-run AUP improvement is obtained for the 14B model using additional mixed GPT-5/GPT-5.4 data.
- Improvements on MLAgentBench and AutoResearch indicate transfer of experimentation, debugging, and feedback-driven refinement beyond MLGym.
- Limitations: Supervised imitation teaches feedback-driven execution and refinement rather than explicitly optimizing exploration or scientific novelty.
- Limitations: Task generation and verification remain mediated by GPT-5, while generating and evaluating long-horizon trajectories requires substantial compute.
1. Claims
The paper’s abstract and introduction state ML-AutoResearch’s contribution, setting, and evaluation gains consistently with the method and experiments. The manuscript also discusses limitations, while reproducibility-related claims and expectations are framed through review criteria and supporting details.
- The abstract and introduction identify ML-AutoResearch, its setting, and its reported evaluation gains as the paper’s core claims.
- The claims are expected to match theoretical and experimental results and reflect how far findings generalize beyond evaluated settings.
- A dedicated Limitations paragraph discusses supervision constraints, dependence on a single teacher model, and open scaling questions.
- The review guidance asks authors to state strong assumptions and explain implications when those assumptions may be violated in practice.
- Reproducibility guidance covers sufficient algorithm, architecture, model-access, code, data, and execution information for experimental verification.
- The manuscript is noted as lacking anonymized public repository links or release artifacts with full reproduction instructions in the supplement.
6. Experimental setting/details
The appendix specifies many training, compute, and execution details, but the experiments do not include hypothesis tests or conventional error bars because repeated full training runs were infeasible. The work reports task-level outcomes, aggregate metrics, and consistency across reruns instead.
- Training details: The appendix enumerates optimizer, learning-rate schedule, warmup, weight decay, batch sizes, steps, seeds, sequence constraints, and filtering settings.
- Statistical reporting: The review guidance recommends error bars, confidence intervals, or significance tests, alongside definitions of variability and calculation methods.
- Statistical reporting: Full task-level outcomes and aggregate benchmark metrics are reported, with evaluation described as consistent across reruns.
- Statistical reporting: Hypothesis tests are not provided because repeating full training runs was infeasible under the compute budget.
- Compute details: The appendix lists the H100 cluster SKU, GPU counts, parallelism settings, and run-step or time-budget configurations.
- Ethics: The paper’s ethics review reports no deviations from the NeurIPS Code of Ethics for this benchmarked ML-systems research.
10. Broader impacts
The appendix addresses societal impacts, release safeguards, asset licensing, documentation, and responsible research practices. It identifies both potential misuse risks and current documentation limitations.
- The work discusses positive impacts, negative misuse risks, and mitigation context for controlled evaluation settings.
- The manuscript does not release high-risk generative model artifacts or scraped datasets requiring special controlled-release safeguards.
- External assets are inventoried by category, with usage stated to follow their licenses and terms.
- New synthetic tasks and trajectories are not yet accompanied by finalized datasheet-style metadata and access instructions.
- LLM use is declared because a teacher model performs task synthesis, verification, and trajectory generation.
A Appendix
The appendix details benchmark evaluation, robustness experiments, societal context, asset practices, and additional data-construction analyses. Results generally support improved reliability while retaining heterogeneous task-level outcomes.
- Benchmarks: MLGym reports both baseline-beating success and aggregate trajectory quality through AUP, using task-native scalar evaluators.
- Results: ML-AR generally improves execution reliability, but some outcomes regress, including 4B MNLI and 14B Breakout.
- Robustness: 4.4% mean AUP gains occur at 8B and 4.0% at 14B under independent training-seed evaluation.
- Teacher transfer: The open-weight-teacher experiment holds 555 verified environments fixed while generating trajectories with Qwen3-32B.
- Responsible research: The appendix notes dual-use risks, controlled sandboxed evaluation, third-party licensing, and implementation-level training details.
Dataset validation prompt
The dataset-validation prompt guides task generation from topic and metric inputs toward valid, executable configurations. It emphasizes dataset verification, structured YAML outputs, supported entrypoints, and detailed task descriptions.
- Dataset validation prompt: The prompt instructs the model to validate uncertain dataset names through a search tool and output only the final JSON when confident.
- Dataset validation prompt: Generated task JSON must include the topic, metric, description, and an optional dataset matched to a public HuggingFace dataset.
- Dataset validation prompt: The prompt permits refining the topic or avoiding repeated searches when existing results provide a suitable dataset.
- A.11 Task files generation prompt Task files stage 1: config generation: The prompt requires detailed objectives, data formats, examples, submission requirements, metrics, constraints, and correctly formatted output files.
- A.11 Task files generation prompt Task files stage 1: config generation: Task files use one task YAML followed by optional dataset YAML files containing valid dataset paths and detailed descriptions.
- A.11 Task files generation prompt Task files stage 1: config generation: The generated configuration specifies identifiers, descriptions, dataset references, entrypoints, timeouts, requirements, baselines, evaluations, and memory settings.
- A.11 Task files generation prompt Task files stage 1: config generation: Task configurations select among CSVSubmissionTasks, ModelSubmissionTasks, LMSubmissionTasks, and PythonSubmissionTasks according to the submission type.
Task files stage 2: starter code generation
The starter-code stage turns task and dataset configurations into runnable ML environments with baselines, evaluators, and submission-specific interfaces. It constrains dependencies, execution time, file formats, and evaluation behavior.
- The agent is expected to improve a provided baseline model and submit the final solution after planning the task.
- The task configuration records entrypoint, timeout, dependency, baseline, evaluation, and memory-path settings.
- Dataset configurations must use real public HuggingFace datasets, while package choices must match preinstalled or explicitly declared requirements.
- The agent receives task and dataset configurations and must produce runnable starter code for an autonomous ML research task.
- Evaluation must print a single valid JSON object containing numeric metric values, and tasks must finish within 30 minutes on specified hardware.
- The four entrypoints map CSV, model or YAML, distributed language-model, and directly executed Python submissions to distinct evaluation commands.
Error-Recovery Retry Prompt
This section presents a HotpotQA-style multi-hop question-answering task with structured questions, answers, supporting facts, and distractor contexts. It also gives concrete comparison, bridge, and multi-hop examples for evaluating these fields.
- Error-Recovery Retry Prompt: The retry prompt requests a complete revised output after an error.
- A.12 Example synthetic task: The synthetic-task example introduces a task instance containing a question and structured evaluation fields.
- 4. Evaluation code evaluate.py: Dataset documentation describes HotpotQA as a large-scale multi-hop question-answering dataset with questions requiring reasoning across multiple documents.
- 4. Evaluation code evaluate.py: HotpotQA examples encode each item with an identifier, question, answer, type, difficulty level, supporting facts, and candidate context.
- 4. Evaluation code evaluate.py: Supporting facts pair document titles with zero-based sentence indices, while context stores ten candidate documents and their sentence lists.
- 4. Evaluation code evaluate.py: The dataset schema includes train, validation or development, and potentially test splits, with normalized answer evaluation and aligned supporting-fact annotations.
- 4. Evaluation code evaluate.py: The distractor configuration uses ten candidate documents, only a subset of which contains the gold supporting paragraphs or sentences.
- 4. Evaluation code evaluate.py: Examples cover comparison questions requiring facts from two documents and bridge questions linking entities across documents.
hotpotqa_join_facts_qa.yaml
This task requires jointly answering HotpotQA distractor questions and identifying the exact supporting sentences from ten candidate paragraphs. It specifies multitask modeling, official metrics, resource constraints, and strict submission requirements.
- The task builds a multi-hop QA system for HotpotQA’s distractor setting, predicting answers and supporting facts.
- Each example contains a question, ten candidate paragraphs with sentence boundaries, gold supporting-fact pairs, and an answer string.
- The model jointly predicts the answer and performs sentence-level classification over candidate sentences using a weighted loss.
- Supporting facts must reference existing context titles and valid zero-based sentence indices, with the exact gold set required for Supporting Facts EM.
- Joint F1 is the primary metric, supplemented by Answer EM/F1, Supporting Facts EM/F1, and Joint EM.
- Training is expected to take approximately 3–5 hours on one NVIDIA RTX A6000 with eight CPU cores for preprocessing.
- Recommended approaches include hierarchical context encoders, long-sequence models or retrieval, and multitask answer-support prediction.
- The baseline reports joint_f1 of 0.022210986997935424 and joint_em of 0.0013504388926401081.
baseline.py
The baseline loads the HotpotQA distractor validation split, emits structurally valid predictions, and uses simple heuristics for answers and supporting facts.
- The baseline loads the HotpotQA distractor validation split for evaluation.
- For questions beginning with auxiliary verbs, the answer heuristic returns yes; otherwise it returns a fallback title or unknown.
- It iterates through examples and extracts each identifier, question, candidate titles, and sentence lists.
- The baseline selects up to two candidate supporting facts by taking the first available titles and sentence index zero.
- Fallback logic emits a structurally valid supporting-facts prediction even when candidate sentences are unavailable.
- Predictions are written as CSV rows containing the example id, answer, and JSON-encoded supporting facts.
evaluate.py
The evaluator parses submissions and gold annotations, normalizes answers, computes answer and supporting-fact metrics, and aggregates joint scores across examples.
- Answer normalization lowercases text, removes punctuation and articles, and fixes whitespace before scoring.
- Answer F1 is computed from token overlap using precision and recall, while exact match separately checks normalized equality.
- Supporting-fact EM and F1 compare predicted and gold sets of title and sentence-index tuples.
- The evaluator loads predictions from a CSV file containing ids, answers, and supporting-facts cells.
- Gold supporting facts are reconstructed as sets of title and sentence-index pairs from the dataset annotations.
- The evaluator averages answer, supporting-fact, and joint metrics across all gold examples before printing one JSON object.
- Joint EM and Joint F1 combine answer and supporting-fact scores by multiplying their respective component scores.