PaperlayerView exact revision on arXiv

Source-linked AI summary

FML-bench: Benchmarking Machine Learning Agents for Scientific Research

Qiran Zou, Hou Hei Lam, Wenhao Zhao, Yiming Tang, Tingting Chen, Samson Yu, Tianyi Zhang, Chang Liu, Xiangyang Ji, Dianbo Liu

arXiv:2510.10472v2cs.CLcs.AI

TL;DR

Existing benchmarks emphasize application-oriented tasks and final outcomes, leaving agents’ iterative research processes less examined. FML-bench evaluates agents on eight fundamental ML tasks and adds process metrics; its evaluation finds broad exploration associated with higher diversity and better performance.

  • Problem

    Existing benchmarks focus mainly on application-oriented tasks, final performance, and computational cost, overlooking agents’ iterative processes and limiting analysis of their relation to research outcomes.

    Source: §1 · p. 1

  • Method

    FML-bench evaluates agents on eight fundamental ML research tasks and adds process metrics, including proposal-variance-based Exploration Diversity and two measures of agent reliability.

    Source: §1 · pp. 1–2

  • Results

    Broad-exploration agents exhibit higher exploration diversity and superior performance, and diversity positively correlates with performance improvements across multiple tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported findings suggest broad exploration and high exploration diversity are important contributors to effective ML research agents.

    Source: §1 · p. 2

    AIDE’s single-file iterative edits can be insufficient for realistic research codebases that span multiple files.

    Source: §5.3.5 · p. 8

Abstract

from arXiv · show

Large language models (LLMs) have sparked growing interest in machine learning research agents that can autonomously propose ideas and conduct experiments. However, existing benchmarks predominantly adopt an engineering-oriented perspective: they emphasize application-oriented tasks and evaluate primarily on final performance and computational cost, overlooking agents' research processes and limiting assessment of their capabilities in scientific research settings. To more comprehensively evaluate agents in scientific research settings, we introduce FML-bench, a benchmark comprising 8 diverse and fundamental ML research tasks, and further propose complementary metrics, notably Exploration Diversity, which quantifies the variance of proposals across iterations and reveals how exploration patterns influence research outcomes. We evaluate state-of-the-art research agents on FML-bench, showing that agents employing broad exploration strategies exhibit higher exploration diversity and achieve superior performance, and that exploration diversity positively correlates with performance improvements across multiple tasks. We hope these findings and our benchmark inform future agent design and support the community in further investigating agent behavior. Our benchmark is available at https://github.com/qrzou/FML-bench.

1. Introduction

FML-bench addresses the gap between engineering-oriented benchmarks and evaluation of scientific ML research by testing agents on fundamental problems and their iterative research processes. The authors also report that broad exploration is associated with higher exploration diversity and better performance.

1. Introduction

  • Existing benchmarks emphasize application-oriented engineering tasks and final performance or computational cost, while overlooking agents’ iterative research processes.This leaves limited scope for analyzing how agent characteristics relate to research outcomes.Source: §1 · p. 1
  • FML-bench comprises eight diverse, fundamental ML research tasks and evaluates agents as they solve problems through iterative research.The tasks span generalization, data efficiency, representation learning, continual learning, causality, robustness, privacy, and fairness.Source: §1 · p. 1, Figure 1 · p. 2
  • Exploration Diversity measures variance in proposals across iterations, while Step Success Rate and Step Completion Rate assess agent reliability.The benchmark also reports Performance and Cost.Source: §1 · p. 2
  • Tasks use established ML research repositories and provided baselines so agents can focus on algorithmic and architectural advances rather than building codebases.This contrasts with benchmarks that provide raw data or tightly formatted, handcrafted codebases.Source: §1 · p. 2
  • Agents using broad exploration exhibit higher exploration diversity and superior performance, with diversity positively correlating with performance improvements across multiple tasks.The authors report these patterns in their evaluation of state-of-the-art research agents.Source: §1 · p. 2

2. Related Works

Prior work includes agents for research assistance and increasingly autonomous research, while existing ML-agent benchmarks vary in scope and often emphasize engineering or data-science workflows. FML-bench is positioned as a benchmark for diverse, fundamental ML research tasks.

2.1. Automatic Research Agents

  • Earlier research-agent systems support idea generation, literature review, or simulated peer review.Examples include SciMON and Nova for research ideas, AutoSurvey for literature reviews, and AgentReview for simulated peer review.Source: §2.1 · pp. 2–3
  • Recent automatic research agents aim to generate ideas, implement them, run experiments, and refine approaches without human supervision.Examples include AIDE’s iterative code editing, TheAIScientist’s end-to-end research pipeline, AgentLaboratory, and AlphaEvolve’s evolutionary refinement.Source: §2.1 · p. 3

2.2. Benchmarks for ML Agents

  • Existing ML-agent benchmarks cover engineering, Kaggle, or data-science tasks, whereas FML-bench targets eight fundamental ML research problems using real-world codebases.The cited benchmarks also vary in flexibility, task scope, and reliance on fixed starter files or task-specific evaluators.Source: §2.2 · p. 3

3. Benchmark Task Design

The benchmark’s eight tasks represent distinct foundational ML challenges. Datasets and baselines are selected to balance computational tractability, meaningful research challenge, and room for improvement.

3. Benchmark Task Design

  • Datasets must support baseline training and evaluation within two hours while remaining challenging, and baselines must be recognized methods with room for improvement.These choices are intended to enable meaningful comparisons across tasks and magnitudes of improvement.Source: §3 · pp. 3–4
  • Generalization uses ColoredMNIST and ERM to test out-of-domain accuracy on a held-out target domain, while Data Efficiency uses Mini-ImageNet and Prototypical Networks for few-shot accuracy with a frozen backbone.Both tasks ask agents to improve or propose algorithms under the stated training constraints.Source: §3 · p. 4
  • Representation Learning uses CIFAR-10 and MoCo to improve linear probing on frozen encoder features; Continual Learning uses splitMNIST and Synaptic Intelligence to improve accuracy across sequential tasks.The continual-learning task also targets avoiding catastrophic forgetting.Source: §3 · p. 4
  • Causality uses IHDP and DragonNet to minimize treatment-effect estimation error, while Robustness, Privacy, and Fairness tasks target backdoor defense, membership-inference resistance, and equitable outcomes.The latter tasks use dp-instahide, Wide-ResNet-28-2, and Adversarial Debiasing baselines, respectively.Source: §3 · p. 4

4. Evaluation Protocol

The evaluation protocol combines outcome and process metrics to characterize agent effectiveness, exploration, reliability, and cost. FML-bench’s unified input-output interface accommodates diverse repository workflows while retaining their native complexity.

4.1. Evaluation Metrics

  • Performance is the best task-specific held-out-test result across iterations, using the maximum when higher is better and the minimum when lower is better.The evaluation uses protected code to support consistent comparisons across agents.Source: §4.1 · p. 5
  • Exploration Diversity measures dispersion of code embeddings across iterations, with greater dispersion indicating broader exploration of implementation strategies.The embeddings use GraphCodeBERT, which accounts for code structure so differently named snippets with identical logic receive high similarity.Source: §4.1 · p. 5
  • Step Success Rate captures the fraction of executed iterations that produce valid experimental results without errors, while Step Completion Rate measures executed iterations relative to the assigned total.Together, these rates characterize reliability during the research process.Source: §4.1 · p. 5
  • Cost is assessed through total and per-step token consumption and total and per-step wall-clock time.These components characterize computational and temporal efficiency.Source: §4.1 · p. 5

4.2. Unified Input-output Interface

  • The unified interface treats training and evaluation commands as one input unit and supplies task descriptions, repository code, suggested files, protected segments, commands, baseline performance, and targets.This accommodates repositories with different script names, multistage pipelines, and training arguments.Source: §4.2 · p. 5
  • The compared agents use parallel broad exploration, hierarchical tree search balancing exploration and exploitation, or linear refinement.These strategies correspond to TheAIScientist, AIDE, and Claude Code, respectively.Source: Figure 2 · p. 6
  • A post-processing module standardizes performance-metric extraction across diverse output formats while preserving native output mechanisms, and evaluation files are protected from agent modification.Agents must use the protected evaluation files to assess proposed methods.Source: §4.2 · p. 6

5. Experiments

The experiments compare three research agents with distinct search strategies and find that broader exploration accompanies stronger task performance and diversity. They also examine resource use, modification patterns, and implementation limits.

5.1. Settings

  • Each agent ran three independent rounds with a 100-step budget per round, and the best test-set result across rounds was selected.Source: §5.1 · p. 6

5.2. Implementation Details

  • TheAIScientist was adapted for real-repository experiments, AIDE was accessed through Weco, and Claude Code used a prompting scheme to read code and propose experiment-grounded changes.The agents used GPT-5 and Gemini-2.5-Pro for TheAIScientist and AIDE, and native Opus-4.1 for Claude Code.Source: §5.2 · p. 6

5.3.1. COMPARISON OF AGENTS WITH DIFFERENT RESEARCH STRATEGIES

  • TheAIScientist with Gemini-2.5-Pro ranked first on 4 of 8 tasks, while AIDE with Gemini-2.5-Pro ranked second with 2 top results.The authors report that TheAIScientist performs better than AIDE and Claude Code at discovering novel and effective methods.Source: §5.3.1 · p. 6, Table 2 · p. 7
  • TheAIScientist explored broadly but shallowly, AIDE used medium breadth and depth, and Claude Code explored narrowly but deeply.The authors suggest broader exploration may be more effective for finding promising ideas.Source: §5.3.1 · p. 7

5.3.2. ANALYSIS OF EXPLORATION DIVERSITY

  • With the LLM held fixed, exploration diversity was 20.85 vs. 18.41 and best results were 4 vs. 2; greater diversity tended to yield better outcomes, with positive correlations across five tasks.Source: §5.3.2 · p. 7

5.3.3. TOKEN AND TIME CONSUMPTION

  • Claude Code used the most tokens despite lower performance; AIDE and Claude Code had shorter total runs primarily because of premature termination.Per-step durations were similar, differing by around two minutes.Source: Table 3 · p. 7, §5.3.3 · p. 8

5.3.4. CASE ANALYSIS

  • In worst-case analyses, TheAIScientist favored algorithmic modifications (56.2%) with zero implementation errors, while AIDE had more implementation issues (31.2%).Claude Code’s parameter-configuration rate was 37.5%, matching its algorithmic-modification rate.Source: §5.3.4 · p. 8
  • In best-case analyses, TheAIScientist focused on core algorithmic logic, AIDE combined algorithmic innovation with tuning, and Claude Code favored architectural changes.Across LLMs, GPT-5 exclusively prioritized algorithmic modifications (100.0%); Gemini-2.5-Pro included unique data-augmentation efforts (28.6%).Source: §5.3.4 · p. 8

6. Conclusion

The conclusion presents FML-bench as an evaluation resource for scientific ML research agents, combining fundamental research tasks with measures of research-process behavior. Its evaluation links broad exploration and exploration diversity with stronger performance, while positioning the benchmark as a foundation for further agent assessment.

  • FML-bench evaluates ML research agents in scientific settings using eight diverse, fundamental research problems built on recognized academic repositories.Source: §6 · p. 8
  • The benchmark adds process metrics, including Exploration Diversity, Step Success Rate, and Step Completion Rate, alongside performance evaluation.Source: §6 · p. 8
  • Evaluated agents using broad exploration showed higher exploration diversity and superior performance, while diversity positively correlated with performance improvements across multiple tasks.Source: §6 · p. 8

Impact Statement

The impact statement frames FML-bench as a shift toward realistic, process-oriented evaluation and greater transparency in research-agent assessment. It also presents the benchmark’s interfaces as supporting extensible evaluation across repositories, datasets, and baselines.

  • FML-bench shifts agent evaluation from narrow answer-matching benchmarks toward realistic, process-oriented scientific problem solving.Source: Impact Statement · p. 9
  • The benchmark emphasizes investigation of agent behavior to promote more meaningful progress and transparency in research-agent evaluation.Source: Impact Statement · p. 9
  • Its unified input-output interfaces support extensibility across repositories, datasets, and baselines, which the authors believe can encourage diverse and reproducible evaluation.Source: Impact Statement · p. 9

A. Task Design Details

The appendix specifies eight task designs spanning generalization, data efficiency, representation learning, continual learning, causality, robustness, privacy, and fairness. Each task sets a research objective, benchmark setting, and evaluation criterion.

Generalization

  • Generalization uses ColoredMNIST under distribution shift, requiring source-domain training and held-out target-domain evaluation against ERM using out-of-domain accuracy.Source: A. Task Design Details · p. 12

Data Efficiency

  • Data Efficiency uses Mini-ImageNet with a frozen backbone and Prototypical Networks baseline, evaluating few-shot classification accuracy on novel classes.Source: A. Task Design Details · p. 12

Representation Learning

  • Representation Learning uses unlabeled CIFAR-10 pretraining with a ResNet-18 and MoCo baseline, measured by linear-probing accuracy on frozen features.Source: A. Task Design Details · p. 12

Continual Learning

  • Continual Learning tests class-incremental splitMNIST across five sequential tasks, measuring average accuracy while avoiding unfair model-size or compute advantages.Source: A. Task Design Details · p. 12

Causality

Robustness and Reliability

  • Robustness and Reliability uses poisoned MNIST and dp-instahide; its defense score is the harmonic mean of clean accuracy and backdoor resistance accuracy.Source: A. Task Design Details · pp. 12–13

Privacy

  • Privacy uses CIFAR-10, Wide-ResNet-28-2, and MIA or RMIA, aiming for attack AUC near 0.5 while maintaining classification accuracy.Source: A. Task Design Details · p. 13

Fairness and Bias

  • Fairness and Bias uses COMPAS with Adversarial Debiasing, minimizing absolute average odds difference toward zero while maintaining or improving subgroup accuracy.Source: A. Task Design Details · p. 13

B.1. Worst Case Analysis

Worst-case modification patterns vary across agents and LLMs: agents differ in algorithmic focus and implementation issues, while LLMs differ in their preferred modification types.

Worst Case Analysis

  • TheAIScientist favors algorithmic modifications (56.2%) with zero implementation errors, while AIDE has a more balanced distribution and 31.2% implementation and execution issues.TheAIScientist's execution is described as robust; AIDE's 31.2% figure concerns implementation and execution issues.Source: §B.1 · p. 13
  • Claude Code's parameter configuration rate (37.5%) equals its algorithmic modification rate.Source: §B.1 · p. 13
  • Among LLMs, Gemini-2.5-Pro favors algorithmic modifications (68.8%), GPT-5 favors structural modifications (50.0%), and Opus-4.1 balances parameter configuration and algorithmic modifications.Source: §B.1 · p. 13

B.2. Best Case Analysis

Best-case modification patterns differ across agents and LLMs, spanning algorithmic innovation, architectural changes, and data augmentation.

Best Case Analysis

  • TheAIScientist focuses on algorithmic logic, while AIDE combines algorithmic innovation with hyperparameter tuning.Source: §B.2 · p. 13, Table 7 · p. 14
  • Claude Code uniquely favors architectural modifications, including layer additions and normalization adjustments.Source: §B.2 · p. 13, Table 7 · p. 14
  • GPT-5 exclusively prioritizes algorithmic modifications (100.0%), Gemini-2.5-Pro incorporates data augmentation (28.6%), and Opus-4.1 balances architectural and algorithmic modifications.Source: §B.2 · p. 13, Table 8 · p. 14

C. Protecting Evaluation Integrity

Evaluation files are protected through agent-specific measures that prevent or deter agents from modifying them.

Protecting Evaluation Integrity

D. Additional Results

This section provides additional material on agent exploration and efficiency, task prompts and experimental framework, and instructions for executing and reporting iterative experiments.

D. Additional Results

  • The additional analysis compares exploration diversity, step reliability, token use, and runtime across TheAIScientist, AIDE, and Claude Code on eight research problems.It covers exploration diversity, step success and completion rates, total and step-wise token consumption, and total and step-wise time.Source: D. Additional Results · pp. 15–16

E. Prompts Details

  • Task prompts specify researcher roles, technical setups, objectives, constraints, and fairness criteria across eight ML areas, encouraging novel methods as well as incremental improvements.They are anchored in established benchmarks and frameworks, while preserving controlled experimental conditions.Source: E. Prompts Details · p. 18

Generalization; Data Efficiency

  • Generalization and data-efficiency prompts target unseen-domain accuracy on ColoredMNIST and novel-class generalization in Mini-ImageNet few-shot learning.The generalization task uses DomainBed’s ERM baseline and prioritizes unseen-domain accuracy while maintaining in-domain accuracy and efficiency.Source: Generalization · p. 19, Data Efficiency · p. 19

Representation Learning

  • The representation-learning task seeks better CIFAR-10 linear-probe accuracy than MoCo under matched compute and data, without external data.It permits changes to MoCo or the ResNet-18 backbone while keeping parameter count and FLOPs comparable.Source: Representation Learning · p. 19

Continual Learning; Causality

  • Continual-learning prompts seek higher average accuracy and reduced forgetting on splitMNIST, while the causal-inference task seeks more precise IHDP treatment-effect estimates.The continual-learning setup uses Synaptic Intelligence on five two-class tasks; the causal task uses the Dragonnet framework.Source: Continual Learning · p. 19, Causality · p. 20

Robustness and Reliability; Privacy

  • Robustness prompts target stronger dp instahide defenses against poisoning while preserving clean accuracy, and privacy prompts seek lower membership-inference risk without sacrificing task accuracy.The privacy task focuses on defense-side strategies and targets auditor AUC toward 0.5.Source: Robustness and Reliability · p. 20, Privacy · p. 20

Fairness and Bias

  • The fairness prompt seeks to reduce absolute Average Odds Difference while preserving or improving Balanced Accuracy on COMPAS held-out splits and protected subgroups.It uses AIF360’s Adversarial Debiasing as its baseline and allows fair changes to the method, networks, or training pipeline.Source: Fairness and Bias · p. 20

Automatic Research Agent Framework

  • The automatic-agent prompt defines an iterative research loop of proposing plans, editing code, running commands, diagnosing errors, recording results, and refining plans until the iteration limit.It also requires respecting read-only paths and maintaining per-step backups and logs.Source: Fairness and Bias · pp. 20–22

Fairness and Bias

  • Commands must run verbatim, failures should be fixed through code or configuration changes, and evaluation files must remain untouched.The instructions also call for small, testable changes, recording only files changed from the original baseline, and avoiding regressions after successful steps.Source: Fairness and Bias · p. 23
  • The agent must continue the modify-run-diagnose loop while the step count is at or below MAX ITERS.The instructions also require treating non-zero command exit codes as errors and explicitly setting and documenting required environment variables or paths.Source: Fairness and Bias · p. 23
  • Optimization must follow the global metric direction unless a per-metric override is given, and specified target metrics and datasets are exclusive goals.The process start and end timestamps must be logged; the example records a duration of 5573 seconds.Source: Fairness and Bias · p. 23

4. Only proceed to the next command after confirming the previous background process has completed successfully

  • Commands expected to run longer than five minutes must use background execution, and the next command must wait until the prior process completes successfully.The 100-epoch training command is explicitly required to run in the background, with monitoring until completion.Source: Fairness and Bias · p. 24, §4 · p. 24
  • After each iteration, the agent must report its current plan, edited files and rationale, command outcomes and exit codes, successful-run metrics, and next steps.The instructions also provide template variables for task paths, read-only paths, targets, optimization direction, command list, iteration limit, and result directory.Source: §4 · p. 24
Loading 2510.10472v2…