Source-linked AI summary
AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents
Alisia Lupidi, Bhavul Gauri, Thomas Simon Foster, Bassel Al Omari, Despoina Magka, Alberto Pepe, Alexis Audran-Reiss, Muna Aghamelu, Nicolas Baldwin, Lucia Cipolina-Kun, Jean-Christophe Gagnon-Audet, Chee Hau Leow, Sandra Lefdal, Hossam Mossalam, Abhinav Moudgil, Saba Nazir, Emanuel Tewolde, Isabel Urrego, Jordi Armengol Estape, Amar Budhiraja, Gaurav Chaurasia, Abhishek Charnalia, Derek Dunfield, Karen Hambardzumyan, Daniel Izcovich, Martin Josifoski, Ishita Mediratta, Kelvin Niu, Parth Pathak, Michael Shvartsman, Edan Toledo, Anton Protopopov, Roberta Raileanu, Alexander Miller, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach
TL;DR
AI Research Agents require standardized evaluation because existing benchmarks are obscured by contamination, inconsistent environments, and noisy empirical results. AIRS-Bench provides 20 non-contaminated tasks spanning the full research workflow and evaluates agents using generated code across models and scaffolds. Most agents remain below human SOTA, while a few exceed it, indicating substantial remaining headroom.
Problem
Existing AI Research Agent benchmarks provide limited reliable evidence because data contamination, inconsistent environments, and noisy evaluations obscure agent capabilities.
Method
AIRS-Bench evaluates agents on 20 diverse, non-contaminated machine-learning tasks covering the full research workflow without baseline code, using LLMs augmented by scaffolds.
Results
For most tasks, even the best-performing agent remains significantly behind human SOTA, while top agents outperform human SOTA on a few tasks.
Takeaways & Limitations
AIRS-Bench is far from saturated and can expose performance gaps while supporting transparency, reproducibility, and standardized evaluation of AI Research Agents.
Takeaways & Limitations
Agent performance can be reduced by formatting and intermediate-result failures, context overflow, and accumulated issues in code editing and debugging.
Abstract
from arXiv · showhide
LLM agents hold significant promise for advancing scientific research. To accelerate this progress, we introduce AIRS-Bench (the AI Research Science Benchmark), a suite of 20 tasks sourced from state-of-the-art machine learning papers. These tasks span diverse domains, including language modeling, mathematics, bioinformatics, and time series forecasting. AIRS-Bench tasks assess agentic capabilities over the full research lifecycle -- including idea generation, experiment analysis and iterative refinement -- without providing baseline code. The AIRS-Bench task format is versatile, enabling easy integration of new tasks and rigorous comparison across different agentic frameworks. We establish baselines using frontier models paired with both sequential and parallel scaffolds. Our results show that agents exceed human SOTA in four tasks but fail to match it in sixteen others. Even when agents surpass human benchmarks, they do not reach the theoretical performance ceiling for the underlying tasks. These findings indicate that AIRS-Bench is far from saturated and offers substantial room for improvement. We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
1 Introduction
AIRS-Bench addresses evaluation challenges for AI Research Agents with 20 diverse, benchmark-driven tasks covering the research workflow. It standardizes task construction and evaluation while comparing agents across models and scaffolds.
- Motivation: AIRS-Bench targets evaluation problems caused by data contamination, inconsistent environments, and noisy empirical results.These factors make agent performance difficult to interpret reliably.
- Benchmark scope: The benchmark contains 20 tasks from recent state-of-the-art literature across NLP, mathematics, code, biochemical modeling, and time series forecasting.Tasks assess idea generation, methodology design, experiment analysis, and iterative refinement without baseline code.
- Evaluation design: Agents are evaluated by generating code, executing it to train and validate models, and measuring resulting performance.This setup evaluates the agent’s capacity to function as an autonomous research scientist.
- Standardization: A task configuration standard, fixed metrics, and human checks during task creation, review, and verification support reproducible comparisons.The format is intended to reduce runtime and environment inconsistencies and remain extensible across machine-learning problems.
- Empirical analysis: The empirical analysis benchmarks frontier open- and closed-source models with different scaffolds and finds substantial performance variation.The evaluation reports valid submission rates, normalized performance scores, and Elo ratings across tasks.
2 Related Work
AIRS-Bench is positioned among scientific-agent benchmarks by emphasizing broad domain coverage, full-cycle research, long horizons, and no starter solutions. Its comparison framework highlights differences in task composition, scientific stages, baseline access, environments, and compute requirements.
- Task origins: Prior benchmarks derive tasks from GitHub repositories, conference papers, Kaggle competitions, and cross-domain papers.
- Evaluation dimensions: Benchmark comparisons consider task composition and origin, reasoning horizon, scientific stages, baseline access, and compute requirements.
- Full research cycle: AIRS-Bench requires agents to cover hypothesis generation, implementation, experimentation, and analysis across the scientific method.
- Environments: Agent environments range from prompt-and-code setups to gamified environments, physics engines, synthetic tasks, and simulated user interactions.
- Benchmark landscape: Recent scientific-agent benchmarks span more than 20 domains, including bioinformatics, chemistry, and physics.
- Task difficulty: AIRS-Bench targets long-horizon research challenges and withholds starter solutions, requiring agents to make progress independently.
3 Agents, Scaffolds, Harnesses
The paper defines an agent as an LLM combined with a scaffold, with the scaffold instantiated and managed by a harness inside an environment. AIRS-Bench uses both sequential and search-oriented harness designs to explore and improve candidate solutions.
- Definitions: An agent consists of an LLM and a scaffold that enables systematic exploration of a solution space.
- Definitions: A scaffold contains operators and search algorithms, while a harness encapsulates the agent and manages its research process.
- Task categories: AIRS-Bench organizes 20 tasks into seven categories: Code, Math, Molecules & Proteins ML, Question Answering, Text Classification, Text Extraction & Matching, and Time Series.
- Harnesses: AIRA-dojo evolves solutions through operators and search policies such as greedy search, Monte Carlo Tree Search, and evolutionary algorithms.
- Harnesses: MLGym supports sequential ReAct-like improvement using ideation, execution feedback, tools, bash, and optional internet access.
4 Method
AIRS-Bench constructs a reproducible benchmark from machine-learning papers and datasets, spanning seven categories and using standardized task metadata, prompts, preparation scripts, and evaluation code.
- Task construction: 20 tasks are extracted from 17 machine-learning papers and 16 datasets selected using publication, availability, and train/test-split criteria.
- Task composition: The benchmark spans seven categories, with 4 Question Answering, 3 Text Extraction & Matching, 2 Text Classification, 5 Molecules and Proteins ML, 3 Time Series, 2 Code, and 1 Math task.
- Task fields: Each task records its research problem, dataset, configuration, splits, metric, input and scoring columns, category, and SOTA metadata.
- Cross-harness standardization: Task metadata and files are organized for programmatic conversion across harnesses, including AIRA-dojo and MLGym.
- Task specification: Task prompts describe the research problem, dataset, and evaluation setup, while data are pre-downloaded and exported to the agent container.
- Evaluation: Agents submit test predictions as CSV files, which evaluate.py scores against test data using task-specific metric implementations.
- Data handling: Preparation scripts separate agent-visible data from evaluation data by removing test labels during solution development.
5 Experiments
AIRS-Bench evaluates agents using valid submission rate, normalized score, and Elo across heterogeneous research tasks. The experiments define these metrics, standardize comparisons across runs, and use pairwise agent outcomes to estimate relative skill.
- Experimental setup: Each agent combines a base LLM with a scaffold, and evaluations use MLGym or AIRA-dojo under matched resource constraints.Runs last 24 hours on one H-200 GPU, with each task launched at least 10 times.
- Metrics: Valid submission rate measures how often an agent produces a working solution that satisfies task requirements and yields a valid score.It averages successful runs over total runs across the evaluated tasks.
- Metrics: Failed and invalid submissions contribute 0 normalized score when scores are averaged across seeds.This incorporates both unsuccessful runs and submissions without numerical scores into aggregate performance.
- Metrics: Elo ratings estimate relative agent skill by treating pairwise task-score comparisons as games and fitting an order-invariant Bradley–Terry model.Human SOTA scores are included as an additional SOTA agent in the comparisons.
- Results overview: Figure 4 orders the 14 evaluated agents by increasing average normalized score and reports valid submission rate, average normalized score, and Elo rating.The figure provides an overview of the three aggregate benchmark metrics.
6 Results
Across 14 agents, valid submission and strong performance remain difficult, while reasoning models and search-based scaffolds generally perform better. Greedy scaffolds produce scores closer to SOTA than One-Shot agents, yet a sizeable Elo gap remains between the best agent and human SOTA.
- Agent configurations: 14 agents were evaluated using six language models and three scaffolds: One-Shot, Greedy, and ReAct.Agents are defined as LLM–scaffold pairs.
- Aggregate performance: Reasoning models and Greedy/ReAct scaffolds generally performed better, but valid-submission ability did not always correlate with high performance.Relative rankings were similar but not identical across average normalized score and Elo Rating.
- Valid submissions: 6% and 7% were the invalid-submission task fractions for Greedy gpt-oss-120b and Greedy gpt-oss-20b, respectively.These agents led the benchmark with the smallest fractions of tasks yielding no valid submission.
- Valid submissions: 55.1% of total submissions were valid on average, indicating that producing a valid solution remained difficult for the evaluated agents.The reported mean valid submission rate is defined by Equation 1.
- Aggregate performance: Greedy scaffolds distributed closer to SOTA than One-Shot scaffolds across the task-level normalized-score breakdown.Figure 9 averages normalized scores across multiple seeds for each task and ranks tasks by difficulty.
- Task difficulty: The expert-task normalized scores were uniformly low and similar, whereas easier tasks showed greater variability across agents.Tasks are grouped into easy, medium, hard, and expert buckets of five tasks each.
- Human SOTA comparison: The best-performing agent remained significantly below human SOTA in Elo rating, indicating that AIRS-Bench is far from saturated.Human SOTA is included as an additional Elo opponent.
- Human SOTA comparison: Agents surpassed reported human SOTA on four tasks in at least some runs or seeds.These cases were examined through detailed task inspections, including an original solution.
7 Conclusion
AIRS-Bench evaluates autonomous AI research across diverse, end-to-end machine-learning tasks and reveals substantial performance gaps, infrastructure needs, and benchmark-design constraints.
- Benchmark scope: AIRS-Bench evaluates 20 non-contaminated tasks spanning ideation, methodology design, experimentation, and iterative refinement without baseline code.The tasks cover domains including NLP, mathematics and code, biochemical modeling, and time-series forecasting.
- Empirical findings: Agent performance varies substantially with both the underlying LLM and its harness; most best-performing agents remain significantly behind human SOTA, although some exceed it.Tasks where normalized scores exceed 1.0 remain informative because they measure how far agents progress beyond the human baseline.
- Infrastructure: Scaling AIRS-Bench requires shared, machine-readable research infrastructure because tracking current SOTA, reproducing experiments, and validating tasks are human bottlenecks.The authors identify standardized formats, updates, and automatic task-onboarding pipelines as needed infrastructure.
- Limitations: Resource constraints, including compute, time, and token limits, may influence reported results, while lifting restrictions could enable more flexible agent behavior.The benchmark retains restricted conditions to support future extensive ablations.
- Empirical findings: Scaffold design significantly affects solution quality, indicating opportunities to improve agents by better leveraging test-time compute.The benchmark is intended to expose performance gaps and support development of methods for accelerating scientific progress.
A Task Selection
AIRS-Bench reduces an approximately 100-task pool to 20 stratified tasks by optimizing fidelity to agent performance, category coverage, and ranking preservation.
- Selection objective: AIRS-Bench selects 20 tasks from a pool of approximately 100 to reduce GPU requirements and accelerate benchmarking.The subset is designed to mirror the full pool while making evaluation more lightweight.
- Selection objective: The subset preserves agent performance, category distribution, and relative ranking fidelity relative to the full pool.These criteria guide the selection so the reduced benchmark retains the original pool’s discriminative structure.
- Difficulty stratification: Tasks are partitioned into easy, medium, hard, and expert bands, with candidate allocations sampled under band constraints.The evaluated allocations include uniform, medium-skewed, center-skewed, and medium-heavy distributions.
- Optimization procedure: Three subset-selection algorithms are compared: random search, simulated annealing, and a genetic algorithm.The genetic algorithm evolves candidate subsets through selection, crossover, and mutation while minimizing MAE.
- Selection results: 4.0 × 10^-3 MAE is achieved by the genetic algorithm with a medium-skewed allocation, outperforming other tested configurations.Other competitive configurations have MAE values ranging from 4.6 × 10^-3 to 7.9 × 10^-3.
- Selection results: The subset-versus-full-pool difference in average score never exceeds 0.02 in absolute value, while rankings and confidence intervals remain nearly identical.Validation compares agent mean normalized scores and their 95% confidence intervals across the two sets.
B Additional Results
Additional results describe an identity-transform normalized score and visualize task performance by ranking and difficulty, while noting limits of linear progress measurement.
- Normalized scores: The identity transform makes normalized score linearly reflect progress between the worst observed solution and human SOTA for each task.Human SOTA corresponds to a normalized score of 1, and tasks are ranked by average score across agents.
- Normalized scores: This linear normalization may misrepresent progress when the evaluation metric is highly non-linear or the gap to the optimum is very small.The identity-transform scores are provided for completeness in Figure 12, with difficulty-level breakdowns in Figure 13.
- Figures and tables: Figure 12 reports normalized scores per task averaged over seeds, using each task’s worst-performing run as the baseline score.The scores are computed according to Equations 2–8.
- Figures and tables: Figure 13 groups the Figure 12 task ranking into easy, medium, hard, and expert categories with decreasing normalized scores.Table 6 maps task numbers to task names and reports their average scores across seeds and agents.
B.1 Task run files
The task run files define a machine-learning task, dataset and evaluation interface, prediction format, and validation checks for executable submissions.
- Task definition: Each run file frames the task as a machine-learning project requiring a model that solves the specified task using data in the provided directory.The SVAMP example asks agents to solve math word problems and predict numerical answers.
- Task definition: The SVAMP dataset includes identifiers, narrative bodies, questions, equations, answers, problem types, and concatenated question text.The example asks for the size of each banana group and identifies the problem type as Common-Division.
- Evaluation: Evaluation loads labeled answers, compares integer predictions with labels, and returns Accuracy as the metric.The evaluator requires a labeled dataset containing an Answer feature for scoring.
- Submission handling: The submission workflow loads predictions from a file, checks that their row count matches the test-set size, and reports file or loading errors.The example assumes a headerless CSV containing prediction values, subject to adjustment for other formats.
B.1.3 metadata.yaml
This benchmark metadata defines a text-generation math-question-answering task using the SVAMP dataset and specifies its evaluation and agent interaction workflow. Agents explore data, establish validation, reproduce baselines, iteratively edit code, and submit solutions through a constrained interface.
- Task metadata: Math Question Answering uses text generation on the ChilleD/SVAMP dataset, scoring the Answer column with accuracy.
- Task metadata: The task does not use custom gold labels, custom radius classes, or a lower-is-better metric.
- Agent interface: Agents work as autonomous machine-learning researchers through a special command-line interface and file editor.
- Agent interface: The interface supports file navigation, creation, search, editing, insertion, and evaluation submission commands.
- Submission constraints: Submissions use hidden test scores to discourage overfitting, while invalid formatting or indentation can cause parsing or execution failures.
- Environment constraints: The environment does not support interactive session commands such as Python or Vim.
- Machine-learning workflow: The prescribed workflow covers exploration, validation setup, baseline reproduction, and iterative optimization toward the hidden test set.
D Compute Requirements of Benchmarks
The benchmark reports compute, runtime, and cost information, alongside a cache of 193 pretrained HuggingFace models. The cache excludes frontier models and contains no model newer than DeBERTa-v3-large, released in 2021.
- Resources: The benchmark summarizes compute, runtime, and cost information for recent LLM-agent benchmarks.
- Model cache: Agents had access to 193 pretrained HuggingFace models in the run’s cache.
- Model cache: The cache excludes frontier models, with DeBERTa-v3-large released in 2021 as its newest model.
F Distribution of tasks SOTA venue and year
AIRS-Bench tasks are categorized by the publication venue and year of their state-of-the-art reference papers. Figure 14 presents both distributions, while Table 10 provides a detailed venue breakdown.
- Figure 14: Figure 14 breaks down AIRS-Bench tasks by SOTA publication venue and publication year.
- Table 10: Table 10 provides a detailed breakdown of the venues where the SOTA papers were introduced.
G AIRS-Bench Task Description
AIRS-Bench spans coding, code search, language understanding, molecular-property prediction, mathematics, reading comprehension, financial reasoning, sentiment, and semantic evaluation. Tasks pair domain-specific datasets with corresponding generation, classification, regression, ranking, extraction, or similarity metrics.
- Coding and retrieval: APPS evaluates five distinct Python programs per coding problem against hidden test cases.
- Coding and retrieval: CodeXGlue Code Search Adv requires ranking Java code snippets by relevance to natural-language queries.
- Language understanding: WSC and Winogrande test pronoun-reference resolution, using binary classification or accuracy over two answer options.
- Molecular prediction: QM9 and ZINC tasks predict molecular properties from molecular geometry, atomic composition, or graph structure using regression.
- Reasoning and comprehension: SVAMP evaluates numerical answers to narrative math problems involving operations such as addition, subtraction, multiplication, or division.
- Reasoning and comprehension: DuoRC, ELI5, FinQA, and SQuAD cover contextual question answering, long-form generation, financial table reasoning, and answer-span extraction.
- Classification and semantics: Yelp sentiment classification uses five star-rating classes and accuracy, while SICK evaluates entailment classification and semantic relatedness.