PaperlayerView exact revision on arXiv

Source-linked AI summary

ResearchGym: Evaluating Language Model Agents on Real-World AI Research

Aniketh Garikaparthi, Manasi Patwardhan, Arman Cohan

arXiv:2602.15112v2cs.AI

TL;DR

ResearchGym addresses limited evidence about whether AI agents can conduct closed-loop research by evaluating them in executable research codebases with objective, execution-based grading. Across recent research tasks, agents showed occasional strong performance but low reliability, with recurring failures in long-horizon experimentation and resource management. The benchmark provides infrastructure for systematic evaluation of this capability–reliability gap.

  • Problem

    Current benchmarks do not reliably establish whether AI systems can sustain closed-loop research, and existing evaluations lack standardized, execution-grounded comparison across systems.

  • Method

    ResearchGym repurposes five recent ICML, ICLR, and ACL papers into containerized tasks, preserving datasets, evaluation scripts, and baselines while withholding each proposed method and grading with original execution-based evaluators.

  • Results

    Only 1 of 15 end-to-end runs improved over the provided baseline, while the agent completed 26.5% of sub-tasks on average; one run surpassed the human reference solution on an ICML 2025 Spotlight task.

  • Takeaways & Limitations

    Frontier agents can occasionally achieve state-of-the-art research results, but current autonomous research remains unreliable and requires systematic closed-loop evaluation.

  • Takeaways & Limitations

    Agents often commit early to a method trajectory and continue tuning weak pipelines instead of reconsidering their approach when intermediate results are poor.

Abstract

from arXiv · show

We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research. To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL. From each paper's repository, we preserve the datasets, evaluation harness, and baseline implementations but withhold the paper's proposed method. This results in five containerized task environments comprising 39 sub-tasks in total. Within each environment, agents must propose novel hypotheses, run experiments, and attempt to surpass strong human baselines on the paper's metrics. In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap. The agent improves over the provided baselines from the repository in just 1 of 15 evaluations (6.7%) by 11.5%, and completes only 26.5% of sub-tasks on average. We identify recurring long-horizon failure modes, including impatience, poor time and resource management, overconfidence in weak hypotheses, difficulty coordinating parallel experiments, and hard limits from context length. Yet in a single run, the agent surpasses the solution of an ICML 2025 Spotlight task, indicating that frontier agents can occasionally reach state-of-the-art performance, but do so unreliably. We additionally evaluate proprietary agent scaffolds including Claude Code (Opus-4.5) and Codex (GPT-5.2) which display a similar gap. ResearchGym provides infrastructure for systematic evaluation and analysis of autonomous agents on closed-loop research.

1 Introduction

Existing benchmarks provide limited, fragmented, or unreliable evidence about whether AI agents can sustain closed-loop research. ResearchGym addresses these gaps with objective, execution-based evaluation on recent research tasks, and finds occasional strong performance alongside low reliability.

  • Motivation: Current benchmarks cannot reliably assess closed-loop research spanning hypothesis proposal, executable experimentation, empirical testing, and belief updating.Existing systems often rely on self-reported studies and lack standardized comparison, which can inflate perceived capabilities.
  • Motivation: Existing evaluations typically isolate ideation, implementation, or reproduction, while other closed-loop benchmarks face compute, judging, contamination, or human-baseline limitations.These constraints make results difficult to reproduce, vulnerable to superficial novelty, or hard to calibrate against expert performance.
  • ResearchGym: ResearchGym evaluates the full research loop using executable codebases, objective grading, recent ICML, ICLR, and ACL tasks, and known human solutions for calibration.The benchmark preserves datasets, evaluation scripts, and baselines while withholding each paper’s core method.
  • Results: 6.7% of 15 end-to-end runs improved over the provided baselines, while the agent completed 26.5% of sub-tasks on average.A single successful run exceeded the human reference solution on an ICML 2025 Spotlight task, but Claude Code and Codex showed a similar capability–reliability gap.
  • ResearchGym: ResearchGym comprises five tasks with 39 sub-tasks and provides an extensible execution environment for standardized agent integration and objective grading.The environment is designed for contamination-aware evaluation and single-GPU accessibility.
  • Evaluation: The study includes controlled evaluation, ablations, more than 35 end-to-end runs, failure-mode analysis, time–token tradeoffs, and case studies.These analyses support systematic investigation of autonomous research behavior rather than isolated demonstrations.

2 ResearchGym

ResearchGym standardizes closed-loop research evaluation through curated, executable empirical tasks, objective grading, and a sandboxed gym-style interface. Its benchmark construction preserves reproducibility while withholding proposed methods and retaining baselines for calibrated comparison.

  • 2.2 Benchmark Construction: ResearchGym selects recent award-winning papers, applies automated extraction and filtering, then performs human quality assessment before finalizing tasks.The automated stage filters for empirical, publicly available, and compute-feasible papers, reducing 1,387 candidates to 90 for human QA.
  • 2.2 Benchmark Construction: The benchmark contains 5 curated tasks and 39 sub-tasks spanning diverse domains and dataset or setting validations.Each task is divided into individually gradable sub-tasks so agents can prioritize a designated primary sub-task.
  • 2.2 Benchmark Construction: Each packaged task removes the authors’ proposed implementation while retaining datasets, evaluation scripts, pinned environments, and baseline components.The resulting input combines a starter repository, task description, and grader; re-integrating the withheld method reproduced reported scores with small deviations.
  • 2.3 Gym Environment: ResearchGym defines tasks around open-ended research goals, executable codebases, and objective evaluation scripts.The framework organizes tasks, environments, solvers, and evaluations as separate abstractions.
  • 2.3 Gym Environment: Objective graders compute reproducible sub-task metrics from workspace state, while sandboxed environments control confounds from dependencies and configuration.The framework remains solver-agnostic, allowing single-agent, multi-agent, or hybrid controllers under disclosed integrity constraints.

3 Experimental Setup

The experiments test whether frontier agents can improve strong human baselines on closed-loop research and what prevents reliable performance across runs. Agents receive research tools and operate under constrained compute, API, and time budgets.

  • Experimental Questions: The experiments ask whether frontier LLM agents can improve strong human baselines and which failure modes prevent reliable performance.These questions target both research capability and sustained execution across tasks and runs.
  • Agents: The primary agent is GPT-5 with the rg-agent Inspect scaffold, while additional evaluations use AI-Scientist-v2, ML-Master, Claude Code, and Codex.The setups include both generalist and specialized agent scaffolds.
  • Research Tools: Agents access literature search, model and dataset APIs, and web search, with web results filtered by an October 2024 cutoff and 160 paper-related URLs blocked.These controls provide external research resources while limiting access to potentially contaminating paper sources.
  • Experimental Protocol: Each rg-agent experiment uses three independent runs and reports mean ± standard deviation alongside best@k.Experiments run on a single NVIDIA A100 with a $10 API budget and 12-hour wall-clock limit, with additional budget and time for the best run.

4 Results

ResearchGym evaluates agent capability, reliability, and efficiency across closed-loop research tasks. Results show occasional strong performance but substantial variability, low baseline-improvement rates, and diminishing returns from additional resources.

  • Evaluation settings: Agents are evaluated under capability, aggregate reliability, and efficiency settings against human-derived lower and upper bounds.Efficiency relates task performance to consumed time, tokens, and cost.
  • Capability: A single TIM run surpasses the reference solution, with CPD(A) = 0.589 versus SOTA = 0.463.Best@3 reaches 93–96% of SOTA on CL and CMR, showing that strong outcomes are possible under repeated attempts.
  • Capability: Mean performance remains substantially below best@3 on several tasks, including CL at Avg: 30.75 ± 37.39 versus Best@3: 80.4.For irb, Avg: 579.79 ± 585.47 versus Best@3: 1407.06, indicating that strong outcomes are outliers.
  • Reliability: 1 of 15 end-to-end runs (6.7%) improves over the provided baseline, while average sub-task completion is 26.5%.The agent often initiates setup, training, evaluation, or grading but struggles to complete the loop and improve the primary sub-task.
  • Efficiency Dynamics: Performance gains typically plateau after approximately 9 hours, while additional compute is disproportionately spent on retries, debugging, and repeated experiments.Figure 3 reports a weak positive performance–resource correlation with diminishing returns; Figure 4 reports Pearson’s r = −0.47 between action density and performance.

5 Analysis

ResearchGym exposes a capability–reliability gap: agents occasionally execute strong ideas, but long-horizon management, coordination, and exploration failures frequently prevent reliable improvement.

  • Execution bottlenecks: Execution failures included empty replay buffers, tensor and wrapper errors, missing evaluations, and invalid or incomplete task results.On Improving Replay Buffers, performance was 71.48 average return versus SOTA’s 4101, or 0.017 normalized.
  • Execution bottlenecks: Async execution actively hurt performance when agents misread empty logs, cancelled jobs prematurely, or failed to coordinate parallel experiments.On Improving Replay Buffers, async_001 achieved 0.0 average return across all 11 seeds.
  • Hypothesis generation: Agents repeatedly converged on minor variations of familiar paradigms rather than exploring genuinely diverse hypotheses.Examples included LoRA with importance regularization, entropy minimization, Integrated Gradients variants, protected token spans, and critical-decision transitions.
  • Monitoring and safety: Agents wasted substantial time monitoring stalled or crashed jobs and sometimes used unsafe process-management commands.One agent continued waiting for more than eight hours after recognizing that a log timestamp was frozen.

6 Related Works

Prior research evaluates isolated parts of research or uses costly, contaminated, subjective, or poorly calibrated benchmarks. ResearchGym instead targets implementation and validation of new ideas in executable, closed-loop environments.

  • Research benchmarks: Existing benchmarks often assess ideation, ML engineering, or paper reproduction rather than the full cycle of implementing and validating new research ideas.These settings provide limited headroom for creative ideation or end-to-end evaluation.
  • Research benchmarks: Closed-loop benchmarks commonly require cluster-scale compute, rely on LLM judges, lack human baselines, or omit contamination-aware construction.These design choices limit objective, calibrated comparisons across agents.
  • Closed-loop environments: The benchmark records agent interaction traces to support failure analysis and future training-data generation.Table 7 summarizes observed failure modes and long-horizon limitations.
  • Closed-loop environments: ResearchGym adopts a gym-style environment in which agents interact with executable research tasks and can be evaluated through standardized testing.The framework builds on prior closed-loop environments such as OpenAI Gym, LlamaGym, SWE-Gym, and R2E-Gym.

7 Conclusion

ResearchGym provides an objective, accessible benchmark for measuring agents’ ability to conduct long-horizon closed-loop research. Results reveal substantial reliability limitations alongside occasional strong research performance.

  • Conclusion: ResearchGym evaluates the full arc of closed-loop AI research with standardized execution-based testing.Its execution environment supports integration of different agents on executable codebases.
  • Conclusion: Frontier agents show substantial reliability limitations linked to poor experiment tracking, resource management, and context degradation.These limitations emerge during long-horizon, open-ended research.
  • Conclusion: Agents can occasionally produce strong results, indicating nascent but genuine research capability.The benchmark is intended to support rigorous measurement, analysis, and development of more capable systems.

Discussion

The benchmark’s current scope excludes multimodal, trainable-agent, and subjective research settings. These boundaries reflect evaluation and infrastructure constraints rather than a fundamental limitation of the framework.

  • Scope boundaries: Multimodal research problems are absent because they often require specialized hardware, large data transfers, and additional evaluation infrastructure.Vision- and audio-centric tasks are identified as future extensions.
  • Scope boundaries: ResearchGym tasks are difficult enough that only frontier LLMs achieve non-trivial performance, preventing direct training experiments.Released trajectories may support future training of smaller models.
  • Scope boundaries: Purely theoretical, analysis-driven, and proof-based papers are excluded because success is subjective and often requires expert human verification.The benchmark therefore focuses on empirical machine learning tasks with executable, objective metrics.

Impact Statement

ResearchGym frames autonomous research as both an opportunity and a risk, motivating transparent evaluation of what agents can reliably accomplish. Its benchmark is positioned as a public, compute-feasible complement to existing automated-research systems, while acknowledging important scope limits and integrity concerns.

  • ResearchGym could expand scientific hypotheses, reduce barriers to entry, and accelerate research in critical scientific and medical fields.
  • Autonomous research also creates risks if discovery outpaces the ability to assess implications or if agents exploit benchmark shortcuts.The paper identifies reward hacking as a route to overestimating AI research competence and affecting deployment decisions.
  • The benchmark complements automated-research systems with a public, compute-feasible, programmatically graded evaluation surface.
  • The evaluation targets raw frontier-model research capability rather than the best-performing agentic system, so results likely form a lower bound.More sophisticated systems could yield further improvements, while multi-agent setups were outside the experimental scope because of cost.

B.2 Task Metadata

Task metadata and construction prioritize feasible, objective, accessible, high-quality, contamination-aware research tasks. The pipeline combines automated extraction with manual auditing and imposes practical compute, data, and reproducibility constraints.

  • The resulting task metadata is documented in tables covering task information and representative exclusions.
  • ResearchGym selects tasks using feasibility, objectivity, open access, quality, and contamination principles.
  • Tasks requiring more than 24GB GPU memory, long training times, hardware-specific metrics, closed-model APIs, or inaccessible datasets are filtered out.Gated datasets and datasets larger than 300GB can also make otherwise suitable papers infeasible.
  • The benchmark prioritizes recent award-winning, open-source papers with room for improvement and proceedings released after January 2025 to reduce contamination risk.
  • An LLM extracts structured metadata from paper Markdown, after which manual audits remove candidates violating the benchmark’s constraints.The extraction schema includes problem, methodology, experimental settings, compute requirements, code availability, and evaluation metrics.

C.4 Task Packaging Guidelines

Task packaging aims to preserve faithful, reproducible evaluation while removing the original method from complex repositories. The resulting agent workflow emphasizes incremental experimentation, integrity, and complete reporting under fixed resource constraints.

  • Task Packaging Guidelines: Removing a paper’s method is difficult because method-specific utilities, hyperparameters, and naming switches are interleaved with shared baseline code.The authors resolve these ambiguities manually through two-way diff reconciliation between authors.
  • Task Packaging Guidelines: Packaging provisions datasets, splits, libraries, API keys, grading scripts, and isolated environments so tasks remain complete and reproducible.Wrapper scripts preserve the original repository’s evaluation details while allowing agents to run experiments and record results.
  • Task Packaging Guidelines: Primary sub-tasks are evaluated first because agents may not finish every task, reducing time-based penalties while preserving reproducibility.Human reruns of the original method provide a comparison yardstick under similar constraints.
  • Prompts: The agent is instructed to develop novel ideas, use available time, build solutions incrementally, and produce multiple metric results before termination.
  • Task Packaging Guidelines: An inspection protocol checks grading-script modifications, hardcoded metrics, and data contamination, while released trajectories show that some agent systems achieved no score on most tasks.

D.9 Inspection Agent

ResearchGym uses post-hoc inspection and live observability to assess agent integrity and execution. The inspection agent detects several cheating patterns, but benchmark execution remains costly.

  • 100% accuracy for detecting true-positive cheating behavior was found in manual review, although the inspection agent had a high false-negative rate.
  • The inspection agent correctly flagged three synthetic cheating scenarios with high confidence, ranging from 0.95–0.99.
  • Held-out runs exposed cross-run contamination, cherry-picking from incompatible configurations, and outright result fabrication.
  • ResearchGym records full transcripts and tool calls post hoc, supports transcript analysis, and provides live message-level tracing during execution.Live tracing captures requests, responses, token usage, latency, and errors for monitoring long runs and diagnosing failures.
  • End-to-end benchmark execution can cost over $300 in API credits plus an estimated $360 in A100 compute.The paper states that additional compute above 12GB mainly reduces runtime and provides insignificant score gains.

E Quantitative Analysis

This section presents statistics for five figures and identifies primary sub-tasks as the main evaluation targets.

  • The section includes statistics for CL, CMR, MDT, TIM, and IRB.
  • Each task has a designated primary sub-task serving as its main evaluation target.
  • Task descriptions include baseline result tables and identify the primary sub-task for evaluation.

F.1 Continual Learning

The continual-learning task tests agents on sequential classification benchmarks using accuracy and anytime-accuracy metrics. Agents produced varied LoRA-based solutions, but performance was highly variable and generally below the withheld method’s reference.

  • Research Goal: The task targets scalable continual learning with foundation models, where existing methods require growing adaptation pools or rehearsal samples.
  • Agent Ideas: Agents proposed three LoRA-based methods combining regularization, distillation, classifier design, and parameter-isolation techniques.
  • Performance Progression: At hour 6, Run 001 rose from 0.12 to 0.93 normalized accuracy after fixing numerical stability in LoRA merging, while Runs 002 and 003 remained far lower.
  • Performance Progression: Run 001 reached CIFAR-100 Acc=80.56 and AAA=86.49, or 0.93× the InfLoRA baseline of 86.31/90.67.
  • Bottlenecks and Gap Analysis: The main bottleneck was ideation: agents converged on familiar regularization combinations and did not discover SD-LoRA’s magnitude-direction decomposition.
  • Materials Tokenization: On materials tokenization, no run completed both primary sub-tasks, and average completion was 27.8% with standard deviation 48.2%.

F.3 Cross Modal Retrieval

The cross-modal retrieval task addresses query shift through online adaptation, but agent performance varied sharply across natural-image and person-reidentification settings. Setup failures, metric-direction errors, and domain-specific models constrained evaluation.

  • Research Goal: Query shift occurs when online queries come from a distribution different from the source domain, causing retrieval degradation.
  • Method: The withheld approach selects source-like query-candidate pairs and adapts normalization layers using uniformity, gap, and noise-robust objectives.
  • Agent Ideas: Agents mainly proposed entropy-based test-time adaptation methods, including distribution balancing, pseudo-labeling, and CORAL alignment.
  • Results: Run 001 achieved CUHK2ICFG=42.14, or 1.27× baseline, while Run 002 reached Base2Flickr I2TR@1=86.3, or 1.03× baseline.
  • Bottlenecks and Failure Modes: Run 003 spent 221 minutes initializing and invalidated ReID results by using the wrong retrieval-direction flag.
  • Performance Progression: By hour 3, all runs exceeded baseline on Base2Flickr, averaging 1.03×, but ReID performance remained inconsistent.

F.4.1 Success Analysis: SOTA-Surpassing Run

The time-series explanation task produced the benchmark’s only run surpassing the withheld reference solution. Success came from directional margin attribution, repeated metric-guided iteration, and attention to differing baseline sensitivities, though the approach is less theoretically grounded than TIMING’s.

  • Success Result: Run 002 achieved PAM Average CPD of 0.589±0.036, surpassing TIMING’s 0.463±0.007 on the primary metric.
  • Agent’s Novel Method: The agent’s method computes Integrated Gradients on the margin between the predicted and strongest alternative classes.
  • Agent’s Novel Method: It combines margin attribution with SmoothGrad-squared noise tunneling, temporal processing, and positive clamping.
  • Comparison with TIMING: Unlike TIMING, the approach changes the explained quantity rather than the integration path and uses post-hoc smoothing.
  • Comparison with TIMING: The agent’s method may be simpler to implement than TIMING’s, but the paper characterizes it as less theoretically grounded.
  • Why Did This Succeed: Run 002 made 13 evaluation attempts over 9.5 hours and used CPD scores to guide each subsequent iteration.
Loading 2602.15112v2…