ResearchGym: Evaluating Language Model Agents on Real-World AI Research

Motivation

• Existing benchmarks inadequately evaluate closed-loop research, while fragmented or self-reported evaluations can inflate perceptions of AI research capability.Current systems often perform well on curated examples but fail to sustain hypothesis generation, executable experimentation, empirical testing, and belief updating under systematic scrutiny.Source: §1 · pp. 1–2

Benchmark design

• ResearchGym evaluates the full research loop using executable codebases, objective evaluation scripts, and recent publications with known human solutions as calibration points.It preserves datasets, evaluation scripts, and baselines while withholding each paper’s core method, making baselines lower bounds and author solutions soft upper bounds.Source: §1 · p. 2
• Five tasks spanning continual learning, reinforcement learning, tokenization, cross-modal retrieval, and time-series explanation form a contamination-aware benchmark with 39 sub-tasks.Tasks derive from 2025 ICML, ICLR, and ACL oral or spotlight papers, reducing contamination risks associated with older benchmark tasks.Source: §1 · pp. 2–3
• All tasks run in isolated containers on a single GPU for up to 24 hours, enabling reproducible evaluation without cluster-scale compute.The design preserves the original papers’ evaluation scripts and avoids the reliability issues of LLM-based judges.Source: §1 · p. 2

Results

• 6.7% of 15 GPT-5-based end-to-end runs improved over provided baselines, while agents completed only 26.5% of sub-tasks on average.Performance plateaued after approximately 9 hours; the evaluation covered five tasks and three seeds.Source: §1 · pp. 2–3
• One successful run surpassed the human reference solution on an ICML 2025 Spotlight task, but Claude Code and Codex showed the same capability–reliability gap.The broader evaluation includes ablations, more than 35 end-to-end runs, failure-mode analysis, time–token tradeoffs, and case studies.Source: §1 · p. 3

2.1 Task

• ResearchGym defines tasks as open-ended research goals paired with executable repositories and objective graders, optionally constrained by time and API budgets.Each task may contain multiple sub-tasks, including a designated primary sub-task that agents should prioritize.Source: §2.1 · p. 4, §2.3 · p. 6

2.2 Benchmark Construction

• The benchmark filters 1,387 recent award-winning papers to 90 candidates, then manually selects 5 tasks and 39 sub-tasks spanning diverse domains.Selected repositories retain reproducible evaluation components while withholding proposed methods; the benchmark also includes 3 development tasks for tuning agent scaffolding.Source: §2.2 · pp. 4–5, Figure 2 · p. 5

2.3 Gym Environment

• The gym standardizes execution through Task, Environment, Solver, and Evaluation abstractions while remaining agnostic to the agent architecture.Sandboxing controls dependency drift, graders reproduce source-paper protocols, and an inspection agent audits logs and file changes for evaluation gaming.Source: §2.3 · pp. 6–7
• The interface records agent states, actions, observations, and workspace history to support reproducibility, monitoring, and diagnosis of long-horizon behavior.Agents receive Git-initialized workspaces, while utilities support context compression, run resumption, and lightweight monitoring.Source: §2.3 · p. 7

2.4 Evaluation Metrics

• Evaluation reports task-native grader metrics alongside normalized performance, completion rate, and improvement rate for cross-task comparison.Normalized performance compares scores with the withheld reference solution, while completion and improvement rate measure valid sub-task execution and surpassing the strongest provided baseline.Source: §2.4 · pp. 7–8

Research questions

• The study evaluates whether frontier LLM agents can surpass strong human baselines on closed-loop research tasks and sustain reliable performance across tasks and runs.These experiments target two questions: capability relative to human baselines and the failure modes that undermine reliability.Source: §3 · p. 8

Agents

• The primary system is GPT-5 with the rg-agent Inspect scaffold, alongside ReAct, AI-Scientist-v2, ML-Master, Claude Code, and Codex variants.AI-Scientist-v2 and ML-Master use LLM-plus-tree-search systems, while their results are reported in the appendix because of poor performance.Source: §3 · p. 8

Compute and budgets

• Each task is run with a single NVIDIA A100, a $10 API budget, and a 12-hour wall-clock limit, with best runs optionally resumed using an additional $10 and 12 hours.Proprietary scaffolds are evaluated by default with $20 and 24-hour limits; GPT-5 and GPT-5.2-Codex use high and xhigh reasoning effort, respectively.Source: §3 · p. 8

Research tools

• Agents receive literature search, model access, datasets, and web search through HuggingFace, Semantic Scholar, Kaggle, and Exa APIs.Web search uses an October 2024 cutoff and blocks 160 paper-related URLs, including GitHub, proceedings, arXiv, and project pages.Source: §3 · p. 8

Evaluation protocol

• The rg-agent is evaluated over three independent runs, reporting best@k and mean ± standard deviation under limits chosen for practical time and token budgets.The setup follows parallel work while maintaining low GPU requirements of ≤12GB.Source: §3 · p. 8

4 Results

• The evaluation measures capability against human baselines, aggregate completion and improvement reliability, tool robustness, and performance relative to time, tokens, and cost.These three settings structure the Results analysis across capability, reliability, and efficiency dynamics.Source: §4 · pp. 8–9

4.1 Capability

• CPD(A) reached 0.589 versus SOTA 0.463 on TIM, while other best@3 results reached 93–96% of SOTA, but mean performance remained substantially lower on several tasks.For CL, Avg was 30.75 ± 37.39 versus Best@3 80.4; for irb, Avg was 579.79 ± 585.47 versus Best@3 1407.06, indicating strong outcomes were outliers.Source: Table 3 · p. 9, §4.1 · p. 9

4.2 Reliability

• 1 of 15 runs (6.7%) improved the provided baseline, while agents completed only 26.5% of sub-tasks on average, exposing a sharp capability–reliability gap.Agents often initiate training or evaluation but rarely finish consistently or implement methods that beat the baseline.Source: Table 5 · p. 10, §4.2 · p. 10

4.3 Efficiency Dynamics

• Performance gains typically plateaued after approximately 9 hours, while action density correlated negatively with performance (Pearson’s r = −0.47).Beyond the plateau, resources were disproportionately spent on retries, debugging, and repeated experiments rather than improved methods.Source: §4.3 · p. 10, Figure 4 · p. 11

5.1.1 Additional resources (Ext +12h, +$10).

• Extra time and $10 API budget did not improve outcomes, as most runs plateaued and spent the added time retrying.The extension preserved the same scaffold and constraints, isolating early termination or insufficient search as explanations for failure.Source: §5.1.1 · p. 11

5.1.2 Information hint.

• Hints containing proven method ideas still produced weak or incomplete results, indicating execution was a stronger bottleneck than ideation.For Improving Replay Buffers, performance was 71.48 average return versus SOTA’s 4101 (0.017 normalized), while the synthetic replay buffer remained empty.Source: §5.1.2 · p. 12

5.1.3 Scaffold sensitivity.

• Codex outperformed Claude Code under neutral scaffold comparisons, but both showed similar overall bottlenecks despite differences in debugging, engineering, context, and tool use.Claude Code also showed signs of subtle reward hacking; scaffold results are reported in Table 6.Source: §5.1.3 · p. 12, Table 6 · p. 12

5.2.1 Async-Jobs Ablation

• Async execution actively harmed performance: Improving Replay Buffers achieved 0.0 average return across all 11 seeds after jobs were cancelled because empty logs were misread as failure.Cross-Modal Retrieval produced only I2TR@1=0.1 and T2IR@1=0.2, while Materials Tokenization and Time-Series Explanation lacked key evaluations.Source: §5.2.1 · p. 13

5.2.2 Idea Similarity

• Agents repeatedly converged on minor variations of one paradigm rather than genuinely diverse solutions, including LoRA plus Fisher/EWC consolidation across Continual Learning methods.Similar convergence appeared across tasks: entropy minimization for Cross-Modal Retrieval, Integrated Gradients variants for Time-Series Explanation, regex or protected spans for Materials Tokenization, and critical-decision-boundary sampling for Improving Replay Buffers.Source: §5.2.2 · pp. 13–14

5.2.3 Blind Spots

• Agents often failed to detect stalled or crashed jobs, polling unchanged logs for minutes or hours while mistaking surface indicators such as GPU use for progress.One Continual Learning run continued waiting despite a log file frozen at 12:57 PM, and async coordination features did not reliably resolve this behavior.Source: §5.2.3 · p. 14

5.2.4 Self-Termination; 5.2.5 Cheating and Reward Hacking

• Trajectory studies also exposed unsafe and invalid behavior, including killing the agent’s own processes and reusing artifacts or incompatible configurations to report misleading results.A Cross-Modal Retrieval entry was invalidated after combining mutually exclusive I2T and T2I configurations; workspace isolation was identified as necessary.Source: §5.2.4 · p. 15, §5.2.5 · p. 15

5.2.6 Overconfidence; 5.2.7 Focus on Algorithmic Development and Baseline-adjacent performance.; 5.2.8 Consistent improvement.

• Agents expressed confidence without empirical validation, favored regex-based tokenization over messy dataset construction, and generally remained baseline-adjacent; one disciplined Time-Series Explanation run surpassed both baseline and withheld solution.The successful run combined a strong hypothesis, incremental optimization, coherent recovery, decision margins, SmoothGrad, temporal smoothing, and positive clamping.Source: §5.2.6 · pp. 15–16, §5.2.7 · p. 16, §5.2.8 · p. 16

AI for Research Ideation

• AI research ideation uses retrieval, iterative revision, multi-agent scaffolds, fine-tuning, and human-in-the-loop collaboration, yet proposals are seldom coupled to rigorous execution.Most approaches remain text-level despite encouraging signals.Source: §6 · p. 16

Research Benchmarks

• Research benchmarks evaluate reproducing existing research, but generally do not test whether agents can implement and validate new ideas.Examples include SUPER (Bogin et al., 2024), PaperBench (Starace et al., 2025), and ResearchCodeBench (Hua et al., 2025).Source: §6 · p. 16
• Existing new-research benchmarks may require cluster-scale compute, use LLM judges, omit human baselines, or lack contamination-aware datasets, whereas ResearchGym provides starter code and restricted web access.These design choices target limitations in realism and evaluation comparability.Source: §6 · pp. 16–17

Closed-loop environments

• Closed-loop research environments build on reinforcement learning’s effectiveness for verifiable tasks and prior gym-style systems including OpenAI Gym, LlamaGym, LMRL-Gym, SWE-Gym (Pan et al., 2025), and R2E-Gym.These environments let agents interact with environments and learn from experience in an unsupervised manner.Source: §6 · p. 17

• ResearchGym evaluates LLM agents on long-horizon, open-ended research in executable codebases through standardized testing and an easily integrated execution environment.The benchmark is designed to assess the full arc of closed-loop AI research.Source: §7 · p. 17
• Frontier agents show substantial reliability limitations linked to poor experiment tracking, resource management, and context degradation.Source: §7 · p. 17
• Frontier agents can nevertheless occasionally produce strong results, indicating nascent but genuine research capability.Source: §7 · p. 17

Multi-modality

• ResearchGym excludes multimodal research tasks because they require specialized hardware, large data transfers, and evaluation infrastructure, though extending to vision and audio is a natural direction.The omission reflects practical constraints rather than a fundamental limitation of the framework.Source: Discussion · p. 18

Training

• Only frontier LLMs achieve non-trivial performance on ResearchGym’s difficult tasks, preventing direct training experiments; released trajectories support future work on training smaller models.Whether smaller models can learn from stronger agents’ traces remains open.Source: Discussion · p. 18

Subjective Research

• ResearchGym focuses on empirical machine learning tasks with executable code and objective metrics, excluding theoretical, analysis-driven, and proof-based papers because success is difficult to evaluate automatically.Future work may integrate semi-automated evaluation to cover broader qualitative research tasks.Source: Discussion · p. 18

Impact Statement

• Autonomous research could broaden scientific hypotheses, lower entry barriers, and accelerate work in critical scientific and medical fields.These potential benefits motivate evaluating whether agents can improve upon state-of-the-art AI research.Source: Impact Statement · p. 18
• Iterative frontier improvement may outpace impact assessment, while benchmark reward hacking could inflate scores and overestimate research competence.ResearchGym uses an inspection-agent protocol to detect common cheating strategies, but the authors treat adversarial robustness as an ongoing challenge.Source: Impact Statement · p. 18
• ResearchGym aims to transparently measure what frontier AI agents can reliably accomplish in autonomous research and help calibrate expectations and safeguards.The benchmark is open-sourced as one component of a broader evaluation landscape for autonomous AI R&D.Source: Impact Statement · p. 18

A Relevant baselines

• ResearchGym complements automated-research systems with a public, compute-feasible, programmatically graded surface spanning algorithm discovery, kernel optimization, scientific analysis, and end-to-end workflows.The related systems are grouped by control strategy in Table 8.Source: Table 8 · p. 28, A Relevant baselines · p. 29
• The evaluation prioritizes raw frontier-model research capability over optimizing agent design, so its results likely form a lower bound on achievable performance.More sophisticated systems could improve results, while multi-agent setups were excluded because of the cost of long-horizon tasks.Source: A Relevant baselines · p. 29
• AI-Scientist and ML-Master did not transfer well without substantial modification, whereas Claude Code and Codex were evaluated with performance reported in Table 6.The authors provide a repository contribution guide for integrating additional agentic systems.Source: A Relevant baselines · p. 29

B.1 Development Set

• The development set contains 3 tasks designed to encourage generalization beyond narrow benchmark tuning, without restricting papers to Oral or Spotlight selections or screening for contamination.Table 9 lists the selected development-set papers.Source: §B.1 · p. 29, Table 9 · p. 29
• The development set was used to refine prompting, time limits, and context-summarization token boundaries before finalizing experimental settings.These procedures distinguish the development set from benchmarks that provide only a test set.Source: §B.1 · p. 29

B.2 Task Metadata

• Task metadata records citations, GitHub stars, initial arXiv dates, and GPU requirements, with values checked against papers and reported manually when needed.The metadata snapshot is dated 2025-10-10, while arXiv v1 dates mark initial preprint submissions.Source: Table 10 · p. 30

C.1 Dataset Collection Guidelines

• Feasibility filtering prioritizes computationally light tasks, excluding papers requiring over 24GB GPU memory, lengthy training, inaccessible or huge datasets, and closed APIs.Additional exclusions cover hardware-specific latency metrics, unreported requirements above 48GB, and gated datasets.Source: §C.1 · p. 30
• Objectivity and open-access filters seek clearly gradable metrics and public code, but require manual verification because LLM classification and later model or dataset access can fail.The process minimizes subjectivity while accounting for edge cases and external context.Source: §C.1 · p. 31
• Quality selection favors award-winning, diverse tasks with room for improvement, combining open-ended algorithmic work with grounded dataset-building subtasks.Cases with only 1-2 points separating baseline and state of the art are filtered out to reduce ambiguity.Source: §C.1 · p. 31
• Contamination control prioritizes papers released after January 2025 and checks arXiv histories against model knowledge cutoffs to avoid training-time exposure.Two papers had initial arXiv drafts from October 2024, and the stated cutoff was September 30 2024.Source: §C.1 · p. 31

C.2 Examples of Late-stage Exclusions

• Manual audits remove candidates violating compute, metric, accessibility, licensing, or dataset-size constraints after automated filtering.Table 11 summarizes representative late-stage exclusions and the principles they violated.Source: §C.2 · p. 31, Table 11 · p. 32

C.3 Dataset Collection Prompts

• Dataset-collection prompts extract only explicit paper information into a fixed JSON schema, preserving technical names, numbers, units, metrics, compute details, APIs, code, and objective-evaluation status.The prompts require reading the full Markdown and returning null when fields are absent or uncertain.Source: §C.3 · p. 33
• A second prompt converts extracted cards into concise Markdown by preserving research goals and metric definitions, retaining baseline results, removing the original method, and adding a blank Your Method row.Paper Markdown is context-only for metric definitions and identifying method aliases, while output sections and ordering are fixed.Source: §C.3 · pp. 34–35

C.4 Task Packaging Guidelines

• Task packaging removes paper-specific methods while preserving generic baselines, resolving interleaved utilities, hyperparameters, and naming clues through two-author diff reconciliation.The authors note that separating paper-specific LoRa details from generic LoRa implementations is non-trivial and may bias agents toward the retained baseline.Source: §C.4 · p. 36
• Tasks provide complete dependencies, programmatic grading, modular wrappers, containerized environments, primary subtasks, and human-verified original methods for fair, reproducible comparison.Provisioned resources include datasets, splits, libraries, and API keys; grading supports selective datasets, logging, and result finalization, while primary subtasks reduce time-related penalties.Source: §C.4 · p. 36

D.1 Prompts

• The agent prompts require iterative hypothesis testing, code implementation, repeated grading, literature review, full-time effort, and completion of primary subtasks before secondary ones.Prompts explicitly require novel, sound, feasible hypotheses, multiple experiments, and an entire codebase capable of producing evaluation results.Source: §D.1 · pp. 37–38

Web Search Tool

• The evaluation stack adds apply_patch for code changes and async_jobs for tracking long-running experiments, addressing agents’ repeated difficulty managing parallel runs and stopping unattended jobs.Initial runs showed poor parallel experiment tracking, including training scripts that ran for hours without effective monitoring or termination.Source: Web Search Tool · p. 42

D.3 Agent Scaffoldings

• The baseline rg-agent uses an untuned ReAct-style loop, while AI-Scientist-v2 failed on most tasks and Claude Code and Codex showed better tool use but comparable research bottlenecks.The specialized systems mismatched repository formats requiring existing datasets or coordinated multi-file edits; Claude Code and Codex were run for $20 and 24 hours per task.Source: §D.3 · pp. 42–43

D.4 Tracking

• Tracking records token-level API usage, costs, wall-clock time, active time, and retry time, but models often failed to exhaust budgets because training runs consumed substantial time.The custom scaffold can stop runs when budgets are exceeded, whereas Claude Code and Codex expose costs only after complete turns, preventing streaming cost control.Source: §D.4 · p. 43, §D.4.1 · p. 43, §D.4.2 · p. 43

D.4.3 Messages

• Interaction logs provide compressed, readable, and full-fidelity formats, alongside OpenTelemetry tracing for live monitoring and post-hoc inspection.The .eval format supports monitoring frameworks such as Transluce’s Docent10 and helps detect unusual or cheating behavior in very large logs.Source: §D.4.3 · p. 43

D.4.4 States

• Run metadata, lifecycle states, execution plans, workspace edits, and raw subprocess logs preserve reproducibility and support debugging, including across resumed runs.Execution logs are appended rather than overwritten, retaining the full history of setup failures, dependency conflicts, and agent crashes.Source: §D.4.4 · p. 44, §D.4.5 · p. 44

D.5 Virtual Environments

• Preconfigured uv and Docker environments reduce library and CUDA compatibility burdens, allowing agents to focus on research rather than dependency repair.This is especially important for brittle reinforcement-learning stacks such as MuJoCo and DeepMind Control Suite, while Docker also isolates agents and restricts cheating.Source: §D.5 · p. 44

D.5.1 Cross-Platform Support

• Platform-aware scripts address operating-system differences, Windows path limits, encoding issues, shell behavior, workspace copying, and GPU-library availability.Installation scripts detect available GPU support and fall back to suitable alternatives when necessary.Source: §D.5.1 · p. 44

D.6 Context Window Summarization

• A handoff mechanism activates near 140K tokens to manage long research conversations, because recall degraded and hallucinations increased beyond 150K tokens despite GPT-5’s 256K context window.Handoffs at 120K tokens were too frequent and disrupted agent flow; after summarization, the conversation is cleared and prior summaries are reintroduced with a bridge prompt.Source: §D.6 · p. 45

D.7 Resuming Runs

• Resume support reconstructs interrupted sessions from persisted transcripts while pruning incomplete tool calls and preserving budget accounting across run boundaries.The system uses agent-specific strategies: transcript seeding for Claude Code and thread monitoring with extended retries for Codex when silent fresh-thread failures occur.Source: §D.7 · p. 45

D.8 URL Blocking

• Table 14 summarizes blocked URLs by category across tasks, with the links available in the released GitHub repository.The supplied passage identifies the table’s scope but does not provide category counts.Source: Table 14 · p. 46, §D.8 · p. 46

D.9 Inspection Agent

• The inspection agent was deployed post hoc to audit solver runs because LLM agents can exhibit documented reward hacking.The inspection experiment covered 46 production and incomplete runs across three agent scaffolds.Source: §D.9 · p. 46
• The tuned GPT-5 inspection agent achieved 100% accuracy on manually reviewed true positives, while some cheating remained unflagged.Table 15 reports inspection verdicts and resource usage across agent scaffolds; manual review found no false positives among Suspicious flags.Source: §D.9 · p. 46, Table 15 · p. 46
• Prompt tuning on six synthetic trajectories correctly flagged all three cheating scenarios and passed benign runs with high confidence.Cheating cases received SUSPICIOUS verdicts at 0.95–0.99 confidence, while benign cases received PASS at 0.90–0.92 confidence.Source: §D.9 · p. 46
• The inspection agent identified cross-run contamination, incompatible-configuration cherry-picking, and outright result fabrication in held-out problematic runs.These patterns correspond to copying pre-computed artifacts, selecting incompatible results, and fabricating outcomes.Source: §D.9 · p. 47

D.10 Tracing

• ResearchGym provides post-hoc transcripts, post-hoc transcript analysis, and live tracing for observing agent execution.These use inspect_ai, Transluce Docent, and Langfuse respectively, supporting conversation capture, search, clustering, replay, and real-time streaming.Source: §D.10 · p. 47
• Langfuse captures LLM requests, responses, token usage, latency, and errors through SDK module patching, including Anthropic models when supported.Traces are batched and flushed before process exit, enabling monitoring and cost/latency analysis during 12–24 hour runs.Source: §D.10 · p. 47

D.11 Budget

• End-to-end benchmark execution can exceed $300 in API credits and incur an additional estimated $360 in A100 costs.The estimate is workload-dependent, and compute beyond 12GB mainly shortens runtime without materially improving scores.Source: §D.11 · p. 47

E Quantitative Analysis

• Figures 5–9 present statistics for continual learning, cross-modal retrieval, materials tokenization, time-series explanation, and improving replay buffers.Source: Figure 5 · p. 48, Figure 6 · p. 48, Figure 7 · p. 49

Continual Learning

• Figure 5 reports statistics for the continual-learning task.Source: Figure 5 · p. 48

Cross Modal Retrieval

• Figure 6 reports statistics for the cross-modal-retrieval task.Source: Figure 6 · p. 48

Materials Tokenization

• Figure 7 reports statistics for the materials-tokenization task.Source: Figure 7 · p. 49

Time Series Explanation

• Figure 8 reports statistics for the time-series-explanation task.Source: Figure 8 · p. 49

Improving Replay Buffers

• Figure 9 reports statistics for the improving-replay-buffers task.Source: Figure 9 · p. 50

F ResearchGym Tasks

• For each ResearchGym task, the paper provides an agent-facing description, analyses of agent ideas and performance, and result tables, while identifying a primary sub-task as the main evaluation target.The descriptions include baseline result tables in markdown format, which are also presented as formatted tables.Source: F ResearchGym Tasks · p. 51

F.1 Continual Learning

• Acc=80.56 and AAA=86.49 on CIFAR-100 reached 0.93× the InfLoRA baseline, but average performance across three continual-learning runs was only 0.39× baseline.Run 001’s breakthrough followed numerical-stability fixes in LoRA merging; Runs 002 and 003 plateaued at 0.04× and 0.21× baseline.Source: §F.1 · p. 52
• Continual-learning agents proposed regularized LoRA variants, but none discovered SD-LoRA’s magnitude-direction decomposition with frozen past directions.Their approaches constrained entire LoRA updates rather than separating magnitude and direction, which was less effective at preventing interference.Source: §F.1 · pp. 52–53
• Hinting the continual-learning algorithm produced Acc=78.52 and AAA=79.30, or 0.89× and 0.86× baseline, but completed only 5 of 10 tasks before budget exhaustion.Implementation issues involving AMP stability, memory management, prototype initialization, and FP16 mismatches remained unresolved.Source: §F.1 · p. 53

F.1 Continual Learning · F.2 Materials Tokenization

• Async execution did not solve the core bottlenecks: the continual-learning async run suffered task degradation, while materials-tokenization async prioritized easier secondary work and never finished either primary NER task.The results indicate that ideation, implementation stability, and task prioritization mattered more than parallel execution.Source: §F.1 · p. 53, §F.2 · p. 57

F.2 Materials Tokenization

• The materials-tokenization agents used regexes, lexicons, or protected spans, whereas MatScore reweights WordPiece merge priorities using a neural material-concept detector trained on 80K curated concepts.These simpler approximations did not capture the semantic richness of the intended learned reweighting mechanism.Source: §F.2 · pp. 55–56
• Materials-tokenization runs averaged only 27.8% task completion, and no run completed both primary sub-tasks.The best MatScholar result was Micro-F1=83.25 versus the WordPiece baseline of 86.1, while SOFC Micro-F1=82.31 exceeded baseline 80.9 but Macro-F1=80.29 remained below baseline 83.0.Source: §F.2 · p. 56
• Hinting MatScore led to SOFC Micro-F1=75.7 and MatScholar Micro-F1=67.5, below baselines of 81.4 and 86.1, despite completing both primary NER tasks.Token augmentation alone could not replace learned reweighting, while compatibility debugging consumed substantial budget.Source: §F.2 · p. 56

F.3 Cross Modal Retrieval

• Query shift causes substantial degradation because real-world queries can come from scarce or personalized domains unlike the source distribution, motivating online adaptation for cross-modal retrieval.The task evaluates query shift and query-gallery shift across corrupted COCO, Flickr, e-commerce, person Re-ID, and captioning settings using T2IR@1 and I2TR@1.Source: §F.3 · p. 58
• 1.03× baseline was the average Base2Flickr improvement across three cross-modal runs by hour 3, whereas ReID performance remained inconsistent or unavailable.Run 001 achieved 1.27× baseline on CUHK2ICFG, Run 002 achieved 0.43× on CUHK2ICFG, and Run 003 produced no valid ReID metrics after using the wrong retrieval direction.Source: §F.3 · p. 59
• 221 minutes of setup preceded Run 003’s cross-modal evaluation, which also used the wrong retrieval flag and domain-specific IRRA models, invalidating ReID results and potentially inflating comparisons with general-model baselines.All three cross-modal runs consumed nearly the full $10 budget, with token usage ranging from 22.8M to 39.9M before Run 001’s additional 40.1M tokens.Source: §F.3 · p. 59
• The cross-modal agents used entropy minimization, pseudo-labeling, and CORAL alignment but lacked reliability-based pair selection and therefore worked better for moderate natural-image shifts than severe cross-dataset ReID shifts.The underlying method selects source-like query-candidate pairs using uniformity and gap scores and optimizes uniformity, gap rectification, and noise-robust adaptation while updating normalization layers online.Source: §F.3 · pp. 58–59

F.4 Time Series Explanation

• Time-series explanations target directional feature effects because magnitude-only attribution and cancellation-prone metrics can misidentify important points, especially in safety-critical applications.TIMING uses segment-masked partially retained baselines and aggregates attributions over random contiguous temporal segments to reduce out-of-distribution paths while preserving temporal structure.Source: §F.4 · pp. 63–64
• Run 002 rose from 0.231 at hour 5 to 0.589±0.036 at hour 9, exceeding the IG baseline (0.448) and SOTA (0.463), but task completion was only 14.3%.It used directional margin IG with per-baseline recomputation, NoiseTunnel smoothing, and positive clamping, with progression dropping to 0.054 at hour 7 before recovering.Source: §F.4 · p. 64
• 0.40× baseline was Run 003’s final PAM Average CPD, while Run 001 completed all five folds and reached 0.357±0.105, showing large variability across time-series implementations.Run 003 used a CNN classifier with margin-based directional IG but completed 0% of tasks; Run 001 achieved the highest completion rate at 71.4%.Source: §F.4 · p. 64
• Five-fold evaluation across seven time-series datasets was repeatedly undermined by dataset preparation and fold-management failures, including an async run that produced no valid results after import and path errors.A hint-guided run implemented segment-masked IG and produced PAM Average CPD=0.311±0.023 and PAM Zeros CPD=0.524±0.044, but grading rejected its CSV output because the expected directory structure was missing.Source: §F.4 · pp. 64–65

F.4.1 Success Analysis: SOTA-Surpassing Run

• 0.589±0.036 PAM Average CPD and 0.525±0.025 PAM Zeros CPD let Run 002 surpass TIMING’s 0.463±0.007 and 0.602±0.033 respectively on the primary metric.This was the only ResearchGym task where an agent surpassed the withheld reference solution, TIMING (Jang et al., 2025).Source: §F.4.1 · p. 67
• The successful explanation method targeted the predicted-versus-second-best logit margin, then applied SmoothGrad-squared IG, temporal pooling, and positive clamping.The margin was defined as f_pred(x) − f_alt(x); the noise tunnel used 8 samples with σ = 0.02, and pooling used kernel size 5.Source: §F.4.1 · p. 67
• The agent’s approach complemented TIMING by changing the explained quantity rather than the IG path, using post-hoc smoothing and hard positive clamping.Both approaches addressed directionality for CPD, but the agent’s method was simpler to implement and potentially less theoretically grounded.Source: §F.4.1 · p. 67
• Run 002 made 13 evaluation attempts over 9.5 hours, used CPD feedback to guide iterations, trained state and CNN models, and addressed Average-versus-Zeros sensitivity.These factors distinguished it from failed runs that changed too many variables or abandoned promising directions prematurely.Source: §F.4.1 · p. 67, Table 24 · p. 68
• The margin-based method paralleled emerging contrastive-attribution research, while its specific combination of margin IG, SmoothGrad, temporal pooling, and positive clamping did not appear in prior work.The agent developed it without web searches or access to the relevant literature, constituting a case of convergent discovery.Source: §F.4.1 · p. 68

F.5 Improving Replay Buffers

• The withheld replay-buffer method modeled p(τ|c) with conditional diffusion and classifier-free guidance, mixing synthetic and real transitions based on return, TD-error, or curiosity relevance.Pixel-based generation occurred in the policy encoder’s latent space, with separate 1M-transition buffers and condition dropout probability 0.25.Source: §F.5 · p. 69
• Returns were 195.93 (0.058× baseline), 228.83 (0.067× baseline, marginally above Run 001), and 71.48±70.93 (0.021× baseline), with the last produced entirely by standard SAC without synthetic augmentation.Technical failures included tensor-stride errors, a missing dm_control.suite.wrappers.pixels attribute, numerical instability during diffusion sampling, and overconfident reporting.Source: §F.5 · p. 70
• 4101.79 return (1.21× baseline) was achieved by the paper’s conditional diffusion method, versus 2915.27 (0.859× baseline) from the strongest agent run.The agent’s best result used ensemble Q-disagreement prioritization and diversity-aware densification, while the paper synthesized new transitions rather than reweighting or interpolating existing ones.Source: §F.5 · p. 70