Source-linked AI summary
FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights
Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Zhiting Hu, Eric P. Xing
TL;DR
Evaluating whether LLM agents can perform verifiable scientific discovery remains difficult because existing benchmarks rely on subjective judging or narrow metrics. FIRE-Bench addresses this gap by giving agents only high-level questions from verified studies and scoring their independently executed rediscovery process. Results show that full-cycle research remains challenging, with low and variable performance and recurring planning and conclusion failures.
Problem
Existing evaluations provide limited evidence of verifiable full-cycle scientific discovery because they rely on LLM judging of research outputs or isolated performance metrics.
Method
FIRE-Bench evaluates agents by having them rediscover established empirical findings from high-level research questions while withholding original designs, implementations, and conclusions.
Results
Full-cycle scientific research remains challenging: performance is limited and variable across runs, with failures dominated by Research Planning and Conclusion Formation.
Takeaways & Limitations
FIRE-Bench provides a rigorous, diagnostic framework for measuring progress toward reliable agent-driven scientific discovery.
Takeaways & Limitations
Contamination analysis is coarse because knowledge cutoff dates are approximate, training data composition is unknown, and per-category sample sizes are limited.
Abstract
from arXiv · showhide
Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge. Existing benchmarks face a trade-off: they either heavily rely on LLM-as-judge evaluations of automatically generated research outputs or optimize convenient yet isolated performance metrics that provide coarse proxies for scientific insight. To address this gap, we introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings from recent, high-impact machine learning research. Agents are given only a high-level research question extracted from a published, verified study and must autonomously explore ideas, design experiments, implement code, execute their plans, and derive conclusions supported by empirical evidence. We evaluate a range of state-of-the-art agents with frontier LLMs backbones like gpt-5 on FIRE-Bench. Our results show that full-cycle scientific research remains challenging for current agent systems: even the strongest agents achieve limited rediscovery success (<50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning. FIRE-Bench provides a rigorous and diagnostic framework for measuring progress toward reliable agent-driven scientific discovery.
1. Introduction
FIRE-BENCH addresses limitations in evaluating full-cycle scientific agents by testing whether they can rediscover verified empirical findings from high-level questions. It combines constrained, insight-driven evaluation with claim-level scoring and diagnostic analysis of research failures.
- Current benchmarks either rely on subjective LLM judging of generated papers or use isolated metrics that provide limited insight into scientific reasoning.Full-cycle paper evaluation is difficult to validate at scale, while replication-focused benchmarks offer limited visibility into broader reasoning processes.
- FIRE-BENCH evaluates whether agents can complete empirical research and reach verifiable conclusions by rediscovering established findings from recent machine learning studies.Tasks use peer-reviewed, documented, and computationally verifiable findings rather than novel, unverified claims.
- Agents receive only a high-level research question while original experimental designs, implementations, and analytical pathways remain withheld.This constrained yet open-ended setup requires independent planning, experimentation, and evidence analysis rather than direct reproduction.
- FIRE-BENCH evaluates agent conclusions against human-authored findings using claim-level precision, recall, and F1 scores.The benchmark also supports structured analysis of errors across the research workflow.
- FIRE-BENCH introduces constrained rediscovery as an evaluation paradigm combining full-cycle execution, insight-driven evaluation, grounded reference scoring, and methodological exploration.The paradigm occupies a previously empty point in the benchmark design space.
- The benchmark adds a research-problem tree abstraction and a diagnostic framework that identifies failures across research planning, implementation, execution, and conclusion formation.The reported contribution emphasizes recurring dominance of Research Planning and Conclusion Formation failures.
2. Related Work
Related benchmarks cover isolated research stages, metric optimization, automated paper generation, or methodological replication. FIRE-BENCH instead targets end-to-end insight rediscovery from high-level questions while grounding evaluation in established findings.
- Research-agent benchmarks commonly evaluate individual stages such as literature synthesis, idea generation, coding, or data analysis in isolation.This fragmented coverage does not directly assess the complete research workflow.
- Metric-driven benchmarks measure optimization and implementation skill on competitive tasks, but a single performance metric provides limited insight into scientific reasoning.These evaluations emphasize engineering-oriented or leaderboard-driven discovery.
- Automated paper-generation benchmarks evaluate broad research outputs but often rely on LLM-based evaluators because large-scale human review is costly.
- Reproducibility benchmarks use existing publications as ground truth and typically provide the original methodology and expected outcomes for experiment replication.FIRE-BENCH differs by withholding those specifics and providing only a high-level research question.
3. FIRE-BENCH: From Papers to Verifiable Discovery Tasks
FIRE-BENCH converts empirical papers into open-ended but verifiable discovery tasks through research-problem trees, curated source selection, and claim-level comparison against grounded findings.
- Benchmark Construction: The construction pipeline decomposes empirical analysis papers into verifiable discovery tasks that permit exploration while remaining grounded in concrete evidence.
- Benchmark Construction: A research-problem tree represents a paper’s reasoning trajectory from an overarching question through intermediate subproblems to fully specified experimental leaves.Leaf nodes encode datasets, methods or models, evaluation criteria, and reported results.
- Benchmark Construction: An automated fixed-prompt LLM extractor produces structured trees encoding node types, hierarchy, and associated experimental data, with human evaluation used to assess extraction quality.
- Benchmark Construction: Each task uses a parent-node research question while inheriting experimental scope and evaluation criteria from a target leaf, without revealing the original implementation or conclusion.This relaxes methodological specification while preserving a verifiable empirical target.
- Benchmark Construction: The parent-node formulation is harder than leaf-level replication because it requires experimental design rather than implementation of a fully specified experiment.
- Source Paper Selection and Filtering: The benchmark selects 30 empirical LLM-analysis papers through keyword search, LLM classification, and manual review against openness, compute-light execution, and verifiable-insight criteria.The selection process is designed to support feasibility, reproducibility, and evaluative rigor.
- Source Paper Selection and Filtering: FIRE-BENCH also releases 10 cross-domain papers and a parsed pool of 60 additional papers spanning areas including computer vision, code generation, retrieval-augmented generation, and safety.
- Evaluation Protocol: Evaluation decomposes agent and ground-truth conclusions into atomic claims and compares them claim by claim to measure rediscovery of key empirical insights.The same automated extraction procedure is applied to both agent and ground-truth texts.
4. Experiments
FIRE-BENCH evaluates three state-of-the-art coding agents with different LLM backbones in sandboxed CLI environments, using default optimized settings. Claude Code achieves the highest average performance, with substantial variance across runs.
- Agent frameworks & LLMs: Three coding agents with different LLM backbones are evaluated on FIRE-BENCH: OpenHands, Codex, and Claude Code.OpenHands is tested with gpt-o4-mini and gpt-5.
- Experimental details: Experiments primarily conducted in August 2025 use default checkpoints for proprietary agents, which may change over time.The defaults are adopted to reflect optimized settings.
- Overall performance: Claude Code achieves the highest average FIRE-BENCH performance with an F1 score of 46.7 while exhibiting substantial variance across runs.The reported variance highlights sensitivity to execution trajectories.
- Experimental details: Agents run through their command-line interfaces in sandbox environments on a GPU node with eight 80GB A100 GPUs.Each working directory includes task instructions and provided datasets, without preconfigured additional environments.
5. Results & Analyses
FIRE-Bench results show that current agents achieve limited and unstable scientific rediscovery, with performance depending strongly on task structure and experimental-design demands. Errors cluster around planning, control-based reasoning, and evidence-to-conclusion formation, while stronger models improve performance at increased cost.
- Claude Code achieves the highest average F1 at 46.7, followed by Codex at 41.9, OpenHands (gpt-5) at 37.9, and OpenHands (o4-mini) at 31.9.
- Performance varies with task structure: Agents perform better on procedurally direct tasks, including Lost in the Middle, Persona with Catch, and CoT Without Prompting.Best observed F1 scores are 91.7, 88.6, and 82.6, respectively.
- Performance varies with task structure: Performance drops on controlled-comparison tasks because agents often fail to construct baselines that isolate causal factors.In LLM Racial Bias in Medicine, agents introduced race information without establishing the required counterfactual control.
- Frontier models lead overall, but substantial gaps remain: Claude Code leads on 13 of 30 tasks, but no single agent dominates and stronger backbones do not eliminate failures in non-trivial experimental design.Upgrading OpenHands from o4-mini to gpt-5 improves average F1 by 6.1 points, from 31.9 to 37.9.
- Fine-grained error analysis: Research Planning and Conclusion Formation account for most errors, with agents showing broadly similar qualitative error distributions.Planning failures include method and goal deviations, while conclusion failures include unsupported or overgeneralized claims.
- Fine-grained error analysis: Contradictory or Unrelated claims comprise 76.4% to 95.0% of false positives, whereas Alternative conclusions account for only 4.5% to 10.9%.The authors report that deviations are dominated by reasoning or relevance failures rather than independently valid scientific insights.
B. Benchmark Scale Comparison
FIRE-BENCH includes 40 fully executed tasks and 60 additional papers released for community evaluation, placing its executed core among the largest end-to-end research-agent benchmarks.
- 40 fully executed tasks form FIRE-BENCH’s evaluated core, with 60 additional papers released for community evaluation.The additional papers include extracted research-problem trees spanning multiple machine learning subfields.
- FIRE-BENCH’s 30-task core plus 10-task cross-domain extension places it among the largest benchmarks in this category.
D. Cross-Domain Extension
FIRE-BENCH extends beyond LLM-behavior research with 10 fully executed papers spanning computer vision, vision-language modeling, and neural network analysis.
- 10 fully executed cross-domain papers comprise five computer vision and vision-language modeling tasks and five neural network analysis tasks.
- The cross-domain extension’s source papers are listed in Table 10, while performance results are reported in Table 11.
- Table 11 reports claim-level F1 means and standard deviations across three runs, omitting Claude Code because of budget constraints.
E. Extended Contamination Analysis
Across all 40 executed tasks, contamination analysis finds no consistent pre-cutoff performance advantage after controlling for task difficulty.
- No consistent advantage emerges for pre-cutoff tasks once difficulty is controlled across all 40 executed tasks.
- The extended analysis covers the core 30 LLM-behavior tasks and the 10 cross-domain tasks.
- Per-agent cost estimates supplement the analysis, with Codex costs inferred from token counts under an assumed 3:1 input-to-output ratio.Relative cost trends remain stable under other reasonable input-to-output ratios.
G. Full Experiment Results
FIRE-BENCH reports three-trial F1 means with standard deviations and rates task difficulty using conceptual decomposition, confound burden, and measurement complexity.
- F1 scores are averaged over three independent trials, with standard deviations indicating variance across runs.
- Task difficulty combines conceptual decomposition, confound and causality burden, and measurement and analysis complexity.
- The difficulty index sums the three axis scores into S = D + C + M, mapping totals 3–4 to Easy, 5–6 to Medium, and 7–9 to Hard.
I. Problem-Tree Parsing Evaluation
Human validation assessed automatically generated research-problem trees across groundedness, completeness, hallucination control, coherence, and question–conclusion alignment. The results indicate consistently high quality and reliability for the trees used in FIRE-Bench.
- Evaluation protocol: Human annotators scored five problem-tree criteria on a 1–5 scale across five sampled benchmark papers.The criteria were research-question groundedness, experiment completeness, hallucination elimination, structural coherence, and question–conclusion alignment.
- Evaluation criteria: The evaluation covered whether extracted questions reflected stated objectives and whether all key experiments were represented.
- Evaluation criteria: The evaluation also tested whether trees avoided fabricated claims, preserved logical parent–child structure, and aligned leaf conclusions with their questions.
- Results: Table 16 reports consistently high scores across all five criteria, supporting the quality and reliability of the problem trees used in FIRE-Bench.
- Additional validation: A separate evaluation assessed automated claim extraction by comparing RAGChecker’s atomic claims with human annotations using precision, recall, and F1.
J.1. Evaluation Examples
The examples show that rediscovered findings vary by task and model: document position favors early context, self-correction helps some tasks but harms others, and fairness conclusions depend on experimental controls.
- Lost in the Middle: 0.534 EM for early positions exceeded 0.427 for middle and 0.451 for late positions, with early–middle and early–late differences significant.The late–middle comparison was not significant, indicating a mild end-position rebound rather than uniform decline.
- Lost in the Middle: Placing relevant passages first was recommended for maximizing exact-match accuracy in retrieval-augmented generation or multi-document question answering.
- LLM Racial Bias in Medicine: The race-sensitivity experiment achieved 182/500 successfully parsed responses (36.4%), limiting the valid sample available for paired comparisons.
- LLM Racial Bias in Medicine: GPT-3.5 produced no statistically significant race-label differences in predicted costs or length of stay when clinical content was otherwise identical.The experiment reported frequent identical outputs across race conditions, with approximately 80.5% exact ties.
- Self-Correction: Two-round self-correction strongly improved GSM8K for gpt-4o, modestly helped retrieval-free HotpotQA, and was neutral or harmful for CommonSenseQA.For gpt-3.5-turbo, effects were neutral or negative apart from small, inconsistent HotpotQA improvements.
- Cross-example synthesis: Self-correction recommendations were task- and model-dependent, while document-position effects showed a specific early-context advantage and middle-position deficit.
K. Error Analysis Examples
The error examples illustrate how agents can miss established findings by misreading trends, omitting controls, or failing to complete the required research workflow and evidence analysis.
- Lost in the Middle: For relevant information position, the ground truth favors beginning or end positions, whereas performance significantly degrades when information appears in the middle.
- Lost in the Middle: One agent committed an analysis failure by missing the slight accuracy recovery at the end of the context.
- Paper Parsing Prompt: Problem trees decompose papers from broad research questions through intermediate objectives to concrete experimental leaves linked to figures, tables, or result sections.
- Paper Parsing Prompt: The extraction rules require recursive decomposition, preservation of intermediate layers, faithful technical wording, and uncertainty marking when information is missing.
- Agent task setup: The benchmark asks agents to answer high-level research questions using available models, datasets, exact-match evaluation, and a fixed computational budget.
- Error taxonomy: The failure taxonomy distinguishes research-planning, implementation, execution, analysis, and system errors, including method deviation, laziness, loops, and unsupported conclusions.
M. Error Type Definition
Table 17 presents the taxonomy used to categorize agent failure modes during false-negative analysis.
- Error Type Definition: Table 17 lists the taxonomy of agent failure modes for false-negative analysis.