PaperlayerView exact revision on arXiv

Source-linked AI summary

HeurekaBench: A Benchmarking Framework for AI Co-scientist

Siba Smarak Panigrahi, Jovana Videnović, Maria Brbić

arXiv:2601.01678v2cs.LG

TL;DR

Scientific agents need realistic benchmarks for open-ended, end-to-end research rather than isolated computational or retrieval tasks. HeurekaBench builds such benchmarks from validated scientific workflows and instantiates them in single-cell biology, where it also evaluates agent design choices and critic-based improvements.

  • Problem

    Existing benchmarks insufficiently evaluate open-ended scientific discovery grounded in experimental data and multi-step workflows.

  • Method

    HeurekaBench uses a multi-LLM pipeline to extract insights from publications, match them to code workflows, verify reproduced findings, and generate open-ended and multiple-choice questions.

  • Results

    In sc-HEUREKABENCH, the framework compares single-cell agents and finds that critic modules improve ill-formed analyses for open-source agents while narrowing their gap with closed-source models.

  • Takeaways & Limitations

    HeurekaBench provides a workflow-grounded path for evaluating and improving AI co-scientists on open-ended scientific problems.

  • Takeaways & Limitations

    Evaluation relies solely on the final agent response rather than verifying intermediate workflow steps or assigning partial credit.

Abstract

from arXiv · show

LLM-based reasoning models have enabled the development of agentic systems that act as co-scientists, assisting in multi-step scientific analysis. However, evaluating these systems is challenging, as it requires realistic, end-to-end research scenarios that integrate data analysis, interpretation, and the generation of new insights from the experimental data. To address this limitation, we introduce HeurekaBench, a framework to create benchmarks with exploratory, open-ended research questions for experimental datasets. Each such question is grounded in a scientific study and its corresponding code repository, and is created using a semi-automated pipeline that leverages multiple LLMs to extract insights and generate candidate workflows, which are then verified against reported findings. We instantiate the framework in single-cell biology to obtain sc-HeurekaBench benchmark and use it to compare state-of-the-art single-cell agents. We further showcase the benefits of our benchmark for quantitatively analyzing current design choices in agentic systems. We find that the addition of a critic module can improve ill-formed responses for open-source LLM-based agents by up to 22% and close the gap with their closed-source counterparts. Overall, HeurekaBench sets a path toward rigorous, end-to-end evaluation of scientific agents, grounding benchmark construction in real scientific workflows.

1 INTRODUCTION

AI co-scientists promise to support scientific discovery, but existing benchmarks do not adequately test open-ended, data-driven research. HeurekaBench addresses this gap by grounding benchmark construction in validated scientific workflows and using it to analyze single-cell agents.

  • Motivation: Existing benchmarks emphasize factual retrieval, instruction following, or narrow computational tasks rather than exploratory scientific discovery.They often ask agents to answer explicit single-step questions instead of autonomously planning workflow sub-steps.
  • HEUREKABENCH: HeurekaBench constructs open-ended, data-driven questions from scientific insights and code workflows, then evaluates agents’ multi-step analyses against published findings.Its pipeline extracts and validates insights, reformulates them as questions, and assesses data-driven answers.
  • Single-cell instantiation: sc-HEUREKABENCH contains 50 open-ended and 50 multiple-choice questions derived from 41 validated insights in single-cell biology.The benchmark evaluates agents on findings from experimental single-cell RNA-seq datasets.
  • Agent evaluation: Closed-source planners outperform open-source counterparts, but multi-turn reasoning narrows the gap.The benchmark also examines planner, critic, and retriever components in current single-cell agents.
  • Agent evaluation: A critic at the end of the agent loop detects and revises ill-formed responses and analyses.This finding motivates critic-based improvements for open-source agents.

2 RELATED WORK

Prior scientific-agent benchmarks cover reproducibility, general scientific reasoning, or domain-specific subtasks, while single-cell agents increasingly support specialized analyses. These efforts leave a need for end-to-end co-scientist evaluation.

  • Scientific agents: Scientific-discovery agents have been proposed across domains for tasks including drug repurposing and knowledge-graph exploration.These systems illustrate the expanding scope of agentic scientific workflows.
  • Single-cell agents: Single-cell agents span gene-perturbation design, code generation, and evaluation of predefined tasks.Examples include BioDiscoveryAgent and CellAgent, which operate at different levels of specificity.
  • Benchmarking scientific agents: Existing benchmarks assess isolated capabilities such as code reproducibility, scientific reasoning, factual knowledge, or tool use.They do not uniformly evaluate complete exploratory workflows grounded in experimental data.

3 HEUREKABENCH FRAMEWORK

HEUREKABENCH unifies data analysis, tool use, interpretation, and multi-step reasoning in benchmarks grounded in published studies, datasets, and code. Its pipeline validates insights, generates question formats, and evaluates agent responses with task-appropriate metrics.

  • Framework overview: HEUREKABENCH evaluates agents on research questions requiring dataset analysis, insight derivation, workflow selection, and evidence-based reasoning.These questions target findings that cannot be obtained from general knowledge alone.
  • Co-scientist task overview: The benchmark task is represented as (D, Q, A): an experimental dataset, an open-ended research question, and its ground-truth answer.This triplet captures the data, workflow-level question, and reference needed to judge agent responses.
  • Insight generation: The creation pipeline extracts candidate insights, matches them to code workflows, and retains only insights whose reproduced results match reported findings.Human reviewers may make minor code adjustments or add supplementary files during validation.
  • Question generation: Each validated insight yields open-ended and multiple-choice questions, with open-ended questions serving as the primary format for free-form scientific reasoning.Multiple-choice questions provide a lighter evaluation format with challenging distractors.
  • Evaluation: Open-ended answers are scored by GPT-4o using G-Eval ratings from 1 to 5, while multiple-choice evaluation reports accuracy plus precision and recall.The open-ended judge decomposes answers to evaluate scientific content beyond surface-level matching.
  • sc-HEUREKABENCH: The single-cell instantiation uses recent studies, open datasets, and code repositories to produce 50 open-ended and 50 multiple-choice questions.A separate tool-usage benchmark contains 12 open-ended questions involving domain-specific tools.

4 EXPERIMENTS

Experiments evaluate HeurekaBench’s insight-construction pipeline, single-cell agents, and agent design choices involving planners, critics, and retrievers. Results show strong code-file retrieval, performance differences among agents and planners, substantial judge–human alignment, and benefits from carefully placed critics.

  • 4.1 EVALUATION OF INSIGHT CONSTRUCTION: 158 of 215 scripts were correctly matched, with 74.6% of files retrieved correctly across insights.The CodeDescriber and CodeMatcher modules selected relevant files for code generation despite the challenging repository setting.
  • 4.2 EVALUATION OF SINGLE-CELL BIOLOGY AGENTS: BixBench-Agent and Biomni outperform CellVoyager on both benchmark formats.The authors associate this result with more flexible agent loops, while noting CellVoyager’s restrictive code-fixing and feedback-handling capabilities.
  • 4.3.1 PLANNER ABLATIONS: Claude-4-Sonnet achieves the highest overall performance, scoring 2.58 versus 2.08 for the second-best model on OEQs.Performance also improves with Qwen model size, and the thinking variant provides additional OEQ correctness gains.
  • 4.3.1 PLANNER ABLATIONS: Human and GPT-4o scores differ by at most one point for 92% and 96% of questions under mode and median aggregation.Expert–judge Spearman correlations are 0.93 and 0.90, with κ=0.85 for both aggregation methods.
  • 4.3.2 CRITIC ABLATIONS: End-critics raise GPT-OSS-120B’s average correctness to 2.49, whereas Plan-critics reduce scores by 0.13 and 0.19 points for the two open models.End-critic gains are strongest on low-scoring questions, while critic placement and underlying LLM determine the benefit or harm.

5 CONCLUSION AND FUTURE WORK

HEUREKABENCH evaluates AI co-scientists through open-ended, data-driven questions and reports training-free improvements, including critic modules and agentic-task-optimized LLMs. Its current evaluation scores only final responses, while future work should validate intermediate workflows and partial credit.

  • Table 4 reports Biomni retriever-ablation correctness scores on sc-HEUREKABENCH-TU, averaged across three independent runs, with higher scores indicating better performance.
  • HEUREKABENCH evaluates AI co-scientists with open-ended research questions grounded in experimental datasets, publications, and codebases, using an LLM-as-a-judge paradigm.
  • Critic modules enabled an open-source model to achieve performance comparable to a closed-source model, while agentic-task-optimized LLMs consistently yielded better results.
  • The current evaluation relies solely on final agent responses, limiting verification of intermediate workflow steps and assignment of partial credit.

ETHICS STATEMENT

The benchmark evaluates autonomous LLM-based agents that generate code for scientific analysis, so users should prioritize safe execution environments. Its analysis workflow emphasizes rigorous, dataset-grounded, non-fabricated insights for biomedical researchers.

  • Because agents can generate malicious code affecting a user’s system, benchmark evaluations should run in sandboxed environments with necessary safety measures.
  • The insight-generation instructions require conclusions grounded in authors’ analyses and interpretations rather than general background or established biological facts.
  • The workflow instructs agents not to fabricate or simulate insights that are not explicitly present in the paper.
  • The intended audience is biomedical researchers, requiring rigor and accuracy in generated insights.

A.3 ADAPTATION OF HEUREKABENCH FOR OTHER SCIENTIFIC DOMAINS

HEUREKABENCH can be adapted to other scientific domains with largely unchanged stages and minor prompt edits. Human domain experts remain necessary for validating automated outputs, although future automation could remove this semi-automated step.

  • HEUREKABENCH retains its two stages—insight generation and question generation—when adapted to another scientific domain.
  • Adapting the framework requires changing domain names and few-shot examples in prompts, while its LLM-based modules need only minor edits.
  • Human experts with sufficient domain experience are required to understand and analyze automated-module outputs during adaptation.
  • The framework’s minimal changes enable straightforward adaptation to new scientific domains.
  • Question generation still requires additional filtering because LLM-generated questions may need robustness checks.

B DETAILS ON EVALUATION IN HEUREKABENCH FRAMEWORK

HEUREKABENCH evaluates free-form scientific responses against ground-truth conclusions by decomposing them into atomic facts. Scores depend on dataset-grounded coverage and contradictions, using a five-level correctness rubric.

  • The adapted G-Eval evaluates agent responses against ground-truth conclusions while penalizing reliance on pretrained knowledge instead of dataset exploration.
  • The grader decomposes responses into atomic facts and labels each fact PRESENT, PARTIAL, MISSING, or INCORRECT before assigning an overall correctness score.
  • A fact is PRESENT only when its meaning is correct and explicitly tied to dataset-derived quantitative or statistical outputs or cluster/subtype identifiers.
  • Contradictions are judged against the ground truth, whereas omissions and partial coverage affect coverage without counting as contradictions.
  • Correctness is scored on an integer 1–5 scale based on the coverage labels assigned to ground-truth facts.

C DETAILS ON SC-HEUREKABENCH BENCHMARK

sc-HEUREKABENCH is built from validated publications, experimental datasets, and code workflows to generate diverse open-ended and multiple-choice questions across single-cell analysis categories.

  • C.1 SELECTED PUBLICATIONS FOR INSIGHT GENERATION: 50 OEQs and 50 MCQs were created from 13 publications for sc-HEUREKABENCH.
  • C.3 QUESTION CATEGORIES: Questions cover heterogeneity, condition or treatment, and other analyses, with categories sometimes requiring multiple analyses for accurate answers.
  • C.3 QUESTION CATEGORIES: Task-category distributions are similar between sc-HEUREKABENCH and sc-HEUREKABENCH-Lite.
  • C.4 INVALIDATED INSIGHTS: Validation includes minor edits aligning gene identifiers, variables, and metadata names with the experimental dataset.
  • C.4 INVALIDATED INSIGHTS: Manual workflow validation invalidates insights when dataset information is insufficient, workflow requirements mismatch available data, or insights are too generic.

C.5 COMPARATIVE ANALYSIS WITH BAISBENCH

The comparison with BaisBench tests whether questions require data-driven reasoning rather than knowledge recall, using differences in question grounding and answerability.

  • C.5 COMPARATIVE ANALYSIS WITH BAISBENCH: BaisBench generates unvalidated multiple-choice questions with a single LLM, whereas HEUREKABENCH grounds questions in publications, code repositories, and experimental data.
  • C.5 COMPARATIVE ANALYSIS WITH BAISBENCH: GPT-5 answered 53.37% of BaisBench questions correctly versus 34.69% of sc-HEUREKABENCH questions without experimental-data access.

D DETAILS ON EXPERIMENTS

The experiments validate expert-insight extraction by comparing LLM-generated insights with scientist-derived findings using G-Eval and a defined relatedness scale.

  • D DETAILS ON EXPERIMENTS: InsightExtractor validation uses G-Eval to classify the relatedness of generated insights to scientist-derived findings.
  • D DETAILS ON EXPERIMENTS: The validation score ranges from 1 to 3, denoting unrelated, weakly related, or strongly related insights.
  • D DETAILS ON EXPERIMENTS: A score of 3 requires at least one LLM-generated insight to be strongly related to the scientist-derived insight.
  • D DETAILS ON EXPERIMENTS: The evaluation prompt requires a numerical rating and explanation enclosed in specified tags without extra text.

D.3 MANUAL ANALYSIS OF AI CO-SCIENTIST ANSWERS

Manual analysis identifies recurring agent failures and shows that closed-source models outperform open-source models across task categories, with correctness varying by category.

  • D.3 MANUAL ANALYSIS OF AI CO-SCIENTIST ANSWERS: Agents may recall pathway markers instead of using gene-set enrichment tools and may fail to inspect metadata columns needed for analysis.
  • D.3 MANUAL ANALYSIS OF AI CO-SCIENTIST ANSWERS: Other failures include limited environment exploration, incomplete analysis, large non-stepwise code blocks, and failure to incorporate execution errors.
  • D.3 MANUAL ANALYSIS OF AI CO-SCIENTIST ANSWERS: Open-source agents show more failure behaviours than closed-source variants, suggesting post-training could improve their adoption for scientific discovery.
  • D.3 MANUAL ANALYSIS OF AI CO-SCIENTIST ANSWERS: Claude-4-Sonnet outperforms open-source models across categories, reaching up to approximately twice their performance.Claude-4-Sonnet scores 1.90 in key gene analysis and 2.16 in cellular functioning analysis, while Qwen3-235B-THINKING reaches 1.53 in key gene analysis.

E.1 OPEN-ENDED QUESTIONS (OEQS)

The section illustrates sc-HEUREKABENCH’s open-ended and multiple-choice questions, expected answer formats, and benchmark evaluation. It also reports that agentic systems substantially outperform a same-model baseline on both question types.

  • OEQ examples: Benchmark examples span expression changes, cell-state proportions, pathway activity, and stage-specific ligand-receptor interactions.Reported examples include aging muscle, post-chemotherapy neuroblastoma, fibroblast-like stromal cells, and developing limbs.
  • Benchmark variants: The benchmark includes OEQs and MCQs, with sc-HEUREKABENCH-TU containing OEQs only.The TU examples cover age-related pathway changes and developmental ligand-receptor interactions.
  • OEQ task design: OEQs ask agents to analyze single-cell datasets and provide fact-based summaries addressing all key findings relevant to the question.The required output is wrapped in <solution> tags without additional formatting.
  • MCQ task design: MCQs require selecting all correct options from answer choices and returning the selected letters in a comma-separated XML-style format.One example identifies proliferating macrophages as the observed subset reducing after therapy.
  • Benchmark evaluation: Agentic Claude-4-Sonnet substantially improves performance over the same-model baseline on both OEQs and MCQs.Table 7 reports accuracy, recall, and precision, with higher values indicating better performance.
Loading 2601.01678v2…