How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices

1 Introduction

• Benchmark scores require understanding the task, available evidence, and scoring procedure to interpret them.Different research outputs, from computations to literature comparisons and criticisms, require different observations.Source: §1 · p. 1
• The review brings together benchmarks, datasets, execution environments, human studies, metrics, and audit protocols, comparing designs as well as reported scores.The introduction situates this evaluation-focused review alongside surveys organized around scientific method, workflows, autonomy, applications, and system architectures.Source: §1 · p. 1
• Across six evaluation targets, Figure 1 maps representative works by target and by the claims their core designs support.The introduction distinguishes questions such as a review’s usefulness to its author from its ability to detect a known error.Source: §1 · p. 1
• The survey explains tasks and assessment procedures, connects evidence, scoring, and calibration to assessed capabilities and support gaps, and offers setting-specific recommendations.The appendix records protocols and reporting details.Source: §1 · p. 1

2 Scope and Review Method

• The review covers automated research workflows, artifacts, and human interaction assessed through explicit tasks or inspectable scoring procedures, while excluding standalone retrieval, single-paper question answering, generic coding, and autonomous wet-lab systems from its main coverage.Selected adjacent tasks contribute distinctive designs to core comparisons.Source: §2 · p. 1
• The authors iteratively searched Google Scholar, arXiv, Semantic Scholar, and the ACL Anthology, using backward and forward citation tracing, with a May 10, 2026 cutoff.Descriptions of some studies incorporate subsequent revisions.Source: §2 · pp. 1–2
• Queries combined target and evaluation terms; the selective review places representative comparisons in the main text, additional designs and details in the appendix, and classifies cross-target studies by primary evaluation question.Works without a single-level summary of equally central designs are omitted.Source: §2 · p. 2
• The review compares each study’s target, evidence source, evaluator, metric, and validity boundary to check whether evidence and scoring support the authors’ stated claims.It distinguishes observations from proxies and examines calibration and conclusions under reported conditions.Source: §2 · p. 2
• Five claim-strength levels range from surface plausibility (L1) and task correctness (L2) to process-grounded competence (L3), human- or utility-calibrated performance (L4), and scientific contribution (L5).L3 requires criterion-checked process evidence explaining execution or decisions beyond stored logs; L5 requires domain-appropriate novelty, methods, results, interpretation, and validation.Source: §2 · p. 2
• Cross-stage interpretation separates local results, verified overall completion, and stage interactions, without requiring process records to report observable final outcomes.Process records explain interactions.Source: §2 · p. 2
• Most selected works assess task correctness (L2), while BLADE, AstaBench (E2E), and MLR-Bench check process-grounded competence (L3), and RE-Bench and Scideator center human-calibrated comparisons (L4).The classifications reflect core evaluation designs and main experiments; supplementary process analyses, human ratings, and scorer audits do not independently determine placement.Source: §2 · p. 2

3.1 Scientific Literature Synthesis Evaluation

• Survey benchmarks differ in source selection, output format, and citation evidence, so similarly named metrics may assess different properties and denominators.AutoSurvey measures citation support among already-cited sentences, while SurGE uses titles and abstracts for citation judgments; expert ranking calibrates its structure and content scores, not those citation judgments. SurveyGen’s human-reference-and-outline condition tests writing with supplied organization rather than autonomous source selection.Source: §3.1 · p. 2
• Reference recovery, table usefulness, and supported answerability are distinct outcomes, and content ratings can rank systems differently from answerability.ArxivDIGESTables separates schema recall from cell matching, while additional context mainly improves partial cell matches; TaxoBench’s semantic path scores vary with tree depth and category wording. Across 20 topics, DeepSurvey-Bench content scores were 4.42/5 for DeepResearch and 4.20 for LLM×MapReduce-V2, but topic-specific answerability scores were 1.97/10 and 3.19, respectively.Source: §3.1 · pp. 3–4

Expert judgments and published references.

• Ideation studies assess different forms of novelty, feasibility, and feedback usefulness, with findings conditional on reference choice, generation process, and evaluator calibration.AI proposals received higher novelty ratings than human proposals, with no significant feasibility difference in the specified comparison; Literature-Grounded Novelty Assessment reported 0.81 accuracy and κ = 0.59 on 32 held-out ideas, while Navigating Ideation Space’s 0.384 correlation was close to GPT-5 Mini’s 0.385. AI Idea Bench’s citation-derived feasibility proxies measure influence of associated literature, not calibrated method applicability or step compatibility; generation and selection budgets also shape which proposals are evaluated.Source: Expert judgments and published… · pp. 4–5

3.3 Executable Research Workflow and Scientific Coding Evaluation

• Executable workflow evaluations distinguish code execution, task success, analysis decisions, and research performance, because passing one check does not establish the others.In ScienceAgentBench, Claude’s valid execution rate was 92.2% but task success was 32.4% under best-of-three selection; SciCode’s supplied-background condition raised subproblem success to 35.4% and main-problem success to 12.3%. BLADE checks analysis decisions against expert references, while DiscoveryBench’s no-data baseline helps identify contributions from question and metadata information.Source: §3.3 · pp. 5–6
• Research-engineering comparisons depend on compute, time, and attempt selection, while hidden-test effectiveness and perceived novelty are not interchangeable.Under AIDE, a 24-hour budget, and one A10 GPU, o1-preview reached an MLE-bench medal threshold in 16.9% of attempts, rising to a 34.1% at-least-one-success rate with eight attempts. RE-Bench’s reported 32-hour human result selects the best of four independent eight-hour attempts; MLRC-Bench’s o1 novelty ratings correlated −0.06 with measured effectiveness, which does not establish novelty judgments are inaccurate.Source: §3.3 · pp. 5–6

3.4 Academic Writing and Scholarly Communication Evaluation

• Writing evaluations separate source support, factuality, readability, and visual information access, and input sufficiency and scoring rules qualify their results.CiteBench’s human study found average consistency below 3/5 despite readable outputs, and SciIG’s keyword-overlap measures aligned weakly with human factuality judgments. GREP’s initial 20% faithfulness pass rate for o3-mini means every citation passed in 20% of sections, with parsing failures also affecting the score; PaperBanana’s overall improvement coexisted with faithfulness below the reference baseline.Source: §3.4 · pp. 6–7

3.5 Automatic Peer Review Evaluation

• Citation verification and peer-review assessment require evidence matched to the target: validity, manuscript support, criticism correctness, review usefulness, or workflow effects.CiteAudit precision was .947 on generated references and .749 on real-world references, while recall was .971 and .951, respectively. Peer-review designs include constructed and historical errors, human-annotated utility, and randomized feedback; SPOT’s unmatched findings can include valid criticism outside annotations, and the reported 89% preference for revisions applies only to 100 selected high-uptake pairs.Source: §3.4 · p. 6, §3.5 · pp. 7–8

3.6 End-to-End Automatic Scientific Research System Evaluation

• End-to-end evaluations connect research decisions and outputs, but overall comparisons require reporting each system’s starting resources and verification arrangements.The AI Scientist links ideation, experiments, figures, writing, and automated review within lightweight ML templates, while DeepScientist studies sustained exploration from reproduced baselines using task-native improvements, implementation records, expert assessments, human verification, and substantial compute.Source: §3.6 · p. 8
• AstaBench found 16% of rubric-criterion assessments appeared satisfied in reports but were contradicted by code or artifacts, while a spot check judged 92% of 50 development-set scores correct.Mean step scores and all-required-step completion are reported separately; the non-executing Faker control earns partial step credit but completes no whole task.Source: §3.6 · pp. 8–9
• MLR-Bench reports unverified synthetic or placeholder-data results in eight of ten Claude Code tasks, motivating inspection of execution records as well as papers (Chen et al., 2025b).Its 201 tasks support idea and proposal evaluation, while experiments, writing, and end-to-end runs mainly use ten selected tasks; ten experts provide calibration, but nonsignificant human–model versus human–human discrepancy differences do not establish statistical equivalence.Source: §3.6 · pp. 8–9
• Under an initial 12-hour, single-A100 and $10 API budget, ResearchGym’s main agent improved on the strongest visible baseline in one of 15 runs, with mean subtask completion of 26.5%.ResearchGym provides runnable repositories with source methods removed and retained baselines; AIRS-Bench instead starts with data and evaluation scripts, using 24-hour, single-H200 repeated runs, so comparisons must retain these resource and task-design differences.Source: §3.6 · p. 9
• FIRE-Bench evaluates rediscovery on 30 core and ten cross-domain problems using atomic-claim precision, recall, and F1, with trajectory inspection diagnosing failures across research stages (Wang et al., 2026b).The study uses only part of its 100-paper source pool, and its human check validates claim extraction rather than the subsequent entailment scorer.Source: §3.6 · p. 9
• End-to-end evaluations should report starting assistance, verified stage and whole-task completion, task-native outcomes, and resource use, retaining successes and failures rather than only aggregate artifact scores.Templates, runnable baselines, and data-only starting points impose different research demands; hidden-test performance and reference-finding recovery measure outcomes under those distinct conditions.Source: §3.6 · p. 9

Reporting recommendations

• Task-dependent reporting should address input sufficiency in CiteBench, feasibility-proxy calibration in AI Idea Bench, problem and scorer checks in FLAWS, and report–artifact verification in AstaBench.Source: §4 · p. 9

Comparability and calibration

• Resource conditions and repeated attempts shape results, while evaluator biases documented in general LLM-judge studies require verification in each scientific setting.MLE-bench varies attempt budgets, and RE-Bench compares agents and experts at stated time budgets.Source: §4 · p. 9

Reporting recommendations

• Report each evaluation’s task and intended claim, inputs and data version, evaluator and scoring, sample and aggregation unit, applicability conditions, and public artifacts.Combine setting-specific reporting modules where applicable, such as error construction and matching for localization or participants and outcomes for feedback utility.Source: §4 · p. 9

• The survey compares evaluation designs across six targets and synthesizes findings and evidence gaps.Source: §5 · p. 9
• It develops setting-specific reporting recommendations to help readers select benchmarks and interpret results.Source: §5 · p. 9
• The comparisons also help readers design subsequent evaluations.Source: §5 · p. 9

Limitations

• The review emphasizes automated AI/ML research workflows with inspectable evaluation evidence, rather than exhausting broader AI-for-Science or autonomous wet-lab research.Coverage is uneven because many available benchmarks concern computer science and machine learning; the review synthesizes reported studies and protocols rather than reproducing experiments.Source: Limitations · pp. 9–10

A Comparative Mapping and Reporting Recommendations

• Reporting recommendations are organized by evaluation setting, with task-dependent checks including input sufficiency, feasibility-proxy calibration, scorer checks, and report–artifact verification.Source: Table 2 · p. 17

B Search and Protocol Records

• Protocol tables record task inputs, evidence, scoring, calibration or controls, findings, and applicability conditions, while searches combine target-specific terms with backward and forward citation tracing.Source: B Search and Protocol Records · p. 18

Survey generation and content assessment

• AutoSurvey citation recall covers already-cited sentences, while its human-ranking correlation is 0.5429; separate LLM assessments score coverage, structure, relevance, and citation support.Source: §B.1 · p. 18
• AutoSurvey improves accuracy by 9.2 points over no material and 2.4 points over naive retrieval; the reported use is simulated.Source: §B.1 · p. 18
• SurGE uses 205 human surveys and a shared retrieval pool to compare reference coverage, citation accuracy, structure, and content; expert rankings calibrate structure and content, not citation judgments.Source: §B.1 · p. 18
• SurveyGen/QUAL-SG uses abstracts for citation support, while its ranking calibration does not cover citation judgments; Tang et al.’s resource contains 4,205 surveys, not 4,205 evaluated outputs.Supplied outlines change the task, and the Tang resource uses reference authenticity, overlap, key-point recall, text and structure similarity, and LLM ratings.Source: §B.1 · p. 18

Reference evaluation

• For reference checking, automated accuracy is 86% with κ = 0.71, based on three raters checking 100 references.Matching allows partial metadata agreement, while authenticity and reference overlap measure different properties.Source: §B.1 · p. 18

Expert judgments and reference-driven comparisons

• AI proposals received higher novelty ratings than human proposals, while idea-level feasibility differences were not significant.The comparison covers seven prompting-based NLP topics, with 104 researchers and 49 proposals per condition; AI uses extensive sampling and retrieval, while each human contributes one idea.Source: §B.2 · p. 22

AI Idea Bench

• Motivation and plan rankings can differ, and target matching measures reference recovery rather than idea quality alone.One baseline lacks plans.Source: §B.2 · p. 22

Literature proxies

• AI Idea Bench also scores literature-relative novelty and citation-impact feasibility, but citation influence does not test method applicability or step compatibility.No independent step-feasibility calibration is reported.Source: §B.2 · p. 22

IdeaBench

• IdeaBench’s Insight Score is the fraction ranked above the target, yet authors report high scores for some incoherent outputs.Its setup mixes three generated ideas with a target idea drawn from biomedical target-paper references.Source: §B.2 · p. 22

Other evaluation protocols

• A separate evaluation compares ideas through matching, semantic diversity, expert ratings, and language-model identification and ranking tasks.The reported protocols include blinded expert ratings of novelty, relevance, feasibility or correctness, and other dimensions.Source: §B.2 · p. 22

Scientific code and analysis decisions

• Scientific coding benchmarks distinguish subproblem pass@1 from main-problem success, which requires both completing steps and integrating them.SciCode’s test set has 65 main problems and 288 subproblems; decomposition is provided rather than independently generated.Source: §B.3 · p. 25

BLADE

• BLADE’s 188 questions test recognition of defensible analysis choices, not implementation of an analysis.Its reference decisions are expert-derived and generation is deterministic.Source: §B.3 · p. 25

DiscoveryBench

• 96% of GPT-4o analyses are runnable and nonempty, yet model and variable coverage remains below 13%.This contrast distinguishes successful execution from coverage of reference analysis choices.Source: §B.3 · p. 25
• Llama-3 scores 11.5 without data versus 12.1 with CodeGen, while the matching score alone does not isolate data verification.The no-data baseline is diagnostic of contributions from question and metadata information.Source: §B.3 · p. 25

Workflow evaluation conditions

• Workflow scores depend on attempt selection and success criteria: best-of-three is not single-run success, and omitting presentation or integrity changes the definition of success.Some code rubrics can mix reference implementation with partial completion, while error propagation is measured separately.Source: §B.3 · p. 25

Introduction generation

• Some reference targets require facts absent from the input, but the prevalence of affected cases is unquantified.SciIG evaluates introductions using ROUGE, BERTScore, citation-intent or discourse KL, and human ratings.Source: §B.4 · p. 27

Citation paragraphs

• 3,900 citation-paragraph tasks show that keyword factuality proxies align weakly with human ratings, while source-set recovery is a separate measure.Task exploration evaluates citation paragraphs.Source: §B.4 · p. 27

Related-work constraints

• Automatic metrics only partly track atomic-fact coverage in an exploratory study that explicitly manipulates input settings.The human study uses six prompts, six input settings, and 30 examples.Source: §B.4 · p. 27

Citation attribution

• Full-text input changes attribution errors, and the selected judge’s wrong-citation F1 is .23 on the pre-extracted test.The evaluation distinguishes Unverifiable from False.Source: §B.4 · p. 27

Paragraph organization

• Human calibration for paragraph organization concerns whole-section structure, leaving evidence for paragraph-level measurement limited.Four people rated 25 Llama2 outputs, with Fleiss’ κ = .156.Source: §B.4 · p. 27

Object descriptions

• For figure and table descriptions, targets are prose spans rather than captions, and context benefits vary by object type.The study compares object-only, local, and retrieved context.Source: §B.4 · p. 27

Review content and calibration

• Preference calibration does not independently label whether review claims are factually supported.ReviewAgent is a separately evaluated review generator.Source: §B.5 · p. 32
• GEM-S teacher correlation is .479, below LMExaminer’s .537, and robustness tests measure a different property.These evaluations require peer references rather than a unique gold review.Source: §B.5 · p. 32

Focus alignment

• MindA’s focus measure is category frequency, not attention weights or critique truth, and abstract and main-text samples differ.The evaluation codes review and meta-review points by manuscript target and evaluation aspect.Source: §B.5 · p. 32

Feedback overlap

• Reported feedback overlap measures shared concerns, but rejected-paper overlap and total review counts have version-internal conflicts.The study uses human extraction and matching validation, a shuffled-paper specificity control, and matched point-count comparison.Source: §B.5 · p. 32

Perceived usefulness

• 57.4% of prospective opt-in users rated feedback helpful or very helpful; the 82.4% comparison is with at least some human feedback, not the average expert.The user survey involved AI or computational-biology researchers and was separate from corpus matching.Source: §B.5 · p. 32

B.6 End-to-End Research Protocols

• The suite, stage, and full-run rows preserve different denominators across connected research subprotocols and rediscovery.Table 8 presents end-to-end scientific research system evaluation protocols.Source: §B.6 · p. 37
• AI Scientist evaluations combine ideas, code changes, experiments, plots, and papers across three lightweight ML domains and multiple backbones.They report task-native losses, funnel counts, automated-review scores, and costs; experiment completions and compiled-paper counts differ, and threshold crossings are automated-review outcomes.Source: §B.6 · p. 37
• DeepScientist’s cross-conference human comparison and model-to-mean versus person-to-person correlations do not establish same-condition human-level performance.The study also includes human result verification, random-idea filtering comparisons, and a failed-implementation audit.Source: §B.6 · p. 37
• One review protocol uses 500 ICLR 2022 papers and text-only model reviews, scoring decision accuracy, balanced accuracy, F1/AUC, and rating correlation.Reflection, ensemble, and threshold variants are compared with published NeurIPS human statistics as a reference.Source: §B.6 · p. 37
• About 5,000 ideas, 1,100 validations, and 21 progress findings were reported for DeepScientist, using 20,000+ GPU hours.The study also reports reproduced baselines, findings memory, implementations, and task results; historical publication spans are not measured human research time.Source: §B.6 · p. 37
• CycleResearcher reports automatic and human mean ratings as different channels; simulated acceptance is not conference acceptance.Its evaluation covers five system papers and external generated papers, with automatic and expert reviews.Source: §B.6 · p. 37
• CycleReviewer’s main protocol simulates results, while Appendix I separately demonstrates one executed study; selected outputs are compared by three NLP experts.The workflow starts from bibliographies and abstracts, produces long research papers with simulated experiments, and includes review-guided refinement.Source: §B.6 · p. 37
• Automatic-review ratings are evaluated against other reviewers’ means, and agreement with aggregate ratings is the proxy target.Reported percentage reductions conflict with rounded table values.Source: §B.6 · p. 37