AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents

1 INTRODUCTION

• Scientific benchmarks are costly to construct and update, requiring substantial labor and domain expertise as agent capabilities advance and existing benchmarks saturate.The introduction notes the particular stakes of reliable scientific-agent analysis in areas such as drug discovery.Source: §1 · p. 1
• Scientific-agent evaluation must specify environments and operations for acquiring information, then assess reasoning from indirect evidence rather than explicit correctness feedback.Agents may need to re-examine raw expression data, validate intermediate results, and integrate chromatin-contact evidence to assess support for causal regulation.Source: §1 · p. 2
• AutoSciBench uses high-level concepts and low-level recipes to define evaluation requirements and construct tasks, then revises them from solver behavior and reuses refinement histories for later concepts.Refinement shifts tasks toward raw-data re-examination, intermediate-result interpretation, and evidence integration.Source: §1 · pp. 2–3
• Generated benchmarks are more challenging than original manually curated benchmarks and receive higher average quality scores across computational biology, materials science, and clinical imaging.Further analyses find that iterative refinement increases difficulty and experience-guided generation supports increasingly challenging tasks.Source: §1 · p. 3

2 RELATED WORK

• Human-curated agent benchmarks cover software engineering, web and computer environments, and scientific reasoning, data analysis, and domain-specific tool use, but require substantial human effort and expertise.Scientific evaluation can also depend on costly expert annotation, making continuous updates difficult as agent capabilities advance.Source: §2 · p. 3
• Automatic benchmark-generation methods use objectives, quality criteria, staged construction, benchmark transformations, or risk taxonomies, but primarily target LLM evaluation.Examples include AutoBencher and BenchMaker [Li et al., 2025; Yuan et al., 2026], BenchAgents and Benchmark Self-Evolving [Butt et al., 2024], and S-Eval [Yuan et al., 2025].Source: §2 · p. 3
• AutoSciBench addresses scientific-agent needs with high- and low-level planning for question and data construction, plus refinement guided by solver behavior.The target tasks require information acquisition through interaction and scientific reasoning from indirect evidence.Source: §2 · p. 3

3 AUTOSCIBENCH

• AutoSciBench cycles through concept generation, recipe planning, task implementation, solver-based evaluation, and refinement, using solver feedback to revise plans and accumulated experience to inform later concepts.Figure 2 depicts the framework overview.Source: §3 · p. 3, Figure 2 · p. 4

3.1 CONCEPT: HIGH-LEVEL PLAN FOR EVALUATION REQUIREMENTS

• A concept specifies the scientific domain, data modality, and required analytical operations, separating the capability to assess from its concrete task instantiation.Concepts draw on existing benchmarks and scientific literature, including extracted workflows and generalized analytical patterns, as source-grounded references for new concepts.Source: §3.1 · p. 4

3.2 RECIPE: LOW-LEVEL PLAN FOR TASK CONSTRUCTION

• A recipe translates a concept into a question and data design that requires intended analytical operations to acquire answer-relevant information through interaction.It specifies which information the question states and which the solver must obtain from the scientific data.Source: §3.2 · p. 4

3.3 TASK IMPLEMENTATION AND ITERATIVE REFINEMENT

• Task implementation produces a question, scientific data files, and a verified ground-truth answer; solver trials and judge feedback assess verifiability, difficulty, and bypasses of intended reasoning.If a task is not both verifiable and hard, feedback guides recipe changes to the data and question or concept changes to the analytical target.Source: §3.3 · p. 5

3.4 EXPERIENCE-GUIDED CONCEPT GENERATION

• Refinement histories capture task designs, solver behavior, revisions, and unresolved issues; reusable experience cards guide later concept generation alongside source priors and discourage duplicate proposals.Experience is retained whether refinement yields a hard, verifiable task or not.Source: §3.3 · p. 6, §3.4 · p. 6

4.1 BENCHMARK GENERATION DETAILS

• AutoSciBench builds 50 tasks per domain from seed benchmarks and literature-derived workflows, using five experience-guided concept-generation iterations and up to 10 refinement rounds per task.The final benchmark selects one task from each refinement run, prioritizing higher difficulty ratings.Source: §4.1 · p. 6

4.2 MODEL PERFORMANCE ON ORIGINAL AND GENERATED BENCHMARKS

• Generated benchmarks generally lower solver accuracy than human-curated benchmarks in computational biology and materials science; later models score 19.5pp and 15.1pp higher on their respective generated benchmarks.Generated tasks also reveal performance differences less apparent on the original materials-science benchmark.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 BENCHMARK QUALITY ASSESSMENT

• Generated benchmarks receive higher average quality scores than human-curated benchmarks from both judges across all three domains, with GPT-5.6-Sol-generated tasks scoring highest on average.GPT-5.6-Sol-generated benchmarks have high fitness scores across domains; the measured quality criteria do not decline as task difficulty increases.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 EFFECT OF FEEDBACK-GUIDED ITERATIVE REFINEMENT

• The proportion of hard tasks increases over refinement rounds, as initial tasks are predominantly easy and solver feedback guides revisions.In one revised UMI task, all three solver trials treated error-containing sequences as distinct original UMIs and answered incorrectly.Source: Figure 4 · p. 8, §4.4 · p. 8

4.5 EFFECT OF EXPERIENCE-GUIDED CONCEPT GENERATION

• In materials science, experience-guided concept generation reduces solver success from 66.7% to 35.0% and raises the hard-task proportion from 15.0% to 50.0%.Subsequent concepts can change domain and modality while reusing higher-level solving behaviors from prior refinement experience.Source: Table 3 · p. 9, §4.5 · p. 9

4.6 EFFECT OF CONCEPT AND RECIPE PLANNING

• Solver success declines more across refinement rounds with the full concept-and-recipe planning framework than when recipe planning or both planning stages are removed.Concepts specify analytical requirements, while recipes specify question and data design; the results suggest revising these plans increases difficulty more effectively than directly revising instantiated tasks.Source: Figure 5 · p. 9, §4.6 · p. 9

4.7 QUALITATIVE ANALYSES ON GENERATED TASKS

• Representative tasks across all three domains require agents to interrogate scientific data, interpret intermediate observations, and integrate multiple evidence sources to answer.The examples span diverse scientific topics, data modalities, and analytical requirements.Source: §4.7 · p. 9

5 CONCLUSION

• AutoSciBench uses concepts and recipes to define analytical requirements and guide question and data construction, then refines these plans and generates later concepts from prior experience.The conclusion describes iterative construction, solver-based evaluation, refinement, and experience-guided generation.Source: §5 · pp. 9–10
• Across three scientific domains, generated benchmarks expose solver failures and receive higher average quality scores than the original benchmarks.Further analyses suggest that refinement increases task difficulty and shifts tasks toward data re-examination and interpretation.Source: §5 · p. 10
• The results suggest an adaptive evaluation process in which solver behavior guides task refinement and accumulated refinement experience informs new evaluation targets.The conclusion frames this adaptation as occurring as agent capabilities evolve.Source: §5 · p. 10

A.1 SEED BENCHMARKS

• CompBioBench contains 100 agentic computational-biology tasks across six areas, requiring multi-step reasoning, code execution, tool use, and external scientific resources.The areas include genomics, transcriptomics, epigenomics, single-cell analysis, human genetics, and machine-learning workflows.Source: §A.1 · p. 13
• MatTools evaluates computational materials-science tool use through large-scale pymatgen-derived tasks and executable materials-property calculations.Its tasks draw on the pymatgen codebase and documentation.Source: §A.1 · p. 13
• MedCTA uses clinician-validated tasks with radiology images, pathology slides, and clinical reports, asking agents to acquire and integrate evidence without a prescribed tool sequence.An LLM evaluator scores its open-ended answers.Source: §A.1 · p. 13

A.2 CONCEPT INFERENCE FROM EXISTING BENCHMARKS

• Seed questions are represented with associated files; agents identify their scientific domain, data modality, and analytical requirements, while literature-derived workflows and recurring patterns supply further concept inputs.Literature workflows capture inputs, analytical steps, intermediate checks, and target outputs; recurring patterns abstract shared steps and decision points.Source: §A.2 · p. 13, §A.3 · p. 13

A.4 INITIAL CONCEPT GENERATION

• Initial concepts combine benchmark-derived concepts, literature workflows, and analytical patterns, specifying domain, modality, and operations without prescribing a solution procedure.Five concepts are generated per batch, with prior concepts supplied to discourage duplicates and minor variations.Source: §A.4 · p. 13

A.5 RECIPE PLANNING

• Each recipe specifies data strategy and plan, construction and ground-truth verification, answer format, tools, and a question-hiding strategy.For clinical imaging, agents are encouraged to reuse benchmark images or retrieve real images rather than simulate them.Source: §A.5 · pp. 13–14

A.6 TASK IMPLEMENTATION

• Task implementation produces a question, data files, and reference answer, with code used to acquire or generate data and compute and verify a unique answer for exact-match scoring.Questions state the analysis target, input files, and answer format without revealing the solution procedure.Source: §A.6 · p. 14

A.7 SOLVING AND JUDGING

• Three independent solver trials are judged for verifiability, difficulty, and reasoning; valid tasks are easy, moderate, or hard when all, some, or none of the trials succeed.Solver answers are evaluated by exact match against the reference answer.Source: §A.7 · p. 14

A.8 FEEDBACK-GUIDED REFINEMENT

• Tasks that are not both verifiable and hard are revised using judge feedback, with recipe changes followed by concept revision after four unsuccessful recipe rounds, for up to ten rounds.Recipe revisions can change data construction, answer computation and verification, or question design.Source: §A.8 · p. 14

A.9 EXPERIENCE-GUIDED CONCEPT GENERATION

• Completed histories are distilled into design lessons, patterns, anti-patterns, and guidance, which are supplied with source priors and prior concepts for subsequent concept generation.The experience summary abstracts away task-specific answers and data details.Source: §A.9 · p. 14

A.10 GENERATION AND INFERENCE CONFIGURATION

• The pipeline uses Claude Opus 4.8 or GPT-5.6 Sol consistently within each generation condition, with high reasoning effort and a 3,600-second timeout per stage.Experience-guided generation uses up to 12 experience cards and 120 prior concepts, sampled uniformly at random under a fixed seed when records exceed those limits.Source: §A.10 · p. 14

B.1 ADDITIONAL RESULTS ON QUALITY ASSESSMENT

• Generated benchmarks receive higher average scores than human-curated benchmarks across domains; GPT-5.6-Sol generations have the highest average and fitness scores, while Opus generations have higher usefulness in computational biology and materials science.The trends are consistent across both judges, and the authors say the quality differences are not driven solely by one judge model.Source: Table 4 · p. 15, Table 5 · p. 15, §B.1 · p. 15
• Both judges show positive Spearman’s ρ and Kendall’s τb correlations across criteria, indicating general agreement in their relative task-quality rankings.The authors present this as support for the consistency of relative quality assessments.Source: Table 6 · p. 15, §B.1 · p. 15

B.2 EFFECT OF FEEDBACK-GUIDED ITERATIVE REFINEMENT BY GENERATION MODEL

• The proportion of hard tasks increases across refinement rounds for both generation models, with a larger increase for Claude-Opus-4.8.The authors suggest Claude-Opus-4.8 is particularly useful for constructing tasks that challenge its own solver through iterative refinement.Source: §B.2 · p. 15, Figure 6 · p. 16

B.3 CONCEPT EXAMPLES FOR EXPERIENCE-GUIDED GENERATION

• An experience-guided concept retains identification, recovery, and consistency-checking operations while changing the target quantities, measurements, physical models, and data modalities.The table shows the previous concept on the left and the experience-guided concept on the right; matching colors indicate corresponding solving behaviors.Source: §B.3 · p. 16, Table 7 · p. 16

B.4 QUALITATIVE ANALYSIS OF THE MICROSATELLITE INSTABILITY TASK

• Agents must count markers with tumor-acquired repeat-length instability not explained by germline variation or the matched normal.The answer must be a single base-10 integer from 0 to 45.Source: §B.4 · p. 17
• The constructed data include stable homozygous and heterozygous markers, allelic imbalance, contamination, and somatic MSI, with construction labels withheld from solvers.Figure 7 shows normal and tumor read counts and their difference for each marker; each sample has 2,000 reads per marker.Source: §B.4 · p. 18, Figure 7 · p. 18
• For all 45 markers, agents use Python with pysam to extract repeat lengths from reads spanning each tract and compare paired tumor/normal distributions.The workflow uses panel.tsv and reference.fa to locate tracts and flanking sequences, accounting for insertions and deletions.Source: §B.4 · p. 18
• The challenge is interpreting distributional patterns, not merely detecting tumor-normal differences: somatic MSI shifts counts toward shorter repeat lengths in the illustrated example.This pattern must be distinguished from allelic imbalance and contamination; contamination introduces new repeat lengths, whereas the target example redistributes reads among existing lengths.Source: §B.4 · p. 18, Figure 7 · p. 18

B.5 ANALYSIS ON GENERATION COST

• Implementation and solving are the most cost-consuming stages for both models.Table 9 reports the cost proportions for implementation, solving, and their combined total.Source: §B.5 · p. 19, Table 9 · p. 19
• Opus 4.8 costs less than GPT-5.6 Sol across all three domains while its generated benchmarks have lower average solver accuracy.The authors interpret this comparison as indicating greater cost-effectiveness for generating challenging tasks; Table 8 reports cost per question and average benchmark difficulty.Source: Table 8 · p. 19, §B.5 · p. 19

C.1 COMPUTATIONAL BIOLOGY: CLAUDE OPUS 4.8

• The examples ask solvers to identify the highest-predicted-presentation peptide for MHC class I allele HLA-A*02:01 and the genuinely essential gene in a pooled CRISPR knockout screen with copy-number-dependent depletion and non-functional guides.The peptide generator selects 30 candidates using MHCflurry predictions; the screen simulates guide-level read counts before and after selection and creates gene-level copy-number annotations.Source: §C.1 · p. 20, P12 P01375 194 YLGGVFQL P13 P68871 13… · p. 21

C.2 COMPUTATIONAL BIOLOGY: GPT-5.6-SOL

• The GPT-5.6-SOL biology examples require integrating simulated phosphoproteomics evidence to identify a direct kinase–substrate site and combining contig composition, coverage, and paired-read links to detect contamination.The contamination example makes the foreign contig’s sequence composition resemble the majority genome while preserving distinguishing coverage and linkage evidence.Source: §C.2 · pp. 22–23

C.3 MATERIALS SCIENCE: CLAUDE OPUS 4.8

• The Claude Opus 4.8 materials examples require finding a valid BET fitting region in a perturbed adsorption isotherm and inferring the fundamental band gap among defect and higher-energy absorption features.The band-gap task leaves transition type unspecified and places the fundamental edge on an exponential background.Source: §C.3 · pp. 24–25

C.4 MATERIALS SCIENCE: GPT-5.6-SOL

• The GPT-5.6-SOL materials examples require inferring a latent reaction mechanism from reference observations to select a qualifying precursor route, and combining distributed calibration and transformation inputs to compute a sensor-frame stress component.The stress task uses exact rational stiffness and deformation values, with target designation, calibration records, and transformation rules distributed across files.Source: §C.4 · pp. 26–27

C.5 CLINICAL IMAGING: CLAUDE OPUS 4.8

• Two imaging examples ask a solver to calculate cardiothoracic ratio from constructed landmark measurements and determine RECIST 1.1 response at latest follow-up using lesion annotations across three time points.The longitudinal case reuses MedCTA chest and abdominal CT images with constructed lesion identities, measurements, and dates, rather than an actual patient follow-up series.Source: §C.5 · pp. 28–29

C.6 CLINICAL IMAGING: GPT-5.6-SOL

• The GPT-5.6-SOL imaging examples classify edema–enhancement correspondence and coronary distribution from cardiac MRI, and count cell phenotypes in a defined spatial neighborhood from multiplex images and a cell-instance mask.The neighborhood task supplies marker-positivity, phenotype-assignment, and spatial-distance rules in definitions.json.Source: §C.6 · pp. 30–31

D PROMPTS

• The prompt appendix lists prompts for concept generation, task instantiation, solving and judging, refinement, experience-guided generation, and benchmark quality assessment; Figure 8 shows the seed-question concept-extraction prompt.Instance-specific inputs are represented by placeholders.Source: D PROMPTS · p. 32, §D.1 · p. 32

D.1 INITIAL CONCEPT GENERATION

• Initial concept generation draws on benchmark concepts, extracted workflows, and recurring patterns, while literature search prioritizes recent sources that expose actual analytical workflows.The workflow-extraction prompt prefers fewer or no results over filling the output with older, weakly justified sources.Source: §D.1 · p. 32, §D.7 · p. 33, Figure 11 · p. 36

D.5 EXPERIENCE-GUIDED CONCEPT GENERATION

• Experience construction distills reusable design guidance from completed refinement histories, and generation pairs that guidance with the concept-generation prompt.The generation prompt uses source priors when no prior experience cards are available.Source: §D.5 · p. 32, §D.7 · p. 35

D.2 TASK INSTANTIATION

• Recipe planning and task implementation translate a domain–modality–operation concept into a construction recipe, then build a question, data, and verified reference answer.The recipe specifies data strategy, how the answer is made and hidden, and how its unique exact-match ground truth will be computed and checked.Source: §D.7 · pp. 36–37, Figure 12 · p. 37, Figure 13 · p. 38

D.3 MULTI-TRIAL SOLVING AND JUDGING

• Solver attempts are judged separately for verifiability and difficulty, using trial outcomes and reasoning traces to assess whether the answer is unique and how reliably agents solve it.Verifiability is assessed from the solver-facing question and data, not builder-only information.Source: §D.7 · p. 38

D.4 FEEDBACK-GUIDED REFINEMENT

• Feedback-guided refinement revises recipes while preserving the concept, then revises concepts from repeated QA histories to address shortcuts or underspecification.Recipe changes can resolve ambiguity or increase difficulty; concept changes target deeper reasoning while preserving the broad scientific domain when possible.Source: §D.7 · pp. 40–41

D.7 DOMAIN-SPECIFIC PROMPT VARIANTS

• Domain-specific prompts define concepts as domain, modality, and atomic operation, and require unique, exact-match answers supported by concrete data construction and ground-truth verification.They direct concept revisions toward deeper reasoning, evidence integration, or clearer estimands rather than arbitrary noise or procedural detail.Source: §D.7 · pp. 36, 41
• Refined prompts use prior QA to identify what to improve, while requiring a new, independently checkable task that follows the refined recipe rather than preserving an old answer-bearing determinant.Question-hiding guidance should conceal intermediate reasoning, not the task target, input files, or answer format.Source: §D.7 · p. 42
• The appendix identifies prompts for task implementation, experience construction, experience-guided concept generation, quality assessment, and domain-specific substitutions in computational biology, materials science, and clinical imaging.Clinical-imaging prompts also cover recipe planning and revision guidance.Source: Figure 18 · p. 43, Figure 19 · p. 43, Figure 20 · p. 44
• Materials-science variants span domains and modalities such as crystallography, CIF structures, spectra, and simulation trajectories, with operations including space-group inference and structure-property evidence integration.The examples also include consistency reasoning between symmetry and measured diffraction evidence.Source: §D.7 · p. 45

• Fig. 21 presents a prompt for assessing a question’s verifiability, usefulness, and fitness using the question and its associated data files.Source: §D.6 · p. 33