PaperlayerView exact revision on arXiv

Source-linked AI summary

SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

Tianyun Zhong, Wei Wang, Jialin Li, Jianhong Tu, Liyao Li, Chuqiao Kuang, Tianshi Zheng, Songchi Zhou, Weiqi Zhai, Qile Jin, Jiajia Li, Jiasheng Zheng, Boxi Cao, Yaojie Lu, Hongyu Lin, Shiwei Ye, Bing Zhao, Hu Wei, Lin Qu, Xianpei Han

arXiv:2610.32441v1cs.CL

TL;DR

Existing scientific-agent benchmarks often cover selected research activities, limiting systematic assessment across workflows, deliverables, and execution dynamics. SCI-AGENT-EXAM evaluates agents on a workflow-oriented benchmark with hybrid assessment and trajectory analysis. GPT-6 Astra scores highest overall, while results also show uneven task-type capabilities and variation in execution efficiency and stability.

  • Problem

    Existing benchmarks often focus on selected scientific activities and do not consistently assess answers, executable artifacts, and written explanations together, while final-answer scores provide limited insight into execution dynamics.

    Source: §1 · p. 2

  • Method

    SCI-AGENT-EXAM evaluates agents on 120 expert-curated tasks across 11 domains and seven task types, combining deterministic executable-artifact checks, calibrated LLM judgments of textual artifacts, and trajectory analysis.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    GPT-6 Astra achieves the highest overall score at 80.9, 13.6 points above Claude Fable 5, but only 26.0% of its runs reach at least 0.95; capabilities and execution trajectories vary across agents.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Stronger agents tend to make earlier progress, while progress and solution stability capture distinct aspects of agent trajectories.

    Source: §5 · p. 9

    The study primarily evaluates general-purpose foundation models under standardized agent settings, rather than specialized agents with domain-specific tools, memory, planning, and iterative research capabilities.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam.

1 Introduction

Sci-Agent-Exam is a workflow-oriented benchmark designed to assess scientific agents across research activities, required deliverables, and execution dynamics. Evaluation of 10 agents finds substantial performance differences and uneven capabilities across task types.

1 Introduction

  • Existing benchmarks have uneven coverage of scientific activities and do not consistently require and jointly evaluate answers, executable artifacts, and textual artifacts for every task.Final-answer scores alone also provide limited insight into execution efficiency and solution stability.Source: §1 · p. 2
  • 80.9 overall score makes GPT-6 Astra the top agent, 13.6 points ahead of Claude Fable 5, but only 26.0% of its runs reach at least 0.95.Stronger agents generally reach high-quality solutions earlier and undergo fewer regressions during execution.Source: §1 · pp. 2–3
  • Sci-Agent-Exam comprises 120 expert-curated tasks across 11 scientific fields and seven Scientific Workflow Task Taxonomy task types.Each task requires a structured answer, executable artifacts, and textual artifacts; evaluation combines deterministic checks with calibrated, expert-validated LLM judgments.Source: §1 · pp. 2–3
  • Trajectory analysis shows that higher-scoring agents tend to use fewer execution steps and exhibit fewer regression and recovery steps.The benchmark analyzes intermediate workspace snapshots to characterize execution efficiency and solution stability.Source: §1 · pp. 2–3

2 SCI-AGENT-EXAM

Sci-Agent-Exam defines scientific research as executable workflow tasks and combines quality-controlled task packages with hybrid assessment of heterogeneous outputs.

2.1 Task formulation and taxonomy

  • Each task specifies a scientific objective, execution environment, visible resources, deliverables, and evaluation criteria, while evaluator checks and rubrics remain hidden.The agent uses the instruction, environment, visible resources, and deliverable specification to produce artifacts under controlled conditions.Source: §2.1 · p. 3
  • SWTT organizes research into seven task types spanning 11 scientific domains, from literature synthesis and hypothesis validation to error diagnosis and result interpretation.The taxonomy also includes tool orchestration, code generation, and end-to-end analysis.Source: §2.1 · pp. 3–4

2.2 Benchmark Construction

  • The benchmark distributes 120 tasks across 11 domains and seven task types, sourced from research papers, competitions, datasets, and maintained open-source projects.Construction includes expert-built executable packages, reference solutions, deterministic checks, and human-written rubrics; independent expert review and solution checks precede release.Source: §2.2 · pp. 4–5, Table 2 · p. 5

2.3 Hybrid evaluation protocol

  • Tasks run in dedicated Docker containers through a shared command-line harness that collects artifacts and applies deterministic checking and rubric judgment.Programmatic checks rerun artifacts on hidden data, while rubric criteria assess scientific correctness, validation, grounding, communication, and completeness.Source: §2.3 · p. 5
  • Programmatic and rubric scores are combined using weights fixed before evaluation that vary by task type, with both component scores reported alongside the total.Scoring points are weighted and assessed through complementary executable checks and pass/fail rubric criteria.Source: §2.3 · p. 5

3 Experiments

The experiments compare ten agents on final performance, execution and report quality, trajectories, and cost. GPT-6 Astra leads overall, but results also show uneven capabilities and differences in solution stability and run-level efficiency.

3.1 Experimental setup

  • Each model is run independently three times per task under the same visible task package and execution protocol, using Claude Code with high inference effort.Each task has approximately eight programmatic checks and 20 LLM-judge rubric criteria; model scores average three runs before macro-averaging across tasks.Source: §3.1 · p. 6

3.2 Main Results

  • 80.9 is GPT-6 Astra’s highest combined score, 13.6 points above second-ranked Claude Fable 5; only 26.0% of Astra’s runs reach a normalized combined score of at least 0.95.The benchmark covers 120 tasks using the Claude Code harness; Table 3 reports programmatic and LLM Judge scores and Pass@0.95.Source: §3.2 · p. 6, Table 3 · p. 6
  • GPT-6 Astra leads in 10 of 11 domains, while Claude Fable 5 and Kimi K3 show contrasting strengths across domains.Claude Fable 5 scores 80 in Engineering and 49 in Bioinformatics; Kimi K3 scores 81 in Materials Science and 50 in Physical Science. Qwen3.8 Max, GLM-5.3, and Kimi K3 perform relatively well on Literature Synthesis.Source: §3.2 · p. 7, Figure 4 · p. 7

3.3 Hybrid Evaluation of Execution and Report Quality

  • 0.14–0.48 is the range of TL rates across models; GPT-6 Astra has a TR rate of 0.83 and TL rate of 0.14.DeepSeek-V4-Pro has the highest TL rate (0.48) and lowest TR rate (0.25); separate examples show that perfect programmatic scores can coexist with weak judged reports, and vice versa.Source: Figure 3 · p. 6, §3.3 · p. 7

3.4 Trajectory Behavior Analysis

  • GPT-6 Astra and GPT-5.6 Sol have the highest early-completion rates, but Sol’s mean dip rate is 27.0% versus Astra’s 18.3%.Across five of six displayed models, programmatic checks recover more often than Scientific Correctness criteria; Kimi K3 is the exception, with recovery rates of 40.4% and 42.9%, respectively.Source: §3.4 · pp. 7–8, Figure 5 · p. 8

3.5 Efficiency and Cost Analysis

  • GPT-6 Astra achieves the highest combined score (80.9) at $20.79 per run, while GLM-5.3 incurs the highest cost ($57.96) and averages 102.4 turns per run.Claude Opus 4.8, Claude Fable 5, and GPT-6 Astra lie on the cost–performance Pareto frontier; the analysis measures mean API cost, runtime, and interaction length per run.Source: §3.5 · p. 8, Figure 6 · p. 8

4 Related Work

Prior benchmarks cover selected scientific activities or output modalities, while scientific agents increasingly operate across research workflows. Sci-Agent-Exam is positioned as a standardized, stage-resolved evaluation of those workflows and agent behavior.

4.1 Science Benchmarks

  • Existing science benchmarks typically isolate a particular activity and output modality, leaving the span of a full scientific workflow underassessed.Examples include question answering, discovery, computational analysis, executable programs, replication, and research coding.Source: §4.1 · pp. 8–9
  • Sci-Agent-Exam organizes evaluation across seven workflow task types and requires both executable and textual deliverables.This combines assessment of code-level correctness with scientific judgment.Source: §4.1 · p. 9

4.2 Scientific Agents

  • Unlike autonomous systems emphasizing pipeline breadth, Sci-Agent-Exam targets standardized, stage-resolved evaluation of agents on well-scoped scientific workflow tasks.It examines final output quality alongside trajectory-level progress and stability.Source: §4.2 · p. 9

5 Conclusion

The paper presents Sci-Agent-Exam as an expert-curated benchmark for scientific inquiry that evaluates computational execution and scientific reasoning. Its findings emphasize both earlier progress and distinct trajectory stability, supporting evaluation beyond isolated task performance.

5 Conclusion

  • Sci-Agent-Exam evaluates language agents across major stages of scientific inquiry with 120 expert-curated tasks spanning 11 domains and seven SWTT task types.Its hybrid protocol assesses computational execution and scientific reasoning.Source: §5 · p. 9
  • Stronger agents tend to make earlier progress during execution, while progress and solution stability capture distinct aspects of agent trajectories.The paper frames these findings as a reason to evaluate agents beyond isolated task performance.Source: §5 · p. 9
  • The benchmark’s findings provide insights for developing agents with broader capabilities and more effective and stable execution throughout scientific workflows.Source: §5 · p. 9

Limitations

The study focuses on general-purpose foundation models in standardized agent settings, while identifying specialized agents, additional workflows, and evaluator configurations as areas for future extension. It also describes its AI-use practices, ethics safeguards, and benchmark construction.

Limitations

  • The study primarily evaluates general-purpose foundation models under standardized agent settings, with specialized agents and broader configurations left for future work.Suggested extensions include domain-specific tools, memory, planning, iterative research capabilities, additional workflows, and evaluator configurations.Source: Limitations · p. 9

A AI Use Statement

  • Generative AI tools provided manuscript feedback and wording suggestions, but authors reviewed revisions and did not use them to generate benchmark data or results.The tools also were not used to develop the conceptual framework or make final scientific decisions.Source: A AI Use Statement · p. 13

B Ethics Considerations

  • The benchmark uses publicly available sources and expert-authored tasks, collects no personal data, and does not involve human subjects as research participants.Executions run in isolated offline environments; the authors document artifact provenance and licenses and discuss domain-representation limitations.Source: B Ethics Considerations · p. 13

C Benchmark Design and Construction

  • Tasks specify goals, constraints, expected files, and resource limits, while hidden inputs are mounted by the checker at evaluation time.Reference packages include solutions, schemas, weighted checks, and rubrics; external sources are used when their licenses permit redistribution and derivative use.Source: C Benchmark Design and Construction · p. 13

C.1 Task and domain distributions

The section describes the benchmark’s domain and workflow coverage, expert construction and verification, representative tasks, trap injection, scoring, and robustness analyses. Its examples and design emphasize evaluating scientific reasoning alongside executable and written outputs.

C.1 Task and domain distributions

  • The benchmark spans multiple task types in most domains, avoiding strong concentration in any single domain–type combination.The human-written rubrics are authored by task-construction experts after reference solutions are completed, and reference reports must score above 90% for release.Source: §C.1 · p. 13

C.2 Expert construction and verification

  • 156 domain experts constructed and verified tasks through separate writer, independent reviewer, and auditor roles.Reviewers and auditors targeted approximately two and one hours per task, respectively; agreement below an 85% quality threshold triggered further review or calibration.Source: §C.2 · pp. 13–14

C.3 Representative tasks

  • Representative tasks span survival analysis, protein-conformation workflows, code implementation, evidence reconciliation, hypothesis design, and regression diagnosis.The examples illustrate why evaluation needs to assess both execution and scientific presentation of evidence and conclusions.Source: §C.3 · p. 14

C.4 Details of Trap Injection

  • Trap tasks place a locally plausible shortcut or decoy in reach, while credit requires completing the scientific procedure.Where appropriate, input mutation defeats shortcuts mechanically; otherwise, withheld references and hidden reruns are used.Source: §C.4 · p. 14

C.5 Task type weights

  • Task-level scores combine programmatic and LLM-judge scores using task-type-specific weights fixed before model evaluation.The paper reports component scores alongside aggregates; domain averages use task-level scores without reweighting domains.Source: §C.5 · p. 14

D Evaluation Robustness and Sensitivity

D.1 Robustness across Agent Harnesses

Measured performance depends on harness configuration, though effects differ by model and the harness comparisons are descriptive rather than paired estimates. On paired tasks, higher inference effort improves aggregate scores but not every model or evaluation component.

D.1 Robustness across Agent Harnesses

  • Harness choice materially affects measured performance, but the size and direction vary by model; the comparisons are descriptive because each harness uses its own clean tasks.GPT-6 Astra scores 78.9 under claude-code versus 79.5 under codex, while GPT-5.6 Sol scores 14.3 points higher under codex.Source: §D.1 · p. 14

D.2 Sensitivity to Inference Effort

  • Across nine models on paired tasks, Final rises from 61.8 to 65.1 and Pass@0.95 from 1% to 6% at max versus high effort.The gains are uneven: GPT-6 Astra remains at 78.9 Final, while GPT-5.6 Sol decreases from 58.0 to 55.6.Source: §D.2 · p. 14
  • Higher inference effort improves aggregate performance on this subset but does not reliably improve every model or evaluation component.Source: §D.2 · p. 14

E Experimental Details and Cost

The section describes the evaluation runner, scoring components, and reported performance-resource trade-offs. It cautions that observed cost and interaction measures are specific to the harness and do not directly measure reasoning quality.

E Experimental Details and Cost

  • The open-source harbor runner builds each task’s Docker image, invokes the agent, collects files, runs checks and the rubric-based judge, and records an aggregate reward.Source: E Experimental Details and Cost · pp. 14–15
  • Table 4 catalogs recurring trap forms and representative tasks, while Table 5 lists programmatic and LLM-judge weights by task type.Source: Table 4 · p. 15, Table 5 · p. 15
  • GPT-6 Astra records the highest combined score (80.9) at $20.79 per run and 22.1 minutes on average, while Claude Fable 5 scores 67.3 at $7.67 per run.GLM-5.3 averages $57.96 and 102.4 turns per run yet scores 55.9; unit token prices alone do not characterize workflow cost.Source: E Experimental Details and Cost · p. 15
  • Turns are only a partial view of interaction efficiency, not a direct measure of reasoning quality or wasted steps.Tokens per turn, tool use, and provider pricing differ, and the table does not isolate those factors.Source: E Experimental Details and Cost · p. 15

F Results across Task Types and Rubric Dimensions

Agents show distinct strengths across scientific task types, and strong presentation does not necessarily correspond to strong scientific grounding or correctness. Decomposing rubric scores reveals differences that aggregate scores can conceal.

Performance across task types

  • Comparable overall scores mask distinct task profiles: some models excel at Code Generation and Tool Orchestration, while others perform relatively well on Literature Synthesis.GLM-5.3 scores 74 on Literature Synthesis but 41 on Code Generation; task-type performance also varies in how strongly it separates models.Source: F Results across Task Types and Rubric… · p. 16

Performance across rubric dimensions

  • Communication is every model’s highest-scoring rubric dimension, with 81–93% criterion pass rates, while Source Grounding and Scientific Correctness are typically among the weakest.For GPT-6 Astra, Communication reaches 93% and Execution & Validation 90%, compared with 82% for Source Grounding and 83% for Scientific Correctness.Source: F Results across Task Types and Rubric… · pp. 15–16
  • Claude Fable 5 and Qwen3.8 Max both score 74 overall, yet Qwen3.8 Max scores higher on Execution & Validation (82 vs. 78) and Communication (89 vs. 86).The decomposition distinguishes presentation quality from scientific support, which the aggregate LLM-judge score compresses.Source: F Results across Task Types and Rubric… · p. 16

G Trajectory-Level Behavior Patterns

Trajectory analysis shows that solution quality can emerge at different times and regress, so endpoint scores alone do not capture how reliably agents execute. The case study illustrates that executable correctness and scientific explanation can diverge, making the two evaluation axes complementary.

G Trajectory-Level Behavior Patterns

  • Individual runs can improve sharply late, remain nearly unchanged, or temporarily lose and later recover quality, so similar endpoint scores can arise through different paths.Even on the same task, models differ in when executable progress and LLM-judge improvements occur.Source: G Trajectory-Level Behavior Patterns · pp. 16–17
  • GPT-5.6 Sol and GPT-6 Astra have the highest early-completion rates (51.7% and 47.8%), but GPT-5.6 Sol has higher dip (27.0%) and regret (16.0%) than GPT-6 Astra (18.3% and 7.9%).Early completion and preservation of gains are distinct aspects of execution behavior.Source: G Trajectory-Level Behavior Patterns · pp. 17–18
  • Regression is concentrated in scientific content: Scientific Correctness often has the highest dip and regret rates, whereas Communication is consistently more stable.Programmatic criteria can also regress despite often being satisfied early.Source: G Trajectory-Level Behavior Patterns · p. 17
  • End-to-End Analysis has the highest early-completion rate (36.2%) and dip rate (12.2%), while Code Generation and Error Diagnosis rarely reach near-final quality early (8.6% and 3.1%).Their lower dip and regret rates do not necessarily indicate easier or more stable execution, because successful criteria are established later.Source: G Trajectory-Level Behavior Patterns · pp. 17–18, Table 9 · p. 18
  • Trajectory measures add information beyond final scores by showing when quality emerges and whether agents preserve it.These patterns vary across models, evaluation dimensions, and scientific task types.Source: G Trajectory-Level Behavior Patterns · p. 18

H Case Study: When execution and communication decouple

  • The benchmark scores executable artifacts and written explanations on complementary axes, then blends them as 0.55 · prog + 0.45 · judge.The programmatic checker re-executes artifacts and tests perturbed inputs; the judge evaluates reports against a rubric without re-checking numbers.Source: H Case Study: When execution and… · p. 18

H.1 Case 1: one axis alone is biased

  • In the tool-orchestration case, the required pipeline includes spectral decomposition, tree-DP leaf-label selection, and evaluation on the original graph.The task provides intermediate hand-offs and an answer file, with separate programmatic checks and rubric items.Source: §H.1 · p. 19
  • GPT-5.6 Sol earns a judge score of 1.00 but a programmatic score of 0.10, passing only 13 of 129 programmatic-check weight.Its result-bearing checks fail, and the same near-1.0-judge / near-0.1-programmatic split recurs for several other models on this task.Source: §H.1 · p. 19

H.2 Case 2: different models fail on different axes

This section shows how separate programmatic and judge axes expose different model failure modes, then makes a scientific task package concrete through a fixture-analysis task. It covers that task’s precision-gated pipeline-selection policy, required analysis artifacts, and offline environment.

H.2 Case 2: different models fail on different axes

  • GPT-5.6 Sol scored 1.00 on the judge but 0.10 programmatically for cs-expansion-tool-orchestration-02, passing only structural glue checks while result-bearing checks failed.The failed checks covered labeled sets and Ψ values, tree-DP selection, hierarchical bisection, and the perturbed-input rerun.Source: Table 10 · p. 19
  • DeepSeek-V4-Pro scored programmatic 1.00 versus Claude Opus 5’s 0.11, while judge scores reversed at 0.43 versus 1.00, exposing opposite execution and communication failures.A programmatic-only ranking favors DeepSeek-V4-Pro, whereas an LLM-judge-only ranking favors Claude Opus 5.Source: Table 11 · p. 19, §H.2 · p. 20
  • DeepSeek-V4-Pro reproduced the reference computation but barely explained it, whereas Claude Opus 5 gave the correct phase label without reproducible numerical evidence.DeepSeek-V4-Pro passed 10 of 23 rubric items and scored 0.43 on the judge; Claude’s margin was roughly 20× too large and its intermediate tables failed to reproduce.Source: §H.2 · p. 20, Table 12 · p. 20

I An Example of Task Package

  • The appendix presents an end-to-end bioinformatics task package for selecting a metagenomic assembly-and-binning pipeline robust across four contig-level stress axes.Its policy is precision-gated maximin selection grounded in the mounted MAG-E paper.Source: I An Example of Task Package · p. 24

I.1 Instruction (instruction.md, verbatim)

  • The instruction requires a precision-qualified pipeline with robust recovery across high coverage-distance, high composition-distance, prophage, and shared-contig axes, while treating its adaptations as task-specific rather than universal recommendations.The task also requires explaining why the whole-genome F-score and truth-precision leaders are not selected.Source: §I.1 · p. 24
  • Pipelines qualify only if every evaluable sample meets the 0.90 truth-precision floor; eligible pipelines are ranked by their weakest mean recovery axis, with specified tie-breaks.The limiting axis is the lowest of four recovery means: coverage outlier, composition outlier, prophage, or shared.Source: §I.1 · p. 25
  • The task separately requests global whole-genome F-score and truth-precision leaders, plus each ineligible pipeline and its failing sample.The report must explain alternatives’ limiting-axis failures and distinguish the fixture policy from paper conclusions, not present its selected pipeline as a universal recommendation.Source: §I.1 · pp. 25–26
  • Required deliverables include executable analysis, CSV summaries, structured JSON, and a scientific report explaining source-derived choices, results, validation, and limitations.The task also requires confirming that regenerated artifacts agree on the selected pipeline and limiting axis and that precision gating and source-derived handling are applied.Source: §I.1 · pp. 25–26

I.2 Environment

  • The task runs offline in a frozen Docker image with one source paper and fourteen data files, but no reference artifact or checker input mounted for the agent.Source: §I.2 · p. 26

I.3 Reference solution

The reference solution selects MAGE-83 based on its composition-outlier limitation, rather than whole-genome F-score or truth-precision leadership. The grading design pairs a weighted rubric for textual artifacts with a programmatic checker that tests scientific outputs, including on relabeled held-out data.

I.3 Reference solution

  • MAGE-83 is selected for the composition-outlier axis, not as the whole-genome F-score leader MAGE-47 or truth-precision leader MAGE-31.The reference answer identifies MAGE-83 as the decision and names composition-outlier as its limiting axis.Source: §I.3 · p. 26
  • 0.9875 is MAGE-83’s mean truth precision, while 0.2862 is its mean composition-outlier recall and robust-recovery minimum.Its margin over the runner-up is 0.2375.Source: §I.3 · p. 26
  • The reference solution is a deterministic reduced synthetic community fixture, not a full replication of the paper’s 4,800 tasks.Source: §I.3 · p. 26

I.4 Grading, part 1 — the LLM rubric (tests/rubric.json)

  • The LLM rubric scores textual artifacts through observable pass/fail criteria, while exact identifiers and numeric values are assigned to programmatic checks.The complete rubric has 20 criteria across four categories and a total weight of 70.Source: §I.4 · p. 27

I.5 Grading, part 2 — the programmatic checker (tests/check_programmatic.py)

  • The checker reruns agent analysis on held-out data with relabeled identifiers, and this replay is the largest single check at 10 of 30 weight.This makes hard-coding the reference pipeline identifier insufficient.Source: §I.5 · p. 27
Loading 2610.32441v1…