PaperlayerView exact revision on arXiv

Source-linked AI summary

FrontierChallenge: Evaluating Scientific Workflow Completion

Liangcai Su, Zhaopeng Feng, Zhuo Chen, Zhen Zhang, Xiang Lin, Ruilin Li, Handuo Zhang, Ning Wang, Kailong Wen, Yueqi Guo, Feng Xing, Yiling Guo, Chenxiong Qian, Simon Shaolei Du, Lidong Bing, Xinyu Wang

arXiv:2608.24979v1cs.AIcs.CLcs.SE

TL;DR

Many benchmarks emphasize final answers, isolated programs, or single-discipline workflows, leaving limited evidence about agents’ ability to complete heterogeneous scientific work with multiple required deliverables. FrontierChallenge evaluates complete artifact bundles across 97 released workflows in six domains; across models and scaffolds, Pass Rate ranged from 3.1% to 20.6% despite Avg. Scores of 67.5 to 87.9.

  • Problem

    Existing benchmarks often evaluate final answers, interaction traces, single programs, or single-discipline workflows, leaving limited characterization of agents’ ability to complete heterogeneous scientific work with multiple required deliverables.

    Source: §1 · p. 1

  • Method

    FrontierChallenge evaluates complete submitted artifact bundles for fixed-input workflows, releasing and testing 97 of 300 workflows across six scientific domains.

    Source: §1 · p. 2

  • Results

    Pass Rate ranged from 3.1% to 20.6% despite Avg. Scores of 67.5 to 87.9 across evaluated models and scaffolds, indicating a gap between partial progress and complete delivery.

    Source: §5 · p. 9

  • Takeaways & Limitations

    More reliable scientific agents will require explicit contract tracking, cross-artifact validation, and evidence-based completion checks.

    Source: §5 · p. 10

    The six domains are descriptive slices, not probability samples, and performance differences should not be interpreted as intrinsic rankings of disciplinary difficulty.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Scientific agents increasingly analyze data, execute code, and produce research artifacts, yet most benchmarks emphasize final answers, isolated programs, or a single domain. We introduce FrontierChallenge, a cross-domain benchmark comprising 300 end-to-end scientific workflows. In this paper, we release and evaluate 97 of these tasks, spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment. Each task provides fixed inputs and specifies a bundle of required scientific deliverables. We evaluate twelve frontier models with three agent scaffolds. Pass Rate measures the fraction of tasks satisfying the full-completion criterion, while Avg. Score captures partial progress. Each of the best-performing configurations completed only 20 of the 97 released tasks, yielding a Pass Rate of 20.6%. Partial progress translated especially poorly into complete delivery in analytical chemistry and electrochemistry/environment: Avg. Scores reached 87.6 and 94.9, but the highest Pass Rates were only 4% and 0%. Among non-passing Claude Code trajectories, 75.5% still ended with language claiming completion. These findings show that neither high partial scores nor confident claims of completion reliably indicate that a scientific task has been fully delivered, highlighting the need to evaluate end-to-end workflow execution and the completeness of scientific deliverables together.

1 Introduction

FrontierChallenge evaluates whether agents can complete specified scientific workflows and deliver the full set of required artifacts, not merely produce plausible answers. The study finds a substantial gap between partial progress and complete task delivery.

Introduction

  • Existing benchmarks often assess a final answer, interaction trace, single program, or one-discipline workflow, leaving heterogeneous multi-stage work and deliverable completeness insufficiently characterized.Scientific task success can require inspecting varied inputs, running and validating analyses, and delivering consistent code, tables, figures, and prose.Source: §1 · p. 1
  • FrontierChallenge tests execution of a specified workflow with fixed objectives, inputs, and required outputs, rather than autonomous research agenda-setting.The intended handoff is inspectable, reproducible, and mutually consistent across analyses and outputs.Source: §1 · pp. 1–2
  • The benchmark comprises 300 workflows across six domains; 97 are released for evaluation, with 203 retained as an internal held-out set.Tasks use fixed inputs and heterogeneous deliverable contracts, and evaluation covers the complete submitted artifact bundle, including code, analysis, visualizations, and evidence-grounded reports.Source: §1 · p. 2
  • Twelve frontier models and three agent scaffolds are evaluated using Pass Rate for full contract completion and Avg. Score for partial progress.The study also compares machine-observable contract-breach signatures across domains and analyzes final-status and error behavior in 970 Claude Code trajectories.Source: §1 · p. 2
  • The best configurations completed 20 of 97 tasks (20.6% Pass Rate), while the highest Avg. Score was 87.9.The gap was especially pronounced in analytical chemistry (Avg. Score 87.6; highest Pass Rate 4%) and electrochemistry/environment (Avg. Score 94.9; highest Pass Rate 0%).Source: §1 · p. 2

2 Related Work

Existing benchmarks assess general agent capability, scientific knowledge and analysis, or end-to-end research tasks, but FrontierChallenge focuses on completing fixed-input, cross-domain scientific workflows with multiple artifacts and task-specific executable evaluation.

General capability and long-horizon agent benchmarks

  • General agent benchmarks span expert knowledge, tool use, computer interaction, software engineering, long-horizon professional work, and frontier tasks, but do not center cross-domain scientific workflows.Examples include HLE (Phan et al., 2025), AgentBench, GAIA, OSWorld, and SWE-bench (Liu et al., 2024; Mialon et al., 2024; Xie et al., 2024), ALE and Frontier-Bench (Sun et al., 2026b; Harbor Framework, 2026).Source: §2 · p. 2

Scientific knowledge and data analysis

  • Scientific benchmarks assess biological knowledge, data-driven open-ended or verifiable problems, expert-recognized analytical decisions, and executable analyses derived from literature.Examples include LAB-Bench (Laurent et al., 2024), BixBench, BioMysteryBench, and CompBioBench (Mitchener et al., 2025; Anthropic, 2026; Nair et al., 2026), BLADE, and ScienceAgentBench (Gu et al., 2024; Chen et al., 2025).Source: §2 · pp. 2–3
  • Many scientific knowledge and analysis benchmarks evaluate an answer, decision set, or single self-contained program rather than a heterogeneous set of deliverables.These benchmarks increase scientific realism, but their evaluated outputs often remain narrower than a multi-artifact workflow.Source: §2 · p. 3

End-to-end scientific workflows

  • FrontierChallenge complements end-to-end research benchmarks by evaluating reliable completion from fixed objectives and inputs across domains, with multi-artifact outputs and task-specific executable evaluation.Related benchmarks address computational reproduction, full research replication, scientific software use, interactive research environments, bioinformatics and biomedicine, paper-level methods, or cross-domain research tasks.Source: §2 · p. 3

3 FrontierChallenge

FrontierChallenge collects realistic, end-to-end scientific workflows and packages each with fixed inputs, tools, required deliverables, and an evaluation procedure. The paper releases 97 of 300 curated workflows across six reporting domains, while treating domain composition as descriptive rather than representative.

3.1 Task Collection, Curation, and Packaging

  • Tasks were curated for professional representativeness, end-to-end complexity, scientific diversity, and verifiability, excluding single-step, subjective, or irreproducibly scored tasks.The complete collection contains 300 scientific workflows; 97 are released and evaluated in this paper.Source: §3.1 · pp. 3–4
  • Each task package aligns the objective, fixed inputs, available tools, deliverable contract, and evaluation procedure, making completion depend on the consistent artifact bundle rather than a plausible final answer.Deliverables can include reports, structured tables, figures, executable analysis code, simulation products, or multiple mutually consistent artifacts.Source: §3.1 · p. 4

3.2 Dataset Statistics

  • The 97 released tasks were randomly selected from workflows whose official evaluation requires no GPU; 74 are Hard, 23 Medium, and 203 remain held out.The held-out set includes workflows whose official evaluation requires GPUs; none is included in the reported experiments.Source: §3.2 · p. 5
  • The release spans six domains—20 quantum chemistry, 16 molecular dynamics, 22 materials characterization, 23 analytical chemistry, 10 life science, and 6 electrochemistry tasks—and 21 workflow families.Deliverables include scientific reports, structured data, figures, executable code, and simulation products.Source: §3.2 · p. 5
  • Domain slices describe this release, not probability samples or balanced field coverage, so cross-domain performance differences do not establish intrinsic disciplinary difficulty rankings.Source: §3.2 · p. 5

4 Experiments

The experiments are organized around three research questions.

4 Experiments

  • The experiments address three research questions.Source: §4 · p. 5

1. RQ1: How reliably can current frontier models complete specified scientific workflows? · 2. RQ2: How does scientific workflow performance vary across domains?

Strict full completion was uncommon even when average scores were high, and performance varied substantially by domain. Runtime, resource use, and failure analyses further show why aggregate scores or confident completion claims do not establish a complete handoff.

1. RQ1: How reliably can current frontier models complete specified scientific workflows? — 4.1 Agent Scaffolds and Models; 4.2 Task-specific Evaluation and Primary Metrics

  • The evaluation used twelve frontier models across three agent scaffolds, with all models given the same 97 task objectives and task-visible inputs.Task-specific graders checked required files, numerical results, formats, figures, code execution, and cross-artifact consistency; the primary Pass Rate required a score of at least 99.9, while Avg. Score measured partial completion and was a descriptive aggregate across heterogeneous rubrics.Source: §4.1 · p. 5, §4.2 · pp. 5–6

1. RQ1: How reliably can current frontier models complete specified scientific workflows? — 4.3 Overall performance

  • Pass Rates ranged from 3.1% to 20.6%; Codex with GPT-5.6 Sol had the highest Avg. Score (87.9) and shared the highest Pass Rate (20.6%) with Claude Code using Grok 4.6 (Avg. Score 86.6).Eight configurations scored above 80 on Avg. Score, but none completed more than 20.6% of tasks. Frontier Agent (Agent Team) scored 74.5 and 12.4% Pass Rate versus 71.8 and 10.3% for Claude Code, a descriptive comparison that does not isolate a scaffold effect.Source: §4.3 · pp. 6–7, Table 2 · p. 6, Figure 2 · p. 6

2. RQ2: How does scientific workflow performance vary across domains? — 4.4 Domain-level performance

  • In quantum chemistry, Claude Code with Grok 4.6 led Pass Rate at 60%; in molecular dynamics, GPT-5.6 Sol, GPT-5.6 Terra (max), and Grok 4.6 each reached 38%.In molecular dynamics, Terra had the highest Avg. Score (93.7). In quantum chemistry, Codex with GPT-5.6 Sol and Claude Code with Claude Opus 5 followed Grok 4.6 at 55% Pass Rate.Source: §4.4 · p. 7, Figure 3 · p. 7
  • Materials characterization reached an Avg. Score of 88.1 but no Pass Rate above 9%; analytical chemistry peaked at 87.6 Avg. Score but only DeepSeek V4 Pro-0813 completed a task, at 4% Pass Rate.Electrochemistry/environment reached a maximum Avg. Score of 94.9 but had 0% Pass Rate for every configuration; in life science, GPT-5.6 Sol led Avg. Score at 76.8 while Kimi K3 led Pass Rate at 20%.Source: §4.4 · p. 7, Figure 3 · p. 7
  • Domain profiles diverged: GPT-5.6 Sol led Avg. Score in four domains, while Terra exceeded Sol in molecular-dynamics and electrochemistry/environment Avg. Score.GPT-5.6 Sol led Avg. Score in quantum chemistry, materials characterization, analytical chemistry, and life science. Terra scored 75.5 versus 91.7 and completed 35% versus 55% of quantum-chemistry tasks, but exceeded Sol in molecular-dynamics Avg. Score (93.7 versus 88.9) and electrochemistry/environment Avg. Score (94.5 versus 91.8). Frontier Agent (Agent Team) had a 40% quantum-chemistry Pass Rate, 19% in molecular dynamics, and completed no tasks in materials characterization, analytical chemistry, or electrochemistry/environment; Gemini 3.7 Flash completed 25% in quantum chemistry and molecular dynamics and none in those other three domains. Aggregate ranking did not capture these domain-specific strengths and weaknesses.Source: §4.4 · pp. 7–8

2. RQ2: How does scientific workflow performance vary across domains? — 4.5 Resource use and execution time

  • Reported input use ranged from 2.183 million tokens per task for Grok 4.6 with Claude Code to 13.730 million for Apodex 1.1 with Claude Code; mean execution times ranged from 21.8 to 112.8 minutes.Frontier Agent (Agent Team) had the longest mean runtime (112.8 minutes), median (54.9), and 90th percentile (272.2). Token reporting conventions preclude hardware-normalized efficiency interpretation, and runtime summaries describe observed times rather than isolated model speed.Source: §4.5 · p. 8, Figure 4 · p. 8

2. RQ2: How does scientific workflow performance vary across domains? — 4.6 Failure Mode Analysis

  • Among non-passing Claude Code trajectories, 75.5% contained completion language, while tool errors occurred in 80.7% of non-passing and 94.2% of passing runs.Judge-assessed artifact shortfalls appeared in 97% of non-passing materials-characterization submissions, 95% of analytical-chemistry submissions, and 85% of electrochemistry/environment submissions, versus 43% in quantum chemistry; deterministic Grader diagnostics were more common in life science (50%) and quantum chemistry (40%). Only 13 of 849 non-passing runs explicitly said work was ongoing, while completion language occurred in 90.1% of passing runs. Median tool-error rates were 4.4 versus 5.6 per 100 results for non-passing and passing runs, respectively; these descriptive analyses do not assign unique causes, and an error alone did not identify eventual contract breach.Source: §4.6 · pp. 8–9, Figure 5 · p. 9

5 Conclusion

FrontierChallenge finds that complete, contract-level delivery remains an unresolved capability: high partial scores and confident completion language do not reliably indicate that scientific workflows are fully delivered. The conclusions are bounded by the released tasks, evaluated configurations, and other study limitations.

5 Conclusion

  • Pass Rate ranged from 3.1% to 20.6% despite Avg. Scores of 67.5 to 87.9; GPT-5.6 Sol with Codex led Avg. Score and shared the top Pass Rate with Grok 4.6 using Claude Code.The study evaluated 97 released workflows from a 300-workflow benchmark across six scientific domains.Source: §5 · p. 9
  • Partial progress translated especially poorly into full delivery in analytical chemistry and electrochemistry/environment, and 75.5% of non-passing Claude Code trajectories ended with completion language.The authors caution that conclusions are limited to the released task set, evaluated configurations, Claude Code trajectory scope, provider-specific resource accounting, and single runs; tool errors were frequent in both passing and non-passing runs.Source: §5 · p. 9
  • More reliable scientific agents will require explicit contract tracking, cross-artifact validation, and evidence-based completion checks.The study frames complete, contract-level delivery as a distinct and unresolved capability.Source: §5 · pp. 9–10

Appendix · B Illustrative Task Case

The appendix’s three illustrative cases show the benchmark unit as an objective, frozen inputs, required workflow, and artifact contract, focusing on what systems must do and hand off rather than task answers.

B Illustrative Task Case

B.1 Cell-migration wound-healing assay

The wound-healing case tests whether an agent can turn 18 marked microscopy images into a reproducible, statistically analyzed handoff. The inputs cover two groups, three time points, and three biological replicates; no measured results or conclusions are provided.

B.1 Cell-migration wound-healing assay

  • The case supplies 18 bright-field images spanning control and experimental groups, 0-, 12-, and 24-hour time points, and three biological replicates, with unmigrated-region boundaries marked in white.Figure 6 shows one replicate per group across the three time points; the other 12 images provide the remaining replicates.Source: §B.1 · p. 12, Figure 6 · p. 12
  • The workflow must segment and inspect all images, match time points by group and replicate, calculate migration rates relative to 0 hours, and test between-group differences at both follow-up times.The checkable handoff includes a reproducible script, image-level and group-statistics tables, two labeled summary plots, and a methodological report; no measured results or conclusions are shown.Source: §B.1 · p. 13

B.2 TLC monitoring of a Suzuki coupling · B.3 Reaction-calorimetry safety assessment

The sections describe two end-to-end workflow tasks: reconstructing Suzuki-coupling progress from a TLC plate and assessing reaction-calorimetry safety through an auditable analysis handoff.

B.2 TLC monitoring of a Suzuki coupling

  • The TLC task uses an eight-lane UV254 plate with standards, a co-spot, and samples from 0 to 120 minutes, requiring lane-specific Rf calculations for tilted geometry.The frozen input also supplies the plate map, reference Rf windows, response factors, sampling metadata, image geometry, integration settings, and endpoint rule.Source: §B.2 · p. 13
  • The TLC analysis must detect and integrate spots, assign compounds, calculate corrected lane compositions, reconstruct reaction progress, and recommend an endpoint under the supplied rule.The input image shows no derived spot boundaries, assignments, corrected intensities, compositions, or endpoint decision.Source: §B.2 · p. 13
  • The TLC handoff requires a reproducible script, five linked data tables, an annotated plate, lane profiles, and a report.The required tables cover plate QC, spots, lane composition, reaction progress, and the endpoint.Source: §B.2 · p. 13

B.3 Reaction-calorimetry safety assessment

  • The calorimetry task supplies electrical calibration pulses, blank-dosing experiments, reaction runs, protocol metadata, and screening rules for a process-safety assessment.Figure 8 includes representative raw traces without baseline correction, integration, derived metrics, or protocol decisions.Source: §B.3 · pp. 13–14
  • The calorimetry analysis must calibrate and correct heat flow, derive run-level metrics, summarize replicates by protocol, and apply the declared screening criteria.The required handoff links calibration, correction, time-series, run-metric, protocol-summary, and decision tables, with diagnostic figures and a report.Source: §B.3 · p. 14
  • Figure 8’s schematic records the required path from calibration evidence to a complete handoff, while its traces provide raw-signal context rather than an outcome.The figure is an answer-free view of the case.Source: Figure 8 · p. 14

Related papers

Loading 2608.24979v1…