PaperlayerView exact revision on arXiv

Source-linked AI summary

From Research Goals to Scientific Closure: Evaluating Agents Through Evidence-Supported Physics Research

Yijia Mao, Song Dai, Tao Chen, Xiangyu Lu, Linqian Zeng, Chenxi Hu, Sitong Zhou, Dongyang He, Xiaodan Mao, Linxuan Zhang, Tong Zhu, Yafu Li, Fangchen Yu, Peng Ye, Feng He, Yu Cheng, Yun Luo

arXiv:2610.21767v1cs.AI

TL;DR

Existing scientific AI evaluations rarely establish whether an investigation as a whole resolves its specified question with well-supported evidence. PhysClosureBench evaluates goal-specified physics investigations using measures of local accomplishment and Research Closure. Across models, higher accomplishment generally accompanies higher closure, but many high-scoring attempts still fall short of complete closure.

  • Problem

    Existing benchmarks rarely assess whether an investigation as a whole establishes a well-supported answer to its specified research question.

    Source: §1 · p. 2

  • Method

    PhysClosureBench evaluates goal-specified physics investigations with a 20-criterion Atomic Score for local accomplishment and RC0–RC4 for whether evidence, interpretation, and scope resolve the question.

    Source: §1 · p. 2, §2 · p. 5

  • Results

    Higher Atomic Scores generally accompany higher closure, yet 36.2% of attempts scoring at least 8/10 do not reach RC4.

    Source: §5.2 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    Local scientific accomplishment and complete Research Closure are distinct outcomes, and case-level failures include misinterpreted results, unsupported mechanisms, and unresolved applicability limits.

    Source: §6 · p. 9

    Evaluation is limited to evidence in the agent's submission; evaluator checks cannot replace missing analyses or interpretations.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

Can AI agents organize scientific investigations that lead to well-supported conclusions? To evaluate this capability, we introduce PhysClosureBench, a benchmark of 50 expert-verified, goal-specified research tasks spanning strong-field, attosecond, and neighboring physics areas. Each task specifies the research objective and essential scientific constraints, while leaving the agent to organize the investigation and determine how to generate and validate the evidence needed to answer the question. A 20-criterion Atomic Score measures local scientific accomplishment, while Research Closure (RC0--RC4) assesses how fully the investigation resolves the specified research question based on whether its indispensable evidence requirements and applicable scope conditions are satisfied. We evaluate seven frontier LLMs, where Claude-Opus-5 achieves the highest mean Atomic Score (6.29/10) but reaches RC4 in only 32.7% of attempts. Across the benchmark, 36.2% of attempts scoring at least 8/10 still fall short of RC4, while all seven models reach RC4 on substantially fewer tasks in all three runs than in at least one. Case-level analysis further illustrates how misinterpretation of generated results and unresolved physical applicability limits can lead to incomplete closure. Together, these results reveal a persistent gap between AI agents' local scientific accomplishments and their ability to achieve complete research closure.

1 Introduction

Scientific research requires agents to organize investigations that connect questions to well-supported conclusions, not merely produce plausible results. The paper introduces PhysClosureBench and a two-level evaluation, finding that strong local accomplishment does not ensure complete or reliable Research Closure.

Motivation

  • Scientific research requires connecting a specified question to evidence, interpretation, and validation sufficient for a well-supported conclusion.Useful assistance with literature synthesis, programming, or reproducing prior work does not itself show that an agent can independently organize such an investigation.Source: §1 · p. 2

Research gap

  • Existing benchmarks rarely assess whether an investigation as a whole establishes a well-supported answer to its specified research question.Correct results or reproduced findings can leave the broader investigation incomplete.Source: §1 · p. 2

Benchmark and evaluation

  • PhysClosureBench evaluates goal-specified physics tasks with an Atomic Score for local accomplishment and Research Closure for whether evidence, interpretation, and scope resolve the question.The Atomic Score covers 20 task-specific criteria across formulation, output, identification, mechanism, and validation/scope.Source: §1 · p. 2

Main findings

  • Claude-Opus-5 has the highest mean Atomic Score at 6.29/10 and RC4 rate at 32.7%, while Gemini-3.1-Pro has the lowest at 2.85/10 and 0.7%, respectively.For Claude-Opus-5, only 18.0% of tasks reach RC4 in all three independent runs.Source: §1 · p. 2
  • Substantial local accomplishment does not ensure complete closure, and achieving closure in one run does not guarantee it in other independent runs.The paper frames these as gaps between local accomplishment and closure, and between achieving closure and doing so reliably.Source: §1 · p. 3

2 Related Work

Scientific AI benchmarks assess capabilities ranging from expert question answering to research workflows and executable artifacts. PhysClosureBench makes closure of a goal-specified investigation an explicit target.

Prior benchmarks

  • Existing benchmarks cover expert-level reasoning, research subproblems, broader research workflows, scientific code, engineering tasks, and research reproduction.These include benchmarks assessing granular checkpoints or rubric-based scientific problems, as well as agents exploring experiment design, execution, and iterative analysis.Source: §2 · p. 3

Positioning

  • PhysClosureBench assesses whether an agent's evidence, interpretation, and validation suffice to resolve the original question, including consistency across independent attempts.Its target is scientific closure at the level of a goal-specified, agent-directed investigation.Source: §2 · p. 3

3 Benchmark and Evaluation Design

The benchmark leaves investigation strategy open while specifying each physics task's objective and constraints. Its task verification and paired evaluation measures are designed to assess both local scientific work and whether task-specific requirements for a complete answer are met.

3.1 Task Design and Agent-Directed Setting

  • PhysClosureBench contains 50 goal-specified tasks spanning strong-field and attosecond physics, solid-state and condensed-matter physics, quantum optics, and neighboring quantum light–matter regimes.Tasks specify the objective, physical setting, parameters, constraints, and computational resources, while leaving agents to choose implementation, observables, comparisons, controls, and validation.Source: §3.1 · pp. 3–4
  • Tasks run in isolated, network-disabled environments with task-specific CPU and wall-time limits.Agents allocate the available computational effort across planning, exploration, analysis, and validation.Source: §3.1 · p. 4

3.2 Task Reliability and Verification

  • Experts reviewed task construction and rubrics, and source results for every task were manually reproduced under the specified conditions before inclusion.Observed reproduction costs informed resource limits, with additional allowance for exploration and unsuccessful intermediate attempts.Source: §3.2 · p. 4

3.3 Evaluation Framework

  • Together, the measures show where scientific progress occurs and whether the investigation meets requirements for a complete answer.Substantial progress on individual components does not necessarily imply complete closure, and different valid pathways can satisfy task-specific requirements.Source: §3.3 · p. 5, §2 · p. 5
  • The Atomic Score measures local work across 20 task-specific half-point criteria, while RC0–RC4 assess whether task-specific requirements for a complete answer are met.Closure levels depend on the scientific role of unresolved requirements in supporting the main conclusion, not on a fixed workflow or Atomic Score threshold.Source: §3.3 · p. 5, §2 · p. 5

4 Experimental Setup

Seven frontier models completed three independent runs on each of 50 tasks, with submissions graded on the evidence they produced. The evaluation reports local accomplishment, closure, and closure consistency, while expert review provides a comparison with automated grading.

4.1 Models and Evaluation Procedure

  • Seven frontier models each ran three independent attempts on all 50 tasks, yielding 1,050 scored attempts.Each attempt used a separate workspace and an isolated, network-disabled computational environment.Source: §4.1 · p. 5
  • Automated grading used a staged protocol with targeted review when a residual disagreement could affect score, closure, or attempt validity.Five physics-trained experts reviewed 100 evaluation cards; human and automated grades agreed on 81.8% of selected atomic-criterion judgments and 75.9% of closure-linked judgments.Source: §4.1 · pp. 5–6
  • Grading was restricted to evidence produced by the agent, and evaluator checks could not substitute for missing analyses or interpretations.This limits what the evaluation can credit to work represented in the submission.Source: §4.1 · p. 6

4.2 Analysis Measures

  • Model-level reporting uses mean Atomic Score and the distribution of Research Closure levels, including the overall RC4 rate.Functional completion is averaged across attempts for F/O/I/M; Validation/Scope is analyzed separately because applicability depends more directly on the method, evidence, and conclusion.Source: §4.2 · p. 6
  • Closure consistency is measured by the fraction of tasks reaching RC4 in at least one run and in all three runs.Figure 3 presents mean Atomic Score and F/O/I/M functional completion alongside the distribution of closure levels across attempts.Source: §4.2 · p. 6, Figure 3 · p. 6

5 Results and Discussion

Across models, local scientific accomplishment generally tracks Research Closure, but strong scores and successful individual runs do not guarantee complete or repeatable closure. The results show distinct gaps in interpretation, physical explanation, and applicability validation.

5.1 Scientific Accomplishment and Research Closure

  • Claude-Opus-5 led mean Atomic Score at 6.29/10, yet reached RC4 in only 32.7% of attempts; GPT-5.6-Sol and Kimi-K3 reached 22.7% and 19.3%.Research Closure broadly followed the Atomic Score ranking, but complete closure remained uncommon among the leading models.Source: §5.1 · p. 7

5.3 Evidence Pathways toward Scientific Closure

  • In Case 30, Kimi-K3, GPT-5.6-Sol, and Gemini-3.1-Pro reached RC4, RC3, and RC2; Gemini misread intensity trends, while GPT-5.6-Sol left applicability risk unresolved.Kimi-K3 corrected an angle-sign inconsistency, explained the competition between exit radius and screening length, and justified applicability limits.Source: Figure 5 · p. 8, §5.3 · p. 9

5.2 High Atomic Scores and Incomplete Research Closure

  • 36.2% of attempts scoring at least 8/10 fell short of RC4, and 55 of those attempts remained at RC2.These RC2 cases had unresolved requirements for the main scientific conclusion, not merely incomplete validation.Source: §5.2 · p. 8
  • High-scoring attempts below RC4 left 16.0% of Identification and 20.8% of Mechanism criteria unmet, versus 2.1% and 3.0% among RC4 attempts.One investigation produced spectra and separated signal contributions but did not connect them to the underlying physical mechanism.Source: §5.2 · p. 8
  • Scope justification was satisfied in 86.9% of cases, compared with 59.6% for required validation and 47.8% for risk-linked validation.An investigation can establish the main mechanism while leaving an applicability concern unresolved; validation or a narrower conclusion may be needed.Source: §5.2 · p. 8

5.4 Research Closure across Repeated Runs

  • Claude-Opus-5 reached RC4 on 50.0% of tasks at least once but on 18.0% in all three runs, showing that demonstrated closure was not reliably repeated.Figure 6 contrasts tasks reaching RC4 in at least one run with tasks reaching RC4 in all three independent runs.Source: §5.4 · p. 9

6 Conclusion

The paper introduces PhysClosureBench to evaluate scientific agents beyond answer correctness or artifact generation. Across seven frontier models, high local accomplishment did not always yield complete closure, and closure was less consistent across repeated runs.

6 Conclusion

  • PhysClosureBench evaluates scientific agents beyond answer correctness or artifact generation, using goal-specified physics tasks and measures of atomic accomplishment and Research Closure.The benchmark assesses whether investigations progress from research questions to well-supported conclusions.Source: §6 · p. 9
  • Across seven frontier models, high Atomic Scores did not always correspond to complete closure, and case analyses identified misinterpretation, unsupported mechanisms, and unresolved applicability limits as distinct failure reasons.The findings distinguish local scientific work from whether an investigation resolves its question with supported evidence.Source: §6 · p. 9
  • Models reached RC4 on substantially fewer tasks in all three runs than in at least one, separating demonstrated closure from reliable closure across independent investigations.Figure 6 compares the fraction of tasks reaching RC4 at least once with the fraction reaching it in all three runs.Source: §6 · p. 9, Figure 6 · p. 9

A.1 Benchmark Construction and Quality Control

The benchmark was built with expert guidance, bounded task settings, and a controlled execution protocol. Its scoring combines fixed atomic criteria with task-specific closure requirements and conditional applicability rules.

A.1 Benchmark Construction and Quality Control

  • An expert-guided pipeline formulated bounded tasks from published studies, with domain-expert reviews checking task validity, grading criteria, prerequisites, alternative evidence, and benchmark design.Tasks were admitted only after successful reference reproduction, and specifications, resource limits, and rubrics were finalized before evaluation.Source: §A.1 · p. 12

A.2 Task-Specific Resource Limits

  • Reference reproductions calibrated CPU and wall-time budgets, with added allowance for planning, exploratory calculations, method selection, and unsuccessful intermediate attempts.The limits aimed to support scientifically viable investigations within a bounded execution setting.Source: §A.2 · p. 13

A.3 Agent Execution Environment and Examination Protocol

  • Agents worked in separate, isolated, network-disabled environments and received common instructions, case-specific objectives, physical settings, parameters, and constraints.They could choose computational, analytical, and validation procedures within the task requirements and visible limits; standard submissions documented reports, derivations, execution history, and supporting artifacts.Source: §A.3 · pp. 13–14

B.1 Automated Grading and Scoring Semantics

  • Each case rubric links twenty binary half-point atoms to task-specific scientific requirements and prerequisites, with conditional criteria activated by the candidate’s method, evidence, or conclusion.Research Closure depends on indispensable scientific requirements assessed against their specified conditions, not on an Atomic Score threshold.Source: §B.1 · pp. 14–15
  • An N/A criterion retains its fixed 0.5-point contribution to the 10-point Atomic Score but is excluded from applicable-criterion completion and does not imply that the scientific work was performed.For risk-triggered Validation/Scope criteria, N/A means the triggering risk did not arise; terminal-zero runs receive score 0 and RC0 without N/A credit.Source: §B.1 · p. 14

B.2 Human Consistency Check of Selected Grading Decisions

Five physics-trained experts reviewed selected grading decisions across the 50 benchmark cases. Their judgments showed substantial, though not complete, criterion-level agreement with automated grading.

B.2 Human Consistency Check of Selected Grading Decisions

  • Five physics-trained experts reviewed 100 evaluation cards spanning all 50 cases, yielding 200 requested criterion-level judgments.The selected criteria emphasized scientifically consequential, locally assessable decisions rather than an equal-probability sample of all rubric atoms.Source: §B.2 · p. 15
  • Of 200 requested judgments, 192 yielded effective decisions, and 81.8% agreed with automated criterion-level decisions.The cards included candidate evidence, physical context, selected criteria, and accepted equivalent routes.Source: §B.2 · p. 15
  • For RC-relevant criteria, 60 of 79 binary judgments agreed with automated decisions, a 75.9% agreement rate.RC assignment additionally accounts for task-specific trigger conditions and evidence dependencies.Source: §B.2 · p. 15

C Statistical Analysis

The analysis treats each final scored attempt as its unit, including incomplete attempts. It averages attempt-level functional completion and summarizes Validation/Scope separately by applicable criteria.

C Statistical Analysis

  • 1,050 final scored attempts from seven models, 50 cases, and three independent runs per model–case pair form the analysis population.Incomplete attempts, including timeouts, remain; infrastructure-invalid executions and superseded retries do not add observations.Source: C Statistical Analysis · p. 15
  • Functional completion is the fraction of satisfied atoms among applicable atoms for each attempt, then averaged across attempts with equal weight.N/A atoms are excluded from both the numerator and denominator; terminal-zero attempts without atom-level grading contribute zero.Source: C Statistical Analysis · p. 15
  • Validation/Scope is summarized separately as the fraction of applicable V criteria satisfied, excluding N/A criteria and terminal-zero attempts without atom records.Applicability depends on the method, evidence produced, and resulting conclusion.Source: C Statistical Analysis · p. 15
  • Figure 4A cells express percentages of all 1,050 attempts, not Research Closure rates conditional on score band.Source: C Statistical Analysis · p. 16

D.1 Detailed Dimension and Research Closure Results

The appendix reports model- and run-level dimension and Research Closure results, with Validation/Scope treated separately. It also details RC3 cases and shows how Atomic Scores vary across investigations.

D.1 Detailed Dimension and Research Closure Results

  • Tables 6–9 give model- and run-level dimension and Research Closure summaries; RC tables report all five closure levels across three independent runs.F/O/I/M use attempt-level completion conventions, while V uses its applicable-criterion denominator.Source: §D.1 · p. 16, Table 6 · p. 16, Table 7 · p. 16
  • Table 10 gives Validation/Scope category definitions and criterion-level satisfaction across the benchmark, excluding N/A criteria and terminal-zero attempts without atom records from its denominator.Source: §D.1 · p. 17, Table 10 · p. 17
  • All 26 RC3 attempts retain an unmet applicable Validation/Scope criterion, while 25 also satisfy another V criterion.Source: §D.1 · p. 17

D.2 Atomic-score Distributions across Investigations

  • Figure 8 shows Atomic Score distributions across each model’s 150 scored attempts, reflecting variation across tasks and independent runs.The distributions include terminal-zero outcomes, and dashed lines mark model means.Source: §D.2 · p. 17, Figure 8 · p. 18

E Representative Evidence Paths

Case 30 compares three submissions against the evidence needed to answer when post-release scattering explains attoclock offsets, including whether the interpretation transfers to equally bound screened potentials. The examples show that trend interpretation and applicability limits affect closure.

E Representative Evidence Paths

  • Kimi-K3 reaches RC4 after correcting an angle-sign inconsistency, interpreting screening through exit distance and screening length, and limiting physical interpretation to quasistatic sub-barrier exits.Its orbit-integral and trajectory results then differ by 4.53 × 10−7 rad; two saved intensities lie within its stated Coulomb window of approximately 0.0065 < I < 0.0078 a.u., with pronounced screening contrast.Source: E Representative Evidence Paths · pp. 18–19
  • GPT-5.6-Sol reaches RC3: it explains the screening-dependent offset changes but does not reconcile its high-field interpretation with a barrier-suppression diagnostic.At E0 = 0.08 a.u., the diagnostic flags a field above the Coulomb-reference threshold EBSI = 0.0625 a.u.; the passage cautions this is not a universal screened-target threshold.Source: E Representative Evidence Paths · p. 19
  • Gemini’s claim that θ decreases monotonically with increasing laser intensity for all tested potentials is contradicted: λ = 1–3 curves rise, while λ = 5 and 10 rise then fall, yielding RC2.Figure 9 replots the submitted scans for λ = 2 and 5, showing all 100 recorded points per curve with straight connectors and no fitting or smoothing.Source: E Representative Evidence Paths · p. 19, Figure 9 · p. 19

F Supplementary Model-Level Results

The appendix consolidates model-level performance and RC4 consistency, reproduces the Case 30 task and closure rubric, and specifies evidence-based grading procedures. It separates local criterion scoring from Research Closure and requires reviewers to preserve evidence attribution and investigate material validity concerns.

F Supplementary Model-Level Results

  • Table 12 consolidates model-level Atomic Scores, attempt-level RC4 rates, and task-level RC4 consistency across the 50 benchmark tasks.Mean and median scores and attempt-level RC4 rates use 150 attempts per model; both consistency measures use the same 50-task denominator.Source: F Supplementary Model-Level Results · p. 19, Table 12 · p. 19

G.3 Case-specific Research Closure summary

  • Case 30 closure requires a consistent offset calculation, equally bound Coulomb–screened comparisons, an explanation of the screening-dependent intensity response, and justified validation and applicability limits.Its RC0–RC4 levels progress from missing or invalid central calculations through incomplete comparisons, unsupported physical conclusions, or unresolved validity issues to satisfaction of all indispensable requirements.Source: §G.3 · pp. 21–22

Shared Grading Instructions

  • Reviewers distinguish an existing scientific object from its presentation, identification, and physical interpretation; evaluator-generated plots can verify an object but cannot supply omitted scientific acts.Output credit also requires the object to be correct and reconstructible, its relevant structure to be presented, and claims to stay within the evidence.Source: Scientific evidence and attribution · p. 22, Atomic scoring · p. 23
  • Atomic Score sums twenty binary half-point criteria, while Research Closure is assessed independently against indispensable task-specific scientific requirements and applicable scope conditions.An inapplicable criterion retains its 0.5-point contribution but does not imply the corresponding work was performed; RC4 requires all indispensable requirements to be satisfied.Source: Atomic scoring · p. 23, Validation and applicability · p. 24
  • A validation or scope gap warrants RC3 only when a concrete failure mode, its origin and threatened conclusion, insufficient evidence, and equivalent remedies are established.Discriminating evidence or an evidence-based restriction can address the threat; recognizing a risk alone does not resolve it.Source: Validation and applicability · p. 24, Verification of important results · pp. 24–25

Independent First-Pass Grading Prompt

  • First-pass grading records attempt validity, all twenty criterion decisions, Output checks, and Research Closure with supporting rationale; targeted review checks material score, closure, fairness, and validity decisions.Review preserves independently valid work, records changed decisions and remaining uncertainty, and requests adjudication only for material unresolved disagreements.Source: Output · p. 26, Review decisions · p. 26

H.6 Grading Output Format

  • The accepted Atomic Score uses accepted criterion-level decisions and prescribed half-point and N/A rules, with targeted review and adjudication documenting any changes and their evidence.Source: §H.6 · p. 27

Related papers

Loading 2610.21767v1…