PaperlayerView exact revision on arXiv

Source-linked AI summary

SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores

Bowen Liu, Shuo Nie, Bodong Du, Xiaomeng Li

arXiv:2609.06192v1cs.AI

TL;DR

Scientific coding agents can produce plausible code, results, figures, and claims without establishing that the claims are scientifically supported. SCIRIGOR formulates same-run evidence-grounded analysis and evaluates complete computational and visual claim-support paths across a broad benchmark, finding that internal agreement remains high even when results are unfaithful and strict whole-chain success is rare.

  • Problem

    Existing evaluations do not jointly establish whether a newly generated claim is supported by the data, computation, and visualization produced in the same run.

  • Method

    SCIRIGOR reconstructs typed evidence graphs, separates artifact fidelity from relational validity, accepts source-grounded alternative paths, and localizes the earliest unsupported relation.

  • Results

    Claims agree with faithful and unfaithful results at nearly identical rates, 91.8% versus 91.0%, while strict whole-chain success reaches only 18.0%.

  • Takeaways & Limitations

    Internal coherence, execution, visual plausibility, and final-output similarity are not substitutes for independently verified data-to-claim support.

  • Takeaways & Limitations

    The evaluation observes submitted artifacts and runtime evidence, not latent reasoning or agent action trajectories, so it cannot identify whether failures arose from planning, implementation, or unsupported generation.

Abstract

from arXiv · show

Scientific coding agents produce interdependent code, results, figures, and claims, yet evaluating final outputs alone does not establish whether their conclusions are scientifically supported. We formulate evidence-grounded multimodal scientific analysis, requiring agents to produce executable analyses and claims supported by results and visualizations from the same run. We introduce SciRIGOR, an evaluation framework and benchmark comprising 100 cases from scientific articles across six domains and 17 subfields. The framework reconstructs typed evidence graphs, separates artifact fidelity from relational validity, and scores complete claim-support paths while localizing the earliest unsupported relation. Source-grounded alternative paths accommodate scientifically equivalent analyses and visualizations. We evaluate 11 agent/model configurations. On full-benchmark runs, claims agree with faithful and unfaithful results at nearly identical rates (91.8% versus 91.0%). Yet no system exceeds 62.6% on the soft evidence-chain score or 18.0% strict whole-chain success. These findings show that internal coherence does not establish scientific correctness: evaluation must verify support along the complete data-to-claim path.

1 INTRODUCTION

SCIRIGOR addresses the gap between plausible final outputs and scientifically supported conclusions by evaluating complete same-run evidence paths. It introduces a multimodal task and evaluates 11 configurations, finding that internal agreement can persist despite unfaithful results.

  • Motivation: Existing benchmarks typically score programs, stages, artifacts, or final outcomes rather than whether same-run data, computation, and visualization support a claim.This can accept correct-sounding but unsupported conclusions or reject scientifically defensible alternatives.
  • Task formulation: SCIRIGOR requires agents to produce mutually supporting code, numerical results, a figure, and atomic claims for a scientific question.Success requires each primary claim to retain at least one complete computational and visual support path.
  • Evaluation framework: The evaluation uses typed evidence graphs to separate artifact fidelity from edge validity, score complete support paths, and localize the earliest unsupported relation.Source-grounded alternative paths accommodate scientifically equivalent analyses and visual encodings.
  • Benchmark and evaluation: 100 evidence-complete cases span six domains and 17 subfields, and 11 agent/model configurations are evaluated.Cases undergo executable reconstruction and human-audited provenance before evaluation.
  • Findings: 91.8% versus 91.0%: claims agree with faithful and unfaithful results at nearly identical rates, while strict whole-chain success reaches only 18.0%.The result indicates that internal coherence alone does not establish scientific correctness.

2 RELATED WORK

SCIRIGOR connects benchmark levels that are often evaluated separately: executable analyses, end-to-end research workflows, and individual figures or claims. Its scored object is a generated claim together with the computation and visualization supporting it.

  • Benchmark scope: Scientific-agent benchmarks differ in whether they primarily score executable analyses, completed research workflows, or individual figures and claims.SCIRIGOR connects these levels by evaluating a generated claim with its supporting computation and visualization.
  • Scientific programming and data analysis: Scientific programming benchmarks evaluate code, execution, analysis decisions, or reproduction of published results as their primary unit.Examples include NatureBench, ScienceAgentBench, BLADE, and DataSciBench.
  • Scientific discovery and research agents: Research-agent benchmarks broaden evaluation to multi-stage workflows involving discoveries, experiments, conclusions, or complete multimodal research artifacts.Examples include DiscoveryBench, FIRE-Bench, ResearchClawBench, and FrontierChallenge.
  • Visualization and claim verification: Visualization and claim-verification benchmarks isolate links such as analysis visualization, figure-caption agreement, or claim auditing.SCIRIGOR instead jointly evaluates newly generated claims with computational and visual support from the same run.

3 SCIRIGOR

SCIRIGOR treats scientific analysis as a connected evidentiary object whose artifacts and dependencies are verified from execution. Its scoring accepts approved alternatives, applies weakest-link path evaluation, and identifies where incomplete support first breaks.

  • Task formulation: SCIRIGOR defines scientific analysis as a connected object spanning generated code, numerical results, visualization, and atomic claims.The task makes relations among these artifacts part of the output rather than treating each artifact as an independent endpoint.
  • Task formulation: The executable task requires analysis code to regenerate results, visualizations, and claims from analysis-ready data while exposing provenance for verification.Every primary claim needs a verified computational and visual support path from the same run.
  • Reference specification: Reference support specifications encode accepted method families, visual encodings, and claim-support rules as semantic constraints rather than one gold program or image.Separate approved support sets represent alternative methods or visualizations.
  • Evidence-graph evaluation: The evaluator reconstructs an executed dependency graph from observable data, transformations, results, visual marks, and claims rather than agent-authored reasoning traces.Candidate-declared provenance proposes relations, which are checked against runtime observations and independent numerical or visual evidence.
  • Claim-path scoring: SCIRIGOR scores a claim by the strongest approved complete path, with each path governed by its weakest verified node or edge.The scoring also combines computational, visual, and local claim fidelity, while allowing alternative paths.
  • Failure localization: Failure localization reports the earliest topological layer from which no accepted partial path can reach the claim.Downstream presentation quality cannot repair an upstream scientific error.

4 SCIRIGOR BENCHMARK

SCIRIGOR builds evidence-complete, figure-level benchmark cases by aligning scientific questions with source analyses, data, figures, and findings. Its 100 cases span six domains and 17 subfields, with construction safeguards for reproducibility, provenance, and support fidelity.

  • Benchmark construction: 100 evidence-complete cases from 27 empirical articles align each question with code-consumed inputs, executable source analysis, numerical results, visualization, findings, and provenance.Each case is bounded to one figure or coherent panel subset and constructed through source discovery, evidence closure, standardized assembly, reproduction, and audit.
  • Source discovery and evidence closure: Admission required publicly retrievable, freezeable data dependencies, author-released analysis code, an auditable figure, and a bounded Results passage.These components tie the visual target, interpretation, executable analysis, and inputs to the same scientific question.
  • Source discovery and evidence closure: Two clean executions had to reproduce the figure’s quantities, visual marks, groups, units, and uncertainty encodings; substantive disagreements caused exclusion.The frozen author entry point ran against complete inputs, and missing dependencies or unrecoverable semantics also led to exclusion.
  • Benchmark composition: The six domains are economics and policy, computing and information sciences, social and behavioral sciences, cognitive science, life and health sciences, and physical sciences.Their case counts are 24, 23, 22, 15, 9, and 7, respectively.
  • Benchmark composition: 100 cases span six reporting domains and 17 disciplinary subfields, while varying in input assets, result slots, visual requirements, and atomic claims.The benchmark unit is a bounded figure-level question, and these requirements create evidence chains of different lengths.

5 EXPERIMENTS

Across 11 configurations, execution, artifact fidelity, and complete evidence-chain support diverge substantially. Errors commonly arise before faithful results are established, while local self-consistency can remain high despite scientific error.

  • Main results: Execution failures gate every downstream score, with execution rates ranging from 79.0% for Kimi-K3 to 12.0% for Qwen-AgentWorld-35B-A3B.Scores based only on successful outputs would conflate scientific quality with selective completion.
  • Main results: 97.9% Result→Claim consistency coexists with 47.2% result fidelity and 52.5% claim fidelity for GPT-5.6 (low).Self-consistency can therefore reflect coherent propagation of scientifically incorrect intermediate values.
  • Main results: 62.6% SoftECS contrasts with only 18.0% Strict success in the strongest run, showing that locally successful components often fail to form a complete supported path.Kimi-K3 declines from 79.0% execution to 35.0% SoftECS and 1.0% Strict success.
  • Error analysis: 48.2% of 936 non-strict trajectories first fail in data selection or transformation, while 30.1% first fail at the artifact contract or execution gate.Result computation accounts for 11.9%, and visualization, claims, and provenance together account for 9.8% of first failures.
  • Error analysis: Among 718 executed trajectories, transform fidelity fails in 59.6%, result fidelity in 49.3%, and claim fidelity in 47.9%, versus 8.6% for Result→Claim consistency.The comparison conditions on successful execution and separates scientific errors from mechanical non-completion.
  • Error analysis: Result→Claim consistency passes on 91.8% of result-faithful trajectories and 91.0% of result-unfaithful trajectories.Result→Visual consistency is more sensitive, passing on 97.3% of faithful versus 82.2% of unfaithful trajectories.
  • Complexity sensitivity: Increasing input assets from one to at least five lowers Strict RG-EC from 45.7% to 25.0% and RG-EC F1 from 70.9% to 57.8%.The gated-claim score drops from 71.7% to 63.7%, indicating a stronger effect on chain completeness than surface claim plausibility.

6 CONCLUSION

SCIRIGOR evaluates scientific analysis as a verified data-to-claim support path rather than as isolated artifacts or final outputs. Its results show that execution and apparent consistency can coexist with unsupported conclusions, motivating independent verification of every relation.

  • 6 CONCLUSION: SCIRIGOR separates artifact fidelity from cross-artifact consistency and localizes the earliest unsupported relation in a typed evidence graph.The framework reconstructs computational and visual dependencies while accepting source-grounded alternative support paths.
  • 6 CONCLUSION: 78.3% of failed trajectories first break at the contract/execution or data/transformation stage across ten complete 100-case runs.This identifies early pipeline relations as the most common first failure location.
  • 6 CONCLUSION: 91.8% versus 91.0%: Result→Claim consistency passes at nearly the same rate for faithful and unfaithful results.The comparison demonstrates that agreement between results and claims does not by itself establish scientific support.
  • 6 CONCLUSION: The evaluation scores a verified data-to-claim support path, requiring generated code, results, figures, and atomic claims to retain same-run computational and visual support.The scored object is relational and evidence-grounded rather than limited to a program, artifact, or final outcome.
  • 6 CONCLUSION: The main paper presents the task interface and evidence-graph workflow conceptually, while Tables 3 and 4 contain complete ownership and edge-verification rules.These tables provide the detailed evaluator specification underlying the conceptual presentation.

E VALIDATION AND RELEASE DETAILS

The release combines hashed, declared inputs with clean reconstruction, human review, and descriptive disciplinary coverage. Its 100 cases are presented as a curated benchmark scope, not a probability-balanced sample of scientific literature.

  • E VALIDATION AND RELEASE DETAILS: 302 declared input assets were materialized at specified paths, matched to recorded SHA-256 hashes, and confirmed as regular files.File-open traces confirmed that execution consumed only declared inputs.
  • E VALIDATION AND RELEASE DETAILS: 100 cases passed two clean reconstruction runs, including one numerically sensitive case using its pinned SciPy environment.The release also retained the source registry, execution traces, and reconstruction evidence.
  • E VALIDATION AND RELEASE DETAILS: All 100 released cases record approved final human-review status after checks for ambiguity, leakage, shared scientific scope, and unsupported requirements.Unresolved discrepancies triggered revision or rejection before release.
  • E VALIDATION AND RELEASE DETAILS: The 100-case benchmark is organized by reporting domain defined by the primary scientific system or institutional setting, not by statistical method or visualization type.Table 5 summarizes the resulting disciplinary composition.
  • E VALIDATION AND RELEASE DETAILS: The six representative cases use frozen paper figures or source-pixel panel crops to illustrate the public benchmark contract rather than hidden evaluation outputs.Each example specifies the scientific task, frozen evidence, and corresponding executable evidence chain.

F.1 ECONOMICS AND POLICY: NIH FUNDING INEQUALITY

The NIH funding-inequality case decomposes Theil inequality across four grouping dimensions from FY1985 to FY2020. Within-group differences exceed between-group differences throughout, as an accounting result rather than a causal claim.

  • F.1 ECONOMICS AND POLICY: NIH FUNDING INEQUALITY: 144 group–year rows and 432 displayed point instances represent the four annual decomposition tables.The visible input contains complete author data spanning fiscal years 1985–2020.
  • F.1 ECONOMICS AND POLICY: NIH FUNDING INEQUALITY: FY1985–FY2020: the case decomposes total Theil inequality into within- and between-group components across career stage, gender, race, and degree.The executable analysis verifies that the two components sum to the total for every year and grouping.
  • F.1 ECONOMICS AND POLICY: NIH FUNDING INEQUALITY: Within-group differences contribute more to funding inequality than between-group differences for all four groupings.The finding applies to funded RPG principal investigators and is an accounting decomposition, not a causal effect.

F.2 SOCIAL AND BEHAVIORAL SCIENCES: HEAT AND FOOD-DELIVERY WORKLOAD

The food-delivery case transforms log-scale regression outputs into percentage changes relative to a 20 °C baseline across three workload periods. The source analysis reports higher lunch-peak workload in the specified hot-temperature bins, while supporting associations rather than new causal claims.

  • F.2 SOCIAL AND BEHAVIORAL SCIENCES: HEAT AND FOOD-DELIVERY WORKLOAD: 100[exp(x) −1] transforms each coefficient and confidence bound independently relative to the 20 °C baseline.The case uses processed regression outputs for all-day, lunch-peak, and dinner-peak workload across 21 temperature bins.
  • F.2 SOCIAL AND BEHAVIORAL SCIENCES: HEAT AND FOOD-DELIVERY WORKLOAD: 7.3% higher: lunch-peak workload in the (35, 36] °C bin, with a 95% CI of 4.8–9.7%.The normalized executable applies the nonlinear transformation to all estimates and confidence limits before reconstructing the response curves.
  • F.2 SOCIAL AND BEHAVIORAL SCIENCES: HEAT AND FOOD-DELIVERY WORKLOAD: 18.0% higher: lunch-peak workload in the open-ended (40, ∞) °C bin, with a 95% CI of 12.6–23.6%.The reported values come from the bounded source analysis relative to the 20 °C baseline.
  • F.2 SOCIAL AND BEHAVIORAL SCIENCES: HEAT AND FOOD-DELIVERY WORKLOAD: The reconstructed Figure 6 curves and 95% confidence bands summarize transformed estimates across all-day, lunch-peak, and dinner-peak periods.Shading marks the extreme-heat range, while curves and bands encode the transformed estimates and uncertainty.

F.4 COMPUTING AND INFORMATION SCIENCES: BIORXIV PUBLICATION OUTCOMES

The bioRxiv case measures publication linkage across monthly cohorts and subject categories, while the Gleason case compares phenotype scores and PCA structure across grade groups.

  • bioRxiv publication outcomes: 42.0% of bioRxiv preprints were linked to publications overall; evolutionary biology had the highest proportional rate at 51.5%, while neuroscience had the largest linked count at 2,608.The analysis covered 37,648 preprints, 15,797 linked publications, 61 monthly cohorts, and 27 subject categories.
  • bioRxiv publication outcomes: Publication rates were near zero for the most recent preprints and increased over roughly 12–18 months.
  • Gleason phenotype: Proliferation differed across Gleason groups (H = 12.36, p = 0.00207), with a larger high-score tail in the high-Gleason group.The case included 313 patient-specific Boolean models across low, intermediate, and high Gleason groups.
  • Gleason phenotype: Apoptosis also differed across groups (H = 25.55, p = 2.83 × 10−6), but showed no clear grade-wise trend.
  • Gleason phenotype: The Gleason visualization combines PCA phenotype directions and group centroids with complete Proliferation and Apoptosis distributions; its two displayed dimensions explain 38.3% and 27.8% of model variance.

F.6 PHYSICAL SCIENCES: POST-IMPACT FADING OF DIDYMOS

The Didymos case estimates post-impact fading from a weighted fit to selected observations, then derives the return time to pre-impact brightness and projected dust velocity.

  • Post-impact fading: 0.0579 magnitude day−1 is the recomputed Didymos fading rate from the author’s weighted fit to 21 selected observations.The complete plotted dataset contains 27 measurements from 15 observers, while the fit uses the author’s 21-observation subset.
  • Post-impact fading: 23.69 days is the fitted crossing time with the pre-impact absolute magnitude.
  • Post-impact fading: 0.366 m s−1 is the associated projected dust velocity derived from aperture geometry and elapsed time.
  • Post-impact fading: Figure 10 displays the observations, error bars, weighted fit, impact-time marker, and pre-impact magnitude uncertainty band on a reversed magnitude axis.On this axis, upward movement denotes greater brightness.
Loading 2609.06192v1…