PaperlayerView exact revision on arXiv

Source-linked AI summary

When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research

Guijin Son, Jiwoo Hong, Honglu Fan, Heejeong Nam, Hyunwoo Ko, Seungwon Lim, Jinyeop Song, Jinha Choi, Gonçalo Paulo, Youngjae Yu, Stella Biderman

arXiv:2505.11855v1cs.CL

TL;DR

LLMs are increasingly used to generate scientific research, but their role in verifying scientific manuscripts remains underexplored. This paper introduces SPOT, a cross-validated benchmark, and finds that even the strongest evaluated model achieves low precision and recall.

  • Problem

    LLMs are increasingly used as scientific generators, but their credibility as verifiers of scientific research remains underexplored.

    Source: §1 · p. 1

  • Method

    SPOT evaluates multimodal LLMs on 83 scientific manuscripts with 91 errors validated by authors and human annotators.

    Source: §1 · pp. 1–2, §2.1 · p. 3

  • Results

    Among ten models, o3 achieves the highest SPOT scores: 6.1% ± 1.3 precision and 21.1% ± 4.4 recall.

    Source: §3.1 · p. 5

  • Takeaways & Limitations

    The results point to a need for further research in robust verification systems for reliability in AI-driven research workflows.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists. To date, prior work casts these systems as generative co-authors responsible for crafting hypotheses, synthesizing code, or drafting manuscripts. In this work, we explore a complementary application: using LLMs as verifiers to automate the \textbf{academic verification of scientific manuscripts}. To that end, we introduce SPOT, a dataset of 83 published papers paired with 91 errors significant enough to prompt errata or retraction, cross-validated with actual authors and human annotators. Evaluating state-of-the-art LLMs on SPOT, we find that none surpasses 21.1\% recall or 6.1\% precision (o3 achieves the best scores, with all others near zero). Furthermore, confidence estimates are uniformly low, and across eight independent runs, models rarely rediscover the same errors, undermining their reliability. Finally, qualitative analysis with domain experts reveals that even the strongest models make mistakes resembling student-level misconceptions derived from misunderstandings. These findings highlight the substantial gap between current LLM capabilities and the requirements for dependable AI-assisted academic verification.

1 Introduction

The paper introduces SPOT to address the limited evaluation of LLMs as verifiers of scientific research. It describes how the benchmark is curated, characterized, and used to evaluate error detection.

1 Introduction

  • Verification of scientific manuscripts remains underexplored, while prior factual-verification benchmarks focus on everyday knowledge, reference-based checking, or computer science.Source: §1 · p. 1

2 SPOT: Automating Error Detection in Scientific Research

  • SPOT pairs 83 manuscripts across ten scientific fields with multiple annotated errors, testing multimodal models on long, image-rich papers.Source: §1 · pp. 1–2

2.1 Data Curation

  • Curation filters reported errors, retains author-confirmed issues, audits annotations, and normalizes PDFs into text and images with OCR refinement and manual checks.The final benchmark contains 83 manuscripts and 91 annotated errors.Source: §2.1 · p. 3

2.2 Benchmark Statistics

  • The 91 errors span six inductively derived categories; mathematics, computer science, and physics skew toward equation/proof flaws, while biology skews toward figure duplication.Most papers contain one error; retractions are concentrated mostly in equation/proof cases.Source: §2.2 · pp. 3–4

2.3 Evaluation protocol

  • Models receive interleaved manuscript text and images and return located error descriptions; precision, recall, and pass@K measure agreement with benchmark annotations.A prediction counts as a true positive only when its location matches and an LLM confirms its description indicates the same error.Source: §2.3 · pp. 4–5

3 Main Results and Analysis

On SPOT, even the strongest tested models achieve low precision and recall, while performance varies across error categories and multimodal versus text-only conditions. Reported confidence is mostly near zero and correlates only weakly with pass@4.

3 Main Results and Analysis

  • The evaluation compares six proprietary and four open models, selecting the most capable models per family and omitting smaller ones.The authors report that the selected models already score near zero on SPOT.Source: §3 · p. 5

3.1 Main Results

  • o3 leads on SPOT with 6.1% ± 1.3 precision, 21.1% ± 4.4 recall, and 37.8% pass@4.The other reported leading models score lower on each listed metric.Source: §3.1 · p. 5
  • o3 exceeds 80% on MathVista, MMLU-Pro, GPQA Diamond, and MMMU, but scores roughly 20% on HLE and 21.1% on SPOT.The passage describes HLE as a set of frontier, research-level academic questions.Source: §3.1 · p. 6
  • 20.2 pp is the widest o3–Llama-4-Maverick gap across six benchmarks, and Llama-4-Maverick scores 0.9% on SPOT.Figure 3 compares the two models across the six benchmarks.Source: Figure 3 · p. 6, §3.1 · p. 6
  • For Equation/Proof, o3 reaches 62.6% pass@4; for Figure Duplication, GPT-4.1 scores 44.4%, while o3 and Gemini-2.5-Pro score 0%.The results show different leading models across these error categories.Source: §3.1 · p. 6, Figure 4 · p. 6

3.2 Unreliability of Miscalibrated Models.

  • Most models report very low confidence, and confidence correlates only weakly with pass@4 across error categories.Only two of 498 model–instance evaluations showed full confidence; both were from o3.Source: Figure 4 · p. 6, §3.2 · p. 7

3.3 Impact of Multi-Modality in Detecting Scientific Errors

  • Most models improve in recall and pass@4 when figures are removed, while o3 and Gemini-2.5-Pro see a modest drop.The text-only comparison uses 48 figure-independent instances; proprietary models retain a smaller advantage over open models than in the multimodal setting.Source: §3.3 · p. 7, Table 3 · p. 7

4 Case Study

Experts reviewed model-flagged issues in withdrawn mathematics and materials-science papers, consulting original authors when reviewers were unsure. The cases reveal false mathematical critiques and context or reading errors, alongside one suspected genuine typo.

4 Case Study

  • Domain experts reviewed flagged issues from two withdrawn manuscripts and checked disputed issues with the original authors when reviewers remained uncertain.The reviewers also assessed whether unannotated flagged issues were valid flaws.Source: §4 · pp. 7–8

4.1 Mathematics

  • Both models flagged only Section 3, which was unaffected by the paper’s mathematical gap; o3’s Eichler–Shimura criticism was hallucinated, while Gemini made a false claim after reasonable calculations.The false claim resembles a student mistake based on recalling an incorrect fact.Source: §4.1 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

4.2 Material science

  • o3 confused the final-mixture concentration of 0.006–0.016 M with the acid concentration of 0.2 M, a failure attributed to long-range context and multi-hop reasoning.The final-mixture value appears only once, and the manuscript does not show the explicit calculation.Source: §4.2 · pp. 8–9, Figure 7 · p. 8
  • Gemini misattributed a facet pair, but its separate observation of a surface area 10000x smaller than related compounds was suspected to identify a genuine, unannotated typo.This was the only observed instance of an LLM finding an unannotated genuine error.Source: §4.2 · p. 9, Figure 8 · p. 9

5 Related Works

The paper distinguishes SPOT from prior sentence-level fact-checking and review-generation evaluations, then concludes that current models need more robust verification support. Its checklist reports reproducibility provisions and experiment details.

5 Related Works

  • Prior fact-checking benchmarks assess text-only, sentence-level consistency, while automated review systems focus largely on computer science and review matching rather than error detection.SPOT instead uses expert and automated validation and supplies full multimodal papers at inference.Source: §5 · p. 9

6 Conclusion

  • The conclusion reports pass@1 scores of 18.4% for o3 and 7.3% for Gemini 2.5 Pro, and calls for further research in robust verification systems.The expert-led case studies identify shortcomings in long-tail domain knowledge and implicit multi-step calculations.Source: §6 · p. 9

NeurIPS Paper Checklist

  • The checklist says the experiments are reproducible using provided code, with minor variations possible from proprietary model availability or stochasticity.Prompts and inference configurations are provided in Appendix F; the benchmark and codes are available through supplementary materials.Source: §2 · pp. 17–18
  • The checklist reports means and standard deviations across multiple independent trials with bootstrapping, and says API-based experiments cost approximately $5,000.The authors used official model providers where available and OpenRouter otherwise.Source: §2 · pp. 18–19

10. Broader impacts

The checklist marks the societal-impact question as not applicable and says the authors saw no negative societal impact.

10. Broader impacts

  • The authors mark societal-impact discussion not applicable, stating that they see no negative societal impact in this work.Source: §10 · p. 19

11. Safeguards

The checklist states that the SPOT dataset sharing policy depends on the copyright status of each original paper. It also says assets used in the paper are cited through footnotes or references.

11. Safeguards

  • SPOT dataset sharing varies according to the copyright status of each original paper.Source: §11 · p. 20

12. Licenses for existing assets

  • The paper says all assets it uses are mentioned in footnotes or references.Source: §12 · p. 20

13. New assets

The checklist points to Sections 2 and Appendix E for the assets justification, says the paper does not involve human-subject research, and reports LLM use for editing and visualization code.

13. New assets

  • The asset checklist directs readers to Sections 2 and Appendix E for its justification and gives guidance on documenting assets, licenses, limitations, and consent.It also advises including details about dataset, code, or model assets.Source: §13 · p. 21

14. Crowdsourcing and research with human subjects

  • The paper says it does not involve crowdsourcing or human-subject research and points to an annotation-platform image in Appendix E.Source: §14 · p. 21

16. Declaration of LLM usage

  • The checklist answers No for declaring core-method LLM use; ChatGPT was used to draft and edit the manuscript and generate visualization code.Source: §16 · p. 22

A Limitations

The context-length ablation favors shorter segments for most tested models, while o4-mini improves with greater reasoning effort; the section also describes benchmark and annotation limits.

A Limitations

  • SPOT remains relatively modest in size because it prioritizes copyright compliance, contamination prevention, and annotation accuracy, leaving broader coverage to future work.The proposed expansion spans additional scientific disciplines and error categories.Source: A Limitations · p. 23
  • Even author-acknowledged or retraction-confirmed errors may be incompletely annotated because complex manuscripts can contain unannotated true errors.Source: A Limitations · p. 23
  • False negatives can reflect author notes that under-cover affected results or smaller errors unrelated to the main flaw; false positives can include irrelevant feedback on a genuinely erroneous theorem.Source: A Limitations · p. 23
  • The authors recommend secondary expert review, particularly for errors involving complex logical dependencies or deep specialization.Source: A Limitations · p. 23

B.1 Impact of Context Length in Detecting Scientific Errors

  • +13.5 recall and +4.7 precision: Gemini-2.5-Pro gains most in segment-only versus full-paper detection; Qwen2.5-VL-72B-Instruct changes –0.4 precision and –0.1 recall.The ablation used 36 instances, excluding Equation/Proof cases and errors spanning multiple sections.Source: Figure 9 · p. 23, §B.1 · p. 23

B.2 Impact of Test-Time Scaling in Detecting Scientific Errors

  • o4-mini’s error-detection performance increases almost linearly with higher reasoning effort.The tested settings were low, medium, and high reasoning effort.Source: §B.2 · p. 24, Figure 10 · p. 24

C Estimating Confidence for pass@K

C.1 Unbiased Per-Error Confidence

The paper estimates per-error pass@K confidence from independent runs and aggregates those estimates; mathematics case studies show OCR-related false alarms and differing model feedback after pipeline improvement.

C.1 Unbiased Per-Error Confidence

  • Per-error confidence uses detection counts across eight independent runs to estimate the chance of detecting an error in K fresh attempts.Source: §C.1 · p. 24

C.2 Aggregating Confidence and Calibration

  • Per-error confidence estimates are aggregated into an overall self-estimated confidence; full-size Llama-4-Maverick was too large to host for multi-thousand-token generations.Source: §C.2 · p. 24

D.1 Mathematics

D.2 Material science

Across the material-science and environmental-science examples, model feedback often misreads evidence or context; o3 also locates [75]’s logarithm error but overstates its implications.

D.2 Material science

  • The selected and materials-science errors were easier to spot than the mathematics examples; errors in did not affect its conclusions.Most errors in could be spotted by close attention to text and figures, though some required domain knowledge about COFs.Source: §D.2 · p. 27
  • On, o3’s plausible criticisms arose from misunderstandings or ignored information, including overlooking NMR validation, kinetic conventions, and supplementary photocatalysis comparisons.The authors say Table S10 shows the COFs outperforming literature comparisons; o3 did not have access to the supplementary information.Source: §D.2 · p. 27
  • Gemini misread [59]’s reagent discussion, but correctly found a characterization unit error that made the reported measurement 10,000 times smaller.Gemini argued the reagent was usable because it appeared in a figure, despite the authors’ discussion of its poor results.Source: §D.2 · pp. 27–28
  • On, o3’s objections reflected misinterpretations of complementary composition measurements, water contact-angle data, two-liquid surface-energy estimates, and a literature-based barrier formula.The reported parallel trends do not imply a mathematical error, and Figure 11 presents a related quantity rather than misapplying the sign.Source: §D.2 · p. 28
  • Gemini failed to identify visual context in [74], calling present tables missing and reading a log-scaled plot as linear, producing hallucinated values.The tables were present despite Gemini’s claim that they were missing.Source: §D.2 · p. 28

D.3 Environmental Science

  • For [75], o3 correctly identified log10 replacing ln, but overstated the consequences: corrected CO2-doubling projections are 1.5–1.9 °C, while the authors still describe warming as modest.Using log10 reduced ∆F by approximately 2.3× and yielded 0.64–0.81 °C; using ln gives 1.5–1.9 °C.Source: §D.3 · pp. 28–29
  • Gemini-2.5 failed to point out [75]’s specific logarithm error.Source: §D.3 · p. 29

E Additional Details on SPOT

These sections describe SPOT’s corpus access and date choices, point to annotation resources, and outline the evaluation workflow and generation settings.

E Additional Details on SPOT

  • SPOT contains 83 manuscripts across 28 venues; 62 are openly accessible under a CC license, while 21 paywalled papers are not redistributed.Organizations with institutional access to Springer Nature or Elsevier can apply the preprocessing pipeline to create versions of the paywalled papers.Source: E Additional Details on SPOT · p. 29
  • The corpus targets papers published from 2024 onward to reduce parametric-knowledge contamination, retaining three earlier manuscripts whose first error notices appeared in March 2024.The corpus also includes ten papers from 2025.Source: E Additional Details on SPOT · p. 29
  • Annotation guidance and a sample labeling platform are provided; the lightweight app lets contributors review randomly selected papers and append labels to annotations.csv.Source: E Additional Details on SPOT · p. 29, §F.1 · p. 32

F Additional Details on Evaluation

F.1 Generation Configurations

  • Models use provider-recommended parameters when available; otherwise, generation uses temperature 0.6, top-p 0.95, repetition penalty 1.0, and 8–8192 tokens.Source: §F.1 · p. 30

F.2 Prompts

The prompts distinguish identifying manuscript flaws from matching predictions to annotations, with the auditor focused on errors that directly challenge scientific validity.

Generation Prompt

  • The generation prompt asks an auditor to identify flaws in methods, analyses, or conclusions that directly call a specific scientific claim or paper into question.It excludes presentational, rhetorical, stylistic, and citation-practice issues.Source: §F.2 · p. 32
  • The generation response is structured as a has_error field and an errors list containing each flaw’s location and description.Source: §F.2 · p. 32

Evaluation Prompt

  • The evaluation prompt compares predictions with annotations and counts a match only when both location and description match overall.It allows slightly different location names but requires the descriptions to match.Source: §F.2 · p. 33

G Detailed Results

This section indexes model-specific detailed tables for the full benchmark and text-only evaluation, with pass@K summaries broken down by error and paper categories.

Full and text-only evaluations

  • Tables 4–13 provide detailed results for the models in Table 2, while Tables 14–26 provide text-only evaluation details for Table 3.Source: G Detailed Results · p. 34

Detailed table contents

  • The detailed tables report mean and standard deviation of pass@K for K ∈ {1, 2, 4}, broken down by error category and paper category.The captions identify these breakdowns for both the full benchmark and text-only evaluation.Source: Table 4 · p. 34, Table 14 · p. 37

Related papers

Loading 2505.11855v1…