Source-linked AI summary
RefVerifier: Semi-Automated Reference Claim Verification for Scientific Manuscripts
Stefania Mocan, Florian Angermeir, Mark Kreitz
TL;DR
Reviewer time constraints leave cited claims rarely checked, while existing approaches do not verify complete manuscripts against author-designated sources. RefVerifier automates citation-bounded claim verification, and evaluation reports strong claim detection and evidence localization alongside 71% verdict agreement on real-world manuscripts.
Problem
Reviewer workload makes systematic checking of citation-supported claims difficult, and existing tools do not verify complete manuscripts against author-designated sources.
Source: §1 · p. 1Method
RefVerifier extracts and enriches citation-bound claims, checks bibliography metadata, retrieves cited papers, localizes evidence, and predicts verdicts while preserving reviewer control.
Source: §1 · p. 1Results
Claim detection reached an F1 score of 0.990 on CiteWorth, evidence localization hit rates were 98% on abstracts and 68% on full papers, and verdict agreement was 71% on real-world manuscripts.
Source: §4.1 · p. 4, §7 · p. 6Takeaways & Limitations
The interactive interface leaves the final assessment with the reviewer.
Source: §1 · p. 1The eight-paper end-to-end evaluation used one annotator, limiting generalization, and the open-access retrieval chain fails for roughly 42% of citations.
Source: §6 · p. 6
Abstract
from arXiv · showhide
As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing.
1 Introduction
Reviewer workload leaves citation-supported claims rarely checked, while existing tools do not provide end-to-end verification of complete manuscripts against their cited sources. RefVerifier addresses this gap with a citation-bounded workflow for manuscript claims.
1 Introduction
- Review deadlines and heavy workloads mean reviewers rarely verify whether sources support their accompanying assertions.Inaccurate citations, overstatements, and unsubstantiated claims can then pass review and propagate through the literature.Source: §1 · p. 1
2 Related Work
- Open-domain fact-checkers use broad corpora, reference managers check surface metadata, and specialized citation tools may require manually supplied claims.These tools differ from manuscript-level verification against author-designated references.Source: §1 · p. 1, §2.1 · p. 2
1 Introduction
- RefVerifier checks claims against manuscript-cited sources through an end-to-end workflow, while leaving the final assessment to the reviewer.Its pipeline extracts citation-bound claims, checks references, localizes evidence, and predicts verdicts with explanations.Source: §1 · p. 1
2.1 Foundational Definitions
- A citance is a citation-bearing sentence asserting something about a cited work, and it is RefVerifier’s primary unit of verification.Reference claim verification asks whether the author-designated source substantiates the assertion, regardless of external consensus [24].Source: §2.1 · p. 2
2.2 Benchmark Datasets
- Existing scientific citation benchmarks variously limit evidence to abstracts, provide full-paper spans for biomedical citations, or use controlled distortions.The surveyed benchmarks also include tools that check supplied citation text rather than parsing complete manuscripts.Source: §2.2 · p. 2
2.3 Research Gap
- Prior approaches lack citation-bounded verification, manuscript-level claim extraction, an end-to-end workflow, or broad full-paper support.RefVerifier is presented as an end-to-end, citation-bounded tool operating on full manuscripts.Source: §2.3 · p. 2
3 Prototype Design
RefVerifier is implemented as a web application organized around four sequential stages.
3 Prototype Design
- The prototype uses a Python FastAPI backend and React 19 single-page frontend, with four sequential stages in its core architecture.Figure 1 is identified as an overview of the pipeline and interactive reviewer interface.Source: §3 · p. 2
Stage 1: Claim Identification
Claim identification combines deterministic citation detection with LLM enrichment of citation-bearing sentences.
Stage 1: Claim Identification
- A hybrid deterministic–LLM workflow identifies citation-bearing sentences and enriches each citance with structured claim information.Detection uses PDF text or LaTeX citation keys, with OCR as a fallback when the PDF has no extractable text layer. Enrichment covers claim type, hedge level, citation-specific spans and scope, and ambiguity handling.Source: Stage 1: Claim Identification · p. 2
Stage 2: Metadata Check & Evidence Retrieval
Stage 2 checks bibliography metadata and resolves citations to verified full-text PDFs through a multi-source retrieval chain.
Stage 2: Metadata Check & Evidence Retrieval
- Bibliography entries are parsed into structured fields and checked against Semantic Scholar, Crossref, and OpenAlex.References are classified as validated, unvalidatable, or suspicious, including cases with title or author contradictions.Source: Stage 2: Metadata Check & Evidence… · p. 2
- A multi-step resolver searches local files and uploads, direct identifiers, open-access APIs, then Google Scholar, stopping at the first valid PDF.A deterministic right-paper check compares embedded metadata and first-page titles with the reference.Source: Stage 2: Metadata Check & Evidence… · p. 2
Stage 3: Evidence Localization
RefVerifier localizes evidence either through a two-stage retriever with reranking or a merged LLM mode that processes the cited paper directly.
Stage 3: Evidence Localization
- The two-stage retriever divides cited papers into overlapping sentence windows, using longer windows for causal or predictive claims.It ranks windows with semantic and lexical signals, then reranks the top candidates with a cross-encoder.Source: Stage 3: Evidence Localization · p. 3
- Merged LLM Mode processes the cited paper’s full text in one call to identify supporting passages and generate verdicts.It bypasses separate retrieval models and combines evidence localization with verdict generation.Source: Stage 3: Evidence Localization · p. 3
Stage 4: Verdict Prediction
The verdict engine assesses localized evidence against a claim and returns one of four verdicts with a natural-language explanation.
Stage 4: Verdict Prediction
- The four verdicts are Supported, Partially Supported, Not Supported, and Not Enough Information.Partially Supported captures minor overstatements, missing nuance, or scope shifts; Not Enough Information means the tool could not confidently judge from the cited document.Source: Stage 4: Verdict Prediction · p. 3
- Prompts use claim-type rubrics and hedge adjustments when generating the verdict explanation.Source: Stage 4: Verdict Prediction · p. 3
Web Interface
The interface links manuscript citations, claim cards, verdicts, and evidence passages across a split-view review workspace.
Web Interface
- Users can upload a manuscript PDF, arXiv ID or URL, or LaTeX project, then follow processing progress to the results view.Source: Web Interface · p. 3
- Selecting a highlighted citation opens its claim card, while selecting an evidence snippet jumps to the passage in the resolved cited-paper PDF.The split view keeps the manuscript, reference sidebar, and cited-paper pane connected during review.Source: Web Interface · p. 3
4 Evaluation
RefVerifier was evaluated across public datasets and an eight-manuscript end-to-end test, alongside a technology-acceptance survey. Results cover detection, retrieval, localization, verdict prediction, runtime, and reviewer perceptions.
4.1 Quantitative Evaluation
- 0.990 F1: deterministic regex detection on CiteWorth, with 0.980 precision and 1.000 recall.On ten arXiv manuscripts, detection recall on rendered PDFs was 0.93.Source: §4.1 · p. 4, Table 1 · p. 4
- 57.6% (571/992) of references across 23 source papers resolved to full-text PDFs, with Medicine and Biology showing the lowest rates.A title-matching audit found 93% of downloaded PDFs matched the cited work; 54.9% of unresolved references had no free web copy.Source: §4.1 · p. 4, Table 2 · p. 4
- 68% on full papers and 98% on abstracts: the deployed merged LLM mode’s evidence-localization hit rates (MRR 0.96).The mode outperformed the earlier bi-encoder and reranker approach.Source: §4.1 · p. 4, Table 3 · p. 4
- Across four public datasets, verdict-prediction F1 ranged from 0.43 to 0.89, with scores shaped by differences in dataset construction and label distributions.The evaluation used ground-truth evidence to isolate judgment from retrieval errors.Source: §4.1 · pp. 4–5, Table 4 · p. 5
- gpt-5.6-terra performed better than the small models at every tested reasoning level, but at increased inference cost.The comparison used the same 100 Citation-Integrity claims across model configurations and two evidence-input setups.Source: §4.1 · p. 5, Table 5 · p. 5
- Across eight manuscripts, the evaluation covered 153 manually annotated citance-reference rows, including 145 confirmed pairs; 25 of 118 claims were judged Partially Supported by the tool when annotators selected Supported.Verdicts were blindly annotated, while citance text, reference mapping, and evidence passages were refined from tool proposals.Source: §4.1 · p. 5, Table 6 · p. 5
- 634 seconds (11 minutes): median end-to-end runtime, with reference retrieval accounting for 35%–91% of total time and a median above 65%.Claim extraction and verdict generation took at most 6.3 minutes per paper.Source: §4.1 · p. 5
4.2 Qualitative Evaluation
- Survey respondents rated usefulness and ease of learning at median 6, while agreement with accepting a Support verdict without review was lower (median 5).Agreement with re-checking a Not Supported verdict had a median of 7.Source: §4.2 · p. 5
5 Discussion
The discussion frames benchmark scores as stage-specific and argues that adoption depends mainly on socioeconomic and institutional conditions, not technical feasibility alone.
5 Discussion
- Benchmark scores are stage-specific indicators, not directly comparable with published dataset results; broader jointly annotated evaluation is needed for stronger generalization.The evaluation repurposes datasets for pipeline components rather than reproducing their original benchmark settings.Source: §5 · p. 6
- Real-world adoption remains difficult mainly for socioeconomic rather than technical reasons.Source: §5 · p. 6
- Open-access shifts vary by discipline and do little to improve access to literature published before those practices took hold.Source: §5 · p. 6
- Copyright constrains reuse: preprints may expose paywalled work, but are not final versions and may be restricted by copyright.Per-publisher agreements are legally cleanest but do not scale, especially without institutional backing.Source: §5 · p. 6
- Scaling requires institutional support for LLM costs, rights negotiations, and privacy safeguards for unpublished intellectual property.Source: §5 · p. 6
- The peer-review community needs guidance to use AI-assisted verification without over-relying on automated judgments.The tools should complement, not replace, reviewers’ expertise.Source: §5 · p. 6
6 Threats to Validity
The authors identify validity limits in benchmark label fit and the small, single-annotator end-to-end manuscript evaluation.
6 Threats to Validity
- Benchmark labels do not map cleanly to RefVerifier’s four-class verdicts, so cautious Partially Supported predictions can reduce exact-match accuracy despite practical usefulness.Citation-Integrity’s Refuted class combines direct contradictions with softer issues such as oversimplification and weak substantiation.Source: §6 · p. 6
- The end-to-end study covers 8 papers assessed by one annotator, limiting generalization of its 0.71 accuracy across research domains.The dataset is predominantly well-supported, so evaluation of genuine Not Supported citations relies heavily on Citation-Integrity.Source: §6 · p. 6
- The open-access resolution chain fails to retrieve roughly 42% of citations because of paywalls or missing identifiers.Source: §6 · p. 6
7 Conclusion
RefVerifier automates citation-to-verdict stages for scientific manuscripts and reports 71% verdict agreement in real-world manuscripts. Future work targets extraction and rubric refinements, while deployment plans depend on institutional partnerships and reviewer-controlled integration.
7 Conclusion
- 71% verdict agreement on real-world manuscripts, with a median runtime of 11 minutes.Claim extraction and verdict generation take at most 6.3 minutes.Source: §7 · p. 6
- Future work targets citation extraction across superscript and author-year formats and verdict rubrics for subtle overstatements.Source: §7 · p. 6
- Deployment plans depend on institutional partnerships for rights-cleared texts, privacy-preserving inference, sustainable costs, and review-platform integration that keeps reviewers in control.Source: §7 · p. 6