PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding before verifying: Claim normalization for automated citation verification

Yifan He, Mengjia Wu, Siming Deng, Yi Zhang

arXiv:2608.30145v1cs.IR

TL;DR

Existing citation-verification systems use raw citing claims for retrieval and classification, although those claims can create scope, perspective, and proposition problems. The paper introduces CNCV, which normalizes claims before grounded evidence retrieval and classification, and reports higher macro F1 across 18 classifiers, with automatically normalized evidence matching manually annotated evidence.

  • Problem

    Raw citing claims complicate automated citation verification through scope mismatch, perspective mismatch, and proposition entanglement, limiting retrieval and classification performance.

  • Method

    CNCV applies claim normalization, grounded evidence retrieval, and citation classification as three sequential stages.

  • Results

    CNCV improves macro F1 by 12% on average for encoders and 10% for generative LLMs across all 18 classifiers versus the prior two-stage framework.

  • Takeaways & Limitations

    Evidence quality is the dominant experimental factor, and evidence retrieved from automatically normalized claims performs statistically equivalently to manually annotated evidence.

  • Takeaways & Limitations

    The corpus covers only Library and Information Science, and claim normalization and LLM-guided retrieval use GPT-5-mini, so generalization remains untested.

Abstract

from arXiv · show

Citation accuracy has been studied for decades because of its importance to research reliability. Content-level citation verification assesses the reliability of scholarly claims. Recent work adopts a two-stage retrieval-classification framework inherited from fact-checking. However, this design overlooks the complexity of the raw citing claim and introduces three issues into the verification system, namely scope mismatch, perspective mismatch, and proposition entanglement. These issues increase the difficulty of retrieval and classification, thereby limiting model performance. Motivated by this gap, we propose claim normalization, which applies three rewriting strategies to the raw citing claim before retrieval and classification, allowing each downstream model to perform a single, well-defined task. Building on this method, we develop Claim-Normalized Citation Verification (CNCV), a new three-stage framework consisting of claim normalization, evidence retrieval with grounding, and citation classification. We evaluate CNCV across 18 classifiers using a factorial experiment on human-annotated citation instances. Compared with the prior two-stage framework, CNCV improves macro F1 by an average of 12% for encoders and 10% for generative LLMs, driven by improved evidence quality, the dominant factor identified in our experiments. Evidence retrieved from automatically normalized claims yields downstream classification performance statistically equivalent to that obtained with manually annotated evidence.

1 INTRODUCTION

Content-level citation verification checks whether citing text faithfully represents a valid source, but raw claims create scope, perspective, and proposition problems for automated retrieval and classification. The proposed CNCV framework normalizes claims before retrieval and classification, improving macro F1 across classifiers.

  • Content-level verification asks whether a citing statement faithfully represents the source it cites, while undetected miscitations can propagate erroneous knowledge.
  • Raw citing claims can contain unsupported information, a different source perspective, and multiple propositions, creating scope mismatch, perspective mismatch, and proposition entanglement.
  • Claim normalization applies three consecutive rewriting strategies before retrieval and classification to emulate human reviewers’ understand-then-verify process.
  • CNCV combines claim normalization, evidence retrieval with grounding, and citation classification in a three-stage framework.The framework also uses an LLM-based retriever and maps retrieved fragments back to source text through strict textual similarity matching.

2 RELATED WORK

Automated citation verification emerged from biomedical NLP and fact-checking-related work, while claim normalization draws on broader input-reformulation strategies used to make text clearer for downstream tasks.

  • Automated citation verification has a short history and initially developed from biomedical research using natural language processing.
  • CNCV is proposed in response to limitations in existing automated citation-verification approaches.
  • Claim normalization builds on reformulating input text to better suit downstream tasks, including fact-checking, context-independent verification, and decomposition into elementary statements.

3.1 Definition and task formulation

Content-level citation verification evaluates whether a claim associated with one citation marker is supported by evidence from the cited paper. The system assigns a three-class severity label, while six finer categories support annotation.

  • Definition and task formulation: The task processes one citation marker at a time, pairing the citing claim and target marker with candidate evidence spanning the cited paper’s full text.
  • Definition and task formulation: Six fine-grained annotation labels distinguish error mechanisms, with applicable labels prioritized so the most serious error is assigned first.
  • Definition and task formulation: Experiments use the corresponding three-class scheme because the objective is to detect miscitations and grade their severity.

3.2 Data construction and annotation

The dataset comprises manually retrieved citation pairs from open-access Information Science and Library Science articles, producing 1,444 citation instances. Three experts annotated the data using iteratively refined guidelines with substantial agreement.

  • Data construction: 1,444 citation instances were obtained from 957 usable citing–cited article pairs selected from the top 2,000 citation-ranked candidates.The corpus consists of open-access articles indexed under Web of Science’s Information Science & Library Science category.
  • Annotation: Three information-science experts annotated the dataset using guidelines refined through multiple rounds of pilot annotation.
  • Annotation: Fleiss’ κ = 0.748 at six-class granularity and 0.752 at three-class granularity, indicating substantial inter-annotator agreement under the cited interpretation.

3.3 CNCV

CNCV normalizes raw citing claims before retrieving grounded evidence and classifying the citation. Its pipeline combines three ordered rewriting strategies, LLM-based retrieval with source grounding, and encoder- or LLM-based classification.

  • Framework: CNCV sequentially normalizes the raw citing claim, retrieves and grounds evidence, then assigns one of three citation labels.The normalized claim and grounded evidence are paired for classification.
  • Claim normalization: The three normalization strategies delimit supported scope, align perspective with the cited study, and consolidate propositions into one sentence.Their order is fixed because each strategy operates on the preceding strategy’s output.
  • Retrieval and grounding: The retriever uses an LLM rather than BM25 and cross-encoder reranking to exploit semantic and structural information introduced by normalization.Candidate evidence consists of sentences segmented from each cited paper.
  • Retrieval and grounding: Evidence grounding maps generated evidence sentences to verifiable source sentences through a three-step matching strategy.Fewer than 0.1% of generated sentences fail all matching steps and are discarded.
  • Classification: The system evaluates nine encoder classifiers and nine zero-shot LLM classifiers using normalized claims and grounded evidence.Encoder models use five random seeds per model, while LLM classifiers represent six major providers.

3.4 Experimental design and analysis

The experiments use a factorial design crossing claim type and evidence type across 18 conditions to compare CNCV with the prior framework and identify performance mechanisms. Complementary inferential and retrieval analyses assess factor contributions, level differences, and the relationship between evidence quality and classifier performance.

  • Experimental design: The factorial design crosses three claim types with six evidence types, yielding 18 experimental conditions.Claim types are raw, LLM-normalized, and human-normalized claims; evidence levels vary by query, retrieval method, and grounding.
  • Experimental design: Comparing two-stage baseline conditions with three-stage configurations tests normalization gains, while gold-input comparisons identify the best automated configuration.The gold-input baseline combines human-normalized claims with human-annotated evidence.
  • Inferential analysis: A two-way repeated-measures ANOVA estimates the independent and interactive effects of claim type and evidence type across classifiers.Greenhouse–Geisser correction and generalized eta-squared quantify within-subject effects and their relative contributions.
  • Inferential analysis: Post hoc paired t-tests with Bonferroni correction compare factor levels, followed by TOST when nonsignificance does not establish equivalence with gold inputs.Hedges’ g is reported as the effect size for paired comparisons.
  • Retrieval analysis: Retrieval paths are evaluated using macro recall, retrieval macro F1, MRR, and NDCG@10, then descriptively compared with classifier macro F1.The analysis uses E-Gold as the reference for B-raw, B-norm, L-raw, and L-norm evidence.

3.5 Metrics

Macro F1 is the primary classification metric, supplemented by accuracy, precision, and recall. The metrics are listed in Table 5, with supplementary reporting for the additional measures.

  • Evaluation metrics: Macro F1 is the primary classification metric because the dataset is dominated by accurate citations.Accuracy, precision, and recall are also computed for completeness and reported in Supplemental Material F.
  • Evaluation metrics: Table 5 lists the evaluation metrics used in the study.

4 RESULTS AND DISCUSSION

CNCV consistently outperforms the prior two-stage framework, with performance gains driven primarily by evidence quality and especially by LLM-based retrieval of normalized claims. The results also identify practical limits: no universal retrieval threshold is established, LLM scores are zero-shot, and large-scale deployment may incur substantial costs.

  • Effectiveness of CNCV: 12% for encoders and 10% for generative LLMs: CNCV improves mean macro F1 over the best prior two-stage configurations across all 18 classifiers.Every classifier improves, and half of the best three-stage configurations match or exceed their gold-input baselines.
  • Effectiveness of CNCV: L-norm is the winning automated evidence condition for all 18 classifiers, while C-Auto is retained as the classification input in 13.This asymmetry indicates that normalization contributes more consistently through retrieval queries than through classifier inputs.
  • Evidence dominance: Evidence quality has a substantially greater effect on classification performance than claim type, with only a small interaction effect between the factors.For LLMs, the evidence-to-claim effect-size ratio is approximately 11:1, compared with 3.7:1 for encoders.
  • Evidence quality and classifier performance: 64.35% macro recall from L-norm yields classifier performance equivalent to E-Gold, whereas lower-recall conditions remain statistically indistinguishable in macro F1.Macro recall rises from 37.92% for B-raw to 45.43% for B-norm and 50.14% for L-raw without corresponding classifier gains.
  • Evidence quality and classifier performance: Classifier performance improves only when retrieved evidence becomes sufficiently complete and well ranked, particularly with a retriever capable of using the normalized query.The LLM retriever shows larger gains across all four retrieval metrics than BM25-CE.
  • Cost and practical feasibility: CNCV retrieval is the primary cost contributor, costing several times as much as the other two stages combined because each cited article's full text is processed.The per-citation cost is affordable for individual users verifying limited papers but may become substantial for publishers and journals at scale.

5 CONCLUSIONS

CNCV’s gains arise mainly from improving retrieved evidence, while claim normalization also contributes at the anchor stage and supports separable system improvements. The study identifies domain and model-family boundaries for generalization and proposes future work on normalization.

  • 5 CONCLUSIONS: 12% average macro F1 gains for encoders and 10% for generative LLMs over the best two-stage framework were observed across all 18 classifiers.
  • 5 CONCLUSIONS: Evidence type explains substantially more variance in macro F1 than claim type, making retrieval quality the primary performance bottleneck.This difference is particularly pronounced among LLMs, and automatically normalized claims produce classification performance statistically equivalent to manually annotated evidence.
  • 5 CONCLUSIONS: The anchor role contributes less than the query role, although stronger LLM classifiers may benefit from higher-quality normalized claims.For generative LLMs, C-Gold outperforms C-Auto even though C-Auto does not significantly outperform C-Raw.
  • 5 CONCLUSIONS: The small interaction between claim type and evidence type suggests that retrieval and classification channels can largely be improved separately.Classifier choice remains consequential: LLMs are not uniformly better than fine-tuned encoders, and performance is not consistently associated with inference cost.
  • 5 CONCLUSIONS: CNCV addresses content-level citation verification, which checks whether cited papers substantiate invoked claims rather than merely whether references and metadata exist.
  • 5 CONCLUSIONS: The corpus covers only Library and Information Science, and claim normalization and LLM-guided retrieval use GPT-5-mini, leaving cross-domain and cross-model generalization untested.The authors identify cross-model replication as an important next step.

Generative AI Use Statement

Generative language models supported claim normalization, evidence retrieval, and zero-shot citation classification alongside encoder-based models. Other listed models were used only for manuscript language editing, which the authors reviewed and approved.

  • Generative AI Use Statement: Generative large language models were used for claim normalization, evidence retrieval, and zero-shot citation classification alongside encoder-based models.
  • Generative AI Use Statement: GPT-5.6 and Claude Opus 4.8 were used solely for language editing, and the authors reviewed and approved those edits.
Loading 2608.30145v1…