PaperlayerView exact revision on arXiv

Source-linked AI summary

R2VC: Modular Fact-Checking with Retrieval, Verification, and Confidence Calibration

Dhruv Dixit, Paritosh Pandey

arXiv:2609.11955v1cs.CLcs.LG

TL;DR

R2VC addresses the difficulty of diagnosing and trusting end-to-end LLM fact checkers that entangle retrieval, reasoning, and uncertainty estimation. It modularizes retrieval, candidate generation, external verification, and sequence-level calibration to produce cited verdicts with abstention. The full pipeline improves predictive performance and confidence reliability, while retrieval—especially wrong-entity evidence—remains the dominant bottleneck.

  • Problem

    End-to-end LLM fact checking entangles retrieval, reasoning, and uncertainty estimation, making failures difficult to diagnose and confidence difficult to trust.

  • Method

    R2VC combines hybrid Wikipedia retrieval, SFT+DPO candidate generation, external NLI-based selection, and sequence-level calibration for cited verdicts and abstention.

  • Results

    The full R2VC model performs best across FEVER and VitaminC metrics, while verifier-based selection, calibration, and multi-sample generation produce the largest ablation losses.

  • Takeaways & Limitations

    Modular fact-checking improves predictive accuracy and confidence reliability, but calibration cannot fully compensate for retrieval failures involving wrong-entity evidence.

  • Takeaways & Limitations

    The paper identifies retrieval, especially wrong-entity evidence, as the main bottleneck and proposes broader model and benchmark evaluation as future work.

Abstract

from arXiv · show

Large language models are increasingly used for automated fact checking, but end-to-end prompting often entangles evidence retrieval, reasoning, and uncertainty estimation, making failures difficult to diagnose and confidence difficult to trust. We present R2VC, a modular retrieve, reason, verify, calibrate architecture for evidence-grounded fact checking with citations and abstention. R2VC combines hybrid sparse+dense retrieval over Wikipedia, a supervised fine-tuned and DPO-aligned generator that produces diverse structured verdict candidates, an external NLI cross-encoder for evidence-based candidate selection, and a lightweight sequence-level calibrator for confidence estimation and selective abstention. On FEVER, an 8B backbone with R2VC achieves 13.74% higher accuracy than baseline. Ablation studies show that verifier-based candidate selection and confidence calibration are the largest contributors to performance. Removing candidate selection drops FEVER accuracy to 76.24%, while removing calibration nearly doubles the Brier score to 0.161. A manual analysis of 250 errors further shows that retrieval failures, especially wrong-entity evidence, remain the dominant bottleneck. Together, these results show that modular fact-checking pipelines can substantially improve both predictive accuracy and confidence reliability in open-domain verification.

1 Introduction

R2VC addresses brittle, hard-to-diagnose end-to-end fact checking by separating retrieval, reasoning, verification, and calibration. The resulting modular pipeline targets evidence-grounded verdicts with citations and calibrated abstention.

  • End-to-end fact checking remains brittle with incomplete, noisy, or adversarial evidence and entangles retrieval, reasoning, and confidence estimation.This coupling makes verdict correctness, citation faithfulness, and confidence difficult to guarantee or diagnose.
  • R2VC proposes a modular retrieve, reason, verify, calibrate pipeline for evidence-grounded fact checking.The system is designed to predict veracity, return minimally sufficient citations, and expose calibrated correctness probabilities for selective prediction and abstention.
  • Retrieval mistakes, especially wrong-entity evidence, can produce confident errors, while disagreement signals do not directly enforce entailment or contradiction against citations.
  • R2VC is motivated by modular interfaces that make components easier to optimize independently and support tuning under different deployment constraints.The paper evaluates the pipeline on Wikipedia-based fact verification benchmarks.

2 Related Work

Related work develops retrieval-augmented, interactive, and consistency-based approaches to fact checking, while highlighting persistent retrieval, evidence-selection, and calibration challenges. R2VC is situated among these efforts as a modular alternative with explicit verification and calibration stages.

  • Benchmarks and attribution: Benchmarking work frames verification around sentence- or document-level evidence, with retrieval and long-context evidence selection remaining key entailment bottlenecks.
  • Retrieval-augmented fact-checking: Retrieval-augmented systems condition fact-checking outputs on Wikipedia evidence and add stages such as self-critique, revision, or reranking.Examples include Self-RAG, RARR, and Re2G.
  • Modular pipelines vs. tightly-coupled search loops: Interactive and agent-style frameworks couple search, query planning, and verification decisions to mitigate misleading evidence and reduce unnecessary searches.
  • Consistency, calibration, and alignment: Consistency-based methods use generation disagreement as a factual unreliability signal, while calibration research shows post-hoc fixes can remain fragile under distribution shift.
  • Consistency, calibration, and alignment: R2VC combines supervised fine-tuning with DPO and trains a Sequence Likelihood Calibration model to support abstention.

3 Methodology

R2VC decomposes fact checking into retrieval, candidate generation, external verification, and confidence calibration over standardized Wikipedia-based datasets. Its modular pipeline combines hybrid evidence retrieval, diverse structured outputs, verifier-based selection, and sequence-level calibration for abstention.

  • Overview: R2VC uses four stages—hybrid retrieval, candidate generation, external verification, and confidence calibration—to produce evidence-grounded verdicts with calibrated confidence and abstention.The system retrieves Wikipedia evidence, generates structured candidates, selects among them with an NLI verifier, and applies SLC.
  • Datasets: VitaminC supplies supervised training, FEVER supplies evaluation, and both are standardized into a shared ternary claim–evidence–label representation.VitaminC contains claim, single-sentence evidence, and labels mapped to SUPPORTS, REFUTES, and NEI; FEVER retains NOT ENOUGH INFO and flattens evidence sets.
  • Hybrid retrieval: Hybrid retrieval min–max normalizes sparse and dense scores, combines them with interpolation weight α, and retains the top-K evidence items for downstream stages.Missing scores are treated as zero, and later stages operate only on the fused evidence pool.
  • Generator training and candidate generation: The generator produces k = 16 candidates from random evidence subsets and diverse decoding configurations, exposing outputs to different evidence mixtures and reasoning paths.Candidates are structured verdicts with labels, rationales, and citations; the generator is trained with supervised fine-tuning and DPO to improve label correctness and citation alignment.
  • External verification: An external NLI cross-encoder scores candidate entailment, contradiction, coverage, label consistency, and citation alignment before selecting a candidate.The verifier retains its scalar score and component signals for later calibration.
  • Sequence likelihood calibration: A logistic-regression calibrator combines generation, verification, agreement, evidence-use, rationale-length, and citation features to estimate correctness probability.The calibrator is trained on a held-out slice of 10,000 VitaminC examples, while the modular design supports clean component ablations and upgrades.

4 Results

R2VC improves predictive performance and confidence reliability across FEVER and VitaminC, including strong results from 8B backbones and competitiveness with larger systems.

  • 84.71% FEVER accuracy and 85.68 Macro-F1 make Llama-3.1-Nemotron-8B + R2VC the strongest locally evaluated 8B model.Its FEVER Brier and ECE are 0.083 and 0.0125, respectively.
  • Llama-3.1-Nemotron-8B + R2VC trails Llama2-70B by 1.69 FEVER accuracy points and GPT-4 by 4.49 while outperforming several smaller baselines.It clearly outperforms Llama2-13B, Mistral-7B, and BLOOM on FEVER.
  • 99.78% VitaminC accuracy and 99.82 Macro-F1 show an even stronger R2VC advantage for Llama-3.1-Nemotron-8B.The corresponding Brier and ECE are 0.073 and 0.0077.
  • Lower Brier and ECE values show that R2VC improves confidence alignment alongside classification quality in a fixed 8B-parameter budget.This supports more reliable selective prediction and abstention.

5 Ablation Study

Ablations identify verifier-based selection, multi-sample generation, and confidence calibration as the largest contributors to R2VC’s predictive and calibration gains.

  • 84.71% FEVER accuracy and 99.78% VitaminC accuracy are achieved by the full model, which performs best across all four reported metrics.The full model also reaches 85.68/99.82 Macro-F1 and 0.083/0.073 Brier on FEVER/VitaminC.
  • Removing DPO causes smaller but consistent degradation, reducing FEVER accuracy from 84.71% to 83.02% and VitaminC accuracy from 99.78% to 97.78%.The ablation also worsens calibration on both datasets.
  • Removing verifier-based candidate selection reduces FEVER accuracy to 76.24% and VitaminC accuracy to 89.80%, among the largest ablation losses.FEVER Macro-F1, Brier, and ECE become 77.11, 0.093, and 0.0140.
  • Single-candidate generation lowers FEVER accuracy to 78.78% and VitaminC accuracy to 92.80%, showing the value of multi-sample generation with verifier selection.The corresponding FEVER and VitaminC Macro-F1 values are 79.68 and 92.83.
  • Disabling confidence calibration nearly doubles FEVER Brier from 0.083 to 0.161 and raises ECE from 0.0125 to 0.0312.Accuracy and Macro-F1 also fall to 77.48% and 77.16 on FEVER.
  • Relative to the NLI-only baseline, the full model gains 10.17 FEVER accuracy points and 11.97 VitaminC accuracy points while reducing Brier and ECE.The strongest gains are attributed to candidate selection, multi-sample generation, and calibration.

6 Failure Modes and Confidence Diagnostics

The error analysis identifies retrieval, especially wrong-entity evidence, as the dominant failure source, while confidence patterns distinguish retrieval, evidence, verifier, and calibration problems.

  • Selective prediction: Selective prediction evaluates outputs above confidence threshold τ through coverage and risk, summarized by the area under the risk–coverage curve.The covered set is C(τ) = {n : πn ≥ τ}.
  • Failure distribution: Retrieval errors dominate the annotated failures, making evidence acquisition the primary bottleneck relative to generation, verification, and citation selection.The analysis used a stratified sample of 250 erroneous predictions across confidence, verifier score, label, and dataset/source.
  • Failure distribution: Wrong-entity pages primarily drive retrieval failures, while verifier errors often arise from NLI label flips that select incorrect candidates.
  • Confidence diagnostics: Retrieval failures are the most overconfident because internal claim–evidence alignment can remain strong even when retrieved passages are irrelevant or entity-mismatched.
  • Confidence diagnostics: Insufficient or ambiguous evidence produces the lowest confidence and frequently triggers abstention, whereas verifier and calibration errors remain concentrated at higher confidence.

7 Conclusion and Future Work

R2VC combines modular retrieval, generation, verification, and calibration to support evidence-grounded fact checking with citations and abstention. The authors identify retrieval quality, evidence selection, and verifier robustness as the main improvement priorities and propose broader evaluations.

  • Conclusion: R2VC combines hybrid Wikipedia retrieval, SFT+DPO candidate generation, external NLI selection, and sequence-level calibration for fact checking with calibrated abstention.
  • Conclusion: Retrieval remains the main bottleneck, especially wrong-entity evidence that produces confident mistakes, while insufficient evidence more often yields low confidence and abstention.
  • Conclusion: The authors identify retrieval quality, evidence selection, and verifier robustness as the main drivers of further improvement.
  • Future Work: Future work will evaluate broader model families, harder benchmarks, richer error taxonomies, targeted training signals, and dynamic web-based retrieval.

Disclosure of LLM Use

An LLM-based writing assistant revised portions of the manuscript prose from author-written text. The authors retained responsibility for the scientific content and manually checked the revisions for unsupported material.

  • Disclosure of LLM Use: An LLM-based writing assistant revised and improved portions of the manuscript prose based on text originally written by the authors.
  • Disclosure of LLM Use: The authors retained full responsibility for the paper’s research ideas, technical approach, experiments, results, and conclusions.
  • Disclosure of LLM Use: The revisions were inspected for accuracy, and the manuscript was manually checked for hallucinated, fabricated, or unsupported content.

D Proposed Expanded Error Taxonomy

The proposed taxonomy supports finer-grained manual error analysis and subtype-specific training signals.

  • D Proposed Expanded Error Taxonomy: The expanded taxonomy is intended for finer-grained manual error analysis.
  • D Proposed Expanded Error Taxonomy: The taxonomy organizes error analysis around more specific subtypes.
  • D Proposed Expanded Error Taxonomy: Subtype-specific taxonomy entries are intended to provide training signals.

E Error Labels Used in This Work

The work uses a compact error taxonomy for manual annotation, alongside figures examining claim–evidence length and SFT training dynamics.

  • The manual analysis uses a compact error taxonomy with primary categories and coarse subtypes.
  • Figure F.1 compares claim and evidence lengths in words for a random VitaminC subsample.
  • SFT training shows a sharp early loss reduction followed by a smooth decline without divergence or late-stage overfitting.The loss falls from O(1) initialization into the 10^-2 range.

F.2.2 DPO training dynamics

DPO converges rapidly and separates chosen from rejected responses, with widening reward and log-probability margins under a short, low-learning-rate run.

  • DPO optimization: DPO loss falls from ≈0.16 at step 20 to near-zero by step 40 and remains small in the final epoch-level summary.The final summary is around 2.5 × 10^-2 at step 125.
  • Reward behavior: Chosen rewards rise from roughly +1.5 to +4.0 while rejected rewards fall from roughly −1.5 to below −6.5.
  • Preference separation: The widening chosen–rejected reward margin reaches above 10 as preference accuracy rises from ≈0.95 to 1.0 by step 40.
  • Optimization signals: Gradient norms collapse from roughly 1.1 × 10^-1 to O(10^-3), while the chosen–rejected log-probability gap grows from about 8 to above 45 nats.The run uses a learning rate beginning around 4 × 10^-6 and annealing toward zero over approximately 120 steps.
Loading 2609.11955v1…