Source-linked AI summary
GERE: Generative Evidence Retrieval for Fact Verification
Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Yixing Fan, Xueqi Cheng
TL;DR
Fact verification depends on relevant evidence, while common retrieval pipelines require substantial indexing and search and offer limited flexibility in evidence selection. GERE generates document titles and sentence identifiers; on FEVER, it outperforms state-of-the-art baselines and reports lower time and memory costs.
Problem
Common fact-verification retrieval methods require a large document index and costly search, score documents and sentences independently, and select a fixed number of evidence items.
Source: §1 · p. 1Method
GERE uses a transformer encoder–decoder to jointly generate document titles and evidence sentence identifiers, modeling dependencies and varying evidence counts by claim.
Source: §1 · pp. 1–2Results
On FEVER, GERE significantly outperforms state-of-the-art baselines and has lower time costs and memory footprint.
Source: §1 · p. 2Takeaways & Limitations
GERE can be adapted to existing claim-verification models for better claim assessment.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Fact verification (FV) is a challenging task which aims to verify a claim using multiple evidential sentences from trustworthy corpora, e.g., Wikipedia. Most existing approaches follow a three-step pipeline framework, including document retrieval, sentence retrieval and claim verification. High-quality evidences provided by the first two steps are the foundation of the effective reasoning in the last step. Despite being important, high-quality evidences are rarely studied by existing works for FV, which often adopt the off-the-shelf models to retrieve relevant documents and sentences in an "index-retrieve-then-rank" fashion. This classical approach has clear drawbacks as follows: i) a large document index as well as a complicated search process is required, leading to considerable memory and computational overhead; ii) independent scoring paradigms fail to capture the interactions among documents and sentences in ranking; iii) a fixed number of sentences are selected to form the final evidence set. In this work, we propose GERE, the first system that retrieves evidences in a generative fashion, i.e., generating the document titles as well as evidence sentence identifiers. This enables us to mitigate the aforementioned technical issues since: i) the memory and computational cost is greatly reduced because the document index is eliminated and the heavy ranking process is replaced by a light generative process; ii) the dependency between documents and that between sentences could be captured via sequential generation process; iii) the generative formulation allows us to dynamically select a precise set of relevant evidences for each claim. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines, with both time-efficiency and memory-efficiency.
1 INTRODUCTION
Existing fact-verification pipelines depend on retrieved evidence, but common retrieval methods impose costs and constrain evidence selection. GERE generates document titles and sentence identifiers, and FEVER experiments report improved performance, time efficiency, and memory efficiency.
1 INTRODUCTION
- Fact verification aims to assess textual claims using multiple evidential sentences from trustworthy corpora such as Wikipedia.The passage motivates automatic fact checking amid online false information.Source: §1 · p. 1
- Existing index-retrieve-then-rank methods require a large document index, incur search costs, score candidates independently, and select a fixed number of documents and sentences.The passage identifies these as limitations of common evidence retrieval approaches.Source: §1 · p. 1
- GERE jointly generates document titles and evidence sentence identifiers, modeling dependencies and allowing the evidence-set size to vary by claim.Its generative process replaces document-index search and heavy ranking with a lighter process.Source: §1 · pp. 1–2
- On FEVER, GERE significantly outperforms state-of-the-art baselines while using less memory and time.The supplied passage reports these overall outcomes without specifying numerical values.Source: §1 · p. 2
2 RELATED WORK
Prior fact-verification research commonly uses a document-retrieval, sentence-retrieval, and claim-verification pipeline. Generative retrieval models instead predict relevant documents, and this paper adapts a pretrained sequence-to-sequence model to evidence retrieval.
2.1 Fact Verification
- Fact verification commonly uses document retrieval, sentence retrieval, and claim verification, with verification relying on precise retrieved evidence.The passage also describes graph reasoning and pretrained language models in recent claim-verification work.Source: §2.1 · p. 2
- Existing document-retrieval methods use mention-, keyword-, or feature-based approaches, while sentence retrieval uses probabilistic, neural, or pretrained-model ranking.The passage notes that these methods generally require a large document index and model sentence relevance independently, limiting flexible sentence selection.Source: §2.1 · p. 2
2.2 Generative Retrieval Models
- Generative retrieval predicts relevant documents from a query, and this paper makes a first attempt to adapt a pretrained sequence-to-sequence model to FV evidence retrieval.The passage contrasts this approach with the commonly adopted index-retrieve-then-rank blueprint.Source: §2.2 · p. 2
3 OUR APPROACH
GERE is a generative framework for fact-verification evidence retrieval, with a claim encoder and dependent decoders that generate document titles and evidence sentence identifiers.
3.1 Model Overview
- GERE uses a claim encoder, a title decoder that generates relevant documents, and an evidence decoder that generates sentence identifiers based on those documents.Its target is a relevant document and evidence set, rather than the broader sets sought by existing retrieval methods.Source: §3.1 · p. 2
3.2 Claim Encoder
- The claim encoder maps the input claim to a compact representation using BART’s bidirectional Transformer encoder.Source: §3.2 · p. 2
3.3 Title Decoder
- The title decoder generates titles sequentially, conditioning each new title on the claim and previously generated titles, and can predict a different number of documents per claim.Constrained beam search restricts generated titles to the corpus’s document-title set.Source: §3.3 · pp. 2–3
3.4 Evidence Decoder
- Given the claim and relevant documents, the evidence decoder generates sentence identifiers from candidate sentences, modeling dependencies among evidence and allowing a dynamic evidence set.At inference, it uses greedy decoding because the search space is small.Source: §3.4 · p. 3
3.5 Model Training
- GERE fine-tunes BART by jointly maximizing the title-generation and evidence-generation objectives, integrating document and sentence retrieval for use with claim-verification models.The learned framework can provide relevant documents and evidence to existing claim-verification models.Source: §3.5 · p. 3
4 EXPERIMENTS
On FEVER, GERE is evaluated for document and sentence retrieval, claim verification, and resource efficiency. It retrieves more precise evidence with strong retrieval F1, improves verification results across tested models, and reduces memory footprint and inference time.
4.1 Experimental Settings
- FEVER contains 185,455 annotated claims and 5,416,537 Wikipedia documents; each claim averages 1.28 relevant documents and 1.86 evidence sentences.Claims are labeled SUPPORTS, REFUTES, or NOT ENOUGH INFO.Source: §4.1 · p. 3
4.2 Evaluation Metrics
- Retrieval is evaluated with precision, recall, and F1, while claim verification uses FEVER and Label Accuracy, which does not assess evidence validity.F1 is the main retrieval metric; FEVER requires the predicted evidence to cover the gold evidence.Source: §4.2 · p. 3
4.3 Results on Document Retrieval
- GERE significantly outperforms state-of-the-art methods on document-retrieval F1, while its higher precision comes with lower recall.GERE averages 1.91 retrieved documents per claim; the authors suggest its smaller, more precise sets may lower recall.Source: §4.3 · p. 4, Table 1 · p. 4
4.4 Results on Sentence Retrieval
- GERE achieves the best sentence-retrieval precision and F1, but lower recall than baselines.It generates an average of 2.42 sentences; DQN has slightly higher recall but substantially lower precision and F1.Source: §4.4 · p. 4, Table 2 · p. 4
4.5 Results on Claim Verification
- All tested claim-verification models significantly outperform their original versions when given GERE’s evidence.The authors report that original evidence can contain conflicting support and refutation, while GERE evidence is more precise and consistent.Source: §4.5 · pp. 4–5, Table 3 · p. 4
4.6 Memory and Inference Efficiency
- GERE reduces memory footprint and document-retrieval inference time, with a title prefix tree and model parameters replacing a large document index and per-document dense vectors.The authors suggest this makes the framework suitable for resource-limited platforms.Source: §4.6 · p. 5, Table 4 · p. 5
5 CONCLUSION
GERE jointly generates documents and sentences for fact verification, with claimed gains in efficiency, dependency modeling, and dynamic evidence selection. FEVER experiments support its effectiveness, while the authors identify single-decoder retrieval and fully generative fact verification as future directions.
5 CONCLUSION
- GERE jointly addresses document and sentence retrieval for fact verification with a generation-based framework.The conclusion characterizes it as a novel framework.Source: §5 · p. 5
- The generative formulation offers improved time and memory efficiency, models document and sentence dependencies, and allows a dynamic number of evidences.The conclusion also says GERE can be adapted to existing claim-verification models.Source: §5 · p. 5
- FEVER experiments demonstrate GERE’s effectiveness; future work includes using one decoder for both retrieval stages and building a fully generative fact-verification system.Source: §5 · p. 5