Source-linked AI summary
Combining Fact Extraction and Verification with Neural Semantic Matching Networks
Yixin Nie, Haonan Chen, Mohit Bansal
TL;DR
Misinformation has motivated automatic fact checking, with FEVER providing a benchmark for verifying claims against Wikipedia evidence. The paper connects retrieval and verification through homogeneous neural semantic matching networks, and reports state-of-the-art FEVER results.
Problem
FEVER requires reliable claim verification and evidence retrieval from a very large Wikipedia corpus, challenging systems to connect both tasks.
Method
The paper uses connected neural semantic matching networks for document retrieval, sentence selection, and claim verification, adding Pageview, relatedness, and WordNet features.
Results
The unified system achieves state-of-the-art results on the FEVER task, while semantic relatedness features improve FEVER score by 1 point and NOT ENOUGH INFO F1 by almost 3 points.
Takeaways & Limitations
Formalizing all three FEVER subtasks as semantic matching supports a unified fact-extraction and verification system.
Takeaways & Limitations
The FEVER shared-task setting constrains blind-test claims to those verifiable with at most five evidence sentences.
Abstract
from arXiv · showhide
The increasing concern with misinformation has stimulated research efforts on automatic fact checking. The recently-released FEVER dataset introduced a benchmark fact-verification task in which a system is asked to verify a claim using evidential sentences from Wikipedia documents. In this paper, we present a connected system consisting of three homogeneous neural semantic matching models that conduct document retrieval, sentence selection, and claim verification jointly for fact extraction and verification. For evidence retrieval (document retrieval and sentence selection), unlike traditional vector space IR models in which queries and sources are matched in some pre-designed term vector space, we develop neural models to perform deep semantic matching from raw textual input, assuming no intermediate term representation and no access to structured external knowledge bases. We also show that Pageview frequency can also help improve the performance of evidence retrieval results, that later can be matched by using our neural semantic matching network. For claim verification, unlike previous approaches that simply feed upstream retrieved evidence and the claim to a natural language inference (NLI) model, we further enhance the NLI model by providing it with internal semantic relatedness scores (hence integrating it with the evidence retrieval modules) and ontological WordNet features. Experiments on the FEVER dataset indicate that (1) our neural semantic matching method outperforms popular TF-IDF and encoder models, by significant margins on all evidence retrieval metrics, (2) the additional relatedness score and WordNet features improve the NLI model via better semantic awareness, and (3) by formalizing all three subtasks as a similar semantic matching problem and improving on all three stages, the complete model is able to achieve the state-of-the-art results on the FEVER test set.
1 Introduction
FEVER frames automatic fact checking as finding evidential sentences in Wikipedia and labeling claims as SUPPORTS, REFUTES, or NOT ENOUGH INFO. The paper proposes connected neural semantic matching networks for retrieval and verification.
- Misinformation concerns motivate machine-learning approaches to automatic fact checking.
- FEVER requires systems to verify claims using evidence from about 5 million Wikipedia documents and assign one of three labels.
- The proposed system connects document retrieval, sentence selection, and claim verification through three homogeneous neural semantic matching networks.
- Claim verification integrates retrieval-relatedness scores and WordNet ontological features into the neural inference model.
- Neural semantic matching replaces pre-designed term-vector matching for evidence retrieval, while Pageview frequency supplies complementary document-ranking information.
2 Related Works
FEVER combines large-scale evidence retrieval with claim verification, linking it to open-domain question answering and information retrieval while emphasizing semantic rather than term-focused matching.
- Open Domain Question Answering: FEVER shares open-domain question answering’s retrieval problem but ends with claim verification instead of answer extraction.
- Information Retrieval: Information retrieval commonly emphasizes relevance matching in which specific terms play an important role.
- Information Retrieval: The paper approaches FEVER retrieval through natural language inference because evidential sentences should share semantic meaning with claims.
- Compared with other leaderboard systems, the proposed method uses one homogeneous semantic matching network across all subtasks and concatenates retrieved sentences with relatedness scores for verification.
3 FEVER: Fact Extraction and VERification
FEVER asks systems to extract evidence from Wikipedia and use it to classify arbitrary claims as supported, refuted, or non-verifiable. A correct prediction requires both the label and an adequate evidence set.
- FEVER evaluates verification of arbitrary claims using potential evidence extracted from a large collection of Wikipedia documents.
- Each Wikipedia document is represented as an array of sentences, including its title as the first sentence.
- The system receives a claim and the sentences from provided documents, then predicts evidential sentences and a claim label.
- The possible labels are SUPPORTS, REFUTES, and NOT ENOUGH INFO.
- A verification is correct only when the predicted evidence contains the ground-truth evidence and the predicted label matches the ground-truth label.
4 Our Model
The model uses one homogeneous Neural Semantic Matching Network architecture to address document retrieval, sentence selection, and claim verification as related semantic-matching tasks. It combines sequence alignment and matching with task-specific outputs for evidence selection or three-way claim labels.
- Architecture: The system applies a homogeneous NSMN architecture across document retrieval, sentence selection, and claim verification.Each subtask is treated as a similar semantic-matching problem within the connected system.
- Encoding and alignment: NSMN encodes two input sequences with bidirectional LSTMs before computing token-level cross-sequence alignment.Alignment scores connect tokens across the two sequences, and weighted sums provide relevant semantic content from the other sequence.
- Matching and output: The matching layer processes compound aligned representations with a recurrent network, while shortcut connections expose additional input vectors to matching.The model’s output layer max-pools the matching sequences and combines the resulting vectors with their absolute difference and element-wise multiplication.
- Model modification: NSMN modifies ESIM by adding input-to-matching shortcuts and replacing its output layer with max-pooling followed by one rectifier affine layer.The paper states these changes are based on validation results.
- Task-specific outputs: Extraction outputs scores for selecting or discarding evidence, whereas verification outputs scores for SUPPORTS, REFUTES, and NOT ENOUGH INFO.The verification vector contains one score for each of the three FEVER labels.
1. Document Retrieval
Document retrieval first narrows the Wikipedia search space with keyword matching, then uses NSMN scores to rank candidate documents, especially those with disambiguative titles. The highest-ranked documents are retained after thresholding and sorting.
- Candidate generation: About 10% of documents have disambiguation information in their titles, requiring semantic understanding for correct retrieval.The paper calls these documents “disambiguative” documents.
- Candidate generation: Keyword matching builds a candidate subset because semantic matching against the entire Wikipedia collection is computationally intractable.Keyword matching returns an average of 8 pages per claim.
- Semantic ranking: The NSMN represents each document by concatenating its title and first sentence, then computes its matching score against the claim.Disambiguative documents are ranked by comparison with the claim using NSMN.
- Semantic ranking: The score m+ represents a document’s global semantic relatedness to the claim, with higher values indicating greater relatedness.The normalized probability p(x = 1 | ci, j) is derived from the positive and negative scores.
- Selection: Documents with normalized selection probability below P d are filtered out, and the remaining documents are sorted by m+.The top k documents by m+ are added to the resulting list.
2. Sentence Selection
Sentence selection extracts evidence from retrieved documents by comparing every candidate sentence with the claim using NSMN. It filters low-probability sentences and retains the five highest-ranked candidates.
- Evidence extraction: Sentence selection takes a claim and the union of sentences from retrieved documents as input, producing an evidence subset.The evidence set is drawn from sentences associated with the documents selected during retrieval.
- Semantic matching: Because document retrieval has narrowed the search space, the system traverses all retrieved sentences and compares each with the claim using NSMN.This treats sentence selection as semantic matching between candidate sentences and the claim.
- Filtering and ranking: Sentences with p(x = 1 | ci, j) below P s are filtered out before ranking.The probability and m+ value are calculated for every sentence in the retrieved documents.
- Filtering and ranking: The system sorts the remaining sentences by m+ and adds the top 5 sentences to the resulting evidence list.The five-sentence limit is consistent with the FEVER blind-test evidence constraint.
3. Claim Verification
Claim verification performs logical inference from evidential sentences to assign a claim label, using a neural semantic matching model enhanced with retrieval-relatedness and token-level features.
- Claim verification maps an evidence set and claim to one of SUPPORTS, REFUTES, or NOT ENOUGH INFO.The evidence set is denoted Eci, and the output label y belongs to {S, R, NEI}.
- The verifier concatenates upstream evidence sentences as its premise and uses the claim as its hypothesis in an NLI-like architecture.
- WordNet contributes 30-dimensional ontological indicators covering hypernymy, antonymy, and edge-distance phenomena.The indicators are organized into 10 embedding channels and encode whether a relation is present across the evidence and claim sequences.
- GloVe and ELMo provide the verifier’s lexical representations alongside the task-specific token features.Their combination is described as providing comprehensive and contextualized lexical representations.
- Evidence enhancement can optionally augment the current evidence set before evaluation.
5 Implementation and Training Details
Document retrieval is trained as neural ranking over keyword-matched candidates, while experiments compare neural, Pageview, and TF-IDF reranking methods across retrieval settings.
- The document retrieval network optimizes cross-entropy using ground-truth disambiguative documents as positives and other disambiguative documents as negatives.Training uses Adam with batch size 128.
- Retrieval experiments compare keyword matching, dNSMN, Pageview frequency, and TF-IDF reranking methods using k retrieved documents.The difficult subset contains examples with at least one evidence sentence in a disambiguative document and covers more than 10% of the development set.
6 Results and Analysis
The experiments compare retrieval, sentence selection, verification, and complete-system variants on FEVER, showing benefits from neural matching, semantic features, and Pageview reranking.
- Document Retrieval: 25 points: keyword matching loses this much oracle score on the difficult document-retrieval subset.The difficult subset contains examples whose evidence is in disambiguative documents.
- Document Retrieval: dNSMN outperforms Pageview frequency and TF-IDF reranking on all document-retrieval metrics.The comparison covers the entire development set and a difficult subset.
- Sentence Selection: 86.65%: normally trained sNSMN’s oracle FEVER score on the full development set, versus 84.08% for the Max-Pool sentence encoder.On the difficult subset, however, sNSMN recall is 68.34%, below the encoder’s 73.68%.
- Sentence Selection: Annealed sampling improves evidence recall but reduces evidence F1, and the recall gain is important for final FEVER Score.This trade-off motivates its use during sentence-selection training.
- Claim Verification: WordNet and number features raise SUPPORTS F1 from 74.7 to 75.7 and REFUTES F1 from 68.0 to 69.4.Adding sentence-selection semantic relatedness improves FEVER score by 1 point and NOT ENOUGH INFO F1 by almost 3 points.
- Final System: The final Pageview + dNSMN + sNSMN + vNSMN system achieves new state-of-the-art performance on the FEVER blind test.Its results are comparable to the authors’ earlier rank-1 shared-task system.
7 Conclusion
The paper addresses FEVER with consistent neural semantic matching networks across document retrieval, sentence selection, and claim verification, supplemented by Pageview, WordNet, and inter-module features.
- The unified three-stage system combines neural semantic matching with Pageview, WordNet, and inter-module features to achieve state-of-the-art FEVER performance.
A Supplemental Material
The supplemental material describes dataset evaluation, evidence-enhancement procedures, keyword-based retrieval rules, and representative successes and limitations of the system.
- Dataset and evaluation: 185,455 annotated claims and 5,416,537 Wikipedia documents provide roughly 25 million potential evidence sentences for FEVER.
- Dataset and evaluation: FEVER Score requires both the predicted label and an evidence set of at most five sentences to match the annotated evidence.Additional analyses use F1, Precision, Recall, and Accuracy for individual modules.
- Keyword matching: Document retrieval applies exact title matching, removes leading articles when needed, and singularizes claim tokens if earlier matching returns no title.
- Evidence enhancement: Two-hop evidence selection adds documents hyperlinked from existing evidence, reruns document and sentence matching, and adds the highest-scoring enhancement sentence.
- Evidence enhancement: 0.2% OFEVER improvement for evidence retrieval and 0.08% improvement in final FEVER scores result from evidence enhancement.
- Example outputs: The model can select evidence correctly yet fail verification when reasoning requires world knowledge about the exclusiveness of a capitalization relation.The authors identify integrating more world knowledge into NLI as future work.