Combining Fact Extraction and Verification with Neural Semantic Matching Networks

1 Introduction

• FEVER asks systems to verify claims against evidence from about 5 million Wikipedia documents and label them SUPPORTS, REFUTES, or NOT ENOUGH INFO (Thorne et al. 2018).The introduction highlights the difficulty of retrieving evidence efficiently at scale and predicting the evidence–claim relation even when ground-truth evidence is available.Source: §1 · p. 1
• The system connects document retrieval, sentence selection, and claim verification as three stages of a similar neural semantic-matching problem.It ranks documents after keyword matching, selects evidential sentences with an annealed-sampling-trained network, and verifies claims using upstream relatedness features and WordNet knowledge.Source: §1 · pp. 1–2
• Adding sentence-selector relatedness scores and WordNet ontological features improves claim verification over simply passing retrieved evidence to an NLI model.The verifier predicts whether evidence supports or refutes a claim, or is insufficient to decide.Source: §1 · p. 2
• The authors report that neural semantic matching without intermediate term representations significantly outperforms term-weighting methods for evidence retrieval.Pageview frequency also provides comparable and complementary information for document ranking.Source: §1 · p. 2
• The unified system uses raw textual claims and Wikipedia sentences without structured machine-friendly knowledge bases and achieves state-of-the-art results on FEVER.The authors present it as a neural baseline for future large-scale fact-checking work.Source: §1 · p. 2

Open Domain Question Answering

• FEVER shares open-domain QA’s document-retrieval problem, but its end task is claim verification rather than extracting an answer span.Chen et al. (2017a) established the two-stage retrieval and reading-comprehension framework; later work added comprehension feedback or adjusted the number of retrieved documents (Kratzwald and Feuerriegel 2018).Source: §2 · p. 2

Information Retrieval

• Unlike relevance-focused IR, which emphasizes specific-term matching, FEVER retrieval seeks sentences with the same semantic meaning as the claim.The paper therefore approaches retrieval using natural language inference techniques rather than relevance-focused IR methods.Source: §2 · p. 2

Natural Language Inference

• The paper formalizes document retrieval and sentence selection as semantic matching problems and solves all three FEVER subtasks with a homogeneous network.It treats NLI as a typical semantic-matching problem and extends that formulation to retrieval.Source: §2 · p. 2

Automatic Fact Checking Datasets

• Unlike earlier fact-checking datasets and tasks, FEVER jointly requires and measures evidence retrieval and claim verification.The paper characterizes FEVER as the first task with this joint requirement and benchmark evaluation.Source: §2 · p. 2

Other FEVER Systems

• Unlike other top FEVER systems, this method uses homogeneous semantic matching across subtasks and feeds concatenated retrieved sentences with relatedness scores to its verification model.The other systems use different models for different subtasks, compare each evidence sentence with the claim using existing NLI models, and aggregate their outputs.Source: §2 · pp. 2–3

3.1 Task Formalization

• FEVER asks systems to verify arbitrary claims using potential evidence extracted from a large collection of Wikipedia documents, or mark claims non-verifiable when evidence is insufficient.Source: §3.1 · p. 3
• Each example supplies a textual claim and sentences from provided Wikipedia documents; the system returns evidential sentences and a label from S, R, or NEI.Source: §3.1 · p. 3
• A prediction is correct only when its label matches the gold label and its evidence set covers the annotated evidence, with at most five sentences; this is the FEVER Score.Source: §3.1 · p. 3

4 Our Model

• NSMN applies a homogeneous semantic-matching architecture to document retrieval, sentence selection, and claim verification.The section presents the three subtasks as similar matching problems resolved consistently through NSMN.Source: §4 · p. 3

4.1 Neural Semantic Matching Network

• The FEVER blind-test constraint allows claims to be verified with at most five evidence sentences.The paper states this is why the constraint is imposed.Source: §4.1 · p. 3
• NSMN encodes each input sequence with a bidirectional LSTM, then aligns tokens using pairwise scores and weighted representations from the other sequence.The alignment layer combines encoded and aligned representations before matching.Source: §4.1 · p. 3
• The matching layer uses a recurrent network with shortcut inputs, and max-pooling compresses the matched sequences for the output.The shortcut inputs are sub-channels of the original sequences without GloVe, and the pooled vectors, their absolute difference, and their element-wise product are mapped to the output.Source: §4.1 · p. 3
• The model modifies ESIM by adding shortcut connections to the matching layer and using max-pooling plus one rectifier-affine layer at the output.The paper says these modifications were based on validation results.Source: §4.1 · p. 3
• For retrieval and selection, NSMN outputs evidence-selection and discard scores; for verification, it outputs scores for SUPPORTS, REFUTES, and NEI.The three verification labels correspond to predicting whether the claim is supported, refuted, or not enough information is available.Source: §4.1 · p. 4

1. Document Retrieval

• NSMN addresses disambiguative documents, which make up 10% of documents and require semantic understanding for correct retrieval.The system ranks these documents against the claim, while non-disambiguative documents receive the highest priority score.Source: §1 · p. 4
• Keyword matching narrows the search space to an average of 8 pages per claim before semantic matching.This initial step reduces the cost of matching claims against the full document collection.Source: §1 · p. 4
• Each document is represented by its title and first sentence, and NSMN computes its matching score against the claim.The score m+ represents global semantic relatedness: higher values indicate a document is more related to the claim.Source: §1 · p. 4
• The system filters candidates below a probability threshold, then sorts the remaining documents by m+ and adds the top k.NSMN scores are used for disambiguative documents in the candidate set.Source: §1 · p. 4

2. Sentence Selection

• Sentence selection takes a claim and the union of sentences from retrieved documents as input, then outputs a subset as its evidence set.Source: §2 · pp. 4–5
• NSMN scores every sentence in the retrieved documents against the claim, after document retrieval has narrowed the search space.For each sentence, the method calculates p(x = 1 | ci, j) and m+; it filters out sentences below a probability threshold and sorts the remainder by m+.Source: §2 · p. 5
• The method adds the top five sentences ranked by m+ to the resulting list.Source: §2 · p. 5

3. Claim Verification

• The verifier takes the concatenated upstream evidence sentences as premise and the claim as hypothesis, and predicts S, R, or NEI.The model uses GloVe and ELMo embeddings for comprehensive, contextualized lexical representations.Source: §3 · p. 5
• WordNet features encode 10 hypernymy, antonymy, and edge-distance phenomena in 30 dimensions, marking whether each token is in the evidence or claim.Each channel uses three elements to indicate sequence position and whether the feature fires.Source: §3 · p. 5
• Five-dimensional real-value embeddings encode unique number tokens to help the model identify and differentiate numbers.Source: §3 · p. 5
• Two normalized relatedness scores from document and sentence NSMN modules provide evidence-token features, helping verification focus on evidence according to its semantic relatedness to the claim.These scores connect the three subtasks more strongly.Source: §3 · p. 5
• Evidence enhancement optionally augments the evidence set and is applied by default before evaluation.Source: §3 · p. 5

5 Implementation and Training Details

• Document retrieval uses ground-truth evidence documents as positives and other disambiguative documents as negatives, with Adam (Kingma and Ba 2015), batch size 128, and experiments using Pageview resources and TF-IDF reranking after keyword matching.Source: §5 · p. 5
• Sentence selection trains on ground-truth evidence versus candidate-pool negatives with Adam (Kingma and Ba 2015), batch size 128, and annealed sampling that lowers negative inclusion probability from 0.5 by 0.1 per epoch, resetting it to 0.02.The authors also compare TF-IDF and a Max-Pool sentence encoder in place of NSMN.Source: §5 · p. 5
• Claim verification trains on FEVER ground-truth labels with Adam and batch size 32; non-verifiable claims use 3–5 sentences sampled equally from the upstream-selected candidate pool.For verifiable claims, the input is the provided ground-truth supporting or refuting evidence.Source: §5 · p. 5

Document Retrieval Results

• Keyword matching scored 88.86% vs 70.20% oracle at k=5 and 90.69% vs 77.24% at k=10, but lost 25 oracle-score points on the difficult subset.dNSMN outperformed TF-IDF and Pageview on all reported metrics; increasing retrieval from 5 to 10 documents caused only a 1-point retrieval-accuracy drop for dNSMN methods on the full dev set.Source: §6 · p. 6

Sentence Selection Results

• On the full dev set, sNSMN scored 86.65% oracle FEVER versus 84.08% for Max-Pool, but its difficult-set recall was 68.34% versus 73.68%.The difficult subset required greater semantic understanding; annealed sampling was used to improve evidence recall despite reducing evidence F1, which the authors considered important for final FEVER score.Source: §6 · pp. 6–7

Claim Verification Results

• WordNet and number features raised SUPPORTS F1 from 74.7 to 75.7 and REFUTES F1 from 68.0 to 69.4; sentence-selection relatedness improved FEVER by 1 point and NOT ENOUGH INFO F1 by almost 3.The vNSMN scored 66.14% FEVER versus 65.07% for vanilla ESIM; adding document-retrieval relatedness as well as sentence-selection relatedness yielded slightly worse results than sentence-selection relatedness alone.Source: §6 · p. 7

Combination Evaluation and Final Result

• For dev-set final comparisons, the authors selected Pageview + dNSMN over the similar-retrieval KM + dNSMN setup for blind testing.They compared the combinations using the best downstream model; the non-Pageview neural-only system remained comparable.Source: Table 6 · p. 7, §6 · p. 8
• The final Pageview + dNSMN + sNSMN + vNSMN system achieved a new state-of-the-art FEVER blind-test result, comparable to the earlier rank-1 system.On dev results, the authors chose Pageview + dNSMN for document retrieval, although the neural-only model was comparable.Source: Table 7 · p. 8, §6 · p. 8

Claim Verification Results

• Changing the training data for non-verifiable examples significantly reduced FEVER accuracy and NOT ENOUGH INFO F1, underscoring the importance of upstream training-data quality.Source: §6 · p. 8

Noise Tolerance of vNSMN

• Lowering the evidence-filtering threshold from the default 0.5 slightly increased overall FEVER score, suggesting vNSMN tolerated noisy evidence.The system used a threshold of 0.05 for sentence selection, and the authors connected the result to using high-recall evidence.Source: §6 · p. 8

• The system achieves state-of-the-art results on FEVER using Pageview, WordNet, and inter-module features.Source: §7 · p. 8
• Consistent, jointly developed neural semantic matching networks handle document retrieval, sentence selection, and claim verification.Source: §7 · p. 8
• The fact-verification task is organized into three stages: document retrieval, sentence selection, and claim verification.Source: §7 · p. 8

A.1 Dataset Splits and Scoring

• FEVER contains 185,455 annotated claims and 5,416,537 Wikipedia documents with roughly 25 million potential evidence sentences; the dev labels are balanced.The experiments use the FEVER Shared Task dataset partition.Source: §A.1 · p. 9
• A prediction earns FEVER Score only when its label is correct and its evidence set covers the annotated evidence set, with at most five predicted sentences.The paper also evaluates submodules using F1, Precision, Recall, and Accuracy.Source: §A.1 · p. 9

A.2 Keyword Matching

• Document retrieval applies exact title matching, ignoring disambiguation parentheses, retries after removing an initial article, then singularizes claim tokens if needed.Exact matching is case insensitive except for the first letter.Source: §A.2 · p. 9

A.3 Evidence Enhancement

• Two-hop evidence enhancement searches documents hyperlinked from existing evidence, retrieves new evidence with dNSMN and sNSMN, then adds the highest-relatedness sentence.The process iterates over the input sentences in the existing evidence set to build the candidate document set.Source: §A.3 · pp. 9–10
• For the final model, evidence enhancement can improve evidence retrieval by 0.2% OFEVER and improve final FEVER scores by 0.08%, according to the reported results.Source: §A.3 · p. 10

A.4 Example outputs

• The negative example selects evidence correctly but predicts incorrectly because the model misses the exclusiveness of a capitalizing relation and is misled by “capital.”The paper identifies integrating more world knowledge into NLI as future work.Source: §A.4 · p. 10, Figure 4 · p. 10