Source-linked AI summary
BERT for Evidence Retrieval and Claim Verification
Amir Soleimani, Christof Monz, Marcel Worring
TL;DR
Automatic verification is needed because growing online information has accompanied the spread of fake news and false claims, while evidence-based datasets motivate the FEVER focus. The paper uses separate BERT models for evidence retrieval and claim verification, comparing retrieval losses and hard negative mining. Its system achieves a FEVER score of 69.66 without ensembling and ranks second on the official leaderboard.
Problem
The paper addresses the need for automatic evidence-based verification of claims amid the spread of fake news and false claims.
Method
The system uses one BERT model to retrieve evidence sentences and another to verify claims, comparing pointwise and pairwise losses and hard negative mining.
Results
69.66 FEVER score without ensembling places the large system second on the official leaderboard.
Takeaways & Limitations
Pairwise Ranknet achieves the highest recall, but pairwise methods are not necessarily superior when precision is considered.
Takeaways & Limitations
The paper leaves the effect of online hard negative mining on faster training and end-to-end BERT for the entire FEVER pipeline to future work.
Abstract
from arXiv · showhide
Motivated by the promising performance of pre-trained language models, we investigate BERT in an evidence retrieval and claim verification pipeline for the FEVER fact extraction and verification challenge. To this end, we propose to use two BERT models, one for retrieving potential evidence sentences supporting or rejecting claims, and another for verifying claims based on the predicted evidence sets. To train the BERT retrieval system, we use pointwise and pairwise loss functions, and examine the effect of hard negative mining. A second BERT model is trained to classify the samples as supported, refuted, and not enough information. Our system achieves a new state of the art recall of 87.1 for retrieving top five sentences out of the FEVER documents consisting of 50K Wikipedia pages, and scores second in the official leaderboard with the FEVER score of 69.7.
1 Introduction
The paper addresses automatic claim verification by applying BERT within FEVER’s evidence retrieval and claim verification pipeline. It proposes separate BERT models for retrieving evidence and verifying claims, while comparing retrieval losses and hard negative mining.
- 1 Introduction: FEVER evaluates whether claims are supported, refuted, or unsupported by evidence from 50K Wikipedia pages.The benchmark contains 185K generated claims and requires correct evidence for supported and refuted labels.
- 1 Introduction: BERT is applied to both evidence retrieval and claim verification in a three-step FEVER pipeline.The system retrieves evidence sentences with one BERT model and verifies claims against them with another.
- 1 Introduction: The paper compares pointwise cross-entropy with pairwise Hinge and Ranknet losses for BERT sentence retrieval and examines hard negative mining.A separate BERT model verifies claims using the retrieved evidence sentences.
- 1 Introduction: The work reports second place on the official FEVER leaderboard without ensembling.This is listed among the paper’s contributions.
2 Related Work
Prior FEVER systems combine document and sentence retrieval with claim verification using methods including TF-IDF, logistic regression, ESIM, and attention-based models. The related work motivates comparing BERT with these retrieval and inference approaches.
- 2 Related Work: FEVER systems use a three-step pipeline of document retrieval, sentence retrieval, and claim verification.The pipeline first narrows candidate documents, then extracts evidence sentences, and finally verifies the claim.
- 2 Related Work: Natural language inference determines whether a premise entails a hypothesis, with established benchmarks including SNLI and MultiNLI.The related work places FEVER claim verification alongside broader NLI research.
- 2 Related Work: BERT is a pre-trained language model that has achieved strong results across NLP tasks, including natural language inference.Its relevance follows from the similarity between NLI and FEVER claim verification.
- 2 Related Work: Earlier FEVER retrieval methods include TF-IDF similarity, lexical and syntactic features, exact title matching, and logistic regression.These methods support document or evidence sentence extraction.
- 2 Related Work: ESIM-based systems encode and align claim-evidence sentence pairs with bidirectional recurrent networks and attention mechanisms.ESIM was widely used among FEVER participants, including systems using pairwise training.
3 Methods
The system uses BERT-based document, sentence, and claim-verification stages for FEVER, with separate retrieval and verification models. Sentence retrieval compares pointwise and pairwise objectives, while hard negative mining addresses the abundance of non-evidence sentences.
- 3 Methods: BERT represents sentence pairs with token, sentence, and positional embeddings, transformer encoders, and a classification layer based on the [CLS] representation.The default configuration uses BERT base with 12 layers.
- 3 Methods: The proposed FEVER system uses a three-step pipeline with BERT models for sentence retrieval and claim verification.Documents are retrieved first, potential evidence sentences second, and claims are verified against those sentences third.
- 3.2 Sentence Retrieval: Sentence retrieval ranks candidate evidence sentences using either pointwise classification or pairwise comparison objectives.The pointwise model uses cross-entropy classification, while the pairwise approach compares positive and negative samples with RankNet or modified hinge losses.
- 3.2 Sentence Retrieval: At test time, the retrieval model sorts candidate sentences by output scores and selects the top five, with thresholding available to trade recall against precision.Candidate sentences come from documents retrieved for each claim and may include non-evidence sentences.
- 3.2 Sentence Retrieval: Hard negative mining samples high-loss negatives to reduce training on easy examples despite the high ratio of non-evidence to evidence sentences.For pairwise retrieval, the method selects the hardest pairs and increases training epochs because some positive samples may otherwise be omitted.
- 3.3 Claim Verification: A second BERT model classifies claim-evidence pairs into supported, refuted, or not enough information, and aggregates decisions over the five retrieved sentences.The default decision is not enough information unless supporting or rejecting evidence establishes another label.
4 Results
The BERT retrieval and verification pipeline improves FEVER evidence and claim-verification performance, with results favoring recall-oriented retrieval while showing that thresholding and hard-negative mining affect the precision–recall trade-off.
- Recall is prioritized because retrieved sentences train verification and FEVER requires evidence for supported and refuted claims.
- Pointwise methods surpass pairwise methods in recall–precision performance, although pairwise Ranknet with hard-negative mining attains the highest recall.Figure 5 plots the recall–precision trade-off against competing systems and retrieval variants.
- Hard-negative mining improves both pairwise retrieval methods while preserving pointwise performance.
- BERT verification improves label accuracy and FEVER score even when trained on UKP-Athene’s higher-recall retrieval outputs.Training on BERT retrieval predictions further improves verification by supplying more correct evidence sentences and a better training set.
- The best model ranked second on the official FEVER test leaderboard without ensembling.The test results are reported in Table 3, whose supplied caption identifies the test-set comparison.
5 Conclusion
The paper concludes that BERT is effective for both evidence retrieval and claim verification, while pairwise retrieval is not uniformly superior and hard-negative mining yields only modest gains.
- A BERT system scored second with a FEVER score of 69.66 without ensembling.
- Pairwise Ranknet achieved the highest recall, but pairwise methods were not necessarily superior when precision was considered.
- Hard-negative mining slightly improves retrieval performance without requiring additional GPUs.
- The authors leave its possible effect on faster training and an end-to-end BERT FEVER pipeline for future work.