Source-linked AI summary
UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification
Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, Iryna Gurevych
TL;DR
Automated fact-checking requires systems that retrieve evidence and verify claims at scale, as manual checking is no longer feasible. UKP-Athene uses entity linking for document retrieval and two ESIM extensions for ranking evidence and classifying claims from multiple sentences, placing third among 23 FEVER teams. The authors report significant gains over organizer baselines across all subtasks and about 100% improvement for the whole pipeline.
Problem
The FEVER task addresses scalable verification of online claims by retrieving supporting or refuting facts from raw text.
Method
The pipeline uses entity linking for document retrieval and two ESIM extensions for ranking candidate sentences and classifying claims from multiple selected facts.
Results
Third among 23 teams, the system significantly improved over baselines on each subtask, with about 100% improvement for the whole pipeline.
Takeaways & Limitations
The approach provides a competitive FEVER pipeline combining entity-linked retrieval with multi-sentence entailment modeling.
Takeaways & Limitations
The system struggles with unresolved entity references, numerical values, and challenging NotEnoughInfo cases.
Abstract
from arXiv · showhide
The Fact Extraction and VERification (FEVER) shared task was launched to support the development of systems able to verify claims by extracting supporting or refuting facts from raw text. The shared task organizers provide a large-scale dataset for the consecutive steps involved in claim verification, in particular, document retrieval, fact extraction, and claim classification. In this paper, we present our claim verification pipeline approach, which, according to the preliminary results, scored third in the shared task, out of 23 competing systems. For the document retrieval, we implemented a new entity linking approach. In order to be able to rank candidate facts and classify a claim on the basis of several selected facts, we introduce two extensions to the Enhanced LSTM (ESIM).
1 Introduction
The FEVER task addresses the need to automate fact-checking by evaluating claims against extracted Wikipedia evidence. UKP-Athene’s system ranked third among 23 competing teams.
- False or misleading online content has increased, making fact-checking increasingly important.
- Manual fact-checking is no longer feasible because large numbers of fake-news and hyperpartisan articles are published daily.
- FEVER provides 185,445 Wikipedia-based claims with evidence sets supporting, contradicting, or failing to determine each claim’s veracity.
- The shared task comprises document retrieval, sentence selection, and claim classification.
- Third among 23 competing teams, UKP-Athene presented a claim-verification pipeline for the FEVER task.
2 Background
The system treats Wikipedia-based document retrieval as entity linking and uses ESIM as the basis for sentence selection and textual entailment. ESIM builds representations of statement pairs through sequential encoding and inference composition.
- Document retrieval can be framed as entity linking because Wikipedia article titles identify the entities described by articles.
- The linked Wikipedia articles become the retrieved documents supplied to subsequent pipeline steps.
- ESIM, originally developed for SNLI, creates rich representations of statement pairs and is adapted here for claim-sentence pairs.
- ESIM uses BiLSTM-based input encoding followed by inference composition, pooling, concatenation, and multilayer-perceptron classification.
3 Our system for fact extraction and claim verification
UKP-Athene combines entity-linking retrieval with two ESIM extensions: one ranks candidate evidence sentences, and the other classifies claims from multiple selected sentences.
- 3.1 Document retrieval: The retrieval component identifies claim entities and links them to matching Wikipedia article titles.
- 3.1 Document retrieval: Potential entity mentions include noun phrases, words before the main verb, and the whole claim to capture varied entity categories.
- 3.1 Document retrieval: Retrieved articles for all identified mentions are filtered and supplied to the next pipeline step.
- 3.2 Sentence selection: The sentence-selection model extends ESIM to produce ranking scores for claim-sentence pairs and selects the five highest-ranked sentences.
- 3.3 Recognizing textual entailment: The claim-verification model extends ESIM to predict Supported, Refuted, or NotEnoughInfo from multiple input sentences.
- 3.3 Recognizing textual entailment: Attention and pooling compress the five claim-sentence representations into one vector emphasizing information relevant to claim classification.
4 Results
Experiments examine how retrieval and sentence-selection depth affect performance, while comparison with organizer baselines shows substantial gains across the FEVER subtasks.
- Larger numbers of retrieved Wikipedia articles benefit both document retrieval and sentence selection systems.
- The textual-entailment model performs best when using all five selected sentences.
- Table 3 compares the three systems and full pipeline with organizer baselines on the development set.
- The systems significantly improve upon the baseline on each FEVER subtask.
- About 100% improvement over the baseline pipeline is reported for the whole pipeline.
5 Error analysis
The error analysis identifies recurring failures in document retrieval, sentence selection, and textual entailment. These include retrieval mismatches, evidence sets with weakly related sentences, and difficulty interpreting numbers or incomplete information.
- Document retrieval: Document retrieval fails on spelling errors, missing entity mentions, and disambiguated article titles that search does not return.These errors can cause relevant articles to be discarded or omitted before later pipeline stages.
- Sentence selection: Sentence selection often misses evidence when the claim’s entity is absent from the annotated evidence sentence and cannot be resolved from context.The analysis gives “Daggering is nontraditional” as an example where “this dance” cannot be linked to the claim entity.
- Sentence selection: Evidence sets may include a sentence only weakly related to the claim, causing the model to rank it very low.For the claim about Henry II having three cars, the car-related sentence is less directly related than the sentence about Henry II’s death.
- Recognizing textual entailment: Numerical claims are frequently misclassified because GloVe and FastText do not represent numbers distinctly enough.The analysis contrasts resting heart rates of 22 and 72 beats per minute as an example.
- Recognizing textual entailment: Challenging NotEnoughInfo cases arise when related evidence does not exclude the claim, as in the example involving Terry Crews and the Los Angeles Chargers.The evidence mentions other teams, including the Los Angeles Rams and San Diego Chargers, but does not establish the claim as false.
6 Conclusion
The paper presents a FEVER fact-extraction and verification system covering document retrieval, fact extraction, and claim verification. It uses entity linking and two ESIM extensions, and the combined pipeline significantly outperforms the baseline on the development set.
- The system addresses FEVER’s three subtasks: retrieving Wikipedia documents, extracting facts, and verifying claims from those facts.
- Document retrieval is framed as entity linking by identifying claim entities and linking them to Wikipedia articles.
- The authors extend ESIM for sentence ranking and for classifying claims from multiple facts using attention.
- Each individual model and the combined pipeline significantly outperforms the baseline on the development set.