Source-linked AI summary
The Fact Extraction and VERification (FEVER) Shared Task
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, Arpit Mittal
TL;DR
Information extracted from text can contain errors, motivating FEVER’s challenge of verifying human-generated claims against Wikipedia evidence. The paper reports the shared task, dataset, scoring setup, and participating systems; 19 of 23 teams surpassed the prior baseline, with the top system reaching 64.21% FEVER score.
Problem
FEVER addresses the need to verify whether human-generated textual claims are correct using evidence retrieved from Wikipedia.
Method
The paper presents the FEVER task and dataset, summarizes participating systems, and evaluates labels together with complete evidence using the FEVER score.
Results
19 of 23 teams exceeded the prior baseline, and the highest-scoring system achieved a FEVER score of 64.21%.
Takeaways & Limitations
FEVER’s shared-task results highlight common system approaches and identify directions for improving verification from untrusted sources.
Takeaways & Limitations
Human-annotated evidence was limited by a trade-off between annotation velocity and evidence recall, and was often incomplete.
Abstract
from arXiv · showhide
We present the results of the first Fact Extraction and VERification (FEVER) Shared Task. The task challenged participants to classify whether human-written factoid claims could be Supported or Refuted using evidence retrieved from Wikipedia. We received entries from 23 competing teams, 19 of which scored higher than the previously published baseline. The best performing system achieved a FEVER score of 64.21%. In this paper, we present the results of the shared task and a summary of the systems, highlighting commonalities and innovations among participating systems.
1 Introduction
FEVER established a shared task for verifying human-generated claims against Wikipedia evidence, addressing the need to check information extracted from potentially erroneous sources. The paper introduces the dataset, task requirements, annotation limitations, and shared-task scope.
- FEVER asks systems to predict whether human-generated claims are supported, refuted, or unresolved using evidence retrieved from Wikipedia.
- The purpose-built dataset contains 185,445 human-generated claims labeled SUPPORTED, REFUTED, or NOTENOUGHINFO after manual verification against Wikipedia introductions.
- Participants must return both the correct claim label and the sentence-level evidence that justifies it.
- FEVER differs from textual entailment by requiring evidence retrieval from a large corpus and from question answering by requiring reasoning about information absent from the claim.
- Annotator evidence was often incomplete because dataset construction traded annotation velocity against evidence recall, prompting later evidence augmentation.
- The paper describes the task and dataset, summarizes submissions and the leaderboard, and identifies future research directions.
2 Task Description
The task evaluates systems that retrieve Wikipedia sentences and classify claim veracity, using disjoint balanced data splits and a score requiring complete evidence. The dataset and evaluation setup also support blind competition scoring.
- 2 Task Description: Systems receive claims of unknown veracity, retrieve sentence-level Wikipedia evidence, and assign SUPPORTED, REFUTED, or NOTENOUGHINFO.
- 2 Task Description: 16.82% of claims require combining more than one sentence as supporting or refuting evidence.
- 2.1 Data: Training, development, and test splits are disjoint by claim-generating page, while development and test sets are balanced across classes.
- 2.2 Scoring Metric: The FEVER score is label accuracy conditioned on supplying at least one complete evidence set; NOTENOUGHINFO claims require no evidence.
- 2.2 Scoring Metric: 33.33% is the random baseline for label accuracy without the evidence requirement, while the FEVER score is lower for random predictions because supported and refuted claims need evidence.
- 2.3 Submissions: The task used blind Codalab test-set scoring, limited participants to 10 submissions, and provided open-source scoring software with diagnostic metrics.
3 Participants and Results
The shared task attracted broad participation, with most teams surpassing the prior baseline. Systems commonly used a three-stage retrieval-and-inference pipeline, although some integrated or added stages.
- 86 submissions from 23 teams were scored on the blind test set, and 19 teams exceeded the prior baseline.
- UNC-NLP achieved the highest reported FEVER score, 64.21%.
- Most participants followed document selection, sentence selection, and natural language inference stages similar to the baseline.
- Some teams jointly selected sentences and performed inference, while others added a post-inference step to discard inconsistent evidence.
- The paper organizes system descriptions to compare applied models and techniques with their relative performance.
4 Analysis
FEVER systems commonly used staged pipelines for document and sentence selection followed by natural language inference, with substantial variation in retrieval, representation, evidence combination, and training. The highest-scoring and highest-recall systems used different design choices, illustrating distinct precision–recall trade-offs.
- Document Selection: Named entities, noun phrases, and capitalized expressions were commonly extracted from claims for search or matching against Wikipedia.UNC-NLP additionally ranked candidate pages using page-viewership statistics, while GESIS Cologne directly selected sentences with Solr.
- Relative performance: 92.18% precision and 64.85% F1 made Papelo the highest-scoring system on evidence precision and evidence F1.Its reported approach combined TF-IDF document retrieval with named-entity and capitalized-expression string matching.
- Relative performance: 85.19% recall for Athene UKP TU Darmstadt and 82.84% for UCL Machine Reading Group were the highest reported evidence-recall scores.Athene queried Wikipedia using claim noun phrases, while UCL ranked article titles identified within claims using capitalization, sentence position, and token-match features.
- Sentence Selection: Sentence selection used keyword matching, supervised classification, or similarity scoring, with some systems adding aggregation to discard inconsistent evidence.Reported representations included lexical features, universal sentence representations, ELMo, TF-IDF, and sentence-selection similarity scores.
- Pipeline structure: Most submissions used document selection, sentence selection, and natural language inference, although some jointly modeled selection and inference or filtered inconsistent evidence afterward.Evidence combination ranged from concatenation to pairwise classification with aggregation or voting.
- Training: BUPT-NLPer and SWEEPer jointly modeled evidence selection and claim verification through multi-task learning, with SWEEPer also reporting reinforcement-learning parameter tuning.The approach was motivated by the hypothesis that information from each task supplements the other.
5 Additional Annotation
The shared task expanded test-set evidence by annotating system-retrieved sentences for claims that systems labeled incorrectly. Early annotations corrected some labels and added hundreds of new evidence sets, while further annotation remained in progress.
- Sampling: 18,846 claims with at least one system returning an incorrect FEVER label were sampled for additional evidence annotation.Claims were sampled with probability proportional to the number of systems labeling them incorrectly.
- Annotation procedure: Retrieved evidence was sampled according to each system’s FEVER score, focusing annotation on higher-quality candidate evidence.Volunteers judged whether evidence supported or refuted claims and identified usable individual sentences or sentence groups.
- Results: 1,003 annotations covering 618 claims found 3 incorrect SUPPORTED or REFUTED labels and 87 NOTENOUGHINFO claims needing relabeling after new evidence.The 87 relabelings comprised 44 SUPPORTED and 43 REFUTED claims.
- Results: 308 new evidence sets were identified for originally SUPPORTED or REFUTED claims, including 280 single sentences and 28 multi-sentence sets.Further annotation was still in progress at the time of writing.
6 Conclusions
The first FEVER shared task attracted broad participation and improved on the prior baseline across most participating teams. Its results also motivate continued work on incomplete human-annotated evidence and additional verification subtasks.
- Participation and results: 86 submissions from 23 teams were made, and 19 teams exceeded the previously published baseline.The paper summarizes reported system approaches, emphasizing commonalities and features for further exploration.
- Future work: Future work will address limitations in human-annotated evidence and explore other subtasks for predicting information veracity from untrusted sources.This conclusion identifies evidence quality and broader verification needs as ongoing research directions.
A.1 UNC-NLP
UNC-NLP used a three-component pipeline that retrieved candidate documents, selected evidential sentences, and verified claims with a three-way NLI classifier. The verifier also incorporated sentence-selection scores as token-level features.
- Document retrieval: The document retriever selected candidate Wikipedia documents using claim–title keyword matching and external page-view frequency statistics.Page-view information was used for wiki-page ranking.
- Sentence selection: The sentence selector compared claims with candidate-document sentences using a sequence-matching neural network trained as a binary classifier.Ground-truth evidence was positive training data, while other sentences were negative examples with annealed sampling.
- Claim verification: The claim verifier concatenated selected evidence as the premise and classified each claim as support, refute, or not enough info with a three-way neural NLI model.The classifier used WordNet and ELMo features.
- Claim verification: UNC-NLP fed the sentence selector’s similarity score into the claim verifier as an additional token-level feature.This combined the final two modules to improve the verifier’s awareness of selected evidence.
A.2 UCL Machine Reading Group
UCL Machine Reading Group uses a four-stage pipeline that retrieves documents and sentences, applies natural language inference, and aggregates predictions into a final evidence-consistent result.
- UCLMR retrieves documents, selects sentences, performs natural language inference, and aggregates predictions in a four-stage pipeline.The final reranking retains only evidence consistent with the aggregated prediction.
A.3 Athene UKP TU Darmstadt
Athene UKP TU Darmstadt trains an enhanced LSTM for sentence selection and applies it separately to claim-evidence pairs, pooling the resulting representations for classification.
- Athene trains the enhanced LSTM with hinge loss and negative sampling using randomly sampled sentences from retrieved documents as negatives.For each positive claim-evidence pair, negative samples are drawn from the retrieved documents.
- The system applies enhanced LSTM to five claim-evidence pairs, combines representations with average and max pooling, and feeds them to an MLP.
- Five sentences selected for evidence are paired individually with the claim before their representations are aggregated for classification.
A.4 Papelo
The paper surveys FEVER systems that combine document and sentence retrieval with textual entailment or joint learning, reporting varied architectures, scoring methods, and results.
- Papelo uses a pretrained transformer entailment classifier to evaluate broad sets of potential evidence and reports a 57.36% FEVER score.The system also reports 61.08% label accuracy and 64.85% evidence F1 on the FEVER test set.
- One system jointly models sentence extraction and verification after retrieving relevant documents with lexical and syntactic claim features.Its second stage is trained with multi-task learning and tuned with reinforcement learning over extracted sentences.
- Several systems retrieve candidate documents or sentences using entities, claim prefixes, lexical indexes, TF-IDF, embeddings, or similarity scoring.These approaches include Wikipedia APIs, Lucene searches, ELMo embeddings, and cosine similarity.
- Systems classify claims with models including Enhanced LSTM, decomposable attention, graph entailment, gradient-boosted trees, and random forests.Some systems process evidence sentence-wise, while others merge candidate sentences before classification.
- SIRIUS-LTG reports that TF-IDF cosine similarity for sentence selection combined with decomposable attention performed best on development and test data.
- BUPT-NLPer introduces an end-to-end bidirectional-attention multi-task model in which evidence extraction and claim verification provide supplementary information to each other.The model produces claim-aware sentence representations and page-aware claim representations.
- UMBC-FEVER uses frame-based retrieval and a two-layer MLP, achieving 0.3695 Evidence F1 and 0.2376 FEVER score.The submission ranked 10th out of 24 for Evidence F1 and was just below the baseline on FEVER score.