Source-linked AI summary
FEVER: a large-scale dataset for Fact Extraction and VERification
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal
TL;DR
FEVER addresses the lack of large-scale, evidence-based datasets for verifying textual claims by constructing and annotating Wikipedia-derived claims. The authors develop a retrieval-and-entailment pipeline to evaluate the dataset, finding that correct-evidence verification is challenging while still feasible. The dataset also supports research on fact extraction and verification against textual sources.
Problem
Existing verification datasets contained few claims or lacked machine-readable evidence, motivating a large-scale resource for textual claim verification.
Method
FEVER contains 185,445 Wikipedia-derived claims labeled SUPPORTED, REFUTED or NOTENOUGHINFO, with evidence sentences recorded for supported and refuted claims.
Results
31.87% accuracy was achieved when correct evidence was required, compared with 50.91% when evidence correctness was ignored.
Takeaways & Limitations
FEVER provides a publicly available testbed intended to stimulate research on claim extraction and verification against textual sources.
Takeaways & Limitations
Evidence annotations are not guaranteed to be complete, so systems selecting valid information that annotators omitted may be penalized in precision.
Abstract
from arXiv · showhide
In this paper we introduce a new publicly available dataset for verification against textual sources, FEVER: Fact Extraction and VERification. It consists of 185,445 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from. The claims are classified as Supported, Refuted or NotEnoughInfo by annotators achieving 0.6841 in Fleiss $κ$. For the first two classes, the annotators also recorded the sentence(s) forming the necessary evidence for their judgment. To characterize the challenge of the dataset presented, we develop a pipeline approach and compare it to suitably designed oracles. The best accuracy we achieve on labeling a claim accompanied by the correct evidence is 31.87%, while if we ignore the evidence we achieve 50.91%. Thus we believe that FEVER is a challenging testbed that will help stimulate progress on claim verification against textual sources.
1 Introduction
FEVER addresses the shortage of large-scale datasets for verifying textual claims against retrieved evidence by introducing 185,445 Wikipedia-derived claims and a baseline pipeline. The task requires document and sentence retrieval before classification, and remains difficult even when evidence is ignored.
- Motivation: Verification research had been limited by datasets containing only a few hundred claims, far smaller than datasets used for textual entailment and question answering.The paper positions FEVER as a resource intended to stimulate further research on verification.
- Dataset contribution: 185,445 claims comprise FEVER, generated from Wikipedia sentences and classified as SUPPORTED, REFUTED or NOTENOUGHINFO with evidence recorded for the first two classes.Claims were altered in ways that could change their meaning, and verification was conducted without access to the source sentence.
- Evidence requirements: 31.75% of claims were judged to require more than one evidence sentence, with 16.82% requiring evidence composition across multiple sentences.Additionally, 12.15% required evidence from multiple pages.
- Annotation quality: 0.6841 Fleiss κ measured agreement for claim verification classification, while evidence retrieval reached 95.42% precision and 72.36% recall.These results followed the use of annotation guidelines and dedicated interfaces.
- Baseline challenge: 31.87% accuracy was achieved when correct evidence was required, compared with 50.91% when evidence correctness was ignored.Oracle experiments identified selecting evidence sentences as the most challenging pipeline component.
2 Related Works
Prior claim-verification datasets were smaller, often lacked machine-readable evidence, or supplied contextual information unavailable in FEVER. FEVER instead evaluates claims against textual sources without requiring a question or article context.
- Fact-checking datasets: Vlachos and Riedel’s dataset contained 106 claims, while Wang’s extension included 12.8K Politifact claims but ignored non-machine-readable justifications and evidence.The latter classified claims using claim text and speaker metadata.
- News verification: The Fake News Challenge used 50K labeled claim-article pairs built from 300 claims and 2,582 articles for stance classification.Its labels were support, refute, neutral observation, or irrelevance with respect to an article.
- Inference datasets: Natural-logic inference evaluated 1,378 claims derived from eight binary ConceptNet relations, limiting claim variety.Its textual sources included Wikipedia processed with Open Information Extraction.
- Contextual validation: Answer Validation Exercise claims were answers returned to questions, so questions and QA systems supplied context absent from FEVER’s task definition.The cited comparison also notes a dataset size of 1,000 instances per language.
3 Fact extraction and verification dataset
FEVER was constructed by generating and then labeling Wikipedia-based claims, with annotators selecting evidence for supported or refuted judgments. Validation found substantial agreement and retrieval precision, while manual checks identified some guideline and annotation errors.
- Claim generation: The dataset construction process had two stages: claim generation from Wikipedia and claim labeling with evidence selection or a NOTENOUGHINFO decision.The generation stage sampled introductory-section sentences from approximately 50,000 popular pages.
- Claim generation: Annotators generated single-fact claims from sampled sentences and produced mutations through paraphrasing, negation, entity or relation substitution, and generalization or specialization.A dictionary of linked terms and their Wikipedia introductions supplied additional knowledge for more complex claims.
- Claim labeling: Evidence was selected from introductory sections of the main and linked entity pages, with annotators also able to add arbitrary Wikipedia pages.For SUPPORTED and REFUTED labels, annotators recorded the sentences necessary to justify their decisions.
- Validation: 6.63% of generated claims were excluded as too vague or ambiguous, alongside 1.01% skipped claims and 2.11% containing typos.The excluded examples included claims such as “Sons of Anarchy premiered.”
- Validation: 0.6841 Fleiss κ was obtained from five-way annotation of 7,506 claims, while comparison with super-annotators yielded 95.42% precision and 72.36% recall.Super-annotators searched all of Wikipedia for possible evidence.
- Validation: 91.2% of 227 manually checked examples were annotated correctly, while 3% contained claim-generation mistakes not flagged during labeling.Only four examples were labeled incorrectly according to the guidelines; many other errors concerned insufficient evidence.
4 Baseline System Description
The baseline is a three-component pipeline that retrieves documents, selects sentence-level evidence, and classifies entailment. Its components are evaluated separately with oracle experiments before final test-set accuracy is reported.
- Pipeline: The pipeline combines document retrieval, sentence-level evidence selection, and textual entailment.Each component is evaluated in isolation through oracle evaluations on the development set.
- Document retrieval: Document retrieval uses DrQA to return the k nearest documents from cosine similarity over binned unigram and bigram TF-IDF vectors.This component forms the document candidates for later evidence selection.
- Sentence selection: Sentence selection ranks sentences by TF-IDF similarity to the claim and tunes a cutoff using development-set accuracy.Both DrQA and a simple unigram TF-IDF implementation are evaluated.
- Textual entailment: The entailment stage compares a single-hidden-layer MLP baseline using term-frequency and TF-IDF features with a decomposable-attention model.The decomposable-attention model was selected as a state-of-the-art RTE system with publicly available code at development time.
- Textual entailment: NOTENOUGHINFO training instances are simulated by sampling evidence sentences from the nearest page or from Wiki-/claim-predicted sentences.The system must classify irrelevant or uninformative retrieved evidence as NOTENOUGHINFO.
5 Experiments
The experiments evaluate FEVER’s retrieval, evidence selection, and entailment components through oracle and full-pipeline settings. Results show that sentence selection is a major bottleneck, while the complete system achieves substantially higher label accuracy when evidence correctness is ignored.
- 5.5 Recognizing Textual Entailment: NEARESTP was the most accurate training strategy when evidence was considered, whereas RANDOMS harmed NOTENOUGHINFO recall in the full pipeline.Related but uninformative evidence was misclassified as SUPPORTED or REFUTED under RANDOMS.
- 5.7 Evaluating Full Pipeline on Test Set: 31.87% accuracy was achieved on the test set when SUPPORTED/REFUTED predictions required correct evidence, versus 50.91% when evidence correctness was ignored.The gap indicates that the system often predicted the correct label while selecting evidence different from the annotators’ evidence.
- 5.7 Evaluating Full Pipeline on Test Set: 45.89% evidence recall, 10.79% precision, and 17.47% F1 were obtained for complete evidence groups on claims requiring evidence.These metrics evaluate the document and sentence retrieval modules together.
- 5.8 Manual Error Analysis: Manual analysis found that information retrieval failed to identify any correct evidence in 58.27% of sampled incorrect-label or incorrect-evidence cases.Among 961 sampled predictions, 28.51% had the correct label but incorrect evidence, and 13.84% were misclassified despite suitable evidence being found.
6 Discussion
The discussion presents FEVER as a basis for alternative verification and claim-extraction uses beyond its evaluated pipeline. It also notes Wikipedia is not the only intended source, with portability to other textual sources expected.
- Alternative approaches, including natural-logic-style reasoning over a knowledge base or graph, could improve interpretability and support reading-comprehension methods.The proposed approach would read textual sources, construct a knowledge base or graph, and reason over the claim.
- FEVER can support claim extraction by generating concise facts from longer encyclopedic texts containing multiple difficult-to-evaluate claims.The dataset construction also allowed simple claims to be extracted from multiple complex sentences.
- Wikipedia is a chosen source rather than the only source for verification, and systems developed with FEVER are expected to be portable to different textual sources.
7 Conclusions
The paper introduces FEVER as a publicly available dataset for fact extraction and verification against textual sources, alongside annotation insights and further uses. A retrieval-and-entailment pipeline shows that the task is challenging yet feasible.
- FEVER is a publicly available dataset for fact extraction and verification against textual sources, accompanied by data-collection and annotation insights.
- 31.87% accuracy was achieved by the best-performing system in a pipeline combining information retrieval and textual entailment components.
- FEVER is intended as a challenge for claim extraction and verification systems, with further uses and extensions discussed for future work.
A.1 Task 1 Definitions
Task 1 defines claims as single-sentence statements about one aspect of a target entity and uses controlled dictionary knowledge to increase claim complexity. Annotation guidance specifies how to generate and format claims.
- A claim is a single sentence expressing true or mutated information about one aspect of one target entity.The definition distinguishes claims from unconstrained world knowledge and overly simple source-sentence extraction.
- A dictionary supplies additional world knowledge so annotators can generate more complex claims in a structured, evidence-backed manner.
- True claims must be deducible from the source sentence and dictionary, while mutations may negate, substitute, or alter specificity.
- Claims should directly name entities, avoid speculative or vague language, and permit only minor entity-name variations.
- Annotators generate concise factoid claims using the source sentence and dictionary, without incorporating their own knowledge or beliefs.
- Claims should use correct entity capitalization, end with a period, and format numbers in appropriate English forms.
- Unsuitable source sentences or dictionary entries may be skipped or ignored, and extracted text is treated as valid summary material without fact-checking.
A.4 Task 1 (substask 2) Guidelines
Task 1 subtask 2 instructs annotators to create objective, verifiable true or false claim modifications from original claims and supporting resources. Six modification types vary wording, entities, or specificity.
- The task generates true or false claim modifications according to specified instructions, using original claims and the dictionary as their basis.
- Modified claims should directly reference entities, avoid vague language, and remain objective and verifiable using publicly available information.
- Formatting guidance requires direct entity references, correct capitalization, sentence-final periods, and appropriate English number formats.
- Annotators may use their own world knowledge when creating modifications, but leave the response blank when no suitable fact or misinformation can be generated.
- The workflow presents six related mutation types on one annotation page, with examples involving borders, geographic summaries, and environmental descriptions.
- Rephrasing preserves the original claim’s meaning, while other modifications substitute verbs or objects or change claim specificity.
- Annotators may leave a modification blank when generating a claim for a particular modification type is not possible.
A.5 Task 1 (subtask 2) Examples
The examples illustrate how a source claim can be transformed through relation-based mutations, using a toy ontology to distinguish mutation types.
- A source sentence about Barack Obama touring the UK serves as the basis for demonstrating possible claim mutations.
- The mutation examples explain why each altered claim is considered suitable for the task.
- Figure 6 provides the toy ontology used to organize examples of similar and dissimilar mutations.
A.6 Task 2 Guidelines
Task 2 asks annotators to identify Wikipedia evidence that supports or refutes claims, using only the page and dictionary information while applying common-sense reasoning. The guidelines cover direct, multi-sentence, linked-page, and insufficient-evidence cases.
- Evidence identification: Annotators read each claim and Wikipedia page, then select sentences that support or refute the claim.
- Evidence identification: Direct evidence can establish a judgment alone, while partial evidence may require supplementary information from linked or custom Wikipedia pages.
- Decision labels: If the page and dictionary lack enough information, annotators cancel the attempt and ultimately label the claim Not enough information.
- Decision labels: Annotators must use common-sense reasoning but base decisions on information presented in the Wikipedia page and dictionary rather than personal world knowledge.
- Decision labels: Valid justifications include direct, synonymous, antonymous, negating, quantified, entity-specific, or compositional information.
- Evidence identification: Claims requiring multiple sentences are supported by selecting the minimal set of relevant evidence, which is stored in document order.
B Manual Error Analysis
The manual error analysis followed the decision process shown in Figure 8. Recommended actions for newly found evidence were documented but not performed in this dataset version.
- Manual error analysis used the decision process described in Figure 8.
- Recommended actions were listed for cases in which new evidence was found during analysis.
- Those actions were not performed for this dataset version but may support a future pool-based FEVER Shared Task update.