PaperlayerView exact revision on arXiv

Source-linked AI summary

FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information

Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal

arXiv:2106.05707v3cs.CL

TL;DR

Large-scale fact-verification benchmarks have largely overlooked structured evidence such as tables, despite its relevance to misinformation detection. FEVEROUS introduces a verified Wikipedia benchmark combining sentences and table cells, together with a retrieval-and-verification baseline that correctly predicts both for 18% of claims.

  • Problem

    Large-scale fact-verification benchmarks have mostly focused on textual evidence and have overlooked structured information such as tables.

  • Method

    FEVEROUS constructs and verifies Wikipedia claims with sentence and table-cell evidence, while its baseline retrieves and classifies both evidence types.

  • Results

    18% of claims receive both the correct evidence and verdict from the proposed baseline.

  • Takeaways & Limitations

    FEVEROUS provides a benchmark for fact verification requiring systems to handle sentences, tables, and their combination.

  • Takeaways & Limitations

    FEVEROUS uses Wikipedia as its evidence source and does not make judgments about claims' real-world truth.

Abstract

from arXiv · show

Fact verification has attracted a lot of attention in the machine learning and natural language processing communities, as it is one of the key methods for detecting misinformation. Existing large-scale benchmarks for this task have focused mostly on textual sources, i.e. unstructured information, and thus ignored the wealth of information available in structured formats, such as tables. In this paper we introduce a novel dataset and benchmark, Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS), which consists of 87,026 verified claims. Each claim is annotated with evidence in the form of sentences and/or cells from tables in Wikipedia, as well as a label indicating whether this evidence supports, refutes, or does not provide enough information to reach a verdict. Furthermore, we detail our efforts to track and minimize the biases present in the dataset and could be exploited by models, e.g. being able to predict the label without using evidence. Finally, we develop a baseline for verifying claims against text and tables which predicts both the correct evidence and verdict for 18% of the claims.

1 Introduction

FEVEROUS addresses the limited coverage of table-based evidence in large-scale fact-verification benchmarks by combining structured and unstructured Wikipedia evidence. It introduces a manually verified benchmark and a baseline that retrieves and verifies both evidence types.

  • Large-scale fact-verification datasets have mostly focused on textual evidence, while table-based datasets often assume an unrealistic setting.
  • FEVEROUS contains 87,026 manually constructed and verified claims labeled as supported, refuted, or not enough information.Claims may require a sentence, a table cell, or combinations of sentences and cells from different articles.
  • Each claim can require heterogeneous evidence, including multiple sentences and table cells from different Wikipedia pages.
  • The baseline combines entity matching, TF-IDF retrieval, cell extraction, and a RoBERTa classifier to predict evidence and claim veracity.
  • 18% of claims receive both the correct evidence and verdict from the proposed baseline.The retrieval module fully covers 28% of a claim's evidence.

2 Literature Review

Prior fact-verification datasets have usually centered on unstructured text or isolated structured sources, often avoiding evidence retrieval. FEVEROUS extends this literature by requiring retrieval over both tables and text.

  • Many fact-verification datasets omit fine-grained evidence, rely on potentially misleading search results, or extract evidence directly from fact-checking sources.
  • Artificial-claim construction has been used to reduce the cost of curating evidence sets for real-world claims.
  • Existing structured-information datasets mainly use tables or infoboxes, while SEM-TAB-FACTS also identifies evidence locations in tables.
  • FEVEROUS is the first proposed dataset to combine structured and unstructured evidence while explicitly requiring evidence retrieval.
  • HybridQA combines Wikipedia tables with linked-entity text for multi-hop questions but assumes the table is already part of the input.

3 FEVEROUS Dataset and Benchmark

FEVEROUS is a fact-verification dataset and benchmark spanning Wikipedia sentences and structured evidence such as table cells, with claims requiring varied evidence combinations. Its construction and evaluation address annotation quality, labeling ambiguity, dataset artifacts, and baseline feasibility.

  • Dataset and task: FEVEROUS retrieves sentence or table-cell evidence from English Wikipedia and assigns Supports, Refutes, or Not Enough Info.Evidence may come from any non-reference article section and is paired with article, section, and relevant header context.
  • Dataset and task: Claims can require a single sentence, a single cell, or multiple sentences and cells from different articles.Tables include captions, infoboxes, and lists, with cells indexed by row and column.
  • Dataset characteristics: FEVEROUS averages 1.4 sentences and 3.3 cells, or 0.8 tables, per claim, and balances evidence across text-only, table-only, and combined cases.Supported claims comprise 56%, refuted claims 39%, and NotEnoughInfo claims about 5%.
  • Annotation: Claims are generated from controlled Wikipedia highlights and independently verified by different annotators to avoid revealing the original label or evidence.Annotators use sentence or table highlights, generate multiple factual claims, and confirm the selected evidence and verdict before submission.
  • Annotation: Refutation labeling permits annotators to mark claims Refuted when the evidence would mislead them, even without explicit contradictory wording.The authors acknowledge that this rationale introduces potential ambiguity because judgments partly depend on what annotators expect to find on Wikipedia.
  • Quality control: Verdict agreement reaches 0.92 when annotators select identical evidence, while disagreements often arise from evidence retrieval, counting, overlooked details, or task complexity.Overall two-way and three-way κ is 0.65; evidence overlap is at least 70% for 74% of claims.

4 Baseline Model

The baseline retrieves relevant Wikipedia pages, sentences, tables, and cells, then predicts claim veracity from the combined evidence and context. It uses entity matching, TF-IDF, table linearization, sequence labeling, and RoBERTa-based verdict prediction.

  • Entity matching and TF-IDF retrieve candidate Wikipedia pages, sentences, and tables for each claim.The retriever selects up to 5 pages, 5 sentences, and 3 tables.
  • The cell extractor linearizes each retrieved table and treats relevant-cell retrieval as binary sequence labeling.A fine-tuned RoBERTa model receives the claim concatenated with the table, with row sampling used during training.
  • A RoBERTa encoder with a linear layer predicts the verdict from linearized cell and sentence evidence with its context.Linearization allows cross-attention between table cells and sentences through concatenation.
  • Additional NEI training instances are created by removing a sentence or an entire table from annotations containing both evidence types.The procedure addresses the dataset’s 5% NEI-label imbalance.

5 Experiments

FEVEROUS evaluates fact verification by jointly retrieving textual and tabular evidence and predicting claim veracity. Experiments show that combining evidence types improves performance, while retrieval, numerical reasoning, and ecological validity remain important boundaries.

  • Evaluation: The FEVEROUS score requires both the correct verdict and retrieval of at least one complete gold evidence set.Precision is excluded because evidence annotations may not be exhaustive; predictions are instead bounded by s table cells and l sentences.
  • Results: The full baseline combining sentences and tables substantially outperforms sentence-only and table-only baselines.All baselines use TF-IDF retrieval, while the full system integrates both evidence types for verdict prediction.
  • Evidence retrieval: At k = 5, retrieval covers 69% of documents; the top 5 sentences cover 53% of sentences and the top 3 tables cover 56%.The retriever’s overall passage recall is reported as 0.55%, compared with 49% coverage for TF-IDF without entity matching.
  • Evidence extraction: Gold-table cell extraction achieves 0.69 cell recall, rising to 0.74 when a table contains only a single evidence cell.Retrieved cells combined with extracted sentences fully cover the evidence for 29% of development samples.
  • Verdict prediction: NLI pre-training raises verdict prediction to a macro-averaged F1 of 0.70, whereas removing NEI sampling prevents correct recognition of any NEI sample.These verdict-classification results use gold evidence on the development set.
  • Discussion and limitations: The baseline independently retrieves structured and unstructured information, although surrounding sentences and hyperlinks can provide table context and entity disambiguation.The discussion identifies ignored hyperlinks and limited cross-modal retrieval as areas for improvement.

6 Conclusion

FEVEROUS combines sentence and table-cell evidence for fact verification and provides a baseline that predicts evidence and verdicts. The benchmark is challenging, with the baseline outperforming single-source baselines while scoring 0.18.

  • FEVEROUS includes both unstructured and structured information for fact verification.The dataset uses sentences and table cells as evidence.
  • The baseline retrieves sentences and table cells, then predicts the verdict from both evidence types.It is evaluated against sentence-only and table-only baselines.
  • 0.18 baseline score demonstrates that FEVEROUS is a challenging benchmark for fact-checking systems.

7 Supplementary Material

The supplementary material specifies FEVEROUS’s data formats, corpus representation, annotation fields, and challenge categories. It also documents access resources and the structured representation of Wikipedia content.

  • Access resources: Example dataset and retrieval-corpus entries are available through the FEVEROUS dataset website.
  • Access and data formats: FEVEROUS data are distributed in Jsonlines format or an SQLite3 retrieval corpus.SQLite3 supports faster article retrieval by name, while training and development records use one UTF-8 JSON object per line.
  • Annotation schema: Each dataset entry records an ID, label, claim, evidence sets, and annotator operations.Evidence sets contain element IDs and contextual mappings, while operations record how annotators searched and navigated Wikipedia.
  • Retrieval corpus: Wikipedia articles are ordered collections of sections, tables, lists, and sentences with explicit structural fields.Tables store cell values and metadata such as headers and spans; lists store item values, nesting levels, and types.
  • Retrieval corpus: Table records encode content as lists of cells and optionally include captions.Each cell stores an ID, value, header status, row span, and column span.
  • Retrieval corpus: List records distinguish ordered and unordered lists and represent nested items through levels and types.Hyperlinks are represented with double square brackets, including optional anchor text.

7.2 Ethics statement

FEVEROUS was constructed using English Wikipedia, controlled claim-generation guidelines, screened annotators, and documented ethical procedures. Its scope is limited to evidence from Wikipedia and claims that are publicly verifiable and non-subjective.

  • Ethics: The dataset followed University of Cambridge ethics approval and contractor policies covering fair pay, inclusion, privacy, communication, and well-being.
  • Scope and limitations: FEVEROUS labels support or refutation by Wikipedia evidence rather than asserting real-world truth.The authors note that Wikipedia contains mistakes and noise and require systems to identify insufficient evidence.
  • Curation rationale: FEVEROUS uses English Wikipedia as its knowledge base, excluding articles flagged for missing references or citations.Highlights were sampled randomly from the collection of English Wikipedia articles.
  • Speech situation: Claims were generated from a December 2020 Wikipedia corpus between March and May 2021 under detailed guidelines.Claims had to be publicly verifiable, unambiguous, and free of idioms, figures of speech, similes, and verbose language.
  • Text characteristics: The dataset avoids contemporary political or disputed topics in some cases, while claim topics roughly follow Wikipedia’s article distribution.
  • Annotator demographics: Annotators underwent screening and calibration stages and were paid above their local minimum wage.

7.4 Licensing

The supplementary material reports dataset composition, evidence distributions, challenge patterns, and annotation agreement. It shows that verification involves diverse challenges and that agreement depends on evidence overlap and early annotator calibration.

  • Evidence statistics: 69% of evidence pieces are table cells, 29% are sentences, 1% are list items, and 1% are table captions.Evidence retrieval directly matches a claim word or phrase in about 84% of claims.
  • Verification challenges: Verification challenges comprise about 10% numerical reasoning, 16% multi-hop reasoning, 14% combining tables and text, 2% entity disambiguation, and 1.3% search terms not in claim.These distributions are relatively similar across dataset splits.
  • Verification challenges: Claim generators overpredicted Other and rarely correctly predicted entity disambiguation or search terms not in claim.The passage attributes this pattern to generators receiving highlights rather than searching for pages themselves.
  • Annotation agreement: κ agreement is 0.92 for exact evidence matches and 0.11 for completely distinct evidence.Agreement decreases linearly as evidence overlap declines.
  • Annotation agreement: Annotators with calibration scores above 0.9 achieve around 0.8 kappa agreement initially, compared with about 0.5 below 0.6.Across all annotations, agreement becomes more similar across calibration groups, except for annotators above 0.9.

7.6 Dataset Processing & Implementation Details

FEVEROUS processes Wikipedia text, links, and tables into a searchable corpus, then uses retrieval and RoBERTa-based models to extract evidence and predict claim veracity.

  • Corpus processing: Wikipedia articles were sentence-tokenized, extracted directly to handle complex templates, and cleaned by normalizing redirects, URLs, and text.Table parsing complexity motivated direct article extraction rather than relying on template resolution.
  • Annotation infrastructure: The annotation platform used MediaWiki with CirrusSearch and stored annotations in MariaDB.The setup included extracted articles and Wikipedia redirects.
  • Evidence retrieval: The retriever combines exact entity-title matching with TF-IDF cosine similarity over claim and article introductory-section unigram and bigram vectors.Lists were excluded to reduce computation because only 1% of annotated evidence was located in lists.
  • Cell retrieval: The cell retrieval model uses RoBERTabase with row-sampling that prioritizes rows containing relevant evidence before irrelevant rows.Long tables were truncated at RoBERTa’s maximum input length.
  • Verdict prediction: The verdict predictor uses RoBERTalarge pretrained on multiple NLI datasets and receives evidence pieces separated by </s>, with cells linearized using table context and headers.The baseline was fine-tuned for one epoch using fixed hyperparameters rather than extensively tuned for peak performance.
  • Experimental setup: RoBERTa experiments were repeated twice with average results and very low variance around 2e−5.Fine-tuning took about 1.5 hours for cell extraction and 4 hours for verdict prediction, while TF-IDF retrieval took about 10 hours.

7.8 Annotation details

FEVEROUS annotation controls claim construction through Wikipedia highlights, search tools, and explicit requirements for factual, objective, unambiguous claims. Annotators generate claims using highlighted content, broader page or cross-page information, and specified mutations.

  • Annotation setup: The annotation process gives each annotator a Wikipedia highlight and search access through MediaWiki, custom page search, and hyperlinks.Highlights contain either four consecutive sentences or a table, and each page is used only once for claim generation.
  • Claim using highlight only (Type I): Type I claims use only highlighted sentences or table cells and combine multiple pieces of information rather than simply paraphrasing one sentence.Table-based claims may use comparisons, filters, arithmetic, and min-max operations.
  • Annotation interfaces: The claim-generation interface supports writing claims, selecting expected challenges, navigating article highlights, submitting annotations, skipping highlights, and moving among prior claims.The verification interface separately supports evidence management, challenge selection, verdict labels, and Wikipedia search.
  • Claim requirements: Claims must be self-contained, objective, verifiable, unambiguous, concise, and free of idioms, speculative wording, and prohibited contemporary political topics.Annotators were instructed to avoid pronouns, subjective comparisons, vague quantifiers, and verbose language.
  • Claim beyond the highlight (Type II): Type II claims extend highlighted information with content from the same page or other Wikipedia pages, and may combine structured and unstructured sources.Annotators may also use assumed general knowledge, such as interpreting “90s” as 1990–1999.
  • Mutated Claim (Type III): Type III claims modify an earlier claim using one of six mutations: More Specific, Generalization, Negation, Paraphrasing, Entity Substitution, or Tense Shift.The mutation can change whether the claim is supported or refuted, but the result must remain a single sentence.

7.10 Claim Verification

FEVEROUS verification requires annotators to collect sentence, table-cell, caption, and list evidence with automatically added context, while judging whether the evidence supports, refutes, or leaves the claim unresolved.

  • Evidence selection: Annotators can combine evidence from different Wikipedia pages, selecting complete sentences or phrases, individual table cells, and list items.Article titles, section headings, and table headers are automatically included as context.
  • Reasoning guidance: Annotators must use common-sense reasoning without adding unsupported world knowledge, while seeking further Wikipedia evidence when it supplies missing information.General knowledge may be used when it cannot be found on Wikipedia, but annotators should avoid relying on it unnecessarily.
  • Verdict assessment: Annotators assess verdicts by asking whether the selected sentences, cells, and list items provide strong enough reason for support or refutation.When uncertain, they identify missing information and search for additional evidence before deciding.
  • Verdict evidence: Supported claims require evidence for each stated fact, whereas refuted claims require only evidence contradicting the claim.Partially supporting evidence should not be selected for a refuted claim.
  • Evidence overview: The evidence overview displays selected evidence and allows annotators to expand entries to inspect their content or remove them.This supports tracking evidence collected from multiple pages.

Note!

The verification guidelines define detailed handling for universal table claims, hedged or missing evidence, ambiguous claims, reporting, challenge categories, and annotation quality assurance.

  • Table evidence: Universal claims requiring every entry in a table row or column require annotators to highlight every cell in that row or column.This applies to claims such as identifying the country with the highest number of gold medals.
  • Evidence boundaries: Hedged Wikipedia statements cannot serve as evidence, and missing evidence must be recorded with a Not enough Information verdict.Annotators must still submit an annotation when no evidence can be found.
  • Verdict consistency: Supported verdicts require evidence for every fact in longer claims, while refuted verdicts should cite only information contradicting the claim.The guideline decomposes a claim into separately verifiable facts before selecting Supported.
  • Assessment principles: Annotators should ignore possible Wikipedia-editor motives and treat the available evidence directly.The guideline separates evidence assessment from speculation about how claims were generated.
  • Claim reporting: Claims may be reported and skipped when they are personal, implausible, unverifiable, ambiguous, ungrammatical, or otherwise violate generation guidelines.Ambiguous claims may be refuted when the evidence would mislead a reasonable reader, even without direct contradiction.
  • Challenge reporting: Verification challenges are categorized as Multi-hop Reasoning, Numerical Reasoning, Combining Tables and Text, Entity Disambiguation, Search terms not in claim, or Other.These categories describe the main difficulty encountered while finding evidence.
  • Quality assurance: Quality assurance uses checked annotations, production reports, updated guidelines, FAQs, and individual annotator feedback to identify error patterns.Partial agreement or disagreement is highlighted in red in the project-manager interface.

7.11 Author statement

The authors accept responsibility for copyright violations associated with FEVEROUS and document annotation interfaces with anonymized IDs.

  • The authors bear responsibility for any copyright violations associated with the FEVEROUS dataset.
  • Figure 14 presents the question-answer annotation interface used by project managers.
  • The annotator interface is similar to the project-manager interface, with IDs anonymized.
Loading 2106.05707v3…