PaperlayerView exact revision on arXiv

Source-linked AI summary

X-FACT: A New Benchmark Dataset for Multilingual Fact Checking

Ashim Gupta, Vivek Srikumar

arXiv:2106.09248v1cs.CL

TL;DR

Multilingual fact-checking lacks manually annotated benchmarks for many languages, despite misinformation crossing linguistic and national boundaries. X-FACT provides a multilingual dataset and benchmark with three test sets, but the reported models underperform on each.

  • Problem

    Automated fact-checking beyond English is constrained by the absence of manually annotated benchmarks in many languages, despite misinformation crossing language boundaries.

    Source: §1 · p. 1

  • Method

    The authors compile claims and metadata from fact-checkers, map ratings to a shared label scheme, and create in-domain, out-of-domain, and zero-shot test sets.

    Source: §2 · p. 2, §1 · p. 2

  • Results

    The reported models underperform on all three benchmark test sets, while adding search snippets with metadata improves in-domain performance by 2.5 percentage points.

    Source: §1 · p. 2, §3.2 · p. 4

  • Takeaways & Limitations

    X-FACT is an order of magnitude larger than prior multilingual fact-checking datasets, enabling exploration of multilingual transformer-based approaches.

    Source: §4 · p. 5

    Monolingual models were unstable because the amount of data for each language was small.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

In this work, we introduce X-FACT: the largest publicly available multilingual dataset for factual verification of naturally existing real-world claims. The dataset contains short statements in 25 languages and is labeled for veracity by expert fact-checkers. The dataset includes a multilingual evaluation benchmark that measures both out-of-domain generalization, and zero-shot capabilities of the multilingual models. Using state-of-the-art multilingual transformer-based models, we develop several automated fact-checking models that, along with textual claims, make use of additional metadata and evidence from news stories retrieved using a search engine. Empirically, our best model attains an F-score of around 40%, suggesting that our dataset is a challenging benchmark for evaluation of multilingual fact-checking models.

1 Introduction

X-FACT addresses the shortage of multilingual fact-checking resources with a large, fine-grained dataset and benchmark. Models underperform on all three test sets, indicating a need for more robust methods.

1 Introduction

  • Multilingual fact-checking research is constrained by the shortage of manually annotated benchmarks beyond English, while multilingual datasets have remained limited in scope and size.Source: §1 · p. 1
  • X-FACT contains 31,189 general-domain non-English claims from 25 languages, with seven labels capturing different levels of truthfulness.Source: §1 · p. 1
  • Table 1 pairs original ratings with manually mapped labels and translations in examples from X-FACT.Source: Table 1 · p. 2
  • The benchmark includes two challenge sets to evaluate generalization across domains and languages, alongside the standard test set.Source: §1 · p. 2
  • Models underperform on all three benchmark test sets, suggesting a need for more sophisticated and robust fact-checking methods.Source: §1 · p. 2

2 The X-FACT Dataset

X-FACT aggregates claims from non-English fact-checking sources, maps their ratings to a shared seven-label scheme, and provides three test sets for evaluation. Its splits cover 25 languages, including languages held out for zero-shot testing.

2 The X-FACT Dataset

  • Claims were gathered from 85 fact-checkers using Google’s Fact Check Explorer and direct website crawling, with duplicates and claims revealing their labels removed.Source: §2 · p. 2
  • Source ratings were translated and merged when synonymous, then manually mapped to seven shared labels: five truthfulness levels, Unverifiable, and Other.Source: §2 · p. 2
  • The dataset contains 31,189 fact-checks after removing sources with highly imbalanced labels or fewer than 60 examples.Source: §2 · p. 3
  • The three test sets measure in-domain performance, generalization to new sources in training languages, and zero-shot transfer to languages absent from training.Source: §2 · p. 3
  • X-FACT covers 25 languages; its data are split into training (75%), development (10%), and an α1 test set (15%), with 13 languages in the zero-shot test set (α3).Source: §2 · p. 3

3 Experiments and Results

The experiments compare multilingual mBERT claim-only, metadata, and search-evidence models using macro F1. Models perform worse on source- and language-shift challenge sets than in-domain, while evidence gains are modest and snippets often lack enough information.

3.1 Experimental Setting

  • Experiments use multilingual mBERT and macro F1 across four seeds, comparing claim-only input with top-five Google snippets and claim metadata in one multilingual model.Source: §3.1 · pp. 3–4

3.2 Results

  • Claim-only models beat the majority baseline on α1, indicating that many examples are classifiable from wording and that the model learns spurious correlations.Source: §3.2 · p. 4
  • Metadata helps, while adding search snippets with metadata improves in-domain α1 performance by 2.5 percentage points; random snippets perform similarly to claim-only.Source: §3.2 · p. 4
  • Only 45% of 100 sampled Hindi α1 snippets sufficed for classification, versus full web-page text providing sufficient evidence for 83% of examples.Source: §3.2 · p. 4
  • Performance on the α3 challenge set is worse than on α1, and the challenge sets show models overfit source-specific patterns (α2) and language-specific patterns (α3).The challenge sets assess model generalizability and diagnose overfitting.Source: §3.2 · p. 5
  • Adding 12,311 English claims from PolitiFact hurts model performance, possibly because those claims are mostly political while X-FACT covers general claims.Source: §3.2 · p. 5, Table 4 · p. 5

4 Conclusion

X-FACT is presented as the largest multilingual fact-checking dataset, an order of magnitude larger than prior work. The models find it challenging, and the authors envision it as a benchmark for multilingual fact-checking.

4 Conclusion

  • X-FACT is presented as the largest multilingual fact-checking dataset, an order of magnitude larger than prior work.Its size enables exploration of large transformer-based multilingual approaches to fact-checking.Source: §4 · p. 5
  • The multilingual models find X-FACT challenging.Source: §4 · p. 5
  • The authors envision X-FACT as a benchmark for developing and evaluating multilingual fact-checking approaches.Source: §4 · p. 5

A Details on Dataset Construction

X-FACT’s claims, labels, and metadata are obtained through Google’s Fact Check Explorer and crawling fact-checking websites. Some sources were omitted because their sites or claim records could not be reliably extracted or interpreted.

A Details on Dataset Construction

  • Websites were omitted when their pages resisted crawling or did not provide extractable claims and clear ratings.AFP’s template hindered automatic extraction; Faktograf’s Croatian pages did not yield properly extractable claims or clearly stated ratings, and many websites in some languages could not be crawled.Source: A Details on Dataset Construction · p. 7
  • Claims with no Google search results were excluded from the training, development, and test sets.The passage describes these as a very small percentage of claims.Source: A Details on Dataset Construction · p. 7
  • Claims, labels, and metadata come from Google’s Fact Check Explorer or crawling the respective fact-checking websites.The authors also planned to release the label mapping created for the dataset.Source: A Details on Dataset Construction · p. 7

B.1 Models and Code

The experiments use multilingual BERT, implemented in PyTorch with the transformers library.

B.1 Models and Code

  • The authors use multilingual BERT and implement their models in PyTorch with the transformers library.Source: §B.1 · p. 7

B.2 Computing Infrastructure Used

Experiments ran on V100, P100, and A100 GPUs, with evidence-based models requiring the larger-VRAM A100.

B.2 Computing Infrastructure Used

  • Claim-only experiments ran on V100 and P100 GPUs, while evidence-based models used A100 GPUs because they required more VRAM.The listed machines had 16 GB VRAM for V100 and P100, and 40 GB for A100.Source: §B.2 · p. 7

B.3 Hyperparameters and Fine-tuning Details

The experiments use mBERT with a fixed batch size and report results averaged across four seeds. Evidence documents were capped at five, and model selection used development-set performance across ten epochs.

B.3 Hyperparameters and Fine-tuning Details

  • mBERT-base has 12 layers, hidden size 768, 12 attention heads, and 125 million parameters; the batch size is fixed at 8, unlike the other Devlin et al. (2019) hyperparameters.Source: §B.3 · p. 7
  • Results are means across four random seeds; models are fine-tuned for ten epochs, with the development-set best checkpoint selected.Source: §B.3 · p. 7
  • The number of evidence documents is capped at five because of GPU VRAM constraints.Source: §B.3 · p. 7
  • Average training times for the models are presented in Table 5.Source: §B.3 · p. 7
  • Table 6 presents dataset composition across training, development, and three challenge sets for 25 languages from 11 language families.Source: Table 6 · p. 8

Related papers

Loading 2106.09248v1…