Source-linked AI summary
MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims
Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, Jakob Grue Simonsen
TL;DR
Automatic fact checking needs evaluation on naturally occurring claims spanning diverse domains, evidence, and metadata. MultiFC constructs such a dataset and evaluates veracity models, including a joint model that ranks evidence pages while predicting veracity. The best-performing model achieves 49.2% Macro F1, while metadata and evidence encoding improve performance.
Problem
Existing fact-checking efforts use small naturally occurring datasets or artificially constructed claims, limiting the available real-world evaluation setting.
Method
MultiFC collects claims from multiple fact-checking websites with evidence pages and metadata, then jointly ranks evidence pages and predicts veracity across disparate label spaces.
Results
49.2% Macro F1 is achieved by the best-performing model, with evidence encoding and metadata contributing to performance.
Takeaways & Limitations
MultiFC provides a challenging real-world testbed for claim-veracity prediction with naturally occurring claims, evidence, and rich metadata.
Takeaways & Limitations
Website crawling, label availability, metadata availability, and article or full-text availability vary across sources.
Abstract
from arXiv · showhide
We contribute the largest publicly available dataset of naturally occurring factual claims for the purpose of automatic claim verification. It is collected from 26 fact checking websites in English, paired with textual sources and rich metadata, and labelled for veracity by human expert journalists. We present an in-depth analysis of the dataset, highlighting characteristics and challenges. Further, we present results for automatic veracity prediction, both with established baselines and with a novel method for joint ranking of evidence pages and predicting veracity that outperforms all baselines. Significant performance increases are achieved by encoding evidence, and by modelling metadata. Our best-performing model achieves a Macro F1 of 49.2%, showing that this is a challenging testbed for claim veracity prediction.
1 Introduction
MultiFC addresses the difficulty of automating fact checking by introducing a large, naturally occurring, multi-domain claim-verification dataset with evidence and metadata. Its models use these resources to predict veracity, with the best model reaching a Macro F1 of 49.2%.
- The dataset targets automatic veracity prediction for naturally occurring claims, addressing limitations of prior small or artificial datasets.
- 34,918 claims from 26 English fact-checking websites form the dataset, with evidence pages, occurrence context, and rich metadata.
- Evidence pages and metadata significantly contribute to veracity-prediction performance.
- 49.2% Macro F1 is achieved by the best-performing model, indicating that the dataset remains challenging.
2 Related Work
Prior fact-checking datasets are often small, lack evidence annotations, or use artificial claims, while existing evidence-based systems commonly separate evidence identification from prediction. MultiFC contributes a large natural-claim dataset and jointly learns evidence importance for veracity prediction.
- Datasets differ in whether they provide claim context, evidence documents, or annotations for those documents.
- Existing fact-checking datasets are mostly small, and larger datasets often lack evidence documents or use artificially constructed claims.
- MultiFC uniquely combines many naturally occurring claims with rich additional metadata.
- Prior evidence-based models generally use pipelines rather than jointly learning evidence identification with stance or veracity prediction.
- The proposed joint approach learns to weigh evidence pages by their importance for veracity prediction and improves downstream performance.
3 Dataset Construction
The dataset was constructed by crawling English fact-checking sources, collecting claims, metadata, linked evidence pages, and entity annotations, followed by cleansing and filtering. The resulting multi-domain collection contains 34,918 claims with heterogeneous labels and label-stratified train, development, and test splits.
- 3.1 Selection of sources: The authors crawled 43,837 claims and metadata from active English fact-checking websites identified through two source lists.They describe source selection, claim and metadata crawling, evidence-page retrieval, and entity linking as the main collection stages.
- 3.1 Selection of sources: The crawl collected claim labels, URLs, rationales, categories, speakers, checkers, tags, dates, article text, and linked outlink text.For each source, the authors also stored article titles, publication dates, claim dates, and related page content.
- 3.1 Selection of sources: Crawling was incomplete because of SSL/TLS protection, timeouts, PDF URLs, encoding problems, and uneven metadata or full-text availability.Web-search results were also acknowledged to change over time, although temporal effects were outside the paper’s scope.
- 3.1 Selection of sources: After removing duplicate or label-leaking claims and merging clearly equivalent within-domain labels, the collection contained 36,534 claims with metadata.Duplicate claims with differing labels were assigned to the training split to avoid an unfair advantage.
- 3.1 Selection of sources: Filtering labels occurring fewer than 5 times produced 34,918 claims split into 80% training, 10% development, and 10% testing sets.Labels were retained across domains rather than mapped onto a common veracity scale; domains contain between 2 and 27 labels.
4 Claim Veracity Prediction
The paper formulates claim veracity prediction across domains as a multi-task learning problem with disparate label spaces, comparing claim-only and evidence-based models. Its proposed ranked-evidence model learns evidence utility implicitly and combines evidence ranking with veracity prediction.
- Model Variants: The models predict claim veracity either from claim text alone or by additionally encoding evidence pages and optional metadata.The claim-only baseline relies on surface patterns, while evidence-based variants encode 10 search snippets rather than full retrieved pages.
- Multi-Task Formulation: 165 heterogeneous website labels motivate learning label relationships instead of manually mapping labels across domains.Each domain is modelled as a task, and labels are projected into a shared fixed-length embedding space.
- Multi-Task Formulation: The base multi-task model shares parameters across tasks while retaining task-specific softmax output layers and minimizing the sum of task losses.The model is trained with a negative log-likelihood objective.
- Label Embedding Layer: The label embedding layer shares a joint Euclidean space across tasks and applies task-specific masks to produce domain-specific probability distributions.Compatibility between label embeddings and the jointly learned hidden representation is used for prediction.
- Evidence Ranking: The crawled avg variant averages all evidence-page embeddings, whereas crawled ranked learns claim–evidence compatibility and uses the resulting ranking to weight evidence.The ranked model has no direct document-ranking labels, so it learns evidence ranks implicitly from claim veracity supervision.
- Evidence Ranking: The joint model projects claim–evidence representations into a soft ranking and combines ranking scores with label scores through a dot product.Unlike the cited prior matching model, the evidence weights are learned end-to-end without direct labels for individual claim–evidence pairs.
5 Experiments
Experiments compare claim-only and evidence-based veracity prediction across domains, with metadata and multi-task modeling. Evidence ranking and metadata improve performance, although results vary substantially by domain.
- Experimental setup: The base sentence encoder is a BiLSTM with randomly initialized word embeddings, while claim-only variants use BiLSTM or mean-averaged word embeddings.
- Training setup: Label embeddings improve results, and multi-domain models outperform single-task models.
- Veracity prediction: Evidence-based models outperform claim-only models by a large margin.
- Evidence ranking: Crawled ranked achieves the highest Micro F1 and Macro F1 by weighting evidence pages according to their utility.The model learns evidence importance implicitly because individual evidence-ranking labels are unavailable.
- Metadata: Topic tags contribute most among metadata types, while all metadata together yields higher Macro F1 at similar Micro F1 than no metadata.
- Metadata: Metadata encoding hurts domains without metadata but improves domains where metadata is available.The authors suggest an ensemble of models for the two availability conditions.
6 Analysis and Discussion
The analysis identifies which claims and evidence are easiest or hardest for the best model. Classification is strongest for short, specific, evidence-overlapping claims and weaker for longer, ambiguous ones.
- Error analysis: The model struggles more with ‘true’ than ‘false’ claims and often confuses nearby labels such as ‘half-true’ and ‘mostly true’.
- Instance properties: Longer claims are harder to classify correctly, while high token overlap with evidence pages produces high evidence rankings.
- Metadata and entities: Specific tags and entities tend to co-occur with correct predictions, whereas general tags tend to co-occur with incorrect predictions.Examples include ‘brisbane-4000’ and ‘hong-kong’ versus ‘government-and-politics’ and ‘tax’.
- Discussion: Short claims with specific topics and informative evidence are easiest, while longer complex claims with ambiguous evidence remain challenging.
7 Conclusions
The paper introduces a large real-world fact-checking dataset and a joint model that ranks evidence pages while predicting claim veracity. Metadata and evidence encoding improve performance.
- The dataset contains 34,918 claims from 26 fact-checking websites, rich metadata, and 10 retrieved evidence pages per claim.
- The paper introduces a joint model for ranking evidence pages and predicting veracity.
- Encoding metadata and evidence pages helps veracity prediction.