Source-linked AI summary
"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection
William Yang Wang
TL;DR
Fake news detection lacks sufficiently large, manually labeled benchmarks, especially for short political statements. The paper introduces LIAR, a decade-long PolitiFact dataset with fine-grained labels and metadata, and evaluates text-only and hybrid neural models. Combining metadata with text improves fine-grained fake news detection, while the dataset also supports future fact-checking and related political NLP research.
Problem
Fake news detection is difficult for short political statements, and manually labeled benchmark datasets remain a bottleneck for advancing broad-coverage computational models.
Method
The paper constructs LIAR from 12,836 PolitiFact statements with truthfulness labels, contextual and speaker metadata, analysis reports, and source links, then evaluates text-only and hybrid CNN models.
Results
Combining metadata with text significantly improves fine-grained fake news detection over text-only modeling.
Takeaways & Limitations
LIAR supports automatic fake news detection and future automatic fact-checking, as well as stance classification, argument mining, topic modeling, rumor detection, and political NLP research.
Abstract
from arXiv · showhide
Automatic fake news detection is a challenging problem in deception detection, and it has tremendous real-world political and social impacts. However, statistical approaches to combating fake news has been dramatically limited by the lack of labeled benchmark datasets. In this paper, we present liar: a new, publicly available dataset for fake news detection. We collected a decade-long, 12.8K manually labeled short statements in various contexts from PolitiFact.com, which provides detailed analysis report and links to source documents for each case. This dataset can be used for fact-checking research as well. Notably, this new dataset is an order of magnitude larger than previously largest public fake news datasets of similar type. Empirically, we investigate automatic fake news detection based on surface-level linguistic patterns. We have designed a novel, hybrid convolutional neural network to integrate meta-data with text. We show that this hybrid approach can improve a text-only deep learning model.
1 Introduction
Fake news detection is difficult because short political statements differ from crowdsourced review data, while manually labeled benchmark datasets remain scarce. LIAR addresses this gap with a substantially larger, naturally grounded dataset and evaluates text-only and hybrid models.
- Motivation: Short political statements make fake news detection more challenging than deceptive-review detection.Existing review datasets often use simulated positive examples that mismatch real-world testing conditions.
- Benchmark gap: 221 statements in the first public fake news and fact-checking dataset were insufficient for machine-learning assessment.
- Contribution: LIAR introduces 12,836 manually labeled short statements collected from natural political and social-media contexts.Each case includes a detailed analysis report and links to supporting documents.
- Contribution: The dataset is an order of magnitude larger than comparable available resources and spans a decade of statements.
- Evaluation: A hybrid neural architecture integrates text with metadata and improves a strong text-only convolutional neural-network baseline.
2 LIAR: a New Benchmark Dataset
LIAR is a large, naturally grounded benchmark of short political statements with fine-grained truthfulness labels, speaker metadata, and broad topical and contextual coverage.
- Prior resources: Prior fake-news resources contained 221 or 300 labeled items, making them impractical as machine-learning benchmarks.
- Dataset construction: LIAR contains 12.8K human-labeled short statements from PolitiFact.com, each evaluated by an editor for truthfulness.
- Labels: The dataset uses six fine-grained truthfulness labels, with all but 1,050 pants-fire cases ranging from 2,063 to 2,638 instances.
- Metadata: Speaker metadata includes party, job, home state, and historical counts of inaccurate statements, with the current label subtracted during prediction.
- Coverage: Statements span news releases, interviews, campaign speeches, advertisements, tweets, debates, and Facebook posts across diverse political subjects.
3 Automatic Fake News Detection
The paper frames fake news detection as six-way classification of short statements and asks whether surface text and speaker metadata can support accurate prediction. Its hybrid CNN combines text and metadata representations before final classification.
- Task: Fake news detection is framed as a 6-way multiclass text-classification problem.
- Research questions: The study asks how well surface-level linguistic realizations classify short statements into fine-grained fakeness categories.
- Research questions: A second research question asks whether speaker metadata can be integrated with text to enhance detection performance.
- Model: The proposed hybrid CNN encodes metadata, applies convolution and max-pooling, combines metadata with bidirectional-LSTM text representations, and predicts through a softmax layer.
4 LIAR: Benchmark Evaluation
The evaluation compares standard classifiers and neural models under text-only and text-plus-metadata settings. CNNs perform best among text-only models, while the hybrid model achieves the best test result overall.
- Experimental settings: The study compares majority, logistic-regression, SVM, bidirectional-LSTM, CNN, and hybrid models.
- Experimental settings: The evaluation separates text-only models from text-plus-metadata hybrid models.
- Experimental settings: Accuracy is used as the evaluation metric because results were equivalent to F-measures on the balanced dataset.
- Results: 0.270 accuracy is achieved by the text-only CNN on the heldout test set, outperforming the other text-only models.The CNN was significantly better than SVM under a two-tailed paired t-test (p < .0001).
- Results: The model using all metadata and text achieves the best result on the test data.
5 Conclusion
The paper concludes that LIAR expands the resources available for fake news detection and supports broader research using authentic, context-rich statements. It also reports that combining metadata with text improves fine-grained fake news detection.
- LIAR is an order of magnitude larger than prior datasets, enabling statistical and computational approaches to fake news detection.
- Authentic short statements from diverse contexts and speakers support development of broad-coverage fake news detectors.
- Combining metadata with text yields significant improvements for fine-grained fake news detection.