Source-linked AI summary
Automatic Detection of Fake News
Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, Rada Mihalcea
TL;DR
The paper addresses the need for reliable automatic identification of fake online news. It constructs two multi-domain datasets, analyzes linguistic differences, and evaluates feature-based detectors, whose best accuracies are comparable to human performance while dataset scope remains constrained by celebrity-topic diversity and crowdsourced-news challenges.
Problem
Misinformation in social media, news blogs, and online newspapers makes identifying trustworthy online news a pressing computational problem.
Method
The paper builds two fake-news datasets, analyzes linguistic properties, and trains detectors using lexical, syntactic, semantic, psycholinguistic, and readability features.
Results
Best-performing models achieved accuracies comparable to human ability to spot fake content.
Takeaways & Limitations
Fake-news detection can be studied with multi-domain datasets and linguistic models that capture differences between fake and legitimate content.
Takeaways & Limitations
The web dataset has limited topic diversity in celebrity news, while crowdsourcing fake news poses journalistic-style and article-length challenges.
Abstract
from arXiv · showhide
The proliferation of misleading information in everyday access media outlets such as social media feeds, news blogs, and online newspapers have made it challenging to identify trustworthy news sources, thus increasing the need for computational tools able to provide insights into the reliability of online content. In this paper, we focus on the automatic identification of fake content in online news. Our contribution is twofold. First, we introduce two novel datasets for the task of fake news detection, covering seven different news domains. We describe the collection, annotation, and validation process in detail and present several exploratory analysis on the identification of linguistic differences in fake and legitimate news content. Second, we conduct a set of learning experiments to build accurate fake news detectors. In addition, we provide comparative analyses of the automatic and manual identification of fake news.
1 Introduction
The paper addresses the growing need to identify fake online news and develops datasets and linguistic-feature-based detectors for this task.
- Online misinformation across social media, news blogs, and online newspapers makes identifying trustworthy sources a pressing need.The motivation is especially salient because social media is a major news source for U.S. adults.
- Earlier computational approaches often used satirical or fact-checking sources, introducing confounds such as humor, absurdity, or domain concentration.
- The paper introduces two datasets spanning seven news domains, analyzes linguistic differences, and trains fake-news detectors using linguistic features.
- 78% accuracy is achieved by the reported detectors, with results compared against an empirical human baseline.
2 Related Work
Prior work has studied fake news through conceptual categories, satire classification, stylometry, and broader deceptive-content analysis.
- Fake news is distinguished conceptually into serious fabrications, hoaxes, and satire; this paper focuses on serious fabrications.
- Satire-versus-real-news classification achieved an F1 score of 0.87 with absurdity, punctuation, and grammar features.
- Stylometric research has used character and stop-word n-grams, readability indices, external links, and paragraph-length features.
- Deceptive-content studies across forums, reviews, advertising, dating, and crowdfunding have examined linguistic and structural cues.
3 Fake News Datasets
The paper constructs two fake-news datasets designed to reduce prior confounds: a crowdsourced six-domain corpus and a web-collected celebrity corpus.
- Dataset guidelines: The datasets include real and fake text, text-only items, verifiable ground truth, matched length and style, a defined timeframe, public availability, and comparable presentation conditions.
- Crowdsourced dataset: The crowdsourced dataset uses six domains and manually fact-checked legitimate news collected from mainstream websites.The legitimate set contains 240 articles, with 40 articles in each domain.
- Crowdsourced dataset: Workers created fake headlines and bodies from legitimate excerpts while matching topic and length; submissions were manually checked for spam and guideline compliance.
- Crowdsourced dataset: AMT workers successfully mimicked the original reporting style, partly addressing concern that author style could introduce analytical noise.
- Web dataset: The web dataset pairs legitimate and fake celebrity articles, with claims evaluated using gossip-checking sites and cross-source comparisons.
- Web dataset: Celebrity news has limited topic diversity because it concentrates on sensational subjects and follows some celebrities more than others.
- Web dataset: The Celebrity dataset contains 100 fake and 100 legitimate articles, averaging 399 versus 709 words per article, respectively.
4 Linguistic Features
The paper represents news with complementary linguistic feature sets spanning lexical content, punctuation, psycholinguistic categories, readability, and syntax.
- The models use unigram and bigram tf-idf features to represent article content while accounting for differences in text length.
- Punctuation features encode eleven punctuation types, including periods, commas, dashes, question marks, and exclamation marks.
- Psycholinguistic features use LIWC proportions grouped into summary, linguistic-process, and psychological-process categories.
- Readability features capture text-content properties and metrics including Flesch-Kincaid, Flesch Reading Ease, Gunning Fog, and ARI.
- Syntax features are lexicalized CFG production rules from Stanford Parser trees, combined with parent and grandparent nodes and encoded as tf-idf values.
5 Computational Models for Fake News Detection
The paper evaluates linear SVM fake-news detectors across feature sets, datasets, training sizes, and domains. Within-domain models perform above chance, while cross-domain performance drops and varies by domain and feature set.
- Linear SVM classifiers are evaluated with five-fold cross-validation using accuracy, precision, recall, and F1.
- Readability performs best on FakeNewsAMT, whereas Punctuation is most accurate on Celebrity, followed by Ngrams, Complete LIWC, and Syntax.
- Most classifiers perform above the 0.50 random baseline, with feature-set rankings differing between the two datasets.
- Except for Celebrity Readability, learning curves generally improve steadily as training data increases across both datasets.
- Cross-domain evaluation produces a significant accuracy loss relative to within-domain results, and crowdsourced-to-celebrity transfer is biased toward predicting truth.
- A cross-domain Readability classifier reaches 0.61 on FakeNewsAMT compared with 0.70 in the reported comparison.
- Cross-domain accuracy reaches 0.91 for Technology and Politics with Readability, while Sports, Business, and Entertainment are less generalizable.
6 Human Performance
The study compares human judgments with automatic fake-news detection on the FakeNewsAMT and Celebrity datasets. Humans performed better on celebrity news, while the automatic system surpassed humans on more serious and diverse news sources.
- Human annotators evaluated 480 FakeNewsAMT articles and 200 Celebrity articles, with two annotators labeling each dataset.Articles were presented in random order, and agreement statistics were reported for both datasets.
- Human judgments showed moderate agreement, with slightly lower Kappa for FakeNewsAMT than for the Celebrity dataset.
- Humans were better at identifying fake news in the Celebrity domain than in other domains.
- The automatic system outperformed humans when detecting fake news from more serious and diverse news sources.The comparison used accuracy scores for human annotators and the developed system.
7 Further Insights
The analyses identify systematic linguistic differences between fake and legitimate news across general and celebrity domains. Fake content tends toward social, positive, perceptual, certain, and present- or future-oriented language, whereas legitimate content more often expresses cognitive and tentative processes.
- Cross-domain findings: LIWC-based semantic classifiers showed consistently good performance across domains.
- FakeNewsAMT: In FakeNewsAMT, legitimate news used more cognitive-process, function, negation, and relativity language, while fake news used more social, positive, certain, and future-oriented language.Fake news also used more adverbs, verbs, and punctuation characters.
- Celebrity: In the Celebrity dataset, fake content used more perceptual words, positive emotions, the pronoun “I,” and prepositions.
- Celebrity: Legitimate celebrity content more often used cognitive-process words related to insight, cause, discrepancy, and tentative language.
8 Conclusions
The paper develops datasets and classification models for automatic fake-news identification, using lexical, syntactic, semantic, and readability information. Its best models achieved accuracies comparable to human ability to spot fake content.
- The paper introduces two fake-news datasets: one crowdsourced across six news domains and one web-collected from celebrity news.
- Classification models combine lexical, syntactic, semantic, and text-readability features.
- The best-performing models achieved accuracies comparable to human ability to identify fake content.