Source-linked AI summary
Fake News Early Detection: An Interdisciplinary Study
Xinyi Zhou, Atishay Jain, Vir V. Phoha, Reza Zafarani
TL;DR
Fake news can spread widely and threatens democracy, but propagation signals are unavailable before social-media dissemination. This paper develops a theory-driven content model using four language levels and evaluates it with supervised learning. On two real-world datasets, the model generally achieves ∼88% accuracy and F1 score, while maintaining ∼80% and ∼88% as data size and class distribution vary.
Problem
Propagation-based fake news detection cannot support early detection before social-media spread, motivating content-focused approaches.
Method
The paper represents news with manual lexicon-, syntax-, semantic-, and discourse-level features inspired by social and forensic psychology, then applies supervised machine learning.
Results
∼88% accuracy and F1 score generally outperform all baselines, while performance maintains ∼80% and ∼88% under varying data size and fake-to-true-news distributions.
Takeaways & Limitations
The model supports interpretable fake news early detection when social-media propagation information and content information are limited.
Abstract
from arXiv · showhide
Massive dissemination of fake news and its potential to erode democracy has increased the demand for accurate fake news detection. Recent advancements in this area have proposed novel techniques that aim to detect fake news by exploring how it propagates on social networks. Nevertheless, to detect fake news at an early stage, i.e., when it is published on a news outlet but not yet spread on social media, one cannot rely on news propagation information as it does not exist. Hence, there is a strong need to develop approaches that can detect fake news by focusing on news content. In this paper, a theory-driven model is proposed for fake news detection. The method investigates news content at various levels: lexicon-level, syntax-level, semantic-level and discourse-level. We represent news at each level, relying on well-established theories in social and forensic psychology. Fake news detection is then conducted within a supervised machine learning framework. As an interdisciplinary research, our work explores potential fake news patterns, enhances the interpretability in fake news feature engineering, and studies the relationships among fake news, deception/disinformation, and clickbaits. Experiments conducted on two real-world datasets indicate the proposed method can outperform the state-of-the-art and enable fake news early detection when there is limited content information.
1 INTRODUCTION
Fake news poses substantial societal and economic risks, while manual fact-checking cannot scale and propagation-based methods cannot support detection before social-media spread. The paper therefore proposes an interpretable, theory-driven content model for early detection.
- Fake news generated 8,711,000 Facebook shares, reactions, and comments during the 2016 U.S. presidential campaign, exceeding the 7,367,000 for major news websites’ election stories.
- Human deception-detection accuracy is typically 55%-58%, with a 54% mean across 1,000 participants and over 100 experiments.
- Manual fact-checking does not scale with newly created information, especially on social media, motivating automatic fake news detection.
- Propagation-based detection cannot operate before social-media dissemination because limited or no social context exists when an article is posted.
- The proposed model mines news content through manual features spanning lexicon, syntax, semantics, and discourse, guided by social and forensic psychology theories.
- Experiments on real-world datasets validate early detection with limited content information and reveal patterns shared by or unique to fake news, deception, and clickbait.
2 RELATED WORK
Related work divides fake news detection into content-based and propagation-based approaches, while theory-informed content features address interpretability and early-detection constraints. Prior research also connects fake news with deception and clickbait through linguistic and dissemination patterns.
- Content-based Fake News Detection: Content-based methods represent news through knowledge, style, or latent features to assess or classify fake news.
- Content-based Fake News Detection: Knowledge-based methods compare extracted subject-predicate-object tuples with knowledge graphs, but face incompleteness, timeliness, and difficulty distinguishing false from intentionally false news.
- Content-based Fake News Detection: Style features can detect fake news but are often selected without fundamental cross-disciplinary theories, whereas latent features are difficult for the public to comprehend.
- Propagation-based Fake News Detection: Propagation-based methods use social context such as cascades, user roles, stances, sentiments, and relationships among articles, publishers, users, and posts.
- Propagation-based Fake News Detection: Early detection requires mining news content because social context is limited or unavailable before an article spreads on social media.
- Deception and Clickbait: Psychological theories identify linguistic cues involving writing quality, sensory-perceptual information, emotions, cognitive processes, and extreme information quantity in deception.
- Deception and Clickbait: Clickbait headlines create information gaps to attract attention, and clickbaits are closely correlated with fake news, although not every clickbait article is fake.
3 METHODOLOGY
The methodology uses theory-driven, interpretable linguistic features spanning lexicon, syntax, semantics, and discourse, then applies supervised learning to classify fake news. It also models clickbait and disinformation-related attributes to capture potential fake-news patterns.
- 3.2 News Classification: The resulting feature representation is used with a supervised classifier whose parameters are estimated from labeled news articles.The prediction task maps a content feature vector to a fake or true news label.
- 3.1 News Representation: The model represents news writing style at lexicon, syntax, semantic, and discourse levels, following the Undeutsch hypothesis that fake and true news may differ stylistically.These representations are designed to support interpretable fake-news feature engineering.
- 3.1.1 Lexicon-level: Lexicon-level features standardize word frequencies to account for differences in article length.The approach uses word occurrence frequencies while addressing the limitation of absolute Bag-Of-Words counts.
- 3.1.2 Syntax-level: Syntax-level features measure relative frequencies of Part-Of-Speech tags and PCFG rewrite rules extracted from parsing trees.The parsing-tree example illustrates how grammatical productions are represented as rewrite rules.
- 3.1.3 Semantic-level: Semantic-level features capture psycho-linguistic attributes inspired by theories from forensic and social psychology, including sentiment-related signals.The theories primarily concern deception, disinformation, and clickbait rather than fake news specifically.
- 3.1.3 Semantic-level: Clickbait attributes cover dictionary phrases, headline readability, sensationalism, and news-worthiness-related properties.Sensationalism features include sentiment, punctuation, and headline–body similarity; news-worthiness includes quality and informality.
- 3.1.4 Discourse-level: Discourse-level features use standardized frequencies of rhetorical relationships extracted from RST parse trees.The resulting tree represents phrases or sentences as leaves and rhetorical relationships as non-leaf nodes.
4 EXPERIMENTS
The experiments evaluate the proposed fake-news detection model using an experimental setup and performance analysis described in separate sections. The stated evaluation examines the model empirically.
- 4 Experiments: The empirical evaluation specifies its experimental setup in Section 4.1 and presents and evaluates performance in Section 4.2.The supplied passage identifies the organization of the experiments but does not report performance values.
4.1 Experimental Setup
The experimental setup uses two public benchmark datasets with expert-provided labels and compares the proposed method with content-based, propagation-based, and hybrid baselines.
- 4.1.1 Datasets: The experiments use two public benchmark datasets collected from PolitiFact and BuzzFeed, with fake-or-true labels supplied by fact-checking experts.Both datasets also include substantial Twitter user-network and dissemination information.
- 4.1.2 Baselines: The proposed method is compared with state-of-the-art approaches based on news content, social-network dissemination, or both information sources.The comparison spans content-only, propagation-only, and combined detection settings.
- 4.1.2 Baselines: Content-based baselines include a comprehensive linguistic model using n-grams, CFGs, LIWC proportions, and readability features.These features are used within a supervised machine-learning framework.
- 4.1.2 Baselines: Propagation-based baselines exploit user profiles, tweets, topics, URLs, sentiment, hashtags, and propagation trees to assess news credibility.Their features describe both users and how news spreads through social networks.
- 4.1.2 Baselines: Another baseline embeds relationships among news articles, publishers, and social-media spreaders before performing fake-news detection.Its representation combines news content, users, engagements, and publisher relationships.
- 4.1.2 Baselines: Additional baselines use word2vec and Doc2Vec embeddings, which are described as less explainable than style features but empirically useful.These provide latent representations of news articles for classification.
4.2 Performance Evaluation
The proposed content-based model slightly outperforms the hybrid baseline without propagation information, while combining interpretable language-level features improves performance. Experiments also show useful signals from disinformation- and clickbait-related attributes and support early detection with limited content.
- Overall performance: The model slightly outperforms the hybrid baseline and does not rely on propagation information.It can therefore evaluate fake news using content rather than social-network propagation signals.
- Language-level features: Lexicon-level BOWs and deep-syntax CFGs outperform other individual language-level features, exceeding 80% accuracy and F1 score.Semantic-level, shallow-syntax, and discourse-level features perform less strongly, with discourse-level features performing poorly within the framework.
- Language-level features: Combining features across language levels achieves approximately 88% accuracy and F1 score.The combined representation excludes rhetorical-relationship features and outperforms separate level-specific feature sets.
- Disinformation-related attributes: Combining disinformation-related attributes achieves 70%–80% accuracy and F1 on PolitiFact and 60%–70% on BuzzFeed.Individual attributes concerning content quality, sentiment, quantity, and specificity perform similarly at lower ranges.
- Clickbait-related attributes: Headline news-worthiness features achieve around 70% accuracy and F1, outperforming headline readability and sensationalism features.Readability, sensationalism, and news-worthiness all contribute to differentiating fake from true news.
- Early detection: With changing numbers of available training articles, the proposed model performs best in most cases.The authors note that temporal sampling would better evaluate early detection, but the datasets do not fully provide publication-time information.
- Early detection: With limited news content, the model always outperforms latent-content models and beats the linguistic baseline when only headlines are available.With varying partial content from headlines to randomly selected paragraphs, performance is generally comparable to the linguistic model and stronger than latent representations.
5 CONCLUSION
The paper presents an interdisciplinary, theory-driven approach for explainable fake news early detection before social-media propagation. It represents content across four language levels and reports strong performance and robustness across varying data conditions.
- The study targets explainable fake news detection before articles begin propagating on social media.
- The model represents news content at lexicon, syntax, semantic, and discourse levels using theories from social and forensic psychology.
- ∼88% accuracy and F1 score were generally achieved, outperforming content-based, propagation-based, and hybrid baselines.
- Performance remained ∼80% and ∼88% when data size and the fake-to-true news distribution varied.
- The study identifies differences between fake and true news in content style, quality, sentiment, sensationalism, and news-worthiness.
A SEMANTIC-LEVEL FEATURES
The semantic-level feature set combines linguistic, cognitive, perceptual, clickbait, sentiment, readability, and headline–body similarity measures. It includes both established resources and self-implemented counts or indices.
- Semantic-level features include cognitive-process and perceptual-process measures, including certainty, differentiation, seeing, hearing, and feeling.
- Clickbait-related attributes include common clickbait phrases, expressions, and patterns.
- Readability features include Flesch Reading Ease, Flesch-Kincaid Grade Level, Automated Readability, Gunning Fog, and Coleman-Liau indices.
- Sentiment features include negative and emotional word proportions and average word sentiment scores.
- Headline–body similarity is computed using Word2Vec and Sentence2Vec with cosine distance, alongside punctuation features.
- The feature set measures word, syllable, polysyllable, character, long-word, informal-word, and stop-word counts or proportions.