Source-linked AI summary
A Stylometric Inquiry into Hyperpartisan and Fake News
Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, Benno Stein
TL;DR
Fake-news detection is commonly approached through fact checking, but this paper examines whether writing style reveals connections between fake news and hyperpartisan news. Using a journalist-annotated corpus and an adaptation of Unmasking, it finds strong stylistic distinctions for hyperpartisanship and satire, while style alone does not solve fake-news detection.
Problem
The paper asks whether fake news and hyperpartisan news can be studied and detected through writing style rather than relying only on fact checking.
Method
The study analyzes a BuzzFeed journalist-annotated news corpus and adapts Unmasking to compare writing styles across text categories.
Results
The paper finds that hyperpartisan styles differ from mainstream style, left-wing and right-wing styles are similar, satire is distinguishable, and style alone does not detect fake news adequately.
Takeaways & Limitations
Writing style can support distinguishing hyperpartisan and satirical news, but it is insufficient as a standalone fake-news detector.
Takeaways & Limitations
The operational fake-news category combines mostly false articles with mixtures of true and false, while excluding satire and non-factual articles.
Abstract
from arXiv · showhide
This paper reports on a writing style analysis of hyperpartisan (i.e., extremely one-sided) news in connection to fake news. It presents a large corpus of 1,627 articles that were manually fact-checked by professional journalists from BuzzFeed. The articles originated from 9 well-known political publishers, 3 each from the mainstream, the hyperpartisan left-wing, and the hyperpartisan right-wing. In sum, the corpus contains 299 fake news, 97% of which originated from hyperpartisan publishers. We propose and demonstrate a new way of assessing style similarity between text categories via Unmasking---a meta-learning approach originally devised for authorship verification---, revealing that the style of left-wing and right-wing news have a lot more in common than any of the two have with the mainstream. Furthermore, we show that hyperpartisan news can be discriminated well by its style from the mainstream (F1=0.78), as can be satire from both (F1=0.81). Unsurprisingly, style-based fake news detection does not live up to scratch (F1=0.46). Nevertheless, the former results are important to implement pre-screening for fake news detectors.
1 Introduction
The paper frames fake news as highly one-sided, inflammatory, emotional, and often untruthful content amplified by social media, then investigates it through writing style. It introduces style-based experiments, Unmasking, and a journalist-annotated dataset to study hyperpartisan and fake news.
- Social media can amplify fake news beyond real news, making rapid detection important before viral spread becomes difficult to contain.The paper connects this problem to proposed detection and countermeasure workflows.
- The study approaches fake news detection through writing style rather than primarily through automatic fact-checking.It examines style in relation to hyperpartisan news.
- Unmasking is introduced as an approach for assessing and visualizing writing-style similarities between text categories.The paper applies it to compare political news orientations.
- The experiments ask whether hyperpartisan news differs stylistically from mainstream news, whether satire differs from both, and whether style alone detects fake news.These questions define the paper’s central empirical agenda.
- The study uses a publicly shared dataset annotated for fake-news and hyperpartisan labels by BuzzFeed journalists across nine well-known publishers.The publishers include mainstream, left-wing, and right-wing sources.
2 Related Work
Related work organizes fake-news detection around knowledge, context, and style-based paradigms. The paper positions its contribution within style-based detection, connecting deception detection, text categorization, and satire identification.
- Fake-news detection research is organized into knowledge-based, context-based, and style-based paradigms.Knowledge methods use factual resources, context methods study dissemination, and style methods model textual manifestations of faking.
- Knowledge-based approaches compare extracted claims with indexed or structured factual knowledge to identify inconsistencies.These methods rely on information retrieval, semantic-web, or linked-open-data techniques.
- Context-based approaches use social-network mechanisms to study how misinformation and rumors spread and how their dissemination can be limited.Some work combines network analysis with linguistic features.
- Style-based detection sidesteps fact checking and social-network analysis by modeling deception and textual style.Its foundations include forensic-linguistic deception detection and style-based text categorization.
- Prior work separated satire from real news with F-Measure values between 0.82 and 0.87, while this paper studies fake, real, and satire together.The cited prior dataset contained 180 satire and 180 real-news articles.
3 The BuzzFeed-Webis Fake News Corpus 2016
The BuzzFeed-Webis corpus combines articles from mainstream and hyperpartisan publishers with professional manual fact-checking. Its annotations support analyses of factuality, satire, publisher orientation, and article statistics.
- The corpus contains articles from nine publishers: three mainstream, three hyperpartisan left-wing, and three hyperpartisan right-wing publishers.It sampled publisher output during a period close to the US elections.
- 1,627 articles were recovered, including 826 mainstream, 256 left-wing, and 545 right-wing articles.Website changes limited recovery from the archived material.
- Five BuzzFeed journalists manually fact-checked the articles, using multiple factuality categories instead of a binary fake-versus-real distinction.Uncertain ratings received additional review, and mostly false items received a final check by another rater.
- None of the mainstream articles were mostly false, whereas faulty articles occurred among both left-wing and right-wing hyperpartisan sources.The passage reports more than a quarter faulty among left-wing articles and almost 45% among right-wing articles, excluding non-factual articles.
- Mainstream articles averaged about 20 paragraphs, while left-wing and right-wing articles averaged roughly 420 and 400 words, respectively.Left-wing articles also contained almost twice as many links as mainstream and right-wing articles.
- The corpus defines fake news operationally as articles rated mostly false or a mixture of true and false, while mostly true articles are treated as real news.Satire and non-factual articles are excluded from these categories.
4 Methodology
The methodology adapts Unmasking from authorship verification to compare writing styles across political news categories, using lexical, grammatical, readability, and dictionary features. Topic-based and naive baselines provide comparisons for classification experiments.
- Unmasking is adapted from authorship verification to assess stylistic similarity between broader categories such as left-wing, right-wing, and mainstream news.The adaptation targets involuntarily adopted styles associated with political orientation.
- Standard Unmasking chunks documents, removes discriminative features iteratively, measures reconstruction errors, and interprets the resulting curve slope.A steeper decrease indicates greater stylistic similarity between the compared inputs.
- The category-level adaptation uses two document sets as input and plots reconstruction-error curves for visual comparison.It skips the original document-chunking step.
- The style model includes character n-grams, stop words, parts of speech, ten readability scores, and dictionary-frequency features.Features are selected per experiment and low-frequency features are discarded to reduce overfitting.
- A bag-of-words topic classifier serves as a baseline, supplemented by naive classifiers based on class distributions.The paper notes that topic-based classification is less practical because news topics change frequently.
5 Experiments
The experiments compare writing styles across political orientations and news veracity categories. Unmasking reveals shared style between left- and right-wing hyperpartisan news, while style distinguishes hyperpartisan and satire from mainstream or other news better than it detects fake news.
- 5.1 Hyperpartisanship vs. Mainstream: Unmasking curves decrease quickly for all orientation pairs, indicating a shared news-genre style across political categories.The experiments average curves over five runs using sets of 100 documents from each orientation.
- 5.1 Hyperpartisanship vs. Mainstream: Left-wing and right-wing documents share more stylistic similarity with each other than either does with mainstream documents.The comparison is based on the relative steepness of the Unmasking curves, where steeper decreases indicate greater similarity.
- 5.1 Hyperpartisanship vs. Mainstream: 0.74 accuracy on left-wing test documents and 0.66 accuracy on right-wing test documents are achieved when the opposite hyperpartisan side is omitted from training.In both settings, the style-based classifier outperforms the topic-feature baseline.
- 5.1 Hyperpartisanship vs. Mainstream: 0.75 accuracy and 0.89 recall for hyperpartisan news are achieved by the style-based classifier when both hyperpartisan sides appear in training.This classifier outperforms the topic-feature model in the reported comparison.
- 5.2 Fake vs. Real (vs. Satire): Style-based fake-news classifiers slightly improve precision over naive baselines but lose substantial recall and do not outperform them on F-Measure.Orientation-specific classifiers are slightly better for most metrics, but none surpasses the naive baselines in F-Measure.
- 5.2 Fake vs. Real (vs. Satire): 0.82 accuracy and 0.81 F score are achieved by the style-based satire classifier, which significantly outperforms all baselines across the board.It improves over pure topic classification but does not outperform Rubin et al.’s classifier using topic, absurdity, grammar, and punctuation features.
6 Conclusion
The paper argues that writing style offers a useful angle on fake news detection: hyperpartisan and mainstream news differ stylistically, while opposing hyperpartisan orientations share a common style. Style also distinguishes satire, but style analysis alone does not solve fake news detection.
- Writing style provides an alternative to fact-checking for studying fake news in relation to hyperpartisan news.The paper frames style analysis as a complementary angle because writing style is difficult to capture computationally and difficult to manipulate consistently.
- Hyperpartisan news can be fairly easily distinguished from more balanced mainstream news by its writing style.
- Left-wing and right-wing hyperpartisan news have very similar writing styles, suggesting a common style of extremism.
- Satire can be distinguished well from other news, helping prevent humor from being rejected by fake news detection technology.
- Style analysis alone does not solve fake news detection, although the other style-based findings are important for pre-screening fake news detectors.