Source-linked AI summary
Exposing propaganda: an analysis of stylistic cues comparing human annotations and machine classification
Géraud Faye, Benjamin Icard, Morgane Casanova, Julien Chanson, François Maine, François Bancilhon, Guillaume Gadek, Guillaume Gravier, Paul Égré
TL;DR
The paper examines how propaganda’s stylistic language can be distinguished by people and machine classifiers. It builds PPN and compares human annotations with computational analyses; annotators reliably distinguished propaganda from regular articles despite shared topics.
Problem
The paper investigates how propaganda’s language and stylistic features can be analyzed and compared across human and machine classification.
Source: §1 · p. 1Method
The authors present PPN, annotate a sample mixed with regular French press articles, and analyze cues with VAGO and classification models.
Source: §1 · p. 1, §5 · p. 5, §6.2 · p. 7Results
Human annotators reliably distinguished propaganda articles from regular press articles despite their sharing topics.
Source: §1 · p. 1Takeaways & Limitations
The analyses identify exaggeration, reduced descriptive content, inadequate sourcing, vagueness, punctuation, and lexical features as cues associated with propaganda in the studied data.
Source: §7 · p. 9Annotation experiments covered only a subset of the French data, and detection experiments covered only two Romance and Germanic languages; performance for other dataset languages is not guaranteed.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
This paper investigates the language of propaganda and its stylistic features. It presents the PPN dataset, standing for Propagandist Pseudo-News, a multisource, multilingual, multimodal dataset composed of news articles extracted from websites identified as propaganda sources by expert agencies. A limited sample from this set was randomly mixed with papers from the regular French press, and their URL masked, to conduct an annotation-experiment by humans, using 11 distinct labels. The results show that human annotators were able to reliably discriminate between the two types of press across each of the labels. We propose different NLP techniques to identify the cues used by the annotators, and to compare them with machine classification. They include the analyzer VAGO to measure discourse vagueness and subjectivity, a TF-IDF to serve as a baseline, and four different classifiers: two RoBERTa-based models, CATS using syntax, and one XGBoost combining syntactic and semantic features.
1 Introduction
The paper examines propaganda’s language and stylistic features, comparing human and machine classification. It introduces a multilingual dataset and annotation experiment, then uses linguistic analysis and classifiers to investigate cues that distinguish propaganda articles.
1 Introduction
- The paper compares human and machine classification to examine propaganda’s language and stylistic features.Source: §1 · p. 1
- The paper presents PPN, a multisource, multilingual, multimodal dataset of articles from websites identified as propaganda sources by expert agencies.The sources were identified by NewsGuard and Viginum.Source: §1 · p. 1
- Annotators reliably distinguished PPN articles from regular French press articles despite their sharing topics.The experiment randomly mixed the articles for annotation.Source: §1 · p. 1
- VAGO analysis and machine-learning models are used to investigate vagueness, subjectivity, and cues for detecting articles from propagandist sources.The models include text-processing approaches and a model using stylistic and syntactic features; explainability is used to inspect learned features.Source: §1 · pp. 1–2
2 The PPN dataset
PPN brings together articles from five varied sources created after Russia’s 2022 invasion of Ukraine. The dataset presents articles in their original languages, while noting that some sites publish translations.
2 The PPN dataset
- PPN draws on five sources, all created after Russia’s invasion of Ukraine began on February 24, 2022.Their source-level article distribution is presented in Table 1.Source: §2 · p. 2, Table 1 · p. 2
- Three sources focus on opposing claims about the Ukraine war, while two French-language sites publish polarizing news aimed at weakening trust in Western institutions.The sites were created at different times, with the latter two appearing a year after the first three.Source: §2 · p. 2
- The dataset preserves propaganda articles in their original languages, although several sites publish translations.Table 2 presents the distribution of articles by language.Source: §2 · p. 2, Table 2 · p. 2
3 Annotated corpus and labels
The annotation study compares a filtered sample of French PPN articles with articles from five national newspapers, using 11 labels. The sample was checked for topic balance across the two press types.
3 Annotated corpus and labels
- The study selected 48 articles of each press type from the French PPN and five national newspapers, filtering by date, Ukraine- or Russia-related mentions, and length.Articles were limited to 1,000–10,000 characters to reduce length-related bias.Source: §3 · p. 2
- The 11 labels cover manipulative content, jocular or adversarial intent, opinion or description, and the quality of justification.“Fake News” was left without explicit criteria to study its predictors among the other labels; the labels were defined in an annotation manual.Source: §3 · pp. 2–3
- A topic analysis classified articles as directly about the armed conflict or about other topics, to check whether regular and propaganda articles covered roughly similar topics.The distribution is shown in Figure 2.Source: §3 · p. 3, Figure 2 · p. 3
- The annotated set contained 48 propaganda and 47 regular articles, with 27 Related and 21 Unrelated regular articles and 20 Related and 28 Unrelated propaganda articles.The sample was balanced overall; waronfakes had only conflict-related articles, while mediapart contributed only one article under the filters.Source: §3 · pp. 3–4
4 Analysis of the annotations
Human annotations sharply distinguish the two press types across nearly all labels, with stronger agreement on regular-press articles. The label correlations group descriptive and sourcing judgments apart from other labels, while Propaganda aligns most with Fake News, Pejorative, and Exaggeration.
4 Analysis of the annotations
- Removing 16 potentially confounding articles in post-hoc analyses left the same main contrasts; URLs had been removed to limit source bias.Source: §4 · p. 4
- All 10 labels other than Satirical differed significantly between press types at α = .01; Fake News and False Information were never applied to regular articles.Descriptive and Adequate Sources were more common in regular articles, while several manipulative labels were marginal there.Source: Figure 3 · p. 4, §4 · p. 4
- Mean scores exceeded .4 for every label in both press types, and annotator agreement was higher for regular-press articles across labels.Figure 3 reports mean scores and agreement by label.Source: Figure 3 · p. 4, §4 · p. 4
- Descriptive and Adequate Sources correlated strongly with each other and inversely with other labels; Propaganda correlated most with Fake News, Pejorative, and Exaggeration.Satirical, used in about 1.5% of annotations, was excluded from later analysis.Source: §4 · pp. 4–5, Figure 4 · p. 5
5 Analysis with the VAGO tools
VAGO scores show associations between stylistic measures and several human annotation labels, but explain only part of the variation in annotations. The “Vague” label is an exception, plausibly because its definition differs from the vagueness measured by VAGO.
5 Analysis with the VAGO tools
- VAGO calculates vagueness, opinion and relative-detail scores using lexical markers, including four types of vague expressions.Opinion scoring checks implicit and explicit subjectivity markers; relative detail compares named entities with vague terms.Source: §5 · p. 5
- “Subjective”, “Exaggeration” and “Pejorative” correlate positively with VAGO vagueness and opinion scores, and negatively with detail-vs-vagueness scores.Vagueness and opinion scores also correlate negatively with “Descriptive” and “Adequate Sources”.Source: §5 · p. 5
- “Propaganda”, “Dishonest Title”, “Fake News” and “False Information” correlate positively with VAGO vagueness scores only.Source: §5 · p. 5
- VAGO opinion scores have especially strong correlations with “Subjective” and “Descriptive”, as shown in Figure 5.Source: §5 · p. 5, Figure 5 · p. 6
- The human “Vague” annotations do not correlate with VAGO vagueness or detail scores; the authors conjecture that differing definitions may explain this.The human label targets generality, while VAGO covers multiple types of vagueness, including semantic vagueness from one- and multi-dimensional adjectives, which make up 96% of its lexicon.Source: §5 · p. 5
- VAGO scores track the clustering of labels, but explain only part of the variance in the annotations.The polarity of correlations for “Descriptive” and “Adequate Sources” is inverse to the other labels.Source: §5 · pp. 5–6
6 Machine learning for propaganda detection
The section evaluates models for detecting propaganda in Ukraine-invasion coverage and examines the cues they use. It reports strong detection by both large and shallow models, alongside topic-specificity concerns and analyses of stylistic markers.
6.1 Dataset for detecting propaganda related to the conflict
- The larger comparison corpus includes English and French regular-press articles related to Russia and Ukraine, with the English sample drawn from 11 reliable outlets.English articles were post-invasion, 1,000–10,000 characters, and topic-filtered; the French corpus was collected similarly but used fewer sources.Source: §6.1 · p. 6, Table 4 · p. 6, Table 5 · p. 6
6.2 Models
- The models combine text-processing approaches, including RoBERTa-based classifiers, with CATS and TF-IDF; the French suite also includes XGBoost using numerical text features.XGBoost features include sentence length, VAGO scores, sentiment, part-of-speech counts, and dependency counts, aggregated by summation.Source: §6.2 · pp. 6–7
6.3 Results
- Test results show propaganda detection on Ukraine-invasion articles is achievable by LLMs and shallow models, with CATS slightly below RoBERTa and XGBoost lower.CATS has 0.6 million parameters, about 200 times fewer than RoBERTa-base; XGBoost uses high-level text features.Source: Table 6 · p. 7, §6.3 · p. 7
6.4 Identified markers of propaganda
- CATS aggregates token contributions by sentence to identify salient article passages, and Figure 6 highlights those sentences alongside VAGO vocabulary.In the example, the selected sentences include laudatory exaggeration and pejorative or racist insinuation; VAGO identifies some subjective adjectives but misses other cues and complex syntactic markers.Source: §6.4 · pp. 7–8, Figure 6 · p. 8
6.5 Explainability of the XGBoost model
- Punctuation frequency is higher in regular articles, while propaganda articles have more periods but fewer question marks, commas, and quotation marks; regular articles also score higher on VAGO-N detail.Article lengths were normalized; VAGO-N differences in vagueness and opinion were not significant after Bonferroni correction.Source: §6.5 · p. 8, Figure 8 · p. 8, Figure 9 · p. 8
- SHAP analysis ranks punctuation count above sentence length among XGBoost features; clausal modifiers, nominal subjects, and sentence count receive similar weight.Figure 7 displays the five top syntactic features.Source: Figure 7 · p. 8, §6.5 · p. 8
6.6 Potential biases of machine learning models
- Near-perfect LLM accuracy raises questions about shallow learned cues and dataset bias, while TF-IDF's Ukraine-operation performance is not guaranteed on other propaganda topics.Source: §6.6 · pp. 8–9
- Propaganda articles more often use terms such as “état,” “zelensky,” and “kiev,” while regular articles more often use “poutine,” “kyiv,” and “guerre.”The paper also notes differing spellings of Kiev/Kyiv and more precise time markers in the regular corpus, consistent with its higher VAGO detail score.Source: §6.6 · p. 9
7 Conclusion and perspectives
The conclusion reports stylistic cues associated with propagandist articles and emphasizes that detection and explainability remain bounded by the evaluated settings.
7 Conclusion and perspectives
- Exaggeration, reduced descriptive content, and inadequate sourcing were prevalent in assessments of propagandist press, with vague markers significantly correlated with those features.Source: §7 · p. 9
- Classifier analyses also revealed syntactic cues, particularly punctuation, as well as lexical cues.Source: §7 · p. 9
- Machine-learning models detect topic-specific propaganda efficiently, but explainability and generalization to other topics remain limited.Technique classifiers offer more explainability at a performance cost, especially for topic-specific propaganda.Source: §7 · p. 9
- The reported high scores concern topic-specific detection, an easier task than general propaganda detection.The authors say topic-specific models may still help prevent disinformation in conflicts similar to the one studied.Source: §7 · p. 9
- The provided detection model covers English and French propaganda about the Ukraine invasion; strategies may vary across languages and target audiences.The authors encourage using native-language dataset portions and considering multilingual models and further reader-annotation experiments.Source: §7 · p. 9
- Symbolic AI explains part of human and classifier judgments, while proposed improvements include richer lexical and syntactic cues and more finely specified labels.The authors suggest adding a Laudatory label and refining Exaggeration, alongside alternatives such as Precise/Objective rather than Vague/Subjective.Source: §7 · p. 9
Limitations
The annotation experiment covered only a French subset, while detection experiments covered two Romance and Germanic language examples.
Limitations
- Annotation experiments were run only on a subset of the French data.Source: Limitations · p. 10
- English articles received additional manual quality checks, but other languages were not manually reviewed and may contain parsing errors.The authors say native-speaker analysis may be needed before using those dataset portions.Source: Limitations · p. 10
- Detection experiments covered only two Romance and Germanic language examples, and performant language models are not guaranteed for every proposed dataset language.Source: Limitations · p. 10
Ethics statement
The statement frames propaganda detection as a way to protect the information space and describes safeguards for annotators and cautious future dataset use.
Ethics statement
- The paper presents propaganda detection methods as important for keeping the information space clean and safe.Source: Ethics statement · p. 10
- Annotators participated voluntarily, received a content warning, and could stop the experiment at any time.Source: Ethics statement · p. 10
- Future work using the dataset is encouraged to proceed cautiously and on limited portions of the full collection.Source: Ethics statement · p. 10