Source-linked AI summary
Unmasking Propaganda: A Comparative Analysis of Masked and Causal Language Models
Claudiu Creanga, Ioachim Lihor, Liviu P. Dinu
TL;DR
Propaganda detection is challenging because techniques can be subtle and context-dependent, yet identifying them matters in a changing digital news environment. This paper compares masked and causal language models with base and chain-of-thought prompting on SemEval-2020. Gemini Flash 2.5 and DeBERTa V3 Large outperform the reported earlier benchmark scores, with F1 scores of 63.62 and 63.18, respectively.
Problem
Propaganda techniques can be subtle and context-dependent, making them difficult to distinguish from legitimate persuasion, while identifying propaganda is important to informed public discourse in a changing digital information ecosystem.
Source: §1 · pp. 1–2Method
The study compares masked and causal language models for propaganda-technique detection on SemEval-2020, using base and chain-of-thought prompts for causal models.
Source: §1 · p. 2, §4.2 · p. 3Results
Gemini Flash 2.5 achieved 63.62 versus the previous SOTA of 58, and DeBERTa V3 Large achieved F1 63.18 on SemEval-2020 Task 11 Technique Classification.
Source: §5 · p. 3Takeaways & Limitations
The authors suggest that fine-tuning on specific datasets and diverse prompting strategies may further improve detection accuracy.
Source: §5.2 · p. 5Pretraining on internet-scraped data may expose large language models to SemEval test data, potentially inflating performance and obscuring generalization.
Source: §5.3 · p. 5
Abstract
from arXiv · showhide
Propaganda detection is an essential task in natural language processing (NLP), particularly in the context of manipulative political communications. However, identifying specific propaganda techniques presents a significant challenge due to their often subtle nature and reliance on context, making them difficult to distinguish from legitimate persuasive language. Propaganda often involves highlighting certain facts while downplaying or ignoring others to create a desired perception. This biased communication aims to influence attitudes, beliefs, or behaviors towards a particular cause or position. This paper explores advances in detecting propaganda techniques through a comparative analysis of modern language models, using the SemEval-2020 Task 11 dataset. We evaluated both masked language models (based on XLM-RoBERTa or DeBERTa V3) and causal models (from OpenAI, Google, Mistral, Anthropic and Meta), employing two prompting strategies: base and chain-of-thought prompting. Our results demonstrate improvements over state-of-the-art models, with the best-performing MLM achieving an F1 score of 63.18 in technique classification and the best causal model achieving 63.62. We also observed that certain models excel in specific techniques, such as loaded language and name-calling, while struggling with others like bandwagon and black-and-white fallacy. These findings suggest that fine-tuning, ensemble modeling, and the use of larger datasets can further enhance propaganda detection capabilities.
1 Introduction
Propaganda selectively presents and distorts information to shape public attitudes, while its subtle, context-dependent techniques make detection challenging. The paper applies masked and causal language models to propaganda-technique detection using SemEval 2020.
1 Introduction
- Propaganda selectively highlights supportive information while downplaying, omitting, distorting, or fabricating contrary information to influence beliefs and behavior.Its techniques range from loaded language and name-calling to oversimplification, emotional appeals, and logical fallacies.Source: §1 · p. 1
- Propaganda is described as contributing to polarization, reduced critical thinking, and the dehumanization of designated out-groups.The paper links these effects to “us vs. them” tactics and passive acceptance of official narratives.Source: §1 · p. 1
- In a digital information ecosystem shaped by shifting news consumption and a crisis in news trust, identifying propaganda is presented as fundamental to informed public discourse and democratic health.The paper notes that more people, particularly young people, get news online, while many others avoid news altogether.Source: §1 · pp. 1–2
- The study compares masked and causal models on the SemEval 2020 dataset for detecting propaganda techniques, and makes its code publicly available.The introduction describes propaganda as a means of manipulating public opinion and shaping collective perception.Source: §1 · pp. 1–2
2 Related Work
Prior work developed propaganda datasets and evaluated both conventional models and prompted language models, while the paper reports new state-of-the-art results for SemEval technique classification.
2 Related Work
- SemEval-2020 Task 11 defined Span Identification and Technique Classification, and its best reported result was F1 62 with a pre-trained RoBERTa model.Earlier datasets included TSHP-17, Qprop, and PTC, with PTC annotating spans using 18 propaganda techniques.Source: §2 · p. 2
- A prior Qprop study reported F1 97.76 using BERT, ELMo, LSA, FastText, and Word2Vec.That result concerns Qprop, rather than SemEval technique classification.Source: §2 · p. 2
- Earlier SemEval LLM experiments combined GPT models with few-shot and chain-of-thought prompting; GPT-4 with few-shot learning achieved F1 58.11.Source: §2 · p. 2
- Loaded language is the most frequent technique in the corpus label distribution shown across train and test splits.Source: Figure 1 · p. 2
2.1 Contribution
- Gemini Flash 2.5 achieved a macro F1-score of 63.62 using base prompting, while fine-tuned DeBERTa V3 Large achieved 63.18 on SemEval-2020 Task 11 Technique Classification.The authors report that both results surpass the original SemEval winning score.Source: §2.1 · p. 2
3 Data
The study uses the SemEval-2020 Task 11 corpus, a professionally annotated collection of news articles with standard train, development, and test splits and a skewed technique distribution.
3 Data
- The SemEval-2020 corpus contains 536 articles and 8,981 annotated propaganda instances, split into 371 training, 75 development, and 90 test articles.Articles averaged 5,348 characters and 875 tokens; six trained professional annotators used independent annotation followed by expert-reviewed consolidation, with average inter-annotator agreement γ 0.6.Source: §3 · p. 2
- Loaded Language and Name Calling/Labeling are substantially more frequent than other techniques in the corpus.The passage points to Figure 1 for the distribution across techniques and data splits.Source: §3 · p. 2
4 Methodology
The methodology compares fine-tuned masked language models with API-accessed causal language models on article-level, multi-label classification of SemEval propaganda techniques.
4 Methodology
- Rather than classifying pre-identified spans, the adapted task gives models whole articles and asks them to identify all present labels from the SemEval set of 14 techniques.The original Technique Classification task assigned one or more technique labels to spans first identified by Span Identification.Source: §4 · pp. 2–3
4.1 Masked Language Models
- The masked-model approach fine-tunes XLM-RoBERTa and DeBERTa V3 Large in two phases, first training a classifier with the base model frozen and then unfreezing its final layer for joint fine-tuning.The first phase runs for five epochs; the second runs for one additional epoch at a reduced learning rate.Source: §4.1 · p. 3
4.2 Causal Models
- The causal-model comparison accesses models from Google, OpenAI, Anthropic, Mistral AI, and Meta through APIs.The tested models include Gemini, GPT, Claude, Mistral, and Llama variants.Source: §4.2 · p. 3
- Causal models receive either a base prompt requesting technique labels or a chain-of-thought prompt requesting labels with explanations.Both prompts ask for the techniques present in the article, or a “no propaganda detected” response if none are identified.Source: §4.2 · p. 3
5 Results
The results report new Technique Classification benchmarks for masked and causal models, while error analysis shows performance varies across techniques and prompting strategies. The discussion identifies scope boundaries and potential data leakage.
5 Results
- Gemini Flash 2.5 reached 63.62 macro F1, while fine-tuned DeBERTa V3 Large reached 63.18 F1; the causal result beat the previous SOTA score of 58.The base-prompt model-performance table supports the comparison context.Source: §5 · p. 3, Table 1 · p. 4
- Chain-of-thought prompting did not consistently produce the highest F1, although it sometimes boosted recall; Gemini Flash 2.5 ranked highest with both base and chain-of-thought prompting.Gemini Flash 2.5’s base-prompt recall was 77.93 with precision 53.75, while the fine-tuned GPT-3.5 variant had precision 71.68 and recall 45.25.Source: §5 · p. 3, Table 2 · p. 4
5.1 Error Analysis and Discussion
- Models generally performed well on loaded language and name-calling but struggled with clichés, bandwagon, and black-and-white fallacy, with substantial variation across models.Claude-Sonnet achieved 0.89 accuracy on loaded language, compared with GPT-3.5’s 0.49.Source: §5.1 · p. 4, Table 3 · p. 5
- For GPT-4o with base prompting, class 1 had a 73% true-positive rate versus class 0’s 62% true-negative rate, and false positives (38%) exceeded false negatives (27%).The reported average F1 across techniques was 65.61.Source: §5.1 · p. 4, Figure 2 · p. 4
- The chain-of-thought confusion-matrix analysis reported average F1 of 67.34, a slight improvement over base prompting, with 67% true negative and 68% true positive rates.Source: Figure 3 · p. 4, §5.1 · p. 5
5.2 Conclusions and Future Work
- The authors report that dataset-specific fine-tuning and diverse prompting can improve detection accuracy, and that two-step fine-tuning with frozen weights improves masked-model performance.Source: §5.2 · p. 5
- Future work proposes ensembles of complementary models and hybrid systems combining fine-tuned masked models with causal models.Source: §5.2 · p. 5
5.3 Limitations
- The study covers Technique Classification only and English, while LLM test-data exposure could inflate performance and obscure generalization; the authors report no publication-related ethical issues.Future work is proposed for Span Identification and other languages.Source: §5.2 · p. 5, §5.3 · p. 5, §5.4 · p. 5