Source-linked AI summary
Fine-Grained Analysis of Propaganda in News Articles
Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, Preslav Nakov
TL;DR
Existing document- and outlet-level propaganda detection uses noisy labels and often lacks explainability. The paper introduces fragment-level detection of eighteen techniques, a corpus and partial-overlap evaluation measure, and a multi-granularity neural network that outperforms BERT-based baselines.
Problem
Document- or outlet-level propaganda labels can be noisy, and existing systems lack explainability.
Method
The paper creates a professionally annotated corpus with eighteen propaganda techniques, defines sentence- and fragment-level tasks, and proposes a partial-overlap evaluation measure and multi-granularity network.
Results
The multi-granularity neural network outperforms standard BERT-based baselines, while the corpus contains 350K tokens.
Takeaways & Limitations
Fine-grained analysis can complement document-level judgments by supporting aggregated decisions and explanations of why text or outlets are flagged.
Abstract
from arXiv · showhide
Propaganda aims at influencing people's mindset with the purpose of advancing a specific agenda. Previous work has addressed propaganda detection at the document level, typically labelling all articles from a propagandistic news outlet as propaganda. Such noisy gold labels inevitably affect the quality of any learning system trained on them. A further issue with most existing systems is the lack of explainability. To overcome these limitations, we propose a novel task: performing fine-grained analysis of texts by detecting all fragments that contain propaganda techniques as well as their type. In particular, we create a corpus of news articles manually annotated at the fragment level with eighteen propaganda techniques and we propose a suitable evaluation measure. We further design a novel multi-granularity neural network, and we show that it outperforms several strong BERT-based baselines.
1 Introduction
The paper argues that document- or outlet-level propaganda labels are noisy and insufficiently explainable, motivating expert fragment-level annotation of specific techniques. It introduces a fine-grained task, corpus, evaluation measure, and multi-granularity model that outperforms BERT-based baselines.
- Outlet-level labels can be noisy because propagandistic outlets may publish objective articles, while objective outlets may publish propagandistic ones.
- Crowdsourced propaganda judgments may reflect annotators’ personal beliefs, motivating trusted professional annotation.
- Six experts annotated article spans from propagandistic and non-propagandistic outlets with eighteen propaganda techniques.
- The released corpus contains 350K tokens and the authors also release their code.
- The work formulates detection of specific propaganda techniques, builds a large corpus, and proposes a suitable evaluation measure.
- The novel multi-granularity neural network outperforms several strong BERT-based baselines.
2 Propaganda and its Techniques
The paper treats propaganda as persuasive communication conveyed through identifiable techniques and presents a curated schema of eighteen such techniques. These include emotional appeals, reasoning patterns, and devices that redirect or constrain interpretation.
- Propaganda techniques enable analysis beyond a single document-level judgment by examining propaganda at paragraph and sentence levels.
- The authors curate eighteen techniques because scholars differ in which techniques they include and how they define them.
- Loaded language uses emotionally charged words or phrases to influence an audience.
- Name calling or labeling assigns targets descriptions intended to evoke fear, hatred, undesirability, love, or praise.
- Other techniques include repetition, exaggeration or minimization, doubt, appeal to fear or prejudice, and flag-waving.
- Causal oversimplification attributes an issue to one cause or transfers blame to a person or group without examining complexity.
- The schema also covers slogans, appeal to authority, black-and-white fallacy, thought-terminating cliché, whataboutism, and reductio ad Hitlerum.
- Red herring diverts attention from the issue under discussion through irrelevant material, while bandwagon appeals to collective action.
17. Obfuscation, intentional vagueness, confu-
This section defines obfuscation as deliberate use of unclear language that permits multiple interpretations and weakens support for a conclusion. It also situates the annotation process and retrieved article statistics within the corpus construction.
- 17. Obfuscation, intentional vagueness, confu-: Obfuscation or intentional vagueness uses deliberately unclear words so audiences may form their own interpretations.
- 17. Obfuscation, intentional vagueness, confu-: A related example presents an apparently reasonable premise before drawing a conclusion that the premise does not adequately support.
- 17. Obfuscation, intentional vagueness, confu-: The corpus includes statistics for articles retrieved from propagandistic, non-propagandistic, and combined media-source categories.
- 17. Obfuscation, intentional vagueness, confu-: Table 3 reports the number of retrieved articles from outlets deemed propagandistic by Media Bias/Fact Check.
- 17. Obfuscation, intentional vagueness, confu-: The authors supplied professional annotators with definitions, examples, and an annotation schema for manually annotating news articles.
3 Data Creation
The corpus comprises professionally annotated news articles from propagandistic and non-propagandistic outlets, using staged annotation and consolidation to identify technique-specific fragments. The resulting dataset contains thousands of annotated instances, with loaded language and name calling most frequent.
- Corpus: 451 articles from 48 news outlets were retrieved, with 82.5% coming from propagandistic sources.The propagandistic-source articles also tend to be longer.
- Annotation: Six professional annotators marked text spans with eighteen propaganda techniques, supported by definitions, examples, an annotation schema, and a hierarchical flowchart.The annotation task was conducted through a professional annotation company because understanding all techniques was not well-suited to crowdsourcing.
- Agreement: After consolidation, γ agreement increased to as high as 0.74 for spans and 0.76 for spans plus labels, from first-stage values of 0.30 and 0.34.The reported agreements compare annotators before consolidation with annotator–gold agreement afterward.
- Agreement: 53% of instances spotted by only one first-stage annotator were retained as gold annotations, and disagreements generally reflected missed instances rather than different labels.The retained-instance percentages ranged from 31% to 70% across annotators.
- Dataset statistics: 7,485 technique instances occur across 21,230 sentences, with loaded language and name calling accounting for 2,547 and 1,294 occurrences.The average propagandistic fragment is 47 characters; repetitions were inflated because original and repeated instances were both marked.
4 Evaluation Measures
The evaluation task must handle overlapping propaganda spans and highly variable fragment lengths, so the paper introduces character-level partial-credit measures for labeled matches. Its precision, recall, and F1 variants also penalize predicting too many or too few instances.
- Task requirements: The sequence-labeling task involves many techniques with potentially overlapping spans and fragments ranging from single tokens to full sentences or longer.These properties make exact-boundary and exact-label evaluation unsuitable as the sole criterion.
- Fragment comparison: The paper defines a comparison function C that gives credit for character-level partial overlap when predicted and gold fragments have the same label.The label indicator δ equals 1 for matching labels and 0 otherwise, while h normalizes the overlap score.
- Precision and recall: Precision and recall average the fragment-comparison scores over predicted and gold fragment sets, respectively, using predicted or gold fragment lengths as normalization factors.The measures are defined to be zero when the corresponding predicted or gold set is empty.
- Precision and recall: The measures penalize systems that predict too many or too few instances by dividing by the sizes of the predicted and gold sets.For example, adding an imperfect extra prediction can lower precision compared with retaining a better-matching prediction alone.
- F1 and flexibility: The resulting F1 measure combines the proposed precision and recall through their harmonic mean.The separate comparison function also allows the partial-overlap factor to be changed for different tolerance levels.
5 Tasks and Proposed Models
The paper defines sentence- and fragment-level propaganda detection tasks and proposes a multi-granularity network that uses lower-granularity information to guide higher-granularity predictions.
- Tasks: SLC predicts whether a sentence contains propaganda, while FLC identifies propaganda spans and their techniques.The tasks operate at sentence and token granularities, respectively.
- Baselines: The authors establish BERT, BERT-Joint, and BERT-Granularity as baselines for the two tasks.BERT uses token representations for FLC and the [CLS] representation for SLC; the joint variants share or transfer task information.
- Multi-Granularity Network: The proposed network drives higher-granularity FLC using lower-granularity SLC information rather than directly transferring it.The architecture is presented as a general framework for tasks arranged from coarser to finer granularity.
- Multi-Granularity Network: Each granularity has a classification layer whose output is transformed by a trainable gate into a weight for the next task.The gate applies a projection and activation, then multiplies the next layer’s output.
- Gating Behavior: When the gate weight is zero, the next task’s output is zero and the example contributes no final entropy-loss information.In this setting, a sentence classified as non-propagandistic suppresses token-level technique predictions.
- Implementation: The multi-granularity network is placed on top of BERT’s contextualized embedding layer.
6 Experiments and Evaluation
Experiments evaluate the models on fragment- and sentence-level tasks. The multi-granularity approach outperforms the baselines on FLC, while token-level information substantially improves SLC.
- Evaluation: Fragment-level evaluation separately measures span identification and the full task of identifying spans with their propaganda techniques.
- Fragment-Level Classification: Joint learning hurts FLC relative to single-task BERT, whereas BERT-Granularity provides small improvements.
- Fragment-Level Classification: The multi-granularity models outperform all baselines on FLC through higher precision.The authors attribute this to excluding sentences classified as non-propagandistic from token-level classification.
- Sentence-Level Classification: 60.98 F1 is the highest reported sentence-level classification result, indicating that sentence-level information remains noisy.
- Sentence-Level Classification: 8.42% higher recall and a 3.24% higher F1 score are achieved on SLC compared with the BERT baseline when token-level information is added.
- Sentence-Level Classification: ReLU is more effective than Sigmoid for SLC because the token-level classification performance is low and aggressive negative-sample removal is preferred.
7 Related Work
Prior propaganda research largely operates at the article level, while related argumentation work annotates fallacies in arguments. This paper instead annotates minimal propaganda fragments across news articles with eighteen techniques.
- Article-Level Propaganda Detection: Earlier propaganda corpora primarily label entire news articles or assign labels through their sources.
- Computational Argumentation: Computational argumentation studies related fallacies such as ad hominem, red herring, and irrelevant authority.
- Comparison: Unlike related argumentation corpora, this corpus annotates 18 techniques on the same news articles and marks minimal technique-related fragments.
8 Conclusion and Future Work
The paper concludes that fine-grained technique detection can support more reliable and explainable propaganda analysis. It provides an annotated corpus, evaluation measure, and neural architecture, with plans to expand training-oriented use.
- Conclusion: Fine-grained identification of specific propaganda techniques can yield more reliable systems and explain their article-level judgments.
- Resources: The authors provide an 18-technique annotation schema, a sizable annotated dataset, and a task-specific evaluation measure.
- Results: The novel architecture outperforms standard BERT-based baselines, while fine-grained predictions can complement document-level judgments.
- Future Work: The planned online platform would train students to recognize and annotate propaganda techniques while accumulating annotations.
A Annotation Guidelines
Professional annotators manually identified propaganda techniques in news articles using definitions, examples, an annotation schema, and a hierarchical flowchart. Guidelines emphasized unbiased judgment, minimal text spans, and the possibility of multiple techniques in one fragment.
- Annotation process: Annotators received technique definitions, examples, an annotation schema, and instructions for manually labeling news articles.The instructions and flowchart were designed to guide the annotation process.
- Annotation process: A hierarchical flowchart organized propaganda techniques and guided identification through a sequence of questions.Annotators could access technique explanations and examples through the flowchart.
- Span selection: Annotators were encouraged to select the minimal span in which a technique appeared, ranging from single words to sentences.The guidelines allowed flexible span sizes while prioritizing minimal text selection.
- Examples and quality guidance: Examples illustrated name calling, black-and-white fallacy, loaded language, and exaggeration in annotated text.The examples included labels such as “babies,” “obstruction vs progress,” emotionally charged words, and exaggerated descriptions.
- Examples and quality guidance: Annotators were instructed to distance themselves from article content, avoid bias, and resolve difficult sentences by selecting the appropriate technique or techniques.The guidance focused on identifying techniques rather than evaluating the message.
- Labeling rules: A single text fragment could receive multiple technique labels when more than one technique occurred simultaneously.The instructions explicitly allowed overlapping technique annotations.