Fine-Grained Analysis of Propaganda in News Articles

1 Introduction

• Outlet-level labels can be noisy because individual articles may differ from the propagandistic or objective label assigned to their source.Propagandistic outlets may publish objective articles, while generally objective media may occasionally promote an editorial agenda.Source: §1 · p. 1
• The authors report that crowdsourced judgments may reflect annotators’ beliefs, motivating trusted professional annotation at the fragment level.They argue for targeting specific techniques rather than labeling whole documents or outlets.Source: §1 · p. 1
• Six experts annotated 350K tokens from propagandistic and non-propagandistic news outlets, marking text spans with eighteen propaganda techniques.The corpus is presented as the first manually curated evaluation framework for fine-grained propaganda analysis.Source: §1 · p. 1
• The paper formulates a task of detecting specific propaganda techniques in text and proposes a suitable evaluation measure.The evaluation measure is designed for comparing labeled fragments.Source: §1 · p. 1
• The proposed multi-granularity neural network outperforms several strong BERT-based baselines.The paper presents this as one of its contributions.Source: §1 · p. 1
• Fine-grained detection could make explicit why an article is judged propagandistic and help users learn to recognize propaganda techniques.The authors describe these as potential uses of the corpus.Source: §1 · p. 2

2 Propaganda and its Techniques

• The authors select eighteen techniques that can be judged from journalistic articles alone, excluding card stacking because it requires external comparison.They note that scholars’ technique sets vary in size and scope.Source: §2 · p. 2
• Some techniques persuade through emotionally charged wording, labels, repetition, or distorted scale.Loaded language uses words with strong emotional implications; labeling assigns a target something feared, disliked, or praised; repetition seeks acceptance through repeated messages. Exaggeration or minimization makes something seem larger or smaller or less important than it is.Source: §2 · p. 2
• Other techniques challenge credibility, invoke group feeling, simplify causation, or constrain how an issue is framed.Doubt questions credibility; appeal to fear or prejudice builds support by instilling anxiety; flag-waving invokes strong group feeling; causal oversimplification assigns a complex issue a single cause or scapegoat. Slogans use brief, striking phrases; appeals to authority rely on an authority’s support without other evidence; black-and-white fallacy presents two options as exhaustive, with dictatorship prescribing a specific action; thought-terminating clichés discourage critical thought.Source: §2 · pp. 2–3
• Further techniques discredit opponents, divert attention, or encourage conformity.Whataboutism charges an opponent with hypocrisy without directly disproving their argument; reductio ad Hitlerum associates an idea with a hated person or concept. A red herring introduces irrelevant material to divert attention, while bandwagon urges action because others are taking it.Source: §2 · pp. 3–4

17. Obfuscation, intentional vagueness, confu-

• Obfuscation uses deliberately unclear words to invite audience interpretation; an unclear phrase may have multiple meanings and fail to support the argument’s conclusion.The example argues that one should shoot a thief if victims say to do so, following the claim that victims should be heard.Source: §17 · p. 4
• Straw man substitutes an opponent’s proposition with a similar, caricatured one that is easier to refute.The definition is attributed to Walton (1996), with the characterization from Weston (2018, p. 78).Source: §17 · p. 4
• Professional annotators received the definitions, examples, and an annotation schema to manually annotate news articles.Further details are provided in the next section.Source: §17 · p. 4

3.1 Article Retrieval

• Of the retrieved articles, 82.5% came from propagandistic sources, and these articles tended to be longer; overall, the corpus contains 451 news articles from 48 propagandistic and non-propagandistic outlets.The corpus has 350k word tokens.Source: §3.1 · p. 4, §3 · p. 4

3.2 Manual Annotation

• After four annotators’ preliminary agreement was low (γs = 0.34; γsl = 0.31), the team adopted two-stage annotation with independent annotators and consolidators, taking 395 man hours.Two annotator teams each covered about 220 documents; each team discussed cases with a consolidator to produce final annotations.Source: §3.2 · p. 5
• After consolidation, γ agreement reached 0.74 and 0.76, up from 0.30 and 0.34 for span selection before consolidation.Agreement was slightly lower when technique labels were also considered. Table 4 distinguishes first-stage annotator agreement from consolidation-stage agreement with the final gold annotation.Source: Table 4 · p. 5, §3.2 · p. 5
• 53% of instances spotted by only one first-stage annotator were retained as gold, indicating disagreements mainly reflected missed instances rather than different labels on overlapping spans.This was 5,921 of 11,122 instances; per-annotator retention ranged from 31% to 70%.Source: §3.2 · p. 5

3.3 Statistics about the Dataset

• 7,485 technique instances occur across 21,230 sentences, with 35.2% of sentences containing an instance; average fragment and sentence lengths are 47 and 112.5 characters.Source: §3.3 · p. 5
• 2,547 loaded-language and 1,294 name-calling/labeling instances were most common, appearing 6.7 and 4.7 times per article; no other technique appeared more than twice.Counts are inflated because annotators marked both original and repeated instances.Source: §3.3 · p. 5

Evaluation Measures

• The measure gives character-level partial credit for imperfect matches, accommodating propaganda spans that range from single tokens to full sentences or longer.The authors note that strict NER evaluation requiring both exact boundaries and labels is unsuitable for this task.Source: §4 · p. 6
• Text is represented as character-sequence fragments, which may overlap, and each predicted fragment is assigned one of the eighteen technique labels.Gold and predicted annotations are modeled as fragment sets over a document.Source: §4 · p. 6
• The comparison function separates fragment matching from class comparison, allowing class-distance weighting to be customized beyond exact label agreement.The authors suggest that confusing some techniques could be treated as less problematic than confusing others.Source: §4 · p. 6
• Precision and recall use the overlap comparison while dividing by the number of predicted and gold fragments, respectively, penalizing over- and under-prediction; their harmonic mean gives F1.The measures are defined as zero when the corresponding prediction or gold set is empty.Source: §4 · p. 6

5 Tasks and Proposed Models

• SLC predicts whether each sentence contains any propaganda, while FLC identifies propaganda spans and their technique types.The corpus is split into training, development, and test sets containing 293, 57, and 101 articles, respectively.Source: §5 · p. 7

5.1 Baselines

• BERT baselines include independent token and sentence classifiers, joint SLC–FLC training, and a granularity variant that transfers sentence-level features to FLC.The granularity variant concatenates the SLC and FLC representations and adds a 19-way classification layer for FLC.Source: §5.1 · p. 7

5.2 Multi-Granularity Network

• The proposed multi-granularity network uses sentence-level information to guide fragment-level predictions rather than feeding lower-granularity information directly.The network is placed on top of BERT contextualized embeddings.Source: §5.2 · pp. 7–8
• A trainable gate maps lower-granularity task output to a weight that scales each element of the next-granularity task’s output.Each task has a separate classification layer that receives its granularity-specific representation and produces the output used to generate the next task’s weight.Source: §5.2 · p. 7
• When the sentence classifier assigns zero weight, the next task’s output and final entropy loss are zero, so strongly negative examples are ignored at fragment level.Higher-granularity information can contribute selectively when an example is not considered highly negative.Source: §5.2 · p. 7
• The network uses sigmoid cross-entropy for SLC and softmax cross-entropy for FLC, combining task losses with a weighted sum controlled by α.Sigmoid outputs are independent across classes, unlike softmax outputs, which are normalized across dimensions.Source: §5.2 · p. 8

6.1 Experimental Setup

• Models used batch size 16, sequence length 210, Adam at 3e-5, and early stopping on validation F1; reported results average three random-seed runs.Models were tuned on the dev partition and evaluated on the test partition.Source: §6.1 · p. 8

6.2 Fragment-Level Propaganda Detection

• Multi-granularity models outperform all FLC baselines through higher precision, with Sigmoid gating outperforming ReLU; BERT-Joint underperforms single-task BERT.The evaluation includes both span identification alone and the full task of identifying spans and assigning propaganda techniques.Source: §6.2 · p. 8, Table 6 · p. 8

6.3 Sentence-Level Propaganda Detection

• The multi-granularity model raises recall by 8.42% and F1 by 3.24% versus BERT; token predictions help identify positive sentences, and ReLU beats Sigmoid.This is the opposite of FLC, where Sigmoid performs better than ReLU.Source: §6.3 · pp. 8–9, Table 7 · p. 9

• Earlier propaganda identification primarily labeled whole articles, including corpora with four categories and a binarized propaganda-versus-other setup (Rashkin et al., 2017; Barrón-Cedeño et al., 2019).Source: §7 · p. 9
• Outlet-level distant supervision assumes articles from each outlet share a label, inevitably introducing noise (Horne et al., 2018).Source: §7 · p. 9
• Computational argumentation work includes an ad hominem forum corpus and Argotario, which produced 1.3k arguments annotated for five fallacies, including ad hominem, red herring, and irrelevant authority (Habernal et al., 2018b; Habernal et al., 2017, 2018a).Source: §7 · p. 9
• Unlike prior argumentation work, this corpus annotates 18 techniques in the same news articles and identifies minimal technique-related fragments rather than flagging entire arguments.Source: §7 · p. 9

8 Conclusion and Future Work

• Fine-grained identification of propaganda-technique instances can yield more reliable systems and explain why an article was judged propagandistic.The approach focuses on specific technique instances rather than document-level labeling alone.Source: §8 · p. 9
• The authors created an annotation schema covering 18 techniques, annotated a sizable document dataset, designed a task-specific evaluation measure, and made the schema and dataset public.They suggest the corpus may also interest researchers in Argumentation and Sentiment Analysis, particularly for fallacies and emotion-based techniques.Source: §8 · p. 9
• The novel architecture outperforms standard BERT-based baselines, and fine-grained analysis can complement document-level judgments by supporting aggregation and explanations.Such explanations can clarify why an article or news outlet was flagged as potentially propagandistic.Source: §8 · p. 9
• The authors are collaborating with A Data Pro to expand the corpus and plan an online platform where relevant-field professors can train students to recognize and annotate propaganda techniques.They hope training use of the platform will accumulate annotations as a by-product.Source: §8 · p. 9

A Annotation Guidelines

• Annotators received definitions, examples, and an annotation schema to manually label news articles.They followed instructions provided for the annotation process.Source: A Annotation Guidelines · p. 11
• A hierarchical flowchart guided technique identification through a series of questions, beginning with the distinction between logical fallacies and emotional appeals.The flowchart and instructions were displayed for annotators in Figure A.1.Source: A Annotation Guidelines · p. 11, Figure A.1 · p. 12
• The instructions and flowchart remained available to annotators beside the annotation interface.Source: A Annotation Guidelines · p. 11