Detecting Propaganda Techniques in Memes

1 Introduction

• Social media serve as a major channel for information, and propaganda can use them to influence opinions and actions toward a predetermined goal.The paper situates its study amid the growth of computational propaganda on social platforms.Source: §1 · p. 1
• The annotation identifies techniques in memes rather than judging whether a meme is propagandistic, which would require determining intent.Source: §1 · p. 2
• The paper formulates a multimodal task and releases 950 manually annotated memes covering 22 propaganda techniques that may occur in text, images, or both.The corpus and experiment code are made available.Source: §1 · p. 2
• Both modalities are important for detecting propaganda techniques in memes, a finding supported by manual analysis and experiments with state-of-the-art models.The experiments also point to a need for further research.Source: §1 · p. 3

2 Related Work

• Earlier propaganda work examined network structure, malicious accounts, and coordinated inauthentic behavior, while computational propaganda can disseminate misleading information through social media.Source: §2 · p. 3
• Text-based studies span document, sentence, and fragment levels, including technique inventories and tasks for classifying or locating propaganda in text.Prior datasets include document-level and binary labels as well as fragment-level annotation of 18 techniques.Source: §2 · p. 3
• Multimodal research has addressed misleading information, deception, hateful memes, antisemitism, and propaganda in images, among other targets.Hateful-meme systems have used multimodal transformers to classify memes as hateful or not hateful.Source: §2 · pp. 3–4
• This work differs through a 22-technique inventory annotated separately in text and then jointly in text and image, enabling analysis and explainability capabilities.Source: §2 · p. 4

3 Propaganda Techniques

• The authors adapt prior inventories into 22 techniques; the first 20 apply to both text and images, while Appeal to (Strong) Emotions and Transfer are used for image labeling only.Source: §3 · p. 4
• Loaded language, name calling, doubt, exaggeration, fear appeals, slogans, and whataboutism describe word choices or arguments used to influence or discredit an audience.Source: §3 · p. 4
• Flag-waving invokes group feeling; straw man substitutes an opponent’s claim; causal oversimplification assigns a single cause; and appeal to authority relies on an authority’s claim.Source: §3 · p. 4
• Other defined tactics include thought-terminating clichés, black-and-white framing, Red Herring, Bandwagon, Smears, and Glittering Generalities.Reductio ad hitlerum discredits an idea by associating it with a group or concept carrying negative connotations.Source: §3 · pp. 4–5

4 Dataset

• Memes were collected from Facebook groups spanning varied topics and political perspectives over three months, alongside older examples to broaden topic coverage.Collection used private accounts and repeated checks of group feeds.Source: §4 · p. 5

4.1 Annotation Process

• Six English-fluent annotators used a multilabel schema for 22 techniques, with pilot training, independent annotation, and team consolidation.The schema allowed multiple techniques per meme because meme content often expresses several techniques; annotators also marked spans.Source: §4.1 · p. 5

4.1.1 Phase 1: Filtering and Text Editing

• Phase 1 filtered low-quality or out-of-scope memes and manually corrected OCR-extracted text before annotation.The original meme and extracted text were presented to an annotators for filtering and editing.Source: §4.1.1 · p. 6

4.1.2 Phase 2: Text Annotation

• Phase 2 presented edited meme text so annotators could identify propaganda techniques and mark their text spans.Source: §4.1.2 · p. 6

4.1.3 Phase 3: Text Consolidation

• Phase 3 consolidated Phase 2 text labels, supporting quality checks and further team training.Source: §4.1.3 · p. 6

4.1.4 Phase 4: Multimodal Annotation

• Phase 4 showed annotators the meme, edited text, and consolidated text labels so they could add techniques requiring visual content.Source: §4.1.4 · p. 6

4.1.5 Phase 5: Multimodal Consolidation

• During Phase 4 consolidation, annotations were discussed; when actual disagreements remained unresolved, the meme was discarded, with five such cases in total.Annotating propaganda techniques from images alone was impractical because text is embedded in the image pixels.Source: §4.1.5 · p. 6

4.2 Quality of the Annotations

• Inter-annotator agreement ranged from moderate to perfect when individual text and image annotations were compared with consolidated labels.The assessment combined phases 2 and 4 annotations and used multilabel Krippendorff’s α.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 Statistics

• The corpus contains 950 memes, averaging 1.68 sentences and 17.79±11.60 words; Loaded Language and Name Calling skew textual, while Transfer skews image-related.Source: §4.3 · p. 7
• Smears, Loaded Language, and Name calling/Labeling appear in 63%, 51%, and 36% of examples, respectively; most memes contain multiple techniques (mean 2.61; maximum 8).These three techniques also form the most common pairs and triples; Figure 3 shows the distribution of technique counts per meme.Source: §4.3 · p. 7, Figure 3 · p. 7, Table 3 · p. 8
• Techniques appear in both textual and visual content, with span lengths varying from short examples to Whataboutism’s 22-word average.Loaded Language averages about two words per span.Source: §4.3 · p. 7, Table 2 · p. 7

5 Experiments

• The task is to predict all propaganda techniques in a meme’s text and image, following the multimodal Phase 5 annotations.Source: §5 · p. 8

5.1 Models

• Random and majority-class baselines were included, with the majority baseline always predicting Smears.Source: §5.1 · p. 8
• Text-only models were BERT and fastText, image-only modeling used ResNet152, and multimodal approaches combined pretrained unimodal models through early, middle, or late fusion.The experiments also included jointly trained multimodal models ViLBERT and Visual BERT.Source: §5.1 · p. 8

5.2 Experimental Settings

• The data split was 687 training, 63 development, and 200 test examples; micro-average F1 was primary, with macro-average F1 also reported.Results were averaged over three random-seed runs.Source: §5.2 · p. 8

6 Experimental Results

• The visual-only ResNet-152 model performs worse than text-only models, and BERT significantly outperforms fastText.The authors suggest identifying techniques in visual content may be harder than identifying them in text.Source: §6 · p. 8
• BERT + ResNet-152 is the best fusion model, scoring 2 points above fastText + ResNet-152; early fusion models outperform late fusion.Late fusion averages modality results, while early fusion trains separate multilayer perceptrons for visual and textual features.Source: §6 · p. 8
• Every multimodal model outperforms every unimodal model; ViLBERT CC and VisualBERT COCO achieve the best results.The reported comparison spans rows 9–11 against rows 1–8.Source: Table 4 · p. 9, §6 · p. 9

7 Conclusion and Future Work

• The proposed task detects propaganda technique types in memes, using a released corpus of 950 memes annotated with 22 techniques that may appear in text, images, or both.Experiments and corpus analysis indicate that understanding both modalities is essential for detecting these techniques.Source: §7 · p. 9
• Future work plans to enlarge the dataset, including with memes in other languages.The authors also plan to develop new multimodal models tailored to fine-grained propaganda detection.Source: §7 · p. 9
• Planned models target deeper understanding of meme semantics and the relation between text and image.Ideas tried by SemEval-2021 shared-task participants may inform new models.Source: §7 · p. 9

Ethics and Broader Impact

• The dataset contains memes only and no user information.Source: Ethics and Broader Impact · p. 9
• Propaganda-technique annotation is subjective, so biases may remain in gold labels or label distributions despite the schema and high inter-annotator agreement.The authors collected examples from varied users and groups and used a schema with clear definitions.Source: Ethics and Broader Impact · p. 9
• The dataset could be misused to unfairly moderate memes based on biases that may or may not relate to demographics or other textual information.The authors state that human moderation would be required to prevent this outcome.Source: Ethics and Broader Impact · p. 9
• The dataset is presented as a resource for research on harmful memes when used appropriately.Source: Ethics and Broader Impact · p. 9

A.1.1 Choice of memes/Filtering Criteria

• The dataset defined memes as photograph-style images with short text and filtered out 111 items, excluding diagrams and memes unsuitable for multimodal analysis.The excluded items included images with diagrams, graphs, or tables, and memes containing only text or only an image.Source: §A.1.1 · p. 12

A.1.2 Rules for Text Editing

• Google Vision API text extraction required manual checking because its output sometimes contained errors.The extracted meme text was then edited according to the listed rules.Source: §A.1.2 · p. 12
• Editing removed social-account identifiers and interface elements, logo text, figure and table text, nearly unreadable text, and text unrelated to the meme.The rules also remove signed authorship; arrange columns in reading order; and separate sentences and distinct text blocks where possible.Source: §A.1.2 · p. 12
• When possible, editors put one sentence on each line, separated distinct image text blocks, and kept blocks together when they formed one sentence.Column text was ordered from the first column to the next.Source: §A.1.2 · p. 12

A.2 Guidelines for Annotators - Phases 2-5

• The passage says that the annotators were presented with the following guidelines and that annotations in these phases were performed by three annotators.Source: §A.2 · p. 12

A.2.1 Annotation Phase 2

• Text-only annotation identified which techniques appeared in a meme’s text and marked each technique’s exact span.The task used the list of techniques 1–20.Source: §A.2.1 · p. 12

A.2.2 Annotation Phase 4

• Multimodal annotation identified which of 22 techniques appeared in the text, visual content, or both; image context could reveal a text technique.Source: §A.2.2 · p. 12

A.2.3 Consolidation (Phase 3 and 5)

• Annotators and consolidators discussed annotations to reach consensus, checking existing labels and adding newly observed technique instances.Source: §A.2.3 · p. 12

• Training used a learning rate of 5e-5, epsilon of 1e-8, weight decay of 0.01, and maximum sequence length of 128.Source: B Hyper-parameter Values · p. 15
• Models were trained for 37 epochs, with early stopping based on F1-micro on the development set.Source: B Hyper-parameter Values · p. 15
• Reported parameter counts were BERT 110,683,414; fastText + ResNet-152 11,194,398; BERT + ResNet-152 170,983,752; ViLBERT CC 112,044,290; and VisualBERT COCO 247,782,404.Source: B Hyper-parameter Values · p. 15