PaperlayerView exact revision on arXiv

Source-linked AI summary

UkrMeme: Multimodal Explainable Persuasion Detection in Ukrainian Memes

Nazar Andrushko, Vera Schmitt, Veronika Solopova

arXiv:2610.13783v1cs.AIcs.CLcs.CV

TL;DR

The paper asks how models identify persuasion techniques in Ukrainian wartime memes, where visual and textual signals combine in political communication. It constructs two complementary datasets and evaluates models across text, image, and combined inputs, comparing explanation signals. Performance drops substantially on native Ukrainian memes relative to translated benchmark memes, while multimodal inputs improve recall but often over-predict techniques.

  • Problem

    The study asks how models identify persuasion techniques in Ukrainian memes, where visual and textual cues combine in wartime political communication.

    Source: §1 · p. 1

  • Method

    The study builds two complementary Ukrainian meme datasets, evaluates models zero-shot across text, image, and combined inputs, and compares LRP saliency with model-based token attributions.

    Source: §1 · pp. 1–2

  • Results

    Performance drops substantially from PTM-UA (Qwen3-VL-4B image+text micro hF1 0.537) to UkrMeme (Qwen3-VL-8B image+text micro F1 0.342); multimodal inputs improve recall but often over-predict techniques.

    Source: §4.1 · p. 5, §5 · p. 9

  • Takeaways & Limitations

    Cultural knowledge and contextual grounding remain key bottlenecks for analyzing multimodal political discourse in low-resource, culturally specific contexts.

    Source: §5 · p. 9

    The datasets are small (158 translated PTM-UA memes and 200 native UkrMeme memes), limiting statistical power and coverage of Ukrainian meme diversity.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Memes play a central role in modern online communication, especially in contexts of political conflict where visual and textual cues combine to convey persuasive messages. We study persuasive technique detection in Ukrainian memes using vision-language models (VLMs). We introduce UkrMeme, a dataset of 200 organically produced Ukrainian wartime memes annotated for persuasion techniques, and PTM-UA, a Ukrainian translation of a subset of the Propaganda 950 benchmark that preserves original visual content while translating meme text. We evaluate VLMs in a zero-shot prompting setup across image, text, and multimodal inputs using hierarchical F1 metrics. Finally, we compare Layer-wise Relevance Propagation (LRP) and model-generated token attributions and analyze the semantic categories of evidence emphasized by each explanation method.

1 Introduction

The paper studies persuasion techniques in Ukrainian memes, where visual and textual cues combine in wartime communication. It addresses limited cross-lingual and culturally grounded evaluation by building complementary datasets and examining model behavior and explanations.

1 Introduction

  • Existing persuasion benchmarks are largely English-centric, while cross-lingual and culturally grounded analyses—especially for low-resource languages and active-conflict contexts—remain limited.Prior work also emphasizes model performance more than faithfulness and explainability.Source: §1 · p. 1
  • The study treats persuasion labels as an analytical lens for grassroots wartime rhetoric, not as evidence that Ukrainian memes are state-coordinated propaganda.It introduces UkrMeme, 200 organically produced wartime memes annotated by two native speakers with adjudication, and PTM-UA, a Ukrainian translation of benchmark memes that preserves their images.Source: §1 · p. 1
  • The evaluation uses zero-shot prompts across image, text, and multimodal inputs, then compares LRP with model-generated token attributions and analyzes the evidence categories each emphasizes.The paper also presents a pilot empirical perspective on persuasion in grassroots wartime meme culture.Source: §1 · p. 2

2 Related Work

Prior work established taxonomies and benchmarks for persuasion in political text and later extended the task to memes, but Ukrainian and other culturally specific settings remain underexplored. The paper also identifies a gap in prompt-only Ukrainian evaluation and faithfulness-oriented explanation comparisons.

2 Related Work

  • Early work developed fine-grained persuasion taxonomies and supervised benchmarks for political news and speeches, while annotation subjectivity makes low agreement expected.The passage cites Da San Martino et al. (2019a), Dmitriev et al. (2019), and Sahitaj et al. (2025).Source: §2 · p. 2
  • Meme benchmarks model meaning across text and images, but remain predominantly English-centric, leaving cross-lingual transfer and culturally specific wartime meme production underexplored.The cited meme work includes Dimitrov et al. (2021), Kmainasi et al. (2025), and Singh et al. (2025).Source: §2 · p. 2
  • To the authors’ knowledge, prior work had not systematically tested prompt-only Ukrainian persuasion detection or compared text-only and vision-language models using faithfulness-oriented explanation metrics.The paper positions this as a gap in existing evaluation.Source: §2 · p. 2

3 Methods

The methods combine a translated English meme benchmark with a newly collected Ukrainian meme dataset, then evaluate models and compare two forms of token explanation. The setup controls selected comparisons through paired visual content, input modalities, and shared hierarchical metrics.

3.1.1 Translated Propaganda 950 (PTM-UA)

  • PTM-UA translates a Propaganda 950 test subset through text detection, inpainting, Ukrainian translation, and re-rendering, then retains 158 manually reviewed samples from 200.Samples were rejected for poor text placement, loss of important visual information during inpainting, or inaccurate translation; samples without technique annotations are treated as No Propaganda.Source: §3.1.1 · pp. 2–3

3.1.2 UKRMEME

  • UkrMeme contains 200 memes collected from open Ukrainian-language sources and independently labeled by two native speakers, with disagreements resolved by a curator.The dataset adds a No Propaganda label; its label distribution has a larger share of those samples and a different technique ranking than PTM-UA.Source: §3.1.2 · p. 3
  • UKRMEME annotation agreement is moderate overall, with MASIweighted Krippendorff’s α = 0.406 and mean per-class κ = 0.404.Agreement is highest for No Propaganda (κ = 0.776); low κ for some rare classes, such as Loaded Language (κ = 0.149), reflects low prevalence rather than poor annotation quality.Source: §3.1.2 · p. 3

3.2.1 Motivation

  • The study compares LRP, a model-internal gradient-based explanation, with model-generated token attribution, a linguistic explanation that relies on the model’s own reasoning.The authors note that LRP can depend on propagation rules and architecture, potentially reflecting methodological choices rather than the model’s true decision process.Source: §3.2.1 · pp. 3–4

3.2.2 Models

  • Qwen3-VL-4B-Instruct and 8B-Instruct support detection and explanation analysis, while GPT-4.1-mini and Gemma 3 4B are evaluated for detection only.The open-weight Qwen models expose internal activations for LRP, and their size difference supports analysis of scale effects.Source: §3.2.2 · p. 4

3.2.3 Detection Metrics

  • Hierarchical F1 gives partial credit for predicted and gold labels sharing taxonomy ancestors, using extended label sets that include ancestors other than the root.The taxonomy has 22 techniques under categories including Ethos, Pathos, Logos, and Ad Hominem; results include micro-F1, macro-F1, and mean per-sample F1.Source: §3.2.3 · p. 4

3.2.4 Baselines and translation control

  • A weighted-random baseline samples each label independently at its empirical positive frequency, while a paired English-versus-Ukrainian PTM-UA comparison holds meme identity and visual content constant.The baseline is averaged over 30 runs; the translation comparison uses the same four models, input modalities, and prompting settings.Source: §3.2.4 · p. 4

3.2.5 Explanation Setup

  • The evaluation compares image-only, image-plus-extracted-text, and text-only inputs, with prompts requesting labels, relevant words, and supporting image concepts.LRP relevance is derived from predicted-label logits and aligned to OCR words; the analysis compares selected words using Jaccard similarity and categorizes tokens with GPT-4o.Source: §3.2.5 · pp. 4–5

4 Results

Models outperform weighted-random baselines, but results vary across datasets, language conditions, modalities, and scale; analyses also reveal systematic label confusions and distinct token-evidence profiles.

4.1 Performance and Cross-Dataset Gap

  • GPT-4.1-mini achieves the highest image+text micro hF1 on both datasets: 0.613 on PTM-UA and 0.384 on UkrMeme.It outperforms both Qwen models on detection; macro F1 remains below 0.30 on PTM-UA and 0.20 on UkrMeme.Source: §4.1 · pp. 5–6, Table 3 · p. 6

4.2 Language Effect: Original vs. Translated English

  • 0.577 vs. 0.537 (Qwen3-VL-4B image+text) and 0.548 vs. 0.523 (8B) show higher hF1 on original English than translated Ukrainian; GPT-4.1-mini and Gemma differences stay within ±0.03.The paired comparison uses the same 158 memes and holds meme identity and visual content constant while textual condition differs.Source: Table 4 · p. 6, §4.2 · pp. 6–7

4.3 Effect of Input Modality

  • On PTM-UA, image+text bests 4B across averaging methods, while on UkrMeme text-only leads per-sample F1 (0.277 for 4B; 0.388 for 8B) as image modes trade precision for recall.For PTM-UA, the 8B image-only micro F1 of 0.524 is competitive with image+text at 0.523.Source: §4.3 · p. 7

4.4 Effect of Model Scale

  • Scaling from 4B to 8B raises PTM-UA image-only micro F1 from 0.483 to 0.524 and UkrMeme text-only per-sample F1 from 0.277 to 0.388, while multimodal gains are modest.On PTM-UA, the 8B model also achieves the best macro F1 at 0.291.Source: §4.4 · p. 7

4.5 No Propaganda Label Ablation

  • Removing the added No Propaganda option yields the largest gains in text-only mode; of 31 such predictions, 4B replaces 23 with technique labels while 8B makes 22 empty.Only two PTM-UA samples genuinely lack annotations; image and image+text improvements are marginal.Source: §4.5 · p. 7

4.6 Per-Class Confusion Patterns

  • Loaded Language attracts predictions on both datasets (283 column vs. 214 row pairs on PTM-UA; 179 vs. 12 on UkrMeme), while several mid-frequency classes never appear on the diagonal.The UkrMeme No Propaganda row has 170 pairs and drives most false positives, including 36 predicted Loaded Language, 31 Slogans, and 29 Exaggeration/Minimisation.Source: §4.6 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8

4.7 LRP and token attribution

  • Translated memes emphasize original English political-discourse tokens, while native Ukrainian memes center on wartime context; image+text explanations include broader contextual tokens than text-only explanations.In native memes, image+text contributes contextual evidence not reducible to extracted text; the translated condition’s text and image modes are more similar.Source: §4.7 · pp. 7–8, Figure 7 · p. 8
  • Native multimodal memes have the highest median VLM–LRP attribution overlap, yet low-hF1 errors often reflect category confusion rather than missed cues.Some translated text-only cases also show low attribution overlap, associated with possible OCR noise, translation artifacts, or unstable lexical grounding.Source: §4.7 · pp. 8–9, Figure 8 · p. 9

5 Discussion and Conclusion

The results show both potential and limitations for vision–language models analyzing multimodal political discourse. Performance, modality, model scale, and explanation emphasis each reveal distinct challenges for reliable detection and interpretation.

5 Discussion and Conclusion

  • Performance drops substantially on native Ukrainian memes compared with translated memes derived from English benchmarks, highlighting cultural knowledge and contextual grounding as bottlenecks.Models achieve moderate performance on the translated memes.Source: §5 · p. 9
  • Multimodal inputs improve recall but often over-predict persuasion techniques, particularly in emotionally charged wartime content.Source: §5 · p. 9
  • Increasing model scale yields uneven gains, with improvements primarily in text-only settings rather than multimodal reasoning.Source: §5 · p. 9
  • Model-generated token attributions emphasize emotional and rhetorical cues, while LRP-based signals focus more on content words.Source: §5 · p. 9

Limitations

The study’s scope is constrained by small datasets, subjective annotations, translated rather than native content in PTM-UA, and zero-shot experiments without supervised systems.

Limitations

  • PTM-UA contains 158 translated memes and UkrMeme 200 native Ukrainian memes, limiting statistical power and coverage of Ukrainian meme diversity.The authors describe the evaluation as exploratory.Source: Limitations · p. 10
  • PTM-UA retains English memes’ visuals while translating their text, which may introduce artifacts and does not fully reproduce authentic Ukrainian meme discourse.The authors also say propaganda-derived labels may not capture all methods of genuine civic society creation.Source: Limitations · p. 10
  • The experiments use zero-shot prompting, leaving supervised or fine-tuned systems unevaluated; explainability is limited to textual token attribution and two Qwen3-VL models.The authors cite unavailable internal activations for GPT-4.1-mini and computational feasibility for Gemma 3 4B as reasons for excluding them from LRP analysis.Source: Limitations · p. 10

Ethical Considerations

The paper treats persuasion techniques as an analytical lens for Ukrainian grassroots wartime communication, not as evidence that memes are coordinated propaganda. It frames the analysis descriptively and warns against using detection systems to police political speech.

Ethical Considerations

  • The study treats persuasion techniques as an analytical framework for rhetorical strategies in grassroots digital communication, not as proof of coordinated propaganda.Ukrainian wartime memes may serve humor, resilience, and collective coping.Source: Ethical Considerations · p. 10
  • The authors describe their analysis as descriptive rather than normative and caution against using automated detection for censorship or political moderation.They acknowledge potential misuse to suppress legitimate political expression.Source: Ethical Considerations · p. 10
  • The dataset uses memes collected from publicly accessible online sources and excludes private or personally identifiable information.Source: Ethical Considerations · p. 10

A Additional Results

The additional results cover hierarchical F1 on PTM-UA and UkrMeme for Qwen3-VL models, a PTM-UA comparison with and without the No Propaganda label, and a confusion matrix after removing that label.

A Additional Results

  • Table 6 reports hierarchical F1 for Qwen3-VL models on PTM-UA and UkrMeme, with best hF1 per averaging method bolded within each model and dataset.Source: Table 6 · p. 13
  • Table 7 compares PTM-UA results with and without No Propaganda; the largest improvements appear in text-only mode, where that label is most frequently predicted.Source: Table 7 · p. 13
  • Without No Propaganda, the Qwen3-VL-4B image-plus-text confusion matrix redistributes former abstentions into partially correct technique labels.Source: Figure 9 · p. 13

B Model exclusion from Explainability Study

GPT-4.1-mini and Gemma 3 4B are detection baselines only, because explainability analysis was infeasible for both models for different reasons.

B Model exclusion from Explainability Study

C Prompt Templates

The appendix specifies separate prompts for text-containing inputs and image-only inputs, with Ukrainian outputs and instructions for extracting supporting words or visual concepts. It also describes the supplied label set and a separate prompt for categorizing attributed words.

C Prompt Templates

  • Two prompt templates vary by input modality and use the full set of persuasion labels, with NO_PROPAGANDA included when applicable.For the ablation without that option, the label and its corresponding prompt clause are removed.Source: C Prompt Templates · p. 14

C.1 Image+Text and Text-Only Prompt

  • For image+text and text-only inputs, the requested JSON includes labels, supporting individual content words, and up to five ranked visual concepts.The prompt asks for Ukrainian values and concrete, concise visual references rather than abstract scene descriptions.Source: §C.1 · p. 14

C.2 Image-Only Prompt

  • The image-only prompt requests labels and up to five ranked visual concepts, excluding words visible in the image from those concepts.Visual concepts should name concrete elements in Ukrainian.Source: §C.2 · p. 14

C.3 Propaganda Technique Labels

  • The prompts offer 23 options: 22 persuasion techniques plus NO_PROPAGANDA.Source: §C.3 · pp. 14–15

D Category Annotation Prompt

  • A GPT-4o prompt categorizes words flagged as important using the meme text, predicted labels, and word list as context.The prompt assigns one category to each word based on its role in that meme.Source: D Category Annotation Prompt · p. 15
  • The word categories distinguish entities, emotional language, rhetoric, factual content, and other items such as numbers or unclear words.The prompt’s examples include calls to action and slogans under rhetoric, and dates or statistics under content.Source: D Category Annotation Prompt · p. 15

E Hierarchical F1 Definitions

Hierarchical F1 evaluates predictions after extending gold and predicted labels to their taxonomy ancestors, excluding the root. The section defines per-sample, micro-averaged, and macro per-label variants.

E Hierarchical F1 Definitions

Related papers

Loading 2610.13783v1…