PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting and Understanding Harmful Memes: A Survey

Shivam Sharma, Firoj Alam, Md. Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Halevy, Fabrizio Silvestri, Preslav Nakov, Tanmoy Chakraborty

arXiv:2205.04274v2cs.CLcs.AIcs.CV

TL;DR

Harmful online content is a concern, and memes combine modalities in a widely shared form. This survey proposes a literature-based typology and reviews harmful-meme detection research. It finds that some types lack detection studies and that reviewed classification setups are primarily binary.

  • Problem

    Harmful content often combines modalities, as in memes, while research has studied content modalities in isolation.

    Source: §1 · p. 1

  • Method

    The survey analyzes literature on harmful memes, proposes a typology, and summarizes detection research and challenges.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    The typology identifies harmful-meme categories without detection studies, while reviewed classification setups are primarily binary.

    Source: §2 · p. 2, §2.2 · p. 4

  • Takeaways & Limitations

    The survey highlights fine-grained analysis, complex meme abstraction, annotation subjectivity, and multimodal learning as challenges.

    Source: §1 · p. 2

Abstract

from arXiv · show

The automatic identification of harmful content online is of major concern for social media platforms, policymakers, and society. Researchers have studied textual, visual, and audio content, but typically in isolation. Yet, harmful content often combines multiple modalities, as in the case of memes, which are of particular interest due to their viral nature. With this in mind, here we offer a comprehensive survey with a focus on harmful memes. Based on a systematic analysis of recent literature, we first propose a new typology of harmful memes, and then we highlight and summarize the relevant state of the art. One interesting finding is that many types of harmful memes are not really studied, e.g., such featuring self-harm and extremism, partly due to the lack of suitable datasets. We further find that existing datasets mostly capture multi-class scenarios, which are not inclusive of the affective spectrum that memes can represent. Another observation is that memes can propagate globally through repackaging in different languages and that they can also be multilingual, blending different cultures. We conclude by highlighting several challenges related to multimodal semiotics, technological constraints, and non-trivial social engagement, and we present several open-ended aspects such as delineating online harm and empirically examining related frameworks and assistive interventions, which we believe will motivate and drive future research.

1 Introduction

Online content can spread harm and hostility, while memes combine images and embedded text in a prevalent, often humorous form. The survey focuses on harmful memes, proposes a typology, and reviews detection research and its challenges.

1 Introduction

  • Social media enabled free sharing and free speech, alongside the spread of harm and hostility.Source: §1 · p. 1
  • The survey defines harmful memes as image-and-text units with potential to harm individuals, organizations, communities, or society.It examines their detection and analysis.Source: §1 · p. 2
  • The survey proposes a literature-based typology and highlights multimodality, multilinguality, cultural influences, repackaging, and challenges in analysis and annotation.The challenges include fine-grained analysis, meme abstraction, annotation subjectivity, and multimodal learning.Source: §1 · p. 2

2 Harmful Memes

The paper presents a new typology of harmful social-media content focused on meme dissemination. It differs from prior categorizations and includes categories for which the authors found no studies despite their prominence online.

2 Harmful Memes

  • The typology focuses on harmful content disseminated through memes and is inspired by, but differs from, earlier categorizations [Banko et al., 2020; Nakov et al., 2021b; Pramanick et al., 2021a].Source: §2 · p. 2
  • The typology excludes misinformation because it is not always harmful, while disinformation is not a separate category because its examples overlap with the typology’s subcategories [Alam et al., 2021].Some listed subcategories, such as doxing and identity attack, fall under malinformation.Source: §2 · p. 2
  • Some harmful-meme types have no detection studies despite being prominent on social media, as indicated by searches using the typology’s keywords [Sabat et al., 2019].Source: §2 · p. 2

I: Hate

The reviewed hate-meme literature covers racism and misogyny/sexism, including physical appearance, ethnicity, and multimodal computational approaches. Its examples range from qualitative analyses of racist discourse to detection models and studies of how textual cues contribute to sexist-meme classification.

I: Hate

  • Racism is described as systematic out-casting across social, economic, and cultural ecosystems, with race-based policies or behavior producing unfair advantage or harmful treatment.Source: §I · p. 3
  • Studies of online racism used keyword-scraped memes and qualitative or thematic analysis; Williams et al. [2016] found higher likelihood of perceiving online racism among offline victims.Source: §I · p. 3
  • Racist memes targeting African-Americans juxtaposed Black NBA athletes with white NFL players to promote white supremacy.Memes targeting Aboriginal Australians also used slurs, cropped faces, skin tone, and stereotypes.Source: §I · p. 3
  • Ethnicity-focused studies examined anti-Semitic meme spread, derogatory language, nationalism, conspiracy theories, and hostility toward Jews [Zannetto et al., 2020].Other work analyzed racist discourse and triple-parenthesis memes.Source: §I · p. 3
  • Multimodal fusion models yielded 0.71 and 0.90 F1 scores for Twitter and Gab racist-meme datasets, respectively, suggesting greater multimodality propensity in Gab.The study combined ResNET152 and RoBERTa with Multimodal Fusion Architecture Search (MFAS).Source: §I · p. 3
  • Sexist memes use humor and derogatory language, while imagery of confident women can also appear in non-sexist memes, underscoring the importance of textual cues.A late-fusion strategy using textual cues achieved an F1 score of 0.76 for sexist-meme detection.Source: §I · p. 3

II: Offensive Memes

Research on offensive memes has used unimodal and multimodal classifiers, analogy-aware contextualization, and commonsense knowledge. Reported results include accuracy gains from graph-based multimodal entity extraction.

II: Offensive Memes

  • A graph neural network approach using multimodal entity extraction and commonsense knowledge improved accuracy by 1% in both Reddit- and Gab-based scenarios [Shang et al., 2021b].Source: §II · p. 3
  • Analogy-aware multimodality with contextual discourse achieved 0.72 and 0.69 accuracy for Reddit- and Gab-based datasets, respectively [Shang et al., 2021a].Source: §II · p. 3
  • Unimodal and multimodal offensive-meme classifiers reported an F1 score of 0.71 and accuracy of 0.50, respectively [Giri et al., 2021; Suryawanshi et al., 2020b].The multimodal approach used text models alongside VGG16.Source: §II · p. 3

III: Propaganda

The survey describes propaganda memes as using shared styles, symbols, and iconography to promote contrasting ideologies, including recruitment and violent racial supremacy. It also notes a lack of significant automated detection efforts for harassment and cyberbullying in memes.

III: Propaganda

  • Propaganda memes use similar styles, symbolism, and iconography to promote contrasting inclinations, including recruitment and violent racial supremacy.A multimodal, multi-label task has examined fine-grained propaganda techniques in meme text and the full meme.Source: §III · p. 4

IV: Harassment/Cyberbullying

  • Despite the use of memes in cyberbullying, the survey found no significant efforts toward automated detection of harassment or cyberbullying in memes.Prior work found that image and text modalities complement each other in cyberbullying detection.Source: §IV · p. 4

V: Violence

The survey finds that harmful- and hateful-meme results are comparatively better than results for other tasks, while violent and self-inflicted harmful memes lack dedicated automated detection studies. It notes that existing violence research includes surveillance videos and hateful memes, but not work specifically targeting violent-meme detection.

V: Violence

  • Although violent memes have been studied in contexts such as hateful-meme detection, the survey found no work specifically focused on violent-meme detection.Other violence-detection research discussed focuses on surveillance videos using video and audio modalities.Source: §V · p. 4

VI: Self-Inflicted Harm

  • The survey found no literature on automatic detection and analysis of self-inflicted harmful memes.Related automated-detection studies instead analyze textual, visual, and network content for self-harm and eating disorders.Source: §VI · p. 4

2.2 Summary

  • Results are comparatively better for harmful and hateful memes than for remaining tasks, while troll identification is only slightly better than random.The table summarizes experimental results across harmful-meme detection tasks, datasets, and approaches.Source: §2.2 · p. 4, Table 1 · p. 5

3 Repackaging Memes for Harmful Agendas

The survey describes meme remixing and imitation as a way memes spread across languages and cultures, sometimes with harmful intent. Examples include repeated harmful or humorous remakes of the Distracted Boyfriend meme and the Proud Boys' use of Pepe the Frog imagery.

3 Repackaging Memes for Harmful Agendas

  • Remixing or mimicking memes facilitates their adoption across languages and cultures and can carry harmful intent.The Distracted Boyfriend meme has been repackaged with varying intent, including harm and humor.Source: §3 · p. 5
  • The Proud Boys' version of Pepe the Frog depicts the character in their uniform and displaying the OK hand gesture.The survey describes Pepe's use as reinstating affiliation with far-right ideologies.Source: §3 · p. 5

4 Cultural Influence and Multilinguality

Memes can be translated, customized, and distributed globally by ordinary users, and may also blend multiple languages. The cited work describes uneven cultural adaptation across languages, while modelling multilingual memes is complex.

4 Cultural Influence and Multilinguality

  • Shifman [2013] calls the global translation, customization, and distribution of memes by ordinary users “user-generated globalization.”In one studied joke, adaptation worked well in most of the top nine non-English languages, except Arabic; localization was limited in Chinese, German, and Portuguese.Source: §4 · p. 5
  • Memes can use multiple languages, as in the TamilMemes dataset, and modelling them is complex.A study found many European organizations used image genres such as share-posts and templates rather than memes' irreverent, participatory forms.Source: §4 · p. 6

5 Major Challenges

The survey identifies challenges spanning complex multimodal abstraction, subjective annotation, limited data and solutions, rapid meme evolution, missing context, platform constraints, and difficulty identifying intent.

Complex abstraction

  • Memes can hide complex ideas in visual and linguistic nuances, including irony and implicit meaning, making automated analysis and multimodal fusion difficult.Examples include overlapping snippets, patterned text, and triple parentheses that can insinuate targeted entities.Source: §5 · p. 6

Subjectivity in the annotation

  • Annotation is subjective because visual and linguistic interplay and missing context can produce conflicting judgments about harmful memes.For 59 ambiguous misogynistic memes, crowd-sourced workers correctly identified 23%, while domain experts achieved 77% expert agreement [Gasparini et al., 2021].Source: §5 · p. 6

Inadequate solutions

  • Conventional hand-crafted visual features and modeling are inadequate for understanding memes’ visual content.Visual descriptors may be non-discriminatory, while pHash clustering cannot model visual role-play.Source: §5 · p. 6

Insufficient sample size

  • Generalizable meme analysis requires large samples, while keyword-based, platform-dependent collection can bias the sample space toward typical memetic characteristics.Source: §5 · p. 6

Rapid evolution

  • Harmful memes evolve with new events and adversaries’ efforts to bypass detection, while AI systems struggle to generalize from a few examples [Wang et al., 2020].The survey argues for multimodal few-shot learning to adapt to emerging meme types and policies.Source: §5 · p. 6

Contextualization

  • Many memes require common-sense, factual, or situational context that may be conveyed through text, visuals, or both, but is often unavailable for the target meme.Source: §5 · p. 6

Platform restrictions

  • Non-standard accountability and transparency across evolving platforms complicate systematic online-harm research.Anonymity can obscure accountability, and platform governance and community dynamics can complicate analysis [Dickerson, 2016; Matamoros-Fernández, 2020; Zannetto et al., 2020].Source: §5 · p. 6

Identifying real instigators of harm

  • Distinguishing satire from extremism and identifying actual intent or targets remain difficult, including when memes obscure their targets.Regulatory bodies also find it challenging to distinguish anti-democracy extremists from anti-democratic alt-right factions.Source: §5 · p. 6

6 Future Forecasting

The survey points to research on the vehicles and effects of harmful memes, cross-cultural interpretation, richer contextual data, finer-grained classification, and counter-memes.

Characterizing vehicles of harm

  • Research should examine how satire and memes can be weaponized, how violent messages circulate between malicious actors and receptive users, and how rhetoric connects to rooted antisemitic perspectives.Source: §6 · p. 7

Cross-cultural studies

  • Cross-cultural studies could investigate how offline racial experiences relate to online perceptions of racism across cultural contexts.Source: §6 · p. 7

Empirical in addition to theoretical

  • Empirical work should investigate harmful memes’ potency, cross-platform transitions, outcomes, and the prevalence of visual forms and cryptic visual-linguistic semiotics.Source: §6 · p. 7

Rich metadata

  • Meme analysis could incorporate social-media tags, video, user profiles, and intermediate representations that capture abstraction across image and text modalities [Chandra et al., 2021].The survey describes contextual knowledge as indispensable for automated meme analysis [Shang et al., 2021b].Source: §6 · p. 7

Multi-class and multi-label classification

  • Multi-class and multi-label analysis can preserve specificity lost when a single hateful label covers characteristics such as violence and misogyny [Kiela et al., 2020].Related work examines fine-grained hate and propaganda, victims, and the roles of heroes, villains, and victims [Zia et al., 2021; Dimitrov et al., 2021a; Pramanick et al., 2021a; Sharma et al., 2022a,b].Source: §6 · p. 7

Memetic moderation

  • Counter-memes and participatory humor are proposed as ways to raise awareness about racial issues and resist selective targeting and oppression.Source: §6 · p. 7

7 Conclusion

The paper surveys technologies for detecting and understanding harmful memes, proposes a typology from recent literature, synthesizes the state of the art and challenges, and suggests future research directions.

7 Conclusion

  • The survey systematically analyzes recent literature to propose a harmful-meme typology, summarize the state of the art, discuss lessons and challenges, and suggest future research directions.Source: §7 · p. 7

Related papers

Loading 2205.04274v2…