Source-linked AI summary
Detecting Harmful Memes and Their Targets
Shraman Pramanick, Dimitar Dimitrov, Rituparna Mukherjee, Shivam Sharma, Md. Shad Akhtar, Preslav Nakov, Tanmoy Chakraborty
TL;DR
Harmful memes can use satirical, context-dependent image-text combinations to target diverse social entities, challenging off-the-shelf multimodal models. The paper introduces HarMeme and two tasks for detecting harmfulness and identifying targets, then benchmarks unimodal and multimodal models. The evaluation highlights the importance of multimodal signals and reveals limitations and biases in existing models.
Problem
Harmful memes are difficult to interpret because their satirical, context-dependent, and camouflaged semantics can target social entities beyond hate or offense.
Method
The paper introduces HarMeme, a 3,544-meme COVID-19 benchmark, and evaluates ten unimodal and multimodal models on harmfulness and target identification.
Results
Evaluation results highlight the importance of modeling multimodal signals for both harmful meme detection and target identification.
Takeaways & Limitations
The findings indicate a need for more sophisticated methods for harmful meme detection and target identification.
Takeaways & Limitations
The best multimodal model can rely on dataset biases, such as frequent politician names and images, instead of interpreting an individual meme's underlying semantics.
Abstract
from arXiv · showhide
Among the various modes of communication in social media, the use of Internet memes has emerged as a powerful means to convey political, psychological, and socio-cultural opinions. Although memes are typically humorous in nature, recent days have witnessed a proliferation of harmful memes targeted to abuse various social entities. As most harmful memes are highly satirical and abstruse without appropriate contexts, off-the-shelf multimodal models may not be adequate to understand their underlying semantics. In this work, we propose two novel problem formulations: detecting harmful memes and the social entities that these harmful memes target. To this end, we present HarMeme, the first benchmark dataset, containing 3,544 memes related to COVID-19. Each meme went through a rigorous two-stage annotation process. In the first stage, we labeled a meme as very harmful, partially harmful, or harmless; in the second stage, we further annotated the type of target(s) that each harmful meme points to: individual, organization, community, or society/general public/other. The evaluation results using ten unimodal and multimodal models highlight the importance of using multimodal signals for both tasks. We further discuss the limitations of these models and we argue that more research is needed to address these problems.
1 Introduction
Memes increasingly communicate harmful, often context-dependent and camouflaged messages targeting social entities, challenging standard multimodal analysis. HarMeme formulates harmfulness and target identification as separate tasks and benchmarks them with annotated COVID-19 memes and unimodal and multimodal models.
- Motivation: Memes can target individuals, groups, businesses, and society, while multimodal and camouflaged semantics make harmful content difficult to analyze.Context dependence, weak image-text correlation, morphed images, and difficult OCR further limit off-the-shelf tools.
- Motivation: Harmful meme detection extends beyond hate and offense to include harms such as cyberbullying and fake news.The paper notes that a meme can harm a news organization without being hateful or offensive.
- Contributions: The paper defines two problems: classifying memes as very harmful, partially harmful, or harmless, and identifying whether harmful memes target individuals, organizations, communities, or society.The target categories are illustrated in Figure 1 using the [Intensity, Target] label format.
- Contributions: HarMeme is a benchmark dataset of 3,544 real COVID-19 memes collected from the web and annotated under detailed guidelines.The dataset supports both harmfulness classification and harmful-target identification.
- Experiments: Experiments benchmark ten unimodal and multimodal models and indicate that systematically combining multimodal signals is needed for both tasks.The authors also discuss model limitations and report biases in the best multimodal model.
2 Related Work
Prior meme research covers hate, offense, sentiment, emotion, propagation, misinformation, and related phenomena. HarMeme differs by studying broader harm and its targets using real-world memes and a dedicated benchmark.
- Existing studies: Existing datasets and studies examine hate speech, offensive content, sentiment, emotion, misinformation, propaganda, virality, and meme propagation.The surveyed resources span multimodal classification, social-media dissemination, and large-scale meme tracking.
- Differences with existing studies: The closest prior work focuses on hateful memes, whereas HarMeme addresses harmful memes as a broader category.The paper treats hateful and offensive memes as harmful but not all harmful memes as hateful or offensive.
- Differences with existing studies: HarMeme additionally identifies the entities targeted by harmful memes, including individuals, organizations, communities, and society.This target-identification task is presented alongside harmfulness detection.
- Differences with existing studies: Unlike synthetic memes used in the Hateful Memes Challenge, HarMeme contains real-world memes posted on the web.The paper presents dataset and benchmark results for both harmfulness and target identification.
3 Harmful Meme: Definition
The paper defines a harmful meme as an image-text unit with the potential to harm a social entity. Harm is broader than hate or offense and may be concealed, requiring contextual judgment.
- Definition: A harmful meme is an image with embedded text that has the potential to harm an individual, organization, community, or society.The definition treats memes as multimodal units rather than text-only messages.
- Definition: Harm includes mental abuse, defamation, psycho-physiological injury, proprietary damage, emotional disturbance, and compromised public image.These categories broaden the scope beyond conventional hateful or offensive content.
- Harmful vs. hateful/offensive: Harmful is more general than hateful or offensive: hateful and offensive memes are harmful, but some harmful memes are neither hateful nor offensive.The paper gives examples involving Donald Trump and news media.
- Harmful vs. hateful/offensive: Harmful content is often camouflaged and may require critical judgment to determine its potential to cause harm.Unlike hateful memes, harmful memes can target entities without attacking sensitive personal attributes.
4 Dataset
HarMeme was collected from web and social-media sources, filtered and deduplicated, then annotated for harmfulness and target type through trained multi-annotator procedures.
- Data Collection: 5,027 initially collected memes were filtered for quality and deduplicated because independent sources produced duplicate items.Filtering addressed image resolution and meme-text readability.
- Annotation Guidelines: Four target classes were defined for harmful memes: individual, organization, community, and society.The guidelines specify examples and operational meanings for each class.
- Annotation Guidelines: Memes were rejected when they used non-English or code-mixed text, unreadable text, only one modality, or cartoons.These criteria constrained the dataset to readable multimodal English memes without cartoons.
- Annotation Process: 15 annotators worked in groups using PyBossa, and each valid meme was rated by three annotators for both tasks.Annotators were trained before final annotation, with a fourth added when agreement was insufficient.
- Annotation Process: Cohen’s κ increased to 0.695 and 0.797 after consolidation for the two tasks, indicating moderate and high agreement respectively.The authors also report variability in judging harmfulness among human experts.
5 Benchmarking HarMeme dataset
HarMeme is benchmarked with textual, visual, and multimodal models across harmfulness and target-identification tasks. The benchmark includes model families and dataset statistics intended to expose class and lexical patterns.
- Benchmark Setup: Benchmark evaluations cover unimodal textual, unimodal visual, and multimodal models on HarMeme.Experiments use the MMF framework except for unimodal visual models.
- Unimodal Models: Text BERT provides the text-only baseline, while VGG19, DenseNet-161, ResNet-152, and ResNeXt-101 provide pretrained visual-only baselines.Visual features were extracted from each architecture’s last pooling layer and passed to a classifier.
- Multimodal Models: Late Fusion, Concat BERT, MMBT, ViLBERT, and Visual BERT COCO represent multimodal or multimodal-fusion approaches.The listed approaches combine textual and visual information through score fusion, feature concatenation, or multimodal transformer architectures.
- Dataset Statistics: HarMeme statistics report harmfulness classes and four target categories for very harmful and partially harmful memes.The dataset statistics table covers the annotation label structure used in benchmarking.
- Dataset Statistics: Top-5 word statistics provide tf-idf scores separately by class.These lexical summaries support inspection of class-specific language patterns.
6 Experimental Results
Experiments show that multimodal models substantially outperform unimodal approaches on harmful-meme detection and target identification, while human performance remains higher and model bias can distort predictions.
- Harmful Meme Detection: 64.76% accuracy is achieved by the majority-class baseline in harmful meme detection because the test set is imbalanced.
- Harmful Meme Detection: Unimodal visual models perform only marginally above the majority baseline, indicating insufficient semantic understanding of the memes.
- Harmful Meme Detection: Multimodal performance improves noticeably, with sophisticated fusion and multimodal pre-training outperforming simpler unimodal approaches.
- Target Identification for Harmful Memes: V-BERT COCO outperforms ViLBERT CC by 8% of F1 score and nearly 0.3 of MMAE on target identification.
- Human Evaluation: 86%–91% average human accuracy on both tasks exceeds the performance of V-BERT, the best-performing model.
- Side-by-side Diagnostics and Anecdotes: V-BERT misclassifies a harmless meme as very harmful after associating Donald Trump’s image with harmful training examples.
7 Conclusion and Future Work
The paper introduces HarMeme as a COVID-19 benchmark for harmfulness and target identification, finding that multimodal modeling is important while more sophisticated methods remain needed. Future work extends the dataset and addresses dataset and model biases.
- HarMeme contains 3,544 COVID-19 memes annotated for harmfulness and the social entity targeted by the harm.
- The evaluation highlights the importance of modeling multimodal signals for both harmful-meme detection and target identification.
- The authors identify a need for more sophisticated methods and analyze limitations of the best model.
- Future work will add memes from other topics and languages while alleviating dataset and model biases.
Ethics and Broader Impact
The paper describes privacy, bias, misuse, intended-use, and environmental considerations surrounding HarMeme and its computational use.
- User Privacy: The dataset contains memes only and no user information.
- Biases: Harmfulness judgments are subjective, making bias in gold labels and label distributions inevitable.The authors use general COVID-19 keywords, an explicit annotation schema, and high inter-annotator agreement to address these concerns.
- Misuse Potential: HarMeme could be maliciously used to unfairly moderate memes based on demographic or textual biases.The authors state that human moderation is required to prevent this misuse.
- Intended Use: The dataset is distributed for research purposes only and without a commercial-use license.The authors present it as a resource for studying harmful memes when used appropriately.
- Environmental Impact: Large-scale Transformer training requires substantial GPU or TPU computation, but this work fine-tunes models on relatively small datasets.The authors note that CPU inference after fine-tuning is feasible and contributes less to global warming.
A Implementation Details and Hyper-Parameter Values
The authors train the models end-to-end for both classification tasks using PyTorch, pretrained components, random initialization where needed, and imbalance-aware loss weighting.
- Hardware and Software: The experiments use PyTorch on an NVIDIA Tesla T4 GPU with 16 GB memory, CUDA-10, and cuDNN-11.
- Model Initialization: Unimodal models import pretrained weights from PyTorch’s TORCHVISION.MODELS10 subpackage.Non-pretrained weights are initialized with a zero-mean Gaussian distribution with standard deviation 0.02.
- Class Imbalance: Larger loss weights are assigned to reduce the impact of label imbalance during training.
- Training Procedure: All models are trained end-to-end for harmful-meme detection and target identification.Memes labeled Very Harmful or Partially Harmful in the first stage proceed to target identification.
B Annotation Guidelines
The annotation guidelines define harmful memes by their potential social harm, distinguish harm from hate or offensiveness, and specify target categories and handling of ambiguous cases.
- Characteristics: Harmful memes may be offensive, hateful, or biased, but they need not be.Harmful content can expose allegations or negative aspects through verified or unfounded claims, mocks, sarcasm, or satire.
- Characteristics: Harmful content is often implicit and may require critical judgment to establish its potential harm.
- Characteristics: Memes can be classified by harm intensity as very harmful or partially harmful.
- Target Annotation: A harmful meme may target multiple individuals, organizations, or communities, so annotators use their best personal judgment.
- Target Categories: Organizations are purposeful groups, whereas communities are social units sharing norms, values, ideology, customs, identity, or place.
B.3 When do we reject a meme?
The collection process rejects memes that are non-English or unreadable, unimodal, or cartoon-based; rejected examples are shown in Figure B.1, with hyperparameters listed separately.
- Rejection Criteria: The rejection criteria exclude memes whose text is code-mixed or not in English.
- Rejection Criteria: Memes with unreadable text, including blurry or incomplete text, are rejected.
- Rejection Criteria: Unimodal memes containing only textual or only visual content are rejected.
- Rejected Examples: Figure B.1 presents examples of memes rejected during collection and annotation.
- Hyper-Parameters: Table A.1 reports the hyper-parameter values for all models.