Source-linked AI summary
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, Davide Testuggine
TL;DR
The paper addresses whether multimodal systems can detect subtle hate speech in memes without relying on unimodal shortcuts. It introduces a challenge dataset with benign confounders and finds that more sophisticated multimodal models perform better, yet remain far below human performance.
Problem
Existing multimodal tasks can permit strong unimodal priors, while subtle meme meanings remain challenging for AI systems and hate speech requires scalable detection.
Method
The paper constructs a challenge dataset of reconstructed memes with benign confounders that flip labels when either images or captions change, and uses trained annotators for hate-speech labeling.
Results
More advanced fusion models perform better, but trained non-expert annotators achieve 84.7% accuracy and the best multimodal models remain far behind.
Takeaways & Limitations
The challenge measures progress in multimodal understanding and reasoning while supporting progress toward automated hate-speech detection.
Takeaways & Limitations
Improved multimodal systems may contribute to job automation or be used for censorship or nefarious purposes.
Abstract
from arXiv · showhide
This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: difficult examples ("benign confounders") are added to the dataset to make it hard to rely on unimodal signals. The task requires subtle reasoning, yet is straightforward to evaluate as a binary classification problem. We provide baseline performance numbers for unimodal models, as well as for multimodal models with various degrees of sophistication. We find that state-of-the-art methods perform poorly compared to humans (64.73% vs. 84.7% accuracy), illustrating the difficulty of the task and highlighting the challenge that this important problem poses to the community.
1 Introduction
The paper introduces a multimodal hate-speech challenge designed to require genuine image-text reasoning rather than unimodal shortcuts. Benign confounders make subtle meme meanings difficult for AI systems while supporting straightforward evaluation and real-world relevance.
- Motivation: The challenge targets hate speech detection in multimodal memes, where harmless text and images can combine into a mean message.Memes therefore require interpreting the interaction between modalities, not merely classifying either modality independently.
- Challenge design: Benign confounders flip a hateful meme’s label by replacing its image or caption, countering models that exploit unimodal priors.Correctly classifying originals and both confounder types is intended to require sophisticated multimodal reasoning.
- Significance: The challenge addresses a real-world hate-speech problem while measuring progress in multimodal understanding and reasoning.The authors contrast this dual purpose with tasks having limited or indirect real-world use cases.
- Results: Baseline performance follows a hierarchy of multimodal sophistication, while the best-performing model remains far below humans.The authors present this gap as evidence that the challenge can measure progress in multimodal research.
2 The Hateful Memes Challenge Set
The challenge set is constructed by reconstructing licensed memes, annotating hatefulness, and adding contrastive benign confounders. Its 10k examples span multimodal and unimodal hate, confounders, and random benign memes.
- Dataset purpose: The dataset is a challenge set for fine-tuning and testing pretrained large-scale multimodal models, rather than training models from scratch.The construction uses third-party annotators trained on the paper’s hate-speech definition.
- Hatefulness definition: Hate speech is defined as direct or indirect violent, dehumanizing, inferiorizing, exclusionary, or segregationist attacks based on specified protected characteristics.Mocking hate crimes is also included, while attacks unrelated to protected characteristics and attacks on groups perpetrating hate are excluded.
- Meme reconstruction: The construction replaces source-meme images with licensed Getty Images while preserving the original text’s meaning and intent.Reconstruction also avoids releasing original memes with unknown creators and reduces visual and OCR-related biases.
- Annotation process: The annotation pipeline filters source memes, reconstructs them, obtains multiple hatefulness ratings, and creates confounders for hateful examples when possible.The source process begins with more than 1 million images, filters to 162k memes, and uses five annotations per meme after reconstruction.
- Benign confounders: A benign confounder is a minimum image or text replacement that flips a multimodal meme from hateful to non-hateful.This design makes relying on only text or only image information insufficient for solving the task.
- Dataset composition: The final dataset contains exactly 10k memes across multimodal hate, unimodal hate, benign confounders, and random not-hateful examples.Confounder construction also identifies memes whose hatefulness is attributable to one modality, with most unimodal-hate cases found to be text-only.
3 Dataset Analysis
The dataset analysis examines annotation agreement and dataset characteristics, including hate categories, attack types, and lexical statistics. Annotators showed moderate agreement, underscoring the difficulty of identifying hate speech.
- 3.2 Dataset Properties: The section analyzes the distribution of hate categories and attack types in the dataset, which is not necessarily representative.
- 3.1 Inter-Annotator Agreement: Cohen’s kappa was 68.4, indicating moderate agreement among the three annotators who rated hatefulness.The authors interpret this agreement level as evidence that determining whether content constitutes hate speech is difficult.
- 3.1 Inter-Annotator Agreement: The hate speech definition includes exceptions, such as distinguishing attacks on terrorist organizations from attacks equating those organizations with racial or ethnic groups.
- 3.2 Dataset Properties: The dataset analysis also includes lexical statistics covering both textual and visual modalities.
4 Benchmarking Multimodal Classification
The benchmark compares unimodal, unimodally pretrained multimodal, and multimodally pretrained models. More advanced fusion performs better, but the best multimodal models remain far below trained human annotators.
- 4.1 Models: The benchmark evaluates unimodal models, multimodal models using unimodally pretrained components, and models with multimodal pretraining.
- 4.1 Models: The evaluation includes image encoders based on ResNet-152 and Faster-RCNN features, alongside BERT for text.
- 4.1 Models: The compared methods range from late and concatenation-based fusion to early-fusion MMBT, ViLBERT, Visual BERT, and multimodally pretrained variants.
- 4.2 Results: 50 AUROC is achieved by random and majority-class baselines on the balanced test set, while visual models only marginally outperform them and text-only classification performs slightly better than vision-only classification.
- 4.2 Results: More advanced fusion models outperform middle- and late-fusion approaches, forming a performance hierarchy tied to multimodal sophistication.
- 4.2 Results: The difference between unimodally and multimodally pretrained models is relatively small, suggesting that multimodal pretraining can be improved further.
- 4.2 Results: 84.7% accuracy is achieved by trained non-expert annotators, leaving the best multimodal models far behind human performance.
5 Related Work
Related work spans text-based and multimodal hate-speech detection, broader vision-and-language tasks, and other multimodal classification settings. The paper positions its dataset as larger, deliberately difficult for unimodal systems, and focused specifically on hate speech.
- Hate Speech: Prior hate-speech research has primarily used text-only datasets, especially from Twitter, while also documenting classification difficulty and unwanted bias.
- Multimodal Hate Speech: Existing multimodal hate-speech studies report improved classification when image features augment text, but the area has relatively little prior work.
- Multimodal Hate Speech: The paper differs from prior work through a larger dataset designed to challenge unimodal architectures, high-confidence trained annotations, balanced fusion-problem types, and a specific focus on hate speech rather than cyberbullying.
- Vision and Language Tasks: Vision-and-language research includes tasks such as visual question answering, image captioning, visual reasoning, referring expressions, storytelling, and dialogue.
- Vision and Language Tasks: Many vision-and-language tasks emphasize text generation or retrieval, often have one dominant modality, and can be difficult to evaluate compared with real-world multimodal classification.
- Related Multimodal Classification: Other multimodal classification areas include sentiment, food and product classification, and social-media tagging, but lack an agreed standard benchmark in some cases.
6 Conclusion
The paper introduces a challenge dataset and benchmark for hate-speech detection in multimodal memes. Benign confounders make unimodal solutions difficult, while advanced fusion models still trail human performance, establishing a demanding benchmark for multimodal reasoning.
- 6 Conclusion: The work introduces a challenge dataset and benchmark centered on detecting hate speech in multimodal memes.
- 6 Conclusion: Benign confounders flip hateful meme labels and make the task difficult for unimodal models.
- 6 Conclusion: More advanced fusion models perform better, revealing a hierarchy in multimodal sophistication.
- 6 Conclusion: State-of-the-art multimodal models perform relatively poorly and remain far from human performance, highlighting the benchmark’s challenge.
- 6 Conclusion: The authors intend the benchmark to support progress in multimodal reasoning and understanding and in hate-speech detection as a real-world problem.
Broader Impact
The work highlights benefits from using automatic methods to counter hate speech and advance multimodal reasoning, while acknowledging risks including job automation, censorship, and misuse.
- Automatic hate-speech mitigation could have a substantial impact on people’s lives.
- The challenge is intended to spur innovation in multimodal reasoning and understanding across many tasks and applications.
- Potential downsides include job automation, censorship, and nefarious uses of improved multimodal systems.
A Implementation Details & Hyperparameters
The experiments use model-specific hyperparameters selected through grid search and development-set AUROC evaluation, with configurations and reproduction materials documented.
- Hyperparameter selection: Grid search varied batch size, learning rate, and maximum updates to select experimental hyperparameters.
- Hyperparameter selection: 22000 updates was the best-performing setting across the evaluated configurations.
- Evaluation: Models were evaluated every 500 updates on the dev set, and the configuration with best AUROC was evaluated on the test set.
- Model configurations: ViLBERT used 6 visual and 12 linguistic TRM blocks with Conceptual Captions pretrained weights.
- Model configurations: VisualBERT used 12 TRM blocks with COCO pretrained weights, hidden sizes of 768 for text and 1024 for vision, and dropout of 0.1.
B Textual and Visual Lexical Statistics
Lexical analyses examine text length, frequent words, and visual labels across dataset categories. Several frequent words are shared with benign confounders, while visual properties appear relatively balanced.
- Text length: Figure 4 presents normalized text-length histograms by meme type and binary class.
- Textual statistics: The textual analysis reports token-count distributions by hatefulness label and category, plus the most frequent words in the test set.
- Textual statistics: Words such as “dishwashers,” “sandwich makers,” “black,” and “white” are frequent but also occur in benign image-confounder examples.
- Textual statistics: Unimodal-hate examples generally use stronger language and often target religious groups.
- Visual statistics: The visual analysis uses Mask R-CNN bounding-box labels and finds the dataset relatively evenly balanced across properties and objects.
C.1 Hate Categories
The dataset analyzes hate speech across protected categories, with race- and religion-based hate most prevalent, while noting that the category set is not exhaustive.
- Hate speech is analyzed through attacks on protected categories such as gender and nationality.
- One meme can attack multiple protected categories, and race- and religion-based hate are the most prevalent categories observed.
- The nine identified protected categories are useful for describing dataset properties but are neither complete nor exhaustive.
C.2 Types of Attack
The paper categorizes hateful-meme attacks across protected categories, with several recurring attack types appearing broadly across the dev set. Dehumanization is one major category, while criminal comparisons, negative stereotypes, and mocked hate crimes are especially frequent.
- Dehumanization compares protected groups with non-human things such as animals or objects.
- 47.1% of annotations concern race or ethnicity, and 39.3% concern religion.
- Attack types are exclusive per meme, while multiple protected-category labels can apply to one meme.
- Criminal comparisons, negative stereotypes, and mocked hate crimes are the most frequent attack types.Examples include comparisons with terrorists, claims that demographics are sexually attracted to children or animals, and Holocaust mockery.