Source-linked AI summary
Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution
Ron Zhu
TL;DR
Hateful meme detection requires understanding visual and linguistic content as well as real-world knowledge, beyond the direct image-text relations common in multimodal tasks. The paper adapts pretrained visual-linguistic transformers with external labels and model adaptations, reporting 0.845 AUROC on the hateful memes dataset versus 0.71 AUROC for state-of-the-art multimodal models. The report also notes that substantial room for improvement remains.
Problem
Method
The paper extends VL-BERT with external information and adapts pretrained visual-linguistic models, including reusing UNITER’s ITM head.
Source: §3.4 · p. 5, §6 · p. 8Results
0.845 AUROC on the hateful memes dataset, compared with 0.71 AUROC for state-of-the-art multimodal models applied to the challenge.
Source: §1 · p. 1Takeaways & Limitations
The report presents external-information integration and pretrained-model adaptation as approaches for improving hateful meme detection.
Source: §6 · pp. 8–9The authors state that substantial room remains to improve the proposed approaches and point to promising research directions.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Hateful meme detection is a new research area recently brought out that requires both visual, linguistic understanding of the meme and some background knowledge to performing well on the task. This technical report summarises the first place solution of the Hateful Meme Detection Challenge 2020, which extending state-of-the-art visual-linguistic transformers to tackle this problem. At the end of the report, we also point out the shortcomings and possible directions for improving the current methodology.
1 Introduction
Hateful meme detection requires visual and linguistic understanding plus background knowledge, yet state-of-the-art multimodal models reach only 0.71 AUROC on the challenge. The proposed approaches achieve 0.845 AUROC on the dataset.
1 Introduction
- 0.845 AUROC on the hateful memes dataset, compared with 0.71 AUROC for state-of-the-art multimodal models applied to the challenge.Source: §1 · p. 1
- The challenge dataset includes confounders intended to prevent models from exploiting unimodal priors.Source: §1 · p. 1
- The task calls for visual and linguistic understanding of memes together with background knowledge.Source: §1 · p. 1
2 Problem Description
Hateful memes differ from common visual-linguistic tasks because their meaning may depend on subtle wording, external knowledge, and the interaction of image and text. Diverse topics and difficult visual cues further complicate classification.
2 Problem Description
- Hateful meme classification is framed as a visual-linguistic-knowledge problem because hateful meaning often relies on subtle wording or real-world context.Source: §2 · p. 2
- Unlike common multimodal tasks such as VCR [16] and VQA [1], only some hateful memes directly connect image and text in the usual way.Source: §2 · p. 2
- Diverse topics and novel objects make visual cues such as injury, traditional clothing, and historical scenes difficult to recognize, even without language.Source: §2 · p. 2
- Caption placement can alter a meme’s meaning, as placing “wishing machine” over a female subject makes it hateful in the example given.Source: §2 · p. 3
3 Approach
The approach combines visual-linguistic transformer models with added information and model adaptations aimed at hateful meme detection. It uses external labels, an extended VL-BERT framework, a reused UNITER-ITM head, and ERNIE-Vil in the ensemble.
3 Approach
- The solution ensembles VL-BERT, UNITER, VILLA, and ERNIE-Vil, modifying all except ERNIE-Vil and adding information extraction for training and inference.Source: §3 · p. 3
3.1 Background
- Common visual-linguistic transformers still perform poorly on hateful memes, which require understanding images, text, real-world events, and cross-modal interaction.Source: §3.1 · p. 3
3.2 Additional Data Source
- Google Vision Web Entity Detection captures image context amid rapidly changing references; FairFace supplies race and gender labels, while entity background and relations remain unmodeled.Source: §3.2 · pp. 3–4
3.3 Extend VL-BERT Visual-Linguistic Framework
- The extended VL-BERT represents external labels as text tokens linked to image regions through visual features, enabling explicit connections between labels and image content.Source: §3.3 · p. 4
3.4 UNITER-ITM
- The method reuses UNITER’s ITM- and WRA-pretrained head, swaps match and non-match classes to benign and hateful, and expects less linguistic bias because both tasks use visual and linguistic features.Source: §3.4 · p. 5
3.5 ERNIE-Vil
- ERNIE-Vil’s scene-graph pretraining targets fine-grained object attributes and relationships; including it in the ensemble adds prediction diversity and improves generalization.Source: §3.5 · p. 6
4 Experiment
Entity and demographic tags improved extended VL-BERT on dev-seen, but their effect varied across architectures. Reusing UNITER’s ITM head improved performance and may reduce text-modality bias.
4.1 Apply entity/race/gender tags
- Entity tags boosted extended VL-BERT and ERNIE-Vil, but worsened or barely changed UNITER and Oscar results on dev-seen.Figures 4 and 5 present dev-seen AUROC for extended VL-BERT with tags and for the Hateful Memes dev-seen set, respectively.Source: §4.1 · p. 6, Figure 4 · p. 6, Figure 5 · p. 7
- The authors suggest weak explicit visual-linguistic fusion and lower linguistic-representation quality may limit entity-tag effectiveness, especially with a small dataset.They point to VL-BERT’s visual embedding for every token and ERNIE-Vil’s cross-modality layers as fusion mechanisms.Source: §4.1 · p. 6
4.2 Reuse of UNITER ITM head
- For UNITER on dev-seen AUROC, the comparison with and without reuse of the ITM head was 0.778 versus 0.765; the note also describes the model as less easy to overfit.The authors suggest it may suffer less bias toward the text modality because ITM and WRA pretraining require both modalities.Source: §4.2 · p. 7, Figure 6 · p. 7
- UNITER-ITM’s high accuracy early in training may indicate a correlation between its pretraining task and hateful-meme classification.The authors raise the question of whether other pretraining tasks share this correlation.Source: §4.2 · p. 7
5 Feature Direction
Hateful memes can rely on subtle relations among image, text, and external knowledge, motivating knowledge-graph-enhanced visual-linguistic models. Their use requires constructing relevant subgraphs from large knowledge graphs.
5 Feature Direction
- Hateful-meme meaning can depend on subtle links among visual, linguistic, and knowledge-base entities, while many pretrained models use only image and text.The passage notes that implicit knowledge in pretrained language models is hard to update and apply to a different domain.Source: §5 · p. 8
- MHGRN’s plug-and-play design and attention GNN make it a flexible option for adding knowledge graphs after different pretrained transformers and scaling to larger graphs.The proposed direction would use a sizeable graph such as Wikidata or ConceptNet and extract a relevant subgraph for each meme.Source: §5 · p. 8
- Building a useful subgraph is challenging because each meme needs relevant entities and relations from large knowledge graphs, where entities may have dozens of relation types.The subgraph must include enough information while remaining a reasonable size.Source: §5 · p. 8
6 Conclusion
The conclusion presents two approaches to improve pretrained visual-linguistic transformers for hateful-meme detection. It notes that substantial room for improvement and promising research directions remain.
6 Conclusion
- The paper proposes two methods to improve pretrained visual-linguistic transformers for hateful-meme detection.One builds a VL-BERT framework to incorporate information difficult to learn directly from the challenge dataset; the other uses the entire pretrained model for meme classification.Source: §6 · pp. 8–9
- The authors state that the approaches improve performance but leave substantial room for further improvement and promising research directions.They point to future work aimed at better model understanding of memes.Source: §6 · p. 9