PaperlayerView exact revision on arXiv

Source-linked AI summary

Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features

Gokul Karthik Kumar, Karthik Nandakumar

arXiv:2210.05916v3cs.CLcs.CVcs.LGcs.MM

TL;DR

Hateful meme detection is challenging because image and text can convey different meanings in context, and existing methods often use multimodal pretraining or intermediate fusion, but not both. Hate-CLIPper combines aligned CLIP features through a feature interaction matrix; it achieves state-of-the-art HMC performance in 14 epochs with four trainable layers.

  • Problem

    Hateful meme meaning can depend on subtle context across image and text, while most existing methods use multimodal pretraining or intermediate fusion, but not both.

    Source: §1 · pp. 1–2

  • Method

    Hate-CLIPper uses multimodally pretrained CLIP encoders and an outer-product feature interaction matrix to model relationships between image and text features.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    14 epochs: Hate-CLIPper achieved state-of-the-art performance on HMC with four trainable layers and demonstrated similar performance on Propaganda Memes multi-label classification.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Interpretability analysis indicates that salient FIM locations and clustered trigger vectors facilitate learning meaningful concepts.

    Source: §1 · p. 2

    For n = 1024 and m = 1024, the high-dimensional feature interaction matrix requires a model with a billion parameters, creating a computational challenge.

    Source: §7 · p. 9

Abstract

from arXiv · show

Hateful memes are a growing menace on social media. While the image and its corresponding text in a meme are related, they do not necessarily convey the same meaning when viewed individually. Hence, detecting hateful memes requires careful consideration of both visual and textual information. Multimodal pre-training can be beneficial for this task because it effectively captures the relationship between the image and the text by representing them in a similar feature space. Furthermore, it is essential to model the interactions between the image and text features through intermediate fusion. Most existing methods either employ multimodal pre-training or intermediate fusion, but not both. In this work, we propose the Hate-CLIPper architecture, which explicitly models the cross-modal interactions between the image and text representations obtained using Contrastive Language-Image Pre-training (CLIP) encoders via a feature interaction matrix (FIM). A simple classifier based on the FIM representation is able to achieve state-of-the-art performance on the Hateful Memes Challenge (HMC) dataset with an AUROC of 85.8, which even surpasses the human performance of 82.65. Experiments on other meme datasets such as Propaganda Memes and TamilMemes also demonstrate the generalizability of the proposed approach. Finally, we analyze the interpretability of the FIM representation and show that cross-modal interactions can indeed facilitate the learning of meaningful concepts. The code for this work is available at https://github.com/gokulkarthik/hateclipper.

1 Introduction

Hateful meme detection is difficult because an image or text can change meaning in the context of the other modality. The paper proposes combining multimodally aligned CLIP features with intermediate fusion that explicitly represents cross-modal interactions.

1 Introduction

  • Hateful memes can be hard to detect automatically because their meaning depends on subtle context shared between image and text.The Hateful Memes Challenge uses non-hateful confounders that change only the image or text, requiring both modalities for successful identification.Source: §1 · p. 1
  • Early fusion assumes text describes image content, while late fusion combines modality-specific outputs; neither directly models the feature relationships needed for hateful memes.The paper argues that intermediate fusion in feature space is more suitable because hateful meme text may imply something different from the image.Source: §1 · pp. 1–2
  • Independently pretrained image and text encoders may produce divergent feature spaces, motivating multimodal pretraining to align their representations.The paper also identifies the need to characterize fine-grained interactions between image and text attributes.Source: §1 · p. 2
  • HateCLIPper uses intermediate fusion of aligned image and text representations from multimodally pretrained CLIP encoders.The paper names bilinear pooling, or the outer product, as its fusion operation and calls the resulting representation the feature interaction matrix (FIM).Source: §1 · p. 2
  • The authors report that salient FIM locations and clustered trigger vectors indicate that the representation facilitates learning meaningful concepts.The interpretability analysis identifies locations that trigger the classification decision before clustering their associated vectors.Source: §1 · p. 2

2 Related Work

Related work establishes HMC benchmarks and compares unimodal, multimodal, and CLIP-based approaches. The paper positions explicit bilinear interactions between aligned features as underexplored for this task.

2 Related Work

  • Humans reached AUROC 82.65, while Visual BERT was the best baseline at AUROC 75.44 on HMC’s test-seen split.The challenge report later found top-five submissions exceeded baseline methods, primarily using ensembles and/or external data and additional features.Source: §2 · pp. 2–3
  • Zhu (2020) achieved AUROC 84.50 on HMC’s test-unseen split using an ensemble and additional entity, race, and gender information.The reported ensemble combined VL-BERT, UNITER-ITM, VILLA-ITM, and ERNIE-Vil.Source: §2 · p. 3
  • Zia et al. (2021) used CLIP features with concatenation and logistic regression, while MOMENTA added object, face, and text-attribute features and used cross-modality attention fusion.MOMENTA reported results only on HarMeme.Source: §2 · p. 3
  • Bilinear pooling has improved other multimodal tasks, but had not been well experimented with using multimodally pretrained aligned encoders such as CLIP or for hateful meme classification.The paper distinguishes this gap from prior CLIP-feature work based on simple concatenation or cross-modality attention fusion.Source: §2 · p. 3

3 Methodology

Hate-CLIPper uses pretrained CLIP encoders and trainable projection layers to align image and text features, then models their interactions for hateful-meme classification. It includes full and reduced interaction representations and a shallow classifier.

3 Methodology

  • Hate-CLIPper targets end-to-end hateful-meme classification without sophisticated ensembles or external data or labels.The model passes meme images and text through pretrained CLIP encoders, whose weights are frozen after pretraining.Source: §3 · pp. 3–4
  • Trainable projection layers further align CLIP image and text features for the task and map them to vectors with the same dimensionality.The paper favors learning these projection layers over fine-tuning the entire CLIP model on small datasets.Source: §3 · pp. 4–5
  • Cross-fusion forms a feature interaction matrix from the outer product of projected image and text features, then flattens it for classification.The resulting representation explicitly models correlations between feature dimensions, unlike concatenation, which leaves those relationships for the classifier to learn.Source: §3 · p. 5
  • Align-fusion retains only the FIM diagonal, giving an n-dimensional vector of element-wise products that represents alignment between feature dimensions.The diagonal elements sum to the dot product between the projected image and text vectors.Source: §3 · p. 5
  • A shallow fully connected network with ReLU, dropout, and softmax classifies the fusion vector; fine-grained tasks add auxiliary outputs.For binary classification, trainable projection and classifier layers minimize binary cross-entropy; fine-grained classification uses the total loss across tasks.Source: §3 · p. 5

4 Experimental Results

Experiments evaluate Hate-CLIPper on HMC, Propaganda Memes, and TamilMemes, comparing fusion approaches and reporting strong results alongside a Tamil text-alignment limitation.

4.1 Datasets

  • Evaluation covers HMC, 22-class multi-label Propaganda Memes, and TamilMemes troll/non-troll classification with English-transliterated Tamil text.Unlike HMC, TamilMemes was not compiled so that only multimodal information is useful for classification.Source: §4.1 · p. 5

4.2 Setup

  • Models use ViT-Large-Patch14 CLIP features; AUROC is used for HMC and micro F1 for Propaganda Memes and TamilMemes.Hyperparameters were manually tuned against the validation metric, and the reported HMC models took around 30 minutes for combined training and evaluation.Source: §4.2 · pp. 5–6

4.3 Key Findings

  • 85.8 AUROC for Hate-CLIPper’s intermediate fusion exceeds VisualBERT’s 75.44 and the late-fusion method’s 69.3 on HMC test data.The paper reports that all considered intermediate-fusion methods outperform the early- and late-fusion methods in the comparison.Source: Table 4 · p. 6, §4.3 · p. 6
  • Cross-fusion and align-fusion achieve the best AUROC on both HMC evaluation sets and exceed reported human performance.The pattern also holds when ViT-Large-Patch14 is replaced with ViT-Base-Patch32.Source: §4.3 · p. 6
  • Align-fusion outperforms concat-fusion despite using n rather than 2n multimodal features, and is closer to cross-fusion’s n^2 features.The authors suggest align-fusion as a replacement for cross-fusion under low computational-resource conditions.Source: §4.3 · p. 7
  • On Propaganda Memes, the proposed approach is reported as a significant leap over prior literature, while differences among its configurations are marginal.A single projection layer per modality and a shallow, 1- or 3-layer classifier are reported as sufficient for good performance.Source: §4.3 · p. 7

4.4 Multilinguality

  • 59 micro F1 on TamilMemes exceeds the reported 47 micro F1 of the ViT-and-BERT concatenation model.The paper notes that Hate-CLIPper uses the same encoder types, multimodally pretrained with CLIP loss.Source: §4.4 · p. 7
  • On TamilMemes, cross- and align-fusion match concat-fusion, which the authors attribute to misalignment with Tamil-to-English transliterated text.Replacing the CLIP text encoder with multilingual BERT also brought no further improvement; the authors connect this to the lack of multimodal pretraining on paired Tamil text and images.Source: §4.4 · p. 7

5 Interpretability

The interpretability analysis identifies salient FIM interactions, turns them into trigger vectors, and clusters memes to examine whether shared concepts emerge.

5 Interpretability

  • Trigger vectors are formed by binarizing salient gradient and meme-specific FIM elements, multiplying the matrices element-wise, and flattening the result for K-means clustering.The gradient matrix selects the top and bottom 20 percentiles by magnitude, while the meme-specific FIM selects the top and bottom 10 percentiles.Source: §5 · pp. 7, 9
  • Clusters 5, 7, and 11 contain memes related to the same concept, including death expressed in images or text within cluster 5.The authors also identify activated FIM positions for matching concepts across modalities.Source: §5 · p. 9
  • Some clusters are ambiguous or too diverse to explain: cluster 13 mixes concepts, and groups with fewer than 3 or more than 10 memes are less useful.Figure 3 shows 15 K-means clusters based on cross-fusion trigger vectors, with some meme text masked.Source: Figure 3 · p. 8, §5 · p. 9

6 Conclusion

Hate-CLIPper combines intermediate fusion with multimodally pretrained CLIP representations. The authors report state-of-the-art HMC performance in 14 epochs with four trainable layers, along with results on Propaganda Memes and preliminary interpretability studies.

6 Conclusion

  • Hate-CLIPper achieves state-of-the-art HMC performance in 14 epochs with four trainable layers.The four layers are one image projection, one text projection, one pre-output, and one output layer.Source: §6 · p. 9
  • The architecture uses explicit cross-modal CLIP representations and requires no additional features such as object boxes, face detection, or text attributes.Source: §6 · p. 9
  • The authors report similar performance for multi-label classification on Propaganda Memes and preliminary studies of cross-modal interaction interpretability.Source: §6 · p. 9

7 Limitations

The authors identify computational, language-resource, and interpretability limitations, alongside broader concerns about subjective labels and classifier accuracy.

7 Limitations

  • For n = 1024 and m = 1024, cross-fusion requires a model with a billion parameters, while align-fusion performs quite close with O(nm) parameters.Source: §7 · p. 9
  • CLIP encoders trained on massive English data are rarely available for low-resource languages, limiting direct applicability there.Source: §7 · p. 9
  • The multilingual experiment highlights issues from misaligned feature spaces, but more ablations are needed to assess whether learnable projections overcome them.Source: §7 · p. 9
  • The authors describe hate-speech boundaries as subjective and classifier accuracy as unsatisfactory for real-world deployment, even on curated benchmarks.Source: §7 · p. 9
  • Their interpretability assessment is described as simple and ad hoc; they call for systematic explainability evaluation and note that fine-grained labels were unused for FIM interpretation.Source: §7 · p. 9

A Appendix

The appendix reports that the tested variations produced unchanged or slightly worse results.

A Appendix

  • The tested variations produced the same results or slightly degraded performance.Source: A Appendix · p. 12

A.1 Variations of HateCLIPper

The appendix describes several training and architecture modifications to Hate-CLIPper and reports that they produced unchanged or slightly worse performance.

A.1 Variations of HateCLIPper

  • Caption experiments either pretrained encoders using image-caption contrastive loss or incorporated generated captions in three ways during finetuning.Source: §A.1 · p. 12
  • Unimodal classification losses and fine-grained classification losses were each jointly optimized with the original multimodal loss.Source: §A.1 · p. 12
  • Data augmentation masked meme text regions detected with EAST, using either average-pixel masks or inpainting.Source: §A.1 · p. 12
  • Figure 4 clusters hateful memes into 15 K-means groups using Hate-CLIPper cross-fusion trigger vectors.Source: Figure 4 · p. 13

Related papers

Loading 2210.05916v3…