PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive Learning

Yiting Qu, Xinlei He, Shannon Pierson, Michael Backes, Yang Zhang, Savvas Zannettou

arXiv:2212.06573v2cs.SIcs.CRcs.CYcs.LG

TL;DR

Hateful memes can affect people’s online experience and potentially contribute to radicalization or real-world hate crimes, making their detection and moderation pressing. The paper uses CLIP-based semantic relationships to analyze hateful-meme evolution and reports identifying 3.3K Happy Merchant items in 4chan data.

  • Problem

    Detecting and moderating hateful memes is pressing because they can affect online experiences and potentially lead to online radicalization or real-world hate crimes.

    Source: §1 · p. 1

  • Method

    The framework uses CLIP embeddings, clustering, and cluster annotation to identify hateful-content targets and analyze meme evolution through semantic regularities.

    Source: §1 · p. 2

  • Results

    3.3K Happy Merchant items were identified in 4chan data as part of a systematic analysis of hateful-meme evolution.

    Source: §7 · p. 13

  • Takeaways & Limitations

    The authors argue the framework could help moderators detect and moderate emerging hateful variants with increased coverage.

    Source: §7 · p. 14

    Threshold choices trade off false positives against missed variants, while reviewing the top 10 images raises influencer-identification to 61%.

    Source: §5.1 · p. 10

Abstract

from arXiv · show

The dissemination of hateful memes online has adverse effects on social media platforms and the real world. Detecting hateful memes is challenging, one of the reasons being the evolutionary nature of memes; new hateful memes can emerge by fusing hateful connotations with other cultural ideas or symbols. In this paper, we propose a framework that leverages multimodal contrastive learning models, in particular OpenAI's CLIP, to identify targets of hateful content and systematically investigate the evolution of hateful memes. We find that semantic regularities exist in CLIP-generated embeddings that describe semantic relationships within the same modality (images) or across modalities (images and text). Leveraging this property, we study how hateful memes are created by combining visual elements from multiple images or fusing textual information with a hateful image. We demonstrate the capabilities of our framework for analyzing the evolution of hateful memes by focusing on antisemitic memes, particularly the Happy Merchant meme. Using our framework on a dataset extracted from 4chan, we find 3.3K variants of the Happy Merchant meme, with some linked to specific countries, persons, or organizations. We envision that our framework can be used to aid human moderators by flagging new variants of hateful memes so that moderators can manually verify them and mitigate the problem of hateful content online.

1 Introduction

Hateful memes combine visual and textual meaning and continually evolve, making them difficult to detect; new variants may evade existing detection mechanisms. The paper uses CLIP to identify hateful targets and analyze meme evolution, with a focus on Happy Merchant variants.

1 Introduction

  • Hateful memes can affect online experience and potentially contribute to online radicalization or real-world hate crimes, motivating detection and moderation.The paper notes that memes can also be used for malicious purposes such as coordinated hate campaigns.Source: §1 · p. 1
  • New hateful memes can fuse other memes or cultural ideas, and substantially different images may evade hash-based detection despite sharing hateful connotations.The authors argue for tools that identify variants and the themes or symbols behind their creation.Source: §1 · p. 1
  • The framework uses CLIP embeddings and semantic regularities to study hateful targets and meme evolution within images and across images and text.The authors use clustering and automatic cluster annotation to identify potentially hateful groups, and apply the regularities to variants and their influencers.Source: §1 · p. 2
  • The paper presents an automated, scalable method for identifying meme variants and potential influencers, and makes the framework publicly available.The discovered Happy Merchant variants are available upon request for research purposes to reduce the risk of malicious use.Source: §1 · pp. 2–3
  • The authors report passive analysis of public 4chan data, aggregate reporting, privacy protection, and no cross-site user tracking.Manual annotations were performed by the authors, avoiding third-party exposure to hateful memes.Source: §1 · pp. 2–3

2 Background

The study uses 4chan’s /pol/ board as its dataset setting and describes CLIP as a model that connects image and text representations. The dataset combines images shared on /pol/ with the text of their posts.

2 Background

  • The dataset contains 4.3 million images shared on /pol/ from June 30, 2016, to July 31, 2017, paired with post text to yield 12.5 million image-text pairs.The images come from the dataset collected by Zannettou et al. [64], complemented with post text from Papasavva et al. [50].Source: §2 · p. 3
  • The authors chose /pol/ because it is associated with hateful memes and prior online-hate research, while expecting the framework to apply beyond this dataset.They describe /pol/ as a board focused on world events and politics.Source: §2 · p. 3
  • A Figure 2 example uses “A Happy Merchant Meme” as a text query to retrieve an image and then retrieve related text.The figure caption identifies the query as the name of the antisemitic signal.Source: Figure 2 · p. 3
  • CLIP jointly trains image and text encoders on image-text pairs, learning connected representations that support zero-shot transfer learning.The described training dataset contains 400 million image-text pairs collected from publicly available online sources.Source: §2 · p. 3

3 Using CLIP for Online Hate

The paper fine-tunes CLIP on 4chan data and uses its image and text embeddings for retrieval and semantic operations. These operations cover image-to-image and image-to-text relationships relevant to meme evolution.

3.1 Fine-tuning CLIP

  • Fine-tuning the entire CLIP model on 4chan data is intended to better connect platform-specific slang, memes, and related images.The authors train on 10.4 million image-text pairs and test on 2.1 million; they report better recognition of matching image-text pairs than with pre-trained CLIP.Source: §3.1 · pp. 3–4

3.2 CLIP’s Versatile Applications

  • CLIP retrieval uses image and text embeddings to find similar items, and the paper observes semantic regularities that transfer relationships within and across modalities.The authors describe using CLIP for image or text search and present these regularities as a basis for studying online hate.Source: §3.2 · p. 4
  • Visual semantic regularities combine image embeddings, such as Trump and Happy Merchant, to retrieve an image reflecting features from both inputs.The example weights the two embeddings equally and also describes subtraction as another operation.Source: §3.2 · p. 4
  • Visual-linguistic regularities combine image and text embeddings; the example retrieves a dataset image with both frog-like visual features and the text concept “Nazi.”The authors use weights of 0.2 for the image and 0.8 for the text, selected through manual examinations, and retrieve rather than generate images.Source: §3.2 · p. 4
  • The formalization treats a weighted sum of two embeddings as approximately matching a third embedding that preserves semantics from both inputs.The authors generalize this formulation to image and text embeddings to study hateful-meme evolution.Source: §3.2 · p. 5

4 Understanding Hateful Meme Clusters

Single-modality clustering can miss how meme images and comments relate, while fused embeddings combine their semantics to identify and interpret hateful meme clusters. The section builds on 1 million sampled image-text pairs and proceeds from clustering to annotation and hate assessment.

4.1 Clustering

  • The method samples 1 million image-text pairs and sums image and text embeddings before clustering, using DBSCAN to form clusters and identify outliers.The sampled pairs come from 12.5 million in the dataset; sampling reduces computation time, and DBSCAN’s parameters are tuned using manual cluster-quality and noise assessments.Source: §4 · p. 5, §4.1 · p. 5
  • Image, text, and fused embeddings reveal different cluster patterns; fusion groups visually different images when their semantics are related in context.In the Hillary example, fused clustering captures semantically related images that image-only clustering misses, while text-only clusters may contain irrelevant images.Source: §4.1 · p. 5, Figure 4 · p. 6

4.2 Automatic Cluster Annotation

  • For each sufficiently large cluster, the annotation pipeline retrieves similar text using its centroid and extracts key phrases, then manually evaluates sampled annotations.The pipeline discards clusters under 30 samples, retrieves 300 textual posts per remaining cluster, and manually evaluates 50 clusters per clustering type.Source: §4.2 · p. 6
  • Fused clustering places nazi-pepe with Hitler-related themes and Justin Trudeau near Canadian themes, unlike image-based clustering’s visually driven placements.Figure 5 projects top-100 cluster centroids; its nodes represent clusters and are sized by sample count, with commonalities and disparities marked separately.Source: §4.2 · pp. 6–7, Figure 5 · p. 7

4.3 Hate Analysis

  • Jewish-targeting memes are the most prevalent hate target, followed by findings of hate targeting Africans, immigrants and refugees, Muslims, Chinese, and Australians.The reported findings describe antisemitism-related communities as numerous and high-scoring; the communities table ranks 35 communities by hate score.Source: Table 2 · p. 8, §4.3 · pp. 8–9
  • Hate scores count posts flagged by either of two APIs, while a cosine-similarity graph and Louvain detection group fused clusters into 35 communities.The study defines hate as identity attack, excluding abusive language directed at a person without an identity-based target; community modularity averages 0.39.Source: §4.3 · pp. 7–8

5 Hateful Memes Evolution

Using all 12.5M image-text pairs from 4chan, the analysis treats a variant as a hateful signal combined with another element and an influencer as the element used. It applies image-based and image-text semantic regularities to examine Happy Merchant variants and their spread.

5 Hateful Memes Evolution

  • The study examines hateful signals combined with other elements, calling the resulting image a variant and the combining element an influencer.It analyzes the Happy Merchant globally through image influencers and directionally through preselected textual influencers, and also reports a Pepe the Frog analysis in the appendix.Source: §5 · p. 9

5.1 Visual Semantic Regularities

  • The image-based pipeline searches for variants and ranks potential image influencers using embedding similarities, with thresholds selected manually for the use case.The Happy Merchant thresholds were tested manually; the authors report that finding suitable thresholds took approximately 1–2 hours and that thresholds differ by original meme.Source: §5.1 · p. 9
  • 3,321 variant–top-1 influencer pairs were identified; manual review found 78% variant and 53% influencer identification, rising to 61% for top-10 influencers.The reviewers also marked 22% of identified variants as false positives.Source: §5.1 · pp. 9–10
  • A graph of 5,279 images and 6,656 edges showed communities with shared influencers, including variants combined with real people, classic memes, and multiple elements.Examples include Trump, Feels Guy, Pepe the Frog, and combinations of Pepe with Nazi ideology or Trump with the Reddit Meme.Source: §5.1 · pp. 10–11

5.2 Visual-linguistic Semantic Regularities

  • The directed pipeline combines the hateful image embedding with embeddings of named entities from four categories, then retrieves similar images as candidate variants.The study selected the top 30 entities from People, GPE, NORP, and ORG and used the more popular of the top two retrieved images for each entity.Source: §5.2 · p. 11
  • 75 of 116 directed candidates were successfully fused with Happy Merchant and entity semantics after four noisy entities were removed.Corresponding variants were found for 48.3% of People, 76.7% of GPE, 80.0% of NORP, and 44.4% of ORG entities.Source: §5.2 · p. 11
  • Weekly variant counts showed differing temporal patterns: Hillary Clinton remained popular throughout the study, while Justin Trudeau peaked in October 2016 and Donald Trump in April 2017.The analysis used perceptual hashing to group image duplicates and covered June 30, 2016, to June 31, 2017.Source: §5.2 · p. 12

6 Related Work

Prior work studies meme diffusion, multimodal representations, and meme families, while hateful-content research addresses speech and multimodal detection. The section situates this paper’s approach among these strands and contrasts earlier methods for tracking meme evolution.

Understanding and annotating memes

  • Prior meme-understanding work covers diffusion and popularity, multimodal embeddings for template memes, and graph learning to identify meme families.The paper contrasts its multimodal semantics and automatic image-text annotation with earlier reliance on meme websites for labels.Source: §6 · p. 12

Hateful content detection

  • Hateful-content research includes NLP studies of hateful communities and language, plus multimodal frameworks developed for the Hateful Meme Challenge.Identity attack is described as under-explored, and some work uses CLIP to retrieve memes from hateful textual phrases [37].Source: §6 · p. 12

Meme evolution

  • Prior meme-evolution studies use temporal patterns, popularity assessments, perceptual hashing, or visual-textual feature clustering to trace variations.The paper describes these approaches as focused on how memes evolve and mutate over time [33] [64].Source: §6 · pp. 12–13

7 Discussion & Conclusion

The paper applies CLIP to 4chan’s /pol/ to identify hateful targets and analyze Happy Merchant variants, reporting 3.3K variants and category-specific variant rates. It discusses moderator workflows and notes that the case study centers on one meme and platform, while false positives require human review.

7 Discussion & Conclusion

  • 3.3K Happy Merchant variants were identified on 4chan’s /pol/, with rates of 80.0% for nationalities, religious or political entities, 76.7% for countries, 44.4% for organizations, and 48.3% for people.The framework uses multimodal contrastive learning models such as CLIP to examine hateful-meme generation and evolution.Source: §7 · p. 13
  • Existing harmful-content image databases and hashes do not generalize beyond stored instances or encode image semantics, leaving some harmful content undetected or reported late.The discussion presents the multimodal framework as a way to identify variants beyond known instances.Source: §7 · p. 13
  • The authors propose flagging possible hateful variants at scale for human review, then using existing hashing techniques for confirmed variants.They argue this workflow could increase coverage for emerging variants; the framework is also proposed for detecting new targets and temporal spikes in coordinated campaigns.Source: §7 · pp. 13–14
  • The study primarily demonstrates the framework on Happy Merchant and 4chan’s /pol/, and identified variants and influencers include false positives that make human involvement important.The authors also report an analysis of Pepe the Frog as evidence of generalization beyond the main case study.Source: §7 · p. 14

A.1 Fine-tuning CLIP

The authors fine-tune CLIP on the 4chan dataset and compare pre-trained and fine-tuned models using image-to-text recommendation accuracy. Fine-tuning improves accuracy on both sampled training and testing data across the reported top-k range.

A.1 Fine-tuning CLIP

  • A learning rate of 1e-6 and 60,000 iterations, approximately three epochs, were used for fine-tuning CLIP on the 4chan data.The original learning rate of 5e-4 caused gradient explosion during fine-tuning on this dataset; training stopped after loss convergence.Source: §A.1 · p. 16
  • The evaluation sampled 10,000 image-text pairs from each of the training and testing data and counted a recommendation as successful when the original text appeared in an image’s top-k results.The authors use the difference between seen and unseen data to indicate model generalizability.Source: §A.1 · p. 16
  • 0.03–0.07 improvement in accuracy was reported for fine-tuned CLIP over pre-trained CLIP on both training and testing data as top-k increased from 50 to 500.The authors interpret the small training-testing evaluation difference as evidence of strong generalization ability.Source: §A.1 · p. 16

A.2 Probing CLIP’s Semantic Regularities

The section tests whether CLIP’s visual semantic regularities arise from its image encoder architecture or its image-text contrastive training. CLIP-based models show more successful visual semantic capture, while fine-tuning improves sentence recommendations at higher recommendation counts.

A.2 Probing CLIP’s Semantic Regularities

  • Fine-tuned CLIP achieved higher accuracy than pre-trained CLIP when recommending more sentences per image, with trivial training–testing differences.Figure 11 reports the comparison on both training and testing data.Source: Figure 11 · p. 17
  • The controlled experiment compared CLIP-ViT and CLIP-ResNet fine-tuned with image-text contrastive learning against ResNet and ViT trained from scratch with image-image contrastive learning.The latter models used SimCLR because supervised labels were unavailable.Source: §A.2 · pp. 16–17
  • CLIP-based models captured visual semantic regularities in a larger percentage of variants, suggesting image-text contrastive training contributes more than architecture design.The authors conjecture that connecting images and texts leads the image encoder to learn the inherent semantic topology of texts.Source: §A.2 · p. 17

A.3 More Case Study Results

The authors apply visual and visual-linguistic semantic-regularity pipelines to Pepe the Frog variants as a generalizability case study. The results describe variant and influencer identification, entity-linked variants, and temporal patterns in sharing.

A.3 More Case Study Results

  • Pepe the Frog was selected as a popular 4chan meme, and both the visual and visual-linguistic pipelines were applied to identify its variants.The paper notes that the Anti-Defamation League includes Pepe the Frog among hate symbols.Source: §A.3 · p. 17

Visual semantic regularities

  • Among 100 annotated pairs from 6,357 variant–influencer pairs, 94% of variants and 37% of influencers were correctly identified.Variant identification was 16% higher and influencer identification 16% lower than for Happy Merchant; Figure 12 shows the top-20 communities, with a variant and potential influencer for each.Source: §A.3 · p. 17, Figure 12 · p. 18
  • Checking all top-10 similar images increased overall influencer identification to 47%, a 10% performance increase.The authors manually checked the top-10 images for each case.Source: §A.3 · p. 17

Visual-linguistic semantic regularities

  • Merged variants were found for 37.9% of People entities, 83.3% of GPE, 80.0% of NORP, and 63.0% of ORG entities.Figure 13 displays ten variant examples for each entity category.Source: §A.3 · p. 17, Figure 13 · p. 19
  • For both Pepe the Frog and Happy Merchant, GPE and NORP entities had the greatest possibilities of merged variants; Pepe variants linked to the same entity could be more diverse.The authors give Trump- and Donald-influenced Pepe variants as an example of that diversity.Source: §A.3 · p. 17

Temporal analysis

  • Both memes had MAGA variants shared in September 2016 and Mexican or Mexico variants appearing in November 2016, while Pepe-Donald was more popular than Pepe-Hillary.Figure 14 plots weekly posts containing Pepe variants; this politician popularity pattern contrasts with Happy Merchant.Source: §A.3 · p. 18, Figure 14 · p. 19
  • The authors suggest persistent sharing of Canada-linked Pepe variants may indicate dissemination of anti-Canadian sentiments among 4chan users.The observation concerns GPE variants in the temporal analysis.Source: §A.3 · p. 18
Loading 2212.06573v2…