Source-linked AI summary
SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!
Chhavi Sharma, Deepesh Bhageria, William Scott, Srinivas PYKL, Amitava Das, Tanmoy Chakraborty, Viswanath Pulabaigari, Bjorn Gamback
TL;DR
The paper addresses limited attention to automatic emotion analysis for increasingly prevalent multimodal Internet memes. It releases human-annotated memes and defines three classification tasks spanning sentiment, emotion type, and intensity, while examining multimodal modeling and reporting benchmark participation and results. The task highlights conflicting evidence about modality importance and identifies scarce annotated data and English-only coverage as boundaries.
Problem
Automatic meme emotion analysis remains limited, while memes combine visual and textual content and offensive meme detection is more complex than offensive-text detection.
Method
The task releases approximately 10K human-annotated memes and defines sentiment, multi-label emotion, and emotion-intensity classification, with a baseline combining image and text predictions.
Results
The challenge involved 583 participants, and analyses reported conflicting views about whether image, text, or their combination contributes most to meme-emotion modeling.
Takeaways & Limitations
Memotion establishes a benchmark for multimodal meme-emotion analysis while motivating further investigation of how image and text should be modeled together.
Takeaways & Limitations
The released dataset is limited to memes with English text, and properly annotated meme data remains scarce.
Abstract
from arXiv · showhide
Information on social media comprises of various modalities such as textual, visual and audio. NLP and Computer Vision communities often leverage only one prominent modality in isolation to study social media. However, the computational processing of Internet memes needs a hybrid approach. The growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twiter further suggests that we can not ignore such multimodal content anymore. To the best of our knowledge, there is not much attention towards meme emotion analysis. The objective of this proposal is to bring the attention of the research community towards the automatic processing of Internet memes. The task Memotion analysis released approx 10K annotated memes, with human-annotated labels namely sentiment (positive, negative, neutral), type of emotion (sarcastic, funny, offensive, motivation) and their corresponding intensity. The challenge consisted of three subtasks: sentiment (positive, negative, and neutral) analysis of memes, overall emotion (humour, sarcasm, offensive, and motivational) classification of memes, and classifying intensity of meme emotion. The best performances achieved were F1 (macro average) scores of 0.35, 0.51 and 0.32, respectively for each of the three subtasks.
1 Introduction
Internet memes are increasingly central multimodal social-media content, but their emotion and offense require more than text-only analysis. The task addresses this gap by focusing on automated meme processing.
- Memes have become widespread social-media artifacts that reflect and participate in community opinions and social discourse.
- Offensive meme detection is harder than offensive-text detection because it requires interpreting both visual cues and language.The growing volume of multimodal content also makes human-only moderation difficult to scale.
- Prior work addressed meme generation and sentiment extraction, but finer distinctions such as humor and offense remained underexplored.
2 The Memotion Analysis Task
The Memotion Analysis task targets multimodal understanding of Internet memes, where text and images can jointly determine meaning. It defines sentiment, humor-type, and intensity classification tasks over human-annotated memes.
- Task A: Sentiment Classification: Task A classifies an Internet meme as positive, negative, or neutral.
- Task B: Humor Classification: Task B identifies sarcastic, humorous, motivational, and offensive emotions, allowing a meme to receive multiple categories.The example in Figure 3 is both offensive and sarcastic.
- Task C: Scales of Semantic Classes: Task C quantifies the extent to which a particular semantic effect is expressed.The task refers to Table 1 for the relevant quantification scheme.
- Dataset: 10K human-annotated memes are labeled for sentiment, humor type, and corresponding intensity, with meme captions or texts also extracted.The humor-related labels include sarcastic, humorous, offensive, and motivational categories, quantified on a Likert scale.
- Motivation: The task addresses meme interpretation beyond traditional OCR, which recognizes characters but does not understand the associated image context.Meme meaning can depend on the interaction between textual and visual information.
3 Dataset
The dataset was built from categorized, English-language memes, then crowd-annotated for emotion, intensity, and sentiment. After filtering and adjudication, the resulting collection contains 9,871 samples with overlapping emotion labels.
- Data collection and filtering: The dataset began with memes from 52 globally popular categories, filtered to clear background images with embedded English text.Public-domain memes and their URLs were retained as metadata.
- Annotation: 14K samples were annotated by Amazon Mechanical Turk workers for humor, sarcasm, offense, motivation, intensity, and overall sentiment.Sentiment annotations used five levels from very negative to very positive.
- Annotation: Each sample was given to 5 annotators, with final labels adjudicated by majority vote to address subjective perception.The filtered and annotated dataset comprises 9,871 samples.
- Data distribution: The dataset shows substantial emotion overlap: sarcasm is predominant, funny memes are often sarcastic, and some memes are both motivational and offensive.Category distributions are summarized in Tables 2 and 3.
4 Baseline Model
The baseline evaluates text-only, image-only, and combined multimodal approaches for meme emotion and intensity prediction. Its multimodal classifier fuses image and text probabilities, while intensity prediction uses a two-level architecture.
- Multimodal motivation: Meme modeling treats graphical and textual content as complementary modalities whose combination can convey complex ideas more effectively.The baseline therefore examines each modality individually and their combination.
- Using only the textual features: The text baseline feeds 100-D pretrained GloVe embeddings through CNN, max pooling, LSTM, and a sigmoid output for binary emotion classification.Binary cross-entropy is used as the loss function.
- Using only the visual features: The image baseline resizes each image to 224×224×3 and extracts features with VGG-16, selected over VGG-16, ResNet-50, and AlexNet alternatives.The extracted feature vector is flattened before classification.
- Visuo-Lingual modeling for Memotion: The multimodal classifier combines image- and text-model probabilities with weighted averaging and thresholds the result to produce a 1×4 multilabel output.The four outputs correspond to sarcastic, humorous, offensive, and motivational emotions.
- Predicting the Intensity of memes: The intensity model first predicts four emotion classes, then activates class-specific classifiers that produce a final 1×14 intensity vector.The 14 outputs comprise four levels each for humor, sarcasm, and offense, plus two for motivation.
5 Evaluation Metric
The challenge uses macro F1 because its sentiment and emotion tasks include multiple classes or labels. This metric supports analysis of individual class performance rather than relying on aggregate label error.
- Task structure: Task A is multiclass sentiment classification, whereas Tasks B and C are multilabel emotion-classification problems.The sentiment categories are positive, negative, and neutral.
- Metric choice: Macro F1 is used because it helps evaluate and analyze performance for each emotion class.The paper contrasts it with hamming loss, exact match ratio, and micro F1.
- Metric choice: Hamming loss measures the fraction of incorrectly classified labels among all labels, but it is not the metric selected for this challenge.The metric discussion also lists exact match ratio and macro/micro F1.
6 Participation and top performing systems
Memotion attracted broad participation across its three tasks, and submitted systems explored diverse textual, visual, and multimodal architectures. Approaches ranged from simple classifiers to transformer ensembles and independently trained modality models.
- Participation: 583 participants submitted systems, with 31, 26, and 23 submissions for Tasks A, B, and C, respectively.Twenty-seven teams also submitted system description papers.
- Top performing systems: IITK Vkeswani compared simple classifiers including FFNN and Naive Bayes with MMBT and BERT, using text alone or image-text combinations.The systems therefore covered both unimodal and multimodal inputs.
- Top performing systems: Guoym ensembled Bi-GRU, BERT, and ELMo textual features with ResNet50 image features and text-image fusion features.The system combined several pretrained or neural feature sources.
- Top performing systems: Aihaihara concatenated visual and textual features extracted with VGG-16 and an n-gram language model.The approach used direct feature concatenation for multimodal representation.
- Top performing systems: UPB George used ALBERT for text and VGG-16 for images, concatenating their features before softmax humor classification.The model specifically targeted humor associated with memes.
- Top performing systems: Souvik Mishra Kraken applied transfer learning with a hybrid neural Naive-Bayes SVM and logistic-regression approach to humor classification.The reported approach was designed for the humor task and its score.
- Top performing systems: Hitachi independently trained four visual and four textual deep-learning models, then fused their predictions through a modality ensemble.The visual models included Inception-ResNet, Polynet, SENet, and PNASNet; textual models included BERT, GPT-2, Transformer-XL, and XLNet.
7 Results, Analysis, and Takeaway points from Memotion 1.0
Memotion 1.0 compared unimodal and multimodal approaches for meme emotion analysis, finding that participant systems commonly fused image and text, while reported evidence on the best modality combination remained mixed.
- Unimodal vs Multi-modal: Text-only systems used Logistic Regression, BiLSTM with attention, BERT, fastText embeddings, convolution layers, and attention-based LSTMs for humour and intensity prediction.Their results were significant but lower than those of models combining image and text.
- Multimodal approaches: Multimodal systems fused visual and textual features using ensembles, early or late fusion, transfer learning, and pretrained models such as BERT, VGG16, and ResNet.Participants used diverse visual and textual feature extractors and fusion strategies.
- Special mentions: Li Zhen hit-mitlab addressed imbalance and noise with RandAugment and Training Signal Annealing while using ResNet-101 and BERT for image and text features.The system also extracted image features using textual information and concatenated them.
- Special mentions: One system concluded that image and text were more complementary than correlated for sentiment analysis and that either modality alone outperformed their combination.It used fusion with random forest, KNN, and MLP models.
8 Related Work
The related work separates meme understanding into recognizing embedded text and analyzing meme content, reflecting the need to process both textual and visual information.
- Related work: Meme-related prior work is organized into text extraction models and methods for analyzing memes.The paper treats identifying image text as important as understanding image context.
- Text recognition: Earlier studies used CNNs, n-gram correction, literature surveys, and contour-based template matching for character and handwriting recognition.These approaches address recognition and correction of text extracted from images.
- Participating systems: Table 8 provides a checklist of pretrained models and techniques implemented by participating teams.It summarizes methods used in the Memotion task rather than introducing a separate related-work taxonomy.
9 Memotion Analysis - the next Horizon!
The paper identifies unresolved challenges in understanding meme emotions and outlines future extensions involving broader data, recurring shared tasks, and applications such as recommendation and offensive-content flagging.
- Open challenges: Understanding meme emotion remains difficult because image and embedded text interact, especially in domain-specific memes based on movies or television shows.The paper notes that human interpretation generally considers both modalities, even when one modality performs better in some evaluations.
- Future direction: Memes have become prominent vehicles for expressing individual opinions about societal issues, motivating continued work on their emotion classification.The paper frames meme emotion analysis as a next research horizon.
- Future work: Properly annotated meme data remains scarce, and the authors plan to add annotations for memes in languages such as Hinglish and Spanglish.They also propose organizing similar events in the future and extending classification toward recommendation and automated offensive-meme flagging.