Source-linked AI summary
Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, Soujanya Poria
TL;DR
Sarcasm detection has often focused on text despite cues across speech, vision, and dialogue context. The paper introduces MUStARD, an audiovisual dataset with conversational history, and evaluates multimodal models. Multimodal variants significantly outperform unimodal counterparts, with relative error-rate reductions of up to 12.9%.
Problem
Sarcasm can require verbal, non-verbal, multimodal, or conversational cues, while recent detection work has largely focused on textual data.
Method
The paper introduces MUStARD, an audiovisual sarcasm dataset with annotated utterances and historical dialogue context, and develops models using text, speech, and visual signals.
Results
12.9% relative error-rate reductions were achieved by multimodal variants compared with unimodal counterparts in multiple evaluations.
Takeaways & Limitations
MUStARD provides a resource for multimodal and conversational-context sarcasm research.
Takeaways & Limitations
The dataset includes often-multiparty dialogues, creating a future challenge for advanced multimodal techniques.
Abstract
from arXiv · showhide
Sarcasm is often expressed through several verbal and non-verbal cues, e.g., a change of tone, overemphasis in a word, a drawn-out syllable, or a straight looking face. Most of the recent work in sarcasm detection has been carried out on textual data. In this paper, we argue that incorporating multimodal cues can improve the automatic classification of sarcasm. As a first step towards enabling the development of multimodal approaches for sarcasm detection, we propose a new sarcasm dataset, Multimodal Sarcasm Detection Dataset (MUStARD), compiled from popular TV shows. MUStARD consists of audiovisual utterances annotated with sarcasm labels. Each utterance is accompanied by its context of historical utterances in the dialogue, which provides additional information on the scenario where the utterance occurs. Our initial results show that the use of multimodal information can reduce the relative error rate of sarcasm detection by up to 12.9% in F-score when compared to the use of individual modalities. The full dataset is publicly available for use at https://github.com/soujanyaporia/MUStARD
1 Introduction
Sarcasm can rely on verbal, non-verbal, multimodal, and conversational-context cues, motivating research that combines these sources. The paper introduces MUStARD to support multimodal sarcasm detection.
- Sarcasm may be signaled by tone changes, word overemphasis, drawn-out syllables, or a straight-looking face.
- Detecting sarcasm can require additional information from multiple modalities or dialogue context history.
- The paper introduces MUStARD, a dataset for multimodal sarcasm research with high-quality multimodal and conversational-context annotations.
- The paper presents multimodal incongruity examples, baselines, and preceding dialogue turns as context information for future conversational sarcasm detection.
2 Related Work
Prior sarcasm-detection research has examined text, speech, visual content, and contextual information, but video-level multimodal sarcasm in dialogues remains limited. This paper positions its resource as an extension beyond text-only and reader-focused multimodal settings.
- Sarcasm in Text: Existing text-based approaches use rules, lexical and pragmatic features, stylistic features, situational disparity, incongruity, and annotations such as hashtags.
- Sarcasm in Text: Sarcasm resources commonly use Twitter and rely on manual annotation or distant supervision through hashtags.
- Sarcasm in Speech: Speech-based sarcasm detection has focused mainly on prosodic and acoustic cues, including amplitude, speech rate, and tonalities.
- Multimodal Sarcasm: Prior multimodal work often combines text with readers’ gaze or EEG/MEG signals to study sarcasm perception.
- Multimodal Sarcasm: The paper extends multimodal sarcasm research to video-based sarcasm in dialogues and proposes a resource for video-level sarcasm.
3 Dataset
MUStARD is a manually annotated audiovisual sarcasm dataset built from television content, with conversational context accompanying each utterance. The paper documents annotation, transcript preparation, balancing, and qualitative evidence that multimodal and contextual cues support sarcasm analysis.
- Data collection: MUStARD consists of short videos manually annotated for sarcasm, collected from multiple television shows and supplemented with non-sarcastic examples.The initial collection contained 6,421 videos from several sources before annotation and filtering.
- Annotation process: Two annotators labeled videos independently, with disagreements reconciled after low initial agreement and again for the second annotation stage.The second stage covered 624 videos, and a third annotator reconciled differences.
- Dataset composition: The final annotated set contains 345 sarcastic and 6,020 non-sarcastic videos, while the experimental dataset uses a balanced sample of 690 videos.The balanced sample contains equal numbers of sarcastic and non-sarcastic examples.
- Dataset structure: An utterance may contain consecutive multi-sentence dialogue from one speaker and is paired with preceding conversational context.The dataset treats videos as utterances; 61.3% are single sentences, and context turns are manually selected for coherent background.
- Dataset structure: Each utterance and its context include video, audio, transcription, and speaker identifiers, enabling analysis across modalities and speakers.The dataset also reports source, character, and label-ratio statistics.
- Qualitative aspects: Examples show sarcasm arising from incongruent modalities, vocal stress, complementary multimodal cues, and context-dependent interpretations.The paper argues that modality combinations need not conflict and that conversational context can be essential for interpreting sarcastic references.
4 Multimodal Feature Extraction
The paper extracts fixed-length representations from text, audio, and video, then combines modality-specific features for sarcasm detection.
- Text Features: Textual utterances are represented with 768-dimensional BERT sentence embeddings from averaged final-layer [CLS] representations.Averaged 300-dimensional GloVe features performed worse than BERT features.
- Speech Features: Audio processing targets pitch, intonation, and tonal details using Librosa-derived low-level features.Features include MFCC, melspectrogram, spectral centroid, and temporal derivatives extracted from segmented windows.
- Speech Features: Audio windows are averaged to create a fixed-length utterance representation.The averaging follows segmentation into nonoverlapping windows.
- Video Features: Visual features use 2048-dimensional pool5 representations from ImageNet-pretrained ResNet-152 frames, averaged across the utterance.Frames are resized, center-cropped, and normalized before feature extraction.
5 Experiments
The experiments compare individual and combined modalities, with additional evaluations of context and speaker information under speaker-dependent and speaker-independent setups.
- Experimental Design: Experiments evaluate each modality separately, modality combinations, conversational context, and speaker information.The study investigates whether these sources improve sarcasm predictions.
- Evaluation Settings: Randomized stratified five-fold cross-validation produces a speaker-dependent setup because speakers can overlap across training and testing folds.Stratification maintains label balance across folds.
- Evaluation Settings: The speaker-independent setup assigns entire shows to either training or testing, preventing speaker overlap and testing generalization to a new environment.The Big Bang Theory, The Golden Girls, and Sarcasmaholics Anonymous are used for training, while Friends is used for testing.
- Metrics: Performance is measured with class-ratio-weighted precision, recall, and F-score across sarcastic and non-sarcastic classes.Speaker-dependent results are averaged across the five cross-validation folds.
- Baselines: Three baselines are evaluated: Majority, Random, and an RBF-kernel SVM.The Majority baseline predicts non-sarcastic for every instance, while Random samples uniformly.
6 Multimodal Sarcasm Classification
Multimodal performance is strongest in the speaker-dependent setting, while speaker-independent evaluation is harder and changes the relative value of audio, video, context, and speaker information.
- Speaker-dependent Classification: 12.9% relative error-rate reduction in weighted F-score is achieved by combining visual and textual signals over unimodal variants.The text-plus-video model performs best among the reported speaker-dependent variants.
- Speaker-dependent Classification: Text-plus-visual models correctly recover sarcastic utterances lacking explicit textual markers by using additional multimodal cues.The analysis focuses on cases where the textual model fails but the bimodal model succeeds.
- Speaker-independent Classification: The speaker-independent setup requires greater generalization because it excludes speaker overlap and uses an entirely new multimodal environment for testing.The authors describe it as a strong test-bed for multimodal sarcasm research.
- Speaker-independent Classification: In speaker-independent evaluation, multimodal variants do not greatly outperform unimodal counterparts, while audio becomes more important and improves slightly with text.The analysis associates correctly predicted text-plus-audio examples with higher mean pitch.
- Speaker-independent Classification: Video features perform poorly in the speaker-independent setup, which the authors hypothesize may reflect character biases from shallow object-based features.The hypothesis is linked to speaker-specific statistics shown in Figure 10.
- Context and Speaker Information: Context produces only a slight improvement for the best speaker-independent text-plus-audio variant, with no improvement in other models.The authors suggest pooling may discard temporal information.
- Context and Speaker Information: Speaker features improve the speaker-dependent textual model but barely improve the best text-plus-video model, while speaker information does not improve speaker-independent results.The speaker-dependent gain is attributed to overlap across splits and exploitable speaker regularities.
7 Conclusion and Future Work
The paper introduces MUStARD and multimodal models for sarcasm detection, reporting improved performance over unimodal variants. It identifies limitations involving fusion, dialogue modeling, speaker localization, and dataset size.
- Conclusion: MUStARD is a dataset of sarcastic and non-sarcastic videos designed to enable multimodal sarcasm research.The paper also includes multimodal and conversational context features.
- Conclusion: Multimodal variants significantly outperform unimodal counterparts, with relative error rate reductions of up to 12.9%.
- Future Work: The study identifies multimodal fusion, multiparty conversation, neural baselines, conversational context, and main speaker localization as challenges for future work.
- Future Work: Only early fusion was explored, leaving advanced spatiotemporal fusion and methods for modeling cross-modal incongruity for future research.
- Future Work: The dataset’s limited size creates over-fitting risks for complex neural models, and initial experiments found SVM classifiers outperforming CNNs.
- Future Work: Context modeling currently ignores interlocutors’ goals, intents, and dependencies, while visual features are extracted without explicitly localizing the main speaker.