Source-linked AI summary
Detecting Sarcasm in Multimodal Social Platforms
Rossano Schifanella, Paloma de Juan, Joel Tetreault, Liangliang Cao
TL;DR
Sarcasm detection can require information beyond textual sentiment, especially when visual context helps reveal the implied meaning. This paper studies that interaction across Instagram, Tumblr, and Twitter, develops two multimodal frameworks, and evaluates human judgments and automatic detection. Results show that incorporating visual information improves sarcasm detection, though performance varies by platform, dataset, and training-data constraints.
Problem
Text-only sarcasm detection is limited when visual or other contextual information is needed to recover the implied sentiment of multimodal social-media posts.
Method
The paper analyzes text-image relationships across Instagram, Tumblr, and Twitter, uses crowdsourcing to assess image necessity, and evaluates two visual-textual fusion frameworks.
Results
Combining visual and textual information improves automatic sarcasm detection across platforms and methods, while the magnitude and consistency of gains vary across settings.
Takeaways & Limitations
Visuals can serve as situational context for decoding sarcasm, and incorporating them improves automatic detection beyond textual information alone.
Takeaways & Limitations
Twitter performance is lower, which the authors attribute mainly to only 2,000 training posts and Twitter’s more text-centric content.
Abstract
from arXiv · showhide
Sarcasm is a peculiar form of sentiment expression, where the surface sentiment differs from the implied sentiment. The detection of sarcasm in social media platforms has been applied in the past mainly to textual utterances where lexical indicators (such as interjections and intensifiers), linguistic markers, and contextual information (such as user profiles, or past conversations) were used to detect the sarcastic tone. However, modern social media platforms allow to create multimodal messages where audiovisual content is integrated with the text, making the analysis of a mode in isolation partial. In our work, we first study the relationship between the textual and visual aspects in multimodal posts from three major social media platforms, i.e., Instagram, Tumblr and Twitter, and we run a crowdsourcing task to quantify the extent to which images are perceived as necessary by human annotators. Moreover, we propose two different computational frameworks to detect sarcasm that integrate the textual and visual modalities. The first approach exploits visual semantics trained on an external dataset, and concatenates the semantics features with state-of-the-art textual features. The second method adapts a visual neural network initialized with parameters trained on ImageNet to multimodal sarcastic posts. Results show the positive effect of combining modalities for the detection of sarcasm across platforms and methods.
1. INTRODUCTION
Sarcasm detection in social media must recover implied sentiment, which can depend on contextual and visual information beyond the text. The paper studies multimodal posts and proposes two visual-textual frameworks, finding that visual features improve textual models.
- Sarcasm expresses an implied sentiment that differs from the surface sentiment and matters for assessing speakers’ real opinions in social-media applications.
- Text-based methods use lexical, linguistic, hashtag, and emoji cues, but can fail when contextual knowledge is needed to decode sarcasm.
- Previous contextual approaches incorporated author profiles, past posts, and conversations, while linked media had largely been overlooked as contextual information.
- Two frameworks combine modalities: one concatenates externally trained visual semantics with textual features, while the other adapts an ImageNet-initialized visual network to sarcastic posts.
- Visual features boost textual-model performance across platforms and methods.
- The paper studies textual-visual interplay across Instagram, Tumblr, and Twitter and quantifies how humans use images to judge sarcasm.
2. RELATED WORK
Related work treats sarcasm as a linguistic and contextual phenomenon, while emerging multimodal research examines how images interact with written irony. This paper extends that line toward computational sarcasm detection using computer-vision features.
- Sarcasm as linguistic phenomenon: Early sarcasm-recognition systems mainly combined linguistic, sentiment, prosodic, spectral, and contextual cues for classification.
- Sarcasm as contextual phenomenon: Context-based approaches use shared knowledge, author behavior, historical posts, discussion context, and conversations to improve sarcasm prediction.
- Sarcasm beyond text: Research on sarcasm beyond text has examined visual markers and pictorial elements that contribute to identifying verbal irony.
- Sarcasm beyond text: The paper studies curated multimodal posts from three social platforms, whereas prior work often focused on curated advertisements, cartoons, or art.
- Sarcasm beyond text: The paper proposes a computational model that incorporates computer vision into automatic sarcasm detection.
- Making sense of images: Multimodal representation research combines visual and textual information through joint embeddings, image captions, sentiment modeling, and metaphor-oriented fusion.
3. DATA
The dataset combines sarcastic and negative multimodal social-media posts from Instagram, Tumblr, and Twitter after platform-specific collection and filtering. The platforms differ substantially in how often text, images, hashtags, and emojis co-occur.
- Data were collected through public APIs from Instagram, Tumblr, and Twitter, whose intended media formats differ from image-centric Instagram to microblogging Twitter.
- Sarcastic examples were retrieved using the hashtags sarcasm or sarcastic, then filtered for English-language posts and other collection criteria.
- Instagram most often combines text and images, whereas less than 8% of Twitter posts contain images; Tumblr contains both text and photo post types.
- Posts without images or with unavailable images, mentions, or external links were removed during dataset cleaning.
- Instagram users rely heavily on hashtags compared with Twitter users, whose posts have a similar average word count.
- Negative examples were randomly sampled without sarcasm-related terms and processed similarly to the positive examples.
4. CHARACTERIZING THE ROLE OF IMAGES IN SARCASTIC POSTS
The paper categorizes how text and images jointly convey sarcasm and uses crowdsourced judgments to measure when visuals are necessary. Results show that images frequently provide essential context, while author-labeled sarcasm is not always recognized by readers.
- 4.1 Defining a Categorization: The annotation framework asks whether text alone identifies sarcasm and whether the image supplies necessary contextual or sentiment clues.Posts are categorized by the sufficiency of text and the contribution of the image, including Text Only and Text+Image cases.
- 4.1 Defining a Categorization: Text+Image posts require both modalities because the image can depict context that contradicts the text’s literal meaning.In one example, the text reports crowds while the image shows an empty room.
- 4.2 Building a Ground Truth for Sarcasm: The crowdsourcing study uses two independent tasks: text-only judgments followed by multimodal judgments for posts initially marked non-sarcastic.The design prevents awareness of missing images from affecting the first judgment.
- 4.2 Building a Ground Truth for Sarcasm: Contributors below 78% accuracy on test questions are discarded, and annotator agreement averages around 80% across both experiments.Fleiss’ κ indicates a Fair level of agreement.
- 4.2 Building a Ground Truth for Sarcasm: 62.20% of Instagram and 76.40% of Tumblr posts were not decidable from text alone, while images led annotators to identify sarcasm in 60.13% and 58.25% of those cases.Both modalities were necessary in 37.40% of Instagram and 44.50% of Tumblr examples.
- 4.2 Building a Ground Truth for Sarcasm: 24.80% of Instagram and 31.90% of Tumblr posts were not perceived as sarcastic by most annotators, revealing disagreement between author intent and audience interpretation.This disagreement indicates noise in hashtag-based author labeling.
5. AUTOMATED METHODS FOR SARCASM DETECTION
The paper evaluates two multimodal sarcasm-detection frameworks that fuse textual features with visual representations. One uses external visual semantics with an SVM, while the other adapts an ImageNet-initialized neural network to sarcastic posts.
- 5.1 SVM Approach: The first framework combines NLP features with image features in a linear SVM classifier.Textual inputs include lexical, sentiment, subjectivity, n-gram, and word-embedding features.
- 5.1 SVM Approach: The SVM experiments use default toolkit settings because the study’s focus is demonstrating that visual features complement textual features rather than optimizing the detector.The authors acknowledge that parameter and kernel tuning could improve performance.
- 5.1 SVM Approach: Visual Semantics Features use image semantics trained on the YFCC100M dataset to represent complex real-world visual content.These visual features are concatenated with textual features before SVM training.
- 5.2 Deep Learning Approach: The second framework adapts a deep visual representation initialized from an ImageNet model trained on roughly one million images.The original 1,000-object output layer is removed, and the preceding representation is fine-tuned on sarcastic and non-sarcastic data.
- 5.2 Deep Learning Approach: The deep fusion model uses a two-layer unigram perceptron for text because the full NLP feature set could overfit the limited training data.Its hidden layer contains 512 neurons, comparable to the visual representation’s scale.
- 5.2 Deep Learning Approach: The neural adaptation framework concatenates visual and textual representations in a layer with 4,608 neurons and predicts two classes with softmax.Image filters are fixed from ImageNet while other layers begin with random weights and are adapted to the multimodal data.
6. EVALUATION
The evaluation compares text-only, image-only, and fused systems on silver and crowd-curated gold sets. Across SVM and deep-learning frameworks, combining textual and visual signals generally improves sarcasm detection, although gains vary by platform and setting.
- Evaluation settings: The silver evaluation trains on 50% of each dataset and tests on the remaining 50%, using author-labeled posts that include substantial noise in Instagram and Tumblr.Instagram and Tumblr contain 20K posts each, while Twitter contains 4,050 posts; 24.8% of Instagram and 31.8% of Tumblr author-marked sarcastic posts were judged nonsarcastic.
- SVM fusion: 80%+ accuracy was achieved by the combined NLP feature class across all three silver datasets, while visual semantics alone reached about 65%.Visual semantics exceeded lexical and subjectivity features but remained more than 15 points below the best NLP models.
- SVM fusion: 82.3% on Instagram and 81.0% on Tumblr were obtained by combining visual semantics with combination features, while Twitter fusion reached 80.0% versus 80.5% for combination features alone.The visual contribution was therefore positive but small in some silver settings.
- SVM fusion: Gold-set fusion generally performed best: Tumblr reached 87.8% at 80% agreement and 89.7% at 100% agreement, while Instagram fusion improved performance by about 5% across agreement levels.At Tumblr’s 50% agreement level, fusion narrowly missed the combination-only result, 88.5% versus 88.8%.
- Deep-learning fusion: 74.2% in Instagram to 69.7% in Twitter was achieved by deep-network text-image fusion across the silver datasets, which was best in each case.The image-only network also consistently outperformed visual semantics, while Twitter performance was lower than on Instagram and Tumblr.
7. CONCLUSIONS
The study finds that visual information improves sarcasm detection and helps humans decode sarcastic tone, while performance varies across platforms and data conditions. It also identifies limited training data and incomplete contextual integration as directions for further improvement.
- Curated data yields higher predictive accuracy across platforms and almost all agreement levels.This pattern suggests that visual components matter when human judgments are involved.
- Performance differs by platform: Instagram benefits most, whereas images are less relevant for noisy Tumblr and Twitter data.The authors attribute the latter pattern to the text-centric nature of those platforms.
- Visual information improves automatic sarcasm detection across three social media platforms.The study reports a consistent performance increment with deep network adaptation across Instagram, Tumblr, and Twitter.
- Limited training data causes overfitting in network adaptation, while higher feature dimensionality strains GPU memory.The authors state that collecting more training data should at least address the overfitting issue.
- Future work will integrate images with author history and discussion-level sarcasm context, and evaluate visual sentiment frameworks.The goal is to assess which contextual clues have the largest impact per platform.