Source-linked AI summary
GDCNet: Generative Discrepancy Comparison Network for Multimodal Sarcasm Detection
Shuguang Zhang, Junhong Lian, Guoxin Yu, Baoxun Xu, Xiang Ao
TL;DR
Multimodal sarcasm detection remains difficult when visual and textual content are weakly aligned or semantically indirect, and generated sarcastic cues can introduce noise. GDCNet instead uses factual MLLM-generated image captions as anchors, models semantic, sentiment, and fidelity discrepancies, and fuses them with visual and textual features through gating. Experiments establish a new state of the art on MMSD2.0 and report consistent superiority over competitive baselines.
Problem
Existing multimodal sarcasm detectors struggle with weak image–text alignment and subtle ironic cues, while diverse LLM-generated sarcastic signals can introduce noise.
Method
GDCNet uses factually grounded MLLM image captions as semantic anchors, models semantic, sentiment, and visual-textual discrepancies, and adaptively fuses them with visual and textual representations.
Results
GDCNet establishes a new state of the art on MMSD2.0 and consistently outperforms competitive baselines.
Takeaways & Limitations
The results support using image-grounded factual captions and explicit discrepancy modeling to capture cross-modal incongruity in multimodal sarcasm detection.
Abstract
from arXiv · showhide
Multimodal sarcasm detection (MSD) aims to identify sarcasm within image-text pairs by modeling semantic incongruities across modalities. Existing methods often exploit cross-modal embedding misalignment to detect inconsistency but struggle when visual and textual content are loosely related or semantically indirect. While recent approaches leverage large language models (LLMs) to generate sarcastic cues, the inherent diversity and subjectivity of these generations often introduce noise. To address these limitations, we propose the Generative Discrepancy Comparison Network (GDCNet). This framework captures cross-modal conflicts by utilizing descriptive, factually grounded image captions generated by Multimodal LLMs (MLLMs) as stable semantic anchors. Specifically, GDCNet computes semantic and sentiment discrepancies between the generated objective description and the original text, alongside measuring visual-textual fidelity. These discrepancy features are then fused with visual and textual representations via a gated module to adaptively balance modality contributions. Extensive experiments on MSD benchmarks demonstrate GDCNet's superior accuracy and robustness, establishing a new state-of-the-art on the MMSD2.0 benchmark.
1. INTRODUCTION
GDCNet addresses the difficulty of detecting subtle cross-modal irony by using stable, factual image descriptions as semantic anchors and comparing them with text. It combines discrepancy modeling with gated multimodal fusion, and experiments report improved accuracy with a new state of the art on MMSD2.0.
- MSD is challenging because image–text interplay can create irony beyond either modality alone, while broad embedding mismatches miss subtle cues when alignment is weak.
- GDCNet uses MLLM-generated, descriptive, factually grounded image captions as cross-modal semantic bridges for comparison with associated text.
- A gated fusion module integrates discrepancy representations with original visual and textual features to adaptively balance modality contributions.
- Sarcastic explanations generated by LLMs vary across models and prompts, whereas factual image descriptions remain stable and can provide reliable semantic anchors.
- Its GDRM compares generated descriptions with original text through semantic and sentiment discrepancies while assessing image–text fidelity.
- Extensive benchmark experiments report significant accuracy improvements and a new state of the art on MMSD2.0.
2. METHODOLOGY
GDCNet addresses subtle cross-modal incongruities by generating an objective image description and comparing it with the original text. It forms discrepancy features and adaptively fuses them with visual and textual representations for classification.
- MSD classifies image-text pairs as sarcastic or non-sarcastic by identifying subtle mismatches between textual semantics and visual context.
- GDCNet uses an image-only MLLM to generate a faithful, context-independent description as an unbiased semantic representation of the image.Excluding multimodal contextual cues helps avoid sarcasm-related biases in the generated description.
- The Generative Discrepancy Representation Module measures semantic discrepancy, sentiment discrepancy, and visual-textual fidelity between the generated description, original text, and image.Semantic discrepancy uses CLIP cosine dissimilarity, sentiment discrepancy uses RoBERTa probability-distribution L1 distance, and fidelity measures description-image alignment.
- The three discrepancy measures are concatenated into D and processed by an MLP to produce the discrepancy representation FD.
- A gated fusion module assigns learnable modality weights and combines text FT, image FI, and discrepancy FD features into Ffused.Independent classifiers process modality-specific and fused representations before an MLP produces the final prediction.
- GDCNet jointly optimizes binary sarcasm classification and contrastive multimodal alignment, with α controlling their trade-off.The contrastive term aligns paired image-text embeddings in a shared latent space.
3. EXPERIMENTS
Experiments on MMSD2.0 compare GDCNet with unimodal, multimodal, ablated, and LLM-based baselines. GDCNet achieves the highest overall performance, while discrepancy modeling, gated fusion, and richer captions support its results.
- Main Results: GDCNet establishes a new state-of-the-art on MMSD2.0 in both accuracy and F1-score.The reported advantage is consistent across these two metrics.
- Main Results: GDCNet achieves the highest overall performance on MMSD2.0, outperforming unimodal and prior multimodal methods.The benchmark comparison reports the strongest overall performance for GDCNet.
- Main Results: Adaptive gated fusion incorporates cross-modal divergence features to prevent modality dominance and balance multimodal integration.The mechanism combines discrepancy signals with multimodal representations through adaptive weighting.
- Ablation Study: Removing GDRM reduces Accuracy by 2.96% and F1-score by 4.15%, underscoring the role of explicit discrepancy modeling.The ablation removes the full discrepancy module from GDCNet.
- Comparison with LLM-based Methods: GDCNet consistently outperforms LLaVA, Qwen-VL, and GPT-4o under both Zero-Shot and Chain-of-Thought prompting.The comparison covers direct MLLM-based sarcasm detection baselines on MMSD2.0.
- Caption Generator Ablation: LLaVA-NEXT produces richer, more semantically consistent captions than BLIP-2, while richer captions yield better downstream performance.BLIP-2 offers superior inference speed, creating a speed-versus-caption-quality trade-off.
4. CONCLUSION
GDCNet uses image-grounded captions as cross-modal anchors and explicitly models semantic and sentiment discrepancies. On MMSD2.0, it establishes a new state-of-the-art while reducing modality dominance and spurious correlations.
- Conclusion: GDCNet uses LLM-generated, image-grounded captions as anchors for fine-grained semantic and sentiment discrepancy modeling.An adaptive gated fusion module integrates visual, textual, and discrepancy signals.
- Conclusion: GDCNet establishes a new state-of-the-art on MMSD2.0 and consistently outperforms competitive baselines.The conclusion presents this as the main experimental outcome.
- Conclusion: The framework positions LLMs as structural guides for capturing subtle cross-modal incongruities in multimodal understanding tasks.This conclusion extends beyond using LLMs solely as data generators.