Source-linked AI summary
CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension
Abhilash Nandy, Rahul Seetharaman, Aman Bansal, Rounak Saha, Manav Nitin Kapadnis, Millon Madhur Das, Pawan Goyal, Niloy Ganguly
TL;DR
Multimodal humor remains difficult because VLMs often miss subtle social, contextual, and relational interactions. CaRGo-T uses VLM-generated code-based causal reasoning graphs before prediction, improving performance over reasoning baselines across humor understanding and detection tasks. Its zero-shot use is constrained by VLM code-generation capability.
Problem
VLMs struggle to comprehend multimodal humor involving intricate social relations, contextual cues, and relational incongruities, while prior reasoning methods provide limited improvement.
Method
CaRGo-T has a VLM generate an explicit causal reasoning graph in a lightweight code-based script, which the same or another VLM uses for humor prediction.
Results
∼1 −20% gains in Humor Understanding and ∼1 −3% gains in Humor Detection are reported across VLMs and settings, with consistent improvements over reasoning-based baselines.
Takeaways & Limitations
CaRGo-T’s causal graph representations contain more information relevant to target outputs and support systematic causal traversal and compositional inference for multimodal humor.
Takeaways & Limitations
Zero-shot CaRGo-T depends on VLM code-generation capability and parametric knowledge, so the authors mostly use closed-source proprietary VLMs in that setting.
Abstract
from arXiv · showhide
Large-scale vision-language models (VLMs) have demonstrated remarkable versatility across a wide range of multimodal tasks. However, understanding humor remains challenging because humorous content often depends on subtle interactions among entities, events, context, and implicit relationships across image and text modalities. These interactions can involve complex chains of reasoning that are difficult to capture through conventional prompting or linear chain-of-thought reasoning. In this work, we propose CaRGo-T (Causal Reasoning Graph-of-Thought), a reasoning framework that represents the causal and contextual relationships underlying multimodal humor as a lightweight graph-based reasoning structure. The graph is serialized into a code-based representation generated by a VLM, which can subsequently be interpreted by the same or a different VLM to produce the final prediction in zero-shot or in-context learning settings. We evaluate CaRGo-T on humor understanding and humor detection across four datasets spanning diverse forms of comedic content, including satire, sarcasm, and memes. Experiments with state-of-the-art commercial and open-source VLMs show that CaRGo-T consistently improves performance over existing reasoning-based baselines, achieving gains of approximately 1-20% on humor understanding and 1-3% on humor detection. Further analysis using mutual information indicates that the reasoning representations produced by CaRGo-T contain more information relevant to the target output than those generated by baseline reasoning approaches. Code is available at https://github.com/abhi1nandy2/CaRGo-T.
1 Introduction
Multimodal humor depends on subtle social, contextual, and relational interactions that current VLMs and linear reasoning methods often fail to represent. CaRGo-T addresses this gap with explicit causal reasoning graphs and reports consistent gains across humor tasks.
- Motivation: Humor comprehension requires modeling relationships among people, objects, concepts, events, social norms, stereotypes, and contextual cues.These interactions create subtle incongruities that give humorous content its meaning.
- Motivation: Prior VLMs struggle with satire, irony, sarcasm, interpersonal dynamics, action–object incongruities, and multimodal figurative content.Chain-of-Thought prompting also yields negligible improvement on sarcasm and affect recognition benchmarks.
- Approach: Causal reasoning graphs retain cause–effect links and lightweight metadata about objects, concepts, events, and participants without probabilistic parameters.They provide an interpretable, event-centric structure for multimodal scenarios.
- Approach: CARGO-T has a VLM construct an explicit causal graph as a lightweight code-based reasoning script before producing the humor prediction.The graph replaces natural-language reasoning or knowledge-graph triplets with causal traversal and compositional inference.
- Results: ∼1 −20% gains in Humor Understanding and ∼1 −3% gains in Humor Detection are reported across VLMs and settings.The evaluations span diverse humor datasets and reasoning-based baselines.
2 Related Work
Related work establishes strong VLM capabilities but persistent weaknesses in social and contextual reasoning, including multimodal humor. Prior causal-AI research studies causal understanding and benchmarks, but does not use cause–effect relations to improve open-ended multimodal reasoning.
- LLMs and VLMs: Large language and vision models perform broadly across instruction-following and multimodal benchmarks but remain inadequate at intricate social reasoning and human contexts.This limitation motivates specialized reasoning approaches for multimodal humor.
- Humor Comprehension and AI: Computational humor research spans recognition, generation, visual humor forecasting, meme captioning, and video humor detection, yet multimodal humor remains difficult for LLMs.The cited work includes studies of visual and video-based humor tasks.
- Reasoning on Vision and Language: Prior vision-language reasoning benchmarks cover commonsense reasoning, visual question answering, and visio-linguistic compositionality relevant to multimodal humor comprehension.Humor tasks require reasoning over interactions among image and text components.
- Causal Reasoning and AI: Causal-AI research includes causal evaluation frameworks, causal graph benchmarks, intervention-based knowledge-graph completion, and multimodal causal-question benchmarks.The paper positions its contribution as applying cause–effect relations to open-ended multimodal reasoning.
3 CARGO-T Framework
CARGO-T prompts a VLM to construct a code-based causal reasoning graph from multimodal inputs and then generate the final answer. Zero-shot use depends strongly on code-generation ability, while in-context learning supplies graph-and-answer examples without model training.
- Framework: CARGO-T replaces a generic reasoning component with a code-based causal reasoning framework for multimodal humor tasks.The framework operates with pretrained VLMs in zero-shot or in-context learning settings.
- Zero-Shot Setting: The zero-shot prompt asks the VLM to link image and text entities in a causal reasoning graph before returning a final answer.The required output contains separate Code and Final Answer fields.
- Zero-Shot Setting: Zero-shot CARGO-T depends on VLM code-generation capability and parametric knowledge, leading the authors to mostly use proprietary VLMs.The authors report that GPT-4o and GPT-4o-mini outperform open-source models on prior zero-shot code-generation evaluations.
- Curating In-Context Examples: In-context learning provides input-output examples containing images, text, manually rectified causal graphs, and ground-truth answers.The examples are curated from non-test samples, with initial graphs generated using a prompt conditioned on the input and answer.
- Curating In-Context Examples: Manual graph rectification identifies entities, lists their properties, and records cause-effect relations derived from those entities and properties.K-shot prompts then provide these examples for inference without training the VLM.
4 Experiments and Results
The experiments evaluate CARGO-T for humor understanding and detection across multiple datasets, VLMs, prompting settings, and reasoning baselines. CARGO-T generally improves performance, with gains also associated with more task-relevant generated reasoning.
- Tasks and Experimental Setup: The study evaluates humor understanding and humor detection across satire, memes, and sarcasm using multiple VLMs and prompting settings.Understanding tasks generate explanations or captions, while detection tasks predict whether content is humorous.
- Tasks and Experimental Setup: CARGO-T is compared with Vanilla and reasoning-based baselines using BLEU, ROUGE-L, BERTScore, average score, Accuracy, and macro-F1.The comparisons cover zero-shot and few-shot in-context learning settings.
- Humor Detection: CARGO-T performs consistently better for sarcasm and satire detection across varying numbers of in-context examples using GPT-4o.The reported improvement is larger for sarcasm detection, which the authors associate with additional supporting text.
- Reasoning Analysis: The generated reasoning component is evaluated for lexical and semantic novelty and for whether the ground truth can be inferred from it.An LLM-as-a-judge procedure produces INFERSCORE values measuring logical inferability of the target from generated reasoning.
5 Conclusion
CaRGo-T uses causal reasoning graphs to model relationships among events, entities, and contextual cues, improving multimodal humor comprehension across VLMs and reasoning-based baselines.
- CaRGo-T systematically models relationships among events, entities, and contextual cues in multimodal inputs.
- CaRGo-T outperforms CoT, CoD, and CCoT across VLMs in zero-shot and few-shot in-context learning settings.
- Causal Reasoning Graphs provide more relevant reasoning information for understanding and detecting humor in multimodal scenarios.
A Rectified vs. GPT-4o generated Causal Reasoning Graph
Figure 3 compares GPT-4o-generated causal reasoning graphs with rectified versions, highlighting which text spans were removed or modified.
- Figure 3 compares in-context causal reasoning graphs generated by GPT-4o with their rectified versions.
- Rectification removes or modifies marked text spans from the GPT-4o-generated graph to produce the resulting changes.
B Example of Humor Understanding and Humor Detection Tasks
Figure 4 presents examples of humor understanding and detection tasks, showing their inputs, task prompts, and ground-truth answers.
- Figure 4 provides an example for humor understanding and detection, specifically using satire.
- The examples list the input, task, and ground truth for the humor tasks.
C Task-specific Queries
The task-specific queries ask models to explain why satirical images or memes are funny and to classify satire or sarcasm with constrained outputs.
- Satirical Image Understanding asks why an image is funny or satirical.
- Meme Caption Generation asks why a meme is funny.
- Satirical Image Detection requires outputting exactly Y or N for whether an image is extremely funny.
- Multimodal Sarcasm Detection requires answering whether an image with text is funny or sarcastic using YES or NO.
D Example of an in-context example for CoT
The example explains humor through the contrast between fashionable high heels and the physical discomfort they cause. Its rationale identifies this contrast as irony and satirical commentary about sacrificing comfort for appearance.
- The image contrasts the aesthetic appeal of high heels with their physical impact.The premise frames the humor around appearance versus bodily consequences.
- The punchline is that fashionable design produces visible discomfort and strain.
- The irony is that people willingly sacrifice comfort for style and appearance.
- The satire highlights the stark contrast between elegant presentation and harm to the wearer’s feet.
- The ground-truth explanation attributes the humor to fashion-driven high-heel use causing substantial physical discomfort.
E Ablation Analysis
The ablation compares CARGO-T with versions using an added graph definition or unrectified in-context graphs, while also defining its graph representation. Results favor CARGO-T, especially when graphs are manually rectified.
- Ablation results: CARGO-T consistently improves lexical ROUGE-L, BLEU, and average score metrics in the ablation.
- Ablation results: In in-context learning, CARGO-T outperforms UNRECTIFIED across all metrics, highlighting the importance of manually rectifying causal reasoning graphs.
- Ablation results: CARGO-T outperforms WITH DEFN. on 3/4 zero-shot metrics and 4/4 in-context-learning metrics.
- Experimental setting: The ablation evaluates GPT-4o with zero, two, and five in-context examples.
- Graph representation: The causal reasoning graph represents entities with descriptive or non-causal relational properties.
- Graph representation: Events collect entities and their non-causal relationships, while causal relationships connect one event as a cause to another as an effect.
F Further Analysis of CARGO-T’s Reasoning Component
The further analysis describes text-comparison procedures based on token distributions and sentence embeddings. These procedures include preprocessing, tokenization, similarity calculation, and low-similarity sentence counting.
- The analysis preprocesses and tokenizes two input texts before comparing them.
- The KL-divergence procedure builds token-count vocabularies and computes smoothed distributions for the two texts.
- The sentence-level procedure preprocesses each text, tokenizes its sentences, and computes sentence embeddings.
- The low-similarity fraction counts sentences in the first text whose average similarity to the second text falls below an upper bound U.
- Cosine similarities are used in the sentence-comparison procedure.