Source-linked AI summary
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, Mohammed, Hoque
TL;DR
Multimodal humor detection remains understudied despite humor’s expression through text, vision, and acoustic cues and its dependence on context. The paper introduces UR-FUNNY and studies it with a Contextualized Memory Fusion Network. Results show that using all three modalities and modeling both context and punchlines improves humor modeling, while a substantial gap remains between state-of-the-art and human performance.
Problem
Multimodal humor detection lacks sufficient study despite humor’s dependence on text, vision, acoustic cues, and contextual buildup.
Method
The paper constructs UR-FUNNY from diverse TED speakers and topics, using laughter markers, three aligned modalities, and a Contextualized Memory Fusion Network.
Results
Using all three modalities outperforms unimodal and bimodal models, while both context and punchline information contribute to humor detection.
Takeaways & Limitations
UR-FUNNY provides a public resource and establishes challenges for multimodal humor detection and multimodal language analysis.
Takeaways & Limitations
The formulation assumes word-level alignment across text, vision, and acoustic modalities, despite their differing sampling frequencies.
Abstract
from arXiv · showhide
Humor is a unique and creative communicative behavior displayed during social interactions. It is produced in a multimodal manner, through the usage of words (text), gestures (vision) and prosodic cues (acoustic). Understanding humor from these three modalities falls within boundaries of multimodal language; a recent research trend in natural language processing that models natural language as it happens in face-to-face communication. Although humor detection is an established research area in NLP, in a multimodal context it is an understudied area. This paper presents a diverse multimodal dataset, called UR-FUNNY, to open the door to understanding multimodal language used in expressing humor. The dataset and accompanying studies, present a framework in multimodal humor detection for the natural language processing community. UR-FUNNY is publicly available for research.
1 Introduction
Humor is expressed through coordinated text, vision, and acoustic cues, but its creative and context-dependent nature makes computational modeling difficult. UR-FUNNY addresses this gap with a multimodal humor-detection dataset and framework.
- Humor combines words, gestures, and acoustic cues, with styles emerging from their coordination or misalignment.Examples include gradual buildups, sudden punchlines, dry humor, and laughter accompanying speech.
- Computational humor modeling is difficult because creative expression produces idiosyncratic multimodal patterns.Speakers experiment with words, gestures, prosody, and their alignments.
- Humor often depends on context because a punchline may become funny only after a preceding buildup and delivery.Understanding humor therefore requires analyzing the context surrounding the punchline.
- UR-FUNNY introduces a multimodal dataset combining text, vision, and acoustic modalities for humor detection.The paper presents it as the first such multimodal language dataset for humor detection in the NLP community.
2 Background
Prior humor datasets study tasks such as joke and punchline detection, while multimodal language analysis extends NLP across text, vision, and acoustic modalities. UR-FUNNY is positioned within this developing multimodal research landscape and is compared with notable humor datasets.
- Dataset Comparison: Table 1 compares UR-FUNNY with notable humor detection datasets using positive, negative, modality, and speaker categories.The table caption defines “pos,” “neg,” “mod,” and “spk” as positive, negative, modalities, and speaker, respectively.
- Humor Analysis: Existing humor datasets address different tasks, including joke detection and punchline detection.Examples include “16000 One-Liners” and “Pun of the Day” for joke-versus-not-joke classification, and “Ted Laughter” for punchline detection.
- Humor Analysis: Humor detection has used hand-crafted, non-neural, recurrent, convolutional, and highway-network models.The cited applications include Yelp and TED talks, alongside the “16000 One-Liner” and “Pun of the Day” datasets.
- Multimodal Language Analysis: Multimodal language analysis studies natural language through text, vision, and acoustic modalities and is described as a recent NLP research trend.Related work includes multimodal neural architectures, fusion approaches, and multimodal resources.
3 UR-FUNNY Dataset
UR-FUNNY is a diverse multimodal humor dataset built from TED videos, using laughter markers and forced alignment to create speaker- and topic-diverse, balanced instances with synchronized text, audio, and video features.
- Data acquisition: The dataset contains TED videos spanning many speakers and topics, providing diversity for modeling idiosyncratic humor.TED is used because its speakers and presentation topics cover a broad spectrum of humor.
- Data acquisition: Laughter markers identify humorous punchlines, while preceding sentences form context; matched negative samples come from random intervals without an immediately following laughter marker.The resulting dataset has a homogeneous 50% split between positive and negative examples.
- Feature extraction: Forced alignment synchronizes sentence, word, and phoneme boundaries across text, audio, and video, enabling multimodal timing extraction.These alignments provide the timing of both punchline and context for every instance.
- Dataset statistics: UR-FUNNY includes 16,514 humor and non-humor video segments from 1,866 videos, 1,741 speakers, and 417 topics, totaling 90.23 hours.Humor and non-humor instances are equally split, with an average instance duration of 19.67 seconds.
- Dataset statistics: Humor and non-humor cases have similar distributions of punchline length, context length, and context-sentence count, so simple length measures do not distinguish them.At least 90% of punchlines contain fewer than 32 words, but classification requires understanding sentence content.
- Availability: The dataset is publicly available together with its extracted multimodal features.
4 Multimodal Humor Detection
The paper formulates humor detection from aligned text, vision, and acoustic sequences, then extends MFN with context encoders that represent unimodal and cross-modal dependencies before predicting humor.
- Problem formulation: Text, vision, and acoustic modalities are assumed word-aligned, with visual and acoustic descriptors aggregated to match each word.
- Problem formulation: Each sample is a labeled triplet containing a punchline and its preceding multimodal context, with prediction based on the punchline-context pair.The binary label indicates whether the inputs are likely to trigger laughter.
- Unimodal Context Network: The Unimodal Context Network uses one LSTM per modality to encode each context sentence into vectors collected as H.Each LSTM processes the modality’s context at word-level time steps.
- Multimodal Context Network: The Multimodal Context Network applies a six-layer encoder to H to derive a spatio-temporal multimodal representation ˆH.Self-attention is used to model temporal relations among sentences and modalities.
- Memory Fusion Network: C-MFN initializes MFN’s unimodal and multimodal memories with H and ˆH, then models punchline information through recurrent and gated memory components.The final humor prediction uses the last states of the LSTM system and Multi-view Gated Memory.
5 Experiments
Experiments establish UR-FUNNY baselines while isolating the roles of context, punchline, and individual modalities through C-MFN variants and modality combinations.
- Experimental design: The experiments evaluate baseline performance and examine how context, punchline, and individual modalities affect binary humor detection.
- Results: The best binary accuracy is achieved when text, vision, and acoustic modalities are used together.
- Experimental design: C-MFN variants separately use punchline-only or context-only information to study their contributions to humor modeling.The punchline-only variant is equivalent to MFN without contextual initialization, while the context-only variant removes MFN.
- Experimental design: The modality scenarios compare text, text-plus-vision, text-plus-acoustic, vision-plus-acoustic, and all three modalities together.
6 Results and Discussion
The experiments show that context and punchlines both matter, with punchlines more important, while combining text, vision, and acoustic modalities improves humor detection.
- C-MFN outperforms C-MFN (P) and C-MFN (C), showing that both context and punchline information are important.
- Punchline is the most important component because C-MFN (P) performs significantly better than C-MFN (C).
- Models using all modalities (T+A+V) outperform models using only one or two modalities (T, T+A, T+V, A+V).
- Vision and acoustic modalities improve text-only performance in most cases, as shown by the T+V and T+A settings.
- Each neural component of C-MFN is useful for improving humor prediction, and multimodal humor modeling yields successful results.
- 82.5% is the human performance on the UR-FUNNY dataset.
- A state-of-the-art model achieves reasonable success, but a large gap remains between its performance and human-level performance.
7 Conclusion
The paper introduces UR-FUNNY, a publicly available multimodal humor-detection dataset, and studies it with C-MFN using text, vision, and acoustic modalities.
- UR-FUNNY is a new multimodal humor-detection dataset and is presented as the first of its kind in the NLP community.
- The dataset contains text, vision, and acoustic modalities and includes diverse speakers and topics.
- The study analyzes UR-FUNNY through a Contextualized Memory Fusion Network (C-MFN).
- The experiments indicate that using all three modalities together better models humor, while context and punchline are both important.
A Hyperparameter Space Search
The hyperparameter search explores the Uni-modal Context Network's LSTM hidden sizes across language, acoustic, and visual inputs.
- The Uni-modal Context Network contains three LSTMs.
- LSTM_l hidden size is randomly selected from [32,64,88,128,156,256].
- LSTM_a and LSTM_v hidden sizes are randomly selected from [8,16,32,48,64,80].
• Multimodal Context Network:
The multimodal context network search specifies Transformer configurations and dropout ranges for regularizing its outputs.
- • Multimodal Context Network:: The Transformer uses d_model=512, d_k=64, d_v=64, eight attention heads, six encoder layers, 4000 warmup steps, and dropout=0.1.
- • Multimodal Context Network:: The output of D(Ĥ) is regularized with a randomly selected dropout rate from [0.0,0.2,0.5,0.1].
- • Multimodal Context Network:: The outputs D_m(H) use modality-specific randomly selected dropout probabilities for m=l, m=a, and m=v.
- • Multimodal Context Network:: For m=l, m=a, and m=v, the dropout probabilities are selected from [0.0,0.1,0.2,0.5], [0.0,0.2,0.5,0.1], and [0.0,0.2,0.5,0.1], respectively.
• Memory Fusion Network(MFN):
The MFN implementation combines LSTM-based processing, delta memory attention, and multi-view gated memory, with randomly selected architectural and optimization settings.
- • Memory Fusion Network(MFN):: The system uses LSTMs for language, acoustic, and visual streams, with independently selected hidden sizes.Language hidden sizes range from 32 to 256; acoustic and visual sizes range from 8 to 80.
- • Memory Fusion Network(MFN):: Delta Memory Attention uses two affine transformations, NN1 and NN2, each followed by dropout.Projection shapes and dropout rates are randomly selected from the specified sets.
- • Memory Fusion Network(MFN):: Multi-view gated memory uses two affine transformations, Gamma1 and Gamma2, with projection and dropout operations.Both transformations use randomly selected projection shapes and dropout rates.
- • Memory Fusion Network(MFN):: The multi-view gated memory module randomly selects its memory size from 64, 128, 256, 300, or 400.
- • Memory Fusion Network(MFN):: Training uses Adam with β1 = 0.9, β2 = 0.98, and ϵ = 10−9, together with the stated warmup-based learning-rate schedule.The optimizer and scheduler follow Vaswani et al. (2017).