Source-linked AI summary
MIntRec: A New Dataset for Multimodal Intent Recognition
Hanlei Zhang, Hua Xu, Xin Wang, Qianrui Zhou, Shaojie Zhao, Jiayan Teng
TL;DR
Real-world multimodal intent recognition has been constrained by benchmarks that mostly provide text rather than multimodal information. MIntRec introduces a dataset with multimodal intent categories and speaker annotations, and its experiments find stable gains from nonverbal information over text-only recognition, though the best methods remain far from human performance.
Problem
Most existing intent benchmarks contain only text or visual information, leaving a lack of multimodal datasets for intent analysis in real-world dialogue scenes.
Source: §1 · p. 1, §6.1 · p. 8Method
The paper constructs MIntRec from Superstore data with 2,224 multimodally annotated samples, new coarse- and fine-grained intent categories, and automatic speaker bounding-box annotation.
Source: §1 · pp. 2–3Results
1%–2% stable improvements from nonverbal information are reported on binary and multi-class classification, while the best methods remain far from human performance.
Source: §1 · p. 3Takeaways & Limitations
The gap between the best-performing methods and humans indicates that multimodal intent recognition remains challenging.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Multimodal intent recognition is a significant task for understanding human language in real-world multimodal scenes. Most existing intent recognition methods have limitations in leveraging the multimodal information due to the restrictions of the benchmark datasets with only text information. This paper introduces a novel dataset for multimodal intent recognition (MIntRec) to address this issue. It formulates coarse-grained and fine-grained intent taxonomies based on the data collected from the TV series Superstore. The dataset consists of 2,224 high-quality samples with text, video, and audio modalities and has multimodal annotations among twenty intent categories. Furthermore, we provide annotated bounding boxes of speakers in each video segment and achieve an automatic process for speaker annotation. MIntRec is helpful for researchers to mine relationships between different modalities to enhance the capability of intent recognition. We extract features from each modality and model cross-modal interactions by adapting three powerful multimodal fusion methods to build baselines. Extensive experiments show that employing the non-verbal modalities achieves substantial improvements compared with the text-only modality, demonstrating the effectiveness of using multimodal information for intent recognition. The gap between the best-performing methods and humans indicates the challenge and importance of this task for the community. The full dataset and codes are available for use at https://github.com/thuiar/MIntRec.
1 INTRODUCTION
Existing intent benchmarks largely lack multimodal information for real-world dialogue, where expressions, movement, and speech tone can alter how an utterance is understood. MIntRec addresses this gap with a multimodal dataset, new intent categories, automatic speaker annotation, and experiments comparing multimodal with text-only recognition.
1 INTRODUCTION
- Existing intent benchmarks mostly use text or visual information, leaving a lack of multimodal datasets for real-world intent analysis.The paper also distinguishes image-caption intent data from spoken language in real-world scenes.Source: §1 · p. 2
- Multimodal fusion yields stable 1%–2% improvements over text-only intent recognition on binary and multi-class classification, while the best methods remain far from human performance.The paper reports that the gap from human performance indicates the task remains challenging.Source: §1 · p. 3
- MIntRec introduces 2,224 high-quality samples with multimodal intent annotations as a benchmark for real-world multimodal scenes.The paper describes it as the first benchmark dataset for intent recognition in such scenes.Source: §1 · p. 3
- The dataset defines two coarse-grained and twenty fine-grained intent categories for multimodal intent analysis.The coarse-grained categories are emotions or attitudes and goals.Source: §1 · p. 3
- An automatic speaker-annotation process produces speaker bounding boxes, evaluated against more than 12K human-annotated keyframes.The paper says the process saves time and labor and may benefit similar annotation tasks.Source: §1 · p. 3
2 MINTREC DATASET
MIntRec is built from Superstore video, subtitle, audio, and text segments, with intent taxonomies and labels designed for multimodal scenes. Five annotators label samples using all three modalities, while an automatic process identifies speakers and their bounding boxes.
2.1 Data Preparation
- Superstore provides varied characters and scenes; utterance timestamps are used to create corresponding video and audio segments.The paper selects the series for its multimodal signals and diverse settings.Source: §2.1 · pp. 3–4
2.2 Intent Taxonomy Definition
- The taxonomy comprises two coarse-grained groups and twenty fine-grained categories, including eleven emotion-or-attitude intents and nine goal intents.Some categories may be cued by one modality, while others may require combining modalities.Source: §2.2 · p. 4
2.3 Multimodal Intent Annotation
- Five trained workers label samples using text, video, and audio; labels require at least three votes, and the resulting dataset has 2,224 samples.The dataset is imbalanced, although every category contains at least fifty samples.Source: §2.3 · p. 4
2.4 Automatic Speaker Annotation
- Speaker annotation uses person and face detection, face tracking, and audio-visual active-speaker detection to assign bounding boxes.The process automatically generates more than 120K keyframes with speaker bounding boxes without manual intervention.Source: §2.4 · pp. 4–5
- The speaker-annotation process has a 2.3% missing rate, and 90.9% of hit keyframes have predicted bounding boxes with IoU > 0.9.These results are evaluated against human-annotated keyframes.Source: §2.4 · p. 5
3 METHODOLOGY
The methodology extracts features from the prepared text, video, and audio data, then uses those features for multimodal fusion.
3 METHODOLOGY
- The methodology begins with corresponding text, video, and audio data for speakers.The passage frames feature extraction and fusion as the next stages.Source: §3 · p. 5
- Features are extracted separately from each modality before multimodal fusion.Source: §3 · p. 5
- The extracted modality features are used for multimodal fusion.Source: §3 · p. 5
3.1 Feature Extraction
The feature-extraction section uses pretrained models to represent text, video, and audio. Vision features are extracted from annotated bounding-box regions, and the dataset splits use a 3:1:1 training, validation, and testing ratio.
3.1.1 Text.
- BERT supplies text token embeddings from its last hidden layer for each utterance.The embedding dimension is 768.Source: §3.1.1 · p. 5
3.1.2 Vision.
- Faster R-CNN with ResNet-50 extracts keyframe features, which are mapped to annotated regions and pooled into vision embeddings.The passage specifies RoIAlign and average pooling for the region features.Source: §3.1.2 · p. 5
Dataset split
- Training, validation, and testing sets are split in a 3:1:1 ratio.Source: Table 3 · p. 5
3.1.3 Audio.
- Librosa obtains audio at 16,000 Hz, and wav2vec 2.0 supplies acoustic embeddings from its last hidden layer.The acoustic feature dimension is 768.Source: §3.1.3 · p. 5
3.2 Benchmark Multimodal Fusion Methods
The paper benchmarks three multimodal fusion methods that model interactions among text, video, and audio features. MulT uses cross-modal attention, MISA separates shared and modality-specific representations, and MAG-BERT adapts BERT with nonverbal information.
3.2 Benchmark Multimodal Fusion Methods
- The benchmark adapts MulT, MISA, and MAG-BERT to model interactions among extracted modality features for multimodal intent recognition.The methods receive multimodal annotations as classification targets; MAG-BERT’s video and audio features are aligned to text with CTC.Source: §3.2 · p. 5, §3.2.3 · p. 5
3.2.1 MulT.
- MulT handles non-aligned multimodal sequences through pairwise inter-modal attention that captures adaptation knowledge across modalities.It extends the vanilla Transformer with a cross-modal Transformer.Source: §3.2.1 · p. 5
3.2.2 MISA.
- MISA learns shared features across modalities and distinct features for each modality, using similarity, difference, reconstruction, and task-specific losses.Source: §3.2.2 · p. 5
3.2.3 MAG-BERT.
- MAG-BERT inserts a multimodal adaptation gate into BERT to shift its semantic space in response to acoustic and visual information.Because MAG-BERT requires aligned multimodal data, video and audio features are aligned to text at the word level with CTC.Source: §3.2.3 · p. 5
4 EXPERIMENTS
The experiments compare text-only and multimodal baselines on MIntRec using twenty-class and binary classification. Multimodal methods outperform text-only classification, while human performance remains substantially higher.
4.1.1 Dataset Splits.
- Video segments are randomly shuffled and split into training, validation, and testing sets in a 3:1:1 ratio.Source: §4.1.1 · p. 6
4.1.2 Evaluation Metrics.
- Performance is evaluated with accuracy, macro F1, macro precision, and macro recall, with higher values indicating better performance.Source: §4.1.2 · p. 6
4.1.3 Implementation Details.
- Text uses BERT, audio uses wav2vec 2.0, and video uses Faster R-CNN with a ResNet-50 backbone.Sequences are zero-padded to lengths 30, 230, and 480 for text, video, and audio; results average ten runs.Source: §4.1.3 · p. 6
4.2 Baselines
- Text-only classification uses fine-tuned BERT, while the multimodal baselines are MAG-BERT, MulT, and MISA, with two-modality variants included for ablations.Two annotators label the unseen test set for the human-performance comparison.Source: §4.2 · p. 6
4.3 Results
- Multimodal fusion methods improve over the text-only baseline on all metrics for both twenty-class and binary classification, generally by over 1%.All baselines perform better on binary than twenty-class classification; the authors suggest coarse-grained intents are easier to recognize than fine-grained ones.Source: §4.3 · p. 6
- Human performance exceeds text-only performance by 14%–19% on twenty-class classification and 6%–7% on binary classification.The authors report that even the best-performing methods remain far from human performance.Source: §4.3 · pp. 6–7
5 DISCUSSION
The discussion reports gains from adding nonverbal modalities, with three-modality inputs generally outperforming two-modality inputs. Benefits vary across fine-grained intents, and human performance remains strongest in many categories.
5.1.1 Bi-modality.
- Text+audio raises twenty-class accuracy by 1.28%, 0.92%, and 0.72% for MAG-BERT, MulT, and MISA; text+video improves performance in all settings.For text+video, accuracy gains are 1.21%, 1.10%, and 0.65% on twenty-class classification, and 0.36%, 0.70%, and 0.65% on binary classification, respectively.Source: §5.1.1 · p. 7
5.1.2 Tri-modality.
- Three-modality inputs bring more gains than two-modality inputs, although MAG-BERT has a slight precision advantage and performs worse on other metrics.The authors report that both video and audio benefit the task and that the benchmark methods model cross-modal interactions.Source: §5.1.2 · p. 7
5.2 Performance of Fine-grained Intent Classes
- For emotion and attitude intents, multimodal methods improve most classes, with gains over 3% for complain, criticize, taunt, oppose, and joke, but help less or degrade performance on some text-cued classes.Apologise, thank, praise, and agree are examples of classes with clear textual cues.Source: §5.2 · p. 7
- For achieve-goals intents, multimodal methods improve every class by 1%–6%, consistent with the classes’ association with body movements that text alone captures less readily.Source: §5.2 · pp. 7–8
- Humans outperform text-only classification in most fine-grained categories, with over 10% improvements in many, but trail it on apologise, thank, and agree.The paper attributes the exceptions to classes biased toward textual information.Source: §5.2 · p. 8
6 RELATED WORKS
The related-work section surveys multimodal language datasets, intent benchmarks, fusion methods, and active-speaker detection. It positions MIntRec as a real-world dialogue dataset combining text, video, and audio for intent analysis.
6.1 Multimodal Language Datasets
- Prior multimodal language datasets cover tasks including sentiment, emotion, humor, metaphor, sarcasm, dialogue acts, and reasoning, while multimodal intent analysis in real-world dialogue scenes remains underrepresented.Source: §6.1 · p. 8
6.2 Benchmark Datasets for Intent Analysis
- Existing intent benchmarks largely use a single modality; MDID combines image and text, but its Instagram information and taxonomies are described as limited for real-world scenes.MIntRec instead provides text, video, and audio from dialogue scenes.Source: §6.2 · p. 8
6.3 Multimodal Fusion Methods
- Earlier fusion methods model intra- and inter-modality relations, while recent Transformer approaches include MulT, MISA, and MAG-BERT, which this work adapts as benchmark methods.Low-rank multimodal tensors were proposed to address the computational complexity of high-dimensional representations.Source: §6.3 · p. 8
6.4 Audio-visual Active Speaker Detection
- For automatic speaker annotation, the paper uses TalkNet, which combines audio-visual cross-attention with self-attention for long-term speech dependencies.Source: §6.4 · p. 8
7 CONCLUSIONS
MIntRec provides a new multimodal intent-recognition dataset with scene-specific taxonomies and an automatic process for speaker bounding boxes. Experiments find multimodal information superior to text alone, while a gap from human performance remains.
7 CONCLUSIONS
- Comprehensive experiments verify the superiority of multimodal information for intent recognition.The gap between the best-performing multimodal fusion methods and humans indicates the task remains challenging.Source: §7 · p. 8
- MIntRec contains 2,224 high-quality annotated samples with multimodal information and coarse- and fine-grained intent categories designed for real-world scenes.Source: §7 · p. 8
- An automatic process obtains speaker bounding-box information and reduces the annotation burden.Source: §7 · p. 8