Source-linked AI summary
Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
Zhongkai Sun, Prathusha Sarma, William Sethares, Yingyu Liang
TL;DR
Text features often outperform audio and video in multimodal sentiment and emotion tasks, motivating models that learn their hidden cross-modal relationships. ICCN uses outer-product text-based audio and video features with DCCA, and evaluations report state-of-the-art performance with ablations supporting its components.
Problem
Text features often outperform audio and video in sentiment and emotion tasks, motivating the study of hidden relationships between verbal and non-verbal information.
Method
ICCN forms text-audio and text-video outer products and uses CNNs within DCCA to learn correlated multimodal embeddings.
Results
Evaluations on three benchmark datasets report state-of-the-art multimodal sentiment and emotion performance, with ablations confirming the usefulness of the network’s components.
Takeaways & Limitations
ICCN captures useful information from text, audio, and video by learning correlations between text-based audio and text-based video features.
Takeaways & Limitations
The paper identifies room for improvement in modeling dynamic intra-actions and balancing maximum canonical correlation against downstream-task performance.
Abstract
from arXiv · showhide
Multimodal language analysis often considers relationships between features based on text and those based on acoustical and visual properties. Text features typically outperform non-text features in sentiment analysis or emotion recognition tasks in part because the text features are derived from advanced language models or word embeddings trained on massive data sources while audio and video features are human-engineered and comparatively underdeveloped. Given that the text, audio, and video are describing the same utterance in different ways, we hypothesize that the multimodal sentiment analysis and emotion recognition can be improved by learning (hidden) correlations between features extracted from the outer product of text and audio (we call this text-based audio) and analogous text-based video. This paper proposes a novel model, the Interaction Canonical Correlation Network (ICCN), to learn such multimodal embeddings. ICCN learns correlations between all three modes via deep canonical correlation analysis (DCCA) and the proposed embeddings are then tested on several benchmark datasets and against other state-of-the-art multimodal embedding algorithms. Empirical results and ablation studies confirm the effectiveness of ICCN in capturing useful information from all three views.
1 Introduction
Multimodal language analysis combines verbal, acoustic, and visual information, but text features often perform better than non-text features. ICCN addresses this imbalance by learning correlations between text-based audio and text-based video features with outer products and DCCA.
- Text often outperforms audio and video because it carries sentiment information, non-text cues can confuse labels, and text-analysis algorithms are more mature.
- Learning hidden relationships between verbal and non-verbal information is a central problem in multimodal language analysis.
- Outer products represent interactions between feature pairs as arrays that can be processed by convolutional neural networks.
- ICCN builds text-audio and text-video outer-product matrices, extracts features with CNNs, and learns their correlation using DCCA.
- The learned CNN outputs are combined with the original text to form multimodal embeddings for downstream tasks.
2 Related Work
Prior work studies multimodal fusion and cross-modal relationships using concatenation, tensor products, specialized neural architectures, and CCA-based methods. These approaches motivate ICCN’s combination of outer-product interactions with deep canonical correlation.
- The paper’s central themes are multimodal fusion embeddings and cross-modal relationship learning through CCA.
- Early multimodal fusion concatenated audio, video, and text embeddings, while later neural architectures sought to combine all three modalities more selectively.
- Tensor fusion uses outer products of text, audio, and video features to represent comprehensive multimodal interactions, but requires substantial computational resources.
- CCA learns maximum correlation between two variables by mapping them into a new subspace, and DCCA extends this with feedforward neural networks.
- Prior CCA-based studies apply these methods to relationships between domains, languages, word representations, and multimodal features.
3 Methodology
ICCN extracts utterance-level text, audio, and video features, forms text-based audio and video outer products, and uses CNNs within a DCCA objective to maximize their correlation. The resulting features are concatenated with text and passed to downstream classifiers.
- CCA and DCCA: CCA learns linear transformations of two vector sets so their transformed outputs have maximum correlation.
- CCA and DCCA: DCCA replaces CCA’s linear transformations with neural networks, allowing nonlinear transformations while optimizing canonical correlation.
- Text Based Audio Video Interaction Canonical Correlation: Utterance-level audio and video features are obtained from convolutional layers followed by LSTMs before multimodal interaction features are constructed.
- Text Based Audio Video Interaction Canonical Correlation: Text-based audio and text-based video matrices are formed by outer products of text with the corresponding audio and video utterance features.
- Text Based Audio Video Interaction Canonical Correlation: CNNs extract vectors from the two outer-product matrices, and backpropagated CCA loss updates the network weights to maximize their canonical correlation.
- Text Based Audio Video Interaction Canonical Correlation: The training pipeline computes audio and video features, forms both outer products, applies two CNNs, minimizes CCA loss, and outputs the learned vectors.
- Pipe-line for Downstream Tasks: The final embedding concatenates text-based audio, original text, and text-based video features for classifiers such as logistic regression or multilayer perceptrons.
- Pipe-line for Downstream Tasks: Figure 1 depicts independent unimodal extraction, DCCA alignment of text-based audio and video, and concatenation into a downstream-task embedding.
4 Experiment Settings
Experiments compare ICCN and multimodal embedding baselines on three benchmark datasets using shared downstream evaluation protocols and modality-specific features. Ablations test canonical correlation and text–audio/text–video interaction components.
- Datasets: ICCN is evaluated on CMU-MOSI, CMU-MOSEI, and IEMOCAP for multimodal sentiment analysis and emotion recognition.
- Datasets: CMU-MOSI contains 2,198 utterance clips from 93 YouTube movie reviews, labeled from -3 to 3 and split into 1,283 training, 229 validation, and 686 test samples.
- Multi-modal Features: Text features use pretrained BERT for CMU-MOSI and CMU-MOSEI, while IEMOCAP uses InferSent with a BiLSTM and max-pooling sentence encoder.
- Multi-modal Features: Audio features come from COVAREP and video features from Facet, representing acoustic properties and facial-expression features respectively.
- Baseline Methods: Comparisons include unimodal concatenation, Linear CCA, Kernel-CCA, GCCA, DCCA, MFM, TFN, and LMF, using the same downstream classifier or regressor for embedding evaluation.
- Ablation Studies: Ablations replace CCA loss with cosine-similarity loss or replace text–audio/text–video outer products with direct audio and video inputs to DCCA.
- Evaluation Metrics: CMU-MOSI and CMU-MOSEI use binary accuracy, F1-score, mean absolute error, and seven-class sentiment level or correlation; IEMOCAP uses binary accuracy and F1-score.
5 Discussion of Empirical Results
Across CMU-MOSI, CMU-MOSEI, and IEMOCAP, ICCN outperforms unimodal, concatenation, and other CCA-based methods, while matching or surpassing neural baselines. Ablations indicate that both text-based outer products and CCA loss contribute to its performance.
- Benchmark comparisons: ICCN outperforms unimodal features and simple concatenation across all reported criteria on CMU-MOSI, CMU-MOSEI, and IEMOCAP.The reported results also note that text features outperform audio and video, while simple concatenation does not work well.
- Benchmark comparisons: ICCN outperforms other CCA-based methods, whose authors attribute their weaker performance to limited neural-network capacity or omitted text-based interactions.The comparison includes CCA, KCCA, and GCCA, while DCCA without learned text-based audio and video interactions may sacrifice useful information.
- Benchmark comparisons: ICCN achieves better or similar results than neural-network state-of-the-art methods including TFN, LMF, and MFM.These comparisons are reported as evidence of ICCN’s competitive performance.
- Ablation studies: CCA loss performs better than cosine-similarity loss with or without the outer product in ICCN variants on CMU-MOSI and CMU-MOSEI.The experiments also show that maximizing canonical correlation does not necessarily increase cosine similarity, indicating distinct objectives.
- Ablation studies: Learning text-based interactions performs better than applying DCCA directly to audio and video, supporting the combination of text-based outer products with DCCA.The paper explains that audio and video are more correlated when represented relative to the same text.
6 Conclusion and Future Work
ICCN learns multimodal relationships through canonical correlation and achieves state-of-the-art performance on multimodal sentiment analysis and emotion recognition. The authors identify dynamic interactions, performance trade-offs, and interpretability as directions for improvement.
- ICCN uses canonical correlation to analyze hidden relationships among text, audio, and video.
- ICCN’s learned multimodal features achieve state-of-the-art performance on sentiment analysis and emotion recognition tasks.
- Ablation studies confirm the usefulness of different parts of the ICCN network.
- Future Work: Future work includes learning dynamic intra-actions alongside cross-modal interactions.
- Future Work: Future work also includes learning the trade-off between maximum canonical correlation and downstream task performance.
- Future Work: Developing an interpretable end-to-end multimodal canonical correlation model remains a future direction.