Source-linked AI summary
Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, Barnabas Poczos
TL;DR
Multimodal sentiment models often require all modalities at test time, making their representations sensitive to noisy or missing inputs. This paper proposes MCTN, which learns robust joint representations through cyclic translations and uses source-modality input for prediction; it achieves state-of-the-art results across three datasets while becoming more discriminative with additional modalities.
Problem
Existing multimodal joint representations require multiple modalities as input and may therefore be sensitive to noisy or missing modalities at test time.
Method
MCTN learns joint representations by translating from source to target modalities with cyclic translation losses and a coupled translation-prediction objective.
Results
MCTN achieves new state-of-the-art results on CMU-MOSI, ICT-MMMO, and YouTube while learning increasingly discriminative representations with more input modalities.
Takeaways & Limitations
After training on paired multimodal data, MCTN requires only the source modality for sentiment prediction and remains robust to noisy or missing target modalities.
Abstract
from arXiv · showhide
Multimodal sentiment analysis is a core research area that studies speaker sentiment expressed from the language, visual, and acoustic modalities. The central challenge in multimodal learning involves inferring joint representations that can process and relate information from these modalities. However, existing work learns joint representations by requiring all modalities as input and as a result, the learned representations may be sensitive to noisy or missing modalities at test time. With the recent success of sequence to sequence (Seq2Seq) models in machine translation, there is an opportunity to explore new ways of learning joint representations that may not require all input modalities at test time. In this paper, we propose a method to learn robust joint representations by translating between modalities. Our method is based on the key insight that translation from a source to a target modality provides a method of learning joint representations using only the source modality as input. We augment modality translations with a cycle consistency loss to ensure that our joint representations retain maximal information from all modalities. Once our translation model is trained with paired multimodal data, we only need data from the source modality at test time for final sentiment prediction. This ensures that our model remains robust from perturbations or missing information in the other modalities. We train our model with a coupled translation-prediction objective and it achieves new state-of-the-art results on multimodal sentiment analysis datasets: CMU-MOSI, ICT-MMMO, and YouTube. Additional experiments show that our model learns increasingly discriminative joint representations with more input modalities while maintaining robustness to missing or perturbed modalities.
Introduction
Multimodal sentiment analysis seeks joint representations of language, visual, and acoustic information, but representations requiring all modalities can be sensitive to missing or noisy inputs. MCTN addresses this through cyclic modality translations and achieves state-of-the-art sentiment results while learning increasingly discriminative representations with more modalities.
- Motivation: Multimodal sentiment analysis extends text-based sentiment analysis by incorporating visual and acoustic information alongside language.Rich nonverbal behaviors can accompany spoken opinions, motivating multimodal representations.
- Motivation: Existing joint-representation methods require multiple modalities as input, making them sensitive to noisy or missing modalities at test time.This dependence creates the central robustness challenge addressed by the paper.
- Approach: MCTN learns joint representations by translating from a source modality to target modalities and applying cyclic translations in forward and backward directions.The method extends translations to hierarchical settings involving multiple target modalities.
- Approach: The learned representations retain joint information across modalities and support sentiment prediction from translation-based representations.Figure 1 illustrates two- and three-modality translation paths that produce representations for prediction.
- Results: MCTN achieves new state-of-the-art results on CMU-MOSI, ICT-MMMO, and YouTube multimodal sentiment analysis datasets.Additional experiments find increasingly discriminative joint representations with more input modalities during training.
Related Work
Prior multimodal sentiment work developed fusion, filtering, attention, memory, generative, and probabilistic approaches to represent multiple modalities and handle missing inputs. These methods include translating one modality into another and inferring missing modalities from learned probabilistic relationships.
- Joint Representation Learning: Earlier multimodal sentiment methods used feature concatenation and neural models to learn joint representations from multiple modalities.Subsequent models added filtering switches, attention, and memory mechanisms.
- Generative Approaches: Generative approaches learn joint distributions across modalities using GANs, conditional generative models, or modality translation.These approaches provide alternatives to purely supervised multimodal representation learning.
- Missing Modalities: Methods for noisy or missing modalities infer unavailable inputs by modeling probabilistic relationships among modalities.Deep Boltzmann Machines can sample conditional distributions for test-time inference with missing modalities.
Proposed Approach
MCTN learns joint multimodal representations through cyclic modality translation, coupling translation and sentiment prediction during training. At inference, it uses only the source modality, and extends hierarchically to additional target modalities while remaining robust to missing or noisy targets.
- Multimodal Cyclic Translation Network: MCTN translates between a source and target modality to learn a joint representation usable for sentiment prediction from only the source modality at test time.The model is trained end-to-end with a coupled translation-prediction objective.
- Multimodal Cyclic Translation Network: Cycle consistency encourages the learned representation to retain information from both modalities by translating to the target and back to the source.The translation model uses a cycle consistency loss alongside forward translation and prediction losses.
- Multimodal Cyclic Translation Network: The Seq2Seq translation model encodes the source sequence into an intermediate representation and decodes that representation into the target modality.The model accepts variable-length inputs and uses beam search for decoding.
- Coupled Translation-Prediction Objective: Training minimizes a coupled objective combining forward translation, cycle consistency, and sentiment prediction losses, with weighting hyperparameters for the translation terms.Translation losses use mean squared error between ground-truth and translated modalities, while prediction uses a label loss.
- Hierarchical MCTN for Three Modalities: Hierarchical MCTN learns representations from more than two modalities by translating the first joint representation into additional target modalities.For three modalities, the final representation combines the source, first target, and second target, while only the source is required at test time.
- Evaluation: MCTN achieves state-of-the-art sentiment results on CMU-MOSI, ICT-MMMO, and YouTube while using only language during testing on the reported comparison.The CMU-MOSI table caption states that MCTN outperforms the current state of the art across most evaluation metrics.
Experimental Setup
The experiments evaluate MCTN’s learned joint representations on CMU-MOSI, ICT-MMMO, and YouTube using established multimodal features, sentiment metrics, and baseline comparisons.
- Datasets: Experiments evaluate MCTN’s learned joint representations using CMU-MOSI, ICT-MMMO, and YouTube datasets.CMU-MOSI contains 2,199 video segments with sentiment labels in [−3,+3], and speaker overlap between training and testing is avoided.
- Features and alignment: Language, visual, and acoustic inputs use GloVe, Facet, and COVAREP features, respectively, aligned by spoken-word utterance intervals.Visual and acoustic features are averaged over each word’s utterance interval after forced alignment with P2FA.
- Metrics and objectives: CMU-MOSI optimization uses Mean Absolute Error, with MAE and Pearson’s r reported for regression and accuracy and F1 for classification.On ICT-MMMO and YouTube, categorical cross-entropy is used and sentiment classification and F1 are reported; lower MAE and higher values for other metrics indicate stronger performance.
- Results: MCTN outperforms the current state-of-the-art across most ICT-MMMO and YouTube evaluation metrics while using only language during testing.The table caption reports the cross-dataset comparison and language-only testing condition.
- Baselines: Baselines include RMFN, LMF, MFN, and EF-LSTM, representing hierarchical, low-rank tensor, gated-memory, and concatenation-based multimodal approaches.RMFN is identified as the current state-of-the-art on CMU-MOSI before comparison.
Results and Discussion
MCTN outperforms comparison models and benefits from adding modalities and cyclic translations during training. Ablations indicate that shared Seq2Seq models, language as the source modality, and hierarchical two-level translations are important design choices.
- Comparison with Existing Work: MCTN achieves state-of-the-art results on CMU-MOSI, ICT-MMMO, and YouTube while using only language during testing.Other baselines use all three modalities at test time.
- Adding More Modalities: Adding more modalities during cyclic-translation training improves performance and produces increasingly separable joint representations.The t-SNE analysis colors negative-sentiment videos red and positive-sentiment videos blue.
- Ablation Studies: Cyclic-translation models outperform non-cyclic bimodal and trimodal baselines, with the gap especially large for trimodal models.The authors attribute this to symmetry-based regularization and retention of information from all modalities.
- Ablation Studies: Using one shared Seq2Seq model for forward and backward translations performs better than using two separate models in bimodal and hierarchical trimodal settings.The authors hypothesize that two deep models require more data, risk overfitting, and learn separate representations.
- Ablation Studies: Language contributes most to the joint representations, and using language as the source modality gives the best performance across hierarchical translation settings.Language-visual combinations generally outperform language-acoustic combinations, while translating only visual and acoustic modalities sharply reduces performance.
- Ablation Studies: Two translation levels outperform directly translating a concatenated modality pair, indicating that both levels are important in hierarchical MCTN.The authors suggest recursive pairwise translation makes representation learning easier than a single translation across all modalities.
Conclusion
The paper learns joint representations through cyclic translations and requires only the source modality during testing, improving robustness to noisy or missing target modalities.
- Cyclic translations learn joint representations from source-to-target modality mappings.
- At test time, prediction uses only the source modality, maintaining robustness to noisy or missing target modalities.
- MCTN achieves new state-of-the-art results on three multimodal sentiment-analysis datasets.
- The model learns increasingly discriminative joint representations as more input modalities are used during training.
Multimodal Features
The paper extracts language, visual, and acoustic features to represent spoken multimodal data, using word embeddings for the language modality and Facet for visual features.
- Language features use 300-dimensional GloVe word embeddings trained on 840 billion Common Crawl tokens.
- The embeddings represent sequences of transcript words as sequences of word vectors.
- Facet is used to extract visual features.
Multimodal Alignment
The modalities are aligned through forced word-level timing, then video and audio features are averaged over each word’s utterance interval.
- P2FA forced alignment provides the exact utterance timestamp for each word and aligns the three modalities.
- Each word utterance interval defines one language time-step.
- Aligned video and audio features are computed as expectations over each word’s utterance interval.
- The evaluation compares MCTN with stacked, bidirectional, and stacked bidirectional LSTM baselines alongside other multimodal models.
Full Results
MCTN achieves state-of-the-art sentiment-prediction performance across CMU-MOSI, ICT-MMMO, and YouTube while using only language during testing.
- MCTN using all modalities achieves new state-of-the-art binary accuracy, F1 score, and MAE on CMU-MOSI.
- MCTN also achieves state-of-the-art results on ICT-MMMO and YouTube.
- On CMU-MOSI, MCTN outperforms the current state-of-the-art across most evaluation metrics and uses only language during testing.
- On ICT-MMMO and YouTube, MCTN outperforms the current state-of-the-art across most evaluation metrics and uses only language during testing.