Source-linked AI summary
Multimodal Transformer for Unaligned Multimodal Language Sequences
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov
TL;DR
Multimodal language sequences are difficult to model because modalities may be sampled at different rates and exhibit long-range crossmodal dependencies. MulT addresses both challenges with directional pairwise crossmodal attention over unaligned streams, and experiments report state-of-the-art performance in aligned and unaligned settings. The results suggest that its attention captures correlated signals across asynchronous modalities.
Problem
Unaligned multimodal language sequences contain variable-rate streams and long-range crossmodal dependencies, making efficient multimodal fusion challenging.
Method
MulT is an end-to-end Transformer extension that uses directional pairwise crossmodal attention to latently adapt and fuse unaligned modalities.
Results
MulT achieves state-of-the-art results on aligned and unaligned benchmarks, outperforming prior approaches by 5%-15% on most metrics.
Takeaways & Limitations
Crossmodal attention can capture correlated signals across asynchronous modalities without relying on explicit alignment.
Takeaways & Limitations
The formulation assumes modality pairs can be represented as potentially non-aligned sequences with defined feature dimensions and lengths.
Abstract
from arXiv · showhide
Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise crossmodal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.
1 Introduction
Multimodal human language combines linguistic, visual, and acoustic signals whose variable sampling rates create non-alignment and whose interactions can span long ranges. MulT addresses these challenges end-to-end with crossmodal attention, achieving strong results in aligned and unaligned settings.
- Variable sampling rates across audio and vision can prevent optimal mappings between modality streams.The resulting heterogeneity complicates multimodal human-language analysis.
- Unaligned multimodal sequences require efficient fusion of long-range dependencies across modalities.
- MulT learns representations directly from unaligned multimodal streams without explicitly aligning the data.It extends the standard Transformer in an end-to-end model.
- Directional crossmodal attention attends across entire utterances and latently adapts one modality using features from another.The mechanism repeatedly reinforces one modality with information from other modalities regardless of alignment.
- MulT achieves state-of-the-art results in aligned and unaligned settings, outperforming prior approaches by 5%-15% on most metrics.Experiments cover CMU-MOSI, CMU-MOSEI, and IEMOCAP.
2 Related Works
Prior multimodal-language methods commonly assumed word-level alignment and modeled short-term interactions. MulT instead extends Transformer ideas through latent crossmodal adaptation and improves beyond standard Transformer on human multimodal-language tasks.
- Human multimodal language requires fusing time-varying language, visual, and acoustic signals rather than static multimodal attributes.
- Earlier approaches relied on word-level alignment and considered only short-term multimodal interactions.
- MulT draws inspiration from the Transformer but latently adapts elements across modalities instead of explicitly translating between them.Its modality sequences can have vastly different frequencies and are not represented as discrete word embeddings.
- MulT uses multiple stacks of pairwise and bidirectional crossmodal attention blocks that directly attend to low-level features.The architecture removes self-attention and does not use an encoder-decoder structure.
- MulT improves beyond standard Transformer on various human multimodal language tasks.
3 Proposed Method
MulT models unaligned multimodal language sequences by merging directional pairwise crossmodal interactions without explicit alignment. Its architecture uses temporal convolutions, positional embeddings, crossmodal transformers, and subsequent temporal modeling for prediction.
- Crossmodal Transformer Architecture: MulT merges multimodal time-series through multiple directional pairwise crossmodal transformers that repeatedly reinforce each target modality with another modality’s features.Each crossmodal transformer learns attention across two modalities’ features, and the full architecture models every pair of modalities.
- Crossmodal Attention: Crossmodal attention maps a target sequence to a weighted summary of source values, preserving the target length while representing outputs in the source value space.The attention score matrix compares every target time step with every source time step.
- Input Processing: Temporal convolutions capture local sequence structure and project modalities into a common dimension before crossmodal dot-products.This preprocessing accommodates modality-specific sampling rates and makes crossmodal attention dimensionally compatible.
- Input Processing: Positional embeddings augment the convolved modality features so the sequences retain temporal information.The resulting representations are low-level, position-aware features for language, video, and audio.
- Crossmodal Transformer Architecture: Each crossmodal transformer stacks D crossmodal attention blocks with residual connections, feed-forward sublayers, layer normalization, and multi-head attention.The blocks adapt directly from low-level source features rather than relying on self-attention over intermediate representations.
- Overall Architecture: For three modalities, MulT uses six directional crossmodal transformers, concatenates outputs sharing each target modality, and applies self-attention sequence models before prediction.This design treats non-alignment through crossmodal attention rather than manually aligning sequences; classical alignment is a monotonic special case.
4 Experiments
MulT is evaluated across three multimodal language benchmarks in word-aligned and unaligned settings, using modality-specific features and comparisons with prior fusion approaches. The experiments show strong performance and indicate benefits from crossmodal attention, especially for unaligned streams.
- Datasets and evaluation: Experiments cover CMU-MOSI, CMU-MOSEI, and IEMOCAP in both word-aligned and unaligned multimodal settings.The study compares MulT with prior competitive approaches across these benchmark datasets.
- Datasets and evaluation: Word-aligned sequences use P2FA word boundaries to average audio and vision features, whereas unaligned sequences retain the original streams.The aligned sequences have length 50; the unaligned version preserves the original asynchronous sampling structure.
- Quantitative analysis: MulT outperforms competitive approaches on nearly all word-aligned task metrics, except the IEMOCAP “sad” class.The comparison uses models with similar sizes of around 200K parameters.
- Quantitative analysis: 10%-15%: MulT improves upon prior methods with CTC on most attributes in the unaligned benchmark results.MulT also converges faster to better training results, and its performance drop from aligned to unaligned data is smaller than for other approaches.
- Ablation study: 77.4 versus 65.6: the language-only unimodal transformer improves the Acc2 metric over the audio-only model.The language transformer also outperforms the audio- and vision-only alternatives by a large margin.
- Ablation study: Crossmodal attention modules improve over late- and early-fusion transformers on most unaligned CMU-MOSEI metrics, with language as the target modality performing best.Adapting low-level source features performs best, and the ablations associate crossmodal attention with better representation learning.
- Qualitative analysis: MulT’s crossmodal attention assigns stronger attention to emotion-related words and large facial-expression changes across different time steps.This qualitative pattern supports direct modeling of potentially long-range crossmodal interactions without conventional alignment.
5 Discussion
MulT uses crossmodal attention to fuse multimodal information and capture long-range contingencies without requiring alignment. Its results also motivate exploring unaligned multimodal tasks beyond the evaluated setting.
- MulT uses crossmodal attention for latent adaptation, fusing multimodal information by attending directly to low-level features in other modalities.
- Unlike prior approaches focused primarily on aligned streams, MulT is designed to capture long-range contingencies regardless of the alignment assumption.
- The reported results suggest future applications for unaligned multimodal sequences, including Visual Question Answering with static and time-evolving signals.
A Positional Embedding
Because attention alone is order-invariant, positional information is embedded into transformer hidden units. MulT encodes positions with sinusoidal values and adds them to the input sequence.
- Attention-only transformers are order-invariant, so permuting an input sequence does not change their behavior or output.
- Positional embedding encodes a sequence X ∈R^T×d using sine and cosine functions whose frequencies depend on the feature index.
- Each feature dimension of PE follows a sinusoidal pattern, providing positional values across sequence positions.
- The computed positional embedding is added directly to the sequence, so X + PE carries position information at every time step.
B Connectionist Temporal Classification
CTC trains an alignment predictor to infer source-target alignments without supervision, then produces a pseudo-aligned source sequence for multimodal processing.
- CTC alignment predictors can be recurrent networks such as LSTMs that process source sequences and output alignment probabilities.
- For each source element, the predictor assigns probabilities over target words and an empty symbol x.
- CTC minimizes negative log-likelihood over proper alignment paths, including paths with blanks, repeats, or duplicated labels.
- Suboptimal or failure alignment cases are also considered in the CTC explanation.
- When CTC loss is minimized, the source signals are treated as properly aligned to the target signals.
- After training, blank probabilities are excluded and predictor outputs are multiplied with source signals to produce a pseudo-aligned target-length sequence.
C Hyperparameters
MulT configurations are kept near prior model sizes for fair comparison, with selected hyperparameters tuned by grid search and learning rates reduced when validation performance plateaus.
- MulT models are maintained at roughly the same sizes as prior work to support fair comparison.
- Dropout rate and crossmodal-attention head count are selected through a basic grid search.
- The learning rate is decayed by a factor of 10 when validation performance plateaus.
- Table 5 reports task-specific MulT hyperparameters, including crossmodal block and attention-head counts for each transformer.
D Features
The multimodal features comprise pretrained language embeddings, facial action units for vision, and low-level acoustic descriptors. These representations cover textual, facial, and acoustic behavior at 300, 35, and 74 dimensions, respectively.
- Language: Language transcripts are converted into 300-dimensional pretrained GloVe word embeddings.The embedding uses glove.840B.300d.
- Vision: Vision features use Facet to record 35 facial action units representing per-frame basic and advanced emotions.The action units capture facial muscle movement.
- Audio: Audio features are extracted with COVAREP and comprise 74 dimensions of acoustic descriptors.They include MFCCs, pitch and voiced/unvoiced features, glottal source parameters, peak slope parameters, and maxima dispersion quotients.