Source-linked AI summary
Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency
TL;DR
Multimodal fusion seeks to integrate heterogeneous representations, but tensor-based methods can become exponentially costly. LMF uses modality-specific low-rank factors to make fusion efficient while retaining competitive performance across three tasks and reducing computational complexity.
Problem
Tensor-based multimodal fusion is expressive but can incur exponential computation and memory costs as modalities increase.
Method
LMF performs multimodal fusion with modality-specific low-rank factors and decomposed weight tensors.
Results
LMF achieves competitive performance on three multimodal tasks and uses fewer parameters and lower computational complexity than previous tensor-based methods.
Takeaways & Limitations
LMF scales linearly with the number of modalities and improves training and testing efficiency compared with TFN.
Takeaways & Limitations
The related fusion approaches discussed have modality-interaction limitations: early fusion may suppress intra-modal dynamics, while late fusion may inadequately model inter-modal interactions.
Abstract
from arXiv · showhide
Multimodal research is an emerging field of artificial intelligence, and one of the main research problems in this field is multimodal fusion. The fusion of multimodal data is the process of integrating multiple unimodal representations into one compact multimodal representation. Previous research in this field has exploited the expressiveness of tensors for multimodal representation. However, these methods often suffer from exponential increase in dimensions and in computational complexity introduced by transformation of input into tensor. In this paper, we propose the Low-rank Multimodal Fusion method, which performs multimodal fusion using low-rank tensors to improve efficiency. We evaluate our model on three different tasks: multimodal sentiment analysis, speaker trait analysis, and emotion recognition. Our model achieves competitive results on all these tasks while drastically reducing computational complexity. Additional experiments also show that our model can perform robustly for a wide range of low-rank settings, and is indeed much more efficient in both training and inference compared to other methods that utilize tensor representations.
1 Introduction
Multimodal fusion must combine heterogeneous modalities while controlling the computation and memory costs of tensor representations. The paper proposes Low-rank Multimodal Fusion (LMF) and evaluates its performance and efficiency across multimodal tasks.
- Multimodal fusion combines heterogeneous data to leverage complementary information and provide more robust predictions.
- Tensor-based multimodal representations improve performance but incur exponential computation and memory costs as modalities increase.
- LMF uses low-rank weight tensors and modality-specific factors to make multimodal fusion efficient without compromising performance.
- LMF is evaluated on three multimodal tasks using public datasets and compared with state-of-the-art models.
- The model is reported to be computationally efficient, with fewer parameters than previous tensor-based methods.
2 Related Work
Prior fusion strategies differ in how they capture intra- and inter-modal information, while tensor methods improve interaction modeling at high computational cost. LMF applies low-rank factors to retain tensor-based fusion benefits more efficiently.
- Early fusion concatenates modality features but can suppress intra-modal interactions and lose contextual or temporal dependencies.
- Late fusion combines outputs from separate modality models, but usually models inter-modal interactions ineffectively.
- Intermediate tensor methods model inter-modality interactions by forming tensor representations from unimodal features.
- Tensor representations can have exponentially increasing computational complexity because outer products create extremely high-dimensional tensors.
- LMF provides tensor-based multimodal representations with fewer parameters and lower computational complexity across sentiment, speaker-trait, and emotion tasks.
3 Low-rank Multimodal Fusion
Tensor fusion captures multimodal interactions but scales poorly because tensor dimensions and parameters grow exponentially with the number of modalities. Low-rank Multimodal Fusion replaces the full weight tensor with modality-specific low-rank factors, computes fusion without explicit tensorization, and scales linearly in modalities.
- 3.1 Multimodal Fusion using Tensor Representations: Tensor-based fusion maps outer products of unimodal representations through a linear layer to capture multimodal interactions.The input tensor has one dimension per modality, while the output dimension is represented by the extra weight-tensor dimension.
- 3.1 Multimodal Fusion using Tensor Representations: The input tensor and its weight tensor become exponentially larger as the number of modalities increases, increasing computation and overfitting risk.The tensor dimensionality grows as ∏M m=1 dm, and the weight tensor also increases exponentially.
- 3.2 Low-rank Multimodal Fusion with Modality-Specific Factors: LMF parameterizes the fusion layer with modality-specific low-rank factors that recover a low-rank weight tensor instead of using the full tensor.The method starts with a fixed rank r and decomposes each output slice of the weight tensor into rank factors.
- 3.2.1 Low-rank Weight Decomposition: The low-rank factors are grouped by modality, with shared output dimensions enabling decomposition into M modality-specific factors.The factors are constructed by taking outer products only over dimensions that are not shared.
- 3.2.2 Efficient Low-rank Fusion Exploiting Parallel Decomposition: Parallel decomposition of the input and weight tensors lets LMF compute fusion directly from unimodal representations without explicitly constructing the high-dimensional input tensor.The resulting operations are fully differentiable and can add a modality by adding another set of modality-specific factors.
- 3.2.2 Efficient Low-rank Fusion Exploiting Parallel Decomposition: LMF reduces tensorization and fusion complexity from exponential in the number of modalities to linear while implicitly combining tensorization and projection.The method avoids explicitly forming the large input and weight tensors and the associated linear transformations.
4 Experimental Methodology
The experiments evaluate Low-rank Multimodal Fusion against tensor-based and other baselines across three multimodal tasks, using aligned language, visual, and acoustic modalities with standard evaluation metrics.
- Experimental design: Experiments compare LMF with TFN and other baselines on sentiment analysis, speaker trait recognition, and emotion recognition.The study includes CMU-MOSI, POM, and IEMOCAP datasets.
- Evaluation: The study uses speaker-independent training, validation, and test splits to evaluate generalization.No identical speakers appear in both training and test sets.
- Datasets and features: Each dataset contains language, visual, and acoustic modalities aligned at the word level.Visual and acoustic features are averaged over each word interval after P2FA alignment.
- Model architecture: The model extracts acoustic and visual representations with two-layer feed-forward networks and language representations with an LSTM.These subnetworks produce unimodal representations before fusion.
- Evaluation: Evaluation covers multiclass classification and regression, reporting F1, accuracy, mean absolute error, and Pearson correlation.Accuracy is reported as Acc-k, where k denotes the number of classes; higher values are better except for MAE.
5 Results and Discussion
LMF achieves competitive or state-of-the-art performance across the evaluated tasks while reducing model size and improving computational efficiency relative to TFN. Low ranks are generally sufficient, whereas higher ranks produce unstable results.
- Impact of low-rank fusion: LMF significantly outperforms TFN across all datasets and metrics in the direct comparison.TFN has a similar structure but explicitly forms the multimodal tensor.
- State-of-the-art comparison: LMF achieves competitive and consistent results across sentiment analysis, speaker trait recognition, and emotion recognition datasets.The comparison includes state-of-the-art models and baselines.
- State-of-the-art comparison: LMF outperforms the previous state-of-the-art model on both MAE and Corr for multimodal sentiment regression.The multiclass accuracy calculation maps continuous sentiment values into discrete intervals.
- State-of-the-art comparison: LMF achieves state-of-the-art performance across all three evaluation metrics on the POM speaker-trait recognition task.The reported score is averaged over 16 speaker traits.
- State-of-the-art comparison: LMF achieves better F1 results than state-of-the-art models across all emotions on the emotion-recognition task.F1-emotion denotes the F1 score for a particular emotion class.
- Complexity analysis: Under M = 3, d1 = 32, d2 = 32, d3 = 64, r = 4, and dy = 1, LMF has about 1.1e6 parameters versus TFN’s 12.5e6.TFN therefore has nearly 11 times more parameters, counting fusion-stage and subnetwork parameters.
- Complexity analysis: LMF trains at 1134.82 IPS compared with 340.74 IPS for TFN on an NVIDIA Quadro K4200 GPU.The comparison measures data point inferences per second during training.
- Rank settings: As rank increases, training results become unstable, while very low rank achieves fairly competent performance on CMU-MOSI.Figure 4 reports this trend using mean absolute error.
6 Conclusion
The paper concludes that Low-rank Multimodal Fusion uses modality-specific low-rank factors to make tensor-based multimodal fusion more efficient. It scales linearly with the number of modalities while retaining competitive performance across tasks.
- Conclusion: LMF performs multimodal fusion with modality-specific low-rank factors and scales linearly with the number of modalities.The method changes the computational complexity from exponential to linear time.
- Conclusion: LMF achieves competitive results across different multimodal tasks while improving training and testing efficiency compared with TFN.The conclusion attributes the efficiency improvement to the low-rank fusion approach.
- Future work: Future work could apply low-rank tensors to attention models over tensor representations because those models may be more memory- and computation-intensive.This is proposed as a direction for related work.