Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis

• MSA must fuse variable-rate unaligned audio, vision, and text while tolerating random feature loss, which can result from noise, occlusion, or recognition errors.MulT and PMR address unaligned input but retain quadratic scaling; TFR-Net reconstructs missing features but risks shortcut solutions and also scales quadratically.Source: §1 · pp. 1–2
• EMT uses utterance-level representations as global multimodal context to exchange information with local unimodal features, yielding practical linear complexity and performance gains.Source: §1 · p. 2
• DLFR combines low-level reconstruction that implicitly learns semantics with siamese high-level attraction between complete and incomplete views, which the paper describes as complementary.Source: §1 · p. 2
• On CMU-MOSI, CMU-MOSEI, and CH-SIMS, EMT and EMT-DLFR achieve state-of-the-art performance in complete and incomplete modality settings, respectively.Source: §1 · p. 2

2.1 MSA in the Complete Modality Setting

• Utterance-level fusion includes concatenation,,,, attention,,, and tensor methods,,,, while element-level methods model fine-grained interactions but often require manual alignment.Element-level approaches include recurrent methods,,, attention-based methods,, and multimodal-aware word embeddings,.Source: §2.1 · p. 2
• Transformer methods,,, [42],,,, handle unaligned sequences, but MulT, PMR, and TFR-Net have quadratic computational complexity over involved modalities.MulT uses directional pairwise cross-modal attention, PMR adds a message hub, and TFR-Net combines intra-modal and cross-modal Transformers.Source: §2.1 · pp. 2–3
• EMT-DLFR uses global multimodal context for linear scaling in practice and is reported to achieve better performance than prior local-local cross-modal interaction methods.The method's global context interacts with local unimodal features, which mutually promote each other.Source: §2.1 · p. 3

2.2 MSA in the Incomplete Modality Setting

• Incomplete-modality research is limited; prior approaches use feature ablation or modality perturbation, tensor-rank regularization, and modality translation, often requiring aligned inputs or addressing whole-modality loss.Low-level feature reconstruction has also been explored, while EMT-DLFR adds explicit high-level feature attraction; the paper reports that attraction outperforms reconstruction alone and works best in combination with it.Source: §2.2 · p. 3

3 METHOD

• In the incomplete-modality setting, EMT-DLFR uses complete and incomplete views, encodes global and local features, applies EMT, and uses DLFR to support robust sentiment prediction.The complete view supplies information for low-level reconstruction and high-level representation attraction.Source: §3 · p. 3
• EMT captures cross-modal interactions between global multimodal context and local unimodal features; utterance-level intra- and inter-modal features are combined for sentiment prediction.Modality-specific encoders first produce utterance-level global and element-level local intra-modal features.Source: §3 · p. 3
• DLFR simultaneously attracts high-level intra- and inter-modal representations across views and reconstructs low-level modality inputs from the complete view.The attraction is explicit in latent space, while reconstruction implicitly encourages semantic learning from incomplete data.Source: §3 · p. 3
• In the complete-modality setting, DLFR is omitted and training uses only the original prediction loss.Source: §3 · p. 3

3.1 Problem Definition

• MSA predicts a speaker’s video sentiment intensity from text, audio, and vision features, producing a score y ∈ R.Audio and vision inputs are shallow features from open-source tools; text inputs are raw tokens from the BERT tokenizer.Source: §3.1 · p. 3
• Random temporal masks create incomplete sequences: masked audio and vision vectors become zero vectors, while masked text tokens become BERT’s [UNK] token.The model is intended to integrate available multimodal information and predict sentiment intensity in both complete and incomplete settings.Source: §3.1 · p. 3

3.2 Unimodal Feature Encoder

• Text tokens are encoded with BERT, while audio and video sequences use LSTMs [53] to capture temporal dependencies.The text encoder follows prior works,,.Source: §3.2 · pp. 3–4
• In EMT, utterance-level modality representations form global context that interacts with local unimodal features through the Mutual Promotion Unit.The architecture figure identifies this as the efficient multimodal fusion mechanism.Source: Figure 1 · p. 4
• The text [CLS] embedding and the final audio and video time-step features serve as utterance-level representations; linear layers project sequence and utterance features to dimension d.These projections facilitate subsequent multimodal fusion.Source: §3.2 · p. 4

3.3.2 Mutual Promotion Unit

• MPU applies symmetric cross-modal attention to exchange information between feature sequences and self-attention to model temporal dependencies, allowing the sequences to mutually promote one another.Cross-modal attention uses queries from the target modality and keys and values from the source modality.Source: §3.3.2 · p. 4, §3.3.1 · p. 4

3.3.3 Multimodal Fusion Strategy

• OOLL and OALL both model local-local cross-modal interactions and have quadratic scaling over modalities; OALL uses a shared common message, while OOLL applies an MPU to each modality pair.In aligned inputs, both strategies have time complexity O(M^2T^2).Source: §3.3.3 · p. 5
• OAGL uses utterance-level representations as global multimodal context for local-feature interaction, enabling mutual refinement with linear space complexity and computation scaling over modalities.The method is motivated by redundancy in local features, especially high-sampling-rate audio and video, and the risk that excessive local features distract attention or encourage spurious correlations.Source: §3.3.3 · p. 6

3.3.4 Hierarchical Parameter Sharing

• Hierarchical parameter sharing shares parameters within directional sub-MPUs, across modalities, and across layers, improving parameter efficiency and making EMT easier to optimize.The paper also describes the three sharing levels as MPU-level, modality-level, and layer-level sharing.Source: §3.3.4 · p. 6

3.4 Prediction Module

• Flattened final global context yields g ∈R3d, which is concatenated with utterance-level intra-modal representations and passed through an MLP to predict sentiment using L1 loss.The prediction loss follows previous works,.Source: §3.4 · p. 6

3.5 Dual-Level Feature Restoration

• DLFR uses the complete view only during training to guide incomplete-view learning through low-level reconstruction and high-level representation attraction.The complete view is unavailable at test time.Source: §3.5 · p. 6

3.5.1 Low-Level Feature Reconstruction

• LLFR decodes fused incomplete-sequence representations to reconstruct complete modality features, using BERT output embeddings rather than raw text tokens and smooth L1 loss.For text, BERT embeddings avoid a vocabulary projection with about 23.4M parameters.Source: §3.5.1 · p. 6

3.5.2 High-Level Feature Attraction

• To avoid collapse risk from direct cosine attraction,, DLFR uses symmetric SimSiam projector-predictor losses with stop-gradient to align complete and incomplete utterance- and modality-level representations.Prediction losses for complete-view intra- and inter-modal representations are also added to ensure their quality.Source: §3.5.2 · p. 7

3.6 Overall Loss Function

• In the incomplete-modality setting, the task loss is the prediction loss of the incomplete view.It is defined as Ltask = |y−˜y′|.Source: §3.6 · p. 7
• The weights λ1 and λ2 balance the contribution of dual-level feature restoration to the overall loss.Source: §3.6 · p. 7
• In the complete-modality setting, dual-level feature restoration is unnecessary, so the overall loss equals the task loss.It is defined as L = Ltask = |y −˚y′|.Source: §3.6 · p. 7

4.1 Datasets

• The evaluation uses CMU-MOSI (2,199 segments), CMU-MOSEI (23,453), and CH-SIMS (2,281), using only CH-SIMS’ multimodal annotations.CMU-MOSI and CMU-MOSEI are English benchmarks; CH-SIMS is Chinese and includes both multimodal and unimodal annotations.Source: §4.1 · p. 7

4.2 Evaluation Metrics

• Regression is assessed with MAE and Corr, alongside dataset-specific discrete accuracy metrics and F1; MOSI/MOSEI binary scores distinguish negative/non-negative from negative/positive.CH-SIMS uses Acc-5, Acc-3, and Acc-2; all metrics except MAE are higher-is-better.Source: §4.2 · pp. 7–8
• Incomplete-setting performance is summarized with AUILC across missing rates 0.1–1.0 for MOSI/MOSEI and 0.1–0.5 for CH-SIMS.AUILC aggregates performance for each metric as missing rates increase.Source: §4.2 · p. 8

4.3 Feature Extraction

• Experiments use official unaligned features, with BERT text encodings and dataset-specific acoustic and facial feature extraction.MOSI/MOSEI use COVAREP and Facet3; CH-SIMS uses Librosa and OpenFace 2.0.Source: §4.3 · p. 8

4.4 Implementation Details

• The model is trained in PyTorch with Adam and early stopping, tuned by random search, and averaged over five complete-setting runs or three incomplete-setting runs.Early stopping uses patience 8; dataset-specific hyperparameters are summarized in Table 3.Source: §4.4 · p. 8

4.5 Baselines

• Comparisons cover general MSA and random-missing-feature baselines, including TFR-Net, which uses intra- and inter-modal attention with feature reconstruction.The complete- and incomplete-setting evaluations use baselines selected to assess both general performance and missing-modality robustness.Source: §4.5 · pp. 8–9

5.1.1 Complete Modality Setting

• In complete modality settings, EMT surpasses Transformer-based and non-Transformer-based competitors on CMU-MOSI, CMU-MOSEI, and CH-SIMS.It beats TFR-Net by 0.016 MAE on CMU-MOSI and 2.2% Acc-7 on CMU-MOSEI, and MulT by 0.141 MAE on CMU-MOSI and 0.043 Corr on CMU-MOSEI; on CH-SIMS it beats Self-MM by 0.015 MAE, 0.022 Corr, and 1.5% Acc-2.Source: §5.1.1 · p. 9

5.1.2 Incomplete Modality Setting

• As missing rates increase, all methods degrade quasi-linearly, while EMT-DLFR outperforms competitors in most cases, especially at higher rates.The same missing rate is applied across all three modalities in training and testing, with independent random temporal masks for each modality.Source: §5.1.2 · p. 9, Figure 4 · p. 10, Figure 5 · p. 10
• EMT-DLFR leads the incomplete-setting AUILC comparisons across CMU-MOSI, CMU-MOSEI, and CH-SIMS.On CMU-MOSI, it beats the second performer by 0.050 MAE, 0.034 Corr, 4.7% Acc-7, 5.1% Acc-5, and 3.9%/4.1% F1; on CMU-MOSEI it achieves 0.020 higher performance on MAE and 0.032 on Corr; on CH-SIMS it outperforms the previous state-of-the-art by 0.015 MAE and 0.025 Corr.Source: §5.1.2 · pp. 10–11, Table 6 · p. 11, Table 7 · p. 11

5.2.1 Multimodal Fusion Strategy

• OAGL outperforms OOLL and OALL while requiring the fewest fusion MACs, supporting efficient global-local cross-modal interaction.OAGL uses 5× fewer MACs than OALL and 2× fewer than OOLL; its computational complexity scales linearly with modalities, unlike the quadratic scaling of OOLL and OALL. Serial variants perform slightly worse than the parallel designs.Source: §5.2.1 · pp. 11–12, Table 8 · p. 12

5.2.2 Hierarchical Parameter Sharing for EMT

• All-shared EMT retains comparable performance to the full model while reducing EMT parameters by 6M.The all-shared model has 0.5M EMT parameters, compared with the text BERT encoder’s 109.5M parameters; the authors suggest parameter sharing can regularize training.Source: §5.2.2 · p. 12, Table 9 · p. 12

5.2.3 Pooling Layer in EMT

• Attention pooling performs better than average pooling, while MLP-based pooling performs worst among the three pooling choices.Average pooling cannot account for differing modality contributions; MLP-based pooling has more parameters than attention pooling, and the authors suggest it may overfit.Source: §5.2.3 · p. 12, Figure 8 · p. 12

5.2.4 Dual-level Feature Restoration

• Both low-level reconstruction and high-level feature attraction improve robustness, and their combination gives EMT-DLFR the best ablation performance.High-level attraction is more important: EMT-LLFR underperforms EMT-HLFR by 0.022 MAE, 0.034 Corr, 2.0% Acc-7, and 0.9% Acc-2. Vanilla EMT also slightly outperforms TFR-Net on MAE, Acc-7, and Acc-2.Source: §5.2.4 · p. 13, Figure 9 · p. 13

5.2.5 Sensitivity of Loss Weights

• Loss-weight combinations have comparable performance, but excessive weights may hurt results; using λ1 = 1.0 and λ2 = 1.0 gives the best performance.Performance generally decreases when SimSiam is removed from siamese representation learning, supporting its role in preventing collapsed representations.Source: §5.2.5 · p. 13, Figure 10 · p. 13, §5.2.6 · p. 13

6 CONCLUSION

• Experiments on three datasets show that EMT-DLFR achieves state-of-the-art performance in both complete and incomplete modality settings.Empirical visualizations also show interpretable and robust cross-modal correlation signals for sentiment prediction.Source: §6 · p. 15
• EMT uses global multimodal context for efficient global-local cross-modal interaction, avoiding the quadratic scaling cost of local-local methods while improving performance.Hierarchical parameter sharing further improves parameter efficiency and eases training.Source: §6 · p. 15
• For random modality feature missing, DLFR combines implicit low-level feature reconstruction with explicit high-level feature attraction, which are complementary despite reconstruction being more effective.Together, the strategies support robust representation learning from incomplete multimodal data.Source: §6 · p. 15
• Future work could apply EMT-DLFR to entire-modality-missing, noisy-modality, and mixed noise-missing settings.The conclusion identifies these as more challenging settings for further study.Source: §6 · p. 15
• Further research could test EMT-DLFR on other multimodal tasks and datasets and explicitly incorporate modality importance to improve performance..Source: §6 · p. 15
• Although EMT scales linearly with the number of modalities, its input-sequence complexity remains O(MT^2); reducing it to O(MT) is an open research direction.The conclusion also notes that EMT-DLFR does not currently account for the importance of each modality.Source: §6 · p. 15