PaperlayerView exact revision on arXiv

Source-linked AI summary

Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective

Jiaqi Zhang, Zheng Pang, Mengting Li, Yiqi Wang, Guangyuan Dong, Chao Xue, Yusen Wu, Zihao Li, Huy Phan, Sicheng Zhao, Björn W. Schuller, Jiachen Luo

arXiv:2608.30563v1cs.CV

TL;DR

Missing-modality MER methods often treat heterogeneous missing information holistically, mixing recoverable shared semantics with uncertain modality-specific details. PriMD disentangles these components, stores modality-specific primitives, and retrieves them during teacher-student compensation. Across three benchmarks and diverse missing-modality settings, it reports stronger robustness and performance than prior methods.

  • Problem

    Existing incomplete MER methods often generate, align, or distill missing modalities holistically, mixing relatively recoverable shared semantics with uncertain modality-specific details.

  • Method

    PriMD disentangles shared semantics from modality-specific representations, discretizes the latter into primitive memory banks, and dynamically retrieves primitives for constrained student compensation.

  • Results

    PriMD outperforms prior methods under diverse missing-modality settings across three benchmarks, with especially large gains when two modalities are absent.

  • Takeaways & Limitations

    PriMD provides stable representations and predictions by estimating shared semantics while constraining compensation for uncertain modality-specific information.

  • Takeaways & Limitations

    PriMD relies on full-modality teacher data; scarce, biased, or costly full-modality samples may leave primitive memory banks incomplete for the target domain.

Abstract

from arXiv · show

Multimodal Emotion Recognition (MER) systems often suffer from missing modalities in real-world scenarios. Existing methods usually generate, align, or distill missing modalities as a whole, overlooking the heterogeneous nature of the information carried by each modality. Such holistic treatment mixes inferable shared semantics with uncertain modality-specific details, yielding unstable representations and degrading robustness. To address this issue, we propose the Primitive Memory Distillation (PriMD) framework. Unlike existing methods, PriMD takes an intra-modal perspective and focuses on how different types of information within a modality differ in recoverability within each modality. PriMD first disentangles cross-modal shared semantics from modality-specific representations, and then discretizes the latter into learnable semantic primitives to construct modality-specific memory banks. When modalities are missing, PriMD is a teacher-student framework that the student model uses the shared semantics of available modalities as queries to dynamically retrieve primitives. It compensates for missing modality-specific information within a constrained memory space and aligns with the teacher model. Extensive experiments on IEMOCAP, CMU-MOSI, and CMU-MOSEI demonstrate that PriMD achieves state-of-the-art performance and consistently stronger robustness across a wide range of missing-modality settings, while mitigating the instability caused by holistic feature inference. Our code and project website are available at https://github.com/JiaqiZhang-Sengoku/PriMD and https://jiaqizhang-sengoku.github.io/PriMD/, respectively.

1 Introduction

Incomplete MER must handle missing modalities whose shared semantics are relatively recoverable but whose modality-specific details are uncertain. PriMD addresses this by disentangling these information types and compensating for missing specific information through semantic primitives.

  • Missing modalities caused by privacy constraints, sensor failures, or low-quality inputs weaken the robustness of multimodal emotion recognition models.
  • Existing completion, representation-learning, and distillation methods process missing modalities holistically, which can mix shared semantics with uncertain modality-specific details.
  • Shared semantics are generally more reliably estimated across modalities, whereas modality-specific details are tied to one expression form and are more uncertain.
  • PriMD uses SSSD to disentangle shared and modality-specific features, then DPMC and DRAD to construct and retrieve semantic primitives for constrained compensation.
  • PriMD formulates incomplete MER as shared-semantic estimation plus constrained compensation of modality-specific information.
  • Across three benchmarks, PriMD outperforms prior methods under diverse missing-modality settings, with especially large gains when two modalities are absent.

2 Related Work

Incomplete multimodal emotion recognition methods use completion, robust representation learning, or knowledge distillation to address missing modalities. However, holistic processing can introduce irrelevant or conflicting features when available modalities cannot infer missing modality-specific details.

  • Multimodal emotion recognition integrates heterogeneous signals such as text, speech, and vision to learn discriminative emotion representations.
  • Incomplete MER methods are commonly categorized as modality completion, robust representation learning, and knowledge distillation.
  • Modality completion recovers missing features through generators, reconstruction networks, or cross-modal mappings, while representation learning maps missing-modality combinations into a unified latent space.
  • Knowledge distillation transfers supervision from full-modality teachers to missing-modality students through approaches including self-distillation, correlation distillation, and hierarchical mutual-information alignment.
  • Holistic generation, alignment, and distillation pay limited attention to shared and modality-specific information, potentially introducing irrelevant or conflicting features.

3 Methodology

PriMD uses a full-modality teacher to disentangle shared and modality-specific information, discretize the latter into primitive memory banks, and guide a missing-modality student through dynamic retrieval and distillation.

  • Framework overview: PriMD formulates incomplete MER as shared-semantic estimation plus constrained compensation of modality-specific information.The framework separates relatively recoverable cross-modal semantics from more uncertain modality-specific details.
  • Shared-Specific Semantic Decoupling: SSSD projects each modality’s encoded features into shared semantic and modality-specific representations, encouraging cross-modal agreement and statistical independence between them.Shared representations are aligned across modalities, while HSIC-based constraints suppress dependence between shared and modality-specific features.
  • Discrete Primitive Memory Construction: DPMC quantizes modality-specific representations into a finite set of learnable semantic primitives and constructs modality-specific memory banks.Vector quantization with a commitment constraint fits primitives to modality-specific feature distributions, after which the banks are frozen for student retrieval.
  • Dynamic Retrieval-Augmented Distillation: For each missing modality, DRAD fuses available-modality shared semantics, scores memory-bank primitives, and dynamically selects a retrieval number before aggregating compensation features.The retrieval-number predictor adapts the number of primitives to the available semantics and sample conditions.
  • Joint Optimization: The student concatenates available and compensated representations, predicts logits, penalizes excessive retrieval, and distills knowledge from the frozen full-modality teacher.Training uses a two-stage optimization: first the teacher and memory banks are learned with complete inputs, then the student is trained under missing-modality masks.

4 Experiments

Experiments on IEMOCAP, CMU-MOSI, and CMU-MOSEI evaluate PriMD under inter-modal and intra-modal missingness, ablations, efficiency, and sensitivity settings. PriMD consistently improves robustness and performance through shared-specific disentanglement and constrained primitive-based compensation.

  • Inter-modal Missingness: PriMD achieves the best average performance across six missing-modality combinations, including 77.43% WA and 77.28% UA on four-class IEMOCAP.It also reaches 58.13% WA/54.62% UA on six-class IEMOCAP and 79.81%/79.57% ACC/F1 on CMU-MOSEI.
  • Intra-modal Missingness: PriMD maintains higher F1 scores and lower MAE than most methods under most intra-modal missing-rate settings on CMU-MOSI and CMU-MOSEI.Missing rates increase gradually from 0.0 to 0.9; most methods degrade as missingness increases.
  • Ablation Study: Removing SSSD reduces average performance to 74.36% WA and 74.05% UA, showing that entangling shared and modality-specific information weakens robustness.The ablation replaces two projection branches with a single projection and directly mixes the two information types.
  • Decomposition and Compensation: PriMD improves WA/UA from 75.02%/74.81% to 77.43%/77.28% over Holistic-PriMD and reduces cross-combination JS/KL from 0.091/0.121 to 0.066/0.081.Under the same SSSD framework, discrete retrieval also achieves normalized within-class dispersion of 0.64 versus 1.00 for continuous compensation.
  • Efficiency and Sensitivity: PriMD reduces parameter count, training time, and peak CUDA memory by 79.7%, 51.4%, and 82.2%, respectively, while increasing inference latency by 8.1%.Sensitivity analyses show stable performance across a wide range of C and β, while Kmax must balance complementary evidence against weakly relevant primitives.

5 Conclusion

PriMD addresses incomplete multimodal emotion recognition by separating shared and modality-specific information, then using primitive memories for constrained missing-modality compensation. Experiments show robust performance under both inter-modal and intra-modal missingness.

  • PriMD decomposes multimodal representations into cross-modal shared semantics and modality-specific information before discretizing the latter into primitive memories.
  • Under missing modalities, the student retrieves relevant modality-specific primitives using shared semantics from available modalities.
  • The framework combines constrained compensation for uncertain modality-specific cues with distillation from a full-modality teacher.
  • PriMD achieves robust performance under both inter-modal and intra-modal missingness while alleviating instability from holistic feature inference.

Limitations

PriMD depends on full-modality data for learning shared-specific representations and primitive memories, and its experiments mainly simulate fixed sample-level missingness. These constraints leave coverage of target-domain primitives and online missingness as open boundaries.

  • PriMD relies on a full-modality teacher, so scarce, biased, or costly full-modality samples may limit primitive coverage in the target domain.
  • The experiments mainly use sample-level fixed missing masks rather than changing modality availability over time.
  • Real applications may involve temporary sensor failures or gradually degrading input quality during continuous interaction.

A Datasets and Evaluation Metrics

The evaluation uses IEMOCAP, CMU-MOSI, and CMU-MOSEI with task-specific emotion metrics and established comparison protocols. Inter-modal missingness uses six fixed modality combinations, while intra-modal missingness varies masking rates from 0.0 to 0.9.

  • Datasets: IEMOCAP contains five dyadic sessions with ten actors and supports two utterance-level multiclass emotion classification tasks.
  • Datasets: CMU-MOSI contains 2,199 YouTube opinion clips, while CMU-MOSEI contains 22,856 clips from diverse topics, environments, and speakers.
  • Evaluation Metrics: IEMOCAP uses WA and UA, whereas intra-modal CMU-MOSI and CMU-MOSEI experiments report weighted F1 and MAE.
  • Evaluation Protocol: Compared methods are evaluated under matched pre-extracted features, data splits, metrics, and protocols for fair comparison.

B.2 PriMD Implementation

PriMD’s implementation follows established missingness protocols and uses repeated evaluation procedures across datasets. The framework is trained with PyTorch and CUDA on an NVIDIA RTX 5090 GPU.

  • Inter-modal missingness uses six fixed modality combinations, while intra-modal missingness increases from 0.0 to 0.9 in increments of 0.1.
  • IEMOCAP uses five-fold Leave-One-Session-Out cross-validation, providing subject-independent evaluation because session actors do not recur elsewhere.
  • CMU-MOSI and CMU-MOSEI experiments are repeated five times, with averaged results reported.
  • PriMD is trained using PyTorch 2.7.0 with CUDA 12.8 on an NVIDIA RTX 5090 GPU with 32 GB of memory.

C.1 Noise Robustness

PriMD is evaluated under controlled feature noise and through component and retrieval-strategy analyses. The results indicate stable performance under combined noise and missing modalities, while dynamic retrieval is assessed against fixed retrieval budgets.

  • Noise robustness: PriMD maintains more stable average performance than compared methods across CMU-MOSI, CMU-MOSEI, and Four-Class IEMOCAP under varying noise levels.Under strong noise with σ = 20, it achieves 53.67%/53.91% ACC/F1 on MOSI, 61.93%/61.68% ACC/F1 on MOSEI, and 34.50%/34.14% WA/UA on IEMOCAP.
  • Noise robustness: 53.67%/53.91% ACC/F1 on MOSI, 61.93%/61.68% ACC/F1 on MOSEI, and 34.50%/34.14% WA/UA on IEMOCAP are achieved under strong noise with σ = 20.These results outperform the compared methods overall when noise interference and modality missingness coexist.
  • Ablation studies: DPMC ablations compare removing primitive memory, removing discretization, sharing one memory bank, and using a randomly initialized memory bank.The study is designed to analyze the effects of DPMC and its internal mechanisms.
  • Retrieval strategy: Dynamic retrieval adaptively selects primitives according to sample characteristics under a maximum retrieval budget of Kmax = 16, unlike fixed Top-K strategies.Fixed strategies use K = 1, 2, 4, 8, or 16 primitives for every sample.

D.2.2 Retrieval Cost Functions

The retrieval-cost analysis shows that cost functions regulate primitive usage while preserving recognition performance. Exponential cost achieves the strongest reported WA/UA balance, and L2 alignment performs best among the tested shared-alignment objectives.

  • Retrieval cost functions: 77.31%/77.16% WA/UA is obtained with linear cost at an expected retrieval size of 5.9, compared with 77.07%/76.91% without a cost constraint.The unconstrained setting uses 10.6 primitives on average.
  • Retrieval cost functions: 77.43%/77.28% WA/UA is achieved with exponential retrieval cost, using 5.2 primitives on average.This configuration has the highest reported performance among the compared cost functions.
  • Retrieval cost functions: The exponential cost provides the best balance between recognition performance and primitive usage under the current setting.It reduces expected primitive usage while achieving the highest reported WA/UA.
  • Shared-alignment objectives: 77.43%/77.28% WA/UA is achieved by L2 alignment, slightly exceeding the 77.16%/76.98% obtained by cosine alignment.InfoNCE and mutual-information alignment yield lower performance under the naturally paired multimodal setting.
  • Representation analysis: SSSD produces highly similar shared representations across modalities while keeping shared and specific representations clearly separated.The analysis also indicates that modality-specific representations retain nonidentical modality information.

E.2 Primitive Retrieval Patterns

Primitive retrieval is organized by emotion category and differs across modality-specific memory banks. The visualizations further show that PriMD's modules improve class separability in text-only fused representations.

  • Retrieval patterns: Different emotion categories activate distinct high-weight regions across primitive IDs within each modality-specific memory bank.Happy, Sad, Neutral, and Angry correspond to different retrieval patterns.
  • Retrieval patterns: Audio, text, and visual memory banks exhibit nonidentical activation patterns, indicating modality-specific primitive distributions.Retrieval weights are distributed across multiple primitives rather than concentrated on a few fixed primitives.
  • Retrieval patterns: The primitive memory provides differentiated modality-specific information compensation for different emotion categories.This conclusion follows from category-related and diverse retrieval patterns across the memory banks.
  • Fused representations: Complete PriMD representations show more compact same-class clusters and fewer overlaps than representations from ablated variants in text-only t-SNE visualizations.Removing SSSD, DPMC, or DRAD produces more scattered, less distinct, or more entangled class distributions.
Loading 2608.30563v1…