PaperlayerView exact revision on arXiv

Source-linked AI summary

MERP: A Latent-Bottleneck Framework for Robust Multi-Modal Multi-Label Emotion Recognition

Chuangxin Zhao, Meimei Zhang, Canran Xiao, Jiachen Luo, Jiajun He, Kaijie Chen, Huy Phan, Tomoki Toda, Biörn W. Schuller

arXiv:2610.29430v1cs.AIcs.SD

TL;DR

MMER models must preserve evidence for multiple correlated emotions when text, visual, or acoustic streams are unavailable. MERP reasons over a fixed latent bank with class-specific evidence mechanisms, and under 20% test-time modality drop it achieves the best reported micro-F1 on both evaluated datasets.

  • Problem

    MMER must preserve distinct evidence for multiple correlated emotions as modalities disappear, while repeated processing of full input streams remains tied to raw sequence length.

    Source: §I · p. 2

  • Method

    MERP compresses available multimodal evidence into a fixed latent workspace and combines cross-level class-specific readouts, prototype-guided noise-gated enhancement, and confidence-aware class-wise fusion.

    Source: §I · p. 2

  • Results

    0.563 on CMU-MOSEI and 0.780 on M3ED: MERP achieves the best micro-F1 under 20% test-time modality drop.

    Source: §I · p. 2, B. Main Results under Modality Drop · p. 7

  • Takeaways & Limitations

    The results suggest that protecting task-relevant evidence after compact aggregation can be more effective than reconstructing missing streams or assigning a single reliability score.

    Source: §VI · p. 10

Abstract

from arXiv · show

Multimodal multi-label emotion recognition is important for affect-aware systems, but models trained with complete text, visual, and acoustic streams often degrade sharply when some streams are unavailable at deployment. We address this train-test gap by learning a compact representation that preserves emotion-specific evidence from any available subset of modalities. We propose the Multi-modal Emotion Recognition Perceiver (MERP), which maps the joint sequence into a fixed-size latent bank, thereby keeping repeated reasoning independent of raw input length; it extracts label-specific evidence before and after latent reasoning and aligns the two levels through reconstruction. MERP further uses positive and negative class prototypes, a noise-gated feature enhancer with frequency-shaped perturbations during training, and confidence-aware class-wise fusion to strengthen separability, reduce sensitivity to missing inputs, and combine complementary predictors. Under 20 percent test-time random modality drop, MERP achieves micro-F1 scores of 0.563 on CMU-MOSEI and 0.780 on M3ED, exceeding the strongest evaluated baselines by 0.004 and 0.005, respectively; in the severe CMU-MOSEI audio-only setting, it retains 0.473 micro-F1 versus 0.022 for LDDU. These results show that fixed latent reasoning provides a practical, testable route to robust multimodal emotion recognition without requiring missing-modality examples during training. Code is available at https://github.com/Creative-zcx/MERP.

I. INTRODUCTION

Multi-modal multi-label emotion recognition must preserve evidence for several correlated emotions as modalities disappear, while repeated processing of full input streams remains costly. MERP uses a fixed latent workspace and class-specific reasoning; under 20% test-time modality drop, it achieves the best reported micro-F1 on both benchmarks.

A. Multi-Modal Multi-Label Emotion Recognition

  • MMER must distinguish several correlated labels, whereas many related categorical emotion models assign one emotion per instance.Existing MMER approaches also retain input-sequence-dependent computation, even when label-specific decoding or late probabilistic fusion improves prediction.Source: §I · pp. 1–2

B. Robust Learning with Incomplete Modalities

  • Incomplete-input methods commonly use imputation, alignment, denoising, or missing-pattern adaptation, but do not resolve the tension between preserving multiple labels and limiting repeated full-stream processing.Recovery can propagate cross-modal prediction error and adds auxiliary paths whose cost and reliability depend on the missing pattern.Source: §I · p. 2, B. Robust Learning with Incomplete… · pp. 2–3

I. INTRODUCTION

  • MERP compresses available multimodal evidence into a fixed latent workspace and extracts and reinforces emotion-specific evidence across representation levels.The same predictor handles complete and incomplete inputs while bounding deeper computation independently of raw sequence length.Source: §I · p. 2
  • MERP’s design combines a fixed latent bottleneck, cross-level class-specific readouts, prototype-guided noise-gated enhancement, and confidence-aware class-wise fusion.The paper frames robust MMER around preserving class-specific evidence rather than recovering absent modalities or assigning one reliability score per stream.Source: §I · p. 2
  • 0.563 on CMU-MOSEI and 0.780 on M3ED: MERP achieves the best micro-F1 under 20% test-time modality drop.It leads LDDU throughout p_miss ∈[0, 0.5] and avoids its compound-failure collapse, with 5.28 M parameters and 5.28 ms latency on CMU-MOSEI.Source: §I · p. 2

C. Latent-Space Multi-Modal Fusion

Latent-bottleneck fusion decouples repeated processing from raw input length, but existing approaches do not ensure that compact representations preserve evidence for co-occurring emotions. The preliminary study diagnoses how complete-input fusion behaves under modality loss using a frozen six-layer Transformer probe.

C. Latent-Space Multi-Modal Fusion

  • MERP connects fixed-size latent computation to label-attentive readouts, cross-level consistency, and class-wise fusion to preserve separable evidence for co-occurring emotions.Earlier fusion designs retain computation coupled to the number and length of input streams.Source: C. Latent-Space Multi-Modal Fusion · p. 3

A. Diagnostic Protocol

  • The diagnostic uses a six-layer Transformer fusion model trained on complete CMU-MOSEI inputs, excluding MERP’s latent queries, reconstruction, prototypes, enhancement, and adaptive fusion.Metrics are computed on the validation split with probe parameters frozen; corrupted views zero removed streams.Source: A. Diagnostic Protocol · p. 3
  • The protocol compares intact and corrupted representations across fusion depth using linear CKA, while also measuring unavailable-token attention, class-wise geometry, and label instability.The unavailable-token attention ratio is interpreted against a uniform-attention denominator.Source: A. Diagnostic Protocol · pp. 3–4

B. Findings and Design Implications

On the CMU-MOSEI validation split, missing-modality corruption grows more disruptive with fusion depth and varies across emotions and feature channels. The findings motivate compact reasoning that preserves class-specific evidence and responds to class- and feature-dependent corruption.

B. Findings and Design Implications

  • 0.921 to 0.794: mean CKA across single-modality drops falls from layer 1 to layer 6; for single-retained views, it falls from 0.720 to 0.480.Unavailable-token attention rises from 1.049 to 1.240 for single-modality drops and from 1.171 to 1.595 for single-retained views.Source: B. Findings and Design Implications · p. 4
  • 10.9–19.4 percentage points: emotions subjected to the same corruption differ in flip rate, while the most affected 10% of classifier-relevant channels contain 46.1–87.8% of perturbation energy.Across 36 emotion–corruption pairs, margin collapse and label instability are associated at r = 0.94; layer-6 CKA and mean label flip rate are anticorrelated at r = −0.96.Source: B. Findings and Design Implications · p. 4
  • Robust MMER should aggregate available evidence before repeated reasoning, keep emotion-specific evidence identifiable as representations evolve, and make enhancement and fusion responsive to class- and feature-dependent corruption.MERP instantiates these requirements with a fixed latent workspace, cross-level label readouts, prototype-guided feature organization, selective enhancement, and class-wise fusion.Source: B. Findings and Design Implications · p. 4

B. Architecture Overview

MERP compresses available multimodal evidence into fixed-size latent banks, then preserves class-specific evidence across latent depth through readouts and reconstruction. Its deeper reasoning operates on latent tokens rather than the raw sequence.

B. Architecture Overview

  • MERP compresses joint multimodal tokens into low- and high-level latent banks, then uses class-specific readouts, reconstruction, prototypes, enhancement, and class-wise fusion to predict emotions.The overview presents the computation as a sequence from latent encoding through prediction.Source: B. Architecture Overview · p. 4, Figure 3 · p. 5

C. Multi-Modal Preprocessing and Latent Encoding

  • With L=80 latent tokens, attention costs O(LTΣ+NL2), so only the first block depends on input length and deeper reasoning uses the latent bank.The same fixed-size latent computation applies after a modality is dropped.Source: C. Multi-Modal Preprocessing and… · p. 5

D. Label-Specific Readout and Cross-Level Consistency

  • Class-specific readouts extract low- and high-level evidence, while a decoder attends to Zhigh to reconstruct class evidence and align it with the low-level readout.The reconstruction objective preserves class-specific evidence across latent depth.Source: D. Label-Specific Readout and… · p. 5

E. Prototype-Guided Feature Enhancement

MERP conditions high-level class features on positive and negative prototypes, then enhances their concatenation with low-level features using SNGA. Training-time perturbations and contrastive learning further regularize the features.

Prototype-guided features

Feature regularization

  • Supervised contrastive learning regularizes projected high-level and enhanced features using labelled prototypes and a memory queue.The queue construction and objective are specified in Appendix B.Source: E. Prototype-Guided Feature Enhancement · p. 6

Spectral noise-gated enhancement

F. Class-Wise Fusion and Training Objective

MERP combines aggregate, high-level, and enhanced probability branches through learned class-wise preferences calibrated by confidence. Its training objective supervises predictions and regularizes reconstruction and contrastive representations.

Class-wise fusion

  • The final prediction fuses aggregate, high-level, and enhanced probability branches using confidence-calibrated, class-wise gate preferences.The gate is trained with binary cross-entropy on the fused probabilities.Source: F. Class-Wise Fusion and Training… · p. 6

End-to-end procedure

  • The end-to-end procedure forms latent features, readouts, reconstructed and prototype-conditioned features, then enhances and fuses branches; training additionally updates prototypes and the memory queue.At inference, enhancement noise is disabled and predictions are thresholded.Source: F. Class-Wise Fusion and Training… · p. 6

Training objective

  • MERP's total objective combines final and auxiliary supervision with weighted reconstruction and supervised contrastive losses.Auxiliary logit-level supervision covers the low, high, enhanced, and aggregate heads.Source: F. Class-Wise Fusion and Training… · p. 6

V. EXPERIMENTS

Under a shared test-time modality-drop protocol, MERP achieves the highest reported micro-F1 on both datasets and outperforms LDDU across the evaluated CMU-MOSEI missing rates. The experiments also compare inference cost and report the limits of the rate-wise evaluation.

A. Experimental Setup

  • Training uses complete inputs; at test time, one uniformly sampled modality is zeroed with probability pmiss, using the same fixed corruption schedule for all methods.Unless stated otherwise, pmiss=0.2; the reported test examples include 900 CMU-MOSEI utterances and 43 M3ED dialogues with corruption.Source: A. Experimental Setup · pp. 6–7

B. Main Results under Modality Drop

  • 0.563 and 0.780 micro-F1: MERP leads CARAT (0.510 / 0.744) and LDDU (0.559 / 0.758) on CMU-MOSEI and M3ED, respectively, at pmiss=0.2.MERP ranks first on both F0.5 and F2 on CMU-MOSEI, and first on F2 and second on F0.5 on M3ED.Source: B. Main Results under Modality Drop · p. 7

C. Inference Cost

  • 27% and 28% faster than LDDU: MERP's reported inference latency is lower on CMU-MOSEI and M3ED, respectively, and both methods use 2,048 MiB of inference memory.The latent bottleneck reduces the self-attention footprint from O(Ttotal^2) to O(LTtotal + L^2), with L≪Ttotal.Source: C. Inference Cost · p. 7

D. Robustness across Missing Rates on CMU-MOSEI

  • The missing-rate sweep is limited to CMU-MOSEI because M3ED has only 179 test dialogues, too few for a stable rate-wise comparison across six operating points.The sweep uses identical per-example missing schedules for MERP and LDDU.Source: D. Robustness across Missing Rates on… · p. 7
  • At every evaluated CMU-MOSEI missing rate, MERP outperforms LDDU; the micro-F1 gap is 0.014 at pmiss=0 and 0.013 at pmiss=0.5.Across the sweep, MERP loses 2.4 absolute F1 points and LDDU loses 2.3 points.Source: D. Robustness across Missing Rates on… · p. 7

E. Fixed-Combo Modality Drop

Under fixed modality-loss combinations, MERP degrades gracefully and avoids LDDU’s severe text–visual-drop collapse; qualitative examples show stronger label-set agreement in several sparse-input cases. Ablations find the largest effects from removing reconstruction or jointly removing information and enhancement losses.

E. Fixed-Combo Modality Drop

  • 0.473 versus 0.022 micro-F1: with text and visual streams zeroed on CMU-MOSEI, MERP retains far more than LDDU.The absolute gap is 0.451.Source: E. Fixed-Combo Modality Drop · p. 7
  • MERP outperforms LDDU on six of seven fixed modality combinations and is within 0.005 on the seventh, without a double-drop collapse.Single-modality drops cost MERP 2.6–8.6 points and double-modality drops cost 5.1–11.2 points.Source: E. Fixed-Combo Modality Drop · p. 8
  • In six representative CMU-MOSEI cases, MERP most closely matches the ground-truth label sets, including exact matches in visual-only and audio-only examples.In the text-only example, it retains Sad and Surprise but misses Fear, showing recovery is not perfect under extreme information loss.Source: E. Fixed-Combo Modality Drop · pp. 8–9, Figure 6 · p. 9

F. Ablation Study

  • Removing reconstruction lowers micro-F1 from 0.585 to 0.550, while jointly removing information and enhancement losses lowers it to 0.414.Structural settings change the two-seed mean by at most 0.31% relative to the default; these ablations use CMU-MOSEI with pmiss=0.Source: F. Ablation Study · p. 9

VI. CONCLUSION

  • The paper concludes that compact aggregation with preserved task-relevant evidence can outperform missing-stream reconstruction or a single modality-reliability estimate within the evaluated settings.It identifies adaptive latent workspaces for evolving label spaces and longer multimodal contexts as future work.Source: §VI · p. 10

APPENDIX A PREPROCESSING AND ATTENTION DETAILS

The appendix describes modality position encoding and channel alignment, the encoder–decoder attention configuration, and the training and inference use of positive and negative class prototypes.

A. Fourier Positions and Channel Alignment

  • For each modality, position vectors use linearly spaced positions and 32 frequency bands; trainable padding embeddings align modalities to a common width before modality-specific gating.Missingness zeros the content term while retaining the deterministic position code.Source: A. Fourier Positions and Channel… · p. 12

B. Attention Blocks

  • Encoder cross-attention uses the latent bank as queries and aligned inputs as the source, while self-attention uses the hidden representation as both query and source.The encoder uses one cross-attention head and eight self-attention heads; the decoder uses class-broadcast trainable queries with Zhigh as source.Source: B. Attention Blocks · p. 12

A. Dynamic Class Prototypes

  • MERP maintains positive and negative prototypes for each class, updates the selected sign from labelled training pairs, and selects the higher-scoring sign at inference.The opposite-sign prototype is unchanged for each training pair.Source: A. Dynamic Class Prototypes · p. 12

B. Queue-Based Supervised Contrastive Loss

The appendix specifies queue-based supervised contrastive learning, a spectral noise-gated enhancement block, and the classification losses and fusion gate.

B. Queue-Based Supervised Contrastive Loss

  • Projected high-level and enhanced features form two class-conditioned views for supervised contrastive learning, with a FIFO queue of 768 feature–label pairs supplying additional comparisons.Anchors without a positive partner contribute zero to the batch mean.Source: B. Queue-Based Supervised Contrastive… · p. 12

APPENDIX C SPECTRAL NOISE-GATED FEATURE BLOCK

  • The spectral noise-gated block reduces concatenated features, injects independently generated pink feature noise during training, applies a feature gate, and projects a residual representation.The noise has a 1/f power spectrum over feature channels and is set to zero at inference.Source: Appendix C · p. 12

A. Classification Losses

  • The fused-output loss is binary cross-entropy on Eq. (22), while the fusion gate is a 3→32→3 multilayer perceptron with a ReLU hidden activation whose gradients are supplied by Lfinal.Source: A. Classification Losses · p. 13

B. Feature-Consistency and Shuffle Losses

The appendix defines feature-consistency and shuffle regularization and gives the objective coefficients used in training.

B. Feature-Consistency and Shuffle Losses

  • Feature-consistency regularization compares normalized flattened enhanced and low-level representations, while shuffle regularization permutes enhanced features and labels independently for each class before aggregate classification.Source: B. Feature-Consistency and Shuffle… · p. 13

C. Objective Coefficients

Related papers

Loading 2610.29430v1…