PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations

Sijie Mai, Ying Zeng, Haifeng Hu

arXiv:2210.17444v3cs.LG

TL;DR

Multimodal fusion can create redundant joint embeddings while unimodal representations retain noise, complicating effective prediction. MIB applies information bottleneck constraints to unimodal and multimodal representations through three flexible variants. The reported results show state-of-the-art performance on multimodal sentiment analysis and emotion recognition across three datasets.

  • Problem

    Multimodal fusion may produce redundant embeddings and ignore discriminative unimodal information, while unimodal representations can contain noise that interferes with cross-modal learning.

  • Method

    MIB applies information bottleneck constraints to learn minimal sufficient unimodal and multimodal representations, using early-fusion, late-fusion, and complete variants compatible with common fusion strategies.

  • Results

    MIB variants outperform other methods on multiple datasets, with reported state-of-the-art performance across multimodal sentiment analysis and emotion recognition.

  • Takeaways & Limitations

    The framework provides a flexible way to filter noisy unimodal information and reduce redundancy in multimodal representations while retaining simple fusion options.

Abstract

from arXiv · show

Learning effective joint embedding for cross-modal data has always been a focus in the field of multimodal machine learning. We argue that during multimodal fusion, the generated multimodal embedding may be redundant, and the discriminative unimodal information may be ignored, which often interferes with accurate prediction and leads to a higher risk of overfitting. Moreover, unimodal representations also contain noisy information that negatively influences the learning of cross-modal dynamics. To this end, we introduce the multimodal information bottleneck (MIB), aiming to learn a powerful and sufficient multimodal representation that is free of redundancy and to filter out noisy information in unimodal representations. Specifically, inheriting from the general information bottleneck (IB), MIB aims to learn the minimal sufficient representation for a given task by maximizing the mutual information between the representation and the target and simultaneously constraining the mutual information between the representation and the input data. Different from general IB, our MIB regularizes both the multimodal and unimodal representations, which is a comprehensive and flexible framework that is compatible with any fusion methods. We develop three MIB variants, namely, early-fusion MIB, late-fusion MIB, and complete MIB, to focus on different perspectives of information constraints. Experimental results suggest that the proposed method reaches state-of-the-art performance on the tasks of multimodal sentiment analysis and multimodal emotion recognition across three widely used datasets. The codes are available at \url{https://github.com/TmacMai/Multimodal-Information-Bottleneck}.

I. INTRODUCTION

The paper frames multimodal fusion as a representation-learning problem: joint embeddings can retain redundancy while unimodal features contain noise. It proposes MIB and three variants to learn concise, task-relevant representations across multimodal tasks.

  • Complex fusion can produce redundant, high-dimensional multimodal embeddings, increasing overfitting risk and potentially ignoring discriminative unimodal information.
  • Unimodal representations may contain noise, especially in visual and acoustic modalities, that interferes with cross-modal interaction learning.
  • MIB applies the information bottleneck principle by maximizing representation–label mutual information while constraining representation–input mutual information.
  • E-MIB compresses a fused multimodal representation, whereas L-MIB first filters unimodal representations before fusion.
  • C-MIB combines E-MIB and L-MIB to constrain both unimodal noise and multimodal redundancy while remaining compatible with flexible fusion mechanisms.
  • MIB variants outperform other methods across multiple datasets, with visualization experiments indicating more discriminative multimodal representations and strong performance using simple fusion.

B. Information Bottleneck

The paper situates MIB within information-bottleneck and multi-view representation learning, then describes a flexible architecture built from unimodal networks and a fusion network. Its algorithms optimize these components under bottleneck objectives.

  • Information bottleneck methods seek shorter representations that preserve task-relevant information, while variational IB makes the constraints usable in deep learning.
  • Related multi-view IB methods constrain individual views, but MIB also supports arbitrary fusion mechanisms rather than relying only on product-of-experts fusion.
  • MIB consists of three unimodal learning networks and a multimodal fusion network whose optimization is driven by the information bottleneck principle.
  • E-MIB: The E-MIB algorithm initializes unimodal and fusion networks, samples utterance batches, produces stochastic representations, predicts labels, and computes its bottleneck objective.

A. Notations and Task Definition

The tasks use utterances represented through acoustic, visual, and language sequences, which are transformed into multimodal predictions. The framework separates unimodal processing from flexible fusion and supports multiple MIB variants.

  • Each utterance contains acoustic, visual, and language modalities, represented as modality-specific sequences with distinct lengths and feature dimensions.
  • The model learns a joint multimodal representation from the three unimodal sequences and predicts sentiment scores or emotion labels.
  • C-MIB is illustrated as a pipeline whose fusion mechanism is flexible, with DNN denoting deep neural network.
  • The framework includes separate algorithmic variants for late fusion and complete MIB, each using unimodal and multimodal network components.
  • After unimodal networks transform input sequences into representations, the fusion network can use different fusion mechanisms.

B. Multimodal Information Bottleneck

MIB extends the information bottleneck principle to multimodal learning by constraining both multimodal and unimodal representations. Its early-fusion variant compresses a fused representation to preserve label-relevant information while reducing redundancy.

  • Information Bottleneck Principle: MIB applies information bottleneck constraints to multimodal and unimodal representations, targeting minimal sufficient representations that retain discriminative information and filter noise.The framework maximizes target-representation mutual information while constraining representation-input mutual information.
  • Information Bottleneck Principle: The information bottleneck balances predictive power against representation complexity by maximizing I(y; z) and minimizing I(x; z).The scalar β controls the weight of the minimal-information constraint.
  • Early-Fusion MIB: E-MIB first fuses unimodal representations into a primary multimodal representation x, then applies IB to obtain a less redundant representation z.The default fusion concatenates xa, xv, and xl before multimodal compression.
  • Early-Fusion MIB: Unlike prior multi-view bottleneck methods that regularize each view, E-MIB directly regularizes the generated multimodal embedding to reduce its redundancy.This targets the high-dimensional joint representation used for prediction.
  • Early-Fusion MIB: E-MIB optimizes a variational lower bound that encourages z to be discriminative for y while forgetting information from the primary multimodal representation x.For regression, the predictive term corresponds to MAE; for classification, it corresponds to cross-entropy.

3) Procedure of Late-Fusion MIB (L-MIB):

L-MIB filters noise from each modality before fusion by applying the information bottleneck independently to unimodal representations. It then combines the encoded representations using flexible fusion mechanisms, including product-of-experts fusion.

  • Late-Fusion MIB: L-MIB applies the information bottleneck separately to each modality before fusion, filtering information irrelevant to the target.This ordering addresses noise and distribution discrepancies that can hinder cross-modal interaction learning.
  • Late-Fusion MIB: The encoded unimodal representations are fused through a flexible multimodal fusion network to produce the final representation.The framework can integrate different fusion methods rather than requiring one fixed mechanism.
  • Product-of-Experts Fusion: Product-of-experts fusion weights modality means by reciprocal variance, giving greater influence to lower-variance, more confident modalities.The method treats lower uncertainty as greater relevance to the target.
  • Late-Fusion MIB: L-MIB is designed to improve fused representations by filtering noisy acoustic and visual information before complex fusion operations.The paper identifies these modalities as particularly noise-prone in multimodal prediction.

4) Procedure of C-MIB:

C-MIB combines the two bottleneck perspectives by first denoising unimodal representations and then reducing redundancy in their fused multimodal representation.

  • Complete MIB: C-MIB first applies IB to unimodal representations, fuses the encoded outputs into x, and applies IB again to obtain the final representation z.This combines unimodal noise filtering with multimodal redundancy reduction.
  • Complete MIB: C-MIB is presented as combining the advantages of E-MIB and L-MIB.The comparison among E-MIB, L-MIB, and C-MIB is deferred to the experiments.

IV. EXPERIMENTS

The experiments evaluate MIB primarily on multimodal sentiment analysis and additionally test its generalization to multimodal emotion recognition.

  • Experimental Tasks: The evaluation covers multimodal sentiment analysis and multimodal emotion recognition.Sentiment analysis is the primary task, while emotion recognition is included to assess generalization to another multimodal learning task.

A. Datasets

CMU-MOSI is a multimodal sentiment-analysis dataset organized into videos and utterances, with sentiment intensity scored from -3 to +3.

  • CMU-MOSI contains 93 videos, each divided into at most 62 utterances.
  • Each utterance receives a sentiment-intensity score ranging from -3 to +3.The endpoints represent the strongest negative and strongest positive sentiment, respectively.

2) CMU-MOSEI:

The experiments evaluate MIB variants across multimodal sentiment and emotion-recognition datasets, comparing them with established baselines. Results generally favor MIB, while variant performance varies by dataset and fusion strategy.

  • CMU-MOSEI: CMU-MOSEI contains 2,928 videos with utterance-level sentiment and six emotion labels.Sentiment ranges from -3 to +3; the emotion labels are anger, disgust, fear, happy, sad, and surprise.
  • Sentiment analysis: C-MIB improves over MISA by 0.7% on Acc2 and 1.2% on F1 score on CMU-MOSEI.Acc7 and Corr also improve relative to state-of-the-art methods.
  • Variant comparison: Variant performance differs across datasets, with L-MIB weaker than E-MIB on CMU-MOSI but stronger on CMU-MOSEI.The paper attributes part of this difference to randomness in latent-representation generation, especially on smaller datasets.
  • Emotion recognition: On IEMOCAP, all MIB variants outperform baselines when performance is combined across emotions.C-MIB leads on Angry, Sad, and Neutral; L-MIB leads on Happy; E-MIB improves Sad F1 by 2.0% over existing methods.

E. Ablation Study

The ablations show that IB constraints improve multimodal systems, especially by filtering noise from visual and acoustic representations, while C-MIB generally performs best among MIB variants and fusion settings.

  • Unimodal and bimodal ablations: Removing IB from language causes only a slight performance drop, whereas removing it from acoustic or visual modalities causes a significant drop.The authors attribute this difference to greater redundancy and noise in visual and acoustic features, while language is more discriminative.
  • MIB variant comparison: C-MIB achieves the best performance on most average metrics across both datasets, while E-MIB leads on some metrics and L-MIB is worst overall.The average-performance differences between variants are consistent across the two datasets.
  • Fusion strategies: C-MIB improves considerably with tensor and graph fusion because it jointly filters unimodal noise and reduces redundancy in complex multimodal representations.E-MIB cannot explicitly filter unimodal noise, while L-MIB cannot reduce multimodal redundancy.
  • Fusion strategies: PoE performs unfavorably relative to other fusion methods used with L-MIB, supporting compatibility with expressive alternatives such as tensor fusion.The paper presents fusion-method compatibility as an advantage over prior multi-view IB approaches.
  • Fusion strategies: Tensor fusion slightly outperforms graph fusion across the two datasets, while concatenation generally outperforms addition and multiplication among simple fusion methods.The paper links tensor fusion’s advantage to its ability to model complicated interactions, while MIB constrains the resulting redundancy.
  • Unimodal and bimodal ablations: Adding IB constraints consistently improves overall performance, with larger gains for visual, acoustic, and visual-acoustic systems.These results support the role of IB regularization in reducing noisy feature information.

H. Analysis on the Hyperparameter β

The hyperparameter β controls the strength of the minimal information constraint: excessive constraint harms performance, while a small value best balances retaining useful information with filtering redundancy and noise.

  • Effect of β: Performance drops significantly when β=1, indicating that an overly strong minimal information constraint is harmful.The constraint can filter necessary information when it dominates learning.
  • Effect of β: When β=0, binary accuracy drops by 1.5% and 7-class accuracy drops by 2%.Removing the constraint therefore also reduces performance.
  • Effect of β: The model performs best when β=1e-5, although the default setting is β=1e-3 for consistency with L-MIB and E-MIB.The authors conclude that β should remain small so useful information is retained while redundancy and noise are reduced.
  • Constraint interpretation: The minimal information constraint controls information shared between the input x and encoded representation z, and alternative objectives can replace its KL-divergence term.The alternatives include distribution- or similarity-based objectives such as MMD, Deep CORAL, Euclidean distance, cosine distance, and orthogonality.
  • Constraint interpretation: The original KL-divergence term performs best, while alternative objectives remain competitive with removing the minimal information constraint.The paper attributes KL’s advantage to explicitly measuring mutual information between input and encoded representation.

K. Learning Curve for Minimal Information Constraints

Learning curves indicate that acoustic and visual representations contain more information requiring filtering than language representations, while IB regularization adds a relatively small parameter increase to C-MIB.

  • Learning curves: Acoustic and visual minimal information constraint losses decrease much more during training than the language loss.The authors interpret this as evidence that acoustic and visual modalities contain more noise to filter, whereas language retains sufficient discriminative information.
  • Learning curves: The learning curves suggest that the method filters noisy information while retaining useful information in the representations.This interpretation is consistent with the modality-specific loss behavior reported for acoustic, visual, and language inputs.
  • Model complexity: C-MIB has 109,835,748 parameters with IB regularization versus 109,216,699 without the IB constraints.The additional encoders and decoders required by MIB increase the model parameter count.
  • Overall conclusion: The paper concludes that MIB variants learn minimal sufficient unimodal and multimodal representations, remain compatible with fusion mechanisms, and achieve state-of-the-art performance on two multimodal tasks.The conclusion also reports that visualization supports more discriminative multimodal representations and that alternative constraint objectives produce promising results.
Loading 2210.17444v3…