PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Factorized Multimodal Representations

Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, Ruslan Salakhutdinov

arXiv:1806.06176v3cs.LGcs.CLcs.CVstat.ML

TL;DR

Multimodal learning must capture complex cross-modal interactions while handling missing or noisy modalities. MFM jointly optimizes generative and discriminative objectives using factorized shared and modality-specific representations. It achieves state-of-the-art or competitive results on six multimodal datasets and supports missing-modality reconstruction without significantly affecting discriminative performance.

  • Problem

    Multimodal models must learn complex intra-modal and cross-modal interactions for prediction while remaining robust to missing or noisy modalities during testing.

  • Method

    MFM uses a joint generative-discriminative objective and factorizes representations into multimodal discriminative factors and modality-specific generative factors.

  • Results

    MFM achieves state-of-the-art or competitive performance on six multimodal datasets and reconstructs missing modalities without significantly impacting discriminative performance.

  • Takeaways & Limitations

    The factorized model supports flexible generation, missing-modality handling, and interpretation of interactions in multimodal learning.

  • Takeaways & Limitations

    MFM's inference factorization assumes Zy depends on all modalities, whereas each Zai depends only on its corresponding modality.

Abstract

from arXiv · show

Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing. In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels. We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors. Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction. Modality-specific generative factors are unique for each modality and contain the information required for generating data. Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets. Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance. Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.

1 INTRODUCTION

Multimodal representation learning must capture intra-modal and cross-modal interactions while remaining robust to missing or noisy modalities. MFM addresses these challenges by factorizing representations and jointly modeling multimodal data and labels.

  • Multimodal models must learn intra-modal and cross-modal interactions for prediction and remain robust to missing or noisy modalities at test time.
  • MFM optimizes a joint generative-discriminative objective across multimodal data and labels.Its discriminative objective targets predictive features, while its generative objective supports missing-modality inference and noisy-modality handling.
  • MFM factorizes representations into shared multimodal discriminative factors and modality-specific generative factors.The shared factors contain joint features for discriminative tasks, while modality-specific factors contain information needed to generate each modality.
  • MFM defines a joint distribution over multimodal data and incorporates both generative and discriminative aspects through conditional independence assumptions.The factorized design also provides interpretability of the learned representations.
  • MFM achieves state-of-the-art or competitive performance on six multimodal time-series datasets and reconstructs missing modalities without significantly impacting discriminative performance.The model also supports flexible generation conditioned on multimodal discriminative factors and modality-specific generative factors.

2 MULTIMODAL FACTORIZATION MODEL

MFM factorizes multimodal representations into shared discriminative and modality-specific generative factors, and learns them through a joint distribution objective approximated with Wasserstein-based inference. The resulting architecture supports prediction, generation, and reconstruction of missing modalities.

  • MFM is a latent variable model whose independent latent variables generate multimodal discriminative factors and modality-specific generative factors.The discriminative factor contributes to label generation, while the discriminative and modality-specific factors jointly generate each modality.
  • The model approximates posterior inference by minimizing a joint-distribution Wasserstein distance and applying a generalized mean-field factorization.The inference distribution separates the shared latent code, inferred from all modalities, from modality-specific codes inferred from their corresponding modalities.
  • The neural implementation uses multimodal encoders and decoders, including late-fusion CNNs and FCNNs for multimodal image data.The encoder maps multimodal inputs and labels to latent codes, while decoder mappings generate factors, labels, and modalities.
  • MFM jointly optimizes multimodal reconstruction and label prediction, with a prior-matching penalty based on Maximum Mean Discrepancy.The generative term reconstructs multimodal data, while the discriminative term predicts labels from the shared factor.
  • For missing modalities, a surrogate inference network infers latent codes from observed modalities to generate the missing modality and predict labels.This adaptation requires inferring latent codes rather than reconstructing the entire observed modality first.

3 EXPERIMENTS

Experiments evaluate MFM on synthetic images and six real-world multimodal video datasets, testing discrimination, generation, missing-modality robustness, ablations, and interpretability. MFM consistently performs competitively, supports factorized generation and reconstruction, and reveals modality contributions to prediction and generation.

  • Experimental setup: MFM is evaluated on paired SVHN+MNIST images and six multimodal time-series datasets spanning speaker traits, sentiment, and emotion recognition.The synthetic dataset contains 100,000 same-label SVHN–MNIST pairs, with 80,000 used for training; the real-world datasets use language, visual, and acoustic features.
  • 3.1 MULTIMODAL SYNTHETIC IMAGE DATASET: MFM outperforms the multimodal discriminative baseline on SVHN+MNIST classification, indicating improved factorized representations for discriminative tasks.Classification performance improves from unimodal to multimodal baselines, while MFM exceeds the multimodal baseline.
  • 3.1 MULTIMODAL SYNTHETIC IMAGE DATASET: MFM generates SVHN and MNIST images by independently varying latent variables corresponding to labels and modality-specific styles.The model fixes one variable in Z and samples the other two from the prior, supporting conditional generation through factorized representations.
  • Prediction: MFM achieves state-of-the-art or competitive results across all six multimodal datasets and improves Acc_2 on CMU-MOSI for EF-LSTM from 74.3 to 75.2 and TFN from 74.6 to 75.5.The factorization framework is model-agnostic and improves several multimodal encoders.
  • Ablation Study: Ablations show that multimodal discriminative factors, generative capabilities, separate factorization, and modality-specific generative factors improve prediction or reconstruction.The reported comparisons are MD > MC and MB > MA; MC > MA and ME > MB; ME > MD; and MFM > ME.
  • Missing modalities: With missing modalities, MFM outperforms purely generative and discriminative baselines while remaining close to its all-modalities performance.Language absence affects discriminative performance most, whereas acoustic and visual features are more robust; low-level acoustic and visual reconstruction is easier than language reconstruction.
  • Interpretation of Multimodal Representations: MFM’s interpretation methods quantify modality contributions across datasets and time steps, linking sentiment-relevant words and acoustic cues to increases in the discriminative-factor gradient.Information-based analysis compares normalized mutual-information ratios, while gradient-based analysis examines generated modalities over time.

4 RELATED WORK

Prior multimodal representation-learning work has mainly treated discriminative and generative objectives separately, while factorized approaches learn disentangled representations through supervised latent attributes.

  • Discriminative methods model P(Y∣X1:M) without explicitly modeling P(X1:M), allocating parameters efficiently to label prediction.
  • Generative methods model the multimodal data distribution, complementing discriminative approaches that focus on conditional label prediction.
  • Factorized representation learning separates data variations into disentangled factors, often using fixed latent attributes with supervised training.

5 CONCLUSION

The paper proposes MFM, which separates multimodal representations into independent shared discriminative factors and modality-specific generative factors. These factors support competitive performance, controlled generation, missing-modality handling, and interpretation of multimodal interactions.

  • MFM factorizes multimodal representations into independent multimodal discriminative factors and modality-specific generative factors.
  • The multimodal discriminative factor achieves state-of-the-art or competitive results on six multimodal datasets.
  • Modality-specific generative factors enable generation from factorized variables, missing-modality handling, and deeper analysis of multimodal interactions.
  • MFM motivates future extensions to video generation, semi-supervised learning, and unsupervised learning.

A PROOF OF PROPOSITION 1

The proof first establishes deterministic reconstructions and conditional independence, then rewrites the Wasserstein objective using the model’s factorized mappings and multimodal reconstruction costs.

  • The proof begins with the unimodal Wasserstein distance between the true and reconstructed joint distributions.
  • Deterministic mappings make reconstructed outputs conditionally Dirac given latent variables, establishing the first lemma.
  • Conditional independence follows because reconstructed variables are deterministic functions of Z, yielding the equality used in the second lemma.
  • The joint-distribution Wasserstein expression is rewritten with separate reconstruction costs cX(X,ˆX) and cY(Y,ˆY).
  • For multimodal data, the proof sums modality-specific costs with the label cost and applies the tower rule with conditional independence.

B FULL BASELINE MODELS & RESULTS

The appendix details baseline implementations, dataset evaluations, feature extraction, and updated data handling. Across six multimodal datasets, MFM consistently achieves state-of-the-art or competitive results.

  • The baseline suite includes MFN, MARN, TFN, TFN, BC-LSTM, MV-LSTM, EF-LSTM, HCRF variants, THMM, SVM-MD, and random forests.
  • EF-LSTM variants differ by stacking and bidirectionality while operating on concatenated multimodal inputs.
  • EF-HCRF variants use latent HCRF states, including latent discriminative codes and hierarchical sequence summarization.
  • MV-HCRF variants extend HCRFs to multiview data by modeling view-shared and view-specific substructures.
  • MFM consistently achieves state-of-the-art or competitive results across speaker-trait, sentiment, and emotion-recognition datasets.The reported datasets include POM, CMU-MOSI, ICT-MMMO, YouTube, MOUD, and IEMOCAP.
  • Experiments use language, visual, and acoustic features, with updated multimodal features adopted after an alignment discrepancy was identified.All baseline models were retrained with extensive hyperparameter search for fair comparison.

D INFORMATION AND GRADIENT-BASED INTERPRETATION

MFM is interpreted through information-based and gradient-based analyses, while benchmark tables report competitive performance across personality, sentiment, and emotion tasks.

  • Information-Based Interpretation: The information-based analysis uses normalized HSIC as an approximation of mutual information between factor representations and modalities.RBF kernels are used after time-series averaging; the bandwidth is fixed at 1.0 throughout the experiments.
  • Experimental Results: MFM achieves state-of-the-art or competitive performance across personality-trait, sentiment, and emotion-recognition benchmarks.The reported tables cover POM, four sentiment datasets, and the IEMOCAP test set.
  • Information-Based Interpretation: On the POM dataset, language is the most informative modality, while visual and acoustic modalities are nearly equally informative.This agrees with cited behavioral studies describing non-verbal behaviors as informative for personality traits.
  • Gradient-Based Interpretation: The gradient-based analysis reconstructs each modality from modality-specific and multimodal latent factors through modality-specific generators and decoders.The reconstruction uses modality-specific codes sampled from unimodal posteriors and a shared code sampled from the multimodal posterior.

E ENCODER AND DECODER DESIGN FOR MULTIMODAL SYNTHETIC IMAGE DATASET

For the synthetic SVHN+MNIST dataset, MFM uses convolutional and fully connected encoders with deconvolutional and fully connected decoders to separate modality-specific and shared information.

  • Encoder and Decoder Design: Different convolutional layers process SVHN and MNIST separately to learn modality-specific generative factors.The decoder uses deconvolutional and fully connected layers for image reconstruction.
  • Encoder and Decoder Design: Features from additional convolutional layers are concatenated across SVHN and MNIST to learn the multimodal-discriminative factor.This factor combines information from both image modalities after their modality-specific processing.

F ENCODER AND DECODER DESIGN FOR MULTIMODAL TIME SERIES DATASETS

For multimodal time series, MFM combines a multimodal-fusion encoder with recurrent encoders and decoders, and uses surrogate inference when modalities are missing.

  • Encoder and Decoder Design: The multimodal discriminative encoder Q(Zy∣X1:M) can use any multimodal-fusion model; the implementation uses the Memory Fusion Network.This design lets the multimodal encoder be replaced by another fusion architecture.
  • Encoder and Decoder Design: Encoder LSTMs parameterize modality-specific inference, decoder LSTMs parameterize modality generation, and fully connected networks parameterize factor mappings and discrimination.The named functions are Q(Za1:M∣X1:M), F1:M, Gy, Ga{1:M}, and D.
  • Missing-Modality Inference: Surrogate inference estimates latent codes from the modalities that remain present, enabling reconstruction or label prediction despite missing modalities.The inferred codes replace the original inference pathway when only a subset of modalities is available.

H COMPARISON WITH HSU & GLASS (2018)

MFM is compared with Hsu and Glass (2018) across inference, objectives, fusion, prior matching, and experimental scale, with MFM outperforming their model on the reported datasets and metrics.

  • Methodological Differences: MFM uses MMD for prior matching, whereas Hsu and Glass use KL divergence.The comparison identifies the prior-matching discrepancy as one of several architectural and objective differences.
  • Methodological Differences: MFM combines multimodal and unimodal inference in one network, while Hsu and Glass use separate networks with an additional coherence loss.The two approaches therefore differ in how inference pathways and their agreement are modeled.
  • Methodological Differences: MFM predicts labels generatively and supports flexible multimodal fusion, whereas Hsu and Glass use a hinge loss and a fixed fusion encoder.MFM is described as compatible with any multimodal fusion encoder, while the comparison model uses an early-fusion-like fixed encoder.
  • Empirical Comparison: Across CMU-MOSI, ICT-MMMO, YouTube, and MOUD, MFM performs better than the reimplemented Hsu and Glass model on the reported disentanglement and prediction metrics.The authors note that the comparison model’s original code was unavailable at submission time.

I COMPARISON WITH β-VAE

The paper compares MFM with β-VAE for multimodal representation learning. MFM consistently outperforms β-VAE across the evaluated datasets and metrics.

  • Comparison setup: MFM extends β-VAE to multimodal data while retaining modality-specific generative factors and a multimodal discriminative factor.The comparison uses a KL-divergence prior-matching discrepancy and large β values to encourage disentanglement in β-VAE.
  • Results: MFM outperforms the baselines across the reported datasets and metrics.Table 7 summarizes comparisons with Hsu & Glass (2018) for sentiment analysis on the same four datasets.
  • Results: MFM outperforms β-VAE across CMU-MOSI, ICT-MMMO, YouTube, and MOUD.The comparison covers multimodal sentiment analysis on four datasets.
Loading 1806.06176v3…