PaperlayerView exact revision on arXiv

Source-linked AI summary

A survey of multimodal deep generative models

Masahiro Suzuki, Yutaka Matsuo

arXiv:2207.02127v1cs.LGstat.ML

TL;DR

Multimodal learning must infer shared representations and generate across modalities while accounting for heterogeneous data. This paper surveys and categorizes multimodal deep generative models, synthesizing their approaches and applications. It concludes with future directions for the field.

  • Problem

    Multimodal learning needs shared representations and cross-modal generation, but differing feature spaces and distributions make these challenges heterogeneous.

    Source: §1 · pp. 1–2

  • Method

    The paper surveys multimodal deep generative models, classifying them as coordinated or joint models.

    Source: §1 · p. 2

  • Results

    The survey classifies coordinated models by inference-distribution closeness and joint models by challenges, and summarizes benchmark datasets, applications, and future directions.

    Source: §8 · p. 19

  • Takeaways & Limitations

    The authors hope their taxonomy, dataset and application summary, and discussion of future directions will help future research on multimodal deep generative models.

    Source: §8 · p. 19

    Approaches requiring separate inference distributions can incur memory and computational costs that grow exponentially when there are three or more modalities.

    Source: §5.1.1 · p. 13

Abstract

from arXiv · show

Multimodal learning is a framework for building models that make predictions based on different types of modalities. Important challenges in multimodal learning are the inference of shared representations from arbitrary modalities and cross-modal generation via these representations; however, achieving this requires taking the heterogeneous nature of multimodal data into account. In recent years, deep generative models, i.e., generative models in which distributions are parameterized by deep neural networks, have attracted much attention, especially variational autoencoders, which are suitable for accomplishing the above challenges because they can consider heterogeneity and infer good representations of data. Therefore, various multimodal generative models based on variational autoencoders, called multimodal deep generative models, have been proposed in recent years. In this paper, we provide a categorized survey of studies on multimodal deep generative models.

1. Introduction

Multimodal deep generative models address the challenges of learning shared representations and generating across modalities despite multimodal heterogeneity. The paper surveys this field and organizes its methods into coordinated and joint models.

1. Introduction

  • Multimodal learning seeks compact, modality-invariant shared representations without labels and cross-modal generation between modalities.A shared representation can support generation between arbitrary modalities when modalities are embedded into it and generated from it.Source: §1 · p. 1
  • Heterogeneous feature spaces and distributions make multimodal relationships non-deterministic, so models must account for those differences.The survey describes deep generative models as neural-network-parameterized distributions; VAEs enable fast inference through amortized inference.Source: §1 · p. 2
  • The survey organizes multimodal deep generative models into coordinated and joint models, with distinct classification criteria for each category.Coordinated models are classified by how they define closeness between inference distributions; joint models by challenges including missing modalities, modality-specific latent variables, and weak supervision.Source: §1 · p. 2
  • Earlier surveys did not fully cover deep-generative representation methods or translation beyond conditional models; the authors call this the first comprehensive survey of the field, to their knowledge.Baltrušaitis et al. describe five multimodal-learning challenges, but their survey omits those methods and only covers conditional models for translation.Source: §1 · p. 2

2. Multimodal generative models

This section establishes the multimodal generative-model setting, explains heterogeneity and shared-representation requirements, and reviews pre-deep generative models and their limits.

2.1 Notation

  • Notation treats M modalities as data with modality-specific feature spaces and denotes an embedding z in Z as their shared representation.The text also denotes the true joint distribution over modalities and subsets of multimodal data.Source: §2.1 · p. 3

2.2 Problem setting

  • Multimodal generative models usually treat modalities as conditionally independent given a latent variable, with parameters for each modality's conditional distribution.Design must account for modality heterogeneity.Source: §2.2 · p. 3
  • The model aims to learn a shared representation and generate target modalities from arbitrary inputs, but likelihood and posterior inference are intractable.The objective involves marginalizing a latent variable, and the posterior of the shared representation given any modality is also intractable to compute.Source: §2.2 · pp. 3–4

2.3 Heterogeneity in multimodality

  • Multimodal heterogeneity combines differences in feature spaces and distributions; unequal information and non-invertible correspondences mean models must address both.The paper contrasts this with heterogeneous transfer learning, where differing feature spaces define heterogeneity.Source: §2.3 · p. 4

2.4 Requirements for a good shared representation

  • A good shared representation should group similar concepts and remain inferable when modalities are missing, supporting cross-modal generation.The paper presents these as requirements for a good shared representation.Source: §2.4 · p. 5

2.5 Related works of multimodal generative models before deep generative models

  • Pre-deep generative models could handle distribution differences and missing inputs, but MCMC-based models struggled with complex high-dimensional data; topic models required preprocessing such as visual words.The paper notes that these earlier methods could not directly infer from or generate complex data such as images.Source: §2.5 · p. 5

3. Multimodal deep generative models

The section introduces deep generative models and VAEs, then surveys their advantages and organizes multimodal methods by how they infer shared representations.

3.1 Deep generative models and variational autoencoders

  • Deep generative models parameterize generative distributions with deep neural networks, enabling high-dimensional data handling and end-to-end training by backpropagation.Source: §3.1 · p. 5
  • VAEs maximize an ELBO rather than the intractable marginal log-likelihood, using qφ(z|x) to approximate the posterior and pθ(x|z) as decoder.The ELBO combines a negative reconstruction term with encoder regularization.Source: §3.1 · p. 6

3.2 Advantages of VAEs as multimodal generative models

  • VAEs support fast inference and generation for complex inputs, flexible priors for representation learning, and explicit modeling of modality-specific distributions.The section contrasts these properties with ordinary deep autoencoders, which cannot incorporate various priors, and notes that probabilistic VAEs can represent distribution differences.Source: §3.2 · p. 6
  • Most multimodal GAN approaches are conditional, while heterogeneity, flexible priors, and training stability remain challenges; GANs also serve as VAE components [35] [123] [47] [51] [125] [113] [92] [113].Source: §3.2 · p. 6

3.3 Categories of multimodal deep generative models

  • The survey divides models by inference source: coordinated models infer from a single modality, while joint models infer from all modalities.Figure 2 depicts the two categories and their latent-variable relationships.Source: §3.3 · p. 7, Figure 2 · p. 7
  • Coordinated models aim to make inference results from each modality agree, whereas joint models seek agreement across any set of modalities.Source: §3.3 · p. 7
  • The survey’s categories are not exclusive: MVAE appears under both missing-modality handling and weakly supervised learning, while JVAE is omitted as the simplest joint model [112] [91, 104].Source: Figure 3 · p. 8

4. Coordinated model

Coordinated models seek agreement across modality-specific inference distributions, with Lcoor implemented through distribution distance or cross-modal generation loss. The section surveys variants of both criteria.

4. Coordinated model

  • Coordinated models bring modality-conditioned inference distributions closer, using per-modality generators with each modality’s ELBO plus Lcoor; the discussion assumes two modalities.Source: §4 · p. 8

4.1 Distance between inference distributions

  • For distribution-distance coordination, Yin et al. use KL divergence in both directions, while SCAN uses one direction so symbol inference covers image inference [26].SCAN first trains the image VAE, then fixes its parameters while optimizing the symbol ELBO and coordination loss.Source: §4.1 · p. 9
  • Tian et al. map modality-specific VAE or GAN representations into a shared space and optimize sliced Wasserstein distance between embeddings alongside class-label prediction loss [96].Source: §4.1 · p. 9

4.2 Cross-modal generation loss

  • Bi-VCCA uses cross-modal generation loss to encourage either modality’s encoding to generate both modalities; CADA-VAE combines that loss with a distance between inference distributions [109].CADA-VAE uses image features and class attributes for generalized zero-shot learning.Source: §4.2 · p. 10
  • AVAE learns associators that map between modalities’ latent variables, with an objective combining cross-modal generation loss and inference regularization.Source: §4.2 · p. 10

5. Joint model

Joint VAEs infer a shared latent representation from multimodal inputs, but must address missing modalities, modality-specific information, and incomplete paired training data. The survey organizes approaches to these challenges around surrogate or aggregated inference, factorized latent variables, and weakly supervised learning.

5. Joint model

  • JVAE extends the VAE objective to jointly model inference from all modalities and generation of the modalities from a shared latent variable.The survey identifies missing modalities, modality-specific latent variables, and weakly supervised learning as three major challenges.Source: §5 · p. 11

5.1 Handling of missing modalities

  • Missing inputs obstruct inference from arbitrary modality sets, so the survey groups solutions into surrogate unimodal inference and aggregation of unimodal distributions.JVAE’s neural inference can be corrupted when informative inputs are absent.Source: §5.1 · p. 11

5.1.1 Introducing a surrogate unimodal inference distribution

  • JMVAE adds surrogate unimodal inference distributions and KL terms, while TELBO trains unimodal and joint ELBOs in sequence and M2VAE combines their bounds.Preparing inference distributions for all modality combinations makes surrogate approaches costly and grows network size exponentially with three or more modalities.Source: §5.1.1 · pp. 12–13

5.1.2 Aggregating the unimodal inference distributions

  • Bridge encoders approximate joint and unimodal inference with network counts that grow linearly in modality count, but require two training stages.The encoders and generative models are first trained with a JVAE objective, then frozen while bridge encoders are learned.Source: §5.1.2 · pp. 13, 15
  • PoE sharpens joint inference as modalities are added, MoE spreads density across experts, and MoPoE combines their approaches.MoPoE’s number of PoE inferences, and its ELBO computational cost, increase exponentially with modality count.Source: §5.1.2 · pp. 13–14
  • mmJSD replaces KL divergence in the MMVAE objective with JS divergence and is reported to outperform MVAE and MMVAE on cross-modal generation and latent inference.The approach uses a dynamic prior defined by a function set to weighted PoE.Source: §5.1.2 · p. 14

5.2 Modality-specific latent variables

  • Modality-specific latent-variable models separate shared and modality-specific factors; methods add objectives for information inclusion, disentanglement, or cross-modality similarity.PVAE adds multimodal–unimodal coherence and a contrastive objective, while Daunhawer et al. estimate mutual-information and total-correlation terms.Source: §5.2 · pp. 15–16

5.3 Weakly-supervised learning

  • MVAE subsampling permits training with modality subsets, and a contrastive-style method is reported to work well with regular multimodal learning and small amounts of paired data.The latter estimates joint log-likelihood with IWAE or a χ upper bound rather than ELBO.Source: §5.3 · p. 17

6. Benchmark datasets and application

The survey finds no standard multimodal benchmark: studies often adapt image datasets into toy multimodal settings, while others use genuinely multimodal datasets and applications, including robotics.

6. Benchmark datasets and application

  • Many studies create toy multimodal benchmarks from image datasets by pairing images with labels, altered-noise images, another image dataset, or added voice and text.Examples include MNIST, FashionMNIST, and SVHN; these are not originally multimodal datasets.Source: §6 · p. 17
  • True multimodal benchmarks include face and bird images with attributes or descriptions, hand images with keypoints, handwriting motion paired with images, and language-acoustic-visual time series.Examples named include CelebA, CUB-200, CUB-200-2011, Rendered Hand pose, UJI Char Pen, and CMU-MOSI.Source: §6 · p. 17
  • The survey distinguishes its defined setting from other VAE-based multimodal tasks, including audio-visual speech enhancement and joint representations of depth and RGB images.These examples do not infer and generate all modalities in the way required by the survey’s definition.Source: §6 · p. 17
  • Robot applications use multimodal models to reconstruct missing sensor modalities, detect anomalous actions, support robot-arm control, and contribute state or reward representations for multi-agent reinforcement learning.The cited examples use iCub, PR2, simulated robot-arm, and multi-robot settings.Source: §6 · p. 18

7. Future challenges

The survey identifies scaling to more modalities, integrating separate modality modules, extending models toward world models, and overcoming practical deployment barriers in robotics as future challenges.

7. Future challenges

  • Most surveyed models have been validated with only two or at most three modalities, although real-world settings involve many more.The paper points to prior cognitive-architecture work using visual, tactile, auditory, and word information from a robot.Source: §7 · p. 18
  • As modality count and variety grow, end-to-end training becomes harder, motivating separate modality modules whose inferences can be integrated.Coordinated models use two-step approaches but cannot infer shared representations from arbitrary modality sets; SERKET and Neuro-SERKET provide message-passing frameworks, while global-workspace-style communication is another possibility.Source: §7 · p. 18
  • Multimodal deep generative models could extend world models beyond image-only inputs toward prediction and decision-making from multiple kinds of information.The paper cites a proposal to implement multimodal prediction and decision-making using generative models based on human cognitive systems.Source: §7 · p. 18
  • In robotics, these models are not yet mainstream, and generalizing across environments would require large amounts of training data that are difficult to obtain in practice.The paper suggests transfer learning and continuous learning as possible responses to the data difficulty.Source: §7 · pp. 18–19
  • Large environments and many modalities may make implementations larger and more complex, complicating their maintenance and handling.The paper notes probabilistic modeling languages and deep generative modeling libraries as possible aids.Source: §7 · p. 19

8. Conclusion

The paper organizes multimodal deep generative models into coordinated and joint models, reviews their methods and applications, and discusses future directions.

8. Conclusion

  • Coordinated models are classified by how their objectives define closeness between inference distributions.The survey distinguishes distance between inference distributions from cross-modal generation loss.Source: §8 · p. 19
  • Joint models are organized around missing modalities, modality-specific latent variables, and weakly supervised learning.For missing modalities, the survey covers surrogate unimodal inference and aggregation of unimodal inference.Source: §8 · p. 19
  • The survey also summarizes benchmark datasets and applications and discusses future research directions.The authors intend the survey to support future research on multimodal deep generative models.Source: §8 · p. 19

Related papers

Loading 2207.02127v1…