PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models

Yuge Shi, N. Siddharth, Brooks Paige, Philip H. S. Torr

arXiv:1911.03393v1stat.MLcs.LG

TL;DR

Multimodal generative models should capture shared and private factors while supporting coherent joint and cross-generation and improving individual modalities. The paper proposes MMVAE, using a mixture-of-experts variational posterior, and reports that it satisfies these four criteria qualitatively and quantitatively across multimodal datasets, including image-language data.

  • Problem

    Existing multimodal generative approaches do not fully express the desired combination of latent decomposition, coherent generation, cross-generation, and multimodal synergy.

  • Method

    MMVAE uses a mixture-of-experts variational posterior over unimodal encoders within a multimodal variational autoencoder.

  • Results

    MMVAE demonstrates all four criteria qualitatively and quantitatively, including coherent image-caption generation and improved multimodal learning outcomes.

  • Takeaways & Limitations

    The framework supports more useful and generalisable multimodal representations by capturing abstract relationships between modalities for downstream tasks such as classification.

Abstract

from arXiv · show

Learning generative models that span multiple data modalities, such as vision and language, is often motivated by the desire to learn more useful, generalisable representations that faithfully capture common underlying factors between the modalities. In this work, we characterise successful learning of such models as the fulfillment of four criteria: i) implicit latent decomposition into shared and private subspaces, ii) coherent joint generation over all modalities, iii) coherent cross-generation across individual modalities, and iv) improved model learning for individual modalities through multi-modal integration. Here, we propose a mixture-of-experts multimodal variational autoencoder (MMVAE) to learn generative models on different sets of modalities, including a challenging image-language dataset, and demonstrate its ability to satisfy all four criteria, both qualitatively and quantitatively.

1 Introduction

The paper frames multimodal generative learning around four criteria: decomposed shared/private representations, coherent joint and cross-generation, and modality-level synergy. MMVAE addresses these goals for settings with complete multimodal training observations, missing modalities at test time, and challenging many-to-many mappings.

  • Multimodal observations provide correlations that can support shared embedding, reasoning, understanding, and information transfer across visual, linguistic, and physical perspectives.
  • Existing multimodal approaches broadly address the problem but fall short of expressing the complete range of expected behaviours.
  • The four criteria are latent factorisation, coherent joint generation, coherent cross-generation, and synergy.Latent factorisation separates shared and private aspects; synergy means multimodal observations improve single-modality generation.
  • Latent factorisation targets shared and private subspaces, while joint and cross-generation preserve semantic commonality across generated or conditioned modalities.Synergy additionally requires multimodal observations to yield more specific generations than either modality alone.
  • MMVAE uses a mixture-of-experts variational posterior to target all four criteria across multimodal datasets.The proposed setting assumes multiple modalities are always presented during training, supports missing modalities at test time, and evaluates many-to-many mappings including image-language transformation.

2 Related Work

Prior multimodal generative work largely focuses on one-way cross-modal generation, joint embeddings, or explicit joint models. The paper argues these approaches often target only one criterion, add modelling or inference requirements, and primarily address vision-to-vision settings.

  • Cross-modal generation: Cross-modal methods commonly learn a conditional model p(x1 | x2) for attribute-to-image, image-to-caption, or image-to-image translation, with non-interchangeable conditioning and generation modalities.
  • Domain adaptation: Domain-adaptation methods learn multimodal joint embeddings or projection spaces for classification and generation across domains.
  • Joint models: JMVAE learns a shared representation with a joint encoder, while unimodal encoders are trained to match it for missing-data handling.
  • Joint models: The paper characterises prior approaches as often targeting one criterion, requiring extra modelling or inference steps, ignoring induced latent structures, and focusing on vision-to-vision observations.

3 Methods

MMVAE models multimodal data with a VAE whose unimodal encoders form a mixture-of-experts joint posterior. Its objective extends importance-weighted learning through stratified modality sampling, avoiding the dominance and overconfidence concerns associated with products of experts.

  • Background: The multimodal generative model factorises as a prior over z and modality-specific likelihoods pθm(xm | z), with deep neural-network decoders.Because the true posterior is intractable, a variational posterior is optimized through an evidence lower bound using stochastic gradient descent.
  • Background: IWAE provides a tighter lower bound through a weighted multi-sample estimator, and its higher-entropy posteriors encourage each modality encoder to cover regions beyond its modality-specific information.
  • Mixture of experts: MMVAE factorises the joint posterior as an equally weighted mixture of unimodal posteriors, qΦ(z | x1:M) = Σm αm · qφm(z | xm), with αm = 1/M.The equal-weight assumption treats modalities as having comparable complexity.
  • MoE vs. PoE: Unlike POE, MOE lets experts vote rather than allowing one low-density or overconfident expert to dominate the combined prediction.This is presented as better suited to latent factorisation because it remains sensitive to information from all modalities.
  • MOE objective: Stratified sampling averages over modalities so samples from different encoders receive equal gradient weight while estimating tight bounds for individual modality terms.Differential weighting can cause modality dominance; MOE objectives add decoder-computation overhead, while DReG IWAE is used for gradient estimation.

4 Experiments

Experiments evaluate MMVAE on matched MNIST-SVHN digits and challenging CUB image-caption pairs, testing latent factorisation, joint and cross-modal coherence, and multimodal integration. MMVAE qualitatively and quantitatively demonstrates coherent generation and improved multimodal representations across these settings.

  • Datasets: Experiments use matched MNIST-SVHN digits to separate conceptual complexity from perceptual complexity, plus a challenging CUB image-language task.The MNIST-SVHN pairs share digit classes while varying styles and colours; CUB contains detailed bird images and fine-grained captions.
  • MNIST-SVHN results: MMVAE produces coherent joint and cross-generations for MNIST and SVHN, with corresponding outputs matching in digit identity.The comparison evaluates generation quality, digit matching, reconstructions, and cross-generation against MVAE.
  • MNIST-SVHN results: Per-dimension traversals show dimensions affecting only SVHN, only MNIST, or both, indicating a degree of latent factorisation.Per-dimension KL measures further indicate which modalities contribute information to each latent dimension.
  • MNIST-SVHN results: MMVAE’s latent space provides significantly better digit-classification accuracy than the single-VAE, while MVAE is dominated by MNIST representations.MVAE reaches 95.7% MNIST accuracy but only 9.10% SVHN accuracy in the cited single-modality comparison; adding both inputs improves SVHN while slightly reducing MNIST.
  • MNIST-SVHN results: MMVAE yields higher likelihoods and benefits from jointly available modalities, whereas MVAE shows lower multimodal conditional likelihoods than unimodal ones.For MMVAE, p(xm | xm, xn) ≥ p(xm | xm); for MVAE, p(xm | xm, xn) < p(xm | xm).
  • CUB results: On CUB, MMVAE generates image-caption pairs whose attributes align well, with average joint-generation correlation 0.263 versus 0.273 for the data itself.Cross-generation correlations are 0.104 for image-to-caption and 0.135 for caption-to-image.

5 Conclusion

The paper defines successful multimodal generative learning through four criteria and proposes MMVAE to satisfy them. Compared with POE, MMVAE is reported to perform better across these criteria on image↔image and image↔language datasets.

  • MMVAE targets latent decomposition, coherent joint generation, coherent cross-generation, and improved unimodal learning through multimodal observations.

A Tighter lower bound

The MOE construction yields a tighter lower bound than the standard objective, but its importance-weighted gradients can cause posterior collapse and harm cross-modal coherence.

  • Stratified sampling draws K joint-posterior samples by allocating L = K/M samples to each modality-specific posterior.
  • The MOE objective is a tighter lower bound on log pΘ(x1:M) than the comparison objective.
  • The tighter objective can adversely affect cross-modal generation and reduce joint-generation coherence.
  • Importance-weighted gradients can favor samples from different experts unequally, causing the joint posterior to collapse toward one modality.
  • Table 5 evaluates joint and cross-generation coherence using digit-matching probabilities.

B Multi-Modal Importance-Sampled ELBO

The multimodal importance-sampled ELBO uses modality-specific samples and stop-gradient importance ratios to reduce computational and estimation difficulties.

  • The ELBO is formulated with a basic MOE variational posterior over modality-specific encodings.
  • Each modality-specific term estimates other modalities’ likelihoods using samples from the corresponding modality’s encoding distribution.
  • Stop-gradient denominators prevent importance-ratio gradients from propagating through the corresponding posterior terms.
  • A single pass over each encoder and decoder makes the total objective computationally linear in M.
  • The standard IWAE gradient estimator can have high variance, motivating doubly reparameterized gradients for IWAE experiments.
  • Higher-entropy variational posteriors encourage each modality to assign probability beyond regions characterizing only itself.

E Qualitative results of MVAE implementation with MOE posterior

In the MVAE implementation, replacing the POE posterior with MOE enables recognizable cross-modal digit generation from SVHN inputs, while joint-generation coherence remains weak for both models.

  • Figure 9 compares POE and MOE in the MVAE codebase.
  • MOE generates recognizable MNIST digits from SVHN inputs, whereas the original POE-MVAE fails completely at cross-modal generation.
  • Neither MOE nor the original POE-MVAE performs well on coherent joint generation.

F Encoder and decoder architectures

This section identifies the CUB-Language dataset and presents the encoder and decoder architectures used in the experiments.

  • The experiments include the CUB-Language dataset.
  • Table 6 summarizes the encoder and decoder architectures.

G Qualitative results of MMVAE on CUB

This section presents qualitative MMVAE results on the CUB Image-Caption dataset, covering image reconstruction, caption reconstruction, and cross and joint generation.

  • MMVAE qualitative results are evaluated on the CUB Image-Caption dataset.
  • Figure 10 compares ground-truth images with their MMVAE reconstructions.
  • Figure 11 presents MMVAE caption reconstruction results.
  • Figure 12 shows MMVAE cross generation and joint generation from prior samples.

H Qualitative results of Wu and Goodman (2018)’s MVAE on CUB

This section reports qualitative MVAE results on CUB and finds strong vision reconstruction but poor language reconstruction, cross-modal generation, joint coherence, and multimodal improvement.

  • MVAE results cover reconstruction, cross generation, and joint generation across vision and language modalities.For vision generation, image features extracted from ResNet101 are reconstructed and matched to images by nearest-neighbour search.
  • Vision reconstruction is almost perfect, but other tasks perform poorly, especially when language is the input.Language reconstruction substitutes characteristics such as “small sized” with “medium sized” and “blue bird” with “white bird”.
  • Language-to-vision cross generation suffers from mode collapse, producing only two images for any given caption.
  • Language reconstruction and vision-to-language generation fail to capture key caption characteristics, while joint image-caption generation is incoherent.
  • -0.00523 averaged test-set CCA correlation suggests low coherence in the generated multimodal data.
  • Providing both modalities does not significantly improve reconstruction quality, with language generation omitting or inventing important bird characteristics.
Loading 1911.03393v1…