PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Multimodal ELBO

Thomas M. Sutter, Imant Daunhawer, Julia E. Vogt

arXiv:2105.02470v2cs.LGstat.ML

TL;DR

Self-supervised multimodal generative models need posterior approximations that support coherent samples and learning the joint data distribution. MoPoE-VAE generalizes MVAE and MMVAE in a formulation that combines their benefits and achieves state-of-the-art results across reported multimodal measures.

  • Problem

    Multimodal VAEs must represent shared and modality-specific factors and produce semantically coherent samples, requirements not inherent to the unimodal VAE ELBO.

    Source: §1 · p. 1

  • Method

    MoPoE-VAE generalizes MVAE and MMVAE with a Mixture-of-Products-of-Experts posterior approximation that covers all modality subsets.

    Source: §1 · p. 2

  • Results

    MoPoE-VAE is the only method reported to reach state-of-the-art coherence, latent classification accuracies, and test log-likelihoods for all combinations of inputs.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    The generalized formulation encompasses MVAE and MMVAE as special cases and achieves state-of-the-art results in the reported experiments.

    Source: §1 · p. 2, §5 · p. 9

    The framework focuses on scalable models and chooses product-of-experts posterior approximations for modality subsets, though non-scalable fusion methods are also possible.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

Multiple data types naturally co-occur when describing real-world phenomena and learning from them is a long-standing goal in machine learning research. However, existing self-supervised generative models approximating an ELBO are not able to fulfill all desired requirements of multimodal models: their posterior approximation functions lead to a trade-off between the semantic coherence and the ability to learn the joint data distribution. We propose a new, generalized ELBO formulation for multimodal data that overcomes these limitations. The new objective encompasses two previous methods as special cases and combines their benefits without compromises. In extensive experiments, we demonstrate the advantage of the proposed method compared to state-of-the-art models in self-supervised, generative learning tasks.

1 INTRODUCTION

Multimodal learning offers shared self-supervision, but models must capture modality-specific and shared factors while producing coherent samples and learning joint distributions. MoPoE-VAE generalizes MVAE and MMVAE by combining their posterior-approximation approaches and modeling joint posteriors for all modality subsets.

1 INTRODUCTION

  • Multiple data types provide shared information for self-supervision, while labeling them for fully supervised learning is time-consuming and expensive.The introduction motivates models that learn efficiently from multiple data types without supervision.Source: §1 · p. 1
  • Multimodal VAEs must represent modality-specific and shared factors and generate semantically coherent samples, requirements not inherent to the unimodal VAE ELBO.Handling missing modalities naively requires 2M encoders for M data types, motivating the focus on scalable multimodal VAEs.Source: §1 · p. 1
  • MVAE and MMVAE differ in their joint posterior approximation functions, whose choice influences model properties.MVAE uses a geometric mean for a sharp posterior and good joint-distribution approximation, whereas MMVAE uses an arithmetic mean that better learns unimodal and pairwise conditional distributions.Source: §1 · pp. 1–2
  • MoPoE-VAE generalizes the two approaches and combines their benefits without considerable trade-offs.Its Mixture-of-Products-of-Experts formulation encompasses MVAE and MMVAE as special cases.Source: §1 · p. 2
  • MoPoE-VAE achieves state-of-the-art results and approximates the joint posterior for all subsets of modalities.The introduction reports empirical validation in Section 4.Source: §1 · p. 2

2 RELATED WORK

Scalable multimodal VAEs approximate joint distributions and support inference from subsets of modalities, but rely on assumptions about posterior fusion. Related work motivates combining product- and mixture-based posteriors to retain their respective benefits.

2 RELATED WORK

  • Scalable multimodal VAEs use one model for the joint distribution and its marginal and conditional distributions, rather than exponentially many models for modality subsets.The related-work discussion identifies product- and mixture-based fusion as prior approaches and presents a mixture of products as their generalization.Source: §2 · p. 2
  • MoPoE unites product- and mixture-based joint posteriors; applying products across all modality subsets yields a valid lower bound on the joint log-likelihood.The MVAE uses a product of unimodal posteriors, while the MMVAE uses a mixture; the MVAE's subsampling of unimodal log-likelihoods for missing modalities no longer guarantees a valid bound.Source: §2 · p. 2
  • MoPoE computes joint posteriors for all modality subsets, enabling efficient many-to-many translations.MMVAE uses unimodal posteriors during training and cannot take advantage of multiple present modalities.Source: §2 · p. 2
  • Table 1 compares properties of prior scalable multimodal VAEs and MoPoE-VAE, including modality aggregation, multimodal posteriors, and missing-modality handling.The caption notes that MVAE's subsampling of unimodal ELBOs for missing modalities yields an invalid bound on the joint log-likelihood.Source: §2 · p. 3, Table 1 · p. 3

3 METHOD

The method optimizes a joint-data ELBO across available modality subsets, then uses abstract mean functions to make inference scalable. Its Mixture-of-Products-of-Experts formulation combines subset-level PoE posteriors with a MoE over subsets, generalizing MVAE and MMVAE.

3.1 PRELIMINARIES

  • An ELBO is a tractable lower bound on the marginal log-likelihood, becoming exact when the approximate posterior matches the true posterior.Maximizing it minimizes the KL divergence between those posteriors.Source: §3.1 · p. 3

3.2 APPROXIMATING pθ(z|X) IN CASE OF MISSING DATA TYPES

  • Maximizing the ELBO for every available modality subset optimizes a convex combination of KL divergences across the powerset.This objective remains a bound on the joint log-likelihood, unlike a subset-specific bound on log pθ(Xk).Source: §3.2 · p. 4

3.3 SCALABLE INFERENCE USING ABSTRACT MEAN FUNCTIONS

  • The PoE’s geometric mean yields sharp joint posteriors, while the MoE’s arithmetic mean better optimizes individual experts and conditional distributions.Abstract means unify these aggregation choices; f(p)=ap+b gives the arithmetic mean, while f(p)=log p gives the geometric mean.Source: §3.3 · p. 4

3.4 GENERALIZED MULTIMODAL ELBO

  • The MoPoE objective is a valid lower bound on the joint log-likelihood, log pθ(X).Because its posterior approximation is a mixture distribution, it is a valid variational distribution.Source: §3.4 · p. 5

3.5 THE GENERAL FRAMEWORK

  • MoPoE combines unimodal posterior approximations within each subset using a PoE, then combines subset approximations using a MoE.For Gaussian posteriors, the PoE has a closed-form computation; MVAE and MMVAE arise as special cases using the full subset or unimodal subsets, respectively.Source: §3.5 · pp. 5–6

4 EXPERIMENTS & RESULTS

The experiments compare multimodal models on MNIST-SVHN-Text, PolyMNIST, and CelebA using latent classification, generated-sample coherence, and test log-likelihood. Results show dataset-dependent strengths, with MoPoE-VAE performing well across input subsets and matching baselines in some settings.

4 EXPERIMENTS & RESULTS

  • The evaluation covers three datasets and measures latent representations with linear classification, generated-sample coherence with pretrained classifiers, and joint-distribution fit with test log-likelihoods.The experiments use MNIST-SVHN-Text, the five-modality PolyMNIST dataset, and bimodal CelebA.Source: §4 · p. 6

4.1 MNIST-SVHN-TEXT

  • On MNIST-SVHN-Text, MoPoE-VAE is the only model with state-of-the-art coherence, latent classification, and test log-likelihood across input combinations, though MVAE leads SVHN generation.MoPoE-VAE achieves a superior joint-coherence/log-likelihood trade-off overall.Source: §4.1 · p. 6, Figure 1 · p. 6

4.2 POLYMNIST

  • PolyMNIST contains five image modalities sharing a digit label but varying in background and handwriting style, enabling evaluation beyond two modalities.The dataset has 60,000 training tuples and 10,000 test tuples, with no MNIST digit shared between the splits.Source: §4.2 · p. 7
  • Across PolyMNIST input subsets, MoPoE-VAE exceeds alternatives on latent classification and conditional coherence; with five inputs its log-likelihood matches MVAE, and with one input it matches MMVAE.MMVAE has higher joint coherence, while MoPoE-VAE is superior for most subsets and matches prior methods in cases favoring them.Source: §4.2 · pp. 7–8, Figure 3 · p. 8

4.3 BIMODAL CELEBA

  • On CelebA, MoPoE-VAE matches the baselines; common attributes such as gender and smiling are learned well, while subtle and rare attributes are harder to generate consistently.The dataset pairs face images with text describing labeled attributes, and negatively labeled attributes are omitted from the text.Source: §4.3 · p. 8

5 CONCLUSION

The conclusion presents a generalized multimodal ELBO that combines benefits of prior approaches and reports experimental advantages, while matching prior performance on tasks that favor them. The appendix proofs relate KL-divergence objectives to the joint probability and establish a multimodal ELBO lower bound.

5 CONCLUSION

  • The proposed formulation generalizes prior approaches and combines their benefits; experiments report advantages over state-of-the-art models and performance matching prior work on tasks that favor it.The authors propose evaluating extensions and other abstract mean functions in future work.Source: §5 · pp. 8–9
  • Table 5 reports classification and coherence results as mean average precision over attributes for image, text, and joint modalities.Figure 4 shows images conditionally generated by MoPoE-VAE from text.Source: Table 5 · p. 9, Figure 4 · p. 9

A.1 PROOF OF LEMMA 1

  • The proof relates a sum of KL-divergences to the joint probability and uses KL non-negativity to derive a lower bound on the joint log-probability.The proof identifies the KL-divergence sum as usable for describing the joint probability.Source: §A.1 · p. 12

A.2 PROOF OF LEMMA 2

  • The appendix states that LMoPoE is a multimodal ELBO, lower-bounding the joint log-probability.The proof notes that its sums range over all subsets in the modality power set.Source: §A.2 · p. 12

A.3 PROOF OF LEMMA 3

The proof shows that the proposed objective is a valid, tighter lower bound than a convex combination of subset ELBOs, and that maximizing it minimizes a convex combination of KL-divergences.

A.3 PROOF OF LEMMA 3

  • Maximizing the proposed objective minimizes the convex combination of KL-divergences across the modality power set.The proof states this follows by deriving the objective from the KL-divergence expression in Equation (5).Source: §A.3 · pp. 12–13
  • The proposed objective is a valid lower bound on the joint log-probability and tighter than a convex combination of subset ELBOs.The proof derives the inequality using Jensen’s inequality and the convexity of f(t) = t log t.Source: §A.3 · p. 13
  • The proof applies Jensen’s inequality to express a mixture of subset posterior approximations and establish the lower-bound inequality.It uses the convexity of f(t) = t log t to compare the mixture expectation with the sum of expectations.Source: §A.3 · p. 13

B EVALUATION OF EXPERIMENTS

The evaluation considers latent-representation classification, generated-sample coherence, and test-set log-likelihood. It also describes exclusions from prior-work comparisons, an equivalence to Shi et al.’s formulation, and the datasets used.

B EVALUATION OF EXPERIMENTS

  • Models are evaluated on latent-representation classification, generated-sample coherence, and test-set log-likelihood.For conditional generation, coherence requires generated samples to match the input samples under the classifiers’ labels.Source: B EVALUATION OF EXPERIMENTS · p. 14

B.1 COMPARISON TO PREVIOUS WORKS

  • The comparisons omit importance samples used by Shi et al. (2019) and dynamic priors used by Sutter et al. (2020), because these are outside the focus on posterior approximation.The authors state importance samples could be introduced to all objectives, and dynamic priors could be added to all formulations.Source: §B.1 · p. 14

B.1.1 EQUIVALENCE TO ELBO FORMULATION IN SHI ET AL. (2019)

  • The appendix says its Equations (32) and (33) are equivalent to the first equation on page 5 of Shi et al. (2019), with the other formulation differing because it uses importance samples.The passage identifies the equivalence as a reformulation of Equation (10).Source: §B.1.1 · pp. 14–15

B EVALUATION OF EXPERIMENTS

  • Table 6 reports MNIST-SVHN-Text generation coherence, with generated modalities above the horizontal line and their conditioning subsets below it.Joint coherence is the exception to that layout.Source: Table 6 · p. 16

C.1 DATASET

C.2 EXPERIMENTAL SETUP

The experimental setup fixes latent dimensions, training duration, and β, and uses stated architectures, activations, and optimizer settings.

C.2 EXPERIMENTAL SETUP

  • Latent spaces have 20 dimensions for every modality, model, and run, and the models train for 150 epochs.The reported results in Tables 2–4 use β = 5.0.Source: §C.2 · p. 16
  • MNIST uses fully connected encoder and decoder layers, while SVHN and text use feed-forward convolutional layers.The architectures match those used in Sutter et al. (2020).Source: §C.2 · p. 16
  • All layers use ReLU activations, and training uses Adam with an initial learning rate of 0.001.Source: §C.2 · p. 16

C.3 ADDITIONAL RESULTS

The additional results compare conditional-generation coherence with log-likelihood, report qualitative trade-offs among the models, and find competitive performance for MoPoE-VAE against a more expensive objective.

C.3 ADDITIONAL RESULTS

  • Table 6 adds the coherence results with standard deviations across five runs, omitted from the main text because of space restrictions.Source: §C.3 · p. 16
  • The conditional-generation evaluations plot coherence against joint log-likelihood for single-modality generation from two inputs and two-modality generation from one input.Each plot reports means across five runs with standard deviations; the evaluated β values are 0.5, 1.0, 2.5, 5.0, 10.0, and 20.0.Source: §C.3 · p. 16, Figure 5 · p. 17, Figure 6 · p. 17
  • Among 100 randomly generated samples, MVAE best approximates the joint distribution in sample quality but has limited coherence; MoPoE approximates MVAE’s sample quality with state-of-the-art coherence, while MMVAE has higher coherence but limited sample quality.Source: §C.3 · pp. 16–17, Figure 7 · p. 18
  • MoPoE-VAE achieves results competitive with Equation (5), which minimizes ELBOs across every modality subset but is computationally more expensive to optimize.Table 7 reports test set log-likelihoods for the two objectives at β = 2.5.Source: §C.3 · p. 18, Table 7 · p. 18

D.1 DATASET

PolyMNIST is constructed by compositing digit images with modality-specific backgrounds; the appendix also specifies the training setup used for these results.

D.1 DATASET

  • PolyMNIST is made by binarizing each MNIST digit and inverting the colors of a random 28×28 background crop wherever the digit is visible.The listed backgrounds include a fractal tree, The Scream, Lena, a star field, and shingles.Source: §D.1 · p. 19

D.2 EXPERIMENTAL SETUP

  • All models use 512-dimensional latent spaces, β = 2.5, the same architectures, and 300 training epochs with Adam at an initial learning rate of 0.001.Source: §D.2 · p. 19

D.3 QUALITATIVE RESULTS

The appendix presents qualitative comparisons across modalities and describes the CelebA data and model setup, including shared and modality-specific latent spaces.

D.3 QUALITATIVE RESULTS

  • Figures 8–11 compare reconstructions and generated images; conditioning on four modalities produces more coherent samples than conditioning on one in Figure 11.For conditional samples, columns vary posterior samples, ideally changing style without changing digit labels.Source: Figure 8 · p. 19, §D.3 · p. 19, Figure 11 · p. 20

E.1 DATASET

  • CelebA text strings include attributes present in a face but omit absent attributes, making the text modality more challenging.Source: §E.1 · p. 21

E.1.1 MODALITY-SPECIFIC LATENT SPACES

  • The CelebA model separates shared information from modality-specific information, allowing each modality to encode its own specific content in a separate latent space.The shared and modality-specific posterior approximations are assumed Gaussian.Source: §E.1.1 · p. 21

E.2 EXPERIMENTAL SETUP

  • The shared and modality-specific latent spaces each have 32 dimensions, for 64 dimensions per modality; models train for 200 epochs with β = 2.5.The image and text encoders and decoders use residual blocks; classification uses average precision because labels are imbalanced.Source: §E.2 · p. 21

E.3 ADDITIONAL RESULTS

CelebA evaluations show attribute-level differences in generated-sample coherence and learned representations, while random generations are high quality but retain minor artifacts.

E.3 ADDITIONAL RESULTS

  • Attribute-specific evaluations across all modality subsets find differences in average precision for both sample coherence and latent representations.Figure 12 notes that different attributes are not learned equally well; Figure 13 shows learned latent representations.Source: §E.3 · p. 21, Figure 12 · p. 22, Figure 13 · p. 23
  • Randomly generated CelebA samples are high quality and cover a wide variety of attributes, though minor artifacts suggest room for more rigorous hyperparameter search.Figure 14 presents these samples.Source: §E.3 · p. 21, Figure 14 · p. 23

Related papers

Loading 2105.02470v2…