PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Generative Models for Scalable Weakly-Supervised Learning

Mike Wu, Noah Goodman

arXiv:1802.05335v3cs.LGstat.ML

TL;DR

Multimodal learning needs joint representations despite sparse complete observations and missing modalities. The paper introduces an MVAE with product-of-experts inference and sub-sampled training, achieving state-of-the-art performance with fewer parameters and strong results under weak supervision. The method reaches near-fully supervised translation quality with 1% aligned examples, although its translation is not competitive with state-of-the-art systems.

  • Problem

    Complete multimodal observations are scarce, while prior generative approaches either do not learn a joint distribution or require extra computation for missing modalities.

  • Method

    The MVAE combines per-modality inference networks through a product-of-experts structure and sub-sampled ELBO training over complete and partial observations.

  • Results

    The MVAE matches state-of-the-art performance on four bimodal datasets with fewer parameters and describes translation test data almost as well as full supervision using 1% aligned examples.

  • Takeaways & Limitations

    The shared MVAE representation supports arbitrary modality combinations, image-transformation modeling, and weakly supervised translation with incomplete supervision.

  • Takeaways & Limitations

    Translation results are not competitive with state-of-the-art translation, and the MVAE-Q variant requires an added constraint on variances.

Abstract

from arXiv · show

Multiple modalities often co-occur when describing natural phenomena. Learning a joint representation of these modalities should yield deeper and more useful representations. Previous generative approaches to multi-modal input either do not learn a joint distribution or require additional computation to handle missing data. Here, we introduce a multimodal variational autoencoder (MVAE) that uses a product-of-experts inference network and a sub-sampled training paradigm to solve the multi-modal inference problem. Notably, our model shares parameters to efficiently learn under any combination of missing modalities. We apply the MVAE on four datasets and match state-of-the-art performance using many fewer parameters. In addition, we show that the MVAE is directly applicable to weakly-supervised learning, and is robust to incomplete supervision. We then consider two case studies, one of learning image transformations---edge detection, colorization, segmentation---as a set of modalities, followed by one of machine translation between two languages. We find appealing results across this range of tasks.

1 Introduction

The MVAE addresses weakly supervised multimodal learning, where complete observations are scarce but larger unimodal datasets are available. It uses shared multimodal modeling across missing-data settings and performs well across benchmarks and case studies.

  • Motivation: Multimodal data can provide converging information for more invariant representations, but complete observations are expensive and sparse.Weak supervision may therefore consist of few fully observed examples alongside larger datasets containing only subsets of modalities.
  • Approach: The MVAE learns a joint distribution under weak supervision using a product-of-experts inference network and sub-sampled training.The approach avoids an inference network for every modality combination and trains with fully and partially observed examples.
  • Evaluation: The model matches state-of-the-art performance on MNIST, binarized MNIST, MultiMNIST, FashionMNIST, and CelebA while supporting large RNN and CNN encoders.These experiments include datasets with complex modalities such as character sequences and RGB images.
  • Case studies: With 18 CelebA attributes and image transformations such as colorization, edge detection, and segmentation treated as modalities, the MVAE learns shared multimodal structure.The image-transformation dataset is synthesized from CelebA by applying commonly studied computer-vision transformations.
  • Incomplete supervision: The MVAE captures a good joint representation when only a small percentage of examples are multimodal and is applied to weakly supervised machine translation.In translation, each language is treated as a modality.

2 Methods

The MVAE uses a product-of-experts inference network to construct posterior approximations for arbitrary modality subsets, while sub-sampled ELBO training learns the component networks needed for missing-modality inference. Its Gaussian formulation is analytically tractable, and the shared training scheme also supports incomplete supervision.

  • Multimodal inference: Conditional-independence assumptions motivate computing multimodal inference networks from unimodal components instead of training all 2^N networks separately.The construction uses N unimodal inference components to obtain the required subset-specific networks.
  • Gaussian formulation: MVAE-Q additionally uses a quotient by the prior, but its variance constraint can make the distribution difficult to handle in practice.The quotient is well-defined only when the relevant variance ordering holds element-wise.
  • Multimodal inference: The MVAE approximates the joint posterior with a prior expert multiplied by the inference networks for the observed modalities.For modality subset X, q(z|X) ∝ p(z) Q_i∈X ˜q(z|xi).
  • Gaussian formulation: For Gaussian priors and experts, products of experts remain Gaussian and have an analytical mean-and-covariance solution.The solution is expressed through expert precisions Ti = Vi^-1.
  • Training: Sub-sampled training combines ELBO terms for complete and partial observations without evaluating all 2^N modality subsets at every gradient step.This scheme trains individual inference networks for missing-modality test inputs and can sample from whatever modalities are present in weakly supervised examples.

3 Related Work

Prior multimodal generative models either condition in one direction or require separate inference networks and staged training for missing modalities. The MVAE instead combines variational distributions with a product-of-experts rule and trains efficiently beyond two modalities.

  • Conditional VAEs and multimodal autoencoders model p(x2|x1), so they do not support interchangeable conditioning between modalities.
  • JMVAE explicitly models the joint distribution but trains joint and unimodal inference networks with additional divergence terms for missing-data handling.
  • The triplet ELBO combines variational distributions using a product-of-experts rule but requires two-stage training and two new inference networks for missing-data combinations.
  • The MVAE uses product-of-experts inference and single-stage training, enabling efficient exploration of more than two modalities and weakly supervised learning.
  • Compared with RBMs, the MVAE uses gradient descent rather than contrastive divergence and supports faster models that can handle more data.

4 Experiments

Experiments compare the MVAE with established multimodal models on image datasets while varying only objectives and training procedures. The model matches state-of-the-art performance, and its multi-modality configurations are evaluated through parameter counts and CelebA attribute settings.

  • The experiments compare VAE, BiVCCA, JMVAE, and MVAE using consistent network architectures across models.
  • The MVAE matches state-of-the-art performance on MNIST, FashionMNIST, MultiMNIST, and CelebA.
  • Upweighting reconstruction error for low-dimensional modalities is important for learning a good joint distribution.
  • Table 1 compares inference-network parameter counts arising from how models combine modality-specific networks, including 19-network JMVAE19 and MVAE19 settings.
  • CelebA experiments treat either all 18 attributes as one modality or each attribute separately, producing the 19-modality configuration MVAE19.

5 Evaluation

The evaluation measures likelihood, inference quality, generated samples, and weak-supervision performance across image datasets. MVAE matches strong baselines while using fewer parameters and performs especially well when paired supervision is limited.

  • Generative samples: MVAE generates good-quality images and conditional samples that are largely matched to target labels.Unconditional samples use p(z), while conditional reconstructions infer z from the second modality.
  • Likelihood evaluation: MVAE is roughly equivalent to JMVAE in data log-likelihood while using far fewer parameters.CVAE can better capture p(x1|x2) but does not learn a joint distribution.
  • Inference quality: MVAE produces lower variance in log importance weights than other joint-distribution methods, indicating better inference networks.MVAE19 further lowers variance relative to MVAE on CelebA.
  • Training analysis: Increasing the sampled ELBO-term count k has little effect on data log-likelihood but reduces inference-distribution variance.The authors choose a small k as a computation–inference-quality tradeoff.
  • Weak supervision: MVAE surpasses all baselines in the middle supervision region, where deep models can train but supervised networks lack enough paired examples.In FashionMNIST, it equals a fully supervised network with two orders of magnitude fewer paired examples.

6 Case study: Computer Vision Applications

The MVAE models computer-vision transformations and their inverses as conditional distributions across six modalities. On CelebA, it jointly learns transformations including colorization, edge detection, segmentation, completion, and watermark removal.

  • Task design: The study treats the original image and five transformations as six modalities for learning conditional distributions and inverses.The transformations include colorization, edge detection, facial landmark segmentation, image completion, and watermark removal.
  • Data construction: The dataset applies ground-truth transformations to CelebA, including grayscale conversion, masking, watermark overlay, Canny edges, and facial-landmark masks.These procedures construct paired modality views from the original images.
  • Results: The MVAE produces reasonable reconstructions with good facial-landmark and edge extraction across transformation directions.The model encodes one modality and decodes another using its learned generative model.

7 Case study: Machine Translation

The machine-translation case study treats English and Vietnamese as modalities and trains on mostly unpaired data. With very sparse alignment, MVAE achieves near-fully-supervised test description quality, though translations are not state-of-the-art.

  • Task and data: The study frames weakly supervised translation as learning between English and Vietnamese modalities when only a small subset of sentence pairs is aligned.This addresses the cost of obtaining expert-translated bilingual data.
  • Setup: The experiment uses 113K English–Vietnamese sentence pairs, 100 latent dimensions, RNN encoders, and sequences up to 70 tokens.Word dropout and KL annealing are reported as important for preventing latent collapse.
  • Results: With 1% of aligned examples, MVAE describes test data almost as well as with fully supervised training; 5% reaches maximum performance.Table 4 reports test log-likelihoods averaged over three runs.
  • Results: The generated translations often capture the true meaning but are not extremely faithful and do not compete with state-of-the-art translation.The authors suggest combining MVAE with modern architectures such as transformers and attention.

8 Conclusion

The MVAE learns a joint distribution robust to missing data by sharing inference-network parameters through a product-of-experts structure and sub-sampled ELBO training. It matches state-of-the-art performance on four image datasets and shows promise on real-world datasets.

  • The MVAE learns a joint distribution and remains robust to missing data.
  • Product-of-experts inference shares parameters across modality combinations, scaling to an arbitrary number of modalities.
  • Sub-sampled ELBO optimization uses multimodal and unimodal examples to train inference networks efficiently.
  • The product-of-Gaussians inference structure requires variance conditions for a well-defined quotient, unless a regularized importance distribution is used.

D Additional Results using the Joint Inference Network

Additional experiments show that using the joint inference network preserves MVAE's state-of-the-art marginal-probability performance. The model is also fairly robust when modalities are randomly missing.

  • MVAE remains state-of-the-art when marginal probabilities are estimated using q(z|x1, x2).The induced distribution gives an unbiased importance-sampling estimate, so sufficiently many samples should produce similar log-likelihoods.
  • Figure 9 evaluates missing-modality robustness through attribute accuracy, log marginal approximations, and log conditional approximations as p changes.
  • The dataset-specific MVAE architectures use separate inference and decoder networks for image and label modalities.Figures 5 and 6 show architectures for MNIST and FashionMNIST, where x1 is an image and x2 is a label.
  • With p = 0.1 probability of retaining each modality, accuracy remains close to performance with full data.

G Table of Weak Supervision Results

The weak-supervision results document performance across several multimodal datasets and examine how ELBO sampling affects estimator variance. The experiments use consistent model comparisons and report strong robustness to incomplete supervision.

  • The MultiMNIST evaluation measures average accuracy for predicting each digit correctly from strings containing up to four digits.
  • Figure 9 reports close-to-best performance using only 10% of complete data as input features are randomly dropped.
  • Weak-supervision tables report MNIST digit accuracy and FashionMNIST attire-class accuracy as paired-example fractions vary.
  • Model comparisons keep architectures consistent while varying the objective and training procedure.The baselines include VAE, JMVAE, RBM, and a supervised neural-network comparator.
  • Increasing k steadily decreases log-importance-weight variance, even when the same reparameterization noise is reused across ELBO terms.This supports independent contributions from additional ELBO terms and additional sampling randomness.

J More on the Computer Vision Transformations

Additional examples illustrate the MVAE's multimodal image transformations and bidirectional English–Vietnamese generation. The results include reconstructions, transformations, and translations sampled through shared latent representations.

  • Computer vision transformations: The MVAE learns all six image transformations jointly under the product-of-experts inference network.
  • Computer vision transformations: Figure 11 compares ground-truth CelebA images with reconstructions, edge maps, and facial-landscape masks.
  • Computer vision transformations: Figures 12–14 show colorization, filling obscured image regions, and watermark removal by comparing ground truth with reconstructions.
  • Machine translation: Translations are generated in both directions by encoding one language with q(z|x) and decoding the other language from the shared latent variable.
  • Machine translation: Many translations capture close interpretations rather than exact wording, and the MVAE performs better on shorter sentences.
Loading 1802.05335v3…