Source-linked AI summary
Joint Multimodal Learning with Deep Generative Models
Masahiro Suzuki, Kotaro Nakayama, Yutaka Matsuo
TL;DR
The paper addresses the limitation that multimodal VAEs generally model conditioned relations in only one direction, despite the need to exchange heterogeneous modalities bidirectionally. It proposes JMVAE, which conditions modalities independently on a shared latent representation, and JMVAE-kl, which aligns modality-specific and multimodal encoders for missing-modality generation. Experiments report appropriate joint representations, improved generation and reconstruction relative to conventional VAEs, and bidirectional generation across modalities.
Problem
Existing multimodal generative models typically force modalities into conditioned relations that permit generation in only one direction, whereas bidirectional exchange requires a shared representation across heterogeneous modalities.
Method
JMVAE independently conditions all modalities on a shared latent representation and models their joint distribution; JMVAE-kl reduces divergence between the multimodal encoder and modality-specific encoders.
Results
JMVAE obtains appropriate joint representations, generates and reconstructs modalities similarly or more properly than conventional VAEs, and generates multiple modalities bidirectionally.
Takeaways & Limitations
The model supports bidirectional exchange between modalities with substantially different dimensions and structures, including face images and binary attributes.
Takeaways & Limitations
The paper leaves evaluation on multimodal datasets containing three or more modalities for future work.
Abstract
from arXiv · showhide
We investigate deep generative models that can exchange multiple modalities bi-directionally, e.g., generating images from corresponding texts and vice versa. Recently, some studies handle multiple modalities on deep generative models, such as variational autoencoders (VAEs). However, these models typically assume that modalities are forced to have a conditioned relation, i.e., we can only generate modalities in one direction. To achieve our objective, we should extract a joint representation that captures high-level concepts among all modalities and through which we can exchange them bi-directionally. As described herein, we propose a joint multimodal variational autoencoder (JMVAE), in which all modalities are independently conditioned on joint representation. In other words, it models a joint distribution of modalities. Furthermore, to be able to generate missing modalities from the remaining modalities properly, we develop an additional method, JMVAE-kl, that is trained by reducing the divergence between JMVAE's encoder and prepared networks of respective modalities. Our experiments show that our proposed method can obtain appropriate joint representation from multiple modalities and that it can generate and reconstruct them more properly than conventional VAEs. We further demonstrate that JMVAE can generate multiple modalities bi-directionally.
1 INTRODUCTION
The paper targets bidirectional exchange across heterogeneous modalities by learning a shared high-level representation. It introduces JMVAE and JMVAE-kl to model joint modality distributions and improve generation when modalities are missing.
- Motivation: Images, text, and tags have different dimensions and structures, motivating a joint representation that captures high-level concepts across modalities.The paper seeks bidirectional generation between modalities through this shared representation.
- Problem: Existing VAE-based multimodal models typically impose conditional relations, restricting generation to one direction.This limitation contrasts with the paper’s goal of exchanging modalities bidirectionally.
- JMVAE: JMVAE independently conditions all modalities on a latent joint representation z and models their joint distribution.This enables sampling modalities through conditional distributions such as p(x|w) and p(w|x).
- JMVAE-kl: JMVAE-kl prepares modality-specific encoders and reduces their divergence from the multimodal encoder to address collapsed samples when modalities are missing.The method is designed for missing-modality generation, where the inferred latent variable can become incomplete.
- Results: JMVAE extracts appropriate joint distributions and generates or reconstructs modalities similarly or more properly than conventional VAEs.The paper reports both qualitative and quantitative confirmation.
- Results: JMVAE generates multiple modalities bidirectionally despite differences such as high-dimensional color face images and low-dimensional binary attributes.Figure 1 illustrates varied images and attributes produced from an input image using one JMVAE model.
2 RELATED WORK
Prior multimodal generative models include bidirectional DBMs and VAE-based conditional models. The paper positions JMVAE against these approaches by emphasizing joint rather than fixed-direction modeling while retaining VAE scalability for high-dimensional data.
- Earlier multimodal models: Deep autoencoders and DBMs share hidden representations across modalities, and DBMs can generate modalities bidirectionally through joint likelihood learning.DBMs use undirected connections but rely on MCMC training.
- Earlier multimodal models: DBM training is computationally difficult for high-dimensional data such as natural images because it uses MCMC.This motivates using VAEs for large-scale and high-dimensional modalities.
- VAE-based models: Conditional VAEs apply variational methods to multiple modality pairs, including handwriting digits and labels, object images and rotations, faces and attributes, and images and captions.These approaches maximize conditional log-likelihoods.
- VAE-based models: CMMA aligns latent representations from single and multiple inputs but still generates modalities in a fixed direction.Its fixed-direction generation is identified as the main difference from JMVAE.
3 METHODS
The paper develops JMVAE, which models multiple modalities through a shared latent representation and joint distribution, enabling bidirectional generation. JMVAE-kl further trains single-input encoders to match the joint encoder for more reliable inference when modalities are missing.
- 3.2 JOINT MULTIMODAL VARIATIONAL AUTOENCODERS: JMVAE models modalities independently conditioned on a shared latent variable z, representing their joint distribution.For modalities x and w, the generative process factorizes as pθx(x|z)pθw(w|z).
- 3.2 JOINT MULTIMODAL VARIATIONAL AUTOENCODERS: The multimodal objective combines a latent-prior regularization term with expected reconstruction terms for both modalities.The resulting lower bound contains one reconstruction term for x and one for w, with qφ(z|x,w) as the encoder and separate modality decoders.
- 3.2 JOINT MULTIMODAL VARIATIONAL AUTOENCODERS: Different decoder networks and distributions are used because modalities can have different feature representations, such as continuous or binary values.The architecture and output distribution depend on each modality’s representation.
- 3.3 INFERENCE MISSING MODALITIES: At test time, missing modalities are represented by zero inputs, but high-dimensional missing data can make latent inference incomplete and cause sample collapse.This baseline approach is called JMVAE-zero.
- 3.3 INFERENCE MISSING MODALITIES: JMVAE-kl introduces single-input encoders and minimizes their KL divergence from the joint encoder to improve generation when modalities are missing.Its objective subtracts two KL divergence terms, weighted by α, for qφx(z|x) and qφw(z|w).
- 3.3 INFERENCE MISSING MODALITIES: Minimizing variation of information corresponds to reducing two negative conditional log-likelihoods, supporting bidirectional modality exchange.The paper relates this objective to variational inference and defines variation of information using log p(x|w) and log p(w|x).
4 EXPERIMENTS
The experiments evaluate JMVAE on MNIST and CelebA using marginal and conditional log-likelihoods, latent representations, and multimodal generation. Results indicate that JMVAE preserves or improves likelihood relative to relevant baselines, while JMVAE-kl addresses missing-modality reconstruction.
- Datasets: Experiments use MNIST handwriting images with digit labels and CelebA face images with 40 binary attributes as multimodal datasets.MNIST serves as a toy multimodal problem, while CelebA pairs high-dimensional color images with binary attributes.
- Evaluation method: Test marginal and conditional log-likelihoods evaluate generation and reconstruction, with higher test log-likelihood interpreted as better performance.The evaluation compares JMVAE marginal likelihoods with VAEs and conditional likelihoods with CVAEs and CMMAs.
- Interpretation: The evaluation interprets likelihood as evidence about both sample generation and joint-representation quality.A modality’s low likelihood may indicate that its representation is harmed by other modalities, whereas equal or higher likelihood suggests contribution from multimodal learning.
- MNIST: JMVAE-zero matches or slightly exceeds VAE marginal log-likelihood on MNIST, while JMVAE-kl reaches the table’s highest value with α = 0.01 and the single lower bound.For JMVAE-kl, larger α strengthens the regularization term and can reduce marginal test log-likelihood.
- MNIST: JMVAE conditional likelihoods with the multiple bound outperform CMMA except at α = 1, although CVAE achieves the highest conditional log-likelihood.Single-bound JMVAE-zero performs poorly when modality w is missing, whereas JMVAE-kl substantially improves this case.
- CelebA: On CelebA, JMVAE marginal and conditional log-likelihoods exceed those of competing methods, with larger improvement than on MNIST.The authors associate this improvement with joint representations benefiting reconstruction and generation when an input modality is large-dimensional and complicated.
4.4 QUALITATIVE EVALUATION
Qualitative experiments visualize joint latent representations and generate faces from attributes. They show attribute-organized representations, varied faces under fixed attributes, and bidirectional generation between faces and attributes.
- MNIST latent representation: JMVAE obtains a more discriminable MNIST latent representation when digit-label information is added, unlike CVAE representations distributed irrespective of labels.The two-dimensional visualization samples latent codes from VAE, CVAE, and JMVAE encoders with the top hidden layer constrained to two dimensions.
- Faces from attributes: On CelebA, JMVAE-GAN generates average faces and varied random faces conditioned on attribute settings.Average faces use the mean latent representation, while random faces add sampled variation with ζ = 0.6; each row shares the same noise sample.
- Faces from attributes: PCA visualizations show samples gathered by attribute, with variation locations remaining consistent across attributes.The authors interpret this as evidence that joint representation manifold learning works well for images and attributes.
- Bidirectional generation: JMVAE-GAN generates attributes from unlabeled images and changed images conditioned on varied attributes.The procedure infers attributes, generates an average face, modifies selected attributes, and reconstructs the changed image.
- Bidirectional generation: Unlike CMMA, JMVAE-GAN supports generation in both directions between faces and attributes.CMMA is described as generating images from attributes in only one direction.
5 CONCLUSION AND FUTURE WORK
The paper introduces JMVAE for joint multimodal representation learning and proposes JMVAE-kl to address missing modalities; experiments support bidirectional generation, with future evaluation on broader multimodal datasets.
- JMVAE independently conditions modalities on a joint representation, modeling their joint distribution.
- JMVAE-kl reduces divergence between the joint encoder and modality-specific encoders to prevent collapse when modalities are missing.
- Experiments on MNIST and CelebA found appropriate joint representations and high log-likelihoods.
- Future work will evaluate JMVAEs on multimodal datasets containing three or more modalities.
A RELATION BETWEEN THE OBJECTIVE OF JMVAE-KL AND THE VARIATION
The JMVAE-kl objective is connected to variation of information by relating its lower bound to negative variation-of-information terms and a nonnegative KL divergence.
- Variation of information is expressed through the sum of negative log-likelihoods for p(x|w) and p(w|x).
- The derivation introduces KL divergences between the joint encoder and each modality-specific encoder.
- Equation 5 bounds LJMkl(1)(x, w) below by adding the nonnegative divergence DKL(q(z|x, w)||q(z)).
- Maximizing Equation 4 is equivalent to minimizing variation of information within variational inference by maximizing lower bounds of negative variation of information.
B TEST LOWER BOUNDS
The appendix describes lower-bound estimators for marginal and conditional test log-likelihoods, importance-weighted tightening, and the associated MNIST and CelebA evaluation settings.
- Two lower bounds estimate the JMVAE test marginal log-likelihood p(x).
- The same two lower-bound strategies estimate the test conditional log-likelihood p(x|w).
- Nw is set to 5,000 on MNIST and 10 on CelebA for importance sampling.
- Figure 6 compares original and JMVAE-reconstructed images on MNIST and CelebA.
- Table 3 evaluates test log-likelihood on MNIST using marginal and conditional columns.
- Importance-weighted sampling produces tighter log-likelihood bounds from the lower-bound expressions.
- As the number of importance samples becomes extremely large, the multiple and single lower bounds converge to the same value.
C RECONSTRUCTED IMAGES
The JMVAE reconstructed original images properly on MNIST and CelebA using either a multiple or single encoder.
- The JMVAE properly reconstructs original images on both MNIST and CelebA.
- Reconstructions remain proper whether the multiple or single encoder is used.
D TEST JOINT LOG-LIKELIHOOD ON MNIST
On MNIST, JMVAE-zero and JMVAE-kl achieve almost identical joint test log-likelihoods, although performance worsens for JMVAE-kl when α is large. Figure 7 evaluates image generation from a single conditional input using one encoder.
- JMVAE-zero and JMVAE-kl have almost identical MNIST joint test log-likelihoods.JMVAE-zero is slightly lower.
- Figure 7 generates images from conditional distribution p(x|w) using a single encoder p(z|w) for both generations.
- JMVAE-kl’s test log-likelihood becomes much lower when α is large.
E IMAGE GENERATION FROM CONDITIONAL DISTRIBUTION ON MNIST
On MNIST, JMVAE-kl generates conditioned digit images properly from a single input, whereas JMVAE-zero cannot. Qualitative results also indicate that JMVAE-kl models qφx(z|x) properly.
- Figure 7 presents samples of x generated from a single input w.
- JMVAE-kl generates conditioned digit images properly, while JMVAE-zero cannot generate them.
- Qualitative evaluation indicates that JMVAE-kl models qφx(z|x) properly compared with JMVAE-zero.