Source-linked AI summary
Deep Variational Canonical Correlation Analysis
Weiran Wang, Xinchen Yan, Honglak Lee, Karen Livescu
TL;DR
The paper addresses limitations of deep multi-view methods that lack a generative latent-variable interpretation and are difficult to optimize. It introduces VCCA and VCCA-private, using variational inference with shared and view-specific latent variables. Across downstream tasks, the methods are competitive with or improve upon prior state-of-the-art results while being easier to train.
Problem
Existing deep CCA methods can lack a generative latent-space model, reconstruct inputs poorly, and be difficult to optimize because their constraints couple all training samples.
Method
VCCA extends CCA with nonlinear deep observation models and variational latent-posterior inference, while VCCA-private also models view-specific private variables.
Results
VCCA and its variants are competitive with or improve upon prior state-of-the-art results on downstream prediction tasks while being much easier to train.
Takeaways & Limitations
Modeling shared and private latent factors provides higher-quality features and reconstructions while retaining scalable backpropagation-based training.
Takeaways & Limitations
The current model uses standard black-box variational inference and leaves more expressive inference, alternative priors, and other observation models for future work.
Abstract
from arXiv · showhide
We present deep variational canonical correlation analysis (VCCA), a deep multi-view learning model that extends the latent variable model interpretation of linear CCA to nonlinear observation models parameterized by deep neural networks. We derive variational lower bounds of the data likelihood by parameterizing the posterior probability of the latent variables from the view that is available at test time. We also propose a variant of VCCA called VCCA-private that can, in addition to the "common variables" underlying both views, extract the "private variables" within each view, and disentangles the shared and private information for multi-view data without hard supervision. Experimental results on real-world datasets show that our methods are competitive across domains.
1. Introduction
VCCA extends the probabilistic latent-variable interpretation of CCA to nonlinear deep observation models, addressing limitations of discriminative deep CCA while supporting variational inference and generative modeling. VCCA-private additionally separates shared information from view-specific factors.
- Motivation: Multi-view learning seeks features that capture common variation across complementary measurements of the same underlying signal.These features can support exploratory analysis and downstream tasks.
- Prior work: CCA projects two views into lower-dimensional spaces whose projections are maximally correlated, with a probabilistic interpretation based on shared latent variables.Under Gaussian linear assumptions, conditional latent-variable expectations occupy the same spaces as the corresponding linear CCA projections.
- Prior work: DCCA learns nonlinear features with separate deep networks and maximizes the canonical correlation of their outputs, but does not define a generative latent-space sampling model.Its optimization, and that of DCCAE, is also challenging because constraints couple all training samples.
- VCCA: VCCA extends linear CCA's latent-variable model to nonlinear observation models parameterized by deep neural networks.The model's marginal likelihood and latent-variable inference are intractable, motivating variational lower bounds and approximate inference.
- VCCA: VCCA parameterizes latent posteriors from an available input view and uses Monte Carlo sampling, reparameterization, and minibatch stochastic gradient descent to jointly optimize the network weights.These choices provide unbiased gradient estimates for scalable training.
- VCCA-private: VCCA-private augments shared latent variables with view-specific private variables, disentangling shared and private information without hard supervision and enabling high-quality samples for each input view.This variant broadens the model beyond information common to both views.
2. Variational CCA
VCCA extends latent-variable CCA to nonlinear DNN observation models by optimizing a variational likelihood bound, while VCCA-private adds view-specific latent variables for information not captured by shared variables.
- VCCA: VCCA replaces linear observation models with nonlinear DNN-parameterized models, making both marginal likelihood and latent-variable inference intractable.The model assumes the views are conditionally independent given shared latent variables.
- VCCA: VCCA approximates pθ(z|x) with qφ(z|x) and maximizes a variational lower bound on the marginal data log-likelihood.The posterior is parameterized by another DNN, and the bound includes a KL divergence to the latent prior.
- VCCA: Monte Carlo sampling with the reparameterization trick enables unbiased minibatch gradient estimates and joint stochastic optimization of the DNN weights.The expected log-likelihood is approximated using samples from the approximate posterior.
- Connection to MVAE: The VCCA objective regularizes noisy latent representations through a KL penalty, distinguishing it from the multi-view autoencoder objective.As posterior variances approach zero, the KL penalty becomes large and prevents that limiting behavior.
- VCCA-private: VCCA-private introduces view-specific hidden variables h_x and h_y to explain aspects of each view not captured by the common variable z.Its factored posterior uses separate DNN-parameterized factors for z, h_x, and h_y.
- Choice of lower bounds: Using q(z|x) is convenient when only x is available at test time, while bi-VCCA variants combine bounds from q(z|x) and q(z|y).When both views are available, the method concatenates projections from the two posteriors as features.
3. Related work
The paper relates VCCA to deep generative, probabilistic multi-view, and information-bottleneck models, emphasizing DNN parameterization with tractable objectives and efficient training.
- Deep generative models: Deep generative models use expressive neural networks to model complex input distributions and potentially reveal underlying data structure through realistic samples.The related work includes unsupervised and semi-supervised generative models with latent variables for labels and intra-class variability.
- Deep multi-view probabilistic models: Earlier deep multi-view probabilistic models based on restricted Boltzmann machines require difficult inference and learning procedures involving variational approximations and Gibbs sampling.VCCA instead relies on sampling from simple standard distributions.
- Related multi-view models: VCCA and VCCA-private differ from related graphical models by parameterizing probability distributions with DNNs while retaining tractable objectives and efficient end-to-end training.The DNN parameterization is presented as increasing model power relative to previous work.
- Information bottleneck: The information bottleneck is equivalent to linear CCA for Gaussian inputs, while a parallel variational extension differs from basic VCCA by omitting the first-view likelihood and adding a KL trade-off parameter.The comparison concerns the structure of the variational objectives.
4. Experimental results
Experiments across noisy MNIST, XRMB speech-articulation, and MIR-Flickr evaluate learned representations through classification, phone recognition, and retrieval. VCCA and VCCA-private generally provide competitive downstream features while retaining generative and disentangling capabilities.
- Evaluation setup: Experiments compare CCA, DCCA, DCCAE, MVAE, contrastive loss, VCCA, and VCCA-private across image-image, speech-articulation, and image-text tasks.The evaluation uses linear SVM classification error, mean phone error rate, and mean average precision for retrieval.
- Noisy MNIST: Noisy MNIST pairs rotated view 1 images with independently selected same-identity, noisy view 2 images, making digit identity the only shared variable.The dataset contains 60K/10K training/testing images and uses a 50K/10K training/tuning split.
- Noisy MNIST: Dropout improves shared-variable class separation for both VCCA models, with VCCA-private slightly outperforming VCCA; MVAE does not achieve comparable separation.The comparison uses 2D t-SNE visualizations of shared variables on noisy MNIST test data.
- Noisy MNIST: VCCA reconstructs noisy view 2 inputs as prototypical same-digit images, whereas VCCA-private better reconstructs individual styles using private variables.VCCA-private’s reconstruction standard deviation is low within the digit and high outside it, indicating separation of digit content and background noise.
- Noisy MNIST: VCCA-private’s private-variable embeddings group different identities with similar rotations, while rotation changes smoothly and carries little class information.This supports disentanglement of shared class information from private style information.
- XRMB and MIR-Flickr: On broader benchmarks, VCCA and VCCA-private are competitive on XRMB phone error rates and achieve higher MIR-Flickr mAP than the compared methods.On MIR-Flickr, the methods exceed the prior MVRBM state-of-the-art mAP of 0.607; bi-VCCA-private reaches mAP 0.687 with both views at test time.
5. Conclusions
The paper presents VCCA as a scalable deep generative model for multi-view representation learning and reports that VCCA-private can disentangle shared and private variables. Across downstream prediction tasks, the methods are competitive with or improve upon prior state-of-the-art results, while future work targets richer priors, inference methods, and observation models.
- VCCA models multiple views as generated from a small set of shared latent variables and is trained efficiently by backpropagation.
- VCCA-private models view-specific private variables in addition to shared variables, improving disentanglement and reconstruction quality.
- VCCA and its variants are competitive with or improve upon prior state-of-the-art results in downstream prediction tasks and are easier to train.
- Future work considers richer priors, more expressive variational inference, and alternative observation models such as adversarial networks.
A. Derivation of the variational lower bound
The derivation introduces a variational lower bound on the marginal data likelihood using an approximate posterior over latent variables given one view. The bound follows from the nonnegativity of KL divergence.
- The marginal data likelihood is lower-bounded using qφ(z|x), an approximate posterior over the latent variable conditioned on view x.
- The derivation uses the nonnegativity of KL divergence to establish the variational lower bound.
- L(x, y; θ, φ) is a lower bound on the data log-likelihood log pθ(x, y).
B. Derivation of the variational lower bound
The passage indicates that the private-variable lower-bound derivation proceeds analogously to the VCCA derivation.
- The private-variable variational lower bound is derived similarly to the VCCA lower bound.
C. Analysis of orthogonality between shared
The paper measures whether shared and private representations are orthogonal and evaluates how dropout affects this separation on noisy MNIST. Dropout produces better orthogonality than training without dropout.
- Orthogonality is evaluated between shared and private variables using matrices of posterior means over a sample set.
- The orthogonality score is zero for orthogonal variables and reaches 1 when equally dimensional variables are nearly identical.
- On noisy MNIST, dropout achieves better orthogonality between shared and private variables from both views than training without dropout.
D. Additional reconstruction results of noisy
Figure 8 provides additional noisy-MNIST reconstruction examples comparing VCCA and VCCA-private, highlighting the effect of private variables.
- The additional examples are intended to demonstrate how private variables affect reconstruction.
- Figure 8 shows sample reconstructions of view 2 images from the noisy MNIST test set.
- The examples compare reconstructions produced by VCCA and VCCA-private.
E. Additional generation examples for noisy
VCCA-private generates diverse samples by varying private variables while fixing shared variables, preserving input identity while changing orientation.
- Manifold traversal fixes shared variables inferred from an input digit while sampling diverse private variables.Samples are generated as x′ ∼pθ(x|z, hx) with diverse hx ∼p(hx).
- With shared variables fixed, generated samples almost always retain the input digit’s identity.
- The generated samples vary substantially in orientation, the main variation source in the first view.
- Figure 9 arranges inputs in the first column and generated samples in a 10-by-10 matrix, sharing z by row and hx by column.
bi-VCCA-private on MIR-Flickr
On MIR-Flickr, bi-VCCA and bi-VCCA-private are evaluated for unimodal retrieval across different µ values, with improvements from varying µ especially important for private-variable models.
- The evaluation varies µ across bi-VCCA and bi-VCCA-private objectives for unimodal retrieval.
- At µ = 1, the bi-VCCA objectives reduce to VCCA and VCCA-private.
- Different µ values produce a non-trivially important improvement for models with private variables.
- Table 2 reports mean average precision for bi-VCCA and bi-VCCA-private features on the MIR-Flickr validation set.