Source-linked AI summary
On Deep Multi-View Representation Learning: Objectives and Optimization
Weiran Wang, Raman Arora, Karen Livescu, Jeff Bilmes
TL;DR
The paper studies how to learn useful representations from multiple unlabeled views when only one view is available for downstream use. It compares reconstruction- and correlation-based deep objectives, develops variants, and examines minibatch optimization and kernel-versus-neural-network trade-offs. CCA-based approaches generally outperform reconstruction-based alternatives, with DCCAE the best overall performer across several tasks.
Problem
The paper addresses representation learning from multiple training-time views when downstream tasks provide only one view.
Method
The paper compares deep autoencoder, deep CCA, and new hybrid objectives, while analyzing stochastic optimization for DCCA and computational trade-offs with kernel methods.
Results
CCA-based models outperform autoencoder-based and related alternatives on several tasks, and DCCAE is the best overall performer.
Takeaways & Limitations
Correlation-based objectives, especially DCCAE, are favored for multi-view representation learning within the evaluated tasks.
Takeaways & Limitations
Exact KCCA is computationally challenging for large datasets because it requires storing kernel matrices and decomposing an N × N matrix.
Abstract
from arXiv · showhide
We consider learning representations (features) in the setting in which we have access to multiple unlabeled views of the data for learning while only one view is available for downstream tasks. Previous work on this problem has proposed several techniques based on deep neural networks, typically involving either autoencoder-like networks with a reconstruction objective or paired feedforward networks with a batch-style correlation-based objective. We analyze several techniques based on prior work, as well as new variants, and compare them empirically on image, speech, and text tasks. We find an advantage for correlation-based representation learning, while the best results on most tasks are obtained with our new variant, deep canonically correlated autoencoders (DCCAE). We also explore a stochastic optimization procedure for minibatch correlation-based objectives and discuss the time/performance trade-offs for kernel-based and neural network-based implementations.
1. Introduction
Multi-view representation learning uses paired views available during training to improve a representation from the single view available downstream. The paper compares reconstruction- and correlation-based deep objectives, introducing variants and reporting empirical and optimization analyses.
- Motivation: Multiple views can provide complementary information for learning features from the primary view available at test time.Examples include audio-video, audio-articulation, image-text, multilingual text, and words-context.
- Prior approaches: Deep multi-view methods commonly use autoencoder objectives that reconstruct one or more input views from a shared representation.The shared encoder extracts features from the primary view, while separate decoders reconstruct each view.
- Prior approaches: Deep CCA learns nonlinear features for two views by maximizing their canonical correlation.CCA and its kernel and neural extensions are established approaches for multi-view feature learning and dimensionality reduction.
- Paper scope: The paper compares prior and new DNN-based approaches with linear and kernel CCA when the second view is unavailable downstream.The study evaluates methods in unsupervised multi-view feature learning and includes comparisons across several task types.
- Findings: CCA-based approaches generally outperform unconstrained reconstruction-based approaches, while DCCAE is the consistent winner across several tasks.The paper also studies stochastic optimization for deep CCA and releases implementations plus a simulated two-view MNIST benchmark.
2. DNN-based multi-view feature learning
The paper reviews DNN-based objectives for multi-view feature learning, contrasting reconstruction, correlation, and distance-based approaches. It proposes DCCAE, which combines canonical correlation with autoencoder reconstruction terms, and discusses optimization trade-offs.
- Overview: DNN-based multi-view methods learn representations from paired views even though only the primary view is available for downstream tasks.The feature transformation is trained using both views, while the resulting primary-view representation supports later tasks.
- Reconstruction-based methods: SplitAE uses one shared encoder and separate decoders to reconstruct both views from the primary view.Its objective is the sum of reconstruction errors for the two views, and its empirical expectation permits minibatch SGD.
- Correlation-based methods: DCCA uses two DNNs and maximizes canonical correlation between nonlinear features from the two views.The final downstream representation is the primary-view network output after the CCA projection.
- Optimization: DCCA and DCCAE objectives couple samples through whitening constraints, so standard SGD is unavailable; large minibatches can still yield effective stochastic optimization.For DCCAE, minibatch sizes for correlation and autoencoder terms may differ and are selected separately using validation data.
- DCCAE: DCCAE combines canonical correlation between bottleneck representations with reconstruction errors from two autoencoders.The parameter λ controls the trade-off, and the reconstruction terms can be viewed as autoencoder regularization for DCCA.
- Variants: CorrAE removes DCCAE’s uncorrelatedness constraints, while DistAE uses an unconstrained discrepancy-and-reconstruction objective compatible with ordinary small-minibatch SGD.The paper explicitly studies the performance impact of relaxing the CCA constraints.
3. Related work
Related work spans neural, kernel, metric-learning, contrastive, and information-bottleneck approaches to multi-view representation learning. Kernel CCA offers nonlinear mappings but faces substantial computational and scaling challenges.
- Neural approaches: Prior neural approaches include mutual-information objectives, correlation objectives without full CCA constraints, two-step autoencoder-plus-CCA methods, and deep Boltzmann machines.DNN-based models are characterized as having tractable objectives and efficient training compared with probabilistic graphical models.
- Kernel CCA: Kernel CCA extends CCA by using feature maps induced by positive definite kernels and solving for coefficient-based projections through Gram matrices.The optimal projection can be obtained from an N × N eigenvalue problem.
- Kernel CCA: Exact KCCA requires regularization and an eigendecomposition of an N × N matrix, with naive computational cost O(N^3).The method also stores kernel matrices and does not optimize the mappings beyond kernel-parameter selection.
- Kernel approximations: Random Fourier features and Nyström approximations both produce rank-M kernel approximations with complexity O(M^3 + M^2N).Random Fourier features are data-independent and more efficient to generate, whereas Nyström often performs better.
- Connections: CCA relates to metric learning because co-occurring pairs are mapped to similar locations, while whitening constraints prevent collapse and decorrelate projection dimensions.Unlike typical metric learning, the paired views may come from different domains or modalities.
- Connections: Other related objectives include Euclidean-embedding likelihood models, contrastive losses using sampled negative pairs, and the information bottleneck, which shares CCA’s subspace for Gaussian variables.The cited multi-view embedding model differs from CCA in its global constraints and regularization.
4. Experiments
The experiments compare multi-view representation-learning methods across noisy images, speech, and word similarity, including DNN, CCA, and kernel-based approaches. Correlation-based methods generally outperform unconstrained reconstruction methods, while optimization and representation-quality trade-offs vary by task.
- Experimental setup: The evaluation covers noisy digit classification, speech recognition, and word-pair semantic similarity using DNN, linear CCA, and approximate kernel CCA methods.The compared DNN models include SplitAE, CorrAE, DCCA, DCCAE, and DistAE.
- Noisy MNIST digits: All multi-view methods improve over the baseline, while nonlinear CCA methods significantly outperform SplitAE, CorrAE, and DistAE on noisy MNIST.t-SNE visualizations qualitatively agree with the relative clustering and classification performance.
- Noisy MNIST digits: DCCAE produces the cleanest noisy-MNIST embedding, separating digit identities while suppressing rotational variation without label information.Linear CCA approximates this behavior but fails to separate classes as effectively because the input variations are too complex for linear mappings.
- Noisy MNIST digits: CCA constraints and the multi-view objective appear important because reconstruction can preserve unwanted noise, whereas DCCAE selects small λ values such as 10^-3 or 10^-2.LLE satisfies similar uncorrelatedness constraints without the second view but does not separate classes as well.
- Acoustic-articulatory speech: Nonlinear CCA-based algorithms outperform SplitAE, CorrAE, and DistAE on speech phone error rates, while linear CCA features remain highly competitive with a nonlinear recognizer.DCCAE again tends to select a relatively small λ, indicating greater importance for the canonical-correlation term.
- Word similarity: Word-similarity experiments show that reconstructing monolingual word embeddings matters: SplitAE outperforms FKCCA, NKCCA, and DCCA on the reported Spearman correlation.This contrasts with the image and speech patterns where correlation-based methods generally perform better than unconstrained reconstruction approaches.
- DCCA optimization: Stochastic optimization for DCCA can outperform L-BFGS on tuning and test data, although L-BFGS performs well on the training set.The comparison uses the same random initialization and hyperparameter grid.
- Kernel approximations: Approximate KCCA methods require relatively large approximation rank M, with diminishing returns as M increases; NKCCA performs better than FKCCA but becomes costlier to form.The experiment evaluates total canonical correlation on the XRMB JW11 tuning set.
5. Conclusion
Across several tasks, CCA-based models generally outperform reconstruction-, distance-, and correlation-based alternatives, with DCCAE the strongest overall performer. The paper also identifies objective constraints as important and suggests broader uses and architectures for DCCA.
- DCCAE is the best overall performer among the evaluated DNN-based multi-view representation-learning approaches.
- CCA-based models generally outperform SplitAE, DistAE, and CorrAE, especially when views are uncorrelated given the class label.The noisy MNIST setting matches this assumption, and deep CCA achieves its largest margins there.
- The performance gap between DCCA and CorrAE shows that uncorrelated learned dimensions matter, while stronger independence constraints remain a possible improvement.
- DCCA may also serve as a data-dependent regularizer in supervised or semi-supervised learning with labeled data and multi-view observations.
Appendix A. Analysis of stochastic optimization for DCCA
The appendix analyzes why naive minibatch optimization for DCCA is biased and derives an expected-error bound for a modified stochastic procedure under explicit assumptions. It shows how minibatch size and covariance regularization affect the spectral-norm error.
- Objective construction: The stochastic objective is formed by sampling paired examples, feeding them through f and g, estimating output covariances, and summing the top eigenvalues of a minibatch matrix.The full objective instead uses covariance estimates computed over the entire training set.
- Bias of the naive estimator: Naive minibatch DCCA does not provide an unbiased stochastic gradient because nonlinear operations prevent E[Θ(n)] from equaling Θ.The problematic operations include summing singular values, multiplying three matrices, and taking inverse square roots of autocovariance matrices.
- Assumptions: The modified procedure independently samples data for estimating Σxx, Σyy, and Σxy, doubling the computational cost of the original naive procedure.The independence assumption applies separate samples to the two autocovariances and cross-covariance.
- Assumptions: The analysis assumes bounded network outputs, which can follow from bounded activations, bounded inputs, or Lipschitz network functions.For sigmoid or hyperbolic tangent activations, the bound B can be set using the output dimensionality.
- Error analysis: Matrix Bernstein concentration bounds the spectral-norm difference between the minibatch matrix and its population counterpart under lower bounds γx and γy on estimated autocovariance eigenvalues.The theorem explicitly conditions on assumptions A1 and A2 and uses random sample selection as specified by A1.
- Error behavior: The error decreases with larger minibatches and larger γx, γy; increasing regularization constants can improve estimation error, while minibatches should exceed the representation dimension.The appendix notes that larger regularization makes the autocovariance matrices less relevant for estimating T.