PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Generalized Canonical Correlation Analysis

Adrian Benton, Huda Khayrallah, Biman Gujral, Dee Ann Reisinger, Sheng Zhang, Raman Arora

arXiv:1702.02519v2cs.LGcs.AIstat.ML

TL;DR

Existing multiview CCA methods cannot jointly provide nonlinear representations and arbitrary numbers of views. DGCCA combines nonlinear view-specific mappings with GCCA, and its representations show clear gains on phonetic transcription and hashtag recommendation while generally matching or exceeding linear many-view methods.

  • Problem

    Prior CCA-style methods were restricted either to linear transformations of many views or to nonlinear transformations of no more than two views.

  • Method

    DGCCA learns differentiable nonlinear mappings from an arbitrary number of views into a shared representation and optimizes the GCCA objective with stochastic gradient methods.

  • Results

    DGCCA representations clearly outperform prior methods on phonetic transcription and hashtag recommendation, including up to 4% higher heldout accuracy for phonetic transcription.

  • Takeaways & Limitations

    DGCCA overcomes the prior CCA-style trade-off between nonlinear transformations and many-view learning for downstream multiview representation tasks.

  • Takeaways & Limitations

    CCA-style comparisons remain bounded by linearity for GCCA and by two-view input for DCCA, while GCCA also incurs quadratic sample-size scaling and memory constraints.

Abstract

from arXiv · show

We present Deep Generalized Canonical Correlation Analysis (DGCCA) -- a method for learning nonlinear transformations of arbitrarily many views of data, such that the resulting transformations are maximally informative of each other. While methods for nonlinear two-view representation learning (Deep CCA, (Andrew et al., 2013)) and linear many-view representation learning (Generalized CCA (Horst, 1961)) exist, DGCCA is the first CCA-style multiview representation learning technique that combines the flexibility of nonlinear (deep) representation learning with the statistical power of incorporating information from many independent sources, or views. We present the DGCCA formulation as well as an efficient stochastic optimization algorithm for solving it. We learn DGCCA representations on two distinct datasets for three downstream tasks: phonetic transcription from acoustic and articulatory measurements, and recommending hashtags and friends on a dataset of Twitter users. We find that DGCCA representations soundly beat existing methods at phonetic transcription and hashtag recommendation, and in general perform no worse than standard linear many-view techniques.

1 INTRODUCTION

Multiview learning seeks representations that capture variation shared across many data views, but prior methods could not jointly support arbitrary numbers of views and nonlinear mappings. DGCCA addresses this gap and improves downstream performance, including up to 4% higher heldout accuracy for phonetic transcription.

  • Motivation: Multiview representation learning uses multiple modalities or information sources to capture variation common to all views.Examples include audio and images, social-media messages and friendships, and speech with articulatory measurements.
  • Motivation: Multiview representations are motivated by the expectation that fitting many views captures meaningful variation and improves generalization through cross-view regularization.The passages also cite theoretical results showing that some latent-variable models can be optimally learned with multiview spectral techniques.
  • Research gap: Prior methods either modeled many views only with linear mappings or handled nonlinear mappings only for two views.This limitation applies to existing GCCA, kernel CCA, and deep CCA approaches.
  • DGCCA: DGCCA learns nonlinear mappings from arbitrarily many views into a shared representation, requiring only differentiability of those mappings.Its methodological contribution is a gradient update for the GCCA objective, and an implementation was released.
  • Evaluation: DGCCA was evaluated on two datasets and three downstream tasks involving phonetic transcription and Twitter hashtag and friend recommendation.The Twitter setting uses six text and network feature views.
  • Results: Up to 4% improvement in heldout accuracy was achieved for phonetic transcription, while DGCCA gains were task-dependent overall.The paper reports clear gains on tasks previously shown to benefit from more than two views.

2 PRIOR WORK

Prior work extends CCA either toward nonlinear transformations or toward multiple views, but these extensions leave complementary limitations. DCCA remains two-view, while GCCA remains linear and can face sample-size-related memory constraints.

  • CCA: CCA finds maximally correlated linear projections of two random vectors and serves as a foundational multiview learning technique.Its formulation uses within-view covariance matrices and a cross-covariance matrix.
  • CCA: CCA is limited both to linear transformations and to leveraging only two input views.These limitations motivated extensions such as DCCA and GCCA.
  • DCCA: DCCA addresses CCA’s linearity limitation by applying stacked nonlinear representations before performing CCA on two views.The network weights are trained through backpropagation to maximize the CCA objective.
  • DCCA: DCCA remains limited to two input views.Thus it does not resolve CCA’s many-view limitation.
  • GCCA: GCCA addresses the number-of-views limitation by finding a shared representation that reconstructs the representations of J different views.The learned representation has dimensionality r, and each view is represented by a data matrix X_j.
  • GCCA: GCCA requires an eigendecomposition of an N × N matrix, causing quadratic scaling with sample size and memory constraints, and it learns only linear view transformations.Unlike CCA and DCCA, GCCA also learns a view-independent representation G.

3 DEEP GENERALIZED CANONICAL CORRELATION ANALYSIS (DGCCA)

DGCCA combines deep nonlinear mappings with GCCA to learn a shared representation from arbitrarily many views. It optimizes this objective with gradients propagated from GCCA through each view-specific network.

  • DGCCA formulation: DGCCA learns nonlinear mappings for arbitrarily many views while leveraging GCCA to produce a shared representation.Unlike Deep CCA, which is limited to two views, DGCCA combines deep networks with many-view GCCA.
  • DGCCA formulation: Each view passes through multiple nonlinear network layers, whose outputs are linearly transformed and trained to reduce GCCA reconstruction error.At test time, new data are projected by feeding each view through its learned network.
  • Optimization: DGCCA optimization uses mini-batch stochastic gradient descent, backpropagating gradients from GCCA network outputs to update the view-specific weights.The constrained GCCA objective motivates computing gradients with respect to network outputs before applying backpropagation.
  • Optimization: The GCCA objective finds a shared representation G and view-specific transformations by minimizing reconstruction error across J views.The formulation uses output matrices for each view and an auxiliary shared representation constrained by orthonormality.
  • Optimization: The gradient compares the auxiliary representation embedded in each view’s learned subspace with that view’s projected network output.A larger discrepancy produces a larger network-weight update.

4 EXPERIMENTS

The experiments show that DGCCA preserves nonlinear multiview structure and improves downstream phoneme classification over linear GCCA and nonlinear two-view DCCA, while delivering strong but task-dependent Twitter recommendation results.

  • 4.1 SYNTHETIC MULTIVIEW MIXTURE MODEL: DGCCA preserves synthetic mixture structure that linear GCCA loses, making the components linearly separable even after projection onto G's first coordinate.The DGCCA networks used three hidden layers with ten units each and random initialization.
  • 4.1 SYNTHETIC MULTIVIEW MIXTURE MODEL: DGCCA learns view-specific nonlinear mappings that make mixture components linearly separable without direct supervision about component identity.The training signal consists of reconstruction errors between network outputs and the learned shared representation G.
  • 4.2 XRMB PHONEME CLASSIFICATION: DGCCA improves phoneme-classification performance over both linear GCCA and nonlinear two-view DCCA on cross-speaker and speaker-dependent cross-validated tasks.The experiments use acoustic and articulatory measurements as views and KNN classification on projected representations.
  • 4.2 XRMB PHONEME CLASSIFICATION: DGCCA sharply reduces reconstruction error relative to GCCA, supporting better nonlinear modeling of the data.In multiview settings, reconstruction error serves as an analogue to correlation.
  • 4.2 XRMB PHONEME CLASSIFICATION: DGCCA corrects several GCCA phoneme confusions, especially V misclassified as P, R, or B and errors involving S, T, F, K, and SH.Vowels have almost equal accuracy under both methods, while GCCA performs better for UH and DH.
  • 4.3 TWITTER USER RECOMMENDATION: DGCCA outperforms linear methods by a wide margin in hashtag-recommendation recall, but cannot match WGCCA for friend recommendation when only the friend-network view is useful.Twitter experiments use six text and network views and compare DGCCA with PCA, GCCA, and WGCCA.

5 OTHER MULTIVIEW LEARNING WORK

Related multiview methods combine views through pairwise similarity or a shared consensus representation, whereas GCCA is attractive because its shared and view-specific solutions have a globally optimal SVD formulation.

  • 5 OTHER MULTIVIEW LEARNING WORK: Non-CCA multiview methods commonly learn joint representations by maximizing pairwise similarity or alternately optimizing a shared consensus representation and view-specific transformations.These approaches provide two broad strategies for combining nonlinear representation learning with multiple views.
  • 5 OTHER MULTIVIEW LEARNING WORK: GCCA is attractive because singular value decomposition of a single matrix yields globally optimal view-specific projections and shared representation.Consensus-representation approaches typically rely on alternating optimization and strong initialization to avoid bad local optima.

6 CONCLUSION

DGCCA extends CCA-style multiview learning to nonlinear transformations of arbitrarily many views. The paper reports strong downstream performance while identifying a representation-specific limitation in WGCCA.

  • 6 CONCLUSION: DGCCA learns nonlinear multiview representations from an arbitrary number of views.This overcomes prior CCA-style limitations to at most two views or strictly linear transformations.
  • 6 CONCLUSION: DGCCA clearly outperforms prior work when labels are incorporated as a third view.
  • 6 CONCLUSION: DGCCA learns user representations useful for downstream tasks such as Twitter hashtag recommendation.
  • 6 CONCLUSION: WGCCA performance suffers because whitening friend-network data ignores its rapidly decaying spectrum and informative long-tail components.

APPENDIX A DERIVING THE GCCA OBJECTIVE GRADIENT

The appendix derives gradients for optimizing DGCCA networks through the GCCA objective. The gradient compares an auxiliary shared representation with each view’s projection into its network-output subspace.

  • APPENDIX A DERIVING THE GCCA OBJECTIVE GRADIENT: DGCCA backpropagates the GCCA-objective gradient through each view’s neural network to update its weights.
  • APPENDIX A DERIVING THE GCCA OBJECTIVE GRADIENT: GCCA represents each neural-network output Yj = fj(Xj) and learns an auxiliary representation G with dimensionality r.
  • APPENDIX A DERIVING THE GCCA OBJECTIVE GRADIENT: The GCCA solution is obtained from an eigenvalue problem involving view-specific projection matrices and their sum.
  • APPENDIX A DERIVING THE GCCA OBJECTIVE GRADIENT: Minimizing the GCCA objective with respect to network weights maximizes the sum of the top r eigenvalues.
  • APPENDIX A DERIVING THE GCCA OBJECTIVE GRADIENT: The final gradient is the difference between G embedded in the view-specific subspace and the projected network output.

APPENDIX B DGCCA OPTIMIZATION PSEUDOCODE

DGCCA training alternates neural-network forward passes, GCCA computation, and gradient-based weight updates across all views. The appendix presents this procedure as minibatch stochastic optimization.

  • APPENDIX B DGCCA OPTIMIZATION PSEUDOCODE: In practice, DGCCA uses stochastic optimization with minibatches.
  • APPENDIX B DGCCA OPTIMIZATION PSEUDOCODE: The algorithm takes multiview data, an iteration count, and a learning rate, then initializes one weight set per view.
  • APPENDIX B DGCCA OPTIMIZATION PSEUDOCODE: Each iteration forward-propagates every view, mean-centers the outputs, and computes GCCA to obtain U1, ..., UJ and G.
  • APPENDIX B DGCCA OPTIMIZATION PSEUDOCODE: For each view, backpropagation produces a weight gradient and the weights are updated using the learning rate.

APPENDIX C RECONSTRUCTION ERROR AND DOWNSTREAM PERFORMANCE

Reconstruction error is useful for filtering very poor DGCCA models but does not reliably determine downstream hashtag-prediction performance. Figure 6 examines this relationship across hyperparameter settings.

  • APPENDIX C RECONSTRUCTION ERROR AND DOWNSTREAM PERFORMANCE: Figure 6 plots reconstruction error against Recall at 1000 for hashtag prediction across hyperparameter settings.
  • APPENDIX C RECONSTRUCTION ERROR AND DOWNSTREAM PERFORMANCE: Reconstruction error cannot be relied on alone for selecting models for downstream tasks.Its usefulness is limited because intrinsic measures depend on embedding and output-layer widths and may not predict downstream performance.
  • APPENDIX C RECONSTRUCTION ERROR AND DOWNSTREAM PERFORMANCE: Models with tuning reconstruction error greater than 10^3 can safely be ignored, while lower-error models show variable performance.
  • APPENDIX C RECONSTRUCTION ERROR AND DOWNSTREAM PERFORMANCE: High reconstruction error suggests disagreement among views and a likely noisy shared embedding, whereas lower error suggests a stable solution.
Loading 1702.02519v2…