PaperlayerView exact revision on arXiv

Source-linked AI summary

A Tutorial on Canonical Correlation Methods

Viivi Uurtio, João M. Monteiro, Jaz Kandola, John Shawe-Taylor, Delmiro Fernandez-Reyes, Juho Rousu

arXiv:1711.02391v1cs.LGstat.ML

TL;DR

Analyzing relations between paired variable views may improve understanding of an underlying system, while CCA variants address limits of sample size, dimensionality, and linearity. This tutorial explains and illustrates CCA methods, then synthesizes their applicability across data settings and emphasizes evaluating extracted relations.

  • Problem

    CCA was originally developed for linear relations in overdetermined settings, while modern analyses may have insufficient samples, nonlinear relations, or too many variables for easy interpretation.

    Source: §3.4 · p. 22

  • Method

    The tutorial explains original, regularised, kernel, and sparse CCA, briefly reviews Bayesian and deep CCA, and presents numerical examples and evaluation techniques.

    Source: §1 · p. 3, §4 · p. 27

  • Results

    The tutorial brings CCA variants into a unified framework that relates their applicability to sample size, data dimensionality, and the type of relations of interest.

    Source: §4 · p. 29

  • Takeaways & Limitations

    Applying CCA methods entails choosing among variants for the data setting and assessing the statistical significance and generalisability of extracted relations.

    Source: §4 · pp. 27, 29

    Linear CCA requires more observations than variables in both views, and the limit beyond which regularisation cannot recover relations in underdetermined systems remains unresolved.

    Source: §4 · p. 28

Abstract

from arXiv · show

Canonical correlation analysis is a family of multivariate statistical methods for the analysis of paired sets of variables. Since its proposition, canonical correlation analysis has for instance been extended to extract relations between two sets of variables when the sample size is insufficient in relation to the data dimensionality, when the relations have been considered to be non-linear, and when the dimensionality is too large for human interpretation. This tutorial explains the theory of canonical correlation analysis including its regularised, kernel, and sparse variants. Additionally, the deep and Bayesian CCA extensions are briefly reviewed. Together with the numerical examples, this overview provides a coherent compendium on the applicability of the variants of canonical correlation analysis. By bringing together techniques for solving the optimisation problems, evaluating the statistical significance and generalisability of the canonical correlation model, and interpreting the relations, we hope that this article can serve as a hands-on tool for applying canonical correlation methods in data analysis.

1. INTRODUCTION

Canonical correlation methods analyze relations between two views of a process, with potential to improve understanding of the underlying system. The tutorial surveys CCA variants and explains their applicability across data settings.

1. INTRODUCTION

  • Analyzing relations between two views may improve understanding of the underlying system, with medicine offered as one example.The example connects disease symptoms with risk factors that can affect disease incidence.Source: §1 · pp. 1–2
  • CCA has been applied across fields including neuroscience, machine learning, bioinformatics, and feature selection, extraction, fusion, and dimensionality reduction.The range of application domains illustrates CCA’s use in extracting relations between variables.Source: §1 · p. 2
  • CCA variants address underdetermined data, nonlinear relations, and high-dimensional settings where sparse solutions can aid interpretation.The tutorial also briefly reviews Bayesian and deep CCA.Source: §1 · pp. 2–3
  • Semi-paired and multi-view CCA address missing observations in one view and relations among more than two views, respectively, but are outside this tutorial’s scope.The tutorial focuses instead on regularised, kernel, and sparse CCA, with deep and Bayesian variants briefly reviewed.Source: §1 · pp. 2–3

2. CANONICAL CORRELATION ANALYSIS

This section explains how canonical correlation analysis identifies linear relationships between two views through transformations that minimize the angle between their unit-norm images, and how to solve, interpret, and evaluate the resulting model. Evaluation includes testing the statistical significance of canonical correlations and whether relations recur in new observations from the sampling distribution.

2.1. The Basic Principles of CCA

  • CCA transforms two variable sets into unit-norm images and chooses positions that minimize their angle, thereby maximizing the canonical correlation.The variables are assumed to be standardized to zero mean and unit variance; the tutorial also assumes joint multivariate normal sampling.Source: §2.1 · p. 3

2.2. Finding the positions and the images in CCA

  • Standard or generalized eigenvalue problems and SVD can find CCA positions; SVD singular values correspond to canonical correlations.The tutorial presents standard eigenvalue methods, generalized eigenvalue methods [Bach and Jordan 2002; Hardoon et al. 2004], and SVD.Source: §2.2 · pp. 4, 8, 9
  • SVD has time complexity O(mn^2), compared with O(n^3) for eigenvalue methods, making it more tractable for very large datasets.The comparison concerns an SVD input matrix of size m × n and an eigenvalue input matrix of size n × n.Source: §2.2 · p. 9

2.3. Evaluating the Canonical Correlation Model

  • Statistical significance tests assess whether extracted patterns occur non-randomly, while generalisability tests assess whether training-data relations represent broader patterns in the sampling distribution.Bartlett’s sequential procedure can determine the number of statistically significant canonical correlations.Source: §2.3 · pp. 9–10
  • Canonical weight entries identify variable relations: the example’s first pair links a3 and b1 positively, its second links a4 and b3 negatively, and its third links a1 and b2 positively.The first relation corresponds to a canonical correlation of 0.99.Source: §2.3 · p. 9, Figure 1 · p. 10
  • Structure correlations relate original variables to canonical images, and biplot angles indicate positive, negative, or zero correlations as acute, obtuse, or right angles.Changing the image pairs used to compute correlations can enhance relation extraction; the tutorial’s biplots show the example’s relations from different image views.Source: §2.3 · pp. 9–10, Figure 2 · p. 10
  • In the simulated example, Bartlett–Lawley tests rejected zero, one, and two canonical relations (L0 = 296.82, L1 = 154.56, L2 = 70.95), supporting all three patterns.Each test used a 0.01 significance level.Source: §2.3 · p. 11
  • On 40 held-out observations from the same distributions, the test canonical correlation was 0.98, indicating the extracted relations were generalisable.Source: §2.3 · p. 11

3. EXTENSIONS OF CANONICAL CORRELATION ANALYSIS

This section surveys extensions of CCA for settings where linear CCA is difficult or insufficient, including regularised, probabilistic and Bayesian, kernel, deep, and sparse methods. It describes approaches to nonlinear relation analysis and variable selection, and illustrates relation extraction using simulated data.

3.1. Regularisation Techniques in Underdetermined Systems

  • After adding c1I to Caa to make it invertible, the square roots of the first three eigenvalues were ρ1 = 0.98, ρ2 = 0.97, and ρ3 = 0.96.The optimal c1 = 0.09 was obtained through 50 times repeated 5-fold cross-validation.Source: §3.1 · p. 14
  • When either data view has more variables than observations, adding non-negative constants to variance-matrix diagonals improves invertibility and makes CCA solvable.The resulting regularised problem can be solved with standard CCA techniques; leave-one-out cross-validation can select the regularisation parameters, while repeated k-fold validation can reduce variance in average test-fold scores.Source: §3.1 · pp. 12, 13, 15

3.2. Bayesian Approaches for Robustness

  • Replacing normal noise with a multivariate Student’s t distribution can improve robustness to outliers, but high-dimensional inference remains difficult because variance matrices must be inverted.Dimensionality reduction is therefore recommended as preprocessing for probabilistic or Bayesian CCA on high-dimensional data.Source: §3.2 · p. 16
  • Probabilistic and Bayesian CCA use latent variables to interpret paired views, and priors can incorporate underlying data structure, including sparse structure.Bayesian variants use inverse-Wishart covariance priors and automatic relevance determination for transformations; noise can also follow exponential-family distributions rather than only a normal distribution [Klami et al. 2012].Source: §3.2 · p. 16

3.3. Uncovering Linear and Non-Linear Relations

  • Kernel CCA uses kernels to extract nonlinear correlations and replaces the original data matrices with Gram matrices; when those matrices are invertible, resulting canonical correlations all equal one, so regularisation is needed.Source: §3.3 · pp. 17–18
  • 0.87 was the first canonical correlation at the cross-validated regularisation parameters in the kernel CCA example, which extracted exponential, third-order polynomial, and linear relations in successive pairs.Repeated 5-fold cross-validation over 20 repetitions selected c1 = 1.50 and c2 = 0.60.Source: §3.3 · p. 19, Table I · p. 20
  • 0.96 was the first canonical correlation in the large-sample kernel CCA example, which extracted exponential, polynomial, and linear relations in successive pairs.The exponential relation appeared in the first pair, the third-order polynomial in the second, and the linear relation in the third.Source: §3.3 · p. 21, Table II · p. 21
  • Deep CCA learns neural-network transformations of both views to maximize their correlation, using gradient-based optimization of network parameters.The networks apply nonlinear functions across layers, and the objective can use the summed top-k singular values of the transformed views' correlation matrix.Source: §3.3 · pp. 21–22
  • Kernel and deep CCA offer alternatives when relations may be nonlinear; kernel choice depends on the relations sought, while deep CCA maximizes correlation but makes relation types difficult to identify.For kernel CCA, prior knowledge can aid interpretation, and transformed pairs can be tested for correlation with functions of variables.Source: §3.3 · p. 22

3.4. Improving the Interpretability by Enforcing Sparsity

  • Sparse CCA addresses settings with many variables by constraining position-vector entries to zero, so relations can be inferred from a smaller subset of variables.Proposed approaches include soft-thresholding, elastic-net regularization, penalized matrix decomposition, and convex least squares; the choice depends on the research question and prior knowledge.Source: §3.4 · p. 22
  • Soft-thresholding CCA finds sparse linear relations iteratively, elastic-net CCA adds grouped variable selection, and PMD CCA seeks sparse low-rank approximations of the cross-covariance matrix.In soft-thresholding CCA, nonzero assignments are independent across entries; elastic net combines LASSO and ridge penalties, while PMD extracts successive relations by deflation.Source: §3.4 · p. 23
  • The sparse convex least-squares formulation of Hardoon and Shawe-Taylor [2011] relates sparse data-space variables to sparse observations represented by similarities in kernel space.Its procedure iteratively minimizes the primal–dual Lagrangian gap and uses deflation to extract successive patterns.Source: §3.4 · pp. 24–25
  • 0.91 was the first canonical correlation in the sparse PMD example, which extracted the negative a4–b3 relation first and the positive a1–b2 and a3–b1 relations next.The reported extraction order is corroborated by the positions shown in Figure 5.Source: §3.4 · pp. 24–25, Figure 5 · p. 25

repeat

This example applies sparse CCA to relate variables in one data view to an observation in the other view, selecting the observation that minimises the objective. At k=29, the selected observation is most related to five variables in the data-space view.

repeat

  • The example uses 50 observations, 100 variables in Xa, and 150 in Xb, generated from standard normal distributions and standardised to zero mean and unit variance.Source: repeat · p. 26
  • For view b, the example uses a centred Gaussian kernel with width σ=17.25, chosen by the median trick because the underlying distribution is normal.Source: repeat · p. 26
  • The algorithm tests each observation as a basis, computes a sparse set of remaining observations β, and selects the k with the minimum objective value f.Source: repeat · p. 26
  • This approach is suited to relating variables in one view to the observations corresponding to the combined variables of the other view, rather than explicit variable-to-variable relations.Source: repeat · p. 26
  • f=0.03 at k=29, yielding canonical correlation 0.88; Figure 6 shows the position entries, and the selected observation is most related to a15, a16, a18, a20, and a24.Source: repeat · p. 26, Figure 6 · p. 27

4. DISCUSSION

The discussion unifies CCA variants around the data settings where they apply and reviews how to solve, evaluate, and interpret CCA models. It also identifies important assumptions and unresolved challenges across linear, nonlinear, and sparse approaches.

4. DISCUSSION

  • Linear CCA’s paired position vectors encode relations; the tutorial presents standard and generalised eigenvalue and SVD solutions.These three commonly applied, relatively straightforward approaches also help explain how CCA extensions were developed.Source: §4 · p. 27
  • Worked examples use optimal settings with one-to-one relations, whereas real data may contain many-to-many relations that the examples’ approach can help uncover.The examples were designed for teaching, and their datasets are often much smaller than real datasets.Source: §4 · pp. 27–28
  • Linear CCA requires more observations than variables in both views; regularisation’s limit under severe underdetermination is unresolved, while Bayesian CCA is more robust to outliers.The sample-size condition guarantees non-singular variance matrices.Source: §4 · p. 28
  • Kernel mappings enable nonlinear relations, but kernel choice complicates their identification without prior knowledge; deep CCA also obscures variable relations, and biplots mainly suit linear patterns.The relation types that kernel CCA extracts depend on the selected kernel function.Source: §4 · pp. 28–29
  • Sparse CCA enforces sparsity on position-vector entries but does not account for dependencies among variables, a limitation addressed by structured sparsity.The tutorial’s selected sparse variants were chosen based on their use in the literature.Source: §4 · p. 28
  • The tutorial compares CCA variants by data setting and covers optimisation, significance, generalisability, and relation interpretation.It also emphasises assessing the significance and generalisability of extracted relations.Source: §4 · pp. 27–29

Related papers

Loading 1711.02391v1…