Source-linked AI summary
Tensor Canonical Correlation Analysis for Multi-view Dimension Reduction
Yong Luo, Dacheng Tao, Yonggang Wen, Kotagiri Ramamohanarao, Chao Xu
TL;DR
CCA is effective for two-view dimension reduction but cannot directly handle the many-view data common in applications, and pairwise extensions omit all-view high-order statistics. The paper introduces TCCA, which analyzes a covariance tensor and converts multi-view correlation maximization into rank-1 tensor approximation. Across varied application tasks, the authors report that TCCA outperforms other methods, especially at higher common-subspace dimensions, while its main disadvantage is computational cost.
Problem
CCA is limited to two-view data, while pairwise multi-view extensions ignore high-order correlations that require simultaneous examination of all views.
Method
TCCA maximizes the correlation of multiple views by analyzing their high-order covariance tensor, whose optimization is equivalent to best rank-1 approximation solvable with ALS.
Results
Across varied application tasks, TCCA outperforms other methods, especially when the common-subspace dimension is high.
Takeaways & Limitations
Exploring high-order statistics can produce a more effective common subspace than pairwise approaches, particularly at higher dimensions.
Takeaways & Limitations
TCCA has higher computational cost than CCA and traditional multi-view extensions, mainly because of tensor decomposition.
Abstract
from arXiv · showhide
Canonical correlation analysis (CCA) has proven an effective tool for two-view dimension reduction due to its profound theoretical foundation and success in practical applications. In respect of multi-view learning, however, it is limited by its capability of only handling data represented by two-view features, while in many real-world applications, the number of views is frequently many more. Although the ad hoc way of simultaneously exploring all possible pairs of features can numerically deal with multi-view data, it ignores the high order statistics (correlation information) which can only be discovered by simultaneously exploring all features. Therefore, in this work, we develop tensor CCA (TCCA) which straightforwardly yet naturally generalizes CCA to handle the data of an arbitrary number of views by analyzing the covariance tensor of the different views. TCCA aims to directly maximize the canonical correlation of multiple (more than two) views. Crucially, we prove that the multi-view canonical correlation maximization problem is equivalent to finding the best rank-1 approximation of the data covariance tensor, which can be solved efficiently using the well-known alternating least squares (ALS) algorithm. As a consequence, the high order correlation information contained in the different views is explored and thus a more reliable common subspace shared by all features can be obtained. In addition, a non-linear extension of TCCA is presented. Experiments on various challenge tasks, including large scale biometric structure prediction, internet advertisement classification and web image annotation, demonstrate the effectiveness of the proposed method.
1 Introduction
Multi-view dimension reduction targets a compact common subspace for heterogeneous, high-dimensional features, but traditional CCA extensions handle only pairwise correlations. TCCA addresses this gap by analyzing high-order correlations across all views.
- Real-world mining tasks often combine multiple feature views whose dimensions can reach several hundred or thousand.
- Multi-view dimension reduction seeks a low-dimensional common subspace that reduces dimensionality and improves subsequent learning efficiency.
- Standard CCA handles two views, while common multi-view extensions maximize pairwise correlations and ignore high-order statistics across all features.
- TCCA generalizes CCA to an arbitrary number of views by maximizing all-view canonical correlation through a high-order covariance tensor.
- The multi-view correlation objective is equivalent to best rank-1 covariance-tensor approximation, enabling optimization with alternating least squares.
2 Related Work
Related work spans multi-view dimension reduction, CCA applications, tensor-oriented CCA extensions, and multi-view generalizations such as CCA-MAXVAR and CCA-LS. These methods typically exploit dependencies among views but retain a pairwise-correlation limitation.
- Multi-view Dimension Reduction: Multi-view learning methods include weighted view combination, multi-view dimension reduction, and view agreement exploration.
- Multi-view Dimension Reduction: Multi-view dimension reduction removes irrelevant or redundant information by leveraging dependencies, coherence, and complementarity across multiple feature representations.
- Canonical Correlation Analysis and Its Extensions: CCA has been applied to pattern recognition, classification, and clustering, including kernel CCA combined with SVM and CCA-based multi-view clustering.
- Canonical Correlation Analysis and Its Extensions: Some prior tensor CCA methods analyze relationships between two high-order tensor datasets, unlike this paper’s focus on high-order statistics among multiple vector views.
- Canonical Correlation Analysis and Its Extensions: CCA-MAXVAR combines canonical variables across views, while CCA-LS reformulates the objective as coupled least-squares problems for more efficient adaptive learning.
- Canonical Correlation Analysis and Its Extensions: CCA-MAXVAR and CCA-LS still exploit only pairwise correlations and ignore high-order correlations involving all views.
3 Canonical Correlation Analysis (CCA) and Its Multi-view Generalization
Standard CCA finds projections of two variable sets whose canonical variables are maximally correlated, using covariance and variance matrices. Its multi-view extensions combine canonical variables or solve coupled regression problems while enforcing normalization and orthogonality constraints.
- Canonical Correlation Analysis: CCA finds projection vectors h1 and h2 that maximize the correlation between the corresponding canonical variables.
- Canonical Correlation Analysis: The canonical variables are projected data vectors formed as z1n = x1n^T h1 and z2n = x2n^T h2.
- Canonical Correlation Analysis: CCA uses C11 and C22 as variance matrices and C12 as the cross-covariance matrix of the two stacked data matrices.
- Multi-view Generalization: CCA-MAXVAR generalizes CCA to m views by combining canonical variables from all views to approximate a one-dimensional common representation.
- Multi-view Generalization: The multi-view dimension-reduction pipeline extracts view-specific features, learns transformations, and maps each view into a shared low-dimensional representation.
- Multi-view Generalization: CCA-MAXVAR uses SVD for optimization, whereas CCA-LS reformulates the problem as coupled least-squares regression and obtains successive solutions with orthogonality constraints.
4 Tensor Canonical Correlation Analysis (TCCA)
TCCA generalizes CCA to arbitrary numbers of views by modeling their joint high-order covariance tensor. It maximizes multi-view canonical correlation through rank-1 tensor approximation and supports nonlinear projections through KTCCA.
- TCCA framework: TCCA extracts multiple feature views and uses their covariance tensor to map high-dimensional data into a shared low-dimensional representation.The resulting view-specific projections are concatenated for subsequent learning.
- Problem formulation: The method maximizes correlation among the canonical variables of all views, rather than considering only pairwise correlations.The canonical correlation is expressed through the element-wise product of the view-specific canonical variables.
- Optimization: ALS solves the rank-1 tensor approximation, while HOPM and the tensor power method are alternative optimizers.The authors report that ALS performs best in their experiments.
- Non-linear extension: KTCCA extends TCCA to nonlinear projections by mapping each view into a potentially infinite-dimensional feature space.The projections are represented through combinations of training instances using the Representer Theorem.
- Tensor formulation: The multi-view correlation problem is equivalent to finding the best rank-1 approximation of a transformed covariance tensor.The equivalence provides the tensor formulation underlying the optimization.
- Complexity: TCCA has space complexity O(d1d2 . . . dm) and time complexity O(trd1d2 . . . dm), independent of the number of instances.This scaling is attributed to offline covariance-tensor computation and rank-r ALS decomposition.
5 Experiments
Experiments evaluate TCCA and related multi-view dimension-reduction methods on biometric structure prediction, advertisement classification, and web image annotation, including nonlinear and computational analyses. Across tasks, results depend on subspace dimension, labeled and unlabeled sample sizes, and the amount of high-order correlation information available.
- Experimental setup: Experiments compare TCCA with single-view, concatenation, CCA, CCA-LS, DSE, and SSMVD baselines across three multi-view learning tasks.The tasks are biometric structure prediction, internet advertisement classification, and web image annotation.
- Biometric Structure Prediction: SecStr uses 100 labeled instances and evaluates CCA-based methods with varying amounts of unlabeled data, including 84K and about 1.3M instances.The task predicts protein secondary structure from amino-acid sequence windows, with features divided into three views of dimension 105 each.
- Biometric Structure Prediction: Properly tuned common subspaces outperform the best-single-view and concatenation baselines on SecStr, while CCA-based accuracy increases as more unlabeled data are used.CCA (AVG) improves over CCA (BST), and CCA-LS is superior to CCA (BST) although their best-dimensional performance is comparable.
- Web Image Annotation: On NUS-WIDE mammal annotation, accuracy generally improves with more labeled instances, and TCCA is significantly better than other methods under most dimensionalities.TCCA remains satisfactory at large subspace dimensions, whereas CCA (AVG) and CCA-LS first increase and then decrease.
- Nonlinear extension and computational complexity: KTCCA achieves the best performance under most dimensionalities, while TCCA generally costs more than other CCA-based methods but is more efficient than DSE or SSMVD for large sample sizes and moderate feature dimensions.The nonlinear comparison uses kernel methods, and the computational comparison measures time and memory costs.
6 Conclusion
TCCA addresses the inability of standard CCA and typical extensions to capture high-order correlations across multiple views by analyzing their covariance tensor. Experiments support its effectiveness, while its main drawback is higher computational cost.
- Standard CCA handles two-view data, while typical multi-view extensions ignore high-order correlation information among all feature views.
- TCCA discovers high-order statistics by analyzing the covariance tensor of all views.
- CCA-based common-subspace learning often outperforms simple feature concatenation, especially when feature dimension is high.
- Examining more statistics may require more unlabeled data, and exploring high-order statistics improves TCCA relative to other methods, especially at high subspace dimensions.
- TCCA's main disadvantage is high computational cost, primarily arising from tensor decomposition.The paper suggests efficient tensor decomposition or GPU-based parallel computing as future acceleration directions.
A Proof of Thoerem 1
The proof applies element-wise product and tensor–matrix product definitions to establish the stated result. It concludes after evaluating the relevant tensor entries and products.
- The proof begins from the definition of the element-wise product.
- It identifies vector entries, including z_p^(n), x_p^n, and h, while reusing the same notation across the proof.
- The proof invokes the definition of the p-mode product between a tensor and a matrix.
- A tensor product expression is evaluated at an index whose p-mode entry is fixed to 1.
- The argument ends by stating that the proof is complete.
B Proof of Theorem 3
The proof compares tensor constructions F and G after expanding their entries through outer products and tensor–matrix products. Equality is then concluded by comparing the resulting expressions.
- The proof defines F by multiplying covariance tensor C with feature maps across modes and defines G using a sum of outer products.
- The outer-product expansion gives G's indexed entries as products k_1^n(j_1)k_2^n(j_2)...k_m^n(j_m).
- The proof defines k_p^n(j_p) as the j_p-th element of vector k_p^n.
- It defines φ_p^n(i_p) as the i_p-th element of φ(x_p^n) before applying tensor–matrix products.
- The tensor–matrix product is evaluated at indexed entries, producing an expression for F that matches the outer-product expansion.
- The theorem is completed by comparing the corresponding expressions in equations (B.1) and (B.3).