PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel Multivariate Analysis Framework for Supervised Subspace Learning: A Tutorial on Linear and Kernel Multivariate Methods

Jerónimo Arenas-García, Kaare Brandt Petersen, Gustavo Camps-Valls, Lars Kai Hansen

arXiv:1310.5089v1stat.MLcs.LG

TL;DR

Feature extraction is increasingly important for high-resolution sensing and multimodal data, while linear MVA is limited when input-output relations are nonlinear. The paper reviews linear and kernel MVA, their connections and practical extensions, and reports that kernel approaches outperform linear schemes in the UCI benchmark passage.

  • Problem

    Linear MVA is constrained to linear input-output relations, while kernel MVA faces practical challenges with very large or very small labeled datasets.

    Source: §I · p. 1, C. Kernel Multivariate Analysis · p. 5

  • Method

    The paper unifies reviews of PCA, PLS, CCA, and OPLS with kernel extensions, connections to other methods, and sparse and semisupervised approaches.

    Source: §I · pp. 1–2

  • Results

    The UCI benchmark passage reports that kernel approaches considerably outperform linear schemes.

    Source: A. UCI repository benchmark · p. 8

  • Takeaways & Limitations

    The paper assesses these methods on public datasets and in music-genre prediction and hyperspectral satellite-image applications.

    Source: §I · p. 2

    Direct kernel MVA scales quadratically in memory and cubically in computation with the number of training samples, and can overfit or yield uninformative features with few labeled samples.

    Source: C. Kernel Multivariate Analysis · p. 5

Abstract

from arXiv · show

Feature extraction and dimensionality reduction are important tasks in many fields of science dealing with signal processing and analysis. The relevance of these techniques is increasing as current sensory devices are developed with ever higher resolution, and problems involving multimodal data sources become more common. A plethora of feature extraction methods are available in the literature collectively grouped under the field of Multivariate Analysis (MVA). This paper provides a uniform treatment of several methods: Principal Component Analysis (PCA), Partial Least Squares (PLS), Canonical Correlation Analysis (CCA) and Orthonormalized PLS (OPLS), as well as their non-linear extensions derived by means of the theory of reproducing kernel Hilbert spaces. We also review their connections to other methods for classification and statistical dependence estimation, and introduce some recent developments to deal with the extreme cases of large-scale and low-sized problems. To illustrate the wide applicability of these methods in both classification and regression problems, we analyze their performance in a benchmark of publicly available data sets, and pay special attention to specific real applications involving audio processing for music genre prediction and hyperspectral satellite images for Earth and climate monitoring.

I. INTRODUCTION

The paper frames multivariate analysis (MVA) as a way to extract task-relevant features, then reviews linear and kernel methods and developments for practical constraints. It connects these methods to classification and dependence estimation and evaluates them on public datasets and real applications.

I. INTRODUCTION

  • Higher-resolution sensing and increasingly common combinations of diverse data sources make feature extraction and dimensionality reduction important, especially for high-dimensional signals and heterogeneous features.Source: §I · p. 1
  • PCA maximizes input-projection variance without using targets, while PLS and CCA maximize input-target covariance and correlation, respectively.The paper also considers OPLS, which is suited to supervised problems and optimal for least-squares multiregression.Source: §I · p. 1
  • Kernel MVA maps inputs to a high-dimensional space and uses kernel inner products to obtain nonlinear methods without explicitly computing the mapping.The paper reviews this kernel-based approach rather than methods that make relations among latent variables nonlinear.Source: §I · p. 1
  • The paper reviews sparse and semisupervised kernel approaches for large datasets and settings with few labeled samples, alongside connections to discriminant analysis and kernel dependence estimates.Source: §I · pp. 1–2
  • The methods are assessed for accuracy and robustness to the number of extracted features on public datasets and in music-genre and hyperspectral-image applications.Source: §I · p. 2

II. MULTIVARIATE ANALYSIS IN REPRODUCING KERNEL HILBERT SPACES

  • The paper places linear and kernel MVA methods in a shared framework and points out connections to classification and dependence-estimation methods.Source: §II · p. 2

A. Problem statement and notation

MVA extracts input features aligned with targets, with methods distinguished by the objective they optimize. The section describes the principal linear methods, their solution framework, and the limits of linear transformations.

A. Problem statement and notation

  • MVA projects inputs into a lower-dimensional feature space, with methods distinguished by the objectives they maximize and commonly formulated as eigenvalue problems.The problem setup defines centered input and target matrices and describes least-squares prediction from inputs to targets.Source: A. Problem statement and notation · pp. 2–3

B. Linear Multivariate Analysis

  • PCA selects orthonormal input projections maximizing variance, whereas PLS maximizes covariance between projected inputs and outputs.PLS variants differ in their deflation schemes; PLS2 uses the input latent variable to deflate both input and target data and avoids matrix inversion.Source: B. Linear Multivariate Analysis · p. 3
  • CCA maximizes correlation between projected inputs and outputs, making it less prone than PLS to overemphasizing high-variance directions with weak correlation.Source: B. Linear Multivariate Analysis · p. 3
  • OPLS is optimal for least-squares regression on extracted features and is equivalent to LDA for classification with an appropriate target-labeling scheme.Unlike CCA, OPLS can suit supervised prediction of target values; CCA may be preferable when both data views should be predicted.Source: B. Linear Multivariate Analysis · p. 3
  • CCA and OPLS require covariance-matrix inversions, so rank-deficient matrices call for prior extraction of nonzero-variance dimensions using PCA.Source: B. Linear Multivariate Analysis · p. 3
  • In a three-class toy problem, PCA, PLS, and CCA optimize variance, covariance, and correlation, respectively, while OPLS minimizes target-approximation MSE.Because these methods transform data linearly, they cannot capture nonlinear relations between input variables.Source: B. Linear Multivariate Analysis · p. 4

C. Kernel Multivariate Analysis

Kernel MVA uses inner products in a feature space to extract nonlinear projections with linear-algebra methods. The section introduces kernel versions of MVA methods, illustrates their behavior, and describes practical limits of direct solutions.

C. Kernel Multivariate Analysis

  • Kernel MVA seeks nonlinear projections while working with linear algebra by representing feature-space projections as combinations of training samples and using their kernel matrix.The kernel trick replaces explicit operations in the feature space with inner products computed from input data.Source: C. Kernel Multivariate Analysis · p. 4
  • Kernel versions include kPCA, kCCA, kPLS2, and kOPLS, with kPLS2 deflating the kernel matrix and target data together.The paper retains the original target representation because its focus is extracting input features that represent labels.Source: C. Kernel Multivariate Analysis · p. 4
  • In the toy example, kernel methods use an RBF mapping, and kOPLS selects feature-space directions that best approximate targets by mean-square error.The example states that maximizing variance or covariance need not be best for supervised learning.Source: C. Kernel Multivariate Analysis · pp. 4–5
  • Direct kernel MVA can overfit, requires O(l^2) memory and O(l^3) computation, and needs l kernel evaluations per new pattern; very small labeled samples can also yield uninformative features.The paper notes that feature information content has not been characterized.Source: C. Kernel Multivariate Analysis · p. 5

D. Relations with other methods

The section connects kernel multivariate analysis (kMVA) to discriminant methods and statistical dependence measures. It describes shared formulations and relationships that may help interpret extracted features.

D. Relations with other methods

  • Fisher’s LDA, CCA, PLS, and OPLS are closely connected for classification, and these links extend to their kernel counterparts.Kernel discriminants include binary-problem methods by Mika et al., and multiclass GDA by Baudat and Anouar.Source: D. Relations with other methods · p. 5
  • HSIC estimates dependence through the RKHS cross-covariance norm, while HSCA iteratively maximizes target dependence and minimizes dependence on previously extracted features.With a universal RKHS kernel, HSIC asymptotically tends to zero when input and output data are independent; maximizing source-target dependence alone reduces to kOPLS.Source: D. Relations with other methods · p. 5
  • Under certain conditions, kGV reduces to kMI at θ = 0 and to kCCA at θ = 1.The empirical kGV dependence estimate depends on the entire spectrum of the associated RKHS correlation operator.Source: D. Relations with other methods · pp. 5–6
  • HSIC is intimately related to correntropy, and the authors suggest these dependence-measure connections could help explain extracted features’ informative content.The proposed connection to feature interpretation is framed as a possibility for future work.Source: D. Relations with other methods · p. 6

III. EXTENSIONS FOR LARGE SCALE AND SEMISUPERVISED PROBLEMS

This section introduces extensions intended to address two sample-size extremes that can hamper supervised kernel MVA: large datasets and few labeled samples.

III. EXTENSIONS FOR LARGE SCALE AND SEMISUPERVISED PROBLEMS

  • Supervised kernel MVA can be impractical when labeled samples are either abundant or scarce, motivating extensions for large-scale and semisupervised settings.The section next summarizes approaches for both cases.Source: §III · p. 6
  • Large-scale extensions address the case in which the wealth of labeled samples hampers supervised kernel MVA.The passage identifies large-scale problems as one of the extension settings.Source: §III · p. 6
  • Semisupervised extensions address situations in which few labeled samples hamper supervised kernel MVA.The passage identifies few labeled samples as the other extension setting.Source: §III · p. 6

A. Sparse Kernel Feature Extraction

Sparse kernel feature extraction reduces reliance on dense kernel solutions, addressing computational, memory, and overfitting concerns. The section reviews low-rank and reduced-set strategies and compares their practical trade-offs.

A. Sparse Kernel Feature Extraction

  • Dense kernel solutions require l × l kernel matrices, scale quadratically in memory and cubically in computation, and can overfit; sparse solutions can limit new-sample evaluation to r ≪ l kernels.Sparse solutions represent projections using a subset of training samples.Source: A. Sparse Kernel Feature Extraction · p. 6
  • Sparse kernel methods divide broadly into low-rank approximations using reduced r × r matrices and reduced-set methods using l × r matrices.Nyström approximation is described as a low-rank approach and has been used to approximate feature mappings for sparse kPLS and kCCA.Source: A. Sparse Kernel Feature Extraction · p. 6
  • Sparse kPCA uses a generative normal model with noise, while sparse kPLS uses a fraction of training samples and an ε-insensitive loss, but both still compute the full kernel matrix during training.The sparse kPLS adaptation is described as computationally burdensome.Source: A. Sparse Kernel Feature Extraction · p. 6
  • Reduced-complexity kOPLS represents projections using a subset of r training samples, retains all training data during training, and avoids computing the full kernel matrix.Its storage requirements become quadratic in r, and the sparsity constraint can significantly improve generalization; the same procedure yields rkPCA and rkCCA, but does not straightforwardly extend to kPLS2 because deflation still requires the full kernel matrix.Source: A. Sparse Kernel Feature Extraction · p. 6
  • Sparse Maximal Alignment and Sparse Maximal Covariance select kPLS projections by maximizing kernel alignment or covariance with the true labels, with β constrained to cardinality 1.The method searches all training patterns for β at each iteration, though restricting search to p randomly chosen samples can significantly reduce complexity.Source: A. Sparse Kernel Feature Extraction · pp. 6–7
  • For large datasets, the authors identify linear MVA and sparse approaches from and as affordable; sparse kMVA also reduces kernel evaluations for new data.Kernel choice and parameter tuning remain important, and cross-validation can help avoid overfitting at higher computational cost; sparse methods can regularize the solution.Source: A. Sparse Kernel Feature Extraction · p. 7

B. Semisupervised Kernel Feature Extraction

The section presents graph-regularized and cluster-kernel approaches to semisupervised kernel feature extraction. They incorporate unlabeled samples differently, with distinct practical trade-offs.

B. Semisupervised Kernel Feature Extraction

  • With few labeled samples, extracted features may poorly capture data-manifold structure, leading to poor classification or regression results.The section describes graph-based and generative-clustering-based approaches to use unlabeled information.Source: B. Semisupervised Kernel Feature… · p. 7
  • Semisupervised kCCA uses labeled and unlabeled data in its kernel matrices and regularizes them with a graph Laplacian.For n = l and null regularization, it reduces to standard kCCA; the approach involves several tunable parameters and 2n × 2n matrices, which can make application difficult in practice.Source: B. Semisupervised Kernel Feature… · p. 7
  • The cluster-kernel approach combines a labeled-information kernel with a generative kernel learned by clustering labeled and unlabeled data.The probabilistic cluster kernel averages dot products between posterior probability vectors across clustering runs and cluster counts.Source: B. Semisupervised Kernel Feature… · p. 7
  • The cluster-kernel method accounts for probabilistic similarities at small and large scales while retaining an l × l kernel matrix for kMVA.The passage says clustering need not be computationally demanding and identifies the approach as appealing to practitioners.Source: B. Semisupervised Kernel Feature… · p. 7

IV. EXPERIMENTAL RESULTS

The section illustrates supervised kernel multivariate feature extraction through a UCI classification benchmark and two real applications: satellite imagery and music-genre prediction.

IV. EXPERIMENTAL RESULTS

  • The evaluation combines publicly available UCI classification problems with satellite-image processing and music-genre prediction.The music dataset is large enough to make standard kMVA methods impractical, illustrating sparse extensions.Source: §IV · p. 7
  • The two application examples are satellite-image processing and audio processing for music-genre prediction.The paper presents them as demonstrations of the methods’ potential.Source: §IV · p. 7
  • The large music dataset is used to illustrate the benefits of sparse extensions when standard kMVA is impractical.Source: §IV · p. 7

A. UCI repository benchmark

The UCI benchmark compares linear and kernel MVA methods on classification problems, examining accuracy, feature counts, and computational properties. Kernel methods outperform linear schemes overall, while regularization is important for supervised methods prone to overfitting.

A. UCI repository benchmark

  • Table III organizes dense and sparse methods by free parameters, kernel evaluations, storage, eigensolver type, and kernel calls for new projections.The prose frames these computational and implementation properties as guidance for choosing an algorithm.Source: Table III · p. 8
  • The benchmark uses a 60% training split, or at most 500 training samples, and evaluates classification through least-squares predictions followed by winner-takes-all activation.Class labels use 1-of-c coding; overall accuracy is the reported measure.Source: A. UCI repository benchmark · p. 8
  • Linear OPLS and CCA can overfit low-variance directions when l/d is small, while regularization helps address this small-sample problem.The described approach applies the methods to PCA-extracted projections; OPLS and CCA can yield more discriminative features but may need regularization.Source: A. UCI repository benchmark · p. 8
  • Kernel approaches considerably outperform linear schemes across the benchmark; regularized kOPLS and kCCA reach performance comparable to kPCA with only c −1 features.kPLS2, using the same feature count, has considerably smaller overall accuracies in all problems, though it can extract more projections and is more robust to overfitting.Source: A. UCI repository benchmark · p. 8

B. Remote sensing image analysis

The remote-sensing examples assess MVA methods on hyperspectral image classification and atmospheric temperature retrieval. Kernel methods perform better on the reported classification and RMSE comparisons, and the temperature estimates require substantially less computational time than physical-inversion methods.

B. Remote sensing image analysis

  • The hyperspectral classification test uses 200 spectral bands from a 145 × 145-pixel image with 16 unbalanced classes, splitting labeled pixels into 20% training and 80% testing.Source: B. Remote sensing image analysis · p. 9
  • For hyperspectral classification, linear OPLS performs better than the other linear methods across feature counts, and CCA and OPLS reach virtually the same performance using 15 rather than 200 features for PCA and PLS2.At the maximum number of projections, all methods have the same error.Source: B. Remote sensing image analysis · p. 9
  • In kernel feature space, kOPLS is slightly more accurate than kCCA, and both significantly outperform kPLS2 and kPCA; all methods have similar accuracy at the feature-count limit.For ten-feature maps, higher accuracies correspond to smoother maps and smaller errors in large spatially homogeneous vegetation covers.Source: B. Remote sensing image analysis · p. 9
  • The temperature-retrieval task maps 8,461 IASI spectral channels to 90 atmospheric pressure levels, using 2,000 training samples for the transformation and least-squares regression.The models are trained on synthetic samples and applied to a full orbit of 91,800 pixels.Source: B. Remote sensing image analysis · pp. 9–10
  • Kernel methods outperform linear approaches in atmospheric RMSE, especially in lower atmospheric layers, and yield surface-temperature maps closer to ECMWF estimates.The figure shows pressure-level RMSE, the ECMWF surface-temperature map, and estimated maps for several methods.Source: B. Remote sensing image analysis · p. 10, Figure 3 · p. 10
  • Kernel-based temperature estimates are obtained with a drastic reduction in computational time compared with ECMWF physical-inversion methods.Source: B. Remote sensing image analysis · p. 10

C. Analysis of integrated short-time music features for genre prediction

The audio-only genre task evaluates linear and sparse kernel MVA on short-time music features. OPLS- and CCA-type methods perform best overall, while sparse kernel methods remain applicable to the large data set.

C. Analysis of integrated short-time music features for genre prediction

  • The audio-only benchmark contains 1,317 30-second snippets across 11 genres, with estimated human accuracy around 55%.The data set averages 1.83 songs per artist.Source: C. Analysis of integrated short-time… · p. 10
  • MFCCs are extracted from overlapping 20-ms frames, then 1.2-second multivariate AR models yield 135-length feature vectors.The AR parameters capture temporal correlation and are stacked for each frame.Source: C. Analysis of integrated short-time… · pp. 10–11
  • The 817-song training and 500-song test split produces 57,388 and 36,556 vectors, respectively, motivating sparse kernels alongside linear methods.The study evaluates sparse rkPCA, rkOPLS, and rkCCA, plus subsampled kPLS2, using an RBF kernel and least-squares winner-takes-all classification.Source: C. Analysis of integrated short-time… · p. 11
  • OPLS- and CCA-type methods perform best overall, with kernel schemes outperforming linear methods when the number of features exceeds 5.PCA and rkPCA perform poorly, while subsampled kPLS2 performs worse than its linear counterpart.Source: C. Analysis of integrated short-time… · p. 11
  • rkOPLS and rkCCA perform best at both AR-pattern and song levels; increasing machine size helps, but gains are not very noticeable beyond 250 nodes.The results support using sparse methods to extend kMVA to large data sets.Source: C. Analysis of integrated short-time… · p. 11

V. CONCLUSIONS

The paper unifies supervised feature extraction through linear and kernel multivariate methods, their connections, and extensions for scale and limited labels. It illustrates their use across benchmark and real-world classification and regression problems.

V. CONCLUSIONS

  • The tutorial compares representative linear and kernel MVA methods, their connections to discriminative feature extraction and dependence estimation, and extensions for large-scale or few-label settings.It reports sparse and semi-supervised extensions for most models.Source: §V · p. 11
  • Beyond PCA, supervised methods are generally better suited to supervised applications because they seek projections that maximize alignment with target variables.Source: §V · p. 11
  • The tutorial illustrates applications in music-genre prediction and land-cover classification and climate-variable estimation, alongside a representative UCI subset.The authors describe these real problems as typically exhibiting complex manifolds.Source: §V · p. 11
Loading 1310.5089v1…