PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel Principal Component Analysis and its Applications in Face Recognition and Active Shape Models

Quan Wang

arXiv:1207.3538v3cs.CV

TL;DR

The paper addresses the limits of standard PCA for complex, nonlinear data and the need to reconstruct original-space shapes from kernel-PCA features. It reviews kernel PCA, develops Gaussian-kernel pre-image reconstruction, and applies the method to classification and active shape models. The experiments report more complex structure discovery, lower classification error rates, and promising additional deformation patterns.

  • Problem

    Standard PCA is limited to linear dimensionality reduction, while active shape models require reconstructing shapes from their feature representations.

  • Method

    The paper combines kernel PCA with Gaussian-kernel pre-image reconstruction and applies it to classification and active shape models.

  • Results

    Gaussian kernel PCA revealed more complex data structures, achieved much lower classification error rates, and produced promising additional deformation patterns in active shape models.

  • Takeaways & Limitations

    Kernel PCA can extend PCA-based classification and shape modeling to structures that standard PCA does not capture well.

  • Takeaways & Limitations

    The Gaussian-kernel parameter-selection strategy may require modification for different datasets.

Abstract

from arXiv · show

Principal component analysis (PCA) is a popular tool for linear dimensionality reduction and feature extraction. Kernel PCA is the nonlinear form of PCA, which better exploits the complicated spatial structure of high-dimensional features. In this paper, we first review the basic ideas of PCA and kernel PCA. Then we focus on the reconstruction of pre-images for kernel PCA. We also give an introduction on how PCA is used in active shape models (ASMs), and discuss how kernel PCA can be applied to improve traditional ASMs. Then we show some experimental results to compare the performance of kernel PCA and standard PCA for classification problems. We also implement the kernel PCA-based ASMs, and use it to construct human face models.

1. Introduction

PCA reduces dimensionality by projecting data onto a lower-dimensional linear subspace that captures maximum variance. In active shape models, PCA represents aligned object shapes through principal deformation patterns and weighted shape reconstructions.

  • Principal Component Analysis: PCA seeks a lower-dimensional linear subspace whose projected features have the largest variance.The projection dimension M is smaller than the original dimension D.
  • Principal Component Analysis: The PCA projection directions are eigenvectors of the data covariance matrix, ordered by decreasing eigenvalue.The kth eigenvector corresponds to the kth largest eigenvalue.
  • Active Shape Models: Active shape models use point distribution models to represent object shapes and locate objects in new images.The paper focuses on the statistical shape model rather than searching for shapes in new images.
  • Active Shape Models: A shape with n points is represented as a 2n-dimensional vector containing the coordinates of its points.The shapes are assumed to have been aligned before computing the mean shape and deviations.
  • Active Shape Models: PCA models aligned-shape variation using eigenvectors of the shape covariance matrix and deformation weights.The first t eigenvectors form P, while b contains the weights for the corresponding deformation patterns.

2. Kernel PCA

Kernel PCA extends PCA to nonlinear dimensionality reduction by applying PCA in an implicit feature space through kernel matrices. The paper also addresses reconstructing approximate pre-images, which is needed when kernel-PCA features must be converted back into shapes or other original-space objects.

  • Kernel PCA: Kernel PCA generalizes standard PCA to nonlinear dimensionality reduction for data that cannot be represented well by a linear subspace.It applies a nonlinear transformation before performing PCA.
  • Kernel PCA: Kernel methods avoid explicitly computing the transformed features by constructing a kernel matrix directly from the training data.The method then centers the matrix, solves for eigenvectors, and computes kernel principal components.
  • Kernel PCA: The kernel-PCA procedure computes principal components from eigenvectors and eigenvalues of the centered kernel representation.The centered Gram matrix is used when the transformed data do not have zero mean.
  • Kernel Functions: Kernel PCA uses kernel functions such as polynomial and Gaussian kernels to encode relationships in the transformed feature space.The polynomial kernel includes a positive constant parameter, while the Gaussian kernel is distance-based.
  • Reconstructing Pre-Images: Pre-image reconstruction approximates original-space data from kernel-PCA features when feature extraction alone is insufficient.This reconstruction is required in active shape models, where PCA features must be converted back into object shapes.
  • Reconstructing Pre-Images: For Gaussian kernels, the approximate pre-image is obtained through an iterative procedure based on distances to training points.The paper notes that exact pre-images are difficult to find and uses an approximation instead.

3. Experiments

The paper reports three experiments: two classification studies without pre-image reconstruction and a third combining active shape models with kernel PCA and pre-image reconstruction.

  • Experimental Design: The first two experiments evaluate classification without pre-image reconstruction.
  • Experimental Design: The third experiment combines active shape models with kernel PCA and includes pre-image reconstruction.
  • Experimental Design: Together, the experiments assess kernel PCA for classification and for kernel-PCA-based active shape models.

3.1. Pattern Classification for Synthetic Data

The synthetic experiment uses noisy points on two concentric spheres and compares standard PCA with polynomial and Gaussian kernel PCA after projection to two dimensions. Gaussian kernel PCA completely separates the classes, unlike standard PCA.

  • 3.1.1. Data Description: The dataset contains equal numbers of noisy points on two concentric spheres with radii 40 and 100.The experiment uses N = 1000 points and Gaussian coordinate noise with standard deviation 1.
  • 3.1.2. PCA and Kernel PCA Results: The data are projected from three dimensions into a two-dimensional feature space using standard PCA and two kernel-PCA variants.The variants use a polynomial kernel with d = 5 and a Gaussian kernel with σ = 27.8.
  • 3.1.2. PCA and Kernel PCA Results: The experiment is unsupervised because neither standard PCA nor kernel PCA uses the class labels as input.Different colors mark the classes only for visualization.
  • 3.1.2. PCA and Kernel PCA Results: Standard PCA does not reveal structural information, while polynomial kernel PCA clusters one class but does not make the classes linearly separable.
  • 3.1.2. PCA and Kernel PCA Results: Gaussian kernel PCA makes the two classes completely linearly separable.

3.2. Classification for Aligned Human Face Images

The paper evaluates PCA and Gaussian kernel PCA for classifying aligned Yale Face Database B images using a linear classifier. Gaussian kernel PCA achieves much lower training and testing error rates than standard PCA.

  • Data and setup: The experiment classifies aligned human face images from two Yale Face Database B subjects using PCA or Gaussian kernel PCA features and a linear classifier.The dataset contains 5760 gray-level images of 10 subjects; the experiment uses 51 training and 13 testing images from each of two subjects.
  • Data and setup: Each 168 × 192 image is represented by a 32256-dimensional intensity vector, then reduced to the 9 most significant PCA or Gaussian kernel PCA features.Testing features require only PCA eigenvectors, whereas Gaussian kernel PCA also requires the training data.
  • Classification method: Gaussian kernel PCA uses a Gaussian kernel with σ = 22546, while classification uses the simplest linear classifier.The reported comparison is between standard PCA and Gaussian kernel PCA under the same classification framework.
  • Classification results: Gaussian kernel PCA achieves much lower training and testing error rates than standard PCA.The error-rate comparison is reported in Table 1.

3.3. Kernel PCA-Based Active Shape Models

The paper replaces standard PCA in active shape models with Gaussian kernel PCA to explore nonlinear face deformations. Varying individual kernel-PCA features produces face changes that differ from the orientation and microexpression patterns associated with standard PCA.

  • ASM formulation: Standard PCA extracts principal deformation patterns from ASM shape vectors, while kernel PCA is proposed to discover additional hidden deformation patterns.The ASM represents object shapes with point distribution models.
  • Data description: The ASM experiments use 1521 BioID face images with 20 manually annotated landmarks, giving each normalized shape vector 40 coordinates.Each landmark contributes two coordinates, and the x and y coordinates are restricted to [0, 1] before PCA or kernel PCA.
  • Shape reconstruction: Standard PCA reconstructs shapes from x = ¯x + Pb, while kernel PCA reconstructs shapes through the pre-image reconstruction algorithm Ω(y).The paper studies reconstruction from kernel-PCA features because ASM applications require recovering shapes from feature representations.
  • Standard PCA experiments: Standard PCA varies one deformation coefficient bk at a time within [−3√λk, 3√λk], setting all other coefficients to zero.The first three PCA-feature effects are shown in Figures 7–9.
  • Kernel PCA experiments: Gaussian kernel PCA varies one feature yk within [ȳk − cσyk, ȳk + cσyk] while fixing the others at their means; c = 3 produces Figures 10–12.The Gaussian kernel parameter is σ = 0.3275.
  • Experimental results: The first PCA feature affects face orientation, the second partly controls an amazement-to-calmness microexpression, and Gaussian kernel PCA features produce different microexpressions.These observations compare the deformation patterns revealed by standard and Gaussian kernel PCA.

4. Discussions

The discussion focuses on selecting Gaussian-kernel parameters, interpreting Gaussian kernel PCA through neighborhood structure, and applying it to classification and pre-image reconstruction. The authors report stronger structure discovery than standard PCA while noting that their parameter strategy may require dataset-specific modification.

  • Parameter selection: Parameter selection directly affects kernel PCA performance, with σ governing the Gaussian kernel's distance-based behavior.The discussion emphasizes selecting σ relative to inter-class and inner-class distances.
  • Parameter selection: The proposed strategy chooses σ to capture neighborhood information, ensuring in these experiments that it is large enough for neighboring points but much smaller than inter-class distances.The authors state that this strategy may need modification for different datasets.
  • Results: Gaussian kernel PCA can perfectly separate the synthetic classes when σ is properly selected, whereas standard PCA cannot.The separation is achieved without using class labels.
  • Results: In face-image classification, Gaussian kernel PCA achieves lower training error and much lower testing error than standard PCA.The authors use this comparison to argue that Gaussian kernel PCA reveals more complex hidden data structures.
  • Intuitive explanation: Gaussian kernel PCA is interpreted as using distances among training points, similarly to nearest-neighbor or clustering methods, followed by eigenvector analysis of cluster directions.A suitable σ determines the capture range connecting nearby points in the original feature space.
  • Kernel choice: Gaussian kernel PCA is used because it is intuitive, easy to implement, and supports pre-image reconstruction, although more powerful learned kernel matrices exist.The paper focuses mostly on the Gaussian kernel for these reasons.

5. Conclusion

The paper reviews PCA, kernel PCA, and ASMs, then develops Gaussian kernel PCA methods for pre-image reconstruction and ASM construction. Experiments report more complex structure discovery, lower classification errors, promising deformation modeling, and a parameter-selection method that worked well in the authors' experiments.

  • Contributions: The paper focuses on Gaussian kernel PCA pre-image reconstruction and uses it to design kernel PCA-based active shape models.The work combines theoretical discussion with experiments on synthetic data and human face images.
  • Classification results: Gaussian kernel PCA revealed more complicated data structures than standard PCA and achieved much lower classification error rates in the reported experiments.The conclusion summarizes results from synthetic-data and human-face-image classification experiments.
  • Active shape models: Gaussian kernel PCA-based ASMs were promising for providing more deformation patterns than traditional ASMs on human face images.The authors identify combining both ASM types as a potential application for microexpression recognition.
  • Parameter selection: The proposed parameter-selection method found proper Gaussian-kernel parameters and worked well in the authors' experiments.The conclusion presents parameter selection as an additional contribution.
Loading 1207.3538v3…