Source-linked AI summary
A Tutorial on Principal Component Analysis
Jonathon Shlens
TL;DR
PCA is widely used but can be poorly understood, motivating this tutorial’s combined intuitive and mathematical treatment. The paper explains PCA as a way to identify meaningful bases, reduce redundancy, and derive principal components through linear algebra, while noting that its source-agnostic nature is also a weakness.
Problem
PCA is widely used to extract information and reduce dimensionality, but its operation and assumptions can be poorly understood.
Method
The tutorial builds intuition from examples, formalizes PCA as a linear-algebra problem, and derives solutions using eigenvector decomposition and its relation to SVD.
Results
PCA identifies a meaningful basis, reduces redundant measurements, and represents principal components through the eigenvectors of the covariance matrix.
Takeaways & Limitations
Understanding PCA’s intuition, mathematics, and assumptions provides a foundation for applying it to dimensional reduction and machine learning.
Takeaways & Limitations
Because PCA is agnostic to the source of the data, its non-parametric plug-and-play behavior is also a weakness.
Abstract
from arXiv · showhide
Principal component analysis (PCA) is a mainstay of modern data analysis - a black box that is widely used but (sometimes) poorly understood. The goal of this paper is to dispel the magic behind this black box. This manuscript focuses on building a solid intuition for how and why principal component analysis works. This manuscript crystallizes this knowledge by deriving from simple intuitions, the mathematics behind PCA. This tutorial does not shy away from explaining the ideas informally, nor does it shy away from the mathematics. The hope is that by addressing both aspects, readers of all levels will be able to gain a better understanding of PCA as well as the when, the how and the why of applying this technique.
I. INTRODUCTION
This tutorial explains PCA through both intuition and mathematical derivation, aiming to clarify how, why, and when the technique works. It presents informal explanations alongside linear-algebraic rigor while assuming working knowledge of linear algebra.
- Motivation: PCA is a standard, non-parametric tool for extracting relevant information from confusing data sets.It is used across fields including neuroscience and computer graphics.
- Approach: The tutorial develops intuition first, then derives PCA within linear algebra and relates it to singular value decomposition.It also discusses practical application and underlying assumptions.
- Scope: The paper is primarily educational and informal, with rigorous proofs placed in the Appendix for readers seeking deeper mathematical understanding.The discussion largely builds on linear algebra while avoiding challenging topics in statistics and optimization.
II. MOTIVATION: A TOY EXAMPLE
The toy example illustrates how experimenters confront unknown relevant dimensions, redundant measurements, and noise. PCA is introduced as a way to extract simple underlying dynamics from such complicated observations.
- Motivation: Complex empirical systems can produce unclear and redundant measurements even when their underlying relationships are simple.The paper frames this as a fundamental obstacle in empirical science.
- Toy example: The ideal spring has one-dimensional motion along the x-axis, but an experimenter may not know this in advance.Its dynamics can be expressed as a function of a single variable x.
- Toy example: Three cameras record two-dimensional projections of the ball while the relevant coordinate axes remain unknown.The setup records the ball’s position in a three-dimensional space using camera projections.
- Measurement problem: Real measurements may contain more dimensions than necessary, making it important to identify which recorded quantities reflect the system’s dynamics.The paper contrasts this with the ideal choice of measuring only position along x.
- Measurement problem: Noise from air, imperfect cameras, or friction further obscures the dynamics that PCA aims to recover.The tutorial uses the spring example to represent challenges faced by experimenters.
III. FRAMEWORK: CHANGE OF BASIS
PCA is framed as finding a linear change of basis that re-expresses data in a more meaningful coordinate system. The framework turns recorded measurements into projections onto new basis vectors and poses the remaining basis-selection questions.
- Data representation: Each sample is an m-dimensional vector containing multiple measurements, and the complete data set is represented by matrix X.In the spring example, each camera contributes a two-dimensional position projection.
- Change of basis: PCA seeks a new basis, formed as a linear combination of the original basis, that best re-expresses the data set.This reframes the problem as choosing an appropriate change of basis.
- Change of basis: A linear transformation P maps the original data matrix X to a new representation Y.P can be interpreted as a matrix transformation, rotation and stretch, or change of basis.
- Change of basis: The rows of P serve as new basis vectors, while each coefficient in Y is the projection of a sample onto the corresponding row.This gives the transformed data a direct projection interpretation.
- Assumptions: Under the linearity assumption, the rows of P become the principal components of X.Linearity restricts PCA to re-expressing data as linear combinations of basis vectors.
- Questions remaining: The framework asks how to best re-express X and how to choose a suitable basis P.Additional assumptions beyond linearity are required to obtain a reasonable result.
A. Noise and Rotation
PCA motivates rotation toward directions of high variance because these are assumed to carry stronger signal than low-variance directions. In the camera example, this direction corresponds to the data cloud’s best-fit line.
- Noise: Noise must be sufficiently low relative to signal for any analysis technique to extract information.Signal-to-noise ratio is commonly measured as a ratio of variances.
- Rotation: The largest-variance direction need not align with the camera’s recording axes.In the two-dimensional example, it aligns with the data cloud’s best-fit line instead.
- Noise: For camera A, straight-line motion represents signal, while deviations from that line represent noise.The cloud’s shape reflects the relative signal and noise variances.
- Rotation: Assuming reasonably good measurements, PCA searches for a rotated basis whose directions have the largest variances.Maximizing variance is treated as maximizing signal-to-noise ratio under this assumption.
B. Redundancy
Redundancy occurs when separate measurements contain overlapping, predictable information. Dimensional reduction exploits this overlap by retaining fewer variables while expressing the data more concisely.
- Redundancy: Redundancy is especially evident when multiple sensors record the same dynamic information.The spring example raises whether recording two variables is necessary.
- Redundancy: Uncorrelated measurements cannot predict one another, whereas highly correlated measurements are highly redundant.Figure 3 presents this as a spectrum from low to high redundancy.
- Redundancy: Nearby cameras or measuring the same quantity in different units can produce highly correlated recordings.The examples compare camera positions and meters versus inches.
- Dimensional reduction: Recording one variable instead of two can preserve the information because a best-fit line estimates one response from the other.This reduces the number of sensor recordings from 2 to 1 variables.
C. Covariance Matrix
PCA formalizes signal and redundancy through covariance: variance represents structure, while covariance captures relationships between measurement types. It seeks a decorrelated, variance-ordered representation under linearity and orthogonality assumptions.
- Covariance: Covariance measures the linear relationship between two variables, with its absolute magnitude indicating their degree of redundancy.Positive and negative values indicate the direction of correlation.
- Covariance matrix: For multiple measurement types, X organizes measurement types by rows and trials by columns, while C_X captures covariance across all measurement pairs.C_X is a square symmetric m×m matrix; diagonal terms are variances and off-diagonal terms are covariances.
- PCA objective: PCA aims to maximize variance as signal and minimize covariance magnitude as redundancy.The desired transformed covariance matrix has zero off-diagonal terms and rank-orders dimensions by variance.
- Principal components: PCA selects orthonormal directions by maximizing variance first, then repeating the search in directions orthogonal to previously selected components.The resulting ordered vectors are the principal components.
- Assumptions: The approach assumes linearity, that large variances contain important structure, and that principal components are orthogonal.The paper notes that the large-variance assumption can be strong and sometimes incorrect, while orthogonality enables linear-algebra decompositions.
V. SOLVING PCA USING EIGENVECTOR DECOMPOSITION
PCA can be solved by eigenvector decomposition of the covariance matrix. Choosing covariance eigenvectors as transformation directions diagonalizes the transformed covariance matrix, with each diagonal entry representing variance along a principal direction.
- Eigenvector solution: The principal components of X are the eigenvectors of its covariance matrix C_X.This follows from the eigenvector-decomposition solution for PCA.
- Eigenvector solution: Choosing P with rows equal to covariance-matrix eigenvectors makes the transformed covariance matrix diagonal.The transformation therefore decorrelates the representation.
- Eigenvector solution: The ith diagonal value of C_Y equals the variance of X along principal direction p_i.These variances provide the basis for rank-ordering the principal directions by importance.
- Implementation: In practice, PCA first subtracts the mean of each measurement type and then computes the eigenvectors of C_X.The tutorial provides Matlab code for this procedure in Appendix B.
- SVD connection: The tutorial also presents SVD as a second algebraic solution and describes it as a more general method for understanding change of basis.The SVD section is mathematically involved but can be skipped without much loss of continuity.
A. Singular Value Decomposition
SVD expresses an arbitrary matrix through orthogonal bases and a diagonal matrix of rank-ordered singular values. This construction provides the framework for interpreting PCA.
- Geometric interpretation: The eigenvectors of X^TX form an orthonormal basis, and singular values satisfy σ_i = √λ_i.The singular values are positive real numbers associated with the eigenvalues of X^TX.
- Matrix construction: SVD decomposes an arbitrary matrix X into two orthogonal matrices and one diagonal matrix.The decomposition can be viewed as a rotation, a stretch, and a second rotation.
- Matrix construction: The matrix form is built by rank-ordering singular values and placing their associated vectors into the columns of U and V.The construction yields XV = UΣ before multiplication by V^T gives the final form.
- Matrix construction: The matrices U and V are orthogonal, while Σ is diagonal with non-zero singular values on its diagonal.The matrix equation represents all scalar decomposition equations simultaneously.
- Geometric interpretation: The columns of V span the row space of X, while the rows of U^T span its column space.These orthonormal bases formalize possible inputs and outputs of the matrix.
C. SVD and PCA
PCA and SVD are intimately related through mean-centered data. Computing the SVD of the transformed matrix yields the principal components as eigenvectors of the covariance matrix.
- Connecting SVD and PCA: Mean-centering the data produces a matrix Y whose columns have zero mean.The choice of Y is motivated by analyzing Y^TY.
- Connecting SVD and PCA: Y^TY equals the covariance matrix of X, so the columns of V from SVD are the principal components of X.This provides an SVD-based algorithm for computing PCA.
- Geometric interpretation: The matrix V spans the row space of Y and therefore also spans the column space of 1/√n X.Thus, finding principal components amounts to finding an orthonormal basis for the column space of X.
VII. DISCUSSION
PCA provides an analytical route to dimensional reduction by identifying dimensions that account for variability. Its usefulness depends on assumptions about the data and the meaning of high variance.
- Benefits: PCA reveals simple underlying structures in complex data and provides a brief implementation roadmap.Figure 5 summarizes the steps for applying PCA.
- Benefits: Variance along each principal component quantifies the relative importance of each dimension for describing the data.PCA is useful when a small number of components reasonably characterize the complete data set.
- Limitations: PCA can fail when non-Gaussian data or non-orthogonal axes make the directions of largest variance inappropriate.The ferris-wheel example has dynamics described by the wheel phase, a nonlinear combination of the naive basis.
- Limitations: PCA is non-parametric and requires no parameters to tune, but its agnosticism toward data sources can also be a weakness.The method can fail to recover a meaningful variable when the underlying structure is nonlinear.
A. Limits and Statistics of Dimensional Reduction
PCA decorrelates data by removing second-order dependencies, but this can miss higher-order structure and produce unsatisfying reductions when data lie along non-orthogonal axes. Kernel PCA and ICA address related limitations through nonlinear transformations or stronger independence assumptions.
- PCA searches for orthogonal directions that decorrelate the data by removing second-order dependencies.
- Non-orthogonal data arrangements can yield unsatisfying PCA results because successive principal directions must be perpendicular.
- PCA is optimal under mean squared error for predicting the original data with an orthogonal reduced representation.
- Higher-order dependencies can remain after PCA removes second-order dependencies, leaving important structure unrevealed.
- Kernel PCA applies a nonlinear transformation using prior knowledge, while ICA seeks statistically independent reduced dimensions but can be difficult to compute.
Appendix A: Linear Algebra
The appendix develops linear-algebra results underlying PCA, especially orthogonal diagonalization of symmetric matrices and properties of X^T X. It also connects these foundations to covariance-based and SVD-based implementations.
- Matrix properties: Orthogonal diagonalizability and symmetry are equivalent, although the reverse direction of the proof is left to the reader.
- Linear-algebra theorems: A symmetric matrix is diagonalized by an orthogonal matrix of its eigenvectors.
- Linear-algebra theorems: Distinct eigenvalues of a symmetric matrix have orthogonal eigenvectors, and normalized eigenvectors make the diagonalizing matrix orthogonal.
- Matrix properties: For any matrix X, X^T X is symmetric and its orthonormal eigenvectors produce an orthogonal set of vectors Xv_i with lengths √λ_i.
- PCA implementations: The covariance implementation centers the data, computes covariance, finds eigenvectors and eigenvalues, and sorts variances in decreasing order.
- PCA implementations: A second implementation computes PCA through singular value decomposition rather than through the covariance matrix.