Source-linked AI summary
Optimal whitening and decorrelation
Agnan Kessy, Alex Lewin, Korbinian Strimmer
TL;DR
Whitening admits infinitely many valid transformations, leaving open how to compare and choose among them. The paper analyzes five natural procedures through cross-covariance and cross-correlation, recommending ZCA-cor for maximal similarity and PCA-cor for maximal compression.
Problem
The whitening constraint allows infinitely many transformations producing different sphered variables, creating a need to compare and select procedures for particular situations.
Method
The paper develops the theory of five natural whitening procedures and investigates cross-covariance and cross-correlation between sphered and original variables to discriminate among them.
Results
The paper recommends ZCA-cor whitening for sphered variables maximally similar to the originals and PCA-cor whitening when maximal compression is desirable.
Takeaways & Limitations
Use scale-invariant cross-correlation as the optimization basis, choosing ZCA-cor for similarity and PCA-cor for compression.
Takeaways & Limitations
Cholesky whitening implicitly assumes an ordering of variables, although that ordering can be useful in time-course analysis for accounting for autocorrelation.
Abstract
from arXiv · showhide
Whitening, or sphering, is a common preprocessing step in statistical analysis to transform random variables to orthogonality. However, due to rotational freedom there are infinitely many possible whitening procedures. Consequently, there is a diverse range of sphering methods in use, for example based on principal component analysis (PCA), Cholesky matrix decomposition and zero-phase component analysis (ZCA), among others. Here we provide an overview of the underlying theory and discuss five natural whitening procedures. Subsequently, we demonstrate that investigating the cross-covariance and the cross-correlation matrix between sphered and original variables allows to break the rotational invariance and to identify optimal whitening transformations. As a result we recommend two particular approaches: ZCA-cor whitening to produce sphered variables that are maximally similar to the original variables, and PCA-cor whitening to obtain sphered variables that maximally compress the original variables.
1 Introduction
Whitening transforms a random vector into same-dimensional variables with unit diagonal covariance, simplifying multivariate analysis. Because the whitening constraint permits infinitely many matrices, the paper frames selecting among them as an optimization problem using cross-covariance and cross-correlation.
- Whitening linearly transforms a d-dimensional random vector with positive definite covariance into a same-dimensional vector with unit diagonal covariance.
- The resulting variables have unit variances and no correlations, whereas standardization alone does not remove correlations.
- Mean-centering is often used with whitening but is not necessary to obtain unit variances or a white covariance.
- The whitening constraint does not uniquely determine W, yielding infinitely many transformations and different orthogonal sphered variables.
- The paper addresses how to compare and select whitening transformations by investigating cross-covariance and cross-correlation between sphered and original variables.
2 Notation and useful identities
The paper establishes covariance and correlation matrix decompositions, inverse square roots, and eigenvalue conventions used throughout its whitening analysis.
- The covariance matrix is decomposed into the correlation matrix and diagonal variance matrix, while covariance and correlation matrices are separately eigendecomposed.
- The unique inverse matrix square roots Σ^-1/2 and P^-1/2 are constructed from the corresponding eigendecompositions.
- Eigenvalues are conventionally sorted from largest to smallest, and eigenvectors remain valid after independently changing column signs.
3 Rotational freedom in whitening
Whitening combines covariance-based rescaling with rotations, and orthogonal rotation choices generate the family of valid whitening matrices.
- The whitening constraint permits arbitrary orthogonal matrices in the polar decomposition, exposing rotational freedom in W.
- All whitening matrices share singular values Λ^-1/2, so their common fundamental operation is rescaling by the inverse square roots of covariance eigenvalues.
- In the covariance representation, whitening rotates by U^T, rescales, and may apply a further rotation determined by Q1.
- In the standardized representation, variables are variance-scaled, rotated by G^T, rescaled by correlation eigenvalues, and possibly rotated again.
- The rotations Q1 and Q2 differ between representations and are linked by an orthogonal matrix A that generally cannot be further simplified.
4 Cross-covariance and cross-correlation
Cross-covariance and cross-correlation encode the rotational choices hidden within whitening matrices. Their properties therefore provide criteria for comparing and selecting whitening transformations.
- Cross-covariance and cross-correlation between z and x are linked to the rotation matrices Q1 and Q2 in the two whitening decompositions.
- The rotational freedom in W is directly reflected in Φ and Ψ, enabling whitening procedures to be selected by constraining or optimizing these matrices.
- Neither Φ nor Ψ is generally symmetric unless its associated rotation is the identity, and Ψ's diagonal elements need not equal 1.
- Each original variable is perfectly explained by a linear combination of the uncorrelated whitened variables, giving a squared multiple correlation of 1.
- The column sums of squared cross-correlations equal 1, whereas row sums vary across whitening procedures and inform transformation choice.
5 Five natural whitening procedures
The paper describes five natural whitening procedures, distinguished by their rotations, scaling choices, and constraints on cross-covariance or cross-correlation. ZCA-cor and PCA-cor whiten standardized variables using correlation-based transformations, while Cholesky whitening uniquely imposes a lower-triangular structure.
- ZCA whitening uses the inverse covariance square root and is the unique sphering method with a symmetric whitening matrix.
- PCA whitening rotates variables by the covariance eigenmatrix and scales them by the inverse square roots of the eigenvalues.It is related to ZCA whitening by a rotation back to the original coordinate system.
- PCA whitening remains sign-ambiguous unless eigenvector signs are adjusted, after which positive diagonal cross-covariance and cross-correlation make the transformation unique.
- Cholesky whitening is based on precision-matrix factorization and uniquely yields lower-triangular cross-covariance and cross-correlation matrices with positive diagonals.Unlike the other four methods, it arises from a symmetry constraint and implicitly assumes an ordering of variables.
- ZCA-cor whitening standardizes variables before applying correlation-based ZCA whitening, producing a generally asymmetric matrix distinct from covariance-based ZCA.
- PCA-cor whitening applies PCA whitening to standardized variables, using correlation eigenvectors and eigenvalues; it differs from PCA whitening.ZCA-cor is PCA-cor followed by rotation back to the standardized-variable frame.
6 Optimal whitening
The paper selects whitening transformations by analyzing cross-covariance and cross-correlation between whitened and original variables. These criteria identify ZCA-based methods for similarity and PCA-based methods for compression, while Cholesky whitening reflects an ordering constraint.
- Cross-covariance and cross-correlation reveal how rotational freedom affects whitening procedures and provide objective criteria for selecting among them.The paper finds a corresponding optimality criterion for each of five natural transforms.
- ZCA whitening uniquely maximizes average cross-covariance between corresponding whitened and original variables.It is also uniquely associated with a symmetric cross-covariance matrix.
- ZCA-cor whitening uniquely maximizes corresponding cross-correlations and yields a symmetric cross-correlation matrix.Its scale-invariant objective compares whitened variables with standardized originals.
- PCA whitening uniquely maximizes integration or compression of all original variables within each sphered component under cross-covariance.Its component-wise compression measures are ordered monotonically, matching the decreasing covariance eigenvalues.
- PCA-cor whitening uniquely maximizes integration or compression under cross-correlation, using a scale-invariant measure.
- Cholesky whitening uniquely produces lower-triangular positive-diagonal cross-covariance and cross-correlation matrices, but assumes an ordering of variables.This ordering can be useful in time-course analysis for accounting for autocorrelation.
7 Application
The application evaluates five whitening transforms on the iris data using empirical covariance estimates, comparing cross-correlations and objective-function values. ZCA methods best preserve component-wise similarity, while PCA methods best compress the original variables.
- Data and evaluation: The iris example applies five whitening transforms to d = 4 correlated variables across n = 150 observations.All estimates use the empirical covariance S.
- Data and evaluation: The analysis compares estimated cross-correlations between whitened and original components with values of the corresponding objective functions.The cross-correlations occupy the upper half of Table (2), while objective-function values occupy the lower half.
- Similarity-preserving transforms: ZCA and ZCA-cor produce sphered variables most correlated with the original data component-wise.ZCA is best for the covariance-based objective, whereas ZCA-cor is best for the correlation-based objective.
- Compression-oriented transforms: PCA and PCA-cor maximize simultaneous linkage between each whitened component and all original variables.For these methods, only z1 with x1 and z2 with x2 are highly correlated; later corresponding pairs are effectively uncorrelated.
- Compression-oriented transforms: PCA-cor whitening achieves higher maximum total squared correlation between z1 and all components of x than PCA whitening, indicating better compression.
- Comparison across transforms: Cholesky whitening ranks third and uniquely produces one perfectly correlated pair, (z4, x4), by construction.
8 Conclusion
The conclusion resolves whitening’s rotational ambiguity by comparing whitened and original variables through cross-covariance and cross-correlation. It recommends ZCA-cor for minimal adjustment and PCA-cor for maximal compression, depending on the application context.
- Motivation and contribution: Whitening methods support preprocessing and subsequent statistical analysis, but the whitening constraint permits infinitely many rotationally distinct procedures.
- Motivation and contribution: Cross-covariance Φ and cross-correlation Ψ between whitened and original variables can break rotational freedom and discriminate among whitening transformations.
- Motivation and contribution: The paper studies five natural transforms as optimizers of functions of Φ or Ψ, or as methods satisfying symmetry constraints on those matrices.
- Application context: Whitening choice depends on whether the application prioritizes minimal adjustment with interpretability or compression reflecting intrinsic effective dimension.Variable selection benefits from retaining the original interpretation, whereas compression produces variables without an interpretable relation to the original data.
- Recommendations: ZCA-cor whitening is recommended when sphered variables should be maximally similar to the originals.
- Recommendations: PCA-cor whitening is recommended when maximal compression is desirable.