PaperlayerView exact revision on arXiv

Source-linked AI summary

Principal Component Analysis: A Natural Approach to Data Exploration

Felipe L. Gewers, Gustavo R. Ferreira, Henrique F. de Arruda, Filipi N. Silva, Cesar H. Comin, Diego R. Amancio, Luciano da F. Costa

arXiv:1804.02502v2cs.CEstat.COstat.ME

TL;DR

The paper examines how PCA can organize and simplify varied datasets while addressing the limited systematic evidence about its efficiency across data types and preprocessing choices. It presents PCA theory and applications, surveys representative uses, and experimentally studies variance explanation and dimensionality reduction. The results show that PCA can retain substantial variance with few components, while standardization materially changes outcomes and can emphasize unwanted noise.

  • Problem

    The paper addresses the need to organize large, varied datasets and to investigate PCA efficiency systematically across data types and preprocessing choices.

  • Method

    The paper combines an accessible theoretical treatment of PCA, a survey of representative applications, and experiments on variance explanation and dimensionality reduction.

  • Results

    PCA retains substantial variance with few components in many datasets, while standardization changes variance concentration and can cause noise-dominated variables to appear important.

  • Takeaways & Limitations

    Using PCA effectively requires interpreting variance preservation together with the choice to standardize or retain the original measurements.

Abstract

from arXiv · show

Principal component analysis (PCA) is often used for analyzing data in the most diverse areas. In this work, we report an integrated approach to several theoretical and practical aspects of PCA. We start by providing, in an intuitive and accessible manner, the basic principles underlying PCA and its applications. Next, we present a systematic, though no exclusive, survey of some representative works illustrating the potential of PCA applications to a wide range of areas. An experimental investigation of the ability of PCA for variance explanation and dimensionality reduction is also developed, which confirms the efficacy of PCA and also shows that standardizing or not the original data can have important effects on the obtained results. Overall, we believe the several covered issues can assist researchers from the most diverse areas in using and interpreting PCA.

I. INTRODUCTION

PCA is presented as an accessible approach for organizing, visualizing, and simplifying high-dimensional data, while the paper surveys applications and investigates how preprocessing affects variance preservation. Its central operation rotates feature-space axes toward directions of maximum dispersion, enabling dimensionality reduction when later components contain relatively little variation.

  • Motivation: PCA addresses the challenge of extracting relevant information from increasingly large and varied datasets by organizing and summarizing their measurements.The paper situates PCA within data mining and the broader need to make accumulated data useful.
  • PCA principle: PCA rotates the coordinate axes of an N-dimensional feature space so the first axis captures maximum data dispersion and subsequent axes capture progressively smaller dispersion.In the bean example, the first PCA axis follows the dominant elongation of the point distribution, while PCA2 contains the second-largest one-dimensional dispersion.
  • Dimensionality reduction: When the data distribution is sufficiently elongated, PCA can discard a low-dispersion axis and represent the data in a lower-dimensional space with limited loss of overall variation.The bean example reduces the resulting feature space to M = 1 after discarding PCA2.
  • Dimensionality reduction: PCA simplifies data with minimum loss of overall dispersion, but its largest-variation directions are not guaranteed to be the most useful directions for every study.Typical applications are characterized by M << N, while usefulness remains study-dependent.
  • Paper scope: The paper combines an intuitive presentation of PCA, a survey of applications, and an experimental investigation of dimensionality reduction and variance preservation across data categories.It also includes theoretical material and discusses applications as data exploration or preprocessing for simplified statistical representation.
  • Preprocessing: Standardization translates each feature by subtracting its mean and then scales it by dividing by its standard deviation, producing a zero-mean, unit-variance variable.The paper distinguishes this specific operation from normalization as a broader class of alterations to original measurements.

III. PRINCIPAL COMPONENT ANALYSIS

PCA linearly transforms centered data into a feature space whose axes are covariance- or correlation-matrix eigenvectors, ordering components by explained variance. The transformation preserves total variance while enabling dimensionality reduction when a small number of axes captures the desired variance.

  • III. PRINCIPAL COMPONENT ANALYSIS: PCA represents the dataset as a matrix X and obtains projected data Y by multiplying it by a transformation matrix W.W is constructed from eigenvectors of the data covariance matrix.
  • III. PRINCIPAL COMPONENT ANALYSIS: PCA first centers measurements by subtracting each feature’s empirical mean before computing the covariance matrix.The centered data matrix is denoted X̂.
  • III. PRINCIPAL COMPONENT ANALYSIS: The transformation matrix W is formed by stacking eigenvectors of the covariance matrix after eigenvalues are sorted in decreasing order.The eigenvalues are non-negative and determine the ordering of the corresponding principal axes.
  • III. PRINCIPAL COMPONENT ANALYSIS: The first PCA axis has maximum one-dimensional dispersion, while subsequent axes capture progressively smaller dispersion.Reducing dimensionality retains the first M ≤ N PCA axes.
  • III. PRINCIPAL COMPONENT ANALYSIS: PCA preserves total data variance under axes rotation, and the retained variance ratio determines how many principal axes to keep.For example, preserving approximately 70% requires choosing M so that G ≈ 70%.
  • III. PRINCIPAL COMPONENT ANALYSIS: A reduced representation with M < N is not guaranteed for every dataset because achievable variance preservation depends on the eigenvalue distribution.Highly correlated variables favor variance preservation, and the paper investigates typical preservation across real-world data categories.

IV. TO STANDARDIZE OR NOT TO STANDARDIZE?

The section examines whether original variables should be standardized before PCA. Standardization equalizes variable scales but can amplify noise, so its effect depends on the relative size of meaningful variation and measurement error.

  • IV. TO STANDARDIZE OR NOT TO STANDARDIZE?: Standardization converts variables to dimensionless forms with zero mean and unit standard deviation, giving them similar variation ranges.It can reduce bias from differences in the original scales.
  • IV. TO STANDARDIZE OR NOT TO STANDARDIZE?: Standardization may magnify unwanted small-scale variation when that variation reflects experimental error or noise rather than meaningful structure.The paper recommends reducing the noise or avoiding standardization in such cases.
  • IV. TO STANDARDIZE OR NOT TO STANDARDIZE?: When additive noise dominates a measurement, the measured Pearson correlation between perfectly correlated properties approaches 0 as σP2/σε decreases.With σP2/σε ⪆ 2, the measured correlation is close to the true value of 1 in the reported simulation.
  • IV. TO STANDARDIZE OR NOT TO STANDARDIZE?: If noise completely dominates a standardized measurement, PCA can assign great importance to that meaningless measurement.The first-axis explanation is evaluated as a function of σP2/σε, with error bars from 1000 simulations.

V. OTHER ASPECTS OF PCA

This section discusses PCA’s axis-direction ambiguity and its behavior under general linear rotations. PCA preserves covariance eigenvalues under rotation, and choosing covariance eigenvectors yields diagonal transformed covariance ordered by variance.

  • A. PCA axes direction: PCA axis directions are arbitrarily defined because both an eigenvector v and its negation −v are valid eigenvectors.Consequently, alternative sign orientations of the same PCA projection are all correct.
  • B. PCA and Rotation: A rotation of the data matrix is a linear transformation represented by an orthogonal matrix W.The covariance matrix of the transformed data is obtained from the rotation.
  • B. PCA and Rotation: The eigenvalues of the covariance matrix are conserved under rotation.
  • B. PCA and Rotation: When W contains covariance-matrix eigenvectors, the transformed covariance matrix is diagonal, and sorting eigenvectors by decreasing eigenvalues gives the PCA transformation matrix.

C. Demonstration of Maximum Variance

PCA chooses an orthonormal transformation whose rows are covariance-matrix eigenvectors, ordered so that successive axes preserve as much original variance as possible.

  • Maximum-variance criterion: PCA maximizes each transformed feature’s variance under an orthonormal rotation constraint.The constraint prevents the trivial unbounded solution and requires WW^T = I.
  • Eigenvector solution: The maximizing rows of W are eigenvectors of the covariance matrix.Differentiation of the constrained objective yields the covariance eigenvector condition.
  • Axis ordering: Eigenvectors are placed in W by decreasing eigenvalue, assigning the largest-variance direction to the first row.The first transformed axis uses the eigenvector associated with the largest covariance eigenvalue.
  • Dimensionality reduction: The first M eigenvectors define an M-dimensional space with optimal preservation of variance from the original data.This provides the basis for retaining a reduced set of principal components.
  • Interpretation: Orthogonality and similarity between eigenvectors and the data help explain PCA’s effectiveness at representing variance.The covariance matrix’s symmetry yields orthogonal eigenvectors, while each eigenvector acts as a data prototype.

VI. PCA LOADINGS AND BIPLOTS

PCA loadings and projections connect original measurements to principal components, while projection angles and biplots provide visual cues about measurement relationships and component contributions.

  • PCA loadings: PCA1 can be expressed as a weighted linear combination of the original measurements.In the beans example, Diameter and √Area have weights 0.82 and 0.57, contributing almost equally to PCA1.
  • Variable projections: Projected measurement axes reveal relationships through their angles, with nearly parallel petal-length and petal-width axes indicating similar measurements.The projection is illustrated for the four Iris measurements: sepal length, sepal width, petal length, and petal width.
  • Biplots: A biplot displays normalized PCA components together with vectors representing the original measurements.For the Iris dataset, the biplot uses PCA components divided by their respective standard deviations.
  • Correlation interpretation: The projection of a measurement vector onto a PCA axis corresponds to its Pearson correlation with that component.This relationship motivates the normalized representation used in the biplot.
  • Correlation interpretation: Angles between measurement vectors approximate correlations among measurements and between measurements and PCA components.The biplot therefore offers an intuitive visualization of these relationships.

VII. LDA – ANOTHER PROJECTION METHOD

LDA is presented as a supervised projection method for categorized data, differing from unsupervised PCA by optimizing separation between groups rather than relying on the covariance matrix.

  • Method distinction: LDA operates on categorized objects and is supervised, whereas PCA is unsupervised.Its objective is to maximize separation between predefined groups using scatter distances.
  • Scatter matrices: LDA measures within-group dispersion using the intra-group scatter matrix.The matrix combines dispersion across the individual groups.
  • Scatter matrices: LDA measures dispersion between group centroids using the inter-group scatter matrix.The definition weights group-centroid dispersion by the number of objects in each group.
  • Separation objective: The separation matrix S is formed by multiplying inter-group scatter by the inverse of intra-group scatter.The trace of S provides a measure of group separation.
  • Projection procedure: LDA applies the PCA sequence of operations while replacing the covariance matrix with S.This preserves the projection procedure but changes the matrix governing the projection.

C. Neuroscience

PCA is used across neuroscience and related quantitative applications to analyze high-dimensional biological, clinical, imaging, behavioral, and sports data, often for reduction, classification, or interpretation.

  • Neuroscience applications: Neuroscience applies PCA to brain-activity and morphology data, including EEG, MRI, classification tasks, and psychophysical experiments.These uses reflect the brain’s many components and complex topology.
  • Neuroscience applications: An EEG classification method using PCA on nine selected features reported a 3.2% false alarm rate and 5.2% missed detection rate.The transformed features were supplied to a cosine radial basis function neural-network classifier for normal, interictal, and ictal EEG.
  • Medical imaging: For MRI classification, PCA reduced 65536 features to 1024 while retaining 95.4% of the variance.The reduced data were then used by a kernel support vector machine to infer brain health across several diseases.
  • Neural structure: PCA describes dendritic arborizations by analyzing the relative standard deviations of their transformed three-dimensional point data.The digitalized points represent roots, nodes, tips, and curvatures.
  • Behavior and sports: PCA supports behavioral and performance analyses, including pain recall, facial-expression analysis, athletic synergies, and body-coordinate compression.In one alpine-skiing experiment, four components accounted for 95.5% of the PCA variance.
  • Chemistry and materials science: PCA is also used in chemistry and materials science to visualize metabolomic data, reduce diagnostic features, and aggregate spectra with many peaks.Reported examples include meat-quality analysis, pancreatic-cancer diagnosis, phytomedicine characterization, superconductors, and polystyrene spectra.

H. Engineering

PCA is applied across engineering and computer-vision problems to reduce, extract, and organize complex measurements for monitoring, diagnosis, recognition, and compression. These applications demonstrate broad utility but also show that benefits depend on the dataset and downstream method.

  • Engineering applications: PCA supports engineering applications including bridge damage detection, sensor validation, bearing diagnosis, gearbox monitoring, and batch-process monitoring.Applications use PCA for preprocessing, feature reduction, fault detection, sensor assessment, and reduced-space modeling.
  • Engineering applications: The PCA-based gearbox approach was more robust to environmental changes than the alternative image-analysis approach.PCA computed features and reduced dimensionality from infrared images of gearbox temperatures.
  • Electronics: Two principal axes achieved high variance explanation for transistor-device data, with parameter variation forming clusters within transistor types.A subsequent study used PCA to quantify variability among devices in the same transistor array and confirmed higher within-array uniformity.
  • Computer vision: PCA has been used in computer vision for face recognition, unsupervised image and video classification, texture analysis, and feature combination.Eigenfaces project pixel-intensity data into a lower-dimensional space, while GPCA and clustering support image and video analysis.
  • Trade-offs: PCA can reduce computational cost with minimal accuracy loss, but preprocessing before SVM considerably reduced classification accuracy in one analyzed dataset.The usefulness of combining PCA with classification therefore depends on the classification technique and dataset.
  • Data compression: PCA-based compression achieved a 30:1 compression rate while retaining 95% of variance in stellar spectra.Another cited compression approach outperformed wavelet-based spectral decorrelation for hyperspectral images.

K. Deep Learning

The surveyed deep-learning applications use PCA mainly to compress data, generate features, or preprocess inputs for image and other classification tasks. Results include higher accuracy in one hyperspectral pipeline, while a PCA-based sleep-stage method was not the best-performing approach.

  • Deep-learning preprocessing: PCA is commonly used as a dimensionality-reduction preprocessing step in neural-network and deep-learning applications.Examples include hyperspectral classification, face recognition, and video feature extraction.
  • Hyperspectral classification: A hyperspectral pipeline compressed data with PCA, summarized neighboring regions, flattened the result, and used the resulting features as neural-network inputs.The proposed features achieved higher classification accuracy than some other methods.
  • Image analysis: PCA-based deep-learning approaches include cascaded PCAs for image classification and PCA-assisted feature generation for face recognition.PCANet was applied to handwritten-digit and object-recognition tasks, while other work combined PCA with deep learning for faces.
  • Classification outcomes: The PCA-based sleep-stage classifier was automatic but did not achieve the best accuracy among the compared approaches.It applied PCA after feature selection and used the first five principal components.
  • Economic applications: PCA is also used in economics to combine indicators into smaller sets of values for ranking, comparison, or integrated development indices.Applications include sustainability measures, macroeconomic indices, and regional economic characteristics.
  • Scientometrics: In scientometrics, PCA has been applied to university-ranking metrics, research-impact metrics, and citation-trajectory clustering.For ARWU metrics, the first and second components explained 54% and 30% of variance, respectively.

O. Astronomy

Astronomy and related spatial, environmental, and chemical applications use PCA to compress, visualize, classify, and interpret high-dimensional measurements. The examples include spectral interpretation, geographic grouping, pollution-source analysis, and forecasting-related preprocessing.

  • Astronomy: Astronomical applications of PCA include image analysis, star identification, pulsar-period determination, object classification, and spectral analysis.A pulsar waterfall diagram was analyzed with PCA to determine optimal periods.
  • Astronomy: PCA reduced astronomical input dimensionality and thereby served as a complementary tool to neural-network classification.The cited classification approach used compressed input data to reduce computational cost.
  • Spectral analysis: A type 1 Seyfert-galaxy spectrum was described by one principal component, whereas a type 2 spectrum required at least three.This comparison illustrates differing component requirements across the two spectral classes.
  • Geographical and chemical analysis: PCA projections identified four visually distinct groups of propolis samples according to their chemical measurements.The analysis used a three-dimensional PCA projection after chemical experiments.
  • Environmental applications: PCA was used with clustering and forecasting methods to analyze rainfall, urban growth, wind, solar generation, and other environmental data.Several applications treated PCA as preprocessing for reduced-dimensional inputs or pattern detection.
  • Pollution analysis: Traditional methods for identifying some soil-pollution sources were described as often unreliable or inaccurate.A PCA study of Czech soils focused on Cu, Hg, Ni, Pb, and Zn and used the first three components, accounting for 70% of variance.

T. Arts

PCA is used in arts, archaeology, social sciences, and linguistics to visualize correlated features, reduce dimensionality, classify artifacts or media, and compare cultural or textual patterns. The surveyed examples span physical measurements, chemical composition, socioeconomic indicators, networks, and literary style.

  • Arts: Arts applications use PCA to visualize correlated artistic features and to reduce measurements derived from music, paintings, and image spectroscopy.Features include musical structure, symbolism, Fourier variation, sparsity, spectra, curvatures, areas, and ratios.
  • History and archaeology: Archaeological studies apply PCA to geometric, chemical, geographic, and genetic features for dimensionality reduction, comparison, and pattern analysis.Applications include artifact composition, landscape features, and population admixture.
  • Social sciences: In economics and social sciences, PCA combines indicators, transforms categorical variables, and identifies patterns in social networks and preferences.Examples include financial-development indices, socioeconomic status, cultural interests, and attraction preferences.
  • Text analysis: Stylometric PCA projections found that the unknown work Winter’s Table was more similar to Shakespeare’s work.The cited study also used the projection to examine the consistency of The Tempest’s style with Shakespeare.
  • Text analysis: PCA can support literary-similarity analysis without serving as a discrimination method.The cited authors suggest dimensionality reduction can help quantify similarity between literary works.
  • Text analysis: PCA is also used as preprocessing for text classification after stopword removal, stemming, and tf-idf weighting.The cited application aimed to classify documents across categories with low computational cost.

B. Results and Discussion

The experiments examine PCA variance explanation and dimensionality reduction across datasets, emphasizing how standardization and feature-count normalization affect interpretation. They also assess curve fitting and broader applications, finding that PCA can preserve substantial variance with few axes while dataset characteristics alone show little influence.

  • Variance explanation: Standardized data usually retained more than 50% of dataset variance in the first three principal components, whereas non-standardized data retained 60% in the first two for most datasets.Without standardization, measurements with large values dominate the variance; in the linguistics reviews dataset, one component explained over 50% of variance.
  • Curve fitting: For standardized datasets, accumulated variance-ratio curves were reasonably fitted with an exponential function, but dataset size, class count, and feature count showed no strong correlation with the fitted parameter α.The reported Pearson correlations for number of classes, samples, and measurements were -0.27, 0.15, and -0.38, respectively.
  • Dimensionality reduction: Normalizing by the original number of features changes the relative ordering of dataset curves: high variance may require many components but only a small feature fraction.The weather (el nino) dataset lies nearest the diagonal, where variance explanation is lower.
  • Scope and applications: The work combines theoretical treatment, a systematic review of representative PCA applications, and experiments across real-world databases.The review covers applications across distinct areas, while the experiments address variance explanation and dimensionality reduction.
  • Variance explanation: Standardization can materially change variance attribution, with non-standardized data tending to yield more intense variance explanation than standardized data.The paper cautions elsewhere that standardization may be detrimental when measurement noise dominates the underlying variation.
  • Scope and applications: Overall, the reported results support PCA as a broadly applicable method for explaining variance and reducing dimensionality, while helping researchers apply and interpret it.The authors frame these resources as relevant to researchers across diverse areas.

APPENDIX A - SYMBOLS

For Gaussian data, PCA yields independent principal components and connects variance-based projection with entropy-based dimensionality reduction. The appendix derives these properties from normal-distribution transformations and covariance structure.

  • Consequences of normality: Gaussian principal components are independent because PCA diagonalizes the transformed covariance matrix, making components uncorrelated and, for Gaussian variables, independent.The transformed data remain Gaussian with covariance WΣW^T, whose diagonal form supports the independence result.
  • Motivation: Entropy measures distributional uncertainty, so maximum-entropy projections are relevant to exploratory analysis and dimensionality reduction.Maximizing entropy avoids imposing additional hypotheses and constraints on the data.
  • Entropy: A Gaussian distribution maximizes entropy among distributions with a fixed mean vector and covariance matrix.The appendix extends the scalar maximum-entropy result to the multivariate normal case.
  • Entropy: PCA preserves the entropy of a full Gaussian distribution because the covariance determinant remains invariant under the coordinate transformation.The argument uses |Cov(Y)| = |Σ| and the dependence of Gaussian entropy on the covariance determinant.
  • Entropy: Selecting principal components associated with the largest eigenvalues produces a maximum-entropy projection of normally distributed data.The retained covariance determinant is the product of the selected eigenvalues, and Gaussian entropy increases monotonically with that determinant.

APPENDIX C - BIPLOT BACKGROUND

The appendix relates PCA components and biplot quantities through eigenvectors of the correlation matrix. It describes component values as weighted combinations of original measurements and expresses eigenvector elements using correlations.

  • Each PCA component value is a linear combination of the original measurements weighted by the corresponding eigenvector.
  • The eigenvector v_i = (W_i1, . . . , W_iN) is associated with the correlation matrix C.
  • Each eigenvector element W_ij can be calculated from correlations between standardized variables and the PCA component.
Loading 1804.02502v2…