Source-linked AI summary
Statistical significance of variables driving systematic variation
Neo Christopher Chung, John D. Storey
TL;DR
The paper addresses how to identify genomic variables driving systematic variation estimated from the same data, where conventional significance tests are inflated by over-fitting. It introduces jackstraw resampling to estimate an empirical null distribution for associations with PCs or their subspaces. Simulations show accurate null p-values, and applications identify cell-cycle-regulated genes and molecularly enriched inflammatory responses in trauma data.
Problem
Genomic variables associated with PCA-derived systematic variation are difficult to test because the PCs are constructed from those same variables, inflating conventional significance measures.
Method
Jackstraw resampling replaces a small number of variables with independently permuted synthetic null variables, preserving systematic variation while estimating null association statistics.
Results
The method produced accurate null p-values in simulations and identified statistically significant cell-cycle-regulated genes and inflammatory-related gene-set enrichment in trauma data.
Takeaways & Limitations
Jackstraw extends PCA-based analysis from characterizing systematic variation to gene-level significance analyses of variables associated with latent variables.
Takeaways & Limitations
Accuracy depends on selecting the number of PCs r; setting r too small leaves systematic residual variation, so the authors recommend erring larger and using extra PCs for adjustment.
Abstract
from arXiv · showhide
There are a number of well-established methods such as principal components analysis (PCA) for automatically capturing systematic variation due to latent variables in large-scale genomic data. PCA and related methods may directly provide a quantitative characterization of a complex biological variable that is otherwise difficult to precisely define or model. An unsolved problem in this context is how to systematically identify the genomic variables that are drivers of systematic variation captured by PCA. Principal components (and other estimates of systematic variation) are directly constructed from the genomic variables themselves, making measures of statistical significance artificially inflated when using conventional methods due to over-fitting. We introduce a new approach called the jackstraw that allows one to accurately identify genomic variables that are statistically significantly associated with any subset or linear combination of principal components (PCs). The proposed method can greatly simplify complex significance testing problems encountered in genomics and can be utilized to identify the genomic variables significantly associated with latent variables. Using simulation, we demonstrate that our method attains accurate measures of statistical significance over a range of relevant scenarios. We consider yeast cell-cycle gene expression data, and show that the proposed method can be used to straightforwardly identify statistically significant genes that are cell-cycle regulated. We also analyze gene expression data from post-trauma patients, allowing the gene expression data to provide a molecularly-driven phenotype. We find a greater enrichment for inflammatory-related gene sets compared to using a clinically defined phenotype. The proposed method provides a useful bridge between large-scale quantifications of systematic variation and gene-level significance analyses.
Introduction
PCA captures systematic genomic variation from latent variables, but conventional significance testing cannot reliably identify which genomic variables drive the resulting PCs. The jackstraw addresses this gap by testing associations while accounting for PCA’s over-fitting.
- PCA aggregates variation across thousands of genomic variables to estimate manifestations of otherwise unobserved latent variables.
- Conventional PCA applications do not precisely identify which genomic variables are statistically significant drivers of the resulting PCs.
- The jackstraw tests associations between genomic variables and selected PCs, PC subsets, or linear combinations.
- The method estimates an empirical null distribution by recomputing PCs after independently permuting a few variables, accounting for measurement error and over-fitting.
- The approach is potentially applicable beyond PCA to independent component analysis, K-means clustering, and broader latent-variable or clustering methods.
Statistical Model and Approach
The paper models genomic measurements as systematic variation from latent variables and estimates that variation with principal components. It then tests which genomic variables are associated with the estimated latent-variable structure while addressing noise and over-fitting.
- Statistical model: The model decomposes row-centered expression data into latent-variable-driven systematic variation and residual error.A low-dimensional basis L(z) represents the systematic component, with rank r no greater than the number of observations.
- Association testing: Each row of B quantifies a genomic variable’s relationship with the latent-variable basis, motivating tests of H0: γ_i = 0 versus H1: γ_i ≠ 0.The analogous test uses estimated PC structure because L is unobserved.
- Statistical model: PCA estimates the latent-variable row space by extracting the top r right singular vectors from the SVD of Y.The PCs are rows of D V^T, while the right singular vectors capture the relevant row space.
- Yeast example: In the yeast example, the first two PCs capture systematic variation resembling cell-cycle regulation, while higher PCs do not capture variation of interest.The experiment measured 5,981 genes across 14 synchronized time points.
- Statistical model: Testing against estimated PCs is not equivalent to testing against the latent basis at fixed m because the estimate is noisy and depends on Y itself.This dependence creates over-fitting when hypothesis tests are performed on the same data used to estimate the PCs.
Proposed Algorithms
The jackstraw estimates significance by replacing a small number of variables with permuted null variables while preserving the data’s overall systematic variation. Repeated recalculation of PCs and comparison with null association statistics yields variable-level p-values, with extensions to PC subspaces and rotations.
- Jackstraw resampling: The jackstraw replaces s ≪ m observed rows with independently permuted synthetic null variables while preserving overall systematic variation.This design targets the over-fitting caused by computing PCs from the same variables being tested.
- Jackstraw resampling: For each resampled matrix, association statistics from the synthetic null rows are compared with the observed statistics to obtain significance measures.The procedure repeats this process B times, producing s × B null statistics.
- Algorithm: The algorithm computes observed F-statistics, permutes s rows, recomputes the PCs by SVD, and calculates null F-statistics for the permuted rows.Significant variables are identified from the resulting p-values, for example using false discovery rates.
- Tuning: Choosing s and B balances computational speed against p-value accuracy and can make significance estimates conservative when s increases.For example, s = 1 and B = 10000 yields the same number of null statistics as s = 100 and B = 100.
- Extensions: The association test generalizes to subsets, rotations, and projections of PC subspaces, including selected independent components from ICA.Other PCs can be included as adjustment variables when testing a target subset.
Results
Across simulations and genomic applications, the jackstraw produced accurate significance measures while conventional testing yielded anti-conservative results. It identified cell-cycle-associated genes and revealed stronger inflammation-related gene-set enrichment in post-trauma data than the original analysis.
- Simulation results: In a representative simulation, the conventional F-test had a double KS p-value of 9.71×10^-196, compared with 0.502 for jackstraw with s = 50.The jackstraw accounted for over-fitting caused by estimating PCs from the measured variables.
- Simulation results: Across 16 simulation scenarios, the conventional F-test consistently produced anti-conservative null p-values, whereas jackstraw p-values were accurately distributed.Scenarios varied signal shape, coefficient distribution, dataset size, and the proportion of null variables.
- Simulation results: Increasing s made jackstraw null p-values more conservative, trading reduced conservatism for greater computation when s was lowered.The null p-values were not observed to become prohibitively conservative or to unreasonably diminish power.
- Cell-cycle gene expression: The yeast analysis addressed an unwieldy composite null by estimating manifested periodic variation with PCA and testing gene associations with that variation.The composite null contains infinitely many non-periodic mean time-course trajectories.
- Cell-cycle gene expression: For yeast expression data, 84.0% of 5981 genes were associated with the top two PCs, and 2988 genes were significant at FDR ≤1%.Testing PCs separately while adjusting for the other identified 1643 genes for the first PC and 966 for the second at the same threshold.
- Post-trauma gene expression: In post-trauma data, many inflammation-related gene sets were enriched when variables were tested against the top nine PCs, whereas none was enriched in the original analysis at FDR ≤1%.MAPK Signaling was enriched for every PC except the fifth, and Innate Pathogen Detection for the first, fourth, sixth, and ninth PCs.
Discussion
The jackstraw tests associations between high-dimensional genomic variables and latent-variable manifestations while preserving systematic variation and accounting for PCA over-fitting. It extends latent-variation analysis to gene-level significance, but requires deliberate PC selection and interpretation rather than blind application.
- Method: The jackstraw replaces a small number of observed variables with independently permuted synthetic null variables while maintaining the dataset’s overall systematic variation.The resulting null distribution accounts for over-fitting in latent-variable estimation.
- Validation: Simulations showed that jackstraw produces accurate statistical significance measures, whereas conventional association tests artificially inflate significance.This contrast reflects the over-fitting induced when latent variables are estimated from the same genomic variables being tested.
- Assumptions and use: The method requires specifying r, the number of PCs capturing systematic variation; setting r too small leaves residual variation that reduces accuracy.The authors recommend using a larger r and treating additional PCs as adjustment variables when the number of significant PCs is uncertain.
- Assumptions and use: Users can test subspaces of the top r PCs while adjusting for the remaining PC subspace, because individual PCs may be uninterpretable or contain multiple signals.The authors caution against blindly applying the method to the top r PCs without considering the signals of interest.
- Method: Unlike jackknife and bootstrap procedures that typically resample observations, jackstraw resamples variables to preserve latent-variable systematic variation.This design addresses high-dimensional settings where perturbing observations would intentionally disrupt the variation of interest.
- Implications: The approach enables automatic discovery of complex sources of variation and the genomic variables driving them, extending PCA from exploration and visualization to gene-level significance analysis.The authors suggest applicability to other high-dimensional biological and data-analysis settings, including methods connected to PCA.
Tables
The figures and table evaluate jackstraw significance testing, illustrate its algorithm, and compare it with conventional testing across genomic applications and simulations.
- Yeast application: Figure 2 shows that the top two yeast-expression PCs capture 48% of total variance, while significant genes at FDR ≤1% display diverse time-course trajectories.The top PCs capture major transcriptional regulatory patterns in synchronized yeast cells.
- Significance evaluation: Figure 3 compares null p-values with Uniform(0,1), showing anti-conservative conventional F-test values and uniformly distributed jackstraw values.The red dashed line represents the Uniform(0,1) density.
- Algorithm: The jackstraw algorithm generates synthetic null variables by independently permuting a small number of rows and recalculating PCs, preserving overall systematic variation.Associations between synthetic null variables and recalculated PCs form an empirical null distribution accounting for over-fitting.
- Simulation design: Figure 5 combines four design factors into 16 simulation scenarios, using 500 independent studies per scenario to assess joint-null statistical accuracy.The joint null criterion requires null p-values to be jointly distributed as Uniform(0,1), evaluated with a KS test.
- Simulation results: Figure 6 shows downward QQ-plot displacement for conventional F-test results, whereas jackstraw results are not anti-conservative and adjust for PCA over-fitting.The plots summarize double KS test p-values across the 16 scenarios.
Supporting Information
The supporting material generalizes jackstraw testing from unrotated PCs to hypotheses involving subspaces, rotations, projections, and independent components, with an alternative delete-s approach showing anti-conservative bias.
- Subspace hypotheses: The generalized test represents null and alternative hypotheses as linear constraints on coefficient vectors, enabling tests of selected PC associations.The null space is γC = a, while the alternative is γC ≠ a.
- Subspace hypotheses: To test one PC while adjusting for another, the adjustment PC enters both constrained and unconstrained models, while the target PC enters only the unconstrained model.For two PCs, this construction yields valid jackstraw p-values in simulation, unlike the conventional F-test.
- Rotations of Principal Components: Rotating the top r PCs preserves their row space, and jackstraw can test associations with any such rotation or projection.The same rotation must be applied to observed and synthetic-null PCs during each estimation iteration.
- Rotations of Principal Components: A rotation may encode biological or clinical measurements, allowing tests of variables associated with a specific linear combination while adjusting for other rotated components.The generalized model tests the target rotated component conditionally on the remaining components.
- Independent Components: Independent components can be obtained by rotating PCs to maximize mutual statistical independence, after which jackstraw testing uses the same rotated-null substitution.The rotation is applied to both observed and synthetic-null PCs.
- Generalized Algorithm to Calculate Significance of Variables Associated with PCs: The generalized algorithm obtains PCs, applies a rotation, calculates observed F-statistics, permutes s rows, recalculates PCs, and builds empirical null statistics over B iterations.P-values are computed by comparing observed statistics with synthetic-null statistics, followed by multiple-testing procedures such as false discovery rates.
- Generalized Algorithm to Calculate Significance of Variables Associated with PCs: The framework can test unrotated PCs by setting the rotation matrix to the identity and can preserve non-target PC variation as adjustment variables.Residual permutation or bootstrap procedures can preserve systematic variation in adjustment subspaces.
- An alternative delete-s approach: The delete-s jackstraw produced anti-conservative null p-values under the joint null criterion, with greater bias as s increased.The authors attribute this to complex dependence among p-values from the disjoint variable sets.