Source-linked AI summary
High-dimensional classification using features annealed independence rules
Jianqing Fan, Yingying Fan
TL;DR
High-dimensional classification can fail when noise accumulates across many features, motivating feature selection. The paper proposes FAIR, which selects features using two-sample t-statistics and chooses their number using an error bound. Theory, simulations, and real-data analyses support FAIR, including performance close to oracle-assisted selection and favorable comparisons with nearest shrunken centroids.
Problem
Using all features can make independence classification as poor as random guessing because noise accumulates in estimating population centroids.
Method
FAIR selects statistically significant features with componentwise two-sample t-statistics and applies the independence classifier to the selected subset.
Results
FAIR's classification errors are close to oracle-assisted independence classification in simulations and compare favorably with nearest shrunken centroids across reported analyses.
Takeaways & Limitations
Feature selection is necessary for high-dimensional classification, and FAIR provides a data-driven choice of selected-feature count supported by theory and empirical analyses.
Abstract
from arXiv · showhide
Classification using high-dimensional features arises frequently in many contemporary statistical studies such as tumor classification using microarray or other high-throughput data. The impact of dimensionality on classifications is poorly understood. In a seminal paper, Bickel and Levina [Bernoulli 10 (2004) 989--1010] show that the Fisher discriminant performs poorly due to diverging spectra and they propose to use the independence rule to overcome the problem. We first demonstrate that even for the independence classification rule, classification using all the features can be as poor as the random guessing due to noise accumulation in estimating population centroids in high-dimensional feature space. In fact, we demonstrate further that almost all linear discriminants can perform as poorly as the random guessing. Thus, it is important to select a subset of important features for high-dimensional classification, resulting in Features Annealed Independence Rules (FAIR). The conditions under which all the important features can be selected by the two-sample $t$-statistic are established. The choice of the optimal number of features, or equivalently, the threshold value of the test statistics are proposed based on an upper bound of the classification error. Simulation studies and real data analysis support our theoretical results and demonstrate convincingly the advantage of our new classification procedure.
5.1. Simulation study.
The simulation study evaluates FAIR under a correlated high-dimensional model with sparse, weak signals. It shows that selecting features can substantially reduce misclassification relative to using all features, while FAIR remains close to oracle-assisted selection.
- Simulation design: The simulation uses p = 4500 features, about 90 signals on average, n1 = n2 = 30 training samples, and 400 test samples per repetition.The experiment repeats this setup 100 times using a factor-based error model with within-group and between-group dependence.
- Simulation design: FAIR ranks features by the absolute two-sample t-statistics and evaluates classifiers using m features, with m ranging from 1 to 4500.The study compares t-statistic ordering with oracle ordering by decreasing absolute true mean differences.
- Classification performance: 0.0128 and 0.0020 are the minimum classification errors, compared with 0.2522 when all m = 4500 features are included.The all-feature error increases because accumulated noise eventually outweighs the signal.
- Comparison with oracle: FAIR's classification errors are close to those of the oracle-assisted independence classifier, while t-statistic ordering has somewhat larger minimum errors.The reported difference between t-statistic and oracle ordering is described as not very large.
- Classification performance: 0.4986 is the average error for independence classification after projection onto randomly chosen directions, with standard deviation 0.0318.Each direction is sampled from the (p − 1)-dimensional unit sphere before applying the Fisher discriminant to the projected data.
- Comparison with nearest shrunken centroids: 28.43 features and error 0.0216 are the nearest shrunken centroids averages, with standard deviation 0.0179; the method selects fewer features but has larger errors than FAIR.This comparison is based on 100 simulations and the feature counts selected by each procedure.
5.2. Real data analysis.
Real-data analyses compare FAIR with NSC and the independence rule without feature selection across leukemia, lung cancer, and prostate datasets. FAIR generally achieves competitive or better classification while often using fewer or more stable feature selections.
- Leukemia data: The leukemia dataset contains 7,129 genes and 72 samples, with 38 training and 34 test samples after standardization.The data comprise 47 ALL and 25 AML samples.
- Leukemia data: FAIR selected 11 genes and made 1 training and 1 test error, compared with NSC’s 21 genes, 1 training error, and 3 test errors.The authors also report FAIR outperforming an earlier unstandardized NSC analysis with 21 genes and 2 test errors.
- Repeated-split evaluation: Across 100 random splits at γ = 0.4, 0.5, and 0.6, test-error distributions were compared for FAIR, NSC, and the independence rule without feature selection.The same split procedure was applied separately to the leukemia, lung, and prostate datasets.
- Lung cancer data: The lung dataset contains 12,533 genes across 181 samples, and FAIR has smaller test error than NSC despite using five more genes in the fixed analysis.Both methods perfectly classify the training samples.
- Lung cancer data: In repeated lung-data splits, FAIR’s classification error is close to NSC’s, while FAIR selects features more stably; weak signal may require more features for optimal performance.The authors caution that estimating the largest eigenvalue becomes inaccurate when many features are used, affecting the estimate of the selected-feature number.
- Prostate cancer data: For prostate cancer, FAIR uses substantially fewer genes than NSC while attaining the same test error and a slightly larger training error; repeated-split performance is reported as better than NSC’s.The prostate training data contain around 12,600 genes, with an independent test set from a different experiment.
APPENDIX
The appendix develops the probabilistic and asymptotic arguments underlying the paper’s classification-error results. Its proofs decompose classification statistics, control noise terms, and establish convergence under stated moment, variance, dimensionality, and threshold conditions.
- Uniform variance control: For Gaussian features, diagonal variance estimates converge uniformly under Condition 1(b) and log p = o(n), with variances bounded away from zero.These controls support later feature-selection and classification arguments.
- Random-projection classification: Theorem 2 concludes that posterior error for projections onto almost all directions is no better than random guessing.The proof represents the projection direction through a Gaussian vector and uses convergence of the resulting classification statistic.
- Two-sample t-test selection: Theorem 3’s feature-selection proof controls false selections among noise features and missed selections among important features using tail bounds and moment conditions.The argument assumes nonzero signal coordinates for important features and zero mean differences for noise features.
- FAIR error bound: Theorem 5 bounds thresholded-classifier terms under conditions controlling log(p − m), the threshold b_n, and the largest complementary signal.The proof shows several remainder terms converge to zero before obtaining an upper bound for classification error.