PaperlayerView exact revision on arXiv

Source-linked AI summary

"Pre-conditioning" for feature selection and regression in high-dimensional problems

Debashis Paul, Eric Bair, Trevor Hastie, Robert Tibshirani

arXiv:math/0703858v1math.STstat.TH

TL;DR

When predictors greatly outnumber observations, model selection is difficult. The paper estimates the response using predictor structure, then applies a selection procedure to that estimate; it reports that pre-conditioning can mitigate noisy-feature effects under a latent-variable model.

  • Problem

    Model selection is difficult when the number of predictors greatly exceeds the number of observations.

    Source: §1 · p. 2

  • Method

    The method estimates the response using predictor correlation structure, then applies a selection procedure such as forward stepwise selection or LASSO to the estimate.

    Source: §1 · pp. 4–5, §2 · p. 6

  • Results

    Pre-conditioning can mitigate noisy-feature effects under a latent-variable model when response-associated predictors grow more slowly than n, even if p grows much faster.

    Source: §1 · p. 5

  • Takeaways & Limitations

    The paper proposes separating response estimation from feature selection and using predictor correlation structure in the two-stage approach.

    Source: §1 · pp. 4–5

    The stated model uses Gaussian latent variables and mean-zero errors independent of the other random variables in their respective models.

    Source: §2.1 · p. 8

Abstract

from arXiv · show

We consider regression problems where the number of predictors greatly exceeds the number of observations. We propose a method for variable selection that first estimates the regression function, yielding a "pre-conditioned" response variable. The primary method used for this initial regression is supervised principal components. Then we apply a standard procedure such as forward stepwise selection or the LASSO to the pre-conditioned response variable. In a number of simulated and real data examples, this two-step procedure outperforms forward stepwise selection or the usual LASSO (applied directly to the raw outcome). We also show that under a certain Gaussian latent variable model, application of the LASSO to the pre-conditioned response variable is consistent as the number of predictors and observations increases. Moreover, when the observational noise is rather large, the suggested procedure can give a more accurate estimate than LASSO. We illustrate our method on some real problems, including survival analysis with microarray data.

1 Introduction

When predictors greatly outnumber samples, fitting a useful predictor and selecting a small feature set are both difficult. The paper proposes separating these tasks: first estimate the response using predictor correlation structure, then apply a selection method to that estimate; under a latent-variable setting, this can mitigate noisy-feature effects.

1 Introduction

  • When predictors greatly outnumber samples, model selection is difficult, and existing penalized methods seek both a good predictor and a small feature set.The introduction identifies LASSO as a widely used variable-selection method.Source: §1 · pp. 2–4
  • Separating prediction from feature selection, then using predictor correlation structure, is the paper’s proposed alternative to solving both tasks at once.The two stages first estimate the response and then apply a model-selection procedure to the pre-conditioned outcome.Source: §1 · pp. 4–5
  • Under a latent-variable setting, pre-conditioning can mitigate noisy-feature effects when response-associated predictors grow more slowly than n, even if p grows much faster.The paper describes this result for applying a selection method to the estimated response rather than the raw outcome.Source: §1 · p. 5

2 Pre-conditioning

Pre-conditioning first estimates the response using latent structure, then applies a variable-selection method to that estimate. Simulations illustrate when this helps and when its marginal-screening assumption can fail.

2 Pre-conditioning

  • The pre-conditioning procedure estimates the response using latent structure, then applies forward stepwise regression or LASSO to the estimated response.Supervised principal components is the primary initial method: select outcome-correlated features, compute their components, and regress the outcome on the first K components.Source: §2 · pp. 6–7

2.1 Example: latent variable model

  • In the latent-variable model, the important features load on an unobserved Gaussian factor, with mean-zero errors independent of the other model variables.The factor, errors, and feature-specific latent terms are assumed standard Gaussian.Source: §2.1 · p. 8

2.2 Example 1

  • With 500 features and 20 samples, correlations with the estimated latent factor distinguished the 20 relevant features from irrelevant ones.The estimated factor was the first supervised principal component.Source: §2.2 · pp. 8–9
  • Pre-conditioning helps forward stepwise selection and LASSO in Example 1; Table 1 compares good-variable counts and test errors, with LASSO outperforming forward selection.The table covers four methods across selection depths.Source: §2.2 · p. 10, Table 1 · p. 11

2.3 Example 2.

  • When a relevant predictor has zero marginal correlation but nonzero partial correlation with the outcome, LASSO selected all three good predictors first in every simulation, while pre-conditioned LASSO missed the third.Supervised principal components screened out that predictor because it was marginally independent of the outcome.Source: §2.3 · p. 12
  • Pre-conditioning assumes important predictors have substantial marginal outcome correlation, an assumption the authors say need not hold in practice.They regard it as a potentially useful working hypothesis in many practical problems.Source: §2.3 · p. 12

2.4 Example 3.

  • With 1,000 predictors and 50 samples, neither method isolated most of the 40 good predictors, but pre-conditioned LASSO found twice as many good predictors as full-model LASSO.The first 40 predictors were correlated with one another, and the outcome depended on them.Source: §2.4 · pp. 13–14

3 Examples

In kidney cancer microarray examples, using supervised-principal-component predictions as the response made selection less greedy in training and improved test-set results. The paper also adapts the approach to other outcomes and applies it to survival analysis.

3.1 Kidney cancer data

  • For tumor-grade prediction, SPC/FS and SPC/LASSO were less greedy in training and yielded higher test-set correlations than their raw-outcome counterparts.The test-set correlations were not spectacularly high, but the authors report better predictions.Source: §3.1 · pp. 14–15, Figure 2 · p. 16

3.2 Application to other regression settings

  • For outcomes beyond ordinary regression, the method changes the supervised-principal-components screening measure to an appropriate association measure, with the likelihood score statistic offered as an option.Source: §3.2 · p. 15

3.3 Survival analysis

  • For survival analysis, SPC/LASSO performed best among the compared procedures and was computationally convenient through LAR on the pre-conditioned response.Forward stepwise Cox regression tended to overfit in training, leaving nonsignificant test-set p-values; the SPC/FS procedure fit more slowly and achieved smaller test-set p-values.Source: §3.3 · pp. 15, 17
  • The authors recommend reporting genes highly correlated with the best 10–15-gene model as potential biological clues or assay surrogates.They note that a gene may be poorly measured on a microarray.Source: §3.3 · p. 17
  • Across the first 1–30 selected features, forward selection and its pre-conditioned version entered features of about the same Cox-score strength, but the pre-conditioned method selected more highly correlated genes.Source: §3.3 · p. 18

4 Asymptotic analysis

This section formulates high-dimensional feature selection under a latent-factor model and gives conditions under which the target predictor set can be identified as dimension grows.

4 Asymptotic analysis

  • The analysis combines supervised principal components with LASSO and establishes asymptotic variable-selection consistency under assumptions on predictor correlations.The section frames this result in a Gaussian linear model with a latent-factor structure.Source: §4 · p. 18

4.1 Model for X

  • Predictors follow a noisy factor model: Gaussian factors contribute structured variation, while independent Gaussian noise contributes the residual variation.The factor vectors and noise matrix are independent under the stated model.Source: §4.1 · p. 21

4.2 Model for Y

  • The response is modeled through latent components, and the section derives marginal correlations and population least-squares coefficients to distinguish feature-selection targets.The passages distinguish the target set D from B, the predictors marginally correlated with Y, and A, the nonzero population regression coefficients.Source: §4.2 · p. 22, §4.3 · pp. 22–23

4.3 Least squares and feature selection

  • Under the stated conditions, the target set A can be identified even as the dimension grows with sample size.The conditions include A being a subset of B, B being identifiable, and B being small compared with n.Source: §4.3 · p. 24

4.4 SPC as a preconditioner

The section uses supervised principal components to exploit predictor covariance structure before applying a standard regression procedure, while noting that this cannot remove all noise.

4.4 SPC as a preconditioner

  • Standard LASSO does not use the data’s covariance structure, motivating SPC pre-conditioning to improve estimation and identify A.This is framed as a possibility when the underlying covariance structure and its parameters can be estimated well.Source: §4.4 · p. 25
  • Even with perfect knowledge of the factor directions and eigenvalues, estimating the latent factors cannot eliminate predictor measurement noise.The estimation can only hope to reduce the effect of measurement noise in Y.Source: §4.4 · p. 25

4.5 Algorithm

  • The algorithm estimates SPC directions using predictors correlated with Y, projects Y onto the estimated components, then fits LASSO to the projected response.The correlation screening uses threshold τn, and LASSO uses penalty µn.Source: §4.5 · p. 26

4.6 Analysis of the projection

The section analyzes the projected response and states assumptions for consistent variable selection, including identifiability and a LASSO condition on predictor relationships.

4.6 Analysis of the projection

  • The projected response is expanded for the subsequent asymptotic analyses and described as an asymptotically unbiased regression.The supplied passage introduces a remainder vector Rn in this expansion.Source: §4.6 · p. 27

4.7 Assumptions

  • Consistency relies on identifiable sets and controlled selection bias when restricting analysis to predictors in B.The passage says excessive δn may prevent consistent variable selection; the stated assumptions also require B to be identifiable.Source: §4.7 · p. 30
  • The LASSO consistency assumptions include an irrepresentable condition for variable selection after projection.The passage attributes that name to Zhao & Yu (2006) and describes the condition as nearly necessary and sufficient for consistency when p and n grow.Source: §4.7 · p. 30

4.8 LASSO solution

This short section excerpt introduces the LASSO penalty notation and a restricted LASSO estimate, with notation following earlier work.

4.8 LASSO solution

  • The paper denotes the LASSO penalty parameter by µ.Source: §4.8 · p. 31
  • The section defines a restricted LASSO estimate of θ for later use.Source: §4.8 · p. 31
  • The notation in this section follows Meinshausen & B¨uhlmann (2006).Source: §4.8 · p. 31

4.9 Consistency of variable selection

Under the stated assumptions, SPC followed by LASSO asymptotically selects the correct predictor set. The analysis also distinguishes a restricted estimator that works over a wider range of dimensions from an unrestricted estimator with a narrower stated regime.

4.9 Consistency of variable selection

  • The restricted estimator is more reliable over a wider range of dimensions, while the unrestricted estimator can consistently select when p_n = O(n^κ) for some finite κ.Source: §4.9 · p. 31
  • The analysis assumes two independent subsamples of size n and a known number K of latent response components.The results are stated to hold uniformly over parameters satisfying assumptions A1–A8.Source: §4.9 · p. 32
  • The preliminary SPC thresholding step permits restricting the analysis to the selected set B before studying the subsequent estimator.Source: §4.9 · p. 32
  • Theorem 1 and Theorem 2 together imply that SPC/LASSO asymptotically selects the correct predictor set under the stated assumptions.Source: §4.9 · p. 33

4.10 Effect of projection

Projection limits the measurement-noise contribution in the pre-conditioned response, and under specified conditions SPC/LASSO can select variables that ordinary LASSO cannot. The section also notes a setting in which the current bound could be sharpened.

4.10 Effect of projection

  • Projection maps measurement noise Z into a K-dimensional space that, under the assumptions, also contains the important predictor components.The passage connects this projection to stable behavior of the projected-response residual.Source: §4.10 · p. 35
  • The bound may improve when the selection-bias term δ_n is smaller-order, but the paper omits the technical second-order eigenvector analysis.Source: §4.10 · pp. 36–37
  • When log p_n ≫ log n, measurement noise contributes less to the maximal empirical partial correlation for the pre-conditioned response than for raw Y.Source: §4.10 · p. 37
  • Under Proposition 5’s assumptions, SPC + LASSO can identify A with an appropriate penalty when LASSO cannot, for any penalty choice.Source: §4.10 · p. 38

5 Classification problems and further topics

For classification, the paper proposes fitting a feature-selection procedure to a transformed classifier output. In ten simulations, this increased the share of good predictors selected but did not improve test error.

5 Classification problems and further topics

  • The approach depends on having a consistent estimator for constructing the pre-conditioned outcome.Source: §5 · p. 38
  • The motivation is to separate finding a good classifier from distilling its features into a smaller, less-correlated set.Source: §5 · pp. 38–39
  • For two-class classification, the procedure applies forward stepwise selection or LASSO to a function of estimated class probabilities, with log[p̂/(1−p̂)] offered as a logical choice.Source: §5 · p. 39
  • Across 10 simulations, NSC/FS increased the number of good predictors found but did not improve test error over FS.Source: §5 · p. 39

Appendix

The paper points readers to a full version that includes the appendix and notes an arXiv archive.

Appendix

  • A full version including the appendix is available, and the paper also notes an arXiv archive.Source: Appendix · p. 41
Loading math/0703858v1…