PaperlayerView exact revision on arXiv

Source-linked AI summary

SCAD-penalized regression in high-dimensional partially linear models

Huiliang Xie, Jian Huang

arXiv:0903.5474v1math.ST

TL;DR

The paper addresses simultaneous variable selection and estimation in sparse partially linear models when the number of linear covariates diverges. It combines SCAD penalization with B-spline estimation of the nonparametric component and shows consistency, oracle asymptotics, and global-rate convergence under stated conditions. The results are established within a scope that requires pn < n.

  • Problem

    The paper studies simultaneous variable selection and estimation in partially linear models with a sparse coefficient vector and a diverging number of linear covariates.

  • Method

    The method applies SCAD penalization to the linear coefficients and uses B-spline bases with partial residual estimation for the nonparametric component.

  • Results

    The PLM-SCAD estimator achieves simultaneous variable-selection and estimation consistency, oracle asymptotics for nonzero coefficients, and consistency for the nonparametric component.

  • Takeaways & Limitations

    Within the stated conditions, linear-variable selection and coefficient estimation can be performed together while the nonparametric estimator retains a global consistency rate.

  • Takeaways & Limitations

    The results require pn < n; extending consistent selection and asymptotic normality to pn > n remains future research without further assumptions on the covariate structure.

Abstract

from arXiv · show

We consider the problem of simultaneous variable selection and estimation in partially linear models with a divergent number of covariates in the linear part, under the assumption that the vector of regression coefficients is sparse. We apply the SCAD penalty to achieve sparsity in the linear part and use polynomial splines to estimate the nonparametric component. Under reasonable conditions, it is shown that consistency in terms of variable selection and estimation can be achieved simultaneously for the linear and nonparametric components. Furthermore, the SCAD-penalized estimators of the nonzero coefficients are shown to have the asymptotic oracle property, in the sense that it is asymptotically normal with the same means and covariances that they would have if the zero coefficients were known in advance. The finite sample behavior of the SCAD-penalized estimators is evaluated with simulation and illustrated with a data set.

1. Introduction.

The paper studies simultaneous variable selection and estimation in partially linear models with a sparse, diverging-dimensional linear component. It uses SCAD penalization for the linear part and B-spline estimation for the nonparametric component, establishing oracle-style asymptotic behavior.

  • Model motivation: Partially linear models combine a linear relationship in X with an unspecified function g of T.This combines nonparametric flexibility with the interpretability and parsimony of linear regression.
  • Research problem: The central problem is simultaneous variable selection and estimation when the number of linear covariates diverges with sample size.The coefficient vector is assumed sparse, with many coefficients equal to zero.
  • Research gap: Earlier partially linear estimators addressed estimation with fixed-dimensional covariates, but did not generally consider penalized variable selection when dimension diverges.Existing results established asymptotic normality for β and optimal-rate convergence for g under fixed-dimensional settings.
  • Approach: The method applies the smoothly clipped absolute deviation penalty to β and partial residual estimation with B-spline bases to g.The resulting estimator is designed to select nonzero coefficients consistently while estimating the nonparametric component.
  • Main contributions: The SCAD-penalized estimator has the oracle property, while the estimator of g retains the optimal global convergence rate.The nonzero coefficient estimators are asymptotically normal with the same means and covariances as if zero coefficients were known.

2. Penalized estimation in PLM with the SCAD penalty.

The estimation procedure represents the nonparametric function with polynomial splines and profiles it out, reducing estimation of the linear coefficients to a SCAD-penalized regression on residualized data.

  • Model structure: The model treats β as a potentially sparse coefficient vector and g as an unknown function of T.The covariates are divided into important nonzero-coefficient variables and trivial zero-coefficient variables.
  • Spline approximation: Polynomial splines approximate g in a spline space whose basis representation converts function estimation into estimation of spline coefficients.The spline space uses piecewise polynomials with continuity constraints and a local B-spline basis.
  • SCAD penalty: The SCAD penalty is singular at zero and has a derivative that vanishes for sufficiently large coefficients.These properties support sparse solutions while avoiding bias for large coefficients.
  • Profile estimation: Profiling out the spline coefficients transforms the joint penalized least-squares problem into an explicit objective for the parametric coefficients.The projection matrix onto the spline-basis column space is used to construct the profile objective.
  • Estimator construction: The fitted nonparametric component is recovered from the estimated spline coefficients, with no penalty imposed directly on g.The procedure focuses penalization on variable selection in the parametric part.

3. Asymptotic properties of the PLM-SCAD estimator.

Under regularity conditions, the PLM-SCAD estimator is consistent for the linear and nonparametric components, selects zero coefficients consistently, and has oracle-style asymptotic inference for nonzero coefficients.

  • Consistency: The estimator of β is consistent under conditions controlling design identifiability, coefficient smoothness, spline approximation, and the tuning parameter.The covariate dimension cannot increase too rapidly relative to sample size and the relevant smoothness parameters.
  • Variable selection: All zero coefficients can be detected simultaneously with probability tending to one under the variable-selection conditions.The selection result requires a tuning parameter that does not converge to zero too quickly.
  • Dimensionality conditions: If pn = o(n1/3), a tuning parameter can satisfy the conditions needed for consistency and variable selection; bounded kn relaxes this to pn = o(n1/2).The latter relaxation also applies when the largest eigenvalue of the design covariance is bounded.
  • Oracle property: The nonzero coefficient estimators are asymptotically normal with the same means and covariances as an estimator knowing the zero coefficients in advance.The asymptotic distribution supports simultaneous confidence intervals for any fixed number of coefficients.
  • Nonparametric convergence: The nonparametric estimator has rate O_p((Mn/n + M_n^-s_g)^(1/2)) in the bounded-kn special case, with an optimal rate when Mn ∼ n^-1/(2s_g+1).Choosing the optimal spline dimension requires sg > 1/2.
  • Diverging sparsity: With diverging kn, the convergence rate depends jointly on kn, pn, Mn, sg, and sθ, and can take the form n^-1/2 kn + k_n^1/(4s_g+2) under appropriate conditions.The nonparametric and parametric asymptotics therefore interact when the number of important covariates grows.

4. Computation.

Computation residualizes the response and covariates against a spline basis, then applies nonconcave penalized regression and selects the tuning parameter by generalized cross-validation.

  • Residualization: Given a spline basis and partition, each column of X and Y is regressed on the basis to obtain residualized covariates and response.This implements the projection approach underlying the profile objective.
  • Optimization: The residualized data are fitted with a nonconcave SCAD-penalized regression using the minorize–maximize algorithm.This converts the original joint optimization into a computationally tractable penalized regression problem.
  • Function recovery: The fitted spline coefficients recover the estimate of g at any t in [0,1].The nonparametric estimate is evaluated through the spline basis representation.
  • Tuning: The tuning parameter λn is chosen by minimizing generalized cross-validation, while the SCAD shape parameter is fixed at a = 3.7.The paper favors GCV for computational advantages and comparable model-selection performance to cross-validation.
  • Implementation choices: The simulations fix cubic splines with m = 4 and use at most three evenly spaced interior knots.The paper notes that data-driven choices of spline order and partition may be desirable.

5. Numerical studies.

The numerical studies evaluate PLM-SCAD through simulations and a workers’ wage dataset, comparing variable selection, coefficient estimation, model error, standard errors, and nonparametric-function recovery.

  • Simulation design: The simulations use n = 100 observations, 100 datasets for each ρ = 0, 0.2, 0.5, 0.8, and compare PLM-SCAD with LS-SCAD, PLM, PLM-AIC, and PLM-LASSO.They examine g(t) = cos(t) and g(t) = cos(2πt), with β_j = j for 1 ≤ j ≤ 4 and β_j = 0 for 5 ≤ j ≤ 10.
  • Simulation design: Table 1 reports coefficient estimates, selection of zero coefficients, selection of T, and model errors, with medians and model-error standard deviations.Model errors are computed as (bβ − β)′ Cov(X)(bβ − β).
  • Simulation results: In Scenario 1, PLM-SCAD estimates the nonzero coefficients accurately, efficiently selects zero-coefficient covariates, and outperforms PLM-AIC in model errors.Across 100 simulations, 83% of zero-coefficient covariates were selected on average, while no nonzero coefficients were incorrectly treated as trivial.
  • Simulation results: In Scenario 2, misspecifying the nonlinear component with LS-SCAD produces biased nonzero-coefficient estimates, with bias increasing as covariate correlation increases.Here g(T) = cos(2πT) is difficult to approximate linearly and has variation large relative to the error term.
  • Simulation results: The sandwich standard-error estimator works well but slightly underestimates sampling variation, while PLM-SCAD estimates of g(·) remain globally close to the true curves and improve as ρ decreases.The standard-error assessment concerns Scenario 1; the nonparametric-function assessment concerns Scenario 2.
  • Real-data example: In the workers’ wage example, PLM-SCAD models wage linearly in 14 covariates while treating years of experience nonparametrically, selecting 10 of the 14 covariates.The analysis uses data from 534 workers and compares PLM-SCAD with unpenalized PLM and Lasso-penalized PLM.

6. Discussion.

The paper establishes SCAD-penalized variable selection and estimation for partially linear models with a divergent number of covariates, and extends the framework to additive nonparametric components. It also identifies a dimensionality boundary: the main results require p_n<n.

  • Discussion: The PLM-SCAD estimator simultaneously selects variables and estimates coefficients in the parametric part while consistently estimating the nonparametric component.The oracle property is established for the parametric estimator, and consistency is shown for the nonparametric estimator.
  • Discussion: B-spline basis functions fit the nonparametric component, while SCAD penalization targets variable selection in the parametric part.No penalty is imposed on the nonparametric estimator because the paper focuses on selecting variables in the parametric component.
  • Partially linear additive extension: A fully multivariate nonparametric extension can suffer from the curse of dimensionality for large d and moderate sample sizes.The paper presents the additive formulation as a more parsimonious alternative.
  • Partially linear additive extension: The partially linear additive model extends the framework from one nonparametric covariate to d additive components.Each additive component is estimated using a separate partition and spline representation, with centering constraints E[g_l(T_l)]=0.
  • Partially linear additive extension: The additive extension attains the same global consistency rate for each component under an absolutely continuous joint distribution with density bounded away from zero and infinity.The stated rate is represented as O_P(k_n M_n^-s_g).
  • Scope boundary: The theoretical results require p_n<n, leaving consistent selection and asymptotic normality for p_n>n as future research.Without additional assumptions on the covariate matrix, the regression parameter is generally not identifiable when p_n>n.

APPENDIX

The appendix develops the theoretical arguments for the PLM-SCAD estimator, using projection-based decompositions, matrix bounds, and probabilistic limit arguments. These steps establish consistency, variable-selection behavior, asymptotic normality, and the nonparametric convergence rate under the stated assumptions.

  • Proof strategy: The proofs seek properties for a global minimizer of the objective rather than the local minimizer studied in related work.The appendix connects this strategy to identifiability and convergence-rate conditions for the partially linear model.
  • Matrix control: The appendix establishes convergence of the normalized sample Gram matrix to Ξ^(n) under assumption (A1).This matrix convergence is used in subsequent control of quadratic forms and estimator behavior.
  • Matrix control: Lemma 2 bounds E[tr(X^(n)′WX^(n))] at order O(np_n).This provides a basic moment bound for the weighted design matrix.
  • Proof strategy: The proof decomposes the objective derivatives and stochastic terms into multiple components, then controls them using inequalities and assumptions (A1)–(A7).The decomposition includes terms labeled I_n1 through I_n4 and related subterms.
  • Asymptotic normality: The parametric limit argument uses a central limit theorem and Slutsky’s theorem to obtain asymptotic normality.The appendix states convergence to N(0_ι,σ²I_ι) after showing remainder terms are negligible.
  • Nonparametric rate: The nonparametric proof analyzes the pointwise estimator through an integrated squared-error decomposition and spline approximation bounds.The appendix invokes results from Stone (1985) to control the component terms.
Loading 0903.5474v1…