PaperlayerView exact revision on arXiv

Source-linked AI summary

Network exploration via the adaptive LASSO and SCAD penalties

Jianqing Fan, Yang Feng, Yichao Wu

arXiv:0908.2053v1stat.AP

TL;DR

Precision-matrix network estimation must address positive-definiteness constraints and bias from penalized likelihood. The paper studies SCAD and adaptive LASSO penalties, with weighted L1 optimization, and reports theoretical and empirical support for the resulting estimates.

  • Problem

    Precision-matrix network estimation uses penalized likelihood but must handle positive-definiteness constraints and bias induced by LASSO-type penalties.

  • Method

    The paper estimates precision matrices with SCAD and adaptive LASSO penalties, using adaptive weights and weighted L1 penalized likelihood optimization.

  • Results

    The SCAD penalized estimator has the oracle property, while real-data analyses report better forecasts and simpler networks for SCAD and adaptive LASSO than for LASSO.

  • Takeaways & Limitations

    SCAD and adaptive LASSO produce less complex network estimates and, in the call-center data, lower forecasting errors than LASSO.

  • Takeaways & Limitations

    The adaptive LASSO requires a consistent initial precision-matrix estimate, which the paper identifies as a drawback.

Abstract

from arXiv · show

Graphical models are frequently used to explore networks, such as genetic networks, among a set of variables. This is usually carried out via exploring the sparsity of the precision matrix of the variables under consideration. Penalized likelihood methods are often used in such explorations. Yet, positive-definiteness constraints of precision matrices make the optimization problem challenging. We introduce nonconcave penalties and the adaptive LASSO penalty to attenuate the bias problem in the network estimation. Through the local linear approximation to the nonconcave penalty functions, the problem of precision matrix estimation is recast as a sequence of penalized likelihood problems with a weighted $L_1$ penalty and solved using the efficient algorithm of Friedman et al. [Biostatistics 9 (2008) 432--441]. Our estimation schemes are applied to two real datasets. Simulation experiments and asymptotic theory are used to justify our proposed methods.

1. Introduction.

The introduction frames sparse precision-matrix estimation as a way to explore conditional dependence networks, while highlighting optimization and bias challenges. It positions penalized likelihood, graphical lasso, SCAD, and adaptive LASSO as relevant approaches to these challenges.

  • Network modeling: Sparse precision-matrix estimation represents conditional dependence networks by identifying nonzero and null entries linked to graph edges.The precision matrix is the inverse covariance matrix, and its structure is used to summarize relationships among variables.
  • Existing approaches: Penalized likelihood can combine model selection and parameter estimation, avoiding the instability of separate two-step procedures.Earlier approaches first selected a model and then estimated its parameters, while penalized likelihood performs both simultaneously.
  • Existing approaches: The graphical lasso uses coordinate descent to estimate sparse inverse covariance matrices with an L1 penalty and is reported to be remarkably fast.Related work also includes semidefinite-programming methods that obey the precision matrix’s positive-definiteness constraint.
  • Challenges: Positive-definiteness constraints make penalized-likelihood optimization difficult and prevent direct application of LARS to precision-matrix estimation.LARS computes the L1-penalized regression solution path efficiently, but does not apply when precision-matrix parameters must satisfy positive-definiteness.
  • Proposed direction: LASSO can produce coefficient bias because its penalty increases linearly, motivating SCAD and adaptive LASSO penalties for precision-matrix estimation.The paper applies local linear approximation to nonconcave penalties, converting the problem into a sequence of weighted L1 penalized-likelihood problems.
  • Paper scope: The paper applies its methods to two real datasets, compares penalty functions through Monte Carlo simulation, and develops asymptotic theory for SCAD and adaptive LASSO.The stated applications involve telephone call-center data and breast-cancer pCR development data.

2. Methods.

The methods estimate sparse precision matrices through penalized likelihood, using SCAD and adaptive LASSO penalties to reduce bias. Local linear approximation converts nonconcave penalties into iteratively reweighted L1 problems solvable by graphical lasso, while cross-validation selects tuning parameters.

  • Penalized likelihood estimation: The sparse precision-matrix estimate solves a penalized likelihood optimization problem with element-specific tuning parameters.The likelihood is expressed using the sample covariance matrix and the trace product tr(Σ̂Ω).
  • Nonconcave penalties: SCAD reduces large-coefficient bias by using a concave penalty that levels off in the tail; the numerical analyses fix a = 3.7.Unlike the L1 penalty, SCAD is designed to avoid substantial bias for large coefficients.
  • Adaptive LASSO: Adaptive LASSO applies weighted L1 penalties with weights w_ij = 1/|ω̃_ij|^γ, using a consistent initial precision-matrix estimate.The analysis fixes γ = 0.5; for p < n, the initial estimate may be the inverse sample covariance matrix, whereas for p ≥ n it may be an L1-based estimate.
  • Iterative reweighted penalized likelihood: Local linear approximation replaces SCAD locally with a symmetric linear function, yielding a sequence of weighted L1-penalized likelihood problems.The weights depend on the penalty derivative and current estimate magnitude: larger estimated magnitudes receive smaller weights.
  • Iterative reweighted penalized likelihood: Each local optimization can be solved by Friedman, Hastie, and Tibshirani’s graphical lasso algorithm, while the penalized likelihood objective increases across LLA iterations.SCAD can allow an element estimated as zero to become nonzero in a later iteration, unlike adaptive LASSO’s progressively sparser updates.
  • Tuning parameter selection: K-fold cross-validation selects λ by grid-searching the value that maximizes the cross-validation score.The selected λ is then used to obtain the final precision-matrix estimate from the full training sample.

3. Application to real data.

The paper applies LASSO, adaptive LASSO, and SCAD precision-matrix estimation to telephone-call forecasting and breast-cancer pCR classification. Across these applications, adaptive LASSO and SCAD produce simpler networks, while SCAD gives the best call-center forecasting performance and adaptive LASSO and SCAD improve selected classification measures over LASSO.

  • Telephone call center data: The call-center analysis estimates a 102×102 precision matrix from 239 daily arrival-pattern observations using three penalty functions.The first 205 days form the training set and 34 days the testing set.
  • Telephone call center data: Penalized estimates forecast better than the sample covariance estimate; SCAD performs best, followed by adaptive LASSO and LASSO.SCAD also produces the smallest forecasting errors and a simpler network than the other penalized estimates.
  • Telephone call center data: SCAD permits a larger penalty parameter and sets more spurious zeros to zero, a pattern also observed in simulations.This provides the paper’s stated explanation for the simpler SCAD network.
  • Breast cancer data: The breast-cancer analysis repeatedly divides the data into 109 training and 21 testing subjects, estimates penalized precision matrices, and evaluates specificity, sensitivity, and MCC.Tuning parameters are selected by six-fold cross-validation, and the sample covariance is unusable because sample size is smaller than dimensionality.
  • Breast cancer data: Adaptive LASSO and SCAD improve specificity and MCC over LASSO while maintaining similar sensitivity and producing simpler models.Their resulting gene networks also show small clusters indicating block diagonal precision matrices.

4. Monte Carlo simulation.

The Monte Carlo study compares LASSO, adaptive LASSO, and SCAD for precision-matrix estimation across sparse, dense-near-zero, and high-dimensional settings. Performance is assessed using estimation losses, errors in identifying zero elements, and recovered sparsity patterns.

  • Simulation settings: The simulations cover p = 30 classical settings and p = 200 high-dimensional settings, including tridiagonal, general sparse, and exponential-decay precision matrices.Examples 4.1–4.3 use p = 30; Examples 4.4–4.5 use p = 200 with n = 120.
  • Evaluation criteria: The study compares the three penalties using entropy loss, quadratic loss, and errors measuring incorrect zero and nonzero estimates.The reported zero1 and zero2 measures correspond to type-I and type-II errors for zero-element recovery.
  • Sparsity recovery: Average sparsity patterns are computed from relative frequencies of nonzero estimates across 100 repetitions and compared with the true pattern.Figures 3–6 place the true pattern in panel A and the LASSO, adaptive LASSO, and SCAD recoveries in panels B, C, and D.
  • Simulation findings: Across the examples, LASSO generally produces more nonzero precision-matrix estimates than adaptive LASSO and SCAD, while adaptive LASSO produces the sparsest pattern.The paper attributes the latter pattern to the specific choice of the adaptive LASSO initial estimate.
  • Simulation findings: Tables 3–7 show improvements for adaptive LASSO and SCAD over LASSO in both loss functions and in the two error measures for zero elements.The improvement is reported as especially pronounced for the second loss type.

5. Theoretical properties.

The paper establishes theoretical guarantees for its LLA-based penalized precision-matrix estimators. Under stated Gaussian and fixed-dimensional assumptions, SCAD and adaptive LASSO estimators have oracle-related properties, while the LLA objective increases at each iteration.

  • Overall theory: The theoretical section provides justification for both the LLA iterations and the oracle properties of the SCAD and adaptive LASSO approaches.The supplied section also identifies asymptotic normality among the reported theoretical conclusions.
  • LLA guarantee: The LLA algorithm increases the penalized log-likelihood at every iteration for differentiable concave penalties.This provides a monotonicity guarantee for the iterative approximation procedure.
  • SCAD theory: For Gaussian observations, the SCAD-penalized precision-matrix optimizer has the oracle property when λ tends to zero under the theorem’s rate conditions.The theorem assumes i.i.d. observations from N(0,Σ0) and uses the sample covariance defined in the paper.
  • SCAD theory: Asymptotically, the SCAD estimator has the same sparsity pattern as the true precision matrix.This is the model-selection component of the stated oracle property.
  • Adaptive LASSO theory: The adaptive LASSO also has the oracle property when its weights use an n-consistent initial estimator and weights of the form w_ij = 1/|˜ω_ij|^γ for γ > 0.The result is stated for the paper’s fixed-p asymptotic setting as n →∞.

APPENDIX

The appendix supplies proofs for the theoretical results by verifying likelihood regularity and concavity conditions. It derives the Gaussian precision-matrix likelihood in trace form and establishes concavity of the log-determinant.

  • Likelihood derivation: The Gaussian density yields the precision-matrix log-likelihood used in the paper, up to an additive constant.The derivation uses tr(·) for the trace operator.
  • Regularity conditions: The appendix verifies the score condition at the true precision matrix using Ω0 = Σ0^-1 and covariance-matrix identities.The argument distinguishes diagonal and off-diagonal elements before concluding that the required expectation condition holds.
  • Regularity conditions: The required likelihood condition follows because the model has common support and is identifiable.This completes the stated verification of condition (A).
  • Concavity proof: Concavity of h(Ω) = log det Ω is established by restricting the function to arbitrary lines Ω = Z + tV and showing g′′(t) ≤ 0.The line is restricted to values of t for which Z + tV remains positive definite.
  • Regularity conditions: The appendix notes that the third-order derivative condition is easy to satisfy because it does not involve x.This addresses another regularity condition in the proof framework.

SUPPLEMENTARY MATERIAL

The supplementary material contains a detailed proof of the adaptive LASSO oracle-properties theorem.

  • Adaptive LASSO proof: The supplementary article provides the detailed proof of the adaptive LASSO oracle properties stated in Theorem 5.3.The cited supplement is identified by DOI 10.1214/08-AOAS215SUPP.

Y. Wu Department of Statistics North Carolina State University Raleigh, North Carolina 27695 USA

This section identifies Y. Wu’s affiliation as the Department of Statistics at North Carolina State University in Raleigh, North Carolina.

  • Affiliation: Y. Wu is affiliated with the Department of Statistics, North Carolina State University, Raleigh, North Carolina 27695, USA.
Loading 0908.2053v1…