PaperlayerView exact revision on arXiv

Source-linked AI summary

High-Dimensional Inference: Confidence Intervals, $p$-Values and R-Software hdi

Ruben Dezeure, Peter Bühlmann, Lukas Meier, Nicolai Meinshausen

arXiv:1408.4026v2stat.ME

TL;DR

High-dimensional point estimation has advanced faster than uncertainty-aware frequentist inference, motivating methods for valid p-values and confidence intervals. The paper reviews these methods, compares them empirically, and introduces the hdi R package; stable procedures appear more reliable for type I error control, although no method is best across all scenarios.

  • Problem

    Frequentist confidence intervals and hypothesis testing remain less developed than point estimation in settings where the number of parameters can exceed the sample size.

  • Method

    The paper selectively reviews recent inference methods, conducts a broad comparative empirical study, and presents the hdi R package for applying them.

  • Results

    Stable Ridge-based or aggregated sample-splitting procedures appear more reliable for type I error control than asymptotically power-optimal methods, with no method best across all scenarios.

  • Takeaways & Limitations

    The paper offers a collection of high-dimensional inference methods and software for researchers rather than recommending a universally superior procedure.

  • Takeaways & Limitations

    Some reviewed methods depend on noncheckable assumptions, and bootstrap approaches can suffer nonuniform convergence, super-efficiency, or poor coverage.

Abstract

from arXiv · show

We present a (selective) review of recent frequentist high-dimensional inference methods for constructing $p$-values and confidence intervals in linear and generalized linear models. We include a broad, comparative empirical study which complements the viewpoint from statistical methodology and theory. Furthermore, we introduce and illustrate the R-package hdi which easily allows the use of different methods and supports reproducibility.

1. INTRODUCTION

High-dimensional point estimation has advanced substantially, but frequentist inference for uncertainty remains less developed. The paper reviews emerging methods, compares them empirically, and provides the hdi R package for practical use.

  • Motivation: High-dimensional statistics has made major advances in point estimation, computation, and applications, while confidence intervals and hypothesis testing remain less developed.The setting allows the number of parameters to greatly exceed the sample size.
  • Contributions: The paper selectively reviews recent frequentist methods for p-values and confidence regions in high-dimensional models.The review covers methods developed for linear and generalized linear models.
  • Contributions: It presents a broad comparative empirical study because existing methods rely on noncheckable or noncomparable assumptions.The study is intended to provide additional insight into procedure reliability and performance.
  • Contributions: The hdi R package makes multiple high-dimensional inference methods available for generalized linear models and supports their practical use.The package is presented as a reproducible way to apply different procedures.
  • Empirical findings: The empirical study indicates that Ridge-based or aggregated sample-splitting procedures control type I error more reliably than asymptotically power-optimal methods.The authors do not identify one method that is nearly best across all considered scenarios.

2. INFERENCE FOR LINEAR MODELS

High-dimensional linear-model inference focuses on confidence intervals and p-values for individual coefficients, where conventional Lasso-based uncertainty quantification is difficult. The paper reviews sample-splitting and projection-based procedures, comparing their assumptions, error control, stability, and power.

  • High-dimensional linear models allow p ≫ n, making coefficient inference substantially less developed than point estimation and computation.The section considers fixed or random designs, responses, and independent mean-zero errors.
  • The paper targets confidence intervals and p-values for individual regression parameters, together with multiple-testing adjustment.Simultaneous inference can support more powerful multiple-testing adjustments than treating the goals separately.
  • The plain Lasso is unsuitable for straightforward uncertainty quantification because its high-dimensional estimator distribution is difficult to characterize and may have a point mass at zero.This noncontinuity makes standard bootstrapping and subsampling delicate, with uniform convergence difficult to achieve.
  • 2.1.1 Multi sample-splitting: Multi sample-splitting aggregates results across repeated splits to avoid the p-value lottery and yields approximately reproducible p-values with familywise error control under stated assumptions.The finite-sample guarantee allows an additional small Bδ term, while the asymptotic formulation can relax Gaussian errors through a central limit theorem.
  • Conceptual differences between the methods: Multi sample-splitting may require beta-min or weaker zonal assumptions, whereas desparsified Lasso and Ridge projection require only sparsity of the underlying regression vector.Desparsified Lasso can be asymptotically efficient under sparse inverse covariance, while Ridge projection applies to general fixed designs but sacrifices efficiency and can trade power for reliable type I error control.

2.2 hdi for Linear Models

The hdi package implements several high-dimensional inference methods and provides accessible functions for p-values and confidence intervals, with options for customization and parallel computation.

  • The hdi package implements Multi sample-splitting, Ridge projection, and desparsified Lasso methods.
  • Users can customize model selection and classical fitting procedures through the multi.split arguments model.selector and classical.fit.Default options are intended for standard usage.
  • All procedures return p-values and individual-parameter confidence intervals.Ridge and desparsified Lasso also return single-testing p-values, whereas Multi sample-splitting returns only multiple-testing corrected p-values.
  • Multiple-testing corrected p-values are provided through pval.corr, with familywise error-rate control as the default correction.
  • The desparsified Lasso can be parallelized across a user-specified number of cores because it is computationally intensive.

2.3 Other Methods

This section surveys additional high-dimensional inference procedures and discusses assumptions and limitations that affect the interpretation of their results.

  • Residual-type bootstrap methods can suffer from nonuniform convergence and super-efficiency, producing poor coverage or excessively long confidence intervals for nonzero parameters.For a true zero parameter, finite bootstrap resampling can instead yield the singleton confidence region {0}.
  • The covariance test uses the Lasso solution path to provide p-values for conditional tests in which relevant variables enter first.
  • Javanmard and Montanari’s procedure replaces the Lasso-based construction of Z with a convex-program solution to relax the design sparsity assumption.
  • The Group-bound method requires no design regularity assumption and can avoid claiming significance when a regression coefficient is not identifiable.Its main motivation is testing groups of correlated variables.
  • The methods require a correct model specification, while hidden variables require adaptations based on approaches such as EM-type estimation, low-rank methods, or FCI.
  • Methods in Section 2.1 require an identifiability condition such as compatibility, unlike the Group-bound procedure.The Group-bound method offers robustness against nonidentifiability by withholding significance claims for unidentifiable components.
  • The methods require sparsity, although weak sparsity can allow many or all coefficients to be nonzero when they are sufficiently small.

2.5 A Broad Comparison

The paper compares multiple high-dimensional inference methods across simulated linear-model settings, evaluating error control, power, and confidence-interval coverage. Performance depends strongly on sparsity and design, with no single method best across all scenarios.

  • The study compares MS-Split, Lasso-Pro, Ridge, Covtest, Jm2013, and bootstrap procedures using corrected p-values and confidence intervals.
  • Simulations use n = 100, p = 500, several covariance designs, active-set sizes s0 ∈ {3,15}, and varied coefficient configurations.RealX additionally uses Bacillus Subtilis gene-expression data with n = 71 and p = 4088.
  • Power worsens as sparsity decreases, while FWER becomes too high for several methods at s0 = 15 under Equi.corr and RealX designs.
  • For s0 = 3, Ridge controls FWER as desired across setups, whereas for s0 = 15, MS-Split performs best relative to the other methods.
  • Good error control generally accompanies lower power, while desparsified Lasso offers less conservative but reasonable FWER control when s0 = 3.Z&Z tuning improves error control, most clearly for Equi.corr.
  • Confidence intervals from MS-Split and Ridge are wider, bootstrap intervals show super-efficiency, and the smallest nonzero coefficient has very poor bootstrap coverage.Coverage of zero coefficients is decent across methods, while nonzero-coefficient coverage aligns with p-value error rates.

3. GENERALIZED LINEAR MODELS

The paper extends high-dimensional inference procedures to generalized linear models, illustrating logistic-regression implementations based on either GLM-specific methods or weighted least squares. In simulations, both approaches perform adequately overall, although Equi.corr challenges Lasso-ProG.

  • For GLMs, the paper targets confidence intervals and statistical tests for unknown parameters in settings where p ≫ n.
  • Sample splitting adapts by using ℓ1-regularized maximum likelihood for screening and low-dimensional MLE asymptotics for inference.
  • The desparsified GLM Lasso uses KKT-based debiasing to obtain an asymptotically Gaussian estimator for confidence intervals and hypothesis tests.
  • The GLM comparison varies Toeplitz and Equi.corr designs, n ∈ {100,200,400}, coefficient distributions, and coefficient-position schemes with p = 500 and s0 = 3.Each setup uses 100 simulated responses, producing 18 settings per method in each plot.
  • Both modified GLM methods and weighted squared-error methods perform adequately, but Equi.corr is challenging for Lasso-ProG.
  • The hdi package provides Ridge projection and desparsified Lasso with weighted squared-error options and supports specifying the response family.

4. HIERARCHICAL INFERENCE IN THE PRESENCE OF HIGHLY CORRELATED VARIABLES

When highly correlated variables cannot be resolved individually, hierarchical group inference tests whether clusters jointly have effects. Group-bound methods remain valid without compatibility assumptions and can retain power for correlated groups, while simulations expose trade-offs in individual-variable power and error control.

  • Highly correlated variables can make individual effects unresolvable, yielding wide confidence intervals and poor power for individual-variable procedures.
  • Hierarchical inference tests the root cluster and proceeds to child clusters only after rejecting their containing hypotheses.
  • 4.1 Group-Bound Confidence Intervals Without Design Assumptions: Group-bound confidence intervals test whether a group’s l1-norm is nonzero and remain valid without the compatibility condition.
  • 4.1 Group-Bound Confidence Intervals Without Design Assumptions: Group-bound detection has an optimal rate under a weaker group-effect compatibility condition than individual-variable detection requires.
  • 4.1 Group-Bound Confidence Intervals Without Design Assumptions: Adding highly or perfectly correlated variables to a group does not affect Group-bound power, although individual-variable power is weaker.
  • 4.4 Illustrations: For highly correlated clusters, Group-bound and hierarchical Multi sample-splitting detected groups when individual variables were no longer rejectable.
  • 4.4 Illustrations: In simulations, Ridge-based power declined with correlation, while Group-bound and Ridge-based methods maintained type I error near or within the nominal 5% level.

5. STABILITY SELECTION AND ILLUSTRATION WITH HDI

Stability selection controls the expected number of false-positive selections and can select variables, groups, graphs, or clusters. Its implementation in hdi supports a user-specified false-positive bound and customizable model selection, but theoretical control relies on exchangeability assumptions.

  • Stability selection controls the expected number of false-positive selections across variables and more general structures such as graphs or clusters.
  • For single-variable selection, the method guarantees E[V] ≤ M for a prespecified false-positive bound M.
  • Stability selection does not require a beta-min assumption for single-variable selection, but its error-control analysis relies on restrictive exchangeability.
  • The hdi implementation accepts a target such as E[V] ≤ 1 and returns selected predictors through its select element.
  • The default selector uses the first q variables entering the Lasso path, while model.selector permits a user-defined selection function.

6. R WORKFLOW EXAMPLE

The hdi workflow applies inference methods to the Riboflavin data, first testing individual variables with Ridge and then testing clustered groups with Group-bound. Individual variables were not significant, whereas one clustering-tree root group was significant.

  • The workflow investigates significance at α = 0.05 and constructs corresponding confidence intervals under a linear model.
  • Ridge projection is chosen for individual-variable inference because the workflow seeks a conservative approach with good error control.
  • No multiple-testing-corrected individual-variable p-value was at most 0.05 in the Riboflavin analysis.
  • The workflow then applies Group-bound with clustering after Ridge finds no significant individual variables.
  • Only the clustering tree’s root node was found significant, consistent with earlier studies of the same dataset.

7. CONCLUDING REMARKS

The paper reviews frequentist high-dimensional inference, compares methods empirically, and provides hdi for reproducible use. Its study suggests a reliability–power trade-off: stable procedures control type I error more reliably, while power-optimal methods are more vulnerable in difficult settings.

  • The paper surveys methods for confidence intervals and p-values in linear and generalized linear models, including methods for groups of highly correlated variables.
  • Its broad empirical study complements methodological and theoretical perspectives by comparing reliability and performance across procedures.
  • Ridge-based or aggregated sample-splitting procedures appear more reliable for type I error control, while asymptotically power-optimal methods can lose control in difficult settings.
  • The paper does not identify a method that is nearly best in every scenario and therefore provides a collection of methods through hdi.
  • Some main assumptions may be unrealistic in practice, motivating alternative routes including post-selection inference and related adjustments.

A.1 Additional Definitions and Descriptions

This section defines the compatibility condition and explains how desparsified Lasso tuning controls bias-correction error. It also describes conservative aggregation of dependent p-values.

  • Compatibility condition: The compatibility condition requires a positive compatibility constant φ0 for vectors whose inactive-coordinate l1 norm is at most three times their active-coordinate norm.The constant φ0 is explicitly called the compatibility constant.
  • Dependent p-values: For dependent p-values, the corrected minimum over empirical quantiles remains a conservative p-value, as does each fixed-γ quantile.The construction uses B valid p-values and searches over γ ∈ (γmin,1).
  • Desparsified Lasso: Under sparsity and the compatibility condition, choosing λj ≍ log(p)/n yields controlled nodewise regression and estimation-error behavior.The supplied passages state that (n−1(X(j))T Z(j))−1 = OP(1) and provide corresponding error bounds.
  • Desparsified Lasso: If s0 ≪ n1/2/log(p), the error from bias correction is asymptotically negligible.The sparsity restriction is also expressed through s0 = O((n/log(p))ξ) for 0 < ξ < 1/2.
  • Desparsified Lasso: The tuning strategy balances decreasing numerator error against a denominator that must remain bounded away from zero over a sufficiently broad λj range.It uses cross-validated Lasso residuals, estimates asymptotic variance, inflates it by 25%, and searches for the smallest admissible λj.
  • Ridge projection: For ridge projection, controlling key quantities is difficult, but fixed-design X makes them fixed and observed.The difficult quantities include maxk≠j |P−1R;jjPR:jk| and the diagonal elements ΩR;jj.

A.2 Confidence Intervals for Multi Sample-Splitting

This section constructs confidence intervals from multi-sample-splitting p-values while preserving their multiplicity correction. It also gives the corresponding single-testing interval and a bisection procedure for locating bounds.

  • Multi-sample-splitting intervals: The confidence intervals are dual to p-values from equation (2.5), so multiplicity correction is incorporated directly.The construction uses information from every sample split through the ordered corrected p-values.
  • Single-testing interval: For single testing without multiplicity adjustment, the corresponding confidence interval is obtained from the unadjusted formulation.The supplied passage distinguishes this interval from the multiplicity-adjusted construction.
  • Numerical construction: When one starting point lies inside and another outside the interval, bisection finds the intervening confidence bound.The method searches between the two starting points until locating the bound.

A.3 Weighted Squared Error Approach for General GLM

For generalized linear models, the approach first fits an l1-penalized model, then converts its local weighted least-squares problem into a form handled by linear-model inference methods.

  • Weighted squared-error approach: The adjusted response and weights are derived using iterative weighted least squares and the fitted inverse link values.The fitted quantities are defined using ˆzi = g−1((Xˆβ)i).
  • Weighted squared-error approach: The GLM procedure applies standard l1-penalized fitting before constructing a weighted least-squares problem at the l1 solution.This weighted problem is the basis for applying the linear-model methods.
  • Variance specification: The resulting weighted least-squares formulation uses the variance function V(z).V(z) is related to the variance of the response Y through the assumed generalized-linear-model distribution.
  • Inference after weighting: Linear-model methods are then applied to the weighted response and design, with ˆσε set to 1.The supplied text denotes the transformed quantities as Yw and Xw.
Loading 1408.4026v2…