PaperlayerView exact revision on arXiv

Source-linked AI summary

Valid post-selection inference

Richard Berk, Lawrence Brown, Andreas Buja, Kai Zhang, Linda Zhao

arXiv:1306.1059v1math.STstat.TH

TL;DR

Data-driven variable selection makes the chosen model stochastic, so classical tests and confidence intervals may lose their guarantees. The paper uses simultaneous inference across coefficients in all submodels to provide valid post-selection inference without requiring the selected model to be correct. The guarantee applies across selection procedures and is less conservative than Scheffé protection.

  • Problem

    Data-driven selection makes the analyzed model stochastic, invalidating classical inference guarantees that assume a model was chosen in advance.

    Source: §1 · pp. 1–2

  • Method

    The paper widens inference through simultaneous coverage for coefficients across all submodels, without requiring the selection procedure to be fully specified.

    Source: §1 · pp. 2–3

  • Results

    PoSI provides valid inference after any variable-selection procedure and is less conservative than Scheffé’s simultaneous inference.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The guarantee does not depend on the selected submodel being correct, so inference remains valid in wrong models.

    Source: §1 · p. 3

    The framework assumes fixed predictors, normal homoscedastic errors, and a valid error estimate; computation for p > 20 remains challenging.

    Source: §1 · p. 3, §7 · p. 27

Abstract

from arXiv · show

It is common practice in statistical data analysis to perform data-driven variable selection and derive statistical inference from the resulting model. Such inference enjoys none of the guarantees that classical statistical theory provides for tests and confidence intervals when the model has been chosen a priori. We propose to produce valid ``post-selection inference'' by reducing the problem to one of simultaneous inference and hence suitably widening conventional confidence and retention intervals. Simultaneity is required for all linear functions that arise as coefficient estimates in all submodels. By purchasing ``simultaneity insurance'' for all possible submodels, the resulting post-selection inference is rendered universally valid under all possible model selection procedures. This inference is therefore generally conservative for particular selection procedures, but it is always less conservative than full Scheffe protection. Importantly it does not depend on the truth of the selected submodel, and hence it produces valid inference even in wrong models. We describe the structure of the simultaneous inference problem and give some asymptotic results.

1. Introduction: The problem with statistical inference after model selection.

Data-driven variable selection makes the chosen model stochastic, undermining classical inference guarantees that presume the model was chosen in advance. The paper proposes simultaneous inference across submodels to support valid inference after any selection procedure, including when the selected model is wrong.

1. Introduction: The problem with statistical inference after model selection.

  • PoSI seeks valid inference after formal, informal, or post-hoc variable selection through a large-scale family-wise error guarantee.The proposal does not require the selection procedure to be fully specified.Source: §1 · p. 2
  • PoSI is less conservative than Scheffé’s simultaneous inference while accounting for selection.Source: §1 · pp. 2–3
  • Simultaneous inference is required for coefficients in every submodel because each selected coefficient is adjusted for the other included predictors, not excluded ones.The framework allows that no eligible model, including the full model, is correct.Source: §1 · p. 3
  • The framework assumes fixed predictors and normal, homoscedastic errors, with a valid error estimate available.Source: §1 · p. 3
  • Simulation-based methods provide satisfactory design-specific accuracy up to p ≈20; universal nonasymptotic upper bounds can be computed for larger p.Source: §1 · p. 4

2. Targets of inference and assumptions.

Selection can distort inference through multiplicity, while submodels raise a separate question about which coefficients inference targets. The paper adopts a submodel interpretation and permits models to be approximations, while retaining distributional assumptions for tests and confidence intervals.

2. Targets of inference and assumptions.

  • Selection favors predictors with strong apparent significance, creating a multiplicity problem that can increase type I errors as the number of candidate predictors grows.Source: §2 · p. 4

2.1. The submodel interpretation of parameters.

  • The paper targets association rather than causal interpretation: each submodel has its own parameters, and excluded predictors are not assigned zero coefficients.The full model need not have special status, since relevant predictors may be unmeasured or redundant.Source: §2 · p. 4, §2.1 · p. 5
  • A predictor’s coefficient depends on which other predictors are included, so different submodels can yield coefficients with different meanings and even opposite signs.The age and income example illustrates this distinction between marginal and income-adjusted slopes.Source: §2.1 · pp. 5–6

2.2. Assumptions, models as approximations, and error estimates.

  • First-order correctness is not assumed: the mean response need not lie in the column space of the predictor matrix, so models may serve as approximations.Source: §2.2 · pp. 6–7
  • Tests and confidence intervals still require homoscedastic, normal errors and a valid error-variance estimate independent of least-squares estimates.The authors note that these assumptions are questionable and defer approaches that avoid them.Source: §2.2 · p. 7
  • Separating model incorrectness from error estimation places n < p within the framework when σ² can be estimated validly and independently.Possible sources include replicated responses, a larger model, another dataset, or a split sample.Source: §2.2 · p. 7

3. Estimation and its targets in submodels.

Regression coefficients can change across submodels when predictors are not perfectly orthogonal, so the paper treats each submodel’s coefficients as distinct targets. It defines and interprets these targets without assuming first-order correctness.

3. Estimation and its targets in submodels.

  • Without perfect orthogonality, coefficients can vary across submodels; the paper frames this as a consequence of partial collinearity and notes that full-model coefficients can proliferate into as many as p2^p−1 submodel coefficients.Each predictor appears in 2^p−1 submodels.Source: §3 · p. 8

3.1. Multiplicity of regression coefficients.

  • The submodel target βM is defined through unbiasedness and is an estimable linear function of β, requiring only that µ = E[Y] exist, not first-order correctness.The target remains defined regardless of how well the submodel approximates µ.Source: §3.1 · pp. 8–9
  • Without first-order correctness, βj·M targets the response’s least-squares approximation in submodel M, not the true µ.Its conventional average-difference interpretation is replaced by an average difference in the response as approximated in M.Source: §3.1 · p. 9

3.2. Interpreting regression coefficients in first-order incorrect models.

  • A leverage plot of Y against the M-adjusted predictor Xj·M, with slope β̂j·M, remains valid without first-order correctness.The adjusted predictor is the residual from regressing Xj on the other predictors in M.Source: §3.2 · p. 9
  • Regression coefficients are weighted averages of case-wise slopes, an interpretation that also holds without first-order assumptions.The weights are based on squared adjusted-predictor values.Source: §3.2 · p. 9

4. Universally valid post-selection confidence intervals.

This section develops universally valid post-selection inference by controlling errors simultaneously across coefficients in allowable submodels, including restricted model universes. It also considers inference for a fixed primary predictor that the selection procedure always retains.

4.1. Test statistics with one error estimate for all submodels.

  • Selection makes both the model and its coefficient targets random; PoSI uses one valid error estimate independent of submodels, avoiding submodel-specific estimates with unknown distributions under misspecification.A common error estimate supports marginal t-based inference when the submodel was not selected; its t-ratio has a central t-distribution under the stated assumptions.Source: §4.1 · p. 10, §4.2 · p. 11

4.3. Post-selection coverage guarantees for confidence intervals.

  • PoSI uses one procedure-independent constant for family-wise coverage of selected predictors, but gives no guarantee for deselected predictors and bypasses procedure-specific constants.The smallest constant satisfying the coverage guarantee can depend on the selection procedure; PoSI instead supplies a constant valid across procedures.Source: §4.3 · pp. 11–12

4.4. Universal validity for all selection procedures.

  • Universal validity can protect analyses with ad hoc or multiple selection choices and establish safe rejections despite unreported data peeking.The authors also motivate this protection by noting concerns about exaggerated or unreproducible empirical findings linked to liberal variable selection.Source: §4.4 · pp. 12–13

4.5. Restricted model selection.

  • A prespecified restricted model universe can reduce conservatism and computation, including universes forcing a primary predictor into every model.PoSI1 concerns inference for such a primary predictor while selection is limited to models containing it.Source: §4.5 · pp. 13–14, §4.10 · p. 17

4.6. Reduction of universally valid post-selection inference to simultaneous inference.

  • Simultaneous family-wise inference for all parameters in all submodels yields valid post-selection guarantees for any model selection procedure.The reduction also provides strong post-selection error control.Source: §4.6 · pp. 14–15

4.7. Computation of the POSI constant.

  • For the basic all-submodels setting, the available algorithm computes PoSI constants for designs with rank(X) ≤20, or slightly higher depending on computing resources.Source: §4.7 · p. 16

4.8. Scheffé protection.

  • The PoSI constant upper-bounds Scheffé protection, while for orthogonal designs it is much smaller than the Scheffé constant.Source: §4.8 · p. 16

4.9. PoSI-sharp model selection—“SPAR.”

  • SPAR attains the significant-triviality bound and is the worst-case selection procedure requiring full PoSI protection.It selects the model containing the most significant effect.Source: §4.6 · p. 14, §4.9 · p. 17, §4.8 · p. 17

4.10. One primary predictor and controls—“PoSI1.”

  • PoSI1 restricts the model family to models containing Xj and targets inference only for that fixed primary predictor.Its relevant test statistic is |tj·M| rather than the maximum over predictors in each model.Source: §4.10 · p. 17
  • The PoSI1 constant Kj· is the quantile of a restricted max-|t| statistic and is dominated by the general PoSI constant K.The restriction to models containing Xj and inference on Xj alone makes the relevant maximum smaller than general PoSI’s.Source: §4.10 · p. 18
  • When selection always includes fixed predictor Xj, Theorem 4.2 gives a post-selection confidence guarantee for inference on that predictor.The guarantee applies to selection procedures whose output belongs to the model family containing Xj.Source: §4.10 · p. 18
  • SPAR1 attains the significant triviality bound and models analyses that choose a predictor of interest a priori to optimize its effect; it requires full PoSI1 protection.The full protection required is the PoSI1 constant Kj·.Source: §4.10 · p. 18
  • In a constructed predictor-matrix example, the PoSI1 constant grows at the Scheffé rate and asymptotically exceeds 63% of KSch, so even SPAR1 may need near-Scheffé protection.The passage says this near-Scheffé protection can be needed even for SPAR1 variable selection.Source: §4.10 · p. 18

5. The structure of the PoSI problem.

The PoSI problem can be represented in the rank-dimensional column space of the predictor matrix, where simultaneous coverage becomes a geometric problem over adjusted predictor directions. This representation exposes structural properties of those directions, the resulting polytope, and the special optimality and duality of orthogonal designs.

5. The structure of the PoSI problem.

  • PoSI coverage reduces from the original n × p representation to the d × p column-space representation, where d = rank(X).Canonical coordinates make the geometric and computational reduction possible.Source: §5.1 · p. 19

5.1. Canonical coordinates.

  • Canonical coordinates preserve submodel slope estimates while allowing the coverage problem to be analyzed in d dimensions.Translation and scale invariance also permit coverage calculations at µ = 0 and σ = 1.Source: §5.1 · p. 19

5.2. PoSI coefficient vectors in canonical coordinates.

  • In canonical coordinates, the PoSI problem is coverage for linear functions of a d-dimensional multivariate t-vector, with coefficient directions given by adjusted predictors.The unit PoSI vectors are the adjusted predictors up to scale.Source: §5.2 · p. 20

5.3. Orthogonalities of PoSI coefficient vectors.

  • PoSI vectors are orthogonal to other predictors in their submodel, and their number ranges from p2^(p−1) with no orthogonal predictor pairs to p for a fully orthogonal design.Nested submodels also yield an orthonormal Gram–Schmidt sequence of PoSI vectors.Source: §5.3 · pp. 20–21

5.4. The PoSI polytope.

  • The PoSI coverage region is an origin-symmetric polytope formed by intersecting slabs, and it contains the Scheffé ball.Its faces are tangent to that ball.Source: §5.4 · p. 21

5.5. PoSI optimality of orthogonal designs.

  • With rank d and a model universe containing a maximal nested sequence, orthogonal designs maximize coverage for fixed K and minimize K for fixed coverage.The result holds for multivariate t-vectors, their Gaussian limits, and arbitrary spherically symmetric distributions.Source: §5.5 · p. 22
  • For the Gaussian limit as d grows, the orthogonal-design asymptotic rate for K is 2log d + o(d).The result concerns X and M satisfying the conditions of Proposition 5.4.Source: §5.5 · p. 22

5.6. A duality property of PoSI vectors.

  • When the design is symmetric in canonical coordinates, replacing X by X^-1 leaves its PoSI vectors, polytope, and constant unchanged.More generally, X and its dual design pose identical PoSI problems for the full model universe.Source: §5.6 · p. 23

6. Illustrative examples and asymptotic results.

The examples study PoSI constants in the classical setting and Gaussian limit, showing that their size varies with design: exchangeable designs remain relatively close to the orthogonal case, while some designs approach the Scheffé rate. A universal asymptotic bound remains below Scheffé’s multiplier, though its attainment for actual designs is unknown.

6. Illustrative examples and asymptotic results.

  • The examples use the classical case d = p with all models considered and take the Gaussian limit, treating σ² as known.Source: §6 · p. 23

6.1. Example 1: Exchangeable designs.

  • The two exchangeable-design limits—collapse to rank 1 or rank p −1—have identical effects on K(X), by duality.As p increases, exchangeable designs approach orthogonal designs even at their most collinear extreme.Source: §6.1 · p. 24

6.2. Example 2: Where K(X) is close to the Scheffé bound.

  • For some designs, PoSI1 constants reach the Scheffé rate asymptotically, and the example shows PoSI constants can greatly exceed common-practice O(1) t-quantiles.The example’s reported lower-rate fraction is 0.6363..., with indications that the supremum over designs may exceed 0.78.Source: §6.2 · pp. 24–25

6.3. Bounding away from Scheffé.

  • A universal asymptotic upper bound for classical PoSI has a multiplier strictly below Scheffé’s, but whether actual designs attain it is unknown.The bound is sharp for unstructured sets L.Source: §6.3 · p. 26

7. Summary and discussion.

The paper establishes valid post-selection inference by simultaneous protection for coefficients across submodels, without assuming any candidate model is correct. It finds that the required constant depends strongly on the predictor matrix, while computation and several extensions remain open challenges.

7. Summary and discussion.

  • Simultaneous protection for all submodel coefficients yields valid post-selection inference without assuming any candidate linear model is correct.The framework allows arbitrary mean vectors, homoscedastic Gaussian errors, rank-deficient predictor matrices, and largely arbitrary model sets.Source: §7 · p. 26
  • The asymptotic PoSI bound varies strongly with the predictor matrix, from the orthogonal-design minimum to a large fraction of the Scheffé bound.In the classical case d = p, the authors’ computational methods are feasible up to about p ≈20.Source: §7 · p. 26
  • In the classical case d = p, the largest attained fraction of the asymptotic Scheffé rate remains unresolved, bounded below by 0.6363 and above by 0.8660.For model sets limited in size as a function of p, better rates can be achieved.Source: §7 · p. 27
  • Computations for p > 20 are challenging, and extensions to generalized linear, mixed-effects, and prediction settings remain desirable.The authors also call for procedure-specific inference when a strict selection protocol is followed.Source: §7 · p. 27

APPENDIX: PROOFS

The proof passage characterizes the vectors l_j·M through their matrix representation and orthogonality to other predictors.

APPENDIX: PROOFS

  • The vectors l_j·M are columns of (X_M^T X_M)^−1 X_M^T, so each lies in the span of the predictors in M.Source: §A.1 · p. 27
  • Each l_j·M is orthogonal to X_j′ for j′ ≠ j, as follows from X_M^T l_j·M = I_p.Source: §A.1 · p. 27
  • The passage ties the orthogonality property to the matrix identity X_M^T l_j·M = I_p.Source: §A.1 · p. 27

A.1. Proof of Proposition 5.3.

The proof establishes geometric properties of normalized vectors associated with regression coefficients across models, including orthogonality for nested model choices.

A.1. Proof of Proposition 5.3.

  • The normalized vectors inherit the stated predictor-span and orthogonality properties of l_j·M.Source: §A.1 · p. 27
  • The normalized vectors along a maximal nested sequence of submodels form an orthonormal basis of R^p.Source: §A.1 · p. 27
  • For nested models M ⊂ M′, vectors indexed by j ∈ M and j′ ∈ M′ \ M are orthogonal.Source: §A.1 · p. 27
  • For any (j0, M0), there are (p −1)2p−2 partner choices (j1, M1) yielding an orthogonal vector pair.Source: §A.1 · p. 27

A.2. Proof of duality: Lemma 5.1 and Theorem 5.1.

The proof uses orthogonality to show that corresponding coefficient vectors from paired models lie in the same one-dimensional subspace, establishing the duality result.

A.2. Proof of duality: Lemma 5.1 and Theorem 5.1.

  • The argument relies on orthogonalities described in Proposition 5.3 and proceeds by comparing vectors orthogonalized against the other predictors in each model.The proof also invokes the relation X^T X* = I_p.Source: §A.2 · pp. 27–28
  • The corresponding coefficient vectors lie in the same one-dimensional subspace, so one is a scalar multiple of the other.The proof establishes this using the models’ column spaces and orthogonalities.Source: §A.2 · pp. 27–28
  • For the normalized vectors, the proof concludes that they are equal up to sign, a result sufficient for Theorem 5.1.The lemma proves a slightly stronger statement involving lengths.Source: §A.2 · p. 28

A.3. Proof of Theorem 6.1.

The proof restricts the parameter range by duality and analyzes Gaussian maxima through bounds and order statistics. It derives the stated asymptotic conclusion using a selected model built from extreme order statistics.

A.3. Proof of Theorem 6.1.

  • By duality, it suffices to consider a ≥ 0, although the parameter a can range from −1/p to infinity.Source: §A.3 · p. 28
  • The proof derives an asymptotic expression with leading scale 2 log p for a quantity evaluated under Z ∼ N(0, I_p).The displayed order is 2 log p(1 + o_p(1)) + O_p(1).Source: §A.3 · p. 30
  • As a tends to infinity, d_a tends to zero and the squared norm of the normalized vector tends to 1 + (m − 1)^−1.Source: §A.3 · p. 30
  • The proof chooses a model containing the index of the smallest Gaussian order statistic and the largest p − m + 1 order statistics.It uses this choice to establish the desired conclusion after letting m grow arbitrarily large.Source: §A.3 · p. 30

A.4. Proof of Theorem 6.2.

The proof analyzes the maximum over models containing a primary predictor by expressing the bound through ordered Gaussian coordinates. It establishes sharpness through models formed from extreme order statistics.

A.4. Proof of Theorem 6.2.

  • The proof examines the maximum of the absolute coefficient contrast over models containing the primary predictor.It treats the design under PoSI1 and assumes known variance σ² = 1.Source: §A.4 · p. 30
  • For fixed model size, the proof maximizes terms involving ordered Gaussian coordinates and reparametrizes model size to obtain a Riemann-sum approximation.Source: §A.4 · p. 31
  • The asymptotic bound is sharp, attained by models containing either the first or last m* order statistics as p tends to infinity and c² approaches 1.Source: §A.4 · p. 32

A.5. Proof of Theorem 6.3.

The proof establishes a uniform asymptotic worst-case bound and identifies a setting attaining it. The supplement also lists additional interpretations, guarantees, adjustments, and illustrations.

A.5. Proof of Theorem 6.3.

  • The proof establishes a uniform asymptotic worst-case bound, attained when |L_p| = ap and the vectors are independent uniform directions on the sphere.The proof derives the corresponding quantile assertions by decomposing Z into its norm and direction.Source: §A.5 · p. 32
  • The proof uses upper and lower probability bounds to establish the asymptotic result under the stated growth condition on |L_p|.The supplied passages report convergence to zero and divergence at nearly geometric speed in the respective bounds.Source: §A.5 · pp. 33–34

Supplement to “Valid post-selection inference”

  • The supplement covers full-model parameter interpretation, omitted variables bias, strong error control, conditional guarantees, and PoSI p-value adjustment.It also includes the PoSI process and figures on polytopes and simulation results.Source: Supplement to “Valid post-selection… · pp. 34–35

Related papers

Loading 1306.1059v1…