PaperlayerView exact revision on arXiv

Source-linked AI summary

High-dimensional regression adjustments in randomized experiments

Stefan Wager, Wenfei Du, Jonathan Taylor, Robert Tibshirani

arXiv:1607.06801v3stat.MEstat.ML

TL;DR

High-dimensional regression adjustments in randomized experiments have less well-understood guarantees than low-dimensional adjustments. The paper develops theory and cross-estimation for these settings, and finds that risk-consistent adjustments can yield efficient estimates while free-intercept adjustments yield unbiased estimates. Its results support using standard prediction-risk model selection tools for these adjustments.

  • Problem

    High-dimensional regression adjustments in randomized experiments have less well-understood guarantees than low-dimensional adjustments, with prior lasso results requiring an ultrasparsity assumption.

    Source: §1 · p. 2

  • Method

    The paper develops theory for high-dimensional regression adjustments and proposes cross-estimation for treatment effect inference using regularized regression or non-parametric machine learning.

    Source: §1 · pp. 2–3

  • Results

    Risk-consistent regression adjustments can yield efficient average treatment effect estimates, and adjustments with a free intercept yield unbiased treatment effect estimates.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Practitioners can use standard model-selection tools that control prediction error, including Mallows’ Cp or cross-validation, when choosing regression adjustments.

    Source: §1 · p. 2

    The analysis assumes centered regression adjustments, and its initial setup uses homoskedastic linear models in each arm.

    Source: §2 · p. 3, §3 · p. 4

Abstract

from arXiv · show

We study the problem of treatment effect estimation in randomized experiments with high-dimensional covariate information, and show that essentially any risk-consistent regression adjustment can be used to obtain efficient estimates of the average treatment effect. Our results considerably extend the range of settings where high-dimensional regression adjustments are guaranteed to provide valid inference about the population average treatment effect. We then propose cross-estimation, a simple method for obtaining finite-sample-unbiased treatment effect estimates that leverages high-dimensional regression adjustments. Our method can be used when the regression model is estimated using the lasso, the elastic net, subset selection, etc. Finally, we extend our analysis to allow for adaptive specification search via cross-validation, and flexible non-parametric regression adjustments with machine learning methods such as random forests or neural networks.

1 Introduction

High-dimensional regression adjustments have weaker guarantees than their low-dimensional counterparts, where they are asymptotically helpful under stated conditions. This paper broadens the high-dimensional results and develops theory and methods for inference using regression adjustments.

1 Introduction

  • Machine-learning regression adjustments designed to minimize test-set error directly induce efficient average treatment effect estimates in randomized experiments.The paper states these methods can be used out of the box, without special-case adjustments.Source: §1 · p. 1
  • Prior high-dimensional guarantees were limited: the cited lasso result established improved efficiency only under an ultrasparsity assumption.This left a gap with the low-dimensional regime, where regression adjustments are asymptotically helpful under the stated conditions.Source: §1 · p. 2
  • Any regression adjustment with a free intercept yields unbiased treatment effect estimates, and risk-consistent adjustments can yield efficient average treatment effect estimates.The results cover methods such as the lasso, elastic net, and subset selection; prediction risk determines estimate precision.Source: §1 · p. 2
  • Unlike randomized experiments, observational settings can have low-risk regression adjustments that yield severely biased treatment effect estimates.The cited observational-study approaches may use special-case procedures that trade off predictive performance to protect against bias.Source: §1 · pp. 2–3
  • The paper develops theory for regularized regression adjustments with high-dimensional Gaussian designs and introduces cross-estimation for average treatment effect inference.Cross-estimation can be paired with high-dimensional regularized regression or non-parametric machine learning methods.Source: §1 · p. 3

2 Setting and notation

The setup uses a potential-outcomes framework for randomized experiments and targets the population average treatment effect. It specifies the predictor and outcome model assumptions used for the analysis.

2 Setting and notation

  • Treatment assignment W_i is independent of pretreatment characteristics in the randomized controlled trials considered.The setup represents each observation using features X_i, outcome Y_i, and assignment W_i, with potential outcomes Y_i(1) and Y_i(0).Source: §2 · p. 3
  • Predictors follow a distribution F in R^p, and the initial model assumes homoskedastic linear outcomes in each treatment arm with mean-zero noise.The text notes that more general models are considered later.Source: §2 · p. 3
  • The target is the population average treatment effect, τ = E[Y(1) − Y(0)], with conditional average treatment effect estimation also considered.Under the homogeneous treatment effects model, τ and the sample average treatment effect coincide.Source: §2 · p. 3

3 Regression adjustments with Gaussian designs

For Gaussian designs, centered regression adjustments let the treatment-effect estimator’s distribution depend on the fitted slopes through their excess predictive error. Risk consistency yields efficient estimation, while optimally tuned ridge can improve on no adjustment even when risk consistency does not hold.

3 Regression adjustments with Gaussian designs

  • The estimator’s distribution depends on fitted slopes through excess predictive error, making low predictive error the key practical criterion for adjustment choice.The result applies to centered regression adjustments on Gaussian designs, including when p exceeds n.Source: §3 · pp. 4–5

3.1 Risk consistency and the lasso

  • Risk-consistent adjustments yield efficient treatment-effect estimates under Theorem 2’s conditions, extending inference beyond coefficient-estimation guarantees.The paper notes that efficient inference about the treatment effect can remain possible where coefficient inference is generally impossible.Source: §3.1 · p. 6
  • For k-sparse coefficients, the lasso is efficient when k log(p)/n → 0, a less restrictive regime than the earlier ultrasparse guarantee.The earlier result cited in the paper requires k ≪ √n/log(p).Source: §3.1 · p. 6

3.2 Inconsistent regression adjustments

  • Optimally tuned ridge yields asymptotically normal treatment-effect estimates when p/n has a positive finite limit and coefficients follow the stated random-effects model.The ridge result concerns a setting where regression adjustment generally reduces, but does not eliminate, excess test-set risk.Source: §3.2 · pp. 6–7
  • Ridge’s limiting variance is strictly smaller than that of the unadjusted estimator because optimally tuned ridge improves on the null model.This comparison is stated for the random-effects setting of the ridge result.Source: §3.2 · p. 7

4 Practical inference with cross-estimation

Cross-estimation combines fold-specific regression adjustments to provide unbiased treatment-effect estimates and, under stated conditions, valid inference. It also supports confidence intervals and a single-round cross-validation procedure for tuning regression adjustments.

4 Practical inference with cross-estimation

  • Cross-estimation splits data into folds, estimates each fold’s regression adjustments on the other folds, and aggregates the fold-specific treatment-effect estimates by fold size.The approach is inspired by data splitting and prior work.Source: §4 · p. 8
  • Cross-estimation yields an estimate unbiased without assumptions; under risk consistency and additional conditions, it supports valid inference, with efficiency in the homoskedastic case.With heteroskedasticity, the estimator is no longer efficient, but can still form the basis for valid inference about the average treatment effect.Source: §4 · p. 8

4.1 Confidence intervals via cross-estimation

  • The confidence intervals are asymptotically exact when regression adjustments are risk-consistent and second moments are bounded.The fold-specific estimates are asymptotically uncorrelated, and the variance estimate uses observable fold-level moments.Source: §4.1 · pp. 8–9

4.2 Cross-validated cross-estimation

  • A single round of cross-validation can tune the regression adjustments while retaining cross-estimation, avoiding the computational expense of rerunning the full cross-validated algorithm for every fold.The procedure centers observations within treatment-by-fold groups, then applies a cross-validated algorithm using the same folds.Source: §4.2 · p. 9

5 Non-parametric machine learning methods

The paper extends cross-estimation to non-parametric regression adjustments, including machine-learning estimates of conditional outcome means. Under jackknife compatibility and risk consistency, the estimator is asymptotically unbiased and efficient, with asymptotically standard Gaussian inference under finite second moments.

5 Non-parametric machine learning methods

  • The estimator uses regression predictions that exclude the observation being evaluated; for random forests, these are out-of-bag predictions.This construction preserves randomization of treatment assignment conditional on the corresponding regression adjustment.Source: §5 · p. 10
  • Under jackknife compatibility, the non-parametric estimator is asymptotically unbiased; with risk-consistent adjustments and finite second moments, it is efficient and asymptotically Gaussian.The paper defines jackknife compatibility through convergence of the expected jackknife variance estimate for the regression estimator to zero, and verifies it for random forests.Source: §5 · pp. 10–11
  • The stated goal is to use machine learning to reduce variance in randomized-experiment treatment-effect estimates, not to estimate heterogeneous treatment effects.The paper describes this aim as more modest than work focused on treatment heterogeneity [46].Source: §5 · pp. 10–11

6 Experiments

In simulations, cross-estimation and the Bloniarz et al. method yield shorter confidence intervals than difference-in-means, while cross-estimation attains nominal coverage. In the welfare-question dataset, both cross-estimation variants achieve near-nominal coverage, and random forests improve precision more than lasso-based cross-estimation.

6 Experiments

  • The authors implement cross-estimation in the open-source R package crossEstimation, built on glmnet and randomForest.The experiments use lasso-based cross-estimation for high-dimensional linear estimation and random-forest adjustments for non-parametric estimation.Source: §6 · pp. 11–12

6.1 Simulations

  • The simulations compare lasso-based cross-estimation with difference-in-means and Bloniarz et al.'s cross-validated lasso adjustment under a Gaussian AR-ρ design.The proposed method uses joint lasso estimation and a different confidence-interval algorithm from Bloniarz et al.Source: §6.1 · p. 12
  • Cross-estimation and the Bloniarz et al. method produce substantially shorter intervals than difference-in-means; cross-estimation attains nominal coverage, unlike Bloniarz et al. in small samples.The simulation results include Figure 2's setting with p = 500 and 1,000 replications.Source: §6.1 · p. 12, Figure 2 · p. 12

6.2 Understanding attitudes towards welfare

  • The welfare-question experiment repeatedly samples n = 2000 from a preprocessed General Social Survey dataset of 28,646 respondents, using 12 regressors.The full dataset's treatment effect is described as known essentially without error.Source: §6.2 · p. 13
  • At 95% nominal coverage, difference-in-means, lasso-based cross-estimation, and random-forest cross-estimation realize 96.3%, 96.5%, and 95.3% coverage, respectively.At 99% nominal coverage, their realized rates are 99.0%, 99.0%, and 99.3%, respectively.Source: §6.2 · p. 13
  • Cross-estimation consistently reduces squared confidence-interval length by a small amount, with random forests improving precision more than lasso-based cross-estimation.The comparison uses individual realizations of the methods' variance estimates against the simple estimator.Source: §6.2 · p. 13

7 Discussion

The discussion frames specification search as a concern for empirical credibility and describes how centered data and cross-estimation can support safe search in randomized trials. It relates this strategy to design-based approaches and presents Figure 3's interval-length comparison.

7 Discussion

  • Specification search—trying many candidate methods and selecting one with a significant result—is a concern because it may reduce empirical credibility.The discussion motivates methods allowing complex model fitting without compromising statistical inference.Source: §7 · p. 13
  • The design-based paradigm fixes an observational design using features and treatment assignments before outcomes are revealed, allowing extensive design refinement without compromising analysis.Examples include matching and inverse-propensity weighting.Source: §7 · p. 13
  • The welfare-question experiment randomly assigned respondents to questions about welfare or assistance to the poor, with treatment effect defined as the change in the proportion answering yes.The data were preprocessed as described in cited work.Source: §7 · p. 13
  • Figure 3 compares random-forest and lasso-based reductions in squared confidence-interval length relative to difference-in-means, aggregated over 1,000 replications.The intervals rely on cross-estimation.Source: Figure 3 · p. 14
  • For randomized trials, class-wise centered data can support analyst specification search while preserving unbiased treatment-effect estimates under a random Gaussian design; cross-estimation extends this idea to non-Gaussian data.The analyst fits adjustments using centered features and outcomes, which are then plugged into the estimator.Source: §7 · p. 14

A Proofs

The appendix proofs use conditional independence and asymptotic inputs for the treatment-effect results, then invoke ridge-risk theory under spectral assumptions. The final proof uses consistency to establish the estimator's asymptotic variance and Gaussian limit.

A.1 Proof of Proposition 1

  • The Proposition 1 proof establishes conditional independence of three summands given arm counts and the fitted coefficient estimate.The argument conditions on n0, n1, and the fitted coefficient quantity.Source: §A.1 · p. 15

A.2 Proof of Theorem 2

  • Given the stated hypotheses, the sample proportions satisfy n0/n →p 1 − π and n1/n →p π, respectively, as sample size n grows.Source: §A.2 · p. 15

A.3 Proof of Theorem 3

  • The Theorem 3 proof assumes convergence of the covariance spectral distribution, invokes optimally tuned ridge-risk convergence, and applies that result separately to treatment and control samples.The proof specifies asymptotically optimal regularization parameters for the two arms.Source: §A.3 · pp. 15–16

A.4 Proof of Theorem 4

The proof uses regression-adjustment consistency to discard a negligible summand and rearrange the estimator into a form with the desired asymptotic variance. Under the stated regularity conditions, that form is asymptotically Gaussian.

A.4 Proof of Theorem 4

  • Regression-adjustment consistency makes the third summand decay faster than 1/√n; rearranging the first two yields the desired asymptotic variance and a Gaussian limit under the stated regularity conditions.Source: §A.4 · p. 17

A.5 Proof of Theorem 5

The proof isolates an unbiased, efficient main component of the estimator and controls the residual: jackknife compatibility makes it asymptotically unbiased, while risk consistency makes it negligible.

A.5 Proof of Theorem 5

  • Jackknife compatibility requires the expected jackknife variance estimate to converge to zero; classical estimators have a(n) = O(1/n), versus O(s/n) for subsampled forests.The passage characterizes this condition as weak.Source: §A.5 · p. 17
  • The main component of τ̂ is unbiased and efficient; the residual is asymptotically unbiased under jackknife compatibility and negligible under risk consistency.These are the proof's conditions for the residual, not a claim of exact finite-sample unbiasedness for the full estimator.Source: §A.5 · p. 18
  • Leave-one-out predictions can have treatment-dependent training sizes, so leave-two-out predictions equalize sample sizes and are independent of Wi conditional on n1.Randomization then gives E[R2] = 0, and risk consistency makes the approximation error o(1/n).Source: §A.5 · p. 18
  • The same result for adjustments not fit separately follows by applying jackknife compatibility to the leave-one-out estimators.Source: §A.5 · p. 19

B Additional simulation results

Additional simulations examine coverage for cross-validated cross-estimation across challenging designs and treatment-heterogeneity settings. Coverage appears promising across the varied simulation settings.

B Additional simulation results

Loading 1607.06801v3…