PaperlayerView exact revision on arXiv

Source-linked AI summary

Covariate Selection for Doubly Robust Double/debiased Machine Learning Estimators for Causal Inference

Muwon Kwon, Peter M. Steiner

arXiv:2609.17238v1stat.MEstat.ML

TL;DR

In high-dimensional settings, machine-learning-based covariate modeling may implicitly rely on assumptions that warrant careful scrutiny. For doubly robust DML estimation, the paper proposes using the union of the two model-specific covariate sets. The union consistently reduces more confounding bias than separate covariate sets, post-Lasso outperforms standard Lasso, and ML-based estimation does not uniformly outperform conventional estimation.

  • Problem

    In high-dimensional settings, machine-learning-based covariate modeling may implicitly rely on assumptions that warrant careful scrutiny.

  • Method

    For doubly robust DML estimation, the paper proposes using the union of the two model-specific covariate sets.

  • Results

    The union consistently reduces more confounding bias than separate covariate sets, post-Lasso outperforms standard Lasso, and ML-based estimation does not uniformly outperform conventional estimation.

  • Takeaways & Limitations

    Causal ML performance depends not only on the algorithm but also on how selected covariates are incorporated into effect estimation.

  • Takeaways & Limitations

    The simulations use a deliberately simple and stylized data-generating process with constant treatment effects.

Abstract

from arXiv · show

High-dimensional data create challenges for causal effect estimation because identifying the covariates needed for correct model specification becomes increasingly difficult. Double/debiased machine learning (DML) facilitates the use of machine learning (ML) for causal inference by mitigating regularization and overfitting bias, but comparatively less attention has been given to covariate selection in relation to the double robustness (DR) property possessed by some DML estimators. In particular, ML-based covariate selection may result in differential covariate selection or in misspecification of both models, thereby limiting the practical utility of the DR property. To address these issues, we propose using the union of the covariates selected by the propensity score (PS) and outcome ML models to re-estimate both models. Simulation results show that using the union consistently reduces more confounding bias than using separate selected covariate sets. The results also show that ML-based estimation does not uniformly outperform conventional DR estimation, even under conditions favorable to the Lasso, and that post-Lasso reduces more confounding bias than standard Lasso. These findings demonstrate that successful use of ML for causal inference depends not only on the ML algorithm but also on how the information obtained through covariate selection is incorporated into causal effect estimation.

Introduction

High-dimensional causal estimation makes covariate selection difficult, while ML introduces regularization and overfitting concerns that DML addresses only partly. The paper studies whether union-based covariate selection preserves double robustness and reduces confounding bias in finite samples.

  • Introduction: DML mitigates regularization bias through Neyman orthogonality and overfitting bias through sample splitting, supporting causal-effect estimation with high-dimensional nuisance models.The treatment-selection and outcome models are estimated on one subset, while the causal effect is estimated on a separate subset.
  • Introduction: Double robustness ensures consistency when either the treatment-selection model or the outcome model is correctly specified, but covariate selection can undermine this protection.Differential selection can prevent the models from jointly accounting for bias-inducing covariates, and both selected models may be misspecified.
  • Introduction: The paper proposes re-estimating both ML nuisance models on the union of covariates selected separately by the propensity-score and outcome models.The union eliminates differential covariate selection and increases the likelihood that at least one model includes covariates needed for approximate correct specification.
  • Introduction: The study focuses on reducing finite-sample confounding bias rather than establishing asymptotic √n-consistency, while designing the approach to preserve that property.It addresses covariate sets but assumes ML can adequately estimate the functional forms of the two models.
  • Introduction: Post-Lasso yields a greater reduction in confounding bias than standard Lasso, and ML-based covariate selection is not uniformly preferable to conventional estimation.The paper examines when ML selection is beneficial, unnecessary, or undesirable.

Double/debiased Machine Learning

DML uses machine learning for causal inference while addressing regularization and overfitting bias. Its setup models treatment assignment and outcomes using observed confounders to estimate the ATE.

  • DML addresses regularization and overfitting bias that arise when machine-learning methods estimate causal effects.The discussion particularly emphasizes regularization bias because its mitigation is closely related to the DR property of some DML estimators.
  • The data-generating process models treatment assignment through the propensity score m(X) and outcomes through g(Z, X).X contains observed confounders satisfying unconfoundedness, while Z is binary treatment and Y is the outcome.
  • The target parameter is the average treatment effect, defined as the expected difference between predicted treatment and control outcomes.

Regularization and Overfitting Bias in ML-based Causal Inference

ML-based causal estimators face regularization bias from systematic shrinkage and overfitting bias from reusing estimation data. For an outcome-only plug-in estimator, first-order outcome-model error can prevent root-n consistency.

  • Regularization Bias: Regularization controls model complexity through penalties or restrictions, improving predictive behavior while introducing systematic estimator bias.Lasso shrinks coefficients toward zero, whereas random forests reduce prediction variance through averaging across trees and predictor subsets.
  • Regularization Bias: Regularized outcome models introduce systematic, non-zero-mean estimation error that can bias the plug-in ATE estimator.The error δg(z, X) reflects regularization-induced distortion in estimating g(z, X).
  • Regularization Bias: The plug-in ATE estimator carries outcome-model error at first order, so slower-than-n^-1/2 convergence can prevent √n-consistent estimation.The sampling-error component has mean zero and typically converges at n^-1/2, whereas the regularization component converges more slowly.

Overfitting Bias

DML combines propensity-score and outcome models to reduce the impact of their estimation errors. Its bias depends on products of model errors, while sample splitting addresses overfitting from data reuse.

  • Double Robustness: The DR-DML estimator uses an augmented inverse probability weighting form whose DR property requires at least one model to be correctly specified.Correct specification concerns both the covariates controlled for and the functional relationship between variables.
  • DML Error Structure: For example, two model errors converging at n^-1/4 yield a product converging at n^-1/2.This rate is compatible with √n-consistent estimation of the causal effect.
  • Neyman Orthogonality: Neyman orthogonality makes the estimator insensitive to small estimation errors in the propensity-score and outcome models.The paper distinguishes this local insensitivity property from double robustness.

Removing Overfitting Bias

DML removes overfitting through sample splitting and cross-fitting, but covariate selection can still undermine the practical value of double robustness. The paper therefore studies how selected covariates are re-used across both nuisance models.

  • Removing Overfitting Bias: Sample splitting estimates nuisance models and the causal effect on separate non-overlapping subsets, reducing the influence of sample-specific overfitting.Cross-fitting reverses sample roles and averages estimates so all observations contribute, recovering efficiency lost through direct sample splitting.
  • Removing Overfitting Bias: Direct sample splitting can lose substantial efficiency because the causal effect uses only part of the available data.Cross-fitting addresses this inefficiency by producing additional estimates from reversed sample roles.
  • Covariate Selection: Double robustness requires correct covariate selection as well as correct functional-form specification, but this study focuses on covariate selection.Potential functional-form misspecification of the propensity-score and outcome models is not considered.
  • Covariate Selection: Separate ML selection can create differential covariate sets or omit variables needed for both models to be correctly specified.These issues correspond to the PS and outcome models adjusting for different covariates, or both selected sets failing to contain all necessary covariates.
  • Double Post-Lasso: Double post-Lasso separately selects covariates with Lasso, then re-estimates the propensity-score model by logistic regression and outcomes by OLS.Post-Lasso removes the initial Lasso shrinkage while retaining the selected covariate sets; omitting re-estimation yields double Lasso.

Differential Covariate Selection

Differential covariate selection can undermine the double robustness of DML estimators because the propensity-score and outcome models may adjust for different or insufficient covariate sets. This problem can produce bias even when one model appears correctly specified, and is especially consequential when both models are misspecified.

  • Differential selection: Different Lasso-selected covariate sets can make the DR property fail because each model remains blind to bias-inducing misspecification in the other.The selected sets differ because covariates can have different predictive strength across treatment and outcome models.
  • Model misspecification: Differential selection can cause biased effect estimates even when one model is correctly specified, because the other model's collider or bias-amplifying misspecification remains unaddressed.The seemingly correct model does not account for problematic covariates retained or omitted by its counterpart.
  • Differential selection: A correctly specified propensity-score model may not protect against outcome-model bias amplified by conditioning on instrumental variables.The propensity-score model can remove confounding bias while remaining blind to bias amplification caused by the outcome model.
  • Model misspecification: When both models are misspecified, the DR property no longer protects against model misspecification, and selected sets may omit covariates needed for approximate specification.Whether the selected sets retain those covariates depends on Lasso performance and conditions such as sample size and approximate sparsity.

Using Union of Covariate Sets for Doubly Robust DML Estimators

The paper proposes combining covariates selected by the propensity-score and outcome models, then re-estimating both models on their union. This removes differential selection and increases the chance that at least one model is correctly specified for DR estimation.

  • Proposed union approach: The union approach re-estimates both propensity-score and outcome models using every covariate selected by either model.The union is formed after separate Lasso selection and then used in conventional regressions for both models.
  • Scope: The union's role in double selection is limited to constant-treatment-effect DGPs and is not used under heterogeneous treatment effects.The paper distinguishes its proposed union-based DR-DML strategy from the conventional role of union sets in double selection.
  • Estimation procedure: The procedure selects covariates separately with logistic and outcome Lasso models, constructs XUnion = XPS ∪ XY0 ∪ XY1, and re-estimates both models on that set.Propensity scores and potential outcomes are then predicted from the re-estimated models and plugged into the ATE estimator.
  • Proposed union approach: Using a common union set eliminates differential covariate selection and retains covariates identified as important for either model.This increases the likelihood that both models include covariates needed to remove confounding bias.
  • Double robustness: The union can preserve DR protection when separately selected sets misspecify both models but the union correctly specifies at least one.The separate procedure fails in this case, whereas union-based re-estimation creates an additional opportunity for correct specification.

Simulation

The simulation varies model specification, dimensionality, sample size, and coefficient-decay conditions to assess union-based covariate selection and related ML choices. It also examines when ML selection helps, when it may not, and whether post-Lasso improves on standard Lasso.

  • Study objectives: The simulation compares conventional DR estimation with ML-based approaches to identify conditions where ML covariate selection is beneficial, unnecessary, or undesirable.This comparison is an explicit secondary objective of the study.
  • Study objectives: Post-Lasso can reduce more confounding bias than standard Lasso by re-estimating selected models without penalization.The stated mechanism is removal of shrinkage imposed on estimated coefficients.
  • Simulation design: Coefficient decay rates r = 0.51 and 1 represent approximate sparsity that is marginally or more strongly satisfied.The coefficients are generated to decay as a function of covariate rank, allowing the study to vary Lasso-favorable conditions.

Estimation of Average Treatment Effect

The study estimates ATEs using conventional DR, double Lasso, double post-Lasso, and union-based double post-Lasso across four model-specification scenarios. The union method is the only approach satisfying the DR conditions in the scenario where both initial models are misspecified.

  • Estimation approaches: The four approaches are conventional DR, doubly robust double Lasso, double post-Lasso, and double post-Lasso using the union set.The Lasso-based approaches use separate selection, while the union variant re-estimates both models on the combined selected covariates.
  • Model-specification scenarios: The four specification scenarios vary whether the initial propensity-score and outcome covariate sets contain the variables needed for correct specification.Scenarios include both correctly specified models, one correctly specified model in either direction, and both misspecified models.
  • Estimation approaches: Double post-Lasso re-estimates conventional regressions on separately selected covariates instead of using shrunken Lasso coefficients for propensity scores and outcomes.The union variant performs the same selection but uses a common union set for both re-estimated models.
  • Model-specification scenarios: In Scenario 4, only union-based double post-Lasso satisfies the conditions required for the DR property to hold.Scenario 4 gives the propensity-score model only A and the outcome model only B, leaving both models misspecified.
  • Union-based estimation: Union-based estimation allows both models to incorporate covariates selected by the other, addressing bias-inducing misspecification from differential selection.Without the union, bias amplified by including B in the outcome model is not addressed by a propensity-score model including only A.
  • Evaluation: The simulation reports the proportion of confounding bias remaining, calculated as estimator bias divided by the original confounding bias.This metric quantifies how much confounding bias each estimator removes.

Results

Across simulation scenarios, DR-DPL(U) generally left the least confounding bias among Lasso-based approaches, while performance depended on sample size, coefficient decay, and model specification.

  • Scenario 1: Initial Sets Correctly Specify Both PS and Outcome Models: When initial covariate sets correctly specified both models, increasing sample size and coefficient decay made selection more reliable and estimates more comparable to conventional DR estimation.The supplied passages describe this pattern through Figure 2 and the accompanying discussion.
  • DR-DPL(U) consistently exhibited the smallest proportion of confounding bias remaining among Lasso-based approaches and often approached conventional DR estimation.Using the union reduced bias relative to separate selected covariate sets.
  • Post-Lasso estimation reduced more confounding bias than standard Lasso by removing coefficient shrinkage during re-estimation.Among non-union approaches, DR-DPL generally outperformed DR-DL.
  • Scenario 2: Initial Sets Correctly Specify the PS Model but Misspecify the Outcome Model: Under outcome-model misspecification, remaining confounding bias was generally larger, and the DR property depended solely on correct specification of the propensity score model.The scenario retained the broader patterns observed when both initial models were correctly specified.

Scenario 4: Initial Sets Misspecify Both PS and Outcome Models

When both initial covariate sets misspecified the propensity score and outcome models, methods without the union retained non-negligible confounding bias, whereas DR-DPL(U) substantially reduced it.

  • Using the union lets both models incorporate covariates selected by the other model, addressing bias-inducing misspecification from differential covariate selection.This construction is intended to correct omissions that remain when each model uses only its own selected set.
  • DR-DPL(U) consistently exhibited substantially less remaining confounding bias than approaches using separate selected covariate sets.Its advantage became more pronounced as sample size increased and, under some conditions, it approached or slightly outperformed conventional DR estimation.

Summary

The study examines how covariate selection affects double robustness in high-dimensional causal inference and proposes re-estimating both nuisance models with their union of selected covariates. Simulations support the union strategy, but show that ML-based estimation depends on sample size, dimensionality, and selection reliability rather than uniformly outperforming conventional DR estimation.

  • High-dimensional causal inference makes it difficult to identify covariates needed for correct propensity score and outcome-model specification, limiting the practical utility of double robustness.Separate ML procedures may select different covariate sets or omit variables required for correct specification.
  • The paper re-estimates both nuisance models using the union of covariates selected by the propensity score and outcome ML models.The union eliminates differential selection at the final specification stage while retaining variables selected by either model.
  • DR-DPL(U) consistently had the smallest remaining confounding bias among Lasso-based approaches, with larger benefits when both models were misspecified.It sometimes approached or slightly outperformed conventional DR estimation.
  • Lasso-based estimation did not uniformly outperform conventional DR estimation, even under simulation conditions favorable to Lasso-based approaches.The value of ML depended on dimensionality, sample size, and the reliability of identifying covariates needed for correct specification.
  • Post-Lasso generally retained less confounding bias than standard Lasso because it preserves selection while removing coefficient shrinkage during causal-effect estimation.The results distinguish covariate selection from subsequent estimation of the nuisance models.
  • Slower coefficient decay made selection harder because predictive information was spread across more covariates, whereas faster decay and larger samples improved reliability.Under those more favorable conditions, Lasso-based estimators increasingly approached conventional DR estimation.

Limitations

The study’s findings are bounded by deliberately simple simulation settings and by its exclusive focus on Lasso-based procedures. Broader evaluation is needed for richer data-generating processes and other machine-learning methods.

  • Simulation scope: The simulations use relatively simple functional forms, independently generated confounder sets, and controlled covariate divisions to isolate the union set’s effects.These design choices were intentional, but they do not fully reflect real-world high-dimensional data.
  • Simulation scope: Real-world settings may involve unconfoundedness or approximate-sparsity violations, nonlinear relationships, and complex dependence among covariates.Future work should evaluate the approach under more realistic data-generating processes where selection and functional-form estimation are both difficult.
  • Predictor representation: When true effects include confounder interactions or higher-order polynomials, candidate predictors may need explicit nonlinear transformations and interaction terms.Without relevant terms among the candidate predictors, Lasso-based selection may not adequately approximate the propensity-score or outcome-regression functions.
  • Machine-learning scope: The study considers only Lasso, so the proposed strategy’s generality across machine-learning methods remains untested.Tree-based ensembles would require rules for combining selected or important predictors across the propensity-score and outcome models, plus learner-specific analogues to approximate sparsity.
  • Assumptions: The effectiveness of Lasso-based procedures depends materially on approximate sparsity, so union-set benefits should be assessed jointly with the learner’s assumptions.Violations of learner-specific conditions may affect both covariate selection and the resulting doubly robust DML estimator.
  • Implications: Overall, the findings support treating covariate selection as part of DR-DML estimation rather than as separate preprocessing for the two models.The union set integrates information from both models and may increase the opportunity for the DR property to hold when model-specific sets are insufficient.
Loading 2609.17238v1…