Source-linked AI summary
When Prediction Error Is Not Enough: Evaluating Nuisance-Function Prediction for Causal Estimation
Cong Cao
TL;DR
The paper asks whether better prediction of nuisance functions also means better causal estimation. Using simulations that compare nuisance-model approaches and examine cluster-level cross-fitting, it finds that prediction performance and causal inferential performance do not always coincide.
Problem
The study asks whether better prediction of nuisance functions also means better causal estimation.
Method
The study compares ordinary least squares and generalized additive models and examines cluster-level cross-fitting when observations within clusters are dependent.
Results
XGBoost achieved the lowest RMSE among non-oracle methods, whereas DML-XGBoost provided better confidence interval coverage.
Takeaways & Limitations
Point-estimation accuracy and inferential calibration were not always aligned, so prediction performance and causal inferential performance do not always coincide.
Takeaways & Limitations
Only one cluster size and one set of variance components were considered, limiting conclusions across wider dependence structures.
Abstract
from arXiv · showhide
Prediction error is widely used to evaluate nuisance-function estimators in causal inference, but its relationship with causal estimator performance may differ across performance measures. We studied this question in a partially linear model using Monte Carlo simulations. We compared ordinary least squares (OLS), generalized additive models (GAMs), XGBoost, and Double Machine Learning with XGBoost (DML-XGBoost), evaluating nuisance-function prediction error, bias, RMSE, and 95\% confidence interval coverage. We also examined a simple joint-error measure based on the absolute cross-product of estimation errors from the exposure and outcome nuisance functions. Across the simulated settings, XGBoost had the lowest RMSE among the non-oracle methods, while DML-XGBoost generally provided better confidence interval coverage. Prediction error did not consistently track causal bias across methods and settings, and the method with the best point-estimation performance did not necessarily have the best confidence interval coverage. The joint-error measure was only weakly associated with causal bias and did not provide a useful standalone measure of causal performance. These results suggest that prediction error is useful for assessing nuisance-function estimation, but it should not be treated as a direct measure of the quality of the resulting causal estimator.
1 Introduction
The paper asks whether nuisance-function prediction error can anticipate causal-estimator performance, especially when multiple nuisance functions jointly enter the estimator. It compares predictive and causal performance across learners and dependence settings.
- Prediction error is useful for evaluating nuisance models, but its relationship with causal performance depends on the learner and causal setting.
- The study specifically asks whether prediction accuracy of fitted nuisance functions anticipates resulting causal-estimator performance.
- Multiple nuisance functions complicate evaluation because exposure- and outcome-regression errors can affect the causal estimator together.
- Orthogonality reduces leading first-order nuisance-estimation effects but does not eliminate finite-sample differences caused by nuisance estimation.
- The analysis also examines whether a simple joint nuisance-function error measure relates to finite-sample causal bias, including under clustered dependence.
- Monte Carlo simulations compare OLS, GAMs, XGBoost, and DML-XGBoost using prediction error, bias, RMSE, and 95% confidence interval coverage.
2 Methods
The study uses Monte Carlo simulations of a partially linear model to compare nuisance learners, causal performance, and inferential behavior under independent and clustered observations.
- Monte Carlo simulations compare nuisance-function prediction error, causal estimation accuracy, and inferential performance across methods and two dependence settings.
- The partially linear model includes a continuous exposure, outcome, and observed baseline covariates, with nuisance functions m0(C) and h0(C).
- Prediction error is evaluated out of sample against known data-generating nuisance functions, rather than using training-set errors.
- The joint-error measure summarizes the absolute average cross-product of exposure- and outcome-nuisance estimation errors.
- OLS, GAMs, and XGBoost estimate exposure and outcome nuisance functions, while DML-XGBoost adds cross-fitting and orthogonalization.
- The clustered scenario uses cluster-level dependence and cross-fitting, but is a secondary analysis based on one cluster size and one set of variance components.
3 Results
Across simulation scenarios, XGBoost delivered the strongest point-estimation performance among non-oracle methods, whereas DML-XGBoost generally achieved better confidence interval coverage. Prediction error and the joint nuisance-error measure did not reliably identify causal performance or bias.
- XGBoost had the lowest RMSE among the non-oracle methods across both simulation settings.
- XGBoost's RMSE was 0.0143 versus 0.0372 for DML-XGBoost in Scenario 1, and 0.0208 versus 0.0522 in Scenario 2.
- XGBoost had confidence interval coverage of 0.856 and 0.838, compared with 0.928 and 0.908 for DML-XGBoost across Scenarios 1 and 2.
- The method with the best point-estimation performance was not the method with the best inferential calibration.
- Prediction error varied in strength and direction across learners and settings, so it did not fully characterize finite-sample causal performance.
- Djoint showed weak associations with absolute causal bias and is better treated as a descriptive diagnostic than as a ranking or selection criterion.
- In the clustered setting, practical learners differed in both point-estimation accuracy and confidence interval coverage, while the oracle remained approximately unbiased with close-to-nominal coverage.
4 Discussion
The simulations show that nuisance-function prediction performance and causal inferential performance do not always coincide. Prediction error remains useful for assessing nuisance-function accuracy, but it should not be treated as a standalone measure of causal estimation quality.
- XGBoost generally achieved the best point-estimation performance, with the lowest RMSE among non-oracle methods, whereas DML-XGBoost provided better confidence interval coverage.Thus, point-estimation accuracy and inferential calibration were not aligned across methods.
- Lower prediction error was associated with better point-estimation performance for XGBoost, but not with better confidence interval coverage.
- Prediction accuracy and sample-to-sample variation describe complementary aspects of nuisance-function estimation and need not rank learning procedures identically.A learner may have low prediction error but substantial sample-to-sample variation, or similar fitted functions with systematic prediction error.
- The joint nuisance-error measure was only weakly related to absolute causal bias, with the direction of association varying across methods.It was more useful as a descriptive summary of the two nuisance-function errors than as a criterion for comparing causal estimators.
- Prediction error remains useful for assessing whether a learner estimates the underlying regression functions adequately, but it does not by itself determine the sampling distribution of the causal estimator.
- The study’s scope is limited by its constant treatment effect, simple partially linear structure, restricted clustered analysis, and inability to isolate the nuisance learner from DML-XGBoost’s estimation procedure.The clustered analysis used one cluster size and one set of variance components, while DML-XGBoost combined XGBoost with cross-fitting and an orthogonal estimating procedure.
Data and Code Availability
The study used simulated data generated from the mechanisms described in the Methods section and did not use human participant data. Simulation and analysis code are publicly available on GitHub.
- The study used simulated data generated from the data-generating mechanisms described in the Methods section.
- No human participant data were used.
- The simulation and analysis code are publicly available on GitHub.
Funding
The author received no specific funding for this work.
- The author received no specific funding for this work.