Source-linked AI summary
De-Biased Machine Learning of Global and Local Parameters Using Regularized Riesz Representers
Victor Chernozhukov, Whitney Newey, Rahul Singh
TL;DR
Many important global and local targets are linear functionals of regression functions, but global targets are typically regular while local targets are non-regular. The paper estimates these functionals using an ℓ1-based method and establishes uniform concentration and approximate normality over large model classes.
Problem
Many important global and local targets, including treatment effects and derivatives, are linear functionals of regression functions, with global targets typically regular and local targets non-regular.
Source: §1 · p. 1Method
The method uses ℓ1 regularization to estimate the Riesz representer and averages estimated orthogonal representations in the DML estimator.
Source: §1 · p. 2, §2.5 · p. 9Results
A single non-asymptotic result covers global and local functionals and implies uniformly valid concentration rates and normal approximations over large sets of distributions.
Source: §4.3 · pp. 17–18Takeaways & Limitations
Under misspecification, the approach estimates an interpretable linear functional of the projection rather than of the regression function.
Source: §1 · p. 2Efficiency is established only for regular functionals with bounded operator norm while the distribution is held fixed; extending this result to diverging operator norms is left to future research.
Source: §4.4 · p. 19
Abstract
from arXiv · showhide
We provide adaptive inference methods, based on $\ell_1$ regularization, for regular (semi-parametric) and non-regular (nonparametric) linear functionals of the conditional expectation function. Examples of regular functionals include average treatment effects, policy effects, and derivatives. Examples of non-regular functionals include average treatment effects, policy effects, and derivatives conditional on a covariate subvector fixed at a point. We construct a Neyman orthogonal equation for the target parameter that is approximately invariant to small perturbations of the nuisance parameters. To achieve this property, we include the Riesz representer for the functional as an additional nuisance parameter. Our analysis yields weak ``double sparsity robustness'': either the approximation to the regression or the approximation to the representer can be ``completely dense'' as long as the other is sufficiently ``sparse''. Our main results are non-asymptotic and imply asymptotic uniform validity over large classes of models, translating into honest confidence bands for both global and local parameters.
1. INTRODUCTION
The paper develops adaptive inference for linear functionals of regressions, including global regular parameters and local non-regular parameters. It estimates the Riesz representer with ℓ1 regularization and studies orthogonal, debiased machine-learning inference under regularization and more general regression learners.
1. INTRODUCTION
- Neyman orthogonal scores correct plug-in bias with a residual-times-Riesz-representer term, while cross-fitting addresses overfitting bias.The score has zero derivative with respect to each first-step learner.Source: §1 · pp. 1–2
- The ℓ1-regularized method estimates the Riesz representer from empirical equations and avoids componentwise estimation, including density derivatives or propensity-score inversions.These examples illustrate cases where the method avoids explicitly estimating components even when a closed-form representer is available.Source: §1 · p. 2
- The analysis gives non-asymptotic Gaussian approximation results for debiased machine learning and treats ℓ1-penalized regression non-asymptotically and other machine-learning regressions asymptotically.The authors describe the Gaussian-approximation analysis as, to their knowledge, the first of its kind for debiased machine learning.Source: §1 · pp. 2–3
- Under misspecification, the approach targets a linear functional of the regression projection rather than the regression itself.The paper states that its current analysis focuses on linear functionals; cited follow-up work extends the approach to nonlinear functionals through linearization.Source: §1 · p. 2
- The approach covers local non-regular functionals, extending debiased machine learning beyond global objects estimable at the 1/√n rate.The paper identifies coverage of local objects estimated more slowly than 1/√n as a contribution.Source: §1 · pp. 1–2
2. OVERVIEW OF TARGET FUNCTIONALS, ORTHOGONAL REPRESENTATION, ESTIMATION, AND INFERENCE
The section develops an orthogonal representation and high-dimensional estimation and inference for global and local regular and non-regular linear functionals of a regression function, using its Riesz representer. It previews adaptive non-asymptotic inference with uniform validity over large classes of probability laws, including guarantees for perfectly localized functionals when localization bias is sufficiently small.
2.1. Target functionals
- Weighted linear functionals of the regression cover causal effects, policy effects, derivatives, and subgroup or localized targets; perfectly localized functionals are not pathwise differentiable.For a causal interpretation, the regression must be invariant to policy-induced distribution changes; the passage gives exogeneity as a sufficient condition.Source: §2.1 · pp. 3–6
2.2. Building an orthogonal representation of the target functional
- The orthogonal representation adds the Riesz representer to the regression nuisance, making the target invariant to local perturbations and yielding double robustness.This provides a basis for point and interval estimation using regularized nuisance estimates.Source: §2.2 · pp. 7–8
2.3. The case of finite-dimensional linear regression
- In finite-dimensional linear regression, the regression and minimal representer use a shared basis, with coefficient vectors defined by minimal ℓ1-norm solutions that yield direct, dual, and orthogonal representations.The paper describes these representations as novel and useful.Source: §2.3 · p. 8
2.4. The case of infinite-dimensional regression
- When the functional’s modulus of continuity L is finite, a unique minimal representer exists and has L2 norm L; the functional extends to the closure of the regression space.The minimal representer is obtained by projecting any finite-norm representer onto that closure.Source: §2.4 · p. 9
2.5. Informal preview of estimation and inference results
- The estimation preview uses generalized Dantzig selector approximations and cross-fitting; either the regression or representer can be completely dense if the other is approximately sparse.The effective sparsity dimension is required to be below √n; the dictionary dimension may exceed the sample size.Source: §2.5 · pp. 9–10
- The de-biased estimator is approximated by the oracle estimator up to o(σ/√n), yielding concentration around the target with normal-law deviations.The non-asymptotic error bound tends to zero as n grows, and the resulting guarantees are uniformly valid over large classes of probability laws P for W.Source: §2.5 · p. 10
- When σ, κ/σ, and L are bounded, the regular case yields 1/√n concentration, adaptation, and Gaussian approximation.In the non-regular case these parameters diverge, and conditions are needed for concentration, adaptation, and Gaussian approximation.Source: §2.5 · p. 10
- Learning and inference guarantees for local functionals transfer to perfectly localized functionals when localization bias is sufficiently small.Source: §2.5 · p. 11
- A single inference theory covers both regular and non-regular cases and provides uniform validity over large classes of P.Source: §2.5 · p. 11
3. APPLICATIONS
The applications estimate global and income-localized effects of 401(k) eligibility on financial assets and of gasoline prices on demand. They report heterogeneous local effects in both settings, alongside comparisons with alternative estimators and specifications.
3.1. Global and local effects of 401(k) eligibility on net financial assets
- The applications use 1991 US survey data for the 401(k) analysis and 1994–1996 Canadian vehicle-use survey data for gasoline demand.The analyses use common-support selection for the 401(k) sample and an interaction-augmented specification for gasoline demand.Source: §3.1 · p. 11, §3.2 · p. 13
- ATE is 7608 (1395) using GDS for both the Riesz representer and regression; localized ATE is positive and heterogeneous except in the second income quintile.The ATE estimate is stable across regression estimators; localized effects are statistically significant in the other quintiles.Source: §3.1 · p. 11, Table 1 · p. 12, Table 2 · p. 12
- The 401(k) ATE is broadly consistent with Chernozhukov et al. (2018a), whose DML estimate is 7170 (1398); this approach avoids an estimated propensity score in the denominator.The authors note that the two estimators are asymptotically equivalent under correct specification, while the estimated denominator could cause numerical instability.Source: §3.1 · pp. 11–12
- Without debiasing, GDS and Lasso estimates of quintile-specific 401(k) effects and gasoline elasticities are attenuated by regularization.The attenuation is smaller for random forest and neural network estimates in both applications.Source: §3.1 · p. 12, §3.2 · p. 14
3.2. Global and local price elasticity of gasoline demand
- GDS estimates an average gasoline price elasticity of −0.28 (0.06), with statistically significant negative localized elasticities in every income quintile and substantial heterogeneity.The localized findings are broadly consistent with Semenova and Chernozhukov (2021a).Source: §3.2 · pp. 13–14, Table 3 · p. 13
- OLS estimates gasoline price elasticity at 0.14 (0.06), opposite the negative sign expected from the stated economic intuition.The linear specification regresses log consumption on log price and listed covariates.Source: §3.2 · p. 13
4. ESTIMATION AND INFERENCE FOR HIGH DIMENSIONAL APPROXIMATELY LINEAR MODELS
The section develops orthogonal, cross-fitted inference for high-dimensional approximately linear models using regularized regression and Riesz-representer estimates, and establishes approximation and Gaussian-inference results under stated conditions. It also derives effective-dimension bounds for GDS estimators that support inference when one nuisance parameter may be dense, provided the other is sufficiently sparse.
4.1. Best linear approximations for the regression function and the Riesz representer
- The score is invariant to small perturbations in the regression and Riesz-representer nuisances, reducing the effect of their biased estimation on the target estimate.The representer is defined through a representation property on the dictionary span; when approximation errors are not sufficiently small, inference instead targets the penultimate parameter.Source: §4.1 · pp. 14–15
4.2. Estimators
- The estimators use generalized Dantzig selectors for the regression and Riesz representer, with cross-fitting to form and average score-based estimates.The nuisance estimates for each fold are obtained from observations outside that fold.Source: §4.2 · pp. 15–17
4.3. Properties of DML: Main result
- Under condition R(δ), DML is close to the oracle estimator and admits approximately Gaussian inference with finite-sample bounds that are uniform over model classes.The results extend to global and non-perfectly localized functionals; uniformly asymptotic oracle equivalence and Gaussianity hold when δ_n → 0 and the stated kurtosis condition is met.Source: §4.3 · pp. 17–18
- With small approximation and localization errors, the results extend to the ultimate target, including for perfectly localized functionals.Otherwise inference targets the interpretable penultimate parameter; the required second-order approximation condition is weaker in the non-regular case.Source: §4.1 · p. 15, §4.3 · p. 19
4.4. Semi-parametric efficiency
- For regular functionals with the distribution fixed, the estimator has the unique influence function and achieves the semiparametric efficiency bound among locally regular estimators.The formal efficiency result does not cover non-regular functionals with an operator norm growing with the sample size.Source: §4.4 · pp. 19–20
4.5. Properties of GDS estimators
- The GDS analysis verifies the main guarantee by bounding population prediction errors and uses a restricted set to control feasible estimation errors.The restricted set imposes both a moment-residual bound and an ℓ1-norm constraint relative to the true parameter.Source: §4.5 · p. 20
- Effective dimension measures estimation difficulty using t0 and the population covariance G, without requiring restricted-eigenvalue or cone-invertibility conditions.For G=I and an s-sparse t0, effective dimension is at most s; the concept does not split t0 into sparse and dense components.Source: §4.5 · pp. 20–21
- In the ATE example, approximate sparsity of projected outcome regressions or inverse propensity scores implies small effective dimensions for the regression or representer.The passage defines the dictionary using basis functions approximating both conditional outcome functions and inverse propensity-score functions.Source: §4.5 · pp. 21–22
- The effective dimension yields a population prediction-norm bound that is the minimum of fast- and slow-rate bounds; when effective dimension is modest, the fast rate is s(t0;ν)ν^2.The result is stated in the population prediction norm and applies under weak assumptions on G.Source: §4.5 · p. 22
- GDS estimation errors for the BLP and Riesz representer are controlled under SC, with a corollary giving sufficient conditions for the guarantee R(δ).The bounds hold with probability at least 1−K4ε; the corollary verifies R(δ) for approximately sparse models.Source: §4.5 · p. 23
- Double-sparsity robustness allows one nuisance parameter to have unbounded effective dimension and be estimated at a slow n^-1/4 rate, if the other is sufficiently sparse.The conditions are described as matching similar conditions for inference on a single coefficient in Gaussian exactly sparse linear regression.Source: §4.5 · p. 23
5. ESTIMATION AND INFERENCE USING GENERAL REGRESSION LEARNERS
The section extends the estimator to general regression learners, including learners used for local functionals, and establishes asymptotic Gaussian inference under convergence-rate conditions. It also places this approach among prior work on debiased estimation and localized functionals.
5. ESTIMATION AND INFERENCE USING GENERAL REGRESSION LEARNERS
- The section’s general-learner theory requires certain L2 convergence properties and covers local as well as regular functionals.The listed learners include GDS, Lasso, neural nets, and boosting.Source: §5 · pp. 23–24
- Asymptotic Gaussian inference follows when regression and Riesz-representer errors converge at rates satisfying the theorem’s conditions, allowing a tradeoff between their speeds.The regression learner needs only a mean-square convergence rate, accommodating a wide variety of machine-learning estimators.Source: §5 · p. 24
A.1. Previous Learning Problems
- The paper’s related-work discussion distinguishes its general linear-functional focus from high-dimensional work centered on regression coefficients and notes that Riesz representers supply exploitable structure.It contrasts the approach with Cai and Guo (2017) and describes regularized Riesz-representer learners as directly targeting functionals and learning the representer from data.Source: §A.1 · p. 30
A.2. De-biased Estimation
- Earlier debiased-estimation analyses often require an explicit Riesz-representer formula; here the representer is learned automatically from its characterizing moment conditions, extending analysis to non-regular functionals.The paper presents this approach as applying to a broader class of linear functionals and allowing relatively slow regression rates, including dense or approximately sparse cases.Source: §A.2 · pp. 30–31
A.3. Localized Functionals
- The paper develops a unified analysis for localized functionals in high-dimensional settings, including effects and derivatives near a value of a low-dimensional covariate subvector.It also analyzes finite-sample Gaussian approximation for general nonpathwise differentiable functionals such as localized ones.Source: §A.3 · pp. 31–32
C. STRUCTURE OF FUNCTIONALS AND THEIR SCORES IN LEADING EXAMPLES
- The appendix identifies the score’s standard deviation, kurtosis, and the functional’s modulus of continuity as key quantities for the main inference results.It develops bounds for these quantities in leading examples.Source: C. STRUCTURE OF FUNCTIONALS AND THEIR… · p. 33
C.1. Structure of Riesz representers for unrestricted and additive models
The section characterizes Riesz representers for unrestricted and additive models, then relates global functional behavior to properties of the representer and score. In additive models, conditional expectation produces minimal representers from universal ones.
C.1. Structure of Riesz representers for unrestricted and additive models
- For key examples, an L2 universal representer ensures continuity, and the minimal representer is its projection onto the regression space when that space is not all of L2(F).The lemma also gives examples where the functional is continuous on a restricted space although the displayed universal representer is unavailable with positive probability.Source: §C.1 · p. 34
- Additive models can have minimal representers different from universal representers, while conditional expectation constructs the minimal ones from universal representers.This setting arises when the regression space is not dense in L2(F).Source: §C.1 · pp. 34–35
- For additive models, conditional expectation of a universal representer yields a minimal representer and contracts Lq norms; using the minimal representer attains full semiparametric efficiency unless the regression space is unrestricted.The conditional expectation operator is also order-preserving.Source: §C.1 · pp. 34–35
C.2. Structure of global functionals and scores in key examples
- In global examples, non-regularity can arise as overlap-related denominators become large, while the section characterizes bounds through the representer, score variance, kurtosis, and modulus of continuity.The discussion focuses on unrestricted or additive models and selected global functional examples.Source: §C.2 · p. 35
C.3. Structure of local and localized functionals and scores in key examples
The section characterizes local functional scores and explains how localization affects non-regularity. It also bounds the bias from approximating a perfectly localized target, with an additional robustness requirement for inference at a point.
C.3. Structure of local and localized functionals and scores in key examples
- Inference for perfectly localized functionals requires the scaled localization bias to be small, making it less robust than inference for local functionals.The paper bounds localization bias using smoothness and kernel order.Source: §C.3 · p. 36
- The section also characterizes local average-effect and local average-derivative scores under model and moment conditions, including cases where the differentiation variable is localized.For local average derivatives, the stated setup imposes density and derivative bounds in a neighborhood of the localization point.Source: §C.3 · pp. 36–37
- When overlap-related quantities are bounded, shrinking bandwidth h is the main source of non-regularity for local functionals.The stated condition bounds the other source associated with the quantities in (3.25).Source: §C.3 · p. 37
- Under the stated smoothness conditions, the bias between a local functional and its perfectly localized target is bounded in order by h^v.Here v is the smoothness order specified in the lemma.Source: §C.3 · pp. 37–38
D. PROOFS FOR SECTION 2
The proof establishes representer existence by extending the functional to the closed regression space and applying the Riesz–Fréchet theorem. A given representer can be projected onto that space to obtain the minimal one.
D. PROOFS FOR SECTION 2
- The proof uses Hahn–Banach and Riesz–Fréchet to represent the extended functional on the closed regression space, with projection yielding a minimal representer.The projection does not increase the L2 norm.Source: §D.1 · p. 38
E.1. Practical implementation details
The appendix outlines an iterative GDS procedure and explains how average partial differences are used when direct derivatives are unavailable or difficult to access.
E.1. Practical implementation details
- GDS initializes coefficients on a low-dimensional dictionary, computes full-dictionary moments, updates diagonal normalization, and iteratively solves a constrained ℓ1 problem.The final estimate is returned after several iterations; the practical choice c = 1 is reported to work from the outset.Source: §E.1 · p. 38
E.2. Partial difference
- Random forests are nondifferentiable, while neural-network derivatives can be difficult to access in black-box implementations, so both use an average partial difference approximation.For this application, the change Δ is set to one fourth of the standard deviation of D.Source: §E.2 · p. 39
- With a dictionary quadratic in d, GDS and Lasso average-derivative estimates equal average partial-difference estimates; the gasoline price-elasticity specification has this property.Table 3's partial-difference estimates therefore coincide with derivative estimates for GDS and Lasso, and approximate them for random forest and neural network.Source: §E.2 · p. 39
E.3. Empirical results without debiasing
This material identifies the contents of the no-debiasing tables and presents proof steps for Theorems 4.1 and 4.2 and Lemmas 4.2 and 4.3.
E.3. Empirical results without debiasing
- The supplement's no-debiasing Tables 4–6 correspond to main-text Tables 1–3, covering 401(k) treatment effects and gasoline-demand derivatives with localized estimates by income quintile.The tables describe GDS/Lasso and random forest/neural-network regressions; the supplied passages do not report their numerical results.Source: §E.3 · p. 40, Table 4 · p. 40, Table 5 · p. 40
F.1. Proof of Theorem 4.1
- Theorem 4.1's proof partitions observations into evaluation and complement sets and expands the estimator around the true regression and Riesz representer.The proof refers to Condition R and analyzes the resulting remainder terms.Source: §F.1 · p. 41
F.2. Proof of Theorem 4.2
- Theorem 4.2's proof identifies the score function as the unique influence function by projecting it onto the score span.The proof also describes a locally asymptotically normal path and its tangent set.Source: §F.2 · p. 43
F.3. Proof of Lemma 4.2
- Lemma 4.2 bounds the off-support estimation error by the on-support error when the remainder term is small relative to the on-support error.The stated condition is 2∥tr∥1 ≤ ∥δM∥1.Source: §F.3 · p. 45
F.4. Proof of Lemma 4.3
- Lemma 4.3 uses an event holding with probability at least 1 − ǫ to bound the estimation error through the sparsity measure and a bound on ∥δ∥1.The passage gives the resulting quadratic-form bounds in terms of s(t0), ν, and B.Source: §F.4 · p. 45
F.5. Proof of Theorem 4.3 and Corollary 4.4
The proof combines estimation-error bounds with a stated size condition to establish the guarantee R(δ) for the specified ε.
F.5. Proof of Theorem 4.3 and Corollary 4.4
- Applying Lemma 4.3 yields estimation-error bounds with probability at least 1 − 4ǫ.The displayed bound itself is not included in the supplied passage.Source: §F.5 · p. 45
- The proof concludes that guarantee R(δ) holds for ε = 1 − K4ǫ, provided the stated condition holds for a sufficiently large absolute constant C.The condition's formula is not included in the passage.Source: §F.5 · p. 45
- The proof refers to r1, r2, and r3 as specified in the corollary when stating the bounds.The passage does not give their formulas.Source: §F.5 · p. 45
G.1. Proof of Theorem 5.1
The proof of Theorem 5.1 defines the score components and uses sample splitting and nuisance-error bounds to control the estimator's remainder.
G.1. Proof of Theorem 5.1
- The proof defines φ(w, γ, α) as α(x)[y − γ(x)] and ψ as θ − m(w, γ) − φ(w, γ, α).These definitions specify the score components used in the proof.Source: §G.1 · p. 46
- Each fold's regression estimate depends only on observations outside that fold, supporting the proof's conditional independence argument.The proof uses this independence to analyze terms indexed by observations within the fold.Source: §G.1 · p. 46
- The proof bounds a nuisance-error product using the L2 errors of the estimated representer and regression.It then concludes that the summed remainder is o_p(σ/√n).Source: §G.1 · pp. 46–47
H.1. Proof of Lemma 3.1
The proofs establish representer integrability and bounds under stated moment and localization conditions, while identifying settings where the representer is undefined.
H.1. Proof of Lemma 3.1
- Square-integrable α0 and γ imply a finite inner product by the Cauchy–Schwarz bound.Source: §H.1 · p. 47
- The representer can be undefined under failing overlap, absent Radon–Nikodym derivatives, or zero conditional density, even when the associated functional is well defined.Source: §H.1 · pp. 47–48
H.2. Proof of Lemma 3.2
- Projection onto L2(F1) is conditional expectation given X1, and its contractive property follows from Jensen’s inequality.Source: §H.2 · p. 48
H.3. Proof of Lemma 3.3
- The proof bounds second moments using bounded-moment assumptions, conditional moments, and the triangle inequality.Source: §H.3 · p. 48
H.4. Proof of Lemma 3.4
- For localized functionals, the proof changes variables, expands in h, and summarizes non-asymptotic bounds for sufficiently small h.Source: §H.4 · p. 49
H.5. Proof of Lemma 3.5
The proofs derive bounds for localized representer expressions and summarize their behavior as the localization bandwidth shrinks, under stated conditions.
H.5. Proof of Lemma 3.5
- The proof identifies v(X) as the conditional density f(D | Z) and uses assumed bounds and lower bounds in its calculations.Source: §H.5 · pp. 49–50
- As h shrinks, the proof summarizes bounds for both cases considered in the argument.Source: §H.5 · p. 51
H.6. Proof of Lemma 3.6
- For h below h1, the proof obtains a bound whose constants depend on K, v, and the stated density-related quantities.Source: §H.6 · p. 52