PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Neural Networks for Estimation and Inference

Max H. Farrell, Tengyuan Liang, Sanjog Misra

arXiv:1809.09953v3econ.EMcs.LGmath.STstat.ML

TL;DR

The paper addresses limited theoretical understanding of valid inference after using deep neural networks for first-step estimation. It develops convergence-rate and inference results for deep feedforward ReLU networks, obtaining rates that support valid second-step inference and illustrating their usefulness in causal and direct-mail applications.

  • Problem

    The paper studies the limited theoretical understanding of inference after deep neural networks are used for first-step estimation, particularly in settings where deep learning is increasingly applied.

  • Method

    It derives nonasymptotic estimation bounds and convergence rates for feedforward deep ReLU networks, then applies them to semiparametric inference without sample splitting or cross fitting.

  • Results

    The rates support valid second-step inference after deep-learning estimation, including inference for causal parameters such as treatment effects, welfare, and decomposition effects.

  • Takeaways & Limitations

    Deep nets provide a valid and useful first-step estimator for semiparametric inference and show excellent performance in the paper’s empirical application.

  • Takeaways & Limitations

    The paper presents its contribution as a first step and does not seek weaker conditions for the semiparametric stage itself.

Abstract

from arXiv · show

We study deep neural networks and their use in semiparametric inference. We establish novel rates of convergence for deep feedforward neural nets. Our new rates are sufficiently fast (in some cases minimax optimal) to allow us to establish valid second-step inference after first-step estimation with deep learning, a result also new to the literature. Our estimation rates and semiparametric inference results handle the current standard architecture: fully connected feedforward neural networks (multi-layer perceptrons), with the now-common rectified linear unit activation function and a depth explicitly diverging with the sample size. We discuss other architectures as well, including fixed-width, very deep networks. We establish nonasymptotic bounds for these deep nets for a general class of nonparametric regression-type loss functions, which includes as special cases least squares, logistic regression, and other generalized linear models. We then apply our theory to develop semiparametric inference, focusing on causal parameters for concreteness, such as treatment effects, expected welfare, and decomposition effects. Inference in many other semiparametric contexts can be readily obtained. We demonstrate the effectiveness of deep learning with a Monte Carlo analysis and an empirical application to direct mail marketing.

1 Introduction

The paper develops theory for modern deep ReLU networks and uses it to establish valid semiparametric inference after deep-learning estimation. It targets a gap in understanding inference with deep networks and illustrates applications including causal analysis and treatment targeting.

  • Research focus: The paper studies deep neural networks with ReLU activations and explicitly diverging depth, focusing on the modern architecture used in current deep learning.The analysis concerns fully connected feedforward networks and contrasts modern deep ReLU networks with earlier shallow sigmoid-based networks.
  • Theoretical contributions: It provides new nonasymptotic bounds and convergence rates for nonparametric estimation under a general class of regression-type loss functions.The loss framework includes generalized linear models, with linear and logistic regression used as concrete causal-inference illustrations.
  • Semiparametric inference: The rates are sufficiently fast to support valid inference on finite-dimensional parameters after first-step estimation with deep learning.The paper emphasizes inference results that are new to the deep-neural-network literature and applies them to semiparametric settings.
  • Causal applications: The causal applications cover average treatment effects, counterfactual expected utility or profits from treatment targeting, and decomposition effects.These results allow planners to compare predetermined or estimated targeting strategies while accounting for targeting costs.
  • Scope: The results extend beyond causal inference to estimands admitting a locally robust semiparametric procedure and to several economic contexts.The paper identifies applications including selection models, games, consumer surplus, and dynamic discrete choice.
  • Empirical illustration: Deep nets are at least as good as, and sometimes better than, the best methods in Hitsch and Misra in the paper’s direct-mail marketing application.The paper also uses numerical analysis to illustrate the utility of deep learning and discusses relevance beyond causal inference.

2 Deep Neural Networks

The paper develops theory for deep ReLU neural networks, focusing on fully connected feedforward architectures, and uses the resulting convergence rates for semiparametric inference. It covers general regression-type losses and several network architectures, including cases attaining optimal rates and very deep fixed-width networks.

  • Semiparametric Inference: The resulting rates are sufficiently fast to support valid second-step semiparametric causal inference after first-step deep-network estimation.The paper presents this inference development as new to the literature and applies it to causal parameters.
  • Estimation Framework: The estimated function maps covariates X ∈ R^d to an outcome Y by minimizing the expected per-observation loss.The framework assumes an unknown smooth target function and allows losses that are Lipschitz in the function and satisfy a curvature condition around the target.
  • Network Architectures and Rates: Certain deep ReLU networks attain the optimal Stone rate, while the standard MLP result is suboptimal but still sufficient for semiparametric inference.The paper also establishes convergence rates for very deep fixed-width MLPs and considers more general feedforward architectures.
  • Bounds and Convergence Rates: The paper establishes nonasymptotic bounds and convergence rates for deep ReLU networks under a general class of nonparametric regression-type losses.The loss class includes least squares, logistic regression, and other generalized linear models.
  • Neural Network Constructions: Fully connected feedforward networks, or multi-layer perceptrons, are the main architecture studied because they are widely used in practice.The analysis focuses on the ReLU activation function and explicitly considers architectures whose depth grows with sample size.

3 Parameters of Interest

The paper defines causal parameters based on potential outcomes and counterfactual treatment policies, including average effects, welfare or profit, subgroup effects, and decompositions. Under standard identification assumptions, these parameters can be represented and estimated using doubly robust influence-function methods.

  • Parameters of Interest: The paper studies average treatment effects, expected utility or profits under targeting policies, subgroup effects, and decomposition effects.Subgroups include treated or untreated groups and groups defined by covariates.
  • Identification: Causal effects are defined from potential outcomes, but each unit reveals only one of its two potential outcomes.The observed outcome is Y = TY (1) + (1 − T)Y (0).
  • Identification: Identification relies on selection on observables and overlap, with propensity score p(x) and outcome regressions µt(x) characterizing treatment assignment and potential outcomes.The assumptions impose conditional independence and bound the propensity score away from zero and one.
  • Policy Parameters: A policy s(x) assigns treatment using covariates, and its expected outcome π(s) evaluates the counterfactual welfare or profit of applying that rule.The policy need not equal the observed treatment assignment.
  • Policy Parameters: A candidate policy improves on a baseline when it targets people with higher average individual treatment effects among those treated differently.The comparison depends on the conditional average treatment effect τ(x).
  • Estimation: Influence-function estimators for these full-population parameters are doubly robust, remaining consistent when either outcome regressions or the propensity score are correctly specified.The framework also covers broader estimands whose locally or doubly robust estimators depend on the paper’s loss class.

4 Asymptotic Inference

The paper uses deep MLP estimators for propensity scores and outcome regressions, then combines them with locally or doubly robust estimators for causal inference. Its main result is that deep ReLU first-step estimates converge fast enough to support valid asymptotic inference, including for randomized treatments and multiple causal parameters.

  • Inference Construction: The inference estimators are sample averages of uncentered influence functions and include standard locally or doubly robust estimators.The average treatment effect estimator is the standard doubly or locally robust estimator.
  • First-Step Estimation: Deep MLPs estimate propensity scores with logistic loss and conditional outcome functions with least-squares loss for each treatment status.The paper also considers joint estimation because treatment groups may share features.
  • Inference Results: The main inference result establishes valid asymptotic inference after deep-learning first-step estimation, including asymptotic normality for the resulting estimators.Theorem 3’s conditions yield asymptotic normality and support variance estimation from sample moments.
  • Rate Conditions: Deep ReLU first-step estimates converge faster than n^-1/4, satisfying a rate sufficient for the subsequent semiparametric inference results.The relevant conditions can be weaker when expressed through products of estimation errors.
  • Inference Results: Localization is used to verify the strongest remainder condition directly, providing a novel approach for second-step inference with deep learning.The authors suggest this approach may be useful in future machine-learning applications.
  • Scope and Limitations: The inference framework uses stronger-than-minimal smoothness assumptions, including βp ∧ βµ > d, and leaves some relaxed-assumption procedures for future work.Cross-fitting and overfitting-robust procedures are discussed as possible alternatives but are not automatic here.
  • Randomized Treatment: For randomized treatment, the propensity score is root-n consistent, so only the remaining outcome-related condition must be verified.The framework applies to randomized experiments, including an illustration with 300,000 units and hundreds of covariates.

5 Empirical Application

The empirical application uses a randomized direct-mail experiment to estimate treatment effects and evaluate targeting policies with several deep-network architectures. Deep nets perform at least as well as established methods, while architecture-specific estimates broadly agree and a prior-spending policy is estimated to be optimal.

  • Data and motivation: The study analyzes a randomized direct-mail experiment involving 292,657 consumers, with two-thirds receiving catalogs and roughly 150 observed covariates.Covariates include demographics, past purchasing, and interactions with the firm.
  • Architecture and estimation: Deep nets are at least as good as, and sometimes better than, the best prior methods, including Causal KNN with Treatment Effect Projections and Causal Forests.The comparison concerns fit and final estimates.
  • Architecture and estimation: Joint estimation of µ0(x) and τ(x) outperforms separately estimating each µt(x), although the approaches are theoretically equivalent.Here τ(x) is the conditional average treatment effect of the catalog mailing.
  • Architecture and estimation: Across eight architectures, conditional-treatment-effect densities differ in shape, but mean and variance estimates are similar.The application uses ReLU-based networks with varied architectures.

6 Monte Carlo Analysis

The Monte Carlo analysis evaluates deep-learning semiparametric inference across randomized and observational designs with linear and nonlinear outcome models. Results show minimal bias, accurate coverage, controlled interval length, and broadly similar performance across architectures, while regularization can worsen bias and interval length.

  • Design: The simulations use n = 10,000 i.i.d. samples and 1,000 replications for each data-generating process.Both randomized and observational treatment-assignment settings are evaluated.
  • Design: The simulations span eight designs combining d = 20 or 100, constant or varying propensity scores, and linear or nonlinear outcome models.Treatment assignment is either randomized with p(x) = 0.5 or observational.
  • Design: Each design uses ReLU-based multilayer perceptrons with architectures varying in depth and width.The architectures are variants of those used in the empirical application.
  • Results: Deep-learning semiparametric inference exhibits minimal bias, accurate coverage, and controlled interval length across the simulated designs.The reported results cover average-treatment-effect inference under randomized and observational data.
  • Results: Most architectures yield similar results, with no architecture dominating and more complex networks showing no systematic coverage or interval-length pattern.Coverage and interval length are fairly similar across architectures.
  • Regularization: Regularization usually preserves coverage but increases bias and interval length relative to the nonregularized results, motivating caution in applications.The authors’ preliminary exploration included dropout and other regularization methods.

7 Conclusion

The paper develops theory and evidence for using deep learning in econometric estimation and inference, including valid inference after first-step deep-learning estimation. It applies these results to causal and other economic settings while identifying unresolved challenges.

  • The authors report that deep-learning methods can offer excellent performance theoretically and empirically, including in causal applications.
  • The paper provides a formal proof that inference can be valid after deep learning is used for first-step estimation.
  • Novel bounds for deep neural networks deliver fast convergence rates that allow second-stage inference on finite-dimensional parameters.
  • The results cover fully connected feedforward networks, fixed-width very deep networks, and general nonparametric regression-type loss functions.
  • The theory supports embedding deep learning into standard econometric models, including linear regressions, generalized linear models, censored regression, and two-step estimators.
  • The paper treats deep learning as a first step toward economic applications, with open questions about density estimation, network architecture, tuning, and further applications.

A Proofs

The proofs establish the main theoretical results through a general architecture argument, then specialize where needed to multilayer perceptrons and impose a large-sample condition.

  • The proof first bounds approximation error and then controls the empirical-process term using localization.
  • The argument is developed for a general network architecture and specializes to multilayer perceptrons only when necessary.
  • The proof section organizes supporting special cases and related results separately from the main argument.
  • The theorem statements require the sample size to exceed a condition involving the envelope constant and pseudo-dimension of the DNN class.

A.1 Main Decomposition and Bias Term

The main decomposition separates empirical-process variation from approximation bias, accounting for the fact that the target function need not belong to the DNN class.

  • The approximation benchmark is the best function realized by the deep ReLU network class.
  • The decomposition is necessary because the target function is not assumed to belong to the DNN class, unlike the estimator and comparison functions.
  • The proof decomposes the estimation error into an empirical-process term and a bias or approximation-error term.
  • Bernstein’s inequality is applied to bound the bias term, while the empirical-process term is handled separately.

A.2 Localization Analysis

The localization analysis controls empirical-process fluctuations using Rademacher complexity and establishes comparison between empirical and population L2 norms beyond a critical radius.

  • The analysis uses scale-insensitive complexity bounds and localization to control the empirical-process term.
  • Rademacher complexity measures the flexibility of a function class for predicting independent random signs.
  • The proof studies a localized quadratic process for squared estimation errors and applies symmetrization and Lipschitz bounds.
  • For radii above the critical radius, the empirical L2 norm is at most twice the corresponding population L2 norm with probability at least 1 −exp(−˜γ).

A.2.2 Step II: One Step Improvement

Step II uses an initial L2 error radius to obtain a sharper high-probability bound through variance-dependent empirical-process control. The resulting improvement produces a smaller radius for subsequent recursive tightening.

  • A.2.2 Step II: One Step Improvement: An initial bound ∥ˆf −f∗∥L2(x) ≤r0 supplies information for sharpening the rate through step-by-step reductions.The variance bound depends on the radius r0, enabling the refinement.
  • A.2.2 Step II: One Step Improvement: Applying Lemma 5 to localized loss differences bounds the empirical-process term with probability at least 1−2e−˜γ.The localized class restricts functions to ∥f−f∗∥L2(x)≤r0.
  • A.2.2 Step II: One Step Improvement: The empirical Rademacher complexity is controlled using entropy bounds when n > Pdim(FDNN), together with a constant K = 6 × 128Cℓ.These controls are inserted into the main decomposition to tighten the error bound.
  • A.2.2 Step II: One Step Improvement: The combined bounds hold with probability at least 1 −4 exp(−˜γ), yielding the tighter result used in later steps.The proof combines the empirical-process and variance calculations through the main decomposition.
  • A.2.2 Step II: One Step Improvement: When ϵn ≪r0, the argument improves the error radius from r0 to r1 ≪r0 with high probability.The improvement can be recursively repeated toward a fixed point/radius r∗.

A.2.3 Step III: Critical Radius

Step III defines a critical radius as a fixed point of localized empirical Rademacher complexity and bounds it using shell-wise localization. The resulting radius is then combined with approximation and architecture choices to prove the neural-network rate.

  • A.2.3 Step III: Critical Radius: The critical radius r∗ is the largest fixed point satisfying 18MERn{f −f∗: f ∈F, ∥f −f∗∥L2(x) ≤s} < s2 for all s ≥r.This radius summarizes the fundamental difficulty of the localized estimation problem.
  • A.2.3 Step III: Critical Radius: The event controlling the localization argument holds with probability at least 1 −n−1.By construction, the same fixed-point condition also applies to 2r∗.
  • A.2.3 Step III: Critical Radius: The proof partitions FDNN into L2 shells B(f∗, ¯r), B(f∗, 2¯r)\B(f∗, ¯r), and successive dyadic shells.The shell index l is chosen as the largest integer no greater than log2((log n)/n).
  • A.2.3 Step III: Critical Radius: Shell-by-shell application of Steps I and II, together with a union bound, yields simultaneous high-probability control across the localized classes.The argument uses variance dependence within each shell and requires two stated conditions.
  • A.2.3 Step III: Critical Radius: The final critical-radius bound is obtained by choosing γ = −log(6l) + ˜γ and combining the upper bound on r∗ with the shell argument.The proof concludes with a constant C′ > 0 independent of n.
  • A.2.3 Step III: Critical Radius: For the MLP architecture, the parameter-count relation W ≤C · H2L is substituted into the general bound before optimizing ϵn, H, and L.The optimization uses the MLP-specific approximation rate and the embedding result.

A.4 Proof of Corollaries 1 and 2

The corollary proofs optimize approximation and estimation terms using known approximation results for deep networks. They obtain rates by selecting width and depth as functions of n, β, and d.

  • A.4 Proof of Corollaries 1 and 2: For Corollary 1, the approximation result yields ϵ ≍n−β/(2β+d) with W, U ≍nd/(2β+d) and L ≍log n.The selected architecture balances approximation accuracy with the estimation bound.
  • A.4 Proof of Corollaries 1 and 2: For Corollary 2, choosing ϵ ≍n−1/(2+d) gives L ≍n−d/(4+2d).This depth choice follows from the cited MLP approximation result.

B Supporting Lemmas

The supporting lemmas establish architectural embedding, complexity control, concentration, approximation, and loss-function conditions used in the main rate proofs. They also verify applicability to least-squares, logistic, and broader generalized linear losses.

  • Architecture and embedding: Any feedforward network in FDNN can be embedded in an MLP with H ≤WL + U by adding auxiliary hidden nodes.The auxiliary units pass information across nonadjacent layers while preserving the represented function.
  • Architecture and embedding: The embedding preserves the function, with gθ′ = fθ, and auxiliary units can be implemented using ReLU biases and subtraction in the receiving unit.This makes the construction compatible with the standard ReLU MLP architecture.
  • Complexity and concentration: Contraction, Dudley chaining, entropy bounds, and pseudo-dimension control provide the Rademacher-complexity tools used for localization.The pseudo-dimension is related to metric entropy and bounded through VC-dimension results.
  • Complexity and concentration: Variance-dependent symmetrization controls empirical-process deviations under boundedness and variance assumptions with exponentially high probability.The variance dependence is specifically retained because it is crucial for localization.
  • Network complexity and approximation: For ReLU networks, pseudo-dimension bounds apply to FDNN(W, L, U), while approximation classes achieve depth O(log(1/ϵ)) and width and units O(ϵ−d).The approximation constant depends only on d and β.
  • Loss functions: Least-squares and logistic losses satisfy the required curvature and Lipschitz conditions, and the framework extends to Poisson, Gamma, and multinomial logistic regression.For multinomial logistic regression, bounded f yields explicit lower and upper eigenvalue bounds for the Hessian.
Loading 1809.09953v3…