PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Generalized Method of Moments for Instrumental Variable Analysis

Andrew Bennett, Nathan Kallus, Tobias Schnabel

arXiv:1905.12495v2stat.MLcs.LGecon.EM

TL;DR

Complex causal relationships and high-dimensional treatments or instruments make conventional IV methods difficult to apply, especially when many moment conditions are needed. DeepGMM addresses this with a variational, optimally weighted GMM framework implemented as a smooth game with neural networks. It matches or exceeds tuned methods in standard benchmarks and remains effective in high-dimensional settings where recent methods break.

  • Problem

    IV estimation is impeded by confounding, complex causal relationships, high-dimensional variables, and the failure of conventional GMM when too many moment conditions are required.

  • Method

    DeepGMM combines neural networks with a variational reformulation of optimally weighted GMM, solving a smooth prediction-versus-critic game with practical optimization and validation procedures.

  • Results

    DeepGMM performs on par with or better than existing approaches in standard benchmarks and continues to work in high-dimensional settings where even recent methods break.

  • Takeaways & Limitations

    DeepGMM provides a computationally tractable approach for IV analysis with high-dimensional variables and complex relationships.

  • Takeaways & Limitations

    The consistency result assumes identification by the function class and bounded complexity, while finite-moment identification may be too strong for highly complex parameters.

Abstract

from arXiv · show

Instrumental variable analysis is a powerful tool for estimating causal effects when randomization or full control of confounders is not possible. The application of standard methods such as 2SLS, GMM, and more recent variants are significantly impeded when the causal effects are complex, the instruments are high-dimensional, and/or the treatment is high-dimensional. In this paper, we propose the DeepGMM algorithm to overcome this. Our algorithm is based on a new variational reformulation of GMM with optimal inverse-covariance weighting that allows us to efficiently control very many moment conditions. We further develop practical techniques for optimization and model selection that make it particularly successful in practice. Our algorithm is also computationally tractable and can handle large-scale datasets. Numerical results show our algorithm matches the performance of the best tuned methods in standard settings and continues to work in high-dimensional settings where even recent methods break.

1 Introduction

Instrumental variables address confounding by using a relevant instrument that does not directly affect the outcome, but flexible causal relationships and high-dimensional variables require more capable methods. DeepGMM combines neural networks with optimally weighted GMM and performs competitively in standard settings while remaining effective in high-dimensional settings.

  • Confounding can make supervised learning estimate correlations rather than counterfactual treatment effects, as illustrated by a neural network missing the true response curve.The β-blocker example links adherence with health consciousness, which also predicts heart health.
  • Adjusting for confounders requires observing all factors that jointly determine treatment and outcome, which is often impractical.In the β-blocker example, latent determinants of adherence and general healthfulness may not be perfectly measured.
  • Instrumental variables identify causal effects when the instrument influences treatment but does not directly affect the outcome.They enable natural-experiment analyses under mild assumptions.
  • High-dimensional instruments, treatments, and complex causal or interaction effects motivate flexible neural-network methods for IV analysis.
  • DeepGMM uses a variational reformulation of optimally weighted GMM to estimate complex causal effects with neural networks and supports practical optimization and model validation.Its estimator is formulated as a smooth game, with approximate equilibria shown to provide consistent causal-parameter estimates.

2 Setup and Notation

The setup allows endogenous treatments and identifies a causal response function through instrument-based conditional moment restrictions. The section reviews two-stage and moment-based estimators, emphasizing that conventional GMM weighting fails when flexible models require many moments.

  • Setup and notation: The model permits treatment endogeneity, so the causal response function need not equal the conditional expectation of outcome given treatment.Identification uses an instrument Z that is relevant for X and satisfies the model’s instrument conditions.
  • Setup and notation: The target response function may be linear, neural, or nonparametric, and performance is measured by mean squared error against the true response.
  • Setup and notation: Exogenous context variables can be included in both treatment and instrument, allowing heterogeneous effects over observed demographics.In the β-blocker example, adherence and demographics enter X, while co-payment and demographics enter Z.
  • Existing methods for IV estimation: Two-stage least squares and related flexible approaches can be limited by an unknown sufficient basis or high-dimensional treatment, while DeepIV may suffer from density-fitting sensitivity.
  • Existing methods for IV estimation: GMM estimates parameters by making empirical instrument-response moments small, but unweighted norms can be inefficient when many moments are redundant or irrelevant.Optimal inverse-covariance weighting yields minimal asymptotic variance when the covariance can be consistently estimated.
  • Existing methods for IV estimation: With flexible models, standard GMM and optimally weighted GMM fail when too many moments are needed because equal weighting is inefficient and the empirical covariance becomes singular.Using a pseudoinverse would retain at most n moment conditions, preventing straightforward use of infinitely many moments.
  • Existing methods for IV estimation: Finite-moment identification assumptions may be too strong for highly complex parameters, although they can yield statistically efficient estimators.

3 Methodology

DeepGMM reformulates optimally weighted GMM as a saddle-point problem, then replaces finite moment-function spans with flexible neural-network classes for both response and critic functions. Its consistency guarantee applies under identification, bounded-complexity, continuity, boundedness, and related assumptions when the game is approximately solved.

  • 3.1 Reformulating OWGMM: The variational reformulation converts the optimally weighted GMM objective into an equivalent saddle-point formulation.This provides the basis for optimizing the estimator through a zero-sum game.
  • 3.2 DeepGMM: DeepGMM uses neural-network classes for both moment functions and response functions, enabling flexible models for high-dimensional causal variables.The critic class is rich but non-linear rather than a linear subspace, while the response class consists of neural networks.
  • 3.2 DeepGMM: Using a rich non-subspace critic class avoids the failure of naively enumerating many or infinitely many moment conditions, while bounded complexity supports consistency.The paper distinguishes this construction from GMM with a high- or infinite-dimensional linear moment space.
  • 3.3 Consistency: The consistency result assumes identification, vanishing Rademacher complexity, absolute star-shapedness, continuity, and bounded random variables.The theorem is stated for generic bounded-complexity functional classes, not only neural networks.
  • 3.3 Consistency: Approximately solving the zero-sum game with eventually vanishing approximation error guarantees consistency under the stated assumptions.Without identification, the analysis instead approaches an identified parameter satisfying all moment conditions.
  • 3.3 Consistency: The boundedness assumption is presented as technical and potentially relaxable to sub-Gaussian tails, with weight bounds or bounded output activations offering practical controls.The authors do not find enforcing these controls necessary in practice.

4 Practical Considerations in Implementing DeepGMM

DeepGMM is implemented by solving a smooth zero-sum game with first-order methods, using OAdam for stability and PyTorch for scalable gradient computation. Validation-based hyperparameter selection and early stopping use a surrogate of the variational objective.

  • Solving the Smooth Zero-Sum Game: OAdam was more stable than alternating SGD or Adam for solving DeepGMM’s smooth game objective.The implementation uses OAdam in the experiments.
  • Solving the Smooth Zero-Sum Game: First-order iterative optimization and PyTorch gradients allow DeepGMM to handle very large datasets with high-dimensional features and instruments.The paper states that other methods fail in such settings.
  • Choosing the reference parameter: The reference parameter is updated using the previous response-function iterate while remaining constant in the response-function gradient.This recursively simulates multiple choices of the reference parameter during game optimization.
  • Hyperparameter Optimization: Hyperparameters are selected by maximizing a validation surrogate of the variational objective over encountered critic iterates.The tuned parameters include neural-network architectures and optimization settings such as learning rate.
  • Early Stopping: Early stopping returns the iterate with the best periodically evaluated validation surrogate.The same surrogate supports both hyperparameter selection and stopping decisions.

5 Experiments

Experiments compare DeepGMM with several IV baselines in low-dimensional and MNIST-based high-dimensional scenarios. DeepGMM competes with the best tuned methods in low dimensions and remains effective and tractable when other methods fail in high dimensions.

  • Experimental design: The evaluation used test-set MSE across low-dimensional cases and MNIST scenarios where X, Z, or both were high-dimensional.The high-dimensional experiments used 20,000 points for each train, validation, and test set and averaged results over ten random seeds.
  • Low-dimensional scenarios: DirectNN failed badly under confounding, making IV information necessary for the low-dimensional experiments.The data-generating process includes a confounder that breaks independence between X and the residual Y − g0(X).
  • Low-dimensional scenarios: In low-dimensional scenarios, DeepGMM attained the lowest MSE except for a slight disadvantage in the linear case against methods using linear models.The authors conclude that it adapts across scenarios and competes with the best tuned method for each scenario.
  • Low-dimensional scenarios: GMM+NN performed well but below DeepGMM, while AGMM performed similarly to GMM+NN and lacked DeepGMM’s optimal weighting.The authors suggest AGMM’s norm choice and lack of optimal weighting may explain its worse performance.
  • Low-dimensional scenarios: DeepIV was consistently outperformed by Poly2SLS across low-dimensional scenarios, despite correct specification of its first-stage Gaussian-mixture model.The authors discuss computational difficulty and sensitivity to first-stage density-fitting errors as possible explanations.
  • High-dimensional scenarios: DeepGMM achieved the lowest MSE in every high-dimensional scenario and remained computationally tractable where other methods failed.AGMM crashed, DeepIV produced NaN outcomes with high-dimensional X, and Poly2SLS could not be run on the large examples.

6 Conclusions

The paper presents DeepGMM for IV analysis with high-dimensional variables and complex relationships. Its variational optimal-weight GMM formulation performs competitively in low-dimensional settings and remains effective in high-dimensional settings where recent methods break.

  • 6 Conclusions: DeepGMM addresses IV analysis with high-dimensional variables and complex relationships through a variational reformulation of optimally weighted GMM.The method is formulated as a smooth zero-sum game intended to handle many moment conditions.
  • 6 Conclusions: Empirically, DeepGMM competes with the best tuned method in low-dimensional settings and performs well in high-dimensional settings where even recent methods break.

A Omitted Proofs

The omitted proofs establish the variational norm identity, uniform convergence of the empirical objective, and consistency of the estimated parameter under the stated assumptions.

  • Proof of Lemma 1: The variational norm admits a dual optimization representation derived through Lagrangian duality and strong duality.The derivation optimizes over v and then maps vectors to functions in span(F).
  • Proof of Theorem 2: The empirical objective converges uniformly to its population counterpart in probability, with separate bounds for sampling and plug-in errors.Rademacher complexity controls the first error, while boundedness and dominated convergence control the second.
  • Proof of Theorem 2: The population objective has θ0 as its unique minimizer because valid alternatives admit a function producing a strictly positive objective value.Scaling the separating function by sufficiently small λ preserves positivity through the quadratic expression.
  • Proof of Theorem 2: Approximate optimization combined with uniform convergence and identification yields θ̂_n →p θ0.The proof bounds the probability that the estimator remains outside any δ-neighborhood of θ0.
  • Proof of Lemma 2: When parameters are observationally equivalent, collapsing Θ0 into a single pseudo-parameter restores identification and gives infθ0∈Θ0 d(θ̂_n, θ0) → 0 in probability.The redefined metric preserves continuity and the other assumptions needed to invoke Theorem 2.

B.1 Hyperparameter Optimization Procedure

Hyperparameter selection evaluates saved training trajectories on validation data, approximating the variational objective while storing only network outputs for tractability.

  • B.1 Hyperparameter Optimization Procedure: Each hyperparameter candidate is trained for fixed epochs, with parameter states saved periodically to form candidate trajectories.The procedure creates finite sets of f functions and θ values for every candidate.
  • B.1 Hyperparameter Optimization Procedure: The validation approximation selects the hyperparameter set whose trajectory minimizes the approximated variational objective over its saved θ values.This approximates the objective that early stopping would produce for each hyperparameter choice.
  • B.1 Hyperparameter Optimization Procedure: Storing validation outputs instead of full neural-network parameters makes validation tractable for complex deep networks.The saved objects are vectors of f(Zval, τ) and g(Xval, θ).

B.2 Hyperparameter Details

The experiments use neural-network parameterizations for f and g, with architecture choices summarized across experimental scenarios in Table 3.

  • B.2 Hyperparameter Details: The experiments parameterize f and g with fully connected networks using leaky ReLU activations, while MNIST also uses a fixed deep convolutional architecture.The CNN is designed for non-causal inference on MNIST.
  • B.2 Hyperparameter Details: The implementation varies hidden-layer configurations and uses scenario-specific choices for the f and g networks.Exact CNN construction details are deferred to the code release.
  • B.2 Hyperparameter Details: Architecture and hyperparameter choices for each experimental scenario are summarized in Table 3.The same hyperparameter choices are used across all low-dimensional scenarios.

B.2.1 Low-dimensional Scenarios

In low-dimensional scenarios, f and g use small fully connected networks, and learning-rate selection is performed over a specified grid.

  • B.2.1 Low-dimensional Scenarios: The low-dimensional f network has one hidden layer of size 20, while g has two hidden layers of sizes 20 and 3.Both networks are fully connected.
  • B.2.1 Low-dimensional Scenarios: The g learning rate is selected from [5 ∗10^-4, 2 ∗10^-4, 1 ∗10^-3] during hyperparameter optimization.The f learning rate is set by multiplying the selected g learning rate by 5.
  • B.2.1 Low-dimensional Scenarios: The low-dimensional network specifications are part of the common experimental hyperparameter configuration.

C One-Step GMM Using All Square Integrable Moments

Using all square-integrable instrument functions with one-step GMM yields the same loss-minimization problem as non-causal least squares under a continuous-instrument assumption.

  • The analysis uses the infinity norm for the infinite collection of moment functions and integrates with respect to the empirical measure.The admissible instrument functions satisfy Pn f(Zi)^2 = 1.
  • The supremum over instrument functions is attained at the conditional expectation of the residual given Z by Cauchy-Schwarz.
  • Under continuous Z, one-step GMM using all square-integrable instrument functions is equivalent to non-causal least squares.Both methods select θ by minimizing the same loss function.
Loading 1905.12495v2…