PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel Instrumental Variable Regression

Rahul Singh, Maneesh Sahani, Arthur Gretton

arXiv:1906.00232v6cs.LGecon.EMmath.FAmath.STstat.ML

TL;DR

Confounded observational data make nonlinear causal effects difficult to estimate, while 2SLS imposes linearity. KIV uses nonlinear RKHS models, proves consistency and minimax-rate conditions, and outperforms state-of-the-art nonparametric IV methods in simulations.

  • Problem

    2SLS simplifies confounded causal estimation by assuming linear relationships, although relationships among X, Y, and Z may be nonlinear.

  • Method

    KIV generalizes 2SLS nonparametrically by estimating conditional mean embeddings and applying kernel ridge regression in RKHSs.

  • Results

    KIV is consistent under mild assumptions and can achieve the minimax optimal rate of unconfounded single-stage RKHS regression.

  • Takeaways & Limitations

    KIV outperforms state-of-the-art alternatives for nonparametric IV regression in simulations.

  • Takeaways & Limitations

    KIV relies on characteristic and prior assumptions, while weak instruments can make IV estimation highly erratic.

Abstract

from arXiv · show

Instrumental variable (IV) regression is a strategy for learning causal relationships in observational data. If measurements of input X and output Y are confounded, the causal relationship can nonetheless be identified if an instrumental variable Z is available that influences X directly, but is conditionally independent of Y given X and the unmeasured confounder. The classic two-stage least squares algorithm (2SLS) simplifies the estimation problem by modeling all relationships as linear functions. We propose kernel instrumental variable regression (KIV), a nonparametric generalization of 2SLS, modeling relations among X, Y, and Z as nonlinear functions in reproducing kernel Hilbert spaces (RKHSs). We prove the consistency of KIV under mild assumptions, and derive conditions under which convergence occurs at the minimax optimal rate for unconfounded, single-stage RKHS regression. In doing so, we obtain an efficient ratio between training sample sizes used in the algorithm's first and second stages. In experiments, KIV outperforms state of the art alternatives for nonparametric IV regression.

1 Introduction

Instrumental variables identify counterfactual effects under confounding, but classical 2SLS assumes linear relationships. KIV generalizes 2SLS nonparametrically with RKHSs and provides consistency and stage-sample allocation guarantees.

  • Instrumental variable regression identifies counterfactual effects when Z influences Y only through X despite unmeasured confounding.
  • 2SLS uses linear regressions in both stages, although practical relationships between Y and X or X and Z may be nonlinear.
  • KIV is an easily implemented nonlinear generalization of 2SLS that models relationships in reproducing kernel Hilbert spaces.
  • Stage 1 estimates a conditional mean embedding whose characteristic RKHS representation captures the full conditional distribution of X given Z.
  • The analysis proves convergence of the stage 1 conditional mean embedding estimator in RKHS norm, including infinite-dimensional RKHSs.
  • KIV theory derives a stage 1-to-stage 2 sample ratio for minimax-optimal second-stage rates, depending on the difficulty of each stage.

2 Related work

Prior nonlinear IV methods use basis approximations, density estimation, or neural networks, while KIV replaces these with RKHS-based estimation under characteristic and prior assumptions. Weak instruments remain a practical source of instability.

  • Sieve IV approximates relationships with basis functions, but selecting an appropriate finite dictionary is challenging.
  • Other approaches estimate the conditional distribution of X given Z with density ratios before applying ridge regression.
  • Deep IV uses neural networks for both stages and supports complex high-dimensional data such as images.
  • Compared with density-based approaches, KIV’s conditional mean embeddings have a closed-form stage 1 solution and lower variance than sample averaging.
  • KIV replaces completeness with the characteristic property and replaces stability and link assumptions with the concept of prior.
  • Weak instruments make IV estimation highly erratic because their relationship with X is negligible.

3 Problem setting and definitions

The IV model identifies a structural function through an operator equation linking the reduced form to conditional expectations. RKHS operators and embeddings represent the two stages, while regularization addresses ill-posed inversion.

  • The structural model defines h as the causal function of interest and e as unmeasured confounding noise.
  • The exclusion restriction and relevance condition make the IV model tractable and ensure that the instrument is informative.
  • The IV hypothesis yields the operator equation E[Y|Z] = E_X|Z h(X), linking the reduced form to the stage 1 conditional expectation and structural function.
  • Solving for h requires inverting an infinite-dimensional compact operator, making the problem ill-posed and motivating smoothness and Tikhonov regularization.
  • The conditional expectation operator E and conditional mean embedding µ are equivalent stage 1 objects, with µ(z) = E*φ(z).
  • The structural operator H and function h are equivalent stage 2 objects, with h(x) = Hψ(x).

4 Learning problem and algorithm

KIV splits estimation into two RKHS ridge-regression stages: estimating the conditional expectation operator and regressing outcomes on its embeddings. The construction yields closed-form computation and supports theoretical regularization choices.

  • Sample splitting estimates stage 1 from n observations and stage 2 from m remaining observations, with an efficient ratio given by Theorem 4.
  • 4.1 Stage 1: A vector-valued RKHS transforms stage 1 into kernel ridge regression for the conditional expectation operator E.
  • Regularized target operators and empirical estimators encode Tikhonov regularization for both stages.
  • KIV’s RKHS construction enables kernel ridge estimators for both stages, unlike prior density-estimation approaches.
  • 4.2 Stage 2: The stage 2 problem becomes scalar-valued kernel ridge regression on estimated conditional mean embeddings.
  • Repeated representer-theorem applications produce a closed-form KIV estimator using empirical kernel matrices.
  • Theorems 2 and 4 determine efficient rates for the stage 1 parameter λ and stage 2 parameter ξ.

5 Consistency

KIV’s consistency analysis establishes finite-sample convergence for both stages under RKHS assumptions, then combines the rates to characterize end-to-end performance and sample allocation.

  • Stage 1: The stage 1 estimator converges in RKHS norm to the true conditional expectation operator under stated hypotheses.The analysis assumes correctly specified vector-valued RKHS regression, bounded continuous kernels, characteristic kX, and distributional smoothness conditions.
  • Stage 1: The stage 1 convergence rate is calibrated by c1, which measures the smoothness of the conditional expectation operator.The paper relates the conditional mean embedding to the adjoint conditional expectation operator.
  • Stage 2: Stage 2 analyzes kernel ridge regression of outputs on conditional mean embeddings using integral-operator tools and assumptions on the structural operator.The structural operator is assumed to lie in the scalar-valued RKHS, with additional boundedness, continuity, and prior conditions.
  • End-to-end convergence: KIV achieves the minimax optimal rate of single-stage RKHS regression for unconfounded data under the stated assumptions.The result equates the achievable learning rate for confounded two-stage estimation with that of unconfounded single-stage RKHS regression.
  • Sample allocation: The statistically efficient stage 1-to-stage 2 sample ratio depends on the difficulty of each stage and implies n > m.The difficulty is characterized by effective input dimension and structural-operator smoothness; the asymmetric split differs from prior analyses assuming n = m.

6 Experiments

The experiments evaluate KIV against four nonparametric IV alternatives across linear, sigmoid, and demand designs. KernelIV performs best in the sigmoid design and, among implementable estimators, at larger demand-design sample sizes.

  • The study compares KIV with KernelReg, SmoothIV, SieveIV, and DeepIV on three simulation designs.Each algorithm, design, and sample size uses 40 simulations, with MSE measured against the true structural function h.
  • KernelIV performs best across sample sizes in the sigmoid design.
  • SmoothIV performs best at n + m = 1000 in the demand design, while KernelIV performs best among implementable estimators at n + m = 5000 and n + m = 10000.SmoothIV was not implemented at larger sample sizes because of running time.
  • KernelReg becomes more biased as demand-design sample size increases because it ignores the instrument and is misled by confounded data.
  • KernelIV uses implicit infinite basis functions, kernel ridge regression in both stages, and direct stage-1 mean-embedding estimation.These choices distinguish it from SieveIV, SmoothIV, and DeepIV, respectively.

7 Conclusion

The paper introduces KIV for nonlinear causal learning from confounded observational data and reports theoretical and empirical support for the approach. KIV is presented as easily implemented and minimax optimal, with simulations showing it outperforms state-of-the-art nonparametric IV methods.

  • KIV learns nonlinear causal relationships from confounded observational data.
  • The paper establishes KIV as easily implemented and minimax optimal.
  • KIV estimates the stage-1 conditional expectation operator, an infinite-by-infinite-dimensional object, using kernel ridge regression.
  • Simulations show that KIV outperforms state-of-the-art algorithms for nonparametric IV regression.
  • The results suggest that RKHS methods may bridge econometrics and machine learning for causal inference and ill-posed inverse problems.

A.1.1 Comparison of IV assumptions

The paper contrasts IV assumptions expressed through conditional independence, potential outcomes, and moment restrictions. Its chosen formulation uses additive separability and a mean-independence condition for tractability.

  • Conditional independence: The conditional-independence formulation requires exclusion, an unconfounded instrument, and relevance.The relevance condition means that the conditional distribution of X given Z is not constant in z.
  • Conditional independence: A DAG encodes the conditional-independence assumptions used in the introduction.
  • Potential outcomes: The potential-outcomes formulation is noted as an alternative, but its relation to DAGs is outside the work’s scope.
  • Moment restrictions: The paper instead uses a moment-restriction formulation that encodes causal assumptions through functional restrictions and conditional expectations set to zero.
  • Moment restrictions: Hypothesis 1 imposes additive separability of confounding noise and E[e|Z] = 0, relaxing independence to mean independence.

A.1.2 Linear vignette

The linear vignette shows why 2SLS targets the structural parameter when observations are confounded, unlike ordinary least squares. It motivates KIV as the nonlinear feature-space analogue of this two-stage construction.

  • With confounded observations, least squares and 2SLS have different estimands, and 2SLS targets the structural parameter of interest.
  • In stage 1, 2SLS projects X onto the conditional mean X̄(Z) = E[X|Z], then projects Y onto that quantity.
  • Taking conditional expectations with respect to Z removes the confounding contribution and yields the structural relationship between X̄(Z) and Ȳ(Z).
  • The resulting identification formula expresses β through moments of X̄(Z) and the original outputs Y.
  • KIV replaces X and Z with feature maps ψ(X) and φ(Z), regressing Y on the conditional mean embedding µ(Z) = E[ψ(X)|Z].

A.2 Comparison of nonparametric IV bounds

This section compares KIV with alternative nonparametric instrumental-variable methods that provide statistical guarantees.

  • KIV is evaluated against nonlinear IV approaches discussed in the related-work literature.
  • The comparison focuses on methods with established statistical guarantees.
  • These alternatives are subsequently compared with KIV in the experiments.

A.2.1 Nadaraya-Watson IV

The section contrasts prior two-stage IV analyses with KIV's conditional-mean-embedding framework, rate guarantees, and training-sample allocation results. It also highlights differences in assumptions, stage-1 estimation, and empirical performance.

  • Nadaraya-Watson IV: The prior two-stage bound combines Nadaraya-Watson density estimation with ridge regression in a square-integrable function space.Its ridge penalty is not directly imposed on an RKHS norm.
  • Nadaraya-Watson IV: Prior work estimates stage-1 conditional densities, whereas KIV estimates conditional mean embeddings sufficient for stage-2 regression.Density estimation becomes more difficult with increasing dimension, while KIV's stage-1 difficulty depends on regression-function smoothness.
  • Nadaraya-Watson IV: KIV provides a minimax-optimal stage-2 guarantee and an efficient ratio of stage-1 to stage-2 training samples.The comparison notes that earlier analyses did not discuss optimality or an efficient sample ratio.
  • Predictive state representations: The PSR bound has a slower-than-minimax stage-2 rate and requires strong smoothness assumptions on the stage-2 input.Its additional Hilbert-Schmidt assumption is also insufficient for the stated bound.
  • Predictive state representations: The PSR assumptions do not apply to KIV's pointwise counterfactual prediction setting.The required test feature representation holds only in a trivial single-point-mass case, so neither cited bound applies here.
  • KIV analysis: KIV's analysis links stage-1 convergence to stage 2 through an RKHS-norm and uniform bound for the conditional mean embedding.This bound is used in the subsequent stage-2 theorem.
  • Empirical comparison: In experiments, KernelIV performs best across sigmoid-design sample sizes and at n + m = 5000 and n + m = 10000 in the demand design.SmoothIV performs best for n + m = 1000 in the demand design; in the linear design, KernelIV performs about as well as regularized SieveIV.
Loading 1906.00232v6…