PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Network Attributions: A Causal Perspective

Aditya Chattopadhyay, Piyushi Manupriya, Anirban Sarkar, Vineeth N Balasubramanian

arXiv:1902.02302v4cs.LGstat.ML

TL;DR

Deep neural networks are difficult to interpret, and existing attribution methods may capture perturbational effects or correlations rather than causal influence. This paper models the network as a Structural Causal Model to compute feature-level causal effects, including scalable and recurrent-network variants. It reports empirical evidence for the method’s usefulness and presents causal attribution visualizations on latent variables.

  • Problem

    Existing attribution methods do not necessarily capture the causal influence of an input on a particular output neuron and can be biased by correlations or other input values.

  • Method

    The method models a neural network as a Structural Causal Model and computes Average Causal Effects, using marginalization, causal regressors, scalable approximations, and recurrent-network extensions.

  • Results

    The paper reports empirical results showing the usefulness of causal attribution and compares the method with a state-of-the-art gradient-based method; latent-variable visualizations identify rotation and scaling effects.

  • Takeaways & Limitations

    Causal attribution provides feature-level and global perspectives on neural-network decisions while addressing implicit biases in existing attribution methods.

  • Takeaways & Limitations

    The recurrent setting violates the assumption that input neurons are not causally related, requiring marginalization of hidden variables in the reduced causal model.

Abstract

from arXiv · show

We propose a new attribution method for neural networks developed using first principles of causality (to the best of our knowledge, the first such). The neural network architecture is viewed as a Structural Causal Model, and a methodology to compute the causal effect of each feature on the output is presented. With reasonable assumptions on the causal structure of the input data, we propose algorithms to efficiently compute the causal effects, as well as scale the approach to data with large dimensionality. We also show how this method can be used for recurrent neural networks. We report experimental results on both simulated and real datasets showcasing the promise and usefulness of the proposed algorithm.

1. Introduction

Deep learning’s success is tempered by interpretability problems, motivating attribution methods that explain how inputs affect predictions. This work develops a causal neural-network attribution framework, extends it to recurrent networks, and targets scalable computation.

  • Motivation: Black-box neural networks make troubleshooting difficult and can obscure whether predictions reflect learned concepts or dataset artifacts.Training data may contain human errors or unwarranted correlations, and may differ from real-world data.
  • Motivation: Attribution-based methods explain the effect of an input feature on a prediction function’s output.Existing approaches backpropagate signals to inputs or approximate local decision boundaries with interpretable regressors.
  • Motivation: Gradients measure output changes under perturbation but do not capture an input’s causal influence on a particular output neuron.Regression-based approaches can map correlations rather than causation and may therefore introduce artifacts.
  • Approach: The method views neural networks as Structural Causal Models and computes the Average Causal Effect of an input neuron on an output neuron.The proposed setting treats input neurons as non-causally related while allowing joint effects from latent confounders such as data-generating mechanisms.
  • Scope and contributions: The framework extends from feedforward networks to recurrent neural networks and includes an approximate strategy for high-dimensional data.The work distinguishes causal influence on a learned output from structure learning, which seeks causal relations among features.
  • Scope and contributions: The paper introduces causal regressors, scaling methods, recurrent-network extensions, and empirical comparisons with a state-of-the-art gradient-based method.These contributions are presented as a methodology for causal attribution from first principles.

2. Prior Work and Motivation

Prior attribution methods can be biased by correlations, baselines, and local perturbations that obscure causal influence. The paper motivates marginalizing over other inputs and using causal regressors to obtain more faithful and global attribution.

  • Existing attribution methods: Saliency-map methods typically condition attributions on the given input data.The section situates these methods within broader efforts to identify effects of input neurons on output neurons.
  • Existing attribution methods: Gradient and local-perturbation methods can assign zero attribution in flat regions even when a feature matters.This behavior violates the sensitivity criterion for attribution methods.
  • Causal perspective: Individual Causal Effect measures the output difference after intervening on one input neuron for a particular individual input.Under a no-input-causation assumption, infinitesimal interventions connect ICE to partial derivatives.
  • Causal perspective: Analyzing ICE alone can conceal the real importance of a feature because of complex inter-feature interactions.This limitation motivates methods that account for effects beyond a single intervention at one input.
  • Implicit biases: For f(a,b)=ab, Integrated Gradients assigns different attribution to a=3 when b changes from 5 to 100 despite identical baselines and a values.The example highlights sensitivity to other input values; the passage states that the change in output is due to intervention on b rather than a.
  • Proposed motivation: The proposed causal approach marginalizes over other input parameters and uses causal regressors to provide a global perspective unavailable from existing attribution methods.The authors state that the approach satisfies all listed axioms except conservativeness, which they deem irrelevant in a causal setting.
  • Proposed motivation: Regression-based explanations can learn correlations that differ markedly from causation, including in closely related causal explanation methods.The paper distinguishes its objective—causal influence on a learned neural function—from inferring dependencies through local regression or latent concepts.

3. Background: Neural Networks as Structural Causal Models (SCMs)

The paper models neural-network architectures as Structural Causal Models, representing layerwise computations and exogenous causes of inputs. Feedforward networks reduce to input-output SCMs by marginalizing hidden neurons, while recurrent networks require time-unfolding because feedback creates cycles and input dependencies.

  • Feedforward networks: Neural-network layers form an SCM whose directed edges encode causal functions from lower layers to higher layers.The input layer has exogenous causal factors, and the output depends on hierarchical interactions among lower-level nodes.
  • Feedforward networks: Exogenous variables may jointly cause multiple input features, so the reduced causal Bayesian network need not be causally sufficient.This represents latent common causes rather than causal relationships among the input features themselves.
  • Feedforward networks: Hidden neurons can be marginalized to obtain a reduced SCM containing only observable inputs and outputs.Recursive substitution creates directed connections from hidden-neuron parents to output-layer children.
  • Recurrent networks: Directly defining an SCM for an RNN introduces feedback loops, making its causal Bayesian network non-acyclic.The paper instead relies on a time-unfolded RNN and carefully defines the reduced SCM after marginalizing hidden states.
  • Recurrent networks: In sequence prediction, recurrent output-to-input connections violate the assumption that input neurons are causally unrelated.After hidden-state marginalization, inputs at earlier timesteps can connect to outputs and subsequent inputs.

4. Causal Attributions for Neural Networks

The paper defines attribution as the causal effect of an input neuron on an output neuron, estimated through interventions in a neural-network SCM. It addresses high-variance and costly intervention estimation using causal assumptions and Taylor-based computation.

  • Causal attribution: The attribution problem asks for the causal effect of a particular input neuron on a particular output neuron.The required information is encoded in a reduced SCM consistent with the learned neural-network architecture.
  • Causal attribution: For continuous inputs, causal attribution is defined using the Average Causal Effect under interventions do(x_i = α).The method contrasts the interventional expectation at α with an adaptive baseline based on average interventional effects.
  • Causal attribution: The method does not require attributions to sum to the difference between the network output and a baseline because it measures causal strength toward one output neuron.The paper reports that the causal attribution satisfies several attribution axioms, including invariance to constant input shifts.
  • Baseline: The baseline for each input is the average interventional expectation across that input’s values, rather than a fixed reference such as a zero vector.This choice is intended to avoid sensitivity to a fixed baseline and is estimated by perturbing the input over an interval.
  • Interventional expectations: Naively estimating interventional expectations by sampling other features is high-variance in high dimensions and expensive when repeated across training data.The paper therefore proposes an alternative mechanism based on the reduced SCM and interventional expectations.
  • Interventional expectations: Under the feedforward SCM assumptions, intervention on one input leaves the other inputs’ distribution unchanged and sets the intervened input to α.The resulting interventional means and covariances support Taylor-based evaluation of the output expectation.

Appendix A.3.3 provides the proof for Proposition 2.

The appendix describes assumptions and computational strategies for estimating causal expectations and scaling attribution, including causal regressors, recurrent interventions, and a timestep-dependent reduction in complexity.

  • Feedforward computation: For feedforward networks, intervention leaves non-intervened input distributions unchanged, so observational means and covariances can be reused.The intervened feature has mean α and zero covariance with the other input features.
  • Causal regressors: The method assumes the interventional expectation as a function of one input can be represented by a polynomial, while allowing other function classes.Bayesian model selection chooses the polynomial order and predictive variance indicates confidence in the estimate.
  • Causal regressors: Causal regressors approximate interventional expectations for later on-the-fly attribution queries, reducing repeated costly calculations.The attribution evaluates the learned regressor at α and subtracts it from the input-specific baseline.
  • Overall procedure: The overall feedforward procedure has two phases: compute interventional expectations, then learn causal regressors and estimate baselines.Phase I uses empirical input statistics, while Phase II fits Bayesian linear-regression models to intervention results.
  • Recurrent computation: For recurrent networks, interventional means and covariances are estimated after evaluating each training sequence with the intervention applied.This is necessary because recurrent connections change later input distributions after intervention.
  • Recurrent computation: The recurrent attribution computation costs O(n × num) per input neuron and scales linearly with the relevant timelag τ.A single backward pass computes τ and reduces the complexity from O(n.num.t.k) to O(n.num.τ.k).
  • Large-dimensional data: 0.04s for the approximation v/s 3.04s per computation on MNIST with a four-hidden-layer deep neural network.The approximation uses a Taylor expansion and covariance eigendecomposition without explicitly requiring the Hessian.

5. Experiments and Results

Experiments apply causal attributions to Iris classification, recurrent sequence data, flight trajectories, and disentangled MNIST representations. The method provides local and global explanations, with causal attributions aligning with known or reported causal structure.

  • Iris dataset: The Iris experiment uses causal regressors and decision-tree comparisons to expose feature-level effects across the three classes.Petal length and width are positively causal for Iris-setosa, moderate values favor Iris-versicolor, and higher values favor Iris-virginica.
  • Attribution method: The approach computes local saliency maps from the causal effect of each input neuron on an output neuron.Its implementation is publicly available as ACE.
  • Simulated data: 98.94% accuracy was achieved by the GRU on simulated sequences whose first three features were designed to determine the class.The experiment compares causal attributions with Integrated Gradients and imputation-based prediction changes.
  • Simulated data: 1956 and 9 prediction changes occurred when imputing x0 and x1, while imputing x3 had no effect over 1M sequences.The imputation experiment indicates that the learned GRU ignored the third feature, whereas Integrated Gradients missed this because of correlations.
  • Airplane data: Causal attributions for an anomalous flight identified effects in LATG, LONG, PTCH, and ROLL up to 7 seconds before the measured outcome.The pattern agreed with the Flight Data Recorder report, while the normal flight showed no specific saliency structure and IG lacked a stark anomaly contrast.
  • Disentangled representations: In the β-VAE experiment, class-specific latent ck values were positively causal for pixels corresponding to their respective digits.The continuous latents z0 and z6 appeared to control rotation and scaling of digit 8, respectively.

6. Conclusions

The paper presents neural-network attribution from a causal perspective by modeling networks as Structural Causal Models and computing Average Causal Effects. It provides scalable and recurrent extensions, with experiments showing promise on synthetic and real-world data.

  • Contributions: The approach defines and efficiently computes Average Causal Effect for neural networks while extending the methodology to high-dimensional data and RNNs.The conclusion frames these extensions as central components of the presented methodology.
  • Implications: Experiments on synthetic and real-world datasets show promise for eliciting causal effects of neural-network inputs on outputs.The authors propose that this causal lens can support further questions about counterfactuals, causal chains, and whether predictions are causal.

A.1. Causality Preliminaries

The appendix introduces Structural Causal Models, graph representations, d-separation, interventions, and Average Causal Effect as foundations for causal analysis.

  • Structural Causal Models: A Structural Causal Model consists of endogenous variables, exogenous noise, structural functions, and a distribution over the exogenous variables.Endogenous variables are deterministically fixed by functions of exogenous and other endogenous variables.
  • Graph representation: An SCM can be represented as a directed graph whose vertices are observables and whose edges encode causal mechanisms between variables.A parent variable has a directed edge to the variable whose structural function depends on it.
  • d-separation: Two variables are d-separated given Z when every connecting path is blocked by an unconditioned collider or a conditioned noncollider.The definitions characterize conditional independence in the corresponding graphical model.
  • Interventions: The do operator identifies interventional effects by replacing a variable’s causal mechanism with an externally imposed value.In graph form, intervention removes incoming edges to the intervened variable; in an SCM, its mechanism becomes constant.
  • Average Causal Effect: For a binary x and outcome y, ACE is E(y|do(x = 1)) − E(y|do(x = 0)).The quantity compares outcome expectations under two interventions rather than ordinary conditioning.

A.2. More on Prior Work

Prior attribution methods include gradient-based and local regression-based approaches, but their quantities need not represent which input caused a particular output. Causal attribution instead targets interventional effects rather than correlations or input-conditioned sensitivities.

  • Existing approaches: Existing attribution methods are broadly divided into gradient-based and local regression-based approaches.Both families are used to explain the effect of an input neuron on a specific output neuron.
  • Gradient methods: Gradients measure how perturbing an input may affect output, but they do not determine which input influenced a particular output neuron.Partial derivatives can be biased by the values of other input features.
  • Individual effects: Perturbations and gradients can be viewed as Individual Causal Effects when other input neurons are fixed to an instance’s values.The expectation is over the learned neural function’s unobservable noise.
  • Individual effects: Inter-feature interactions can conceal an input feature’s importance when only its Individual Causal Effect is analyzed.The computation varies one feature while holding the remaining features at their instance values.
  • Regression methods: Regression-based attribution can map correlations rather than causation because conditioning E[y|x1, ..., xk] differs from the interventional quantity E[y|do(xi)].Regression identifies the causal quantity only under additional conditions involving backdoor variables.

A.3.1. PROOF OF PROPOSITION 1

The method computes interventional expectations for neural-network inputs and uses covariance structure to approximate costly second-order terms. Experiments report substantially faster approximation with near-perfect agreement with exact calculations, while latent interventions reveal interpretable causal effects.

  • Interventional computation: Interventional expectations E(y|do(x_i)) are computed by varying an input neuron across evenly spaced intervention values.The number of interventions controls the sampled α values and affects the accuracy of subsequent causal regressors.
  • Scaling to Large Data: The high-dimensional approximation uses covariance eigendecomposition and second-order directional derivatives instead of explicitly computing the full Hessian.The second-order term can be evaluated with three forward passes using µ, µ+ϵV, and µ−ϵV.
  • Scaling to Large Data: 0.04s for the approximation versus 3.04s per exact computation on MNIST with a four-hidden-layer network.The reported timing compares approximate and exact second-order calculations for the same experiment.
  • Scaling to Large Data: The approximate and exact second-order terms agree almost perfectly across intervention values in the MNIST experiment.The comparison follows the y=x line in Figure 6.
  • Visualizing Causal Effects: Latent interventions identify rotation in z0, scaling in z6, and no discernable causal effect in z2.The decoded-image trends are consistent with the corresponding causal attributions.
Loading 1902.02302v4…