Neural Network Attributions: A Causal Perspective

Introduction

• Black-box models make troubleshooting difficult and can leave uncertainty about whether high accuracy reflects learned concepts or dataset artifacts.This interpretability problem motivates explainable machine learning.Source: §1 · p. 1
• Gradient methods measure how perturbing an input affects output but not its causal influence, while interpretable regressions can capture correlation rather than causation.The paper frames attribution as an inherently causal question (Sundararajan et al., 2017).Source: §1 · p. 1
• The method treats neural networks as Structural Causal Models and estimates the Average Causal Effect of an input neuron on an output neuron.The authors present it as a first-principles causal approach to neural network attribution, to the best of their knowledge.Source: §1 · pp. 1–2
• Assuming inputs are not causally related but may share a latent confounder makes the causal-effect problem tractable in settings such as images.The authors note that this assumption may be violated by recurrent neural networks.Source: §1 · pp. 1–2
• The goal is to estimate an input’s causal influence on a neural network’s output, not to discover causal structure among the input features.The authors distinguish this objective from structure learning, which asks which features cause others.Source: §1 · p. 2
• The approach adds causal regressors to improve causal-effect estimates and provide a global perspective on those effects.The paper also proposes an approximate computation strategy for high-dimensional data and extends the methodology to recurrent neural networks.Source: §1 · p. 2
• The paper presents empirical results to demonstrate the method’s usefulness and compare it with a state-of-the-art gradient-based method.No specific results or winning comparison are given in the supplied introduction.Source: §1 · p. 2

• Perturbation and infinitesimal-gradient methods violate sensitivity and can assign zero attribution in flat regions, while erroneous heatmaps cannot be distinguished from poor learned representations.Source: §2 · p. 2
• Gradient and perturbation methods can be viewed as Individual Causal Effect special cases when inputs do not cause one another, but isolated effects can miss importance under complex feature interactions.Source: §2 · p. 2
• DeepLIFT and LRP address sensitivity with baseline-based discrete differences but break implementation invariance, while Integrated Gradients averages gradients along a baseline-to-input path yet may remain biased without marginalizing other inputs.Source: §2 · p. 2
• Integrated Gradients assigns different attributions to feature a despite the same baseline and a value, although the example attributes the entire output change to interventions on b.Source: §2 · pp. 2–3
• The paper proposes causal attribution that marginalizes over other input parameters, satisfies all listed axioms except conservativeness (irrelevant in a causal setting), and uses causal regressors for a global model perspective.Source: §2 · p. 3
• The closest prior causal explanation method extends LIME with local perturbed-sample regression, but learns correlations rather than causation; latent-concept approaches likewise do not explain effects causally (Alvarez-Melis & Jaakkola, 2017; Alvarez-Melis & Jaakkola, 2018; Li et al., 2018).Source: §2 · p. 3

3. Background: Neural Networks as Structural Causal Models (SCMs)

• Feedforward networks correspond to SCMs whose directed edges run from lower to higher layers, encoding hierarchical interactions without inferring causal direction (Kocaoglu et al., 2017).Each layer's neurons have causal functions, while exogenous random variables act as causal factors for the input layer.Source: §3 · p. 3
• Marginalizing hidden neurons reduces the feedforward SCM to observable input and output layers, with recursive substitution connecting deleted neurons' parents to their children.The resulting reduced SCM is formed by marginalizing hidden neurons from the layer-by-layer model.Source: §3 · p. 3, Figure 1 · p. 3
• Time-unfolded RNNs admit SCMs, but hidden-state marginalization creates cross-time causes that violate input noncausality when outputs feed later inputs.Direct cyclic SCMs may be ambiguous and fail to define a unique probability distribution (Bongers et al., 2016).Source: §3 · pp. 3–4, Figure 1 · p. 3

4.1. Causal Attributions

• The attribution need not sum to the output difference from a baseline because it measures causal strength, and it is unchanged by constant input shifts between equivalent functions.The method satisfies attribution axioms (ii)–(vi), but not the conservation axiom (i).Source: §4.1 · p. 4
• Causal attribution measures the effect of setting input feature xi to α on output y relative to its average interventional expectation, using a uniform intervention distribution over xi’s domain when no prior information is available.Domain knowledge can instead specify an intervention distribution or a significant baseline point.Source: §4.1 · p. 4

4.2. Calculating Interventional Expectations

• To avoid the high variance and cost of naive empirical estimates, the method approximates interventional expectations with a second-order Taylor expansion using interventional input means and covariances.For feedforward networks, the proposition states that the intervened input neuron is d-separated from all other input neurons.Source: §4.2 · pp. 4–5

Appendix A.3.3 provides the proof for Proposition 2.

• Under Corollary 2.1, intervening on input neuron xi leaves other input-neuron distributions unchanged; interventional means and covariances match observational values except for xi.After do(xi = α), E[xi] is α and Cov(xi, xj) is zero for every input neuron xj.Source: Appendix A.3.3 · p. 5

4.3. Computing ACE using Causal Regressors

• Causal regressors fit polynomial functions to interventional expectations, use Bayesian model selection to choose polynomial order, and enable on-the-fly expectation estimates.The baseline is computed by analytic integration; predictive variance estimates confidence and can signal a need for more intervention samples.Source: §4.3 · pp. 5–6

4.4. Overall Methodology

• The overall feedforward procedure computes interventional expectations in Phase I and learns causal regressors and estimates the baseline in Phase II.Phase I has complexity O(num × k); Phase II trains polynomial functions using Phase I outputs and Bayesian linear regression.Source: §4.4 · p. 6

4.5. Causal Attribution in RNNs

• For RNNs with output-to-input recurrence, intervention changes subsequent input distributions, so means and covariances are estimated from intervened training sequences.The method reduces attribution complexity from O(n.num.t.k) to O(n.num.τ.k), with τ computed per sample in one backward pass.Source: §4.5 · p. 6

4.6. Scaling to Large Data

• 0.04s versus 3.04s per computation on MNIST: the Taylor-series and eigendecomposition approximation was significantly faster than exact calculations.The experiment used a deep neural network with four hidden layers.Source: §4.6 · p. 6

5.2. Simulated data

• Over 1M sequences, imputing x0 and x1 changed predictions 1956 and 9 times, while imputing x3 had no effect; IG missed this model behavior under correlated inputs.The GRU achieved 98.94% accuracy, yet did not use the third feature despite its causal role in the simulated data-generating function.Source: Figure 3 · p. 7, §5.2 · p. 7

5.1. Iris dataset

• On Iris, causal regressors show higher petal widths favoring virginica over versicolor and setosa, while sepal-width trends reverse; this global view is unavailable from other attribution methods.The data is nearly separable with axis-aligned boundaries, and the decision tree thresholds petal width to predict each class.Source: §5.1 · pp. 6–7, Figure 2 · p. 7

5.2. Simulated data

• Local causal attributions ACE_y do(x=input[x]) can serve as saliency maps for explaining individual neural decisions.The simulated-data experiment used a GRU with one input, hidden, and output neuron and sigmoid activations.Source: §5.2 · p. 7

5.3. Airplane Data

• For the FDR-identified anomalous flight, attributions highlight acceleration, pitch, and roll up to 7 seconds before touchdown, unlike the normal flight; IG shows no comparable contrast.The anomalous flight was associated with slippery runway conditions and delayed braking, and the causal attribution pattern agreed with the FDR report.Source: Figure 4 · p. 7, §5.3 · p. 8

5.4. Visualizing Causal Effect

• On MNIST β-VAE latents, c_k interventions positively affect pixels at the corresponding digit location, while z0 and z6 control rotation and scaling, respectively.With c8 = 1, z0 and z6 were varied from -3.0 to 3.0; other continuous latents such as z2 showed no discernable causal effect.Source: §5.4 · p. 8, Figure 5 · p. 8

6. Conclusions

• The method models a neural network as a structural causal model and defines a mechanism to compute average causal effect (ACE) in neural networks.The paper presents an efficient strategy for high-dimensional data and extends the methodology to recurrent neural networks.Source: §6 · p. 8
• Experiments on synthetic and real-world data show promise for eliciting causal effects of neural-network inputs on outputs.Source: §6 · p. 8
• Future work includes extending the method to architectures such as ConvNets, studying other baselines, and investigating further causal questions about neural networks.The questions include whether causal chains exist in neural networks and whether their predictions are causal.Source: §6 · pp. 8–9

A.1. Causality Preliminaries

• An SCM comprises endogenous variables, exogenous noise, deterministic causal functions, and a noise distribution; feedback causal models are excluded.Each endogenous variable is determined by its function of other endogenous variables and exogenous noise; variables need not have unique or shared noise.Source: §A.1 · p. 11
• An SCM maps to a causal Bayesian network whose edges encode causal mechanisms, and each vertex’s distribution depends only on its parents.Vertices represent observable endogenous variables, and an edge runs from each parent variable to the variable determined by its causal function.Source: §A.1 · p. 11
• d-separation indicates conditional independence in a faithful graphical model, with paths blocked by noncolliders in Z or colliders outside Anc(Z).Two vertices are d-separated when every connecting path is blocked; Anc(Z) comprises vertices with directed paths to a vertex in Z.Source: §A.1 · p. 11
• The do operator distinguishes an intervention-based expectation, E(y|do(x) = 1), from the conditional expectation E(y|x = 1) (Pearl, 2009; 2012).The intervention expectation is taken over the interventional distribution P(y|do(x) = 1).Source: §A.1 · p. 11
• The average causal effect of binary x on y is E(y|do(x = 1)) − E(y|do(x = 0)).Source: §A.1 · p. 11
• In a causal Bayesian network, intervening on variables removes their incoming edges and omits their factors from the interventional joint distribution.For an SCM, do(x = x′) replaces x’s causal mechanism with the constant x′ and substitutes x′ into causal-function arguments.Source: §A.1 · p. 11

A.2. More on Prior Work

• For f(x1,x2)=x1x2 at [5,1000], gradients imply x1 is stronger despite the paper's claim that x2 contributes more; gradients suit perturbation sensitivity, not output influence.Partial derivatives depend on the values of other input features.Source: §A.2 · pp. 11–12
• Perturbations and gradients can capture individual causal effects (ICE), but inter-feature interactions can conceal a feature's importance when ICE alone is analyzed.ICE conditions other inputs to their training-instance values and compares the intervened output with a baseline at α−ϵ.Source: §A.2 · p. 12
• Local regression can mistake correlations for causation: regressing y on inputs estimates E[y|x1,...,xk], not the causal effect E[y|do(xi)].Regression recovers E[y|do(xi)] only when all backdoor variables are controlled and weighted according to their distribution.Source: §A.2 · p. 12

A.3. Proofs

• A feedforward network can be represented as a structural causal model, and recursively substituting layer functions yields output neurons expressed as functions of the input layer.The input neurons are modeled as functions of independent noise variables.Source: §A.3.1 · p. 12, §A.3.2 · p. 12

A.3.3. PROOF OF PROPOSITION 2

• Intervening on an input neuron blocks paths to other input neurons, making the intervened neuron d-separated from them.The proof attributes this to paths passing through exogenous variables or output-neuron colliders, with intervention removing incoming edges to the intervened neuron.Source: §A.3.3 · p. 12

A.3.4. PROOF OF PROPOSITION 3

• For recurrent networks, the lookback considers timesteps with a nonzero Jacobian determinant, and the dataset-level τ is the expected maximum such lookback.The proof frames this criterion around whether the output depends on an earlier input through a one-to-one mapping.Source: §A.3.4 · pp. 12–13

A.4. Algorithms/Pseudocode

• Phase I estimates interventional expectations using evenly spaced intervention values; its feedforward computation scales as O(k × num), compared with O(n × num) for naive estimation.The parameter num determines the number of intervention values and affects the accuracy of the learned polynomial functions in Phase II.Source: §A.4.1 · p. 13

A.4.2. ALGORITHM FOR PHASE I IN RECURRENT NETWORKS

• For recurrent networks, the algorithm intervenes on a feature at a selected timestep, propagates the sequence, and estimates interventional expectations from input means and covariances.The input data is arranged as samples × time × features.Source: §A.4.1 · p. 13, §A.4.2 · p. 13

A.5. Scaling to Large Data

• 0.04s for the approximation versus 3.04s per exact computation on MNIST, with the methods agreeing almost perfectly across intervention values.The approximation computes the second-order term with three forward passes using inputs μ, μ+ϵV, and μ−ϵV, with ϵ = 10^-6.Source: Figure 6 · p. 14, §A.5 · p. 14

A.6.1. GENERATION OF SYNTHETIC DATASET

• Synthetic sequences have lengths from 10 to 15, and their first three values are sampled from N(1, 0.2) or N(−1, 0.2) to assign class 1 or 0, respectively.The two classes are sampled with equal probability.Source: §A.6.1 · p. 14

A.6.3. ADDITIONAL RESULTS: VISUALIZING CAUSAL EFFECT

• Interventions on VAE latents show that z0 controls rotation and z6 controls scaling, while z2 and other tested dimensions show no discernable causal effect.The reported visualizations include class-specific latents for digits 0, 2, and 3.Source: §A.6.3 · p. 15, Figure 7 · p. 16, Figure 8 · p. 16