PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining Deep Learning Models using Causal Inference

Tanmayee Narendra, Anush Sankaran, Deepak Vijaykeerthy, Senthil Mani

arXiv:1811.04376v1cs.LGcs.AIstat.ML

TL;DR

Deep neural networks are difficult to interpret, and existing methods do not explain how structural changes affect predictions. This paper builds a Structural Causal Model abstraction for reasoning about CNN components and ranking filters by counterfactual importance. It illustrates the approach on LeNet-5, VGG-19, and ResNet-32, while reporting that the binary transformation yields SCM accuracy below random choice.

  • Problem

    DNNs’ complex representations are difficult to understand, and existing interpretability methods do not explain how structural perturbations affect predictions.

    Source: Introduction · p. 1

  • Method

    The paper builds and validates a causal abstraction of a DNN and uses it to rank model components by importance, illustrated on VGG-19, ResNet-32, and LeNet-5 with CIFAR10.

    Source: Introduction · p. 2, Conclusion and Future Work · p. 7

  • Results

    The binary transformation’s SCM accuracy is below random choice, which the authors hypothesize reflects excessive information loss.

    Source: Sanity Check of the Causal Model · p. 6

  • Takeaways & Limitations

    The framework is illustrated as a way to build, verify, and use causal abstractions to rank filters in CNN layers.

    Source: Conclusion and Future Work · p. 7

    The current formulation largely assumes a fixed dataset, and the authors suggest more expressive structural-equation models and more representative transformations for complex queries.

    Source: Conclusion and Future Work · p. 7

Abstract

from arXiv · show

Although deep learning models have been successfully applied to a variety of tasks, due to the millions of parameters, they are becoming increasingly opaque and complex. In order to establish trust for their widespread commercial use, it is important to formalize a principled framework to reason over these models. In this work, we use ideas from causal inference to describe a general framework to reason over CNN models. Specifically, we build a Structural Causal Model (SCM) as an abstraction over a specific aspect of the CNN. We also formulate a method to quantitatively rank the filters of a convolution layer according to their counterfactual importance. We illustrate our approach with popular CNN architectures such as LeNet5, VGG19, and ResNet32.

Introduction

Deep neural networks’ complex representations make their inner workings difficult to understand, while existing interpretability methods do not explain how structural changes affect predictions. The paper proposes a causal abstraction for reasoning about such changes and quantitatively ranking model components, illustrated on image-classification models.

Introduction

  • Deep networks’ expressive power relies on complex higher-dimensional representations whose intermediate features are difficult to understand.The paper motivates interpretability in light of DNN use in both consumer-facing and sensitive applications.Source: Introduction · p. 1
  • Existing CNN explanation methods generate image saliency maps but do not explain how structural perturbations, such as removing a filter, affect predictions.The methods discussed focus on the relevance of image pixels to the CNN output.Source: Introduction · p. 1
  • The framework builds a causal abstraction that supports interventions and queries about how DNN components affect predictions and performance.It is intended to trace effects from inputs through filters and intermediate representations to outputs, including counterfactual questions.Source: Introduction · p. 1
  • Ablation testing measures component importance by removing components, retraining on the same dataset, and comparing performance, but can be computationally expensive.The proposed approach requires one-time construction of the causal abstraction and no retraining to identify a filter’s influence on predictions.Source: Introduction · pp. 1–2
  • The work contributes a general causal-model framework, a validation approach, and a method to quantitatively rank DNN components by importance.It illustrates the framework with VGG-19, ResNet-32, and LeNet-5 on CIFAR10.Source: Introduction · p. 2

Background

The background distinguishes observational prediction from intervention and counterfactual questions, then introduces Structural Causal Models as a representation of causal mechanisms. For this work’s intervention estimates, it uses direct interventions over a fixed dataset instead of marginal-probability estimation.

Background

  • Causal models add structure beyond statistical prediction, supporting questions about interventions and distribution changes.The background describes causal models as subsuming statistical models.Source: Causal Theory · p. 2

Causal Theory

  • Causal questions extend beyond prediction: interventions set variables to specified values, while counterfactuals ask retrospective “what if” questions.The paper frames causal models as a way to reason about changes such as setting a filter response to zero.Source: Causal Theory · p. 2

Structural Causal Models (SCM)

Estimating Interventions

  • The paper estimates intervention effects by intervening directly and averaging outcomes over a fixed dataset, rather than computing marginal probabilities.It avoids the latter because high graph connectivity and many variables make marginal-probability estimation computationally expensive.Source: Estimating Interventions · p. 2

Existing Work

Prior causal-model applications span advertising, astronomy, fairness, and deep learning. The authors position their work as the first, to their knowledge, to use causal inference to reason over DNN model components.

Existing Work

  • A causal model of computational advertising was used to address confounding and reason inexpensively about ad-placement interventions without randomized trials.The quantity of interest was advertisement clickthrough rate.Source: Existing Work · p. 2
  • Causal models have been applied to isolate Kepler telescope noise in exoplanet discovery [Sch¨olkopf et al. 2016] and to quantify model bias counterfactually [Kusner et al. 2017].The paper also cites causal explanations over human-understandable image abstractions and gender-bias removal in NLP [Harradon, Druce, and Ruttenberg 2018; Lu et al. 2018].Source: Existing Work · p. 3
  • The authors state that, to their knowledge, this is the first use of causal inference to reason over DNN model components.They place the claim against prior causal applications in other domains and earlier deep-learning work.Source: Existing Work · p. 3

Proposed Approach

The approach builds a structural causal model (SCM) to reason about a selected aspect of a deep neural network. It derives a graph from the network, transforms filter responses, and estimates structural equations.

Proposed Approach

  • The approach constructs a DAG, applies a suitable transformation, and estimates structural equations to build a causal model of a deep learning network.Source: Proposed Approach · p. 3

Building the Structural Causal Model

DAG Structure

  • At filter granularity, each filter in layer i receives incoming edges from every filter in layer i−1; the study restricts its analysis to filters.Source: DAG Structure · p. 3

Applying a Suitable Transformation

Learning the Equations of the SCM

  • Because conditional probability tables become prohibitively large in the highly connected CNN graph, the method instead learns each node’s function from its parents as a regression.Source: Learning the Equations of the SCM · p. 3
  • For each filter in layer i, its structural function is approximated using the preceding layer’s filters, and the functions together with the DAG form the SCM.Source: Learning the Equations of the SCM · p. 3

Experimental Set Up

The experiments apply the causal framework to CNNs on CIFAR10 and compare transformations for representing filter responses. The SCM uses a CNN-derived fixed DAG, estimated structural equations, and intervention data.

Implementation Details

  • Experiments apply the framework to VGG19, ResNet32, and LeNet5 on CIFAR10, with all models trained using Keras and TensorFlow.Source: Implementation Details · p. 4

Building the Causal Model

  • The causal DAG is inferred from the CNN and remains fixed for that model, while the transformation φ and structural-equation estimation method can vary.Source: Building the Causal Model · p. 4

Transformation φ

  • The binary variance transformation marks filters as high- or low-variance, assuming that this distinction is a useful measure of importance despite substantial information loss.The paper notes prior observations that low-variance filters contain more information than higher-variance filters.Source: Transformation φ · p. 4
  • The Frobenius-norm transformation loses less information and supports queries about the effect of setting a filter response to zero.Source: Transformation φ · pp. 4–5
  • Before fitting regressions, the dataset is augmented by randomly zeroing 10% of filters and transforming downstream responses so the SCM learns from interventions.Source: Transformation φ · p. 5

Learning the Structural Equations

  • Binary-transformation structural equations use logistic regression or random forests, with 98% average accuracy; Frobenius-norm equations use linear models and are assessed with average and maximum MSE.Source: Learning the Structural Equations · p. 5

Evaluation

The evaluation checks whether the chosen transformation and structural equations reproduce CNN performance. Inference estimates intervention effects on individual samples or averages them across the dataset.

Sanity Check of the Causal Model

Inference on the Causal Model

  • For an intervention on filters X set to x, the observed filter Y must not be in X; the intervention effect is estimated from the SCM structural equations.Source: Inference on the Causal Model · p. 6
  • Dataset-level intervention effects are estimated by predicting Y for each sample and averaging those estimates across the dataset.Source: Inference on the Causal Model · p. 6

Results

The SCM is evaluated by prediction accuracy and then used for intervention-based filter-importance estimates. Reported accuracy is satisfactory for the listed CNNs, while the binary transformation performs below random choice.

Sanity Check of the Causal Model

Inference on the Causal Model

  • This preliminary work restricts its questions to feature importance for correct classification and uses the Frobenius norm to model setting a filter response to zero.Source: Inference on the Causal Model · p. 6
  • Filter importance is estimated from the model-accuracy drop after setting a filter to zero; a smaller drop corresponds to a lower-ranked filter.Source: Inference on the Causal Model · p. 6

Discussion

The authors propose using the SCM to predict model performance without retraining across compression, transfer learning, and hyperparameter changes, and note potential applications beyond these examples.

Discussion

  • The framework is described as generic and potentially applicable to a wide variety of applications.Source: Discussion · p. 6
  • For model compression, the SCM could predict post-compression performance without retraining after removing nodes or layers.The passage contrasts this with the typical practice of retraining to verify compression loss.Source: Discussion · p. 6
  • For transfer learning, the learnt SCM could predict a pretrained model’s performance on a target task without finetuning its weights.The described setting starts with a model pretrained on a large-scale dataset such as ImageNet.Source: Discussion · p. 6
  • For model accuracy prediction, the learnt SCM could predict accuracy under a modified hyperparameter configuration without retraining.The passage contrasts this with retraining models for each hyperparameter combination during an extensive search.Source: Discussion · pp. 6–7

Conclusion and Future Work

The conclusion presents a causal-inference framework for building, verifying, and using an abstraction of a DNN, illustrated through filter ranking. Future work could strengthen the causal models and extend the fixed-dataset formulation.

Conclusion and Future Work

  • The framework builds and verifies a causal abstraction of a specific DNN aspect, then uses it for inference and filter ranking across CNN layers.Source: Conclusion and Future Work · p. 7
  • Using more expressive models than linear approximations for structural equations could strengthen causal models and support more complex queries.Source: Conclusion and Future Work · p. 7
  • The work uses two basic transformation methods, while other transformations might represent the deep learning model more representatively.Source: Conclusion and Future Work · p. 7
  • The current formulation largely assumes a fixed dataset; extending it to generalize across datasets is proposed as future work.Source: Conclusion and Future Work · p. 7
Loading 1811.04376v1…