PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable Deep Learning: A Field Guide for the Uninitiated

Gabrielle Ras, Ning Xie, Marcel van Gerven, Derek Doran

arXiv:2004.14545v2cs.LGcs.AIstat.ML

TL;DR

DNNs are difficult to diagnose and justify in consequential applications, while explainable deep learning has become broad and fragmented. This field guide organizes foundational methods with a three-dimensional taxonomy, reviews evaluation and related research, and discusses user-oriented design and future directions. It aims to help newcomers navigate the literature and identify research problems.

  • Problem

    DNNs remain black boxes whose input drivers are difficult to diagnose, making satisfactory explanations important for trusting and justifying recommendations in consequential domains.

  • Method

    The guide synthesizes explainable deep learning through a three-dimensional taxonomy of foundational methods, evaluation discussions, related topics, user-oriented design, and future directions.

  • Results

    The guide organizes a broad explainable deep learning literature and connects its methods with evaluation, related research areas, practical design, and future directions.

  • Takeaways & Limitations

    The guide is intended as an easy-to-digest starting point for researchers and practitioners entering explainable deep learning.

  • Takeaways & Limitations

    The paper cautions that DNNs can be misused and recommends testing them carefully and considering alternative algorithms before production or commercial use.

Abstract

from arXiv · show

Deep neural networks (DNNs) have become a proven and indispensable machine learning tool. As a black-box model, it remains difficult to diagnose what aspects of the model's input drive the decisions of a DNN. In countless real-world domains, from legislation and law enforcement to healthcare, such diagnosis is essential to ensure that DNN decisions are driven by aspects appropriate in the context of its use. The development of methods and studies enabling the explanation of a DNN's decisions has thus blossomed into an active, broad area of research. A practitioner wanting to study explainable deep learning may be intimidated by the plethora of orthogonal directions the field has taken. This complexity is further exacerbated by competing definitions of what it means ``to explain'' the actions of a DNN and to evaluate an approach's ``ability to explain''. This article offers a field guide to explore the space of explainable deep learning aimed at those uninitiated in the field. The field guide: i) Introduces three simple dimensions defining the space of foundational methods that contribute to explainable deep learning, ii) discusses the evaluations for model explanations, iii) places explainability in the context of other related deep learning research areas, and iv) finally elaborates on user-oriented explanation designing and potential future directions on explainable deep learning. We hope the guide is used as an easy-to-digest starting point for those just embarking on research in this field.

1. Introduction

Explainability is needed because DNN recommendations in high-stakes domains are difficult to justify, while newcomers face a fragmented field with competing definitions and many methods. This guide organizes the field, its evaluations, related topics, design considerations, and future directions.

  • Motivation: DNN recommendations in domains including healthcare, finance, law enforcement, and insurance raise accountability questions when users cannot access satisfactory explanations.The paper links explanation access to trust and justification for consequential decisions.
  • Motivation: Newcomers may find explainable deep learning daunting because the field contains many directions and competing definitions of explanation and evaluation.The guide is intended to lower the barrier to entering this research area.
  • Guide scope: The field guide introduces dimensions for foundational methods, methods for evaluating explanations, complementary research topics, user-oriented design considerations, and future directions.Its scope spans technical methods and practical concerns for building explainable DNN systems.
  • Guide scope: The guide specifically targets deep learning explanations, uninitiated researchers, a simple categorization scheme, and connections to related research areas.These are presented as distinguishing contributions relative to broader or narrower reviews.
  • A word of caution: DNNs should be tested carefully before deployment, and other algorithms should be considered when they can serve the same purpose.The paper gives facial recognition disparities across demographic groups as an example of harmful premature use.
  • What is an explanation?: An explanation is information that helps users understand why a model exhibits a decision pattern and how individual decisions arise.The paper distinguishes explanations about training and generalization from explanations about individual predictions.
  • What is an explanation?: Explanation representations often match the input data type, such as heatmaps for images and highlighted words for text, but practitioners may interpret them differently.The paper cautions that interpretation depends partly on knowledge of the explanation method.

2. Methods for Explaining DNNs

The guide organizes foundational explainable DNN methods in a three-dimensional space based on how explanations are produced. It covers visualization, model distillation, and intrinsic methods, while noting that method choice depends on users and explainability goals.

  • Taxonomy: Relative advantages among methods depend on the explainability goal and the differing requirements of users at different software-pipeline stages.The discussion takes expert users as its reference audience.
  • Taxonomy: The taxonomy presents a simple three-dimensional space for organizing foundational explainable deep learning work.The dimensions summarize core aspects inspiring or underlying much current explainable DNN research.
  • Visualization methods: Visualization methods highlight input characteristics that strongly influence a DNN output.They commonly use saliency maps or heatmaps to represent feature attribution.
  • Model distillation: Model distillation trains a separate white-box model to mimic a DNN's input-output behavior and identify influential features or decision rules.Decision trees are one example of an interpretable representation used for this purpose.
  • Intrinsic methods: Intrinsic methods are DNNs designed to produce explanations with their outputs and can jointly optimize predictive performance and explanation quality.Their explanations are part of the model or model output rather than derived only after training.

2.1 Visualization Methods

Visualization methods explain DNN decisions by associating input or hidden-state features with saliency scores, commonly shown as heatmaps. They include backpropagation-based methods, perturbation-based methods, and techniques for inspecting learned internal representations.

  • Visualization methods associate feature importance with a DNN decision, often displaying salient image regions through overlaid heatmaps.The visualized element can be model input X or hidden states H, with saliency scores varying by method.
  • Backpropagation-based methods: Backpropagation-based methods use gradient signals to estimate how input features affect the network output.A baseline approach visualizes the output derivative with respect to each input feature, scaled by that feature’s value.
  • Backpropagation-based methods: Activation maximization optimizes the input while keeping network parameters fixed to visualize features that maximize selected units.This can reveal whether learned concepts are human interpretable and provide insight into training and generalization, but not individual predictions.
  • Backpropagation-based methods: Deconvolution reverses convolutional-network operations to project learned filters or higher-layer features back into the input space.It can use stored max-pooling switches during unpooling and supports visualization of filters in any network layer.
  • Backpropagation-based methods: CAM produces class-specific activation maps under a GAP(Conv) →FC →softmax architecture, whereas Grad-CAM uses gradients and applies to a broader range of CNNs.Grad-CAM combines feature-map importance scores, applies ReLU, and upsamples the resulting map to input-image dimensions.
  • Perturbation-based Methods: Perturbation-based methods alter or remove input features and compare the resulting network output with the original prediction.Occlusion sensitivity is model-agnostic but trades finer heatmap resolution for longer computation and cannot isolate jointly occurring features when only one region is occluded.

2.2 Model Distillation

Model distillation explains an opaque DNN by training a more interpretable model to mimic its behavior. Local approximations explain limited regions, while model translations replicate behavior across an entire dataset.

  • Distillation framework: Model distillation trains an interpretable model g to mimic the input/output behavior of an opaque DNN f, so explaining g serves as a surrogate explanation of f.The distilled model approximates f as g(x) ≈ f(x).
  • Local approximation: Local approximation learns a simple model that mimics DNN decisions for a small subset of similar inputs.The approach treats the DNN's behavior within a small area of the data manifold as simpler than its global behavior.
  • Model translation: Model translation trains a smaller alternative model to mimic the DNN across an entire dataset rather than only local subsets.The alternative may be directly explainable, easier to deploy, or further analyzed for insight into replicated input/output behavior.
  • Local approximation: LIME combines model-agnostic local explanations with an interpretable model trained on locally perturbed samples.Once trained, one local LIME model can explain similar inputs without retraining when the input domain remains stable.
  • Local approximation: SHAP represents Shapley-value feature contributions as coefficients in a linear model and can aggregate local explanations into global explanations.With sufficient Shapley values, SHAP can explain any instance while retaining faithfulness to local explanations.
  • Other distilled representations: Distillation into finite-state automata or graphs can expose sequential states or relational input structure in objective, visual forms.Finite-state automata can be represented graphically, while graphs can appear as heatmaps or feature-connection diagrams.

2.3 Intrinsic Methods

Intrinsic methods incorporate explanations into the model or its training process rather than deriving them afterward. The field guide highlights attention-based explanations and jointly trained explanation tasks, while noting added complexity and explanation inconsistencies.

  • Intrinsic explainability: Intrinsic methods make explanations part of the model or its output, integrating explanation generation with model training.This differs from post-hoc methods, which separate model training from explanation generation.
  • Limitations: Intrinsic methods require specialized application knowledge, multiple models can complicate optimization, and generated explanations can exhibit inconsistencies that undermine trust.Explanation association may also require specialized labeled datasets that are difficult to obtain.
  • Method trends: The literature follows two intrinsic-method trends: adding attention mechanisms or jointly training an additional explanation task with the original task.Both trends aim to provide explanations during or directly from the model process.
  • Attention mechanisms: Attention mechanisms learn conditional distributions over input units and produce weights that can be visualized to show feature weighting during inference.In translation, attention weights align encoder and decoder hidden states and can be displayed as heatmaps.
  • Attention mechanisms: Attention heatmaps can help practitioners inspect predictions during training and identify undesirable dataset correlations.In single-modal tasks, attention matrices relate input elements to outputs; multimodal methods align and fuse different feature spaces.
  • Attention mechanisms: Multimodal interaction methods combine attention mechanisms with supplementary tasks to increase interpretability and support application-specific explanations.Their greater combination of components and tasks makes them more difficult to apply than single-modal weighting.
  • Joint training: Joint training minimizes at least a prediction loss and an explanation-component loss, with weighting controlling the balance between prediction quality and explanation quality.The explanation task may provide text explanations or other direct or indirect explanation outputs.

2.4 A Methods Lookup Table

The guide’s lookup tables help practitioners match explanation methods to their DNN architecture, data type, and problem. This addresses the difficulty of choosing an appropriate method for a specific application.

  • Selection challenge: Choosing the right explanation method depends on the DNN architecture, data, problem, and desired explanation form.The taxonomy organizes method classes but does not by itself resolve application-specific selection.
  • Lookup tables: Tables 4 and 5 provide a starting point for selecting explanation methods by organizing papers under their main explanation-method categories.Each table includes subcategory, reference, data type, problem type, and the specific DNN model used.
  • Using the tables: Practitioners can match their data type, problem, and DNN architecture to the corresponding entries in the tables to identify candidate explanation methods.The tables are intended as a practical matching aid rather than a definitive selection rule.

3. Evaluating Explanations

Explanation evaluation combines objective benchmarks with human studies, examining factual quality, fidelity, stability, consistency, comprehensibility, and practical usefulness. Evidence across heatmap and NLP studies shows that explanation quality and interpretability depend on the evaluation method and may not align with model accuracy or user performance.

  • Why Evaluate Explanations: Incorrect explanations can lead to bad outcomes, making factual quality and trustworthiness central evaluation concerns.The paper notes that people often make better decisions with correct explanations, while incorrect explanations can produce bad outcomes.
  • Evaluation Approaches: Explanation evaluation uses objective metrics or benchmarks and human assessments as its two main approaches.Objective methods enable comparisons across explanation techniques, while human studies assess properties such as usefulness and comprehensibility.
  • Evaluating Heatmaps: ROAR modifies training and test images consistently, retrains models, and uses resulting accuracy changes to assess whether removed features caused classification degradation.Applying the same modifications across both datasets addresses concerns that perturbation-based evaluation introduces out-of-distribution images.
  • Evaluating NLP Explanations: Attention weights correlate weakly with feature-importance measures, and their interpretability varies across tasks, documents, and model initializations.Some studies identify conditions where attention provides insight, but single-sequence inputs can make attention function as a gating unit.
  • Using Humans to Evaluate Explanations: Human-centered evaluations found that subjective ratings may not predict usefulness, model alignment can worsen with more parameters, and explanations can reduce decision time while slightly lowering accuracy.The reported findings separate simulatability, alignment, and user-performance outcomes rather than treating interpretability as a single property.

4. Topics Associated with Explainability

Research associated with explainability spans learning mechanisms, model debugging, adversarial robustness, and fairness. These areas examine how DNNs learn, fail, respond to perturbations, and treat different groups, thereby connecting explanation research with broader questions about model behavior.

  • Learning Mechanism: Learning-mechanism research studies parameter evolution, convergence, generalization, and memorization to explain how DNNs reach decision-making processes.Semantic analyses associate internal filters or representations with human-interpretable concepts, while statistical analyses track training dynamics.
  • Learning Mechanism: SVCCA compares layer activation representations during training, and reported results show earlier layers converge faster than later layers.Each neuron is represented by activations across a dataset, and canonical correlation measures alignment between layer representations.
  • Model Debugging: Model debugging applies probes, assertions, and interactive inspection tools to identify architecture, data-processing, training, and prediction errors.Model assertions verify consistency with input-output observations, while ModelTracker supports inspecting outliers and correcting mislabeled samples.
  • Adversarial Examples: Adversarial-example research connects explanation with input-space structure, class boundaries, attacks, and defenses against imperceptible or physical perturbations.Model explanations that are unreasonable given inputs and outputs may indicate adversarial examples.
  • Adversarial Examples: C&W frames adversarial attacks as optimization problems, while adversarial training uses a min-max objective to generate strong examples and improve model robustness.Defense methods also include filtering perturbations by projecting corrupted inputs toward pseudo-original images with GANs.
  • Model Fairness: Fairness methods are commonly organized into pre-processing, in-process, and post-process approaches that modify data, training constraints, or predictions.These methods address bias and sensitive-attribute concerns at different stages of the modeling pipeline.

5. Designing Explanations for Users

Useful explanations require more than foundational methods: engineers must tailor presentation to the user, task, decision impact, and reuse requirements. The guide highlights trade-offs between fidelity, interpretability, computational cost, and modularity.

  • User and task: Explanation presentation should be designed case by case for the intended user and task.Foundational methods are seldom sufficient for explanations that users can apply in practice.
  • User and task: User expertise determines the appropriate trade-off between explanation fidelity and parsimony.Existing designs range from low-level technical explanations for deep learning experts to high-level reasoning for normal users.
  • Decision impact: Decision impact determines whether explanations should prioritize rapid processing or deep inspection.Time-critical settings require simple, computationally efficient explanations, whereas decision-critical settings require trustworthy evidence and post-hoc inspection.
  • Decision impact: Many current explanations are poorly suited to time- and decision-critical scenarios because they are computationally costly or require extra human effort.The guide identifies this as an under-investigated practical concern.
  • Design extendability: Extendable explanation designs should support multiple users, problem types, or DNN models rather than serving only one narrow setting.Modularity reduces implementation and deployment overhead, while reusability can maintain consistent explanation fidelity across applications.
  • Design extendability: Model-agnostic explanation methods are modular by definition, while model-specific methods support expert debugging but lack modularity.The literature has not generally focused on modularity or reusability as design goals.

6. Future Directions

The guide identifies future work aimed at unifying explainability theory, making explanations accessible to nonexperts, producing them efficiently, and improving trustworthiness. These directions respond to persistent gaps in general theory, usability, real-time deployment, and DNN vulnerability.

  • A Unifying Approach to Explainability: A systematic general theory for explaining DNNs remains lacking because existing frameworks examine the literature from different perspectives.The guide calls for a unifying approach to explainability.
  • User-friendly Explanations: User-friendly explanations are needed because most current methods cater to experts and require knowledge of DNN processes.The guide links this direction to varied user backgrounds and increasing societal and legal attention to opaque models.
  • Producing Explanations Efficiently: Efficient explanation generation is important for time- and decision-critical settings and for applications with limited computing resources.Efficiency can also reduce computational power requirements in industrial applications.
  • Developing Methods for Trustworthiness: Trustworthy explanation methods are needed as adversarial examples and poisoned training sets raise concerns about DNN trustworthiness.The demand is especially relevant in decision-critical scenarios where undesired decisions may have severe consequences.

7. Conclusions

The guide frames explainable deep learning as a response to opaque DNNs and growing demands for transparency and trustworthiness. It organizes methods and related topics for newcomers while acknowledging that rapid field growth prevents complete coverage.

  • 7. Conclusions: Opaque DNN systems and increasing societal demand for transparency and trustworthiness motivate explainable deep learning.Government policies such as the GDPR also point toward explainability becoming a legal concern.
  • 7. Conclusions: The field guide organizes explainable deep learning methods through a novel categorization scheme and reviews related topics including debugging, adversarial attacks, fairness, and bias.It is intended to provide an accessible overview for researchers beginning in the field.
  • 7. Conclusions: The authors acknowledge that the guide cannot cover every paper or every aspect of the rapidly developing field.Hierarchical categories are used to visualize the field’s underlying structure.
  • 7. Conclusions: The guide’s concluding aim is to support appropriate explanations for the right audience and contribute to fairer, safer, and more confident use of deep learning.The stated goal is to match explanations to the audience that needs them most.
Loading 2004.14545v2…