PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable Deep Learning: A Field Guide for the Uninitiated

Gabrielle Ras, Ning Xie, Marcel van Gerven, Derek Doran

arXiv:2004.14545v2cs.LGcs.AIstat.ML

TL;DR

Explainable deep learning is difficult to navigate because DNN decisions are opaque and the field lacks uniform definitions of explanation and evaluation. This field guide organizes foundational methods, evaluation approaches, related topics, and user-oriented design into an accessible taxonomy for newcomers. It presents explainability research clearly while identifying practical cautions and future directions.

  • Problem

    DNNs make it difficult to diagnose which input aspects drive decisions, while competing definitions of explanation and evaluation complicate research navigation.

    Source: §1 · p. 2, §3 · p. 35

  • Method

    The paper develops a taxonomy of explainable DNN methods, reviews explanation evaluation and related research, and discusses user-oriented system design.

    Source: §1 · pp. 3–4

  • Results

    The field guide organizes a wide range of explainability methods through a novel categorization scheme and presents associated topics, concerns, and future directions.

    Source: §1 · p. 3, §7 · p. 51

  • Takeaways & Limitations

    The guide provides an accessible starting point for researchers entering explainable deep learning and frames explanation design within broader DNN research concerns.

    Source: §1 · p. 4, §7 · p. 51

    The paper cautions that DNNs can be misused when applied prematurely or incorrectly and should be tested before production or commercial use.

    Source: §1.1 · p. 4

Abstract

from arXiv · show

Deep neural networks (DNNs) have become a proven and indispensable machine learning tool. As a black-box model, it remains difficult to diagnose what aspects of the model's input drive the decisions of a DNN. In countless real-world domains, from legislation and law enforcement to healthcare, such diagnosis is essential to ensure that DNN decisions are driven by aspects appropriate in the context of its use. The development of methods and studies enabling the explanation of a DNN's decisions has thus blossomed into an active, broad area of research. A practitioner wanting to study explainable deep learning may be intimidated by the plethora of orthogonal directions the field has taken. This complexity is further exacerbated by competing definitions of what it means ``to explain'' the actions of a DNN and to evaluate an approach's ``ability to explain''. This article offers a field guide to explore the space of explainable deep learning aimed at those uninitiated in the field. The field guide: i) Introduces three simple dimensions defining the space of foundational methods that contribute to explainable deep learning, ii) discusses the evaluations for model explanations, iii) places explainability in the context of other related deep learning research areas, and iv) finally elaborates on user-oriented explanation designing and potential future directions on explainable deep learning. We hope the guide is used as an easy-to-digest starting point for those just embarking on research in this field.

1. Introduction

This field guide frames explainable deep learning as necessary for trustworthy, accountable use of DNNs and organizes the field for newcomers. It surveys foundational methods, evaluation, related topics, system design, and future directions while defining explanations broadly as information that helps users understand model decisions.

1. Introduction

  • DNNs increasingly affect high-stakes decisions, but accountability requires recommendations supported by explanations that users can interpret, relate to context, and trust.The need is illustrated by medical systems making life-altering predictions and by DNN use across finance, healthcare, law enforcement, and other domains.Source: §1 · p. 2
  • The guide lowers the entry barrier to explainable deep learning by organizing foundational methods, explanation evaluation, related research, end-user system design, and future directions.Its taxonomy targets deep learning specifically, introduces a simple categorization of explanation methods, and connects explainability with transparency, robustness, and reliability.Source: §1 · pp. 2–3

1.1 A Word of Caution

  • DNNs can be beneficial, but they require cautious testing and consideration of alternative algorithms because premature or incorrect deployment can cause harmful outcomes.The paper highlights facial recognition’s lower accuracy for non-white and non-male persons and its harmful real-world use by federal law enforcement agencies.Source: §1.1 · p. 4

1.2 But What is an Explanation?

  • An explanation is information that helps users understand and communicate why a model makes decisions, either by revealing training and generalization or by explaining individual predictions.Prediction explanations commonly relate decisions to inputs and include counterfactual and contrastive forms, while representations vary by data type, such as heatmaps for images, highlighted words for text, and rules for tabular attributes.Source: §1.2 · pp. 4–6

2. Methods for Explaining DNNs

This section organizes foundational methods for explaining DNNs into a simple three-dimensional space, while emphasizing that method advantages depend on the explainability goal and user requirements.

User requirements

  • Relative advantages depend on the explainability goal and the requirements of different users at different software-pipeline stages, including expert users [Ras et al., 2018].Source: §2 · p. 6

Foundational method space

  • The framework categorizes foundational explainable DNN methods into visualization methods, model distillation, and intrinsic methods.These methods are foundational because they are often used in practice or introduce concepts underlying modern approaches.Source: §2 · p. 6

Visualization methods

  • Visualization methods highlight input characteristics that strongly influence a DNN’s output through scientific visualization.Source: §2 · p. 6

Model distillation

  • Model distillation trains a separate white-box model to mimic DNN input-output behavior and identify influential decision rules or input features.Source: §2 · p. 6

Intrinsic methods

  • Intrinsic methods design DNNs to produce explanations alongside outputs, jointly optimizing predictive performance and explanation quality.Source: §2 · p. 6

2.1 Visualization Methods

Visualization methods attribute a DNN decision to input features using saliency maps or heatmaps, divided into backpropagation- and perturbation-based approaches. These methods differ in whether they infer relevance from gradients and internal representations or from output changes after altering inputs.

2.1 Visualization Methods

  • Visualization methods associate input-feature attribution with DNN decisions, commonly displaying salient features as transparent heatmaps over the original input [Yosinski et al., 2015, Ozbulak, 2019, Olah et al., 2017, Olah et al., 2018, Carter et al., 2019].The field guide divides them into backpropagation-based and perturbation-based methods.Source: §2.1 · pp. 6, 8

2.1.1 Backpropagation-based methods

  • Backpropagation methods infer saliency from gradients or reverse-propagated signals, while activation maximization and deconvolution visualize learned internal features rather than individual predictions.Activation maximization optimizes the input with network parameters fixed; deconvolution reverses convolutional and pooling operations to project higher-layer features into input space, supporting analysis of learned concepts, information extraction, training, and generalization.Source: §2.1.1 · pp. 8–9
  • CAM-family methods localize class evidence, whereas LRP, DeepLIFT, and integrated gradients provide pixel-wise or baseline-relative attributions with differing constraints on references, signs, and axiomatic properties.CAM requires a GAP(Conv) →FC →softmax architecture, while Grad-CAM only requires a differentiable final activation; LRP and DeepLIFT can show negative attribution but depend on reference choices, and integrated gradients satisfies sensitivity and implementation invariance.Source: §2.1.1 · pp. 10, 11, 13, 14

2.1.2 Perturbation-based Methods

  • Perturbation methods estimate feature relevance by altering or removing inputs and comparing the resulting network outputs, enabling model-agnostic sensitivity analysis but often trading resolution for computation.Occlusion sensitivity cannot capture co-occurring feature effects because it removes one region at a time; representation erasure can test combined word or phrase removals, and meaningful perturbation produces more concentrated, less spurious heatmaps by minimizing perturbation.Source: §2.1.2 · pp. 15–16
  • Prediction difference analysis measures both positive and negative evidence by conditionally perturbing patches or intermediate activations, while meaningful perturbation keeps changes minimal to yield more interpretable regions.Its improvements include patch sampling, patch removal, and analysis of intermediate layers; the resulting heatmaps indicate regions for and against a predicted class.Source: §2.1.2 · pp. 16–17

2.2 Model Distillation

Model distillation explains an opaque DNN by training an interpretable model to imitate its input/output behavior, providing a surrogate view of features, correlations, and rules. It comprises local approximations for selected regions and model translations covering entire datasets, with forms including trees, automata, graphs, causal models, and rules.

2.2 Model Distillation

  • Model distillation trains an interpretable model g to mimic a trained DNN f, such that g(x) ≈ f(x), and explains f through g’s input/output mapping.The distilled model can reveal discriminatory features, correlations, and relational rules, but not the DNN’s internal representations or learning process.Source: §2.2 · pp. 17–18

2.2.1 Local Approximation

  • Local approximation learns a simple model for a small, similar-input region, motivated by the possibility that DNN behavior is simpler locally than across the full data manifold.LIME samples perturbed inputs, weights them by proximity, and fits a complexity-controlled interpretable model; its local model can be reused for similar inputs, while precision and coverage for unseen instances are not guaranteed.Source: §2.2 · p. 19, §2.2.1 · pp. 19–20

2.2.2 Model Translation

  • Model translation trains a smaller model to replicate a DNN across an entire dataset, potentially improving deployability or convergence and enabling explanation through trees, finite-state automata, graphs, causal models, or rules.Unlike local approximation, translation targets total dataset behavior rather than a small subset.Source: §2.2 · p. 19, §2.2.2 · p. 24

2.2.1 Local Approximation

  • SHAP assigns game-theoretic additive feature contributions, yielding a unique solution for additive attribution methods and enabling global explanations that remain faithful to local explanations.Human-practitioner evaluations found SHAP explanations better aligned with human intuition than several alternatives, including LIME.Source: §2.2.1 · pp. 22–24

2.2.2 Model Translation

  • Translated representations provide objective, structured explanations: finite-state automata expose RNN state transitions, graphs represent relational feature structure, and rules or trees support direct interpretation and downstream decision-system processing.These approaches can clarify internal mechanisms or semantic concepts, but causal-model explanations may be coarse and require practitioner-supplied concept labels.Source: §2.2.2 · pp. 25–26

2.3 Intrinsic Methods

Intrinsic methods integrate explanations into the model architecture, outputs, or training process, providing information during training that post-hoc approaches do not. They mainly use attention mechanisms or jointly trained explanation tasks, although they require specialized knowledge and often greater computational and optimization effort.

2.3 Intrinsic Methods

  • Intrinsic methods make explanations part of the model or its outputs during training, enabling explanation fidelity to influence model design and providing information unavailable from post-hoc derivation.They are generally harder to apply because they require application-specific knowledge, multiple jointly optimized models, and likely more computing power.Source: §2.3 · pp. 26–27

2.3.1 Attention Mechanisms

  • Attention mechanisms produce learned input-weighting visualizations that can explain single-modal processing and multimodal feature alignment while preserving or improving performance on some tasks.Attention weights can be visualized as heatmaps to monitor predictions and possible undesirable correlations, but multimodal systems are more difficult to apply because of their greater component and task complexity; whether attention qualifies as explanation remains debated [Jain and Wallace, 2019, Wiegreffe and Pinter, 2019].Source: §2.3.1 · pp. 27–30

2.3.2 Joint Training

  • Joint training adds an explanation task alongside prediction, generating text explanations, associating model elements with human-understandable concepts, or learning prototypes for traceable reasoning.The objective balances prediction and explanation losses; text explanations are user-friendly but require labeled data and can be inconsistent [Oana-Maria et al., 2019], whereas concept associations and prototypes provide interpretable internal reasoning paths but may require model modifications, specialized datasets, and multi-model pipelines.Source: §2.3.2 · pp. 30, 31, 33, 34

2.4 A Methods Lookup Table

The lookup tables organize explanation methods by category and help practitioners match methods to their data, problem, DNN architecture, and desired explanation form.

A Methods Lookup Table

  • The field guide’s method categories clarify the distinct philosophies used to elicit and express explanations, but categorization alone does not determine the right method for a specific application.The organization is intended to make method classes easier to understand while leaving application-specific selection to the practitioner.Source: §2.4 · pp. 34–35
  • Selecting an appropriate explanation method depends on the DNN architecture, data, problem, and desired form of explanation.The lookup addresses the difficulty of matching explanation methods to particular situations because suitability depends on multiple variables.Source: §2.4 · p. 35
  • Tables 4 and 5 provide a practitioner-oriented lookup organized by explanation-method subcategory, paper reference, data type, problem type, and specific DNN model.The tables compile papers from Figure 3 and use icons to summarize data and problems; their final column identifies the DNN model used.Source: §2.4 · p. 35
  • Practitioners can match their situation to table entries by comparing data type, problem type, and DNN architecture, such as selecting rows marked for images, classification, and CNNs.“No specific requirements” means any DNN is acceptable, while “model agnostic” means the model type does not matter and need not be a DNN.Source: §2.4 · p. 35

3. Evaluating Explanations

Evaluating explanations requires matching evaluation criteria to the user, context, model, data, and desired explanation form, while recognizing that ground truth is often unavailable. Research therefore uses objective benchmarks, human judgments, and task-specific analyses, with results showing that explanation quality and usefulness can vary substantially.

3.1 What Makes a Good Explanation?

  • Good explanations are commonly assessed by fidelity, consistency, stability, and comprehensibility, although the appropriate criteria depend on the user, context, model, data, and explanation form.The lack of ground truth for most cases makes explanation evaluation difficult, and the preferred metric varies with the evaluation goal and user group.Source: §3 · p. 35, §3.1 · p. 38

3.2 Methods for Evaluating Explanation Methods and their Explanations

  • Objective benchmarks compare explanation methods without human intervention, whereas human evaluations assess clarity, parsimony, comprehensibility, importance, and practical usefulness.Objective evaluations have focused especially on heatmaps for image classifiers, with a smaller body of work addressing NLP explanations.Source: §3.2 · p. 38, §3.2.3 · p. 40

3.2.1 Evaluating Heatmaps

  • Heatmap evaluations find that LRP better matches class features than sensitivity analysis and deconvolution, while many visualization methods are unreliable because they lack input invariance.c-Eval also reports quality differences between black-box methods such as SHAP and LIME and back-propagation methods such as LRP and DeepLIFT; sanity checks test sensitivity to model parameters and randomized labels.Source: §3.2.1 · pp. 38–39
  • ROAR addresses distribution-shift concerns by removing heatmap-identified pixels from both training and test images, retraining models, and attributing accuracy drops to removed information.Earlier perturbation-based evaluations replace highlighted regions with random data, while ROAR applies corresponding modifications across both datasets.Source: §3.2.1 · pp. 38–39

3.2.2 Evaluating NLP Explanations

  • NLP studies question attention as a universal explanation: attention often correlates weakly with feature-importance measures, varies across documents and initializations, and only sometimes tracks predictions.Other experiments identify conditions where attention is interpretable, while single-sequence inputs can make attention function as a gating unit.Source: §3.2.2 · pp. 39–40

3.2.3 Using Humans to Evaluate Explanations

  • Human-centered evaluations show that explanation usefulness is not reliably captured by subjective ratings: LIME improves simulatability on tabular models, but ratings did not predict actual usefulness.The perceived usefulness of explanations can differ from their actual usefulness, and findings vary across people, contexts, and needs.Source: §3.2.3 · p. 40
  • Human studies find mixed effects of explanations: BERT-based models have the highest NLI alignment, larger models align worse, explanations can reduce decision time, and LIME or SHAP can slightly reduce accuracy.Explanation complexity influences interpretability, and regularizers can optimize interpretability, while explanation-assisted accuracy differences may be nonsignificant.Source: §3.2.3 · pp. 40–41

4. Topics Associated with Explainability

The paper situates explainable deep learning alongside four related research areas: learning mechanisms, model debugging, adversarial robustness, and fairness and bias. These areas respectively explain training dynamics, diagnose failures, expose vulnerabilities, and constrain undesirable decision behavior.

4. Topics Associated with Explainability

  • Four related research classes connect to explainability: learning mechanism, model debugging, adversarial attack and defense, and fairness and bias in DNNs.They address how models learn, why they fail, how they withstand unexpected inputs, and whether decisions over-emphasize undesirable features.Source: §4 · p. 41

4.1 Learning Mechanism

  • Learning-mechanism research links internal filters or representations to semantic concepts and analyzes training dynamics, layer convergence, generalization, and memorization.TCAV measures how positively a concept influences classification, while SVCCA compares layer representations and shows earlier layers converge faster than later ones; randomization studies examine fitting and generalization.Source: §4.1 · pp. 41–43

4.2 Model Debugging

  • Model debugging uses probes, assertions, interactive inspection, and auxiliary branches to identify data, representation, architecture, and training errors and improve DNNs.Neural stethoscopes can promote or suppress information through a parallel branch, improving network performance and mitigating dataset biases.Source: §4.2 · pp. 43–44

4.3 Adversarial Attack and Defense

  • Adversarial attack and defense studies input-space boundaries and unexpected activation patterns to explain behavior and improve robustness against crafted examples.C&W generates high-confidence attacks against defensive distillation, PGD provides direct distortion control, and GAN-based filtering is effective against black-box and white-box attacks.Source: §4.3 · pp. 45–46

4.4 Fairness and Bias

  • Fairness research addresses bias through group and individual fairness notions, using pre-processing, in-process constraints, or post-processing to reduce discriminatory decisions.These methods respectively transform inputs, penalize unfair decisions during training, or adjust predictions afterward to target goals such as equalized odds and equal opportunity.Source: §4.4 · pp. 46–47

5. Designing Explanations for Users

Useful DNN explanations must be designed case-by-case around the user, task, decision impact, and desired balance between fidelity and parsimony. The section emphasizes user-specific representations, scenario-appropriate computational and evidential requirements, and extendable explanation designs.

5. Designing Explanations for Users

  • Explanation quality depends on user- and context-specific utility, so engineers must refine explanatory methods for each user and task.Users may prefer visual or textual explanations and may prioritize fidelity or parsimony differently.Source: §5 · p. 47

5.1 Understanding the End User

  • User expertise and domain requirements determine whether explanations should expose technical model details or provide high-level reasoning and intuitive domain-specific evidence.No method automatically tailors explanations to a domain; domain-expert knowledge bases may help interpret input features.Source: §5.1 · p. 48

5.2 The Impact of DNN Decisions

  • Time-critical decisions require rapidly generated, simple explanations, whereas decision-critical decisions require trustworthy evidence and deep inspection of failures and root causes.Few current explanations address both constraints because many are computationally costly, labor-intensive, or unsuitable for real-time inputs.Source: §5.2 · p. 49

5.3 Design Extendability

  • Extendable explanation designs should be modular and reusable across models and domains, reducing deployment overhead while helping preserve consistent explanatory fidelity.Modularity supports interchangeable explanatory components, while reusability supports applying established explanations across problem domains.Source: §5.3 · pp. 49–50
  • Model-agnostic explanation methods are modular by definition, while model-specific methods support expert debugging but inherently lack modularity.This trade-off reflects the tension between broad applicability and explanations tied closely to particular model mechanisms.Source: §5.3 · p. 50

6. Future Directions

Future explainable deep learning research should develop a unifying theory, user-friendly and efficient explanations, and methods that improve model trustworthiness. These directions address formalization, accessibility, computational constraints, and vulnerabilities in real-world deployment.

A Unifying Approach to Explainability

  • A systematic theory is needed to unify explainability methods, though formalizing AI explanation concepts remains difficult or impossible.Existing frameworks examine the literature from different perspectives without sufficiently unifying methods [Gilpin et al., 2018, Doshi-Velez and Kim, 2017, Doshi-Velez and Kim, 2018].Source: §6 · p. 50

User-friendly Explanations

  • User-friendly explanations should reduce the technical knowledge required for correct interpretation across users with varied backgrounds.Most current methods target experts, limiting adoption because real-world end users are rarely machine learning specialists [Ras et al., 2018].Source: §6 · pp. 50–51

Producing Explanations Efficiently

  • Efficient explanation generation is needed for time-critical decisions and resource-constrained industrial environments.Reducing computation gives users enough time to react while conserving power where computing resources are limited [Grigorescu et al., 2020, Ahmad et al., 2018, Nemati et al., 2018].Source: §6 · p. 51

Developing Methods for Trustworthiness

  • Trustworthiness research should address DNN vulnerability to adversarial examples and poisoned training sets, especially in decision-critical applications.This research direction is only beginning to develop, despite the potentially severe consequences of undesired decisions [Yuan et al., 2019, Goodfellow et al., 2014, Carlini and Wagner, 2017, Madry et al., 2017, Saha et al., 2019, Jiang et al., 2018, Heo et al., 2018].Source: §6 · p. 51

7. Conclusions

The conclusion frames explainable deep learning as increasingly important for transparency, trustworthiness, and potential legal compliance, while presenting the field guide as a structured introduction to methods, related work, user-oriented design, and future directions.

Motivation

  • Explainable deep learning is motivated by opaque DNN systems and growing societal demands for transparency and trustworthiness, including potential legal concerns under the EU’s GDPR [Goodman and Flaxman, 2017].These concerns arise as deep learning expands across applications such as facial recognition, voice assistance, and driving systems.Source: §7 · p. 51

Contributions

  • The field guide organizes a wide range of explainability methods through a hierarchical categorization scheme while acknowledging that it cannot cover every paper or aspect of the rapidly developing field.This structure is intended to make the field’s overall skeleton clear and straightforward for newcomers.Source: §7 · p. 51

Related Work and Future Directions

  • The guide reviews model learning mechanisms, debugging, adversarial attack and defense, fairness and bias, user-oriented explanation design, and future trends as topics closely associated with DNN explainability.Together, these areas broaden explainability beyond explanation methods alone.Source: §7 · p. 51

Outlook

  • Effective explanations should be delivered to the right person in the right way at the right time, supporting hopes for fairer, safer, and more confident use of deep learning across society.The conclusion emphasizes matching explanations to the needs of their intended audience.Source: §7 · p. 52

Related papers

Loading 2004.14545v2…