PaperlayerView exact revision on arXiv

Source-linked AI summary

Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey

Arun Das, Paul Rad

arXiv:2006.11371v2cs.CVcs.AIcs.LG

TL;DR

Deep-learning systems’ black-box nature creates challenges for mission-critical use and limits end-user understanding of model decisions. The paper surveys XAI approaches and evaluates explanation maps, finding that XAI evaluation remains difficult and that human-grounded evaluation is promising.

  • Problem

    Deep-learning models can be difficult to explain, while end-users of API-based services may know only the input and not the model itself, challenging explainability and trust.

  • Method

    The paper discusses XAI approaches and perspectives, describes explanation-map methods, and compares visualizations produced by methods including GradCAM and SHAP.

  • Results

    XAI research is concentrated in model-agnostic post-hoc methods, additive and local surrogate models, and concept-based explanations, while explanation evaluation remains an open challenge.

  • Takeaways & Limitations

    XAI methods should be developed and selected carefully because explanation maps alone may not support decision-making and human interpretation can hinder use in mission-critical applications.

  • Takeaways & Limitations

    Guided Backprop and Guided GradCAM failed checks, suggesting they can generate explanations without proper model training.

Abstract

from arXiv · show

Nowadays, deep neural networks are widely used in mission critical systems such as healthcare, self-driving vehicles, and military which have direct impact on human lives. However, the black-box nature of deep neural networks challenges its use in mission critical applications, raising ethical and judicial concerns inducing lack of trust. Explainable Artificial Intelligence (XAI) is a field of Artificial Intelligence (AI) that promotes a set of tools, techniques, and algorithms that can generate high-quality interpretable, intuitive, human-understandable explanations of AI decisions. In addition to providing a holistic view of the current XAI landscape in deep learning, this paper provides mathematical summaries of seminal work. We start by proposing a taxonomy and categorizing the XAI techniques based on their scope of explanations, methodology behind the algorithms, and explanation level or usage which helps build trustworthy, interpretable, and self-explanatory deep learning models. We then describe the main principles used in XAI research and present the historical timeline for landmark studies in XAI from 2007 to 2020. After explaining each category of algorithms and approaches in detail, we then evaluate the explanation maps generated by eight XAI algorithms on image data, discuss the limitations of this approach, and provide potential future directions to improve XAI evaluation.

I. INTRODUCTION

The survey frames XAI as a response to the opacity and interpretability challenges of deep learning, especially in consequential applications. It organizes XAI methods by explanation scope, implementation methodology, and usage, while synthesizing seminal algorithms, evaluations, and future directions.

  • Motivation: Deep neural networks are difficult to interpret because their many parameters can learn representations that humans may not consider important.Understanding their decisions also requires knowledge of internal operations that non-experts and end-users typically lack.
  • Explanations: XAI explanations can provide feature-level or decision-level information intended to make AI outputs more understandable and support verification of model decisions.Examples include pixel maps for cancer detection, time-frequency information for speech recognition, and surrogate or gradient-based explanations.
  • Motivation: Black-box API-based services intensify explainability challenges because end-users often know only the input and model output, not the architecture or parameters.Intrinsic methods depend on the model architecture, whereas post-hoc methods can be applied to already trained neural networks.
  • Survey scope: The paper adds mathematical summaries of seminal XAI algorithms, a timeline of important publications, and discussion of evaluation strategies, limitations, and future directions.Its literature base covers published research from 2007 to 2020 across multiple scholarly sources.
  • Taxonomy: The survey taxonomizes XAI techniques by scope, methodology, and usage, covering local or global explanations, backpropagation- or perturbation-based methods, and intrinsic or post-hoc application.The taxonomy is intended to improve clarity and accessibility across explainable deep-learning approaches.

III. DEFINITIONS AND PRELIMINARIES

The paper defines interpretability, interpretation, explanation, and model transparency while formalizing deep-learning inference and explanation maps. It distinguishes white-box and black-box settings according to whether model architecture and parameters are available.

  • Model inference: A standard deep-learning inference scenario takes an input x and produces an output ȳ without generating metadata or an explanation.The model is therefore represented as a blob of information whose observable result is the classification output.
  • Model formalization: A deep-learning model maps an input x in R^d to C output classes, with prediction ȳ = f(θ, x).Here, θ denotes model parameters and ȳ denotes the output prediction.
  • Core definitions: Interpretability is a desirable algorithmic quality that provides enough expressive information to understand how the algorithm works.Interpretation is described as simplifying a complex model output into meaningful, human-understandable concepts.
  • Core definitions: An explanation is additional metadata describing the importance or relevance of an input feature to a particular output classification.For images it may be a pixel map, while for text it may be word-level influence scores.
  • Model access: A white-box model exposes its parameters and architecture, whereas a black-box model hides them from the end-user.White-box access can improve debugging and trust, but architecture and parameters alone do not make a model explainable.

A. Why Is Research on XAI Important?

The paper presents XAI as important for trustworthy use of deep learning in high-stakes settings, particularly for transparency, trustability, and bias and fairness. It also emphasizes that explanation evaluation remains challenging and that visual explanations can be insufficient on their own.

  • Why XAI matters: XAI is motivated by ethical, judicial, and safety concerns arising from deep-learning use in healthcare, credit scoring, loan acceptance, and other consequential applications.The paper identifies trustability, transparency, and bias and fairness as the three most important concerns.
  • Transparency: Human-understandable justifications can improve transparency, help assess prediction quality, and potentially deter adversarial examples when used properly.The paper links transparency to understanding whether model outputs are suitable and to warding off adversaries.
  • Evaluation: Typical explanation maps alone might not aid decision making, and human bias in interpreting visual explanations can hinder their use in mission-critical applications.Human-grounded evaluations are described as a promising direction for improving XAI evaluation.
  • Trustability: Explanations can support trust by giving stakeholders reasons for particular decisions, including sub-optimal decisions, rather than only highly confident outputs.The paper specifically identifies end-users, subject-matter experts, developers, lawmakers, and laypersons as relevant stakeholders.
  • Bias and fairness: XAI can help trace prediction discriminations back to input features and thereby support understanding of feature correlations related to class-wise decisions and fairness.The paper presents this as a way to examine biases arising from data or model deficiencies.

1) Activation Maximization:

This section surveys activation- and gradient-based explanations, which visualize influential input patterns, pixels, or features by examining neural-network activations and derivatives.

  • Activation Maximization: Activation Maximization optimizes input patterns to maximize hidden-unit activations, revealing feature importance associated with particular layers.The method can average local minima into an explanation map or select the activation-maximizing result.
  • Saliency Maps: Saliency maps compute output-class gradients with respect to input images to summarize pixel importance through positive gradients.The survey distinguishes class-model visualizations from image-specific class visualizations.
  • Layer-wise Relevance Propagation: Layer-wise Relevance Propagation decomposes DNN predictions into input-feature relevance scores by backpropagating class scores under a conservation property.Relevance is redistributed from output activations toward corresponding inputs and summed for each input feature.
  • Layer-wise Relevance Propagation: LRP explanation maps can support global attribution and model pruning by identifying low-importance weights or filters.The survey reports reduced computation and storage costs without significant accuracy drops in the described pruning work.

4) Local Interpretable Model-Agnostic Explanations (LIME):

LIME produces local, human-understandable explanations by perturbing interpretable input components and fitting a simpler model that approximates the black-box prediction locally.

  • LIME: For images, LIME represents contiguous superpixels with binary presence indicators and estimates their importance for the output class.The approach explains a single input at patch level.
  • LIME: LIME perturbs the input, queries the classifier, measures similarity to the original prediction, and fits a locally weighted interpretable model.The fitted model’s coefficients provide the local explanation.
  • LIME: The LIME objective minimizes a locality-aware loss while balancing the faithfulness and complexity of the interpretable explanation model.The explanation model may be a decision tree, linear model, or another interpretable model.
  • LIME Extensions: Extensions adapt LIME through alternative perturbations, nonlinear quadratic approximations, temporal or frequency segmentation, and divergence-based formulations.QLIME reports improved local MSE over LIME’s linear relationship on a global staffing-company dataset.
  • SHAP: SHAP distributes prediction contributions among features represented as coalition-game players, with variants tailored to model classes and input settings.KernelSHAP reduces evaluations for large inputs, while DeepSHAP adapts DeepLIFT to deep networks.

B. Global Explanations

Global explanations summarize black-box behavior across arrays of inputs rather than individual instances, helping characterize feature attributions and general model behavior.

  • Global Explanations: Globally explainable methods operate on arrays of inputs to describe the model as a whole instead of explaining only individual predictions.The survey presents this design at a high level in Figure 10.
  • Global Explanations: Global explanations support understanding model behavior on large input distributions and previously unseen data.Their scope concerns generalized behavior beyond a single input.
  • Class-Model Visualization: Class-model visualization generates numerically computed images representing a target class, providing insight into what the black-box model learned for that class.The survey illustrates such visualizations for goose, ostrich, and limousine classes.

1) Global Surrogate Models:

Global surrogate and concept-based methods explain model-wide behavior by approximating or interrogating learned representations, while their reliability depends on representative concepts and suitable data.

  • Global Surrogate Models: Global surrogate models approximate highly nonlinear AI models with interpretable linear models or decision trees to answer how generalized model behavior is.A common use case trains a linear classifier on feature-rich layer embeddings.
  • Class Model Visualization: Class Model Visualization generates images that maximize a trained ConvNet’s class score, making learned class representations inspectable.The resulting images are described as representative of the selected class.
  • SP-LIME: SP-LIME aggregates local LIME explanations and uses submodular selection to provide a nonredundant global decision-boundary view.The method selects representative features under a budget.
  • Concept Activation Vectors: CAVs interpret neural-network internal states through human-understandable concept vectors derived from positive and contrasting concept examples.TCAV evaluates class sensitivity to a concept using directional derivatives at a selected layer.
  • Concept Activation Vectors: TCAV scores summarize the influence of concepts on inputs associated with a target class.The score is defined over inputs labeled with that class.
  • Concept Activation Vectors: TCAV may produce meaningless CAVs when concepts are poorly selected, highly correlated, or affected by human concept-selection bias.Random concepts can yield poor binary-classification models, while shared objects can reduce efficiency.
  • Concept-Based Extensions: ACE removes human supervision from concept discovery through multiresolution segmentation, activation clustering, and concept stitching.The described InceptionV3 experiment classified stitched images as the correct class category.
  • Concept-Based Extensions: CaCE and ConceptSHAP address concept confounding or quantify concept importance and completeness using causal effects or Shapley values.ConceptSHAP seeks coherent spatial concept clusters and assigns importance to individual concepts.

5) Spectral Relevance Analysis (SpRAy):

SpRAy aggregates local LRP explanations across samples to reveal global patterns, while NAMs learn additive feature-wise explanations with neural networks.

  • SpRAy: SpRAy applies spectral clustering to local LRP relevance maps to identify normal and abnormal model behavior.The maps are resized, clustered in a high-dimensional space, and analyzed with eigenmaps; t-SNE visualization is optional.
  • SpRAy: Global attribution mapping clusters weighted feature rankings to identify explanation differences among subpopulations and trace them to individual samples.It constructs pair-wise rank-distance matrices and minimizes cluster-distance costs when features have well-defined semantics.
  • Neural Additive Models: NAMs train separate neural networks for individual input features and combine their outputs additively to model nonlinear patterns.They extend generalized additive models and can learn feature jumping that traditional tree-based GAMs cannot learn.
  • Neural Additive Models: Exp-centered hidden units are proposed to help NAMs fit jagged functions that standard ReLU networks with standard initialization may fail to learn.The unit is defined as h(x) = f(ew ∗ (x − b)).
  • Neural Additive Models: NAMs generate exact explanations for each feature space with respect to an output prediction.The survey identifies extending this approach to CNNs and other domains such as text as a future direction.

V. DIFFERENCES IN THE METHODOLOGY

Perturbation-based XAI probes models with modified inputs, whereas convolutional visualization methods and counterfactual tests use masks, occlusions, or feature replacements to expose importance.

  • Perturbation-based methods: Perturbation-based methods iteratively vary input features and observe model responses to explain individual feature attributions.Operations include replacement, occlusion, blurring, shifting, and masking.
  • Perturbation-based methods: LIME generates visual explanations by repeatedly supplying input patches and attributing importance to individual superpixels.The method operates on superpixels or other feature groups.
  • Convolutional visualizations: DeConvNets occlude image segments and map convolutional activations back to input pixels using filters and unpooling operations.This produces activation maps that visualize neural feature activity.
  • Perturbation-based methods: Conditional sampling assigns positive and negative relevance values to input features by measuring changes in predictions for targeted image-classification explanations.The approach uses multivariate conditional sampling for CNNs.
  • Perturbation-based methods: RISE multiplies images by randomized masks and computes a confidence-weighted average to estimate black-box saliency maps.Importance maps are estimated through Monte Carlo sampling.
  • Perturbation-based methods: IRT and OSFT replace features with uninformative counterfactuals to identify contextual importance through hypothesis testing.The survey notes that removing features is difficult for deep-learning inputs with strict dimensionality requirements.

B. BackPropagation- or Gradient-Based

Backpropagation- or gradient-based methods use backward information flow to visualize influential activations or feature attributions, including class-specific localization and context-aware relevance maps.

  • Gradient-based methods: Gradient-based methods use the backward pass to estimate neuronal influence and reshape feature attributions into input dimensions.Their natural advantage is producing human-understandable visual explanations.
  • Gradient-based methods: DeConvNet uses backpropagation for activation visualization and gives relative importance to gradient values during propagation.With ReLU, traditional CNN backpropagation can produce zero values for negative gradients.
  • Class activation methods: CAM localizes class-specific image regions in a single forward pass, while Grad-CAM and GradCAM++ extend it to deeper CNNs and improved visualizations.GradCAM also supports class-specific and counterfactual queries and has been applied to classification, segmentation, and VQA.
  • Salient Relevance: Salient Relevance maps combine LRP with context-aware pixel saliency based on differences in color and spatial position across multiple scales.An attention function incorporates immediate image context into the resulting map.
  • Salient Relevance: The SR algorithm computes an LRP map, generates multiscale saliency, applies an attention function, and returns a relevance map.Its inputs are a classifier, an input sample, and a scale factor.
  • Gradient-based methods: Integrated Gradients addresses desirable gradient-method axioms, while related work includes DeepLift, LRP, and Deconvolution methods.The survey presents these methods as part of broader gradient-based explainability research.

4) Attribution Maps:

Attribution maps assign feature relevance by integrating gradients along paths from baselines, but their interpretation depends on baseline choice and gradient-method desiderata.

  • Integrated Gradients: Integrated Gradients summarizes attributions by integrating gradients along the straight-line path from a baseline x′ to the input x.The integral is approximated by summation in computer implementations.
  • Baseline choice: Zero-valued baselines are common for images and text, but arbitrary baseline choices can distort downstream attributions.A black image baseline may diminish the attributed importance of black pixels in the source image.
  • Attribution methods: Attribution priors and Expected Gradients incorporate domain knowledge or training-distribution expectations to reduce baseline influence and improve interpretability.Expected Gradients reformulates an intractable integration over the training distribution as an expectation.
  • Desiderata: Gradient-based methods are evaluated against sensitivity, implementation invariance, completeness, and linearity desiderata.Sensitivity requires non-zero attribution for a feature that changes predictions; completeness requires attributions to sum to the input–baseline output difference.
  • Desiderata: Linearity requires attributions for f3 = a × f1 + b × f2 to equal the weighted sum of attributions for f1 and f2.The survey also notes practical concerns for gradient-based explanation maps in mission-critical applications.

A. Model Intrinsic

Model-intrinsic explainability embeds interpretable elements in the neural-network architecture, making explanations architecture-specific. The section also contrasts these methods with post-hoc approaches and outlines broader interpretability trade-offs and evaluation requirements.

  • Model Intrinsic: Model-intrinsic methods embed interpretable elements into the model and cannot generally be reused across classifier architectures without redesign.Their explanations depend on the model architecture.
  • Model Intrinsic: Decision lists can improve accuracy and interpretability as IF-THEN rules are added, but explanations deteriorate when the list contains many conditions.Bayesian techniques can learn decision lists from frequent rule patterns.
  • Model Intrinsic: Bayesian Rule Lists optimize rule distributions by iteratively adding and editing rules, after which new rules can be sampled from the posterior distribution.Later work improved scalability through theoretical bounds, computational reuse, and tuned language libraries.
  • Model Intrinsic: GA2Ms were introduced to improve generalized additive model accuracy while maintaining interpretability, although accurate GAMs may require millions of decision trees.Over-regularization can also reduce the accuracy of spline-fitted GAMs.
  • Model Intrinsic: Model-intrinsic architectures require careful algorithm development and fine-tuning, making it difficult to transfer their concepts to existing high-accuracy models.The paper identifies this transfer difficulty as a disadvantage of model-intrinsic methods.
  • Post-Hoc: Post-hoc methods are applied after training and can explain pre-trained models through black-box or white-box access, often without depending on architecture.Examples include saliency maps, LRP, Shapley sampling, and activation maximization.
  • Evaluation: XAI evaluation remains immature, with research focusing primarily on human-in-the-loop studies while quantitative general evaluation schemes remain underexplored.Proposed usability constraints include identity, stability, consistency, separability, and similarity.

A. Evaluation Schemes

The survey presents evaluation schemes for assessing explanation quality, human understandability, and attribution correctness. Its case study compares multiple XAI explanation maps on ImageNet images and illustrates differences among gradient- and perturbation-based methods.

  • Evaluation Schemes: System Causability Scale evaluates requirements for explanations within human-facing AI interfaces, including domain-specific medical interfaces.The cited medical example applies SCS to the Framingham Risk Tool.
  • Evaluation Schemes: BAM evaluates attribution correctness using pasted common features and compares models with model contrast score and input dependence rate.Its hypothesis is that models should ignore identical pasted pixel groups occurring in the same spatial location across inputs.
  • Evaluation Schemes: Faithfulness measures the correlation between feature importance scores and the performance effect of incrementally removing important features.Monotonic attribution functions were also proposed as an evaluation approach.
  • Evaluation Schemes: Human-grounded evaluation compares local explanations with weighted explanation maps summarizing human annotations of ImageNet images.Human annotators manually labeled images for particular classes.
  • Case Study: Figure 19 compares saliency maps, gradient times input, integrated gradients, LRP, DeepLIFT, Grad-CAM, LIME, and SHAP on four ImageNet images explained by a pre-trained InceptionV3 model.The compared methods include both gradient-based and perturbation-based techniques.
  • Case Study: Grad-CAM heatmaps range from 0 to 1, while SHAP values range from -0.3 to +0.3, with positive values increasing and negative values decreasing output-class probability.The visualization implementations differ across gradient methods, Grad-CAM, LIME, and SHAP.
  • Case Study: Integrated gradients visually improve over saliency maps and gradient times input, especially for images with lower class probabilities.For the sandbar example, integrated gradients attribute importance to the beach, chairs, and part of the sky.
  • Case Study: Grad-CAM emphasizes the plastic chair and sandy beach, whereas LIME and SHAP generate superpixels that maximize class probability.The comparison illustrates distinct spatial emphases among explanation-map methods.

D. Limitations of XAI Visualizations and Future Directions

The survey identifies substantial limitations in current XAI visualizations and attribution methods, while highlighting concept-based and human-grounded approaches as directions for more meaningful evaluation.

  • Limitations: Human attention may not reliably use XAI maps for decisions, and no quantitative measure establishes explanation completeness or correctness.A human-grounded study found that adding SHAP explanations to class probabilities did not improve alert-processing decisions.
  • Limitations: LIME produced irrelevant attributions and lower explanation precision than weighted explanations from human annotators.The survey recommends application-grounded, human-grounded, and functionally-grounded evaluation to improve explanation maps.
  • Limitations: Small input perturbations can cause large changes in interpretations generated by popular methods such as DeepLIFT and Integrated Gradients.These adversarial examples can disrupt the saliency maps produced by those methods.
  • Limitations: Integrated Gradients and Deep Taylor Decomposition depend strongly on the reference point, while many gradient methods incorrectly attribute constant vector transformations.The survey argues that reference points should be treated as hyperparameters and input invariances should be prerequisites for reliable attributions.
  • Limitations: Randomization tests found that Gradients and GradCAM passed sanity checks, whereas Guided Backprop and Guided GradCAM generated explanations even without proper training.These tests assess dependence on model parameters and the data-generating process.
  • Future directions: Concept-based explanations add meta-information about factors contributing to class predictions and may improve perturbation- and gradient-based methods.Interpretable Basis Decomposition decomposes explanations by objects or scenes, while PatternNet and PatternAttribution estimate data components related to activations and output classes.
Loading 2006.11371v2…