Source-linked AI summary
Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models
Wojciech Samek, Thomas Wiegand, Klaus-Robert Müller
TL;DR
The paper addresses the opacity of highly capable deep learning models and the need to understand their predictions, especially in consequential applications. It surveys explainability and presents sensitivity analysis and layer-wise relevance propagation, finding that LRP produces more informative explanations than sensitivity analysis in the reported evaluations.
Problem
Deep learning models can achieve high accuracy while their nested nonlinear structure conceals what drives individual predictions, creating a transparency problem in applications such as medicine.
Method
The paper surveys explainability research and presents sensitivity analysis and layer-wise relevance propagation to explain predictions in terms of input variables.
Results
LRP provides more informative heatmaps than sensitivity analysis in the text-document evaluation, while image examples show less noisy, feature-focused LRP explanations.
Takeaways & Limitations
Explainability makes model decisions more accessible for verification, bias detection, human insight, and accountability in consequential AI applications.
Takeaways & Limitations
Sensitivity analysis can assign importance to irrelevant background features and performs poorly in the paper’s quantitative evaluation experiments.
Abstract
from arXiv · showhide
With the availability of large databases and recent improvements in deep learning methodology, the performance of AI systems is reaching or even exceeding the human level on an increasing number of complex tasks. Impressive examples of this development can be found in domains such as image classification, sentiment analysis, speech understanding or strategic game playing. However, because of their nested non-linear structure, these highly successful machine learning and artificial intelligence models are usually applied in a black box manner, i.e., no information is provided about what exactly makes them arrive at their predictions. Since this lack of transparency can be a major drawback, e.g., in medical applications, the development of methods for visualizing, explaining and interpreting deep learning models has recently attracted increasing attention. This paper summarizes recent developments in this field and makes a plea for more interpretability in artificial intelligence. Furthermore, it presents two approaches to explaining predictions of deep learning models, one method which computes the sensitivity of the prediction with respect to changes in the input and one approach which meaningfully decomposes the decision in terms of the input variables. These methods are evaluated on three classification tasks.
1. INTRODUCTION
Deep learning has achieved strong performance across complex tasks, but its nested nonlinear structure makes these models difficult to understand. The paper motivates explainability and introduces sensitivity analysis and layer-wise relevance propagation for explaining individual predictions.
- 1. INTRODUCTION: Deep learning systems excel at complex tasks including image recognition, language understanding, speech processing, and strategic games.
- 1. INTRODUCTION: Their nested nonlinear structure makes highly accurate models non-transparent, leaving the basis of predictions unclear.
- 1. INTRODUCTION: Black-box decisions are especially problematic in medicine and self-driving cars, where predictions require human validation or reliable feature use.
- 1. INTRODUCTION: The paper surveys explainability developments and presents sensitivity analysis and layer-wise relevance propagation for explaining individual predictions in terms of input variables.
2. WHY DO WE NEED EXPLAINABLE AI ?
Explainability supports verification, improvement, knowledge transfer, and legal compliance by making AI decisions more accessible to human experts and affected individuals. These needs are particularly acute when biased data or opaque reasoning can produce harmful decisions.
- Verification of the system: Explainable models enable medical experts to verify predictions instead of trusting black-box systems by default.
- Verification of the system: Explanations can expose correlations that lack causal meaning, such as supervised asthma patients appearing less likely to die from pneumonia.
- Improvement of the system: Interpretability helps identify model weaknesses, dataset biases, and differences between competing models.
- Learning from the system: Explainable systems may transfer patterns learned from millions of examples into insights that humans could not obtain from limited examples.
- Compliance to legislation: Legal responsibility and affected individuals’ requests for reasons create regulatory pressure for more explainable AI systems.
3. METHODS FOR VISUALIZING, INTERPRETING AND EXPLAINING DEEP LEARNING MODELS
The paper presents sensitivity analysis and layer-wise relevance propagation for explaining deep-learning predictions through input-variable importance. Sensitivity uses local gradients, whereas LRP decomposes the prediction into relevance scores conserved during redistribution.
- Sensitivity Analysis: Sensitivity analysis explains predictions using the model’s locally evaluated gradient with respect to each input variable.It assumes features receiving the most sensitive output response are most relevant.
- Sensitivity Analysis: Sensitivity analysis can highlight pixels whose changes alter the prediction without identifying pixels pivotal to the prediction itself.The paper notes that this limitation contributes to poor performance in its quantitative evaluation.
- Layer-Wise Relevance Propagation: Layer-wise relevance propagation decomposes a model prediction backward into relevance scores assigned to input variables such as image pixels.LRP applies to several model families, including feedforward networks, bag-of-words models, LSTMs, and Fisher Vector classifiers.
- Layer-Wise Relevance Propagation: LRP conserves the total relevance, equal to the prediction f(x), at every redistribution step.Thus, the relevance scores represent how much input variables contributed to the function value.
- Layer-Wise Relevance Propagation: The simple LRP rule redistributes relevance according to neuron activation and connection strength, using ϵ to prevent division by zero.Relevance conservation holds when ϵ = 0.
- Layer-Wise Relevance Propagation: The alpha-beta rule enforces conservation through α − β = 1 and coincides with deep Taylor decomposition when α = 1 for ReLU networks.The LRP toolbox implements the method in Python and MATLAB and integrates with Caffe and TensorFlow.
4. EVALUATING THE QUALITY OF EXPLANATIONS
The paper evaluates explanation quality through perturbation analysis: input variables are ranked by explanation scores, perturbed iteratively, and assessed by the resulting prediction decline. Model-independent perturbations are used to avoid biases.
- Perturbing highly important input variables should cause a steeper prediction-score decline than perturbing less important variables.
- Explanation methods provide scores for every input variable, allowing variables to be sorted by estimated relevance.
- Average prediction-score or accuracy decline during iterative perturbation provides an objective explanation-quality measure.A larger decline indicates better identification of truly relevant input variables.
- The evaluation uses model-independent perturbations, such as replacing input values with random samples from a uniform distribution, to avoid biases.
5. EXPERIMENTAL EVALUATION
SA and LRP are evaluated on image, text, and video classification tasks, with LRP producing more interpretable explanations across these settings.
- SA and LRP are evaluated on three problems: image annotation, text-document classification, and human-action recognition in videos.
- Image Classification: In image classification, LRP identifies object features while SA produces noisier heatmaps that can emphasize irrelevant background regions.LRP highlights features such as a coffee cup’s ellipsoidal shape and a volcano’s mountain shape.
- Text Document Classification: In text classification, both methods identify medically relevant words, but only LRP distinguishes positive evidence from words supporting another category.The classifier’s decision was “sci.med”; words such as “sickness,” “body,” and “discomfort” supported that prediction, while other words supported “sci.space.”
- Text Document Classification: LRP heatmaps caused a larger decrease in classification accuracy than SA heatmaps when the most important words were progressively deleted.The evaluation covered 4154 documents from the 20Newsgroup dataset.
- Human Action Recognition in Videos: For video action recognition, LRP identifies both spatially relevant body regions and temporal frames associated with upward and downward movement.The example was a correctly classified “sit-up” video using blockwise motion-vector features.
6. CONCLUSION
The conclusion presents explainability as necessary for validating AI decisions in sensitive contexts and as useful for detecting flaws, improving models, and gaining insights. It also identifies theoretical foundations, learned representations, and new application domains as future-work directions.
- Black-box AI models are unacceptable for some applications, especially medical settings where incorrect decisions can be harmful.
- Explainability is presented as a prerequisite for addressing legal responsibility when AI systems fail.
- The European law’s “right to explanation” is expected to foster explainability in AI systems.
- Explainability can help detect model flaws and data biases, verify predictions, improve models, and generate new insights.
- Future work will study theoretical foundations, links between post-hoc and model-incorporated explainability, learned representations, and applications in new domains.The authors specifically mention generalizability, compactness, explainability, and communications.