Source-linked AI summary
Towards Explainable Artificial Intelligence
Wojciech Samek, Klaus-Robert Müller
TL;DR
Black-box deep learning models can provide little transparency into their predictions, creating problems where trust and verifiability matter. This introductory paper surveys explainable AI developments and applications, highlighting how explanations can support understanding, verification, and trust while identifying remaining challenges.
Problem
Deep learning models’ nested nonlinear structure limits transparency into why they make predictions, which matters especially in applications requiring trust and verifiability.
Method
The paper presents an introductory survey of recent explainable artificial intelligence developments, applications, and explanation methods.
Results
The paper shows that explanations can help verify individual predictions, establish trust, uncover Clever Hans predictors, and foster understanding of classifiers and problems.
Takeaways & Limitations
Explainable learning algorithms can support transparent decision making and should receive wider practical attention, particularly where AI decisions require trust and verification.
Takeaways & Limitations
Explanations often remain at a low abstraction level, requiring human interpretation that can be difficult and erroneous.
Abstract
from arXiv · showhide
In recent years, machine learning (ML) has become a key enabling technology for the sciences and industry. Especially through improvements in methodology, the availability of large databases and increased computational power, today's ML algorithms are able to achieve excellent performance (at times even exceeding the human level) on an increasing number of complex tasks. Deep learning models are at the forefront of this development. However, due to their nested non-linear structure, these powerful models have been generally considered "black boxes", not providing any information about what exactly makes them arrive at their predictions. Since in many applications, e.g., in the medical domain, such lack of transparency may be not acceptable, the development of methods for visualizing, explaining and interpreting deep learning models has recently attracted increasing attention. This introductory paper presents recent developments and applications in this field and makes a plea for a wider use of explainable learning algorithms in practice.
1 Introduction
Machine learning and deep learning now achieve strong performance across complex scientific and industrial tasks, but their opacity limits adoption where decisions must be understood and verified. This paper introduces explainable AI as a response to that transparency challenge.
- 1 Introduction: Machine learning systems have become important across everyday applications, science, and industry.Their capabilities include visual recognition, language processing, speech understanding, prediction, simulation, and exploration.
- 1 Introduction: Large datasets, methodological advances, and GPU-based computational gains enabled recent AI successes.
- 1 Introduction: Deep learning’s nested nonlinear structure makes it difficult to determine what drives individual predictions.This opacity is a central reason powerful models are commonly treated as black boxes.
- 1 Introduction: Lack of transparency and explainability reduces trust in and verifiability of AI decisions.The problem is especially consequential in applications such as medicine and autonomous driving.
- 1 Introduction: The paper surveys recent explainable-AI developments and argues for broader practical use of transparent decision-making methods.It covers explanation needs, explanation types, methods, and evaluation questions.
2 Need for Transparency and Trust in AI
Transparency matters because AI systems can exploit spurious correlations, undermine verification, and create legal or ethical concerns. Explanations can reveal such behavior, support trust, and help people understand AI decisions.
- 2 Need for Transparency and Trust in AI: In safety-critical applications, opaque AI decisions may be limiting or disqualifying because errors can endanger people.Transparency requirements are lower for consumer applications whose failures have relatively unspectacular consequences.
- 2.1 Explanations Help to Find “Clever Hans” Predictors: AI models may achieve test-set success by exploiting spurious contextual cues rather than the intended object or concept.Examples include boats associated with water, trains with rails, horses with copyright watermarks, and wolves with snow.
- 2.1 Explanations Help to Find “Clever Hans” Predictors: Explanations help detect dataset or model biases and expose classifier misbehavior, potentially from a single test image.
- 2.2 Explanations Foster Trust and Verifiability: Verifying AI decisions can foster trust in supportive systems such as medical diagnosis and autonomous systems such as self-driving vehicles.In medical settings, explanations can give experts additional information for treatment decisions.
- 2.2 Explanations Foster Trust and Verifiability: Explanations also support human-machine interaction by helping people accept difficult decisions and participate in informed consent.
- 2 Need for Transparency and Trust in AI: AI’s scientific value includes discovering associations that are inaccessible to human experts, motivating interpretable analysis of learned patterns.
- 2 Need for Transparency and Trust in AI: Legal, ethical, responsibility, autonomy, fairness, and anti-discrimination concerns have increased demand for human-understandable AI decisions.The GDPR is described as including a right to explanation in specified articles.
3 Different Facets of an Explanation
Explanations differ in information content, recipient, and purpose, so useful explanations must be matched to the intended audience and question. They can describe learned representations, individual predictions, or broader model behavior.
- Recipients and purposes: Different recipients require different explanation detail, from coarse user-facing summaries to deeper developer-oriented insights.Medical examples include simple patient explanations and more elaborate explanations for medical personnel.
- Information content: Explanation types range from learned representations and individual predictions to distinct prediction strategies and overall model behavior.
- Explaining learned representations: Learned-representation explanations investigate what individual neurons or groups of neurons encode.Prototype construction is presented as one way to study internal representations.
- Explaining individual predictions: Individual-prediction explanations use heatmaps to show pixels that were relevant or sensitive for a model’s decision.Such explanations can help verify predictions and establish trust in correct system functioning.
- Recipients and purposes: Explanation methods answer different questions, including what a model predicted and what evidence supports or opposes that prediction.LRP can adjust the amount of positive and negative evidence shown.
- Recipients and purposes: Explanations may also support model improvement, compression, pruning, certification, and legal use, although several applications remain future work.
4 Methods of Explainable AI
The paper organizes explainable-AI methods by how they access or approximate model behavior: surrogate models, local changes, internal structure, and aggregated meta-explanations. It emphasizes major developments rather than a complete survey.
- Surrogate functions: Model-agnostic surrogate methods locally approximate complex predictors with interpretable functions, as in LIME.LIME samples inputs near the case of interest, evaluates the neural network, and fits a local surrogate.
- Explaining with local perturbations: Local-change methods include gradient-based, perturbation-based, and optimization-based approaches.
- Explaining with local perturbations: Sensitivity Analysis explains changes in predictions rather than predictions themselves and can suffer from gradient shattering and discontinuities.Locally averaged gradients and path integration are described as variants addressing some problems.
- Explaining with local perturbations: Perturbation methods test responses to local changes by masking input regions or conditionally sampling to remove feature information.Occlusion and Prediction Difference Analysis are described as model-agnostic methods.
- Leveraging structure: Propagation-based methods incorporate model structure into explanation generation rather than treating the model as opaque.
- Leveraging structure: LRP propagates an output prediction backward to the input through local redistribution rules and applies to several neural and other model structures.Its structure-aware rules enable explanations at low computational cost, using one forward and one backward pass.
- Leveraging structure: Other structure-based methods identify input-space patterns related to outputs rather than quantifying feature contributions to predictions.Deconvolution and Guided Backpropagation are examples.
- Meta-explanations: Meta-explanations aggregate individual explanations or inspect hidden-unit semantics to study model-wide behavior and human-friendly concepts.SpRAy clusters heatmaps, while network dissection evaluates concepts encoded by hidden units.
5 Evaluating Quality of Explanations
The paper surveys objective and indirect approaches for evaluating explanation quality, including perturbation tests, localization games, task-specific checks, functional uses, and axioms.
- Perturbation-based evaluation: Perturbation analysis evaluates heatmaps by testing whether altering highly relevant input variables causes a steeper prediction-score decline than altering less important variables.The average decline across perturbation rounds, beginning with the most relevant variables, serves as the objective measure.
- Localization-based evaluation: The pointing game tests whether a heatmap’s most relevant point falls on the target object, assuming the model focuses on that object during classification.This evaluates the discriminativeness of explanations for localizing objects of a designated category.
- Task-specific and human evaluation: Task-specific schemes compare explanations with known task structure, such as checking whether NLP relevance identifies the correct subject or a noun with the predicted number.Other evaluation schemes rely on human judgment, which can become impractical for many explanations.
- Ground-truth evaluation: Ground-truth toy tasks can evaluate sequential explanations by correlating assigned relevances with the two input numbers required for addition or subtraction.Because the remaining sequence numbers are random, accurate task performance should correspond to focusing on the relevant pair.
- Functional evaluation: Indirect evaluations assess explanations through downstream tasks, including document classification from word-level explanations and reinforcement learning.These functionally grounded evaluations use the performance of another task as an indicator of explanation quality.
- Axiom-based evaluation: Axiom-based evaluation checks whether explanations satisfy properties such as relevance conservation, explanation continuity, sensitivity, and implementation invariance.Unlike other measures, axioms define properties considered necessary for explanations to fulfill.
6 Challenges and Open Questions
Despite progress, explainable AI still faces methodological, theoretical, and practical challenges. Open questions include richer and more abstract explanations, human-centered use, formal theory, and applications beyond visualization.
- Section 6 Challenges and Open Questions: Explainable AI continues to face challenges on the methods, theory, and practical-use sides, with fundamental theoretical questions requiring more time for satisfactory answers.Objective evaluation and uses beyond visualization are among the challenges researchers have already begun addressing.
- Section 6 Challenges and Open Questions: Current heatmaps provide first-order feature relevance but do not clarify whether features matter independently or through interactions.Higher-order explanations could identify groups of brain regions that act together rather than only important individual voxels.
- Section 6 Challenges and Open Questions: Explanations remain limited by their low abstraction level because pixel-level relevance does not directly express objects or scenes.Users must interpret these heatmaps to understand model behavior, and that interpretation can be difficult and erroneous.
- Section 6 Challenges and Open Questions: Meta-explanations are proposed as a way to aggregate low-level heatmap evidence into more abstract, human-understandable accounts of model behavior.Initial approaches aggregate low-level explanations and quantify neural-representation semantics, while more advanced meta-explanations remain future work.
- Section 6 Challenges and Open Questions: Optimizing explanations for human-machine interaction remains challenging because successful interaction requires the right user focus and appropriately framed questions.Human factors in explainable AI have begun to receive attention, but explanation use is still an open research topic.
- Section 6 Challenges and Open Questions: A formal, universally agreed theory and definition of explainable AI is still lacking.Taylor decomposition and axiomatic approaches are identified as mathematically grounded directions toward such a theory.
- Section 6 Challenges and Open Questions: Using explanations beyond visualization remains a wide-open challenge, including integrating them into optimization processes to improve performance or reduce complexity.The paper identifies this integration as a direction for future work rather than an established capability.