PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining Explanations: An Overview of Interpretability of Machine Learning

Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, Lalana Kagal

arXiv:1806.00069v3cs.AIcs.LGstat.ML

TL;DR

Opaque machine-learning systems often lack transparent accounts of their decisions, raising concerns about trust, bias, fairness, and safety. This survey defines interpretability and explainability, organizes prior work with a taxonomy, and reviews explanatory methods, especially for deep learning. It finds that existing approaches address different facets in silos and calls for diverse, purpose-aligned metrics and combinations of techniques.

  • Problem

    Complex opaque algorithms cannot reliably provide insight into their behavior, while explanatory methods lack standardized definitions and systematic assessment.

  • Method

    The paper surveys explanatory AI, defines key concepts, constructs a taxonomy, reviews deep-learning and classical approaches, and proposes best practices.

  • Results

    Reviewed methods span processing, representation, and explanation-producing categories, but processing and explanation-producing approaches are more numerous than representation-based models.

  • Takeaways & Limitations

    Explainability research should combine techniques across categories and use diverse metrics aligned with the purpose and completeness of the targeted explanation.

Abstract

from arXiv · show

There has recently been a surge of work in explanatory artificial intelligence (XAI). This research area tackles the important problem that complex machines and algorithms often cannot provide insights into their behavior and thought processes. XAI allows users and parts of the internal system to be more transparent, providing explanations of their decisions in some level of detail. These explanations are important to ensure algorithmic fairness, identify potential bias/problems in the training data, and to ensure that the algorithms perform as expected. However, explanations produced by these systems is neither standardized nor systematically assessed. In an effort to create best practices and identify open challenges, we provide our definition of explainability and show how it can be used to classify existing literature. We discuss why current approaches to explanatory methods especially for deep neural networks are insufficient. Finally, based on our survey, we conclude with suggested future research directions for explanatory artificial intelligence.

I. INTRODUCTION

The paper frames explainable AI as a response to mistrust and risks in opaque automated decision systems. It distinguishes interpretability from explainability and surveys definitions, taxonomies, and best practices toward more standardized explanatory work.

  • Motivation: Opaque automated systems can produce unreliable, biased, or misleading decisions, motivating mechanisms that explain their behavior.Examples include racial bias in COMPAS predictions and deep networks fooled by inputs unrelated to the true category.
  • Key concepts: Interpretability concerns comprehending what a model did or might have done, but simplifying outputs or identifying dominant classifiers does not resolve every problem of understanding opaque models.The paper presents interpretability as an important first step while noting its limitations.
  • Key concepts: Explainability extends interpretability by requiring models to summarize reasons, answer questions, defend actions, and support auditing.The paper argues that explainable models are interpretable by default, whereas interpretable models are not always explainable.
  • Motivation: Regulatory, inclusion, fairness, and safety concerns have accelerated interest in explanatory systems for black-box models.The paper links this urgency to the EU’s proposed Right to Explanation and concerns about inequality, bias, and safety-critical AI.
  • Contribution: The survey defines key terms, reviews explanatory approaches, constructs a taxonomy, and proposes best practices for standardizing interpretability and explainability research.It focuses mainly on deep-learning models while also reviewing classical AI approaches and open questions for safety-critical applications.
  • Conceptual background: The paper treats explanations as answers to why and why-should questions, reflecting ongoing philosophical debate about what constitutes a good explanation.These questions help characterize when an algorithmic explanation has answered what a person wants to know.

B. Interpretability vs. Completeness

The paper separates interpretability from completeness and identifies a tension between them. Explanations must balance human comprehensibility with accurate, predictive descriptions while avoiding persuasive but misleading simplifications.

  • Definitions: Interpretability describes system internals in terms that humans can understand using vocabulary meaningful to the user.Its success depends on the user’s cognition, knowledge, and biases.
  • Definitions: Completeness describes system operation accurately and increases when an explanation allows behavior to be anticipated in more situations.For a self-contained program, complete explanation could reveal all mathematical operations and parameters.
  • Tradeoff: The central challenge is achieving interpretability and completeness simultaneously because accurate explanations may be hard to understand, while simple ones may lack predictive power.The paper presents these properties as opposing ends of an explanation tradeoff.
  • Evaluation and ethics: Human evaluations that favor simple descriptions can encourage persuasive systems rather than transparent systems.The paper links this concern to ethical questions about manipulating explanations to persuade users.
  • Evaluation and ethics: Simplified explanations are misleading when users cannot understand their limitations or when the simplification hides undesirable system attributes.The paper warns that such explanations may lead users to dangerous or unfounded conclusions.
  • Evaluation and ethics: Explanation methods should be evaluated across a curve trading maximum interpretability against maximum completeness rather than at a single point.The proposed framing permits more detailed and complete descriptions at a possible cost in interpretability.

C. Explainability of Deep Networks

Deep-network explanations reduce the complexity of large computations by focusing on processing, representations, or architectures designed to produce explanations. The survey emphasizes proxy models and salience maps as approaches to make network behavior more understandable.

  • Explanation targets: Processing explanations ask why an input produces a particular output, whereas representation explanations ask what information the network contains.The paper compares these to an execution trace and internal data structures in a computer program, respectively.
  • Explanation targets: Explanation-producing architectures are designed to simplify interpretation of processing, representations, or other aspects of network operation.This is presented as a third approach alongside explaining processing and explaining representations.
  • Scope: The survey focuses on explainable methods for deep neural architectures because the expanding literature cannot be reviewed comprehensively across all subfields.Other subfields are highlighted only briefly.
  • Complexity: ResNet incorporates about 5×10^7 learned parameters and executes about 10^10 floating point operations to classify one image, illustrating the complexity explanations must reduce.The paper uses this scale to motivate methods that summarize only the most relevant computation.
  • Approaches: Proxy models approximate a black-box network with a simpler model, while salience maps highlight a small portion of computation most relevant to a decision.These strategies reduce the explanatory burden without exposing every operation.
  • Approaches: LIME probes a black-box model on input perturbations and fits a local linear proxy to identify influential input regions near the examined input.The method is described as applicable across multiple model types and problem domains.

1) Linear Proxy Models:

Proxy and rule-extraction methods summarize neural-network decisions through simplified models, input-output mappings, or neuron-level rules. These approaches can improve transparency but may sacrifice faithfulness or scalability.

  • Decision-tree proxies: Decision-tree proxies can construct faithful explanations of deep networks, but their size and computational demands limit scalability.DeepRED constructs complete, closely faithful trees, yet requires substantial time and memory.
  • Rule extraction: Rule-extraction methods span taxonomies based on expressive power, translucency, and rule quality.
  • Decompositional approaches: Decompositional methods extract neuron-level if-then rules, but some approaches are exponential and impractical for deep neural networks.The KT method processes neurons layer by layer, while related work includes polynomial-time alternatives and fuzzy-rule transformations.
  • Pedagogical approaches: Pedagogical approaches treat networks as black boxes and infer rules by mapping inputs to outputs, including through sensitivity analysis and sampling.
  • Rule-extraction variants: Other rule-extraction algorithms identify meaningful neurons or inputs by clustering, ignoring insignificant units, or applying decision-tree learning.
  • Limitations: Rule extraction increases neural-network transparency but may not remain truly faithful to the underlying model.

4) Salience Mapping:

Salience mapping explains network behavior by identifying input regions or internal information that influence outputs. Representation analysis examines network structure across layers, units, and concept-aligned directions.

  • Salience mapping: Salience methods balance highlighting high activation regions against highlighting regions whose perturbation most changes the output.Approaches include LRP, DeepLIFT, CAM, Grad-CAM, integrated gradients, and SmoothGrad.
  • Representation granularity: Deep-network representations can be studied at different granularities, from layers and channels to individual neurons and filters.ResNet’s billions of operations are organized into roughly 100 layers with 64 to 2048 channels per pixel.
  • Layer analysis: Internal layers can be evaluated by testing whether their features transfer to problems beyond the network’s original training task.ImageNet-trained features were reused for difficult image-processing problems, including fine-grained bird-species and scene-image classification.
  • Unit analysis: Individual units can be interpreted through visualizations that maximize their responses or through quantitative transfer and segmentation tests.
  • Unit analysis: Network dissection quantifies whether units locate emergent concepts such as objects, parts, textures, and colors.
  • Related methods: Reviews of visual-CNN interpretability cover visualization, diagnosis, disentanglement, explainable models, and semantic middle-to-end learning.
  • Network simplification: Pruning can reveal smaller subnetworks that solve the same problems and may be more interpretable than large networks.

3) Role of Representation Vectors:

Representation-vector methods seek interpretable structure in internal features, while designed architectures use attention or constrained sensitivity to make network behavior easier to explain.

  • Representation vectors: Concept Activation Vectors interpret neural representations by identifying directions in feature space that align with human-interpretable concepts.
  • Explainable architectures: Explainable architectures can use explicit attention, disentangled representations, or generative explanations.
  • Attention networks: Attention-based networks weight inputs or internal features to steer which information becomes visible to later network components.They have been applied to translation, fine-grained image classification, and visual question answering.
  • Attention networks: Explicitly trained attention adjusts and measures input sensitivity so network behavior conforms to desired explanations.
  • Attention networks: This approach is intended to steer internal reasoning and may support sequences of models that discover alternative solution strategies.

2) Disentangled Representations:

Disentangled and explanation-generating systems aim to make internal factors or decisions more human-understandable. The survey also situates these methods within broader evaluation taxonomies and emphasizes distinct explanation focuses.

  • 2) Disentangled Representations: Disentangled representations assign meaningful, independent factors of variation to separate dimensions.Methods discussed include Beta-VAE, InfoGAN, and specialized losses for interpretable convolutional units.
  • 3) Generated Explanations: Explanation-generating systems produce natural-language “because” sentences alongside tasks such as visual question answering and fine-grained image classification.
  • 3) Generated Explanations: Multimodal systems combine visual pointing and textual explanations, and both forms together score well on user-trust and explanation-quality evaluations.Their explanations are generated from the network’s output after the decision has already been made.
  • A. Interpretability: The paper reviews interpretability and explainability across related review literature and other domains.
  • A. Interpretability: Prior work identified the lack of clear definitions and evaluation criteria as interpretability research expanded.
  • A. Interpretability: A prior taxonomy distinguishes application-grounded, human-grounded, and functionally grounded evaluation modes.
  • A. Interpretability: Application-grounded evaluation uses real humans performing real tasks, such as doctors assessing medical diagnosis systems.
  • A. Interpretability: Human-grounded evaluation uses human metrics on simplified tasks when target communities or complete end goals are difficult to access.

B. Explainable AI for HCI

This section reviews data-driven and HCI-oriented explainable AI research, emphasizing literature structure, practical user needs, and emerging research directions.

  • The reviewed work proposes explainable-by-design systems and uses literature analysis to identify relationships among explainability communities.
  • A citation-network analysis reduced explainable-AI literature to 289 core papers and 12,412 citing papers, identifying 28 significant clusters and 9 distinct areas.
  • Topic modeling found intelligence and ambient systems to be the largest, most central, and best-studied network.
  • The analysis highlights FAT algorithms, interpretable machine learning, and theories of explanations as especially important subnetworks.
  • Identified future directions include machine-learning production rules, rigorous usable intelligibility, and broader use of classical AI methods.

C. Explanations for Black-Box Models

This section surveys explanations for opaque machine-learning systems and related planning, storytelling, verbalization, and abductive approaches.

  • A survey taxonomy classifies black-box explanation methods by problem type, explanatory capability, model type, and input-data type.
  • The methods are grouped into explaining black-box models, explaining black-box outcomes, inspecting black boxes, and designing transparent boxes.
  • Explainable planning distinguishes articulating a plan in natural language from explaining the plan itself.
  • Planning explanations address knowledge that a questioner believes the system has but the questioner lacks.
  • Related work connects automatic explanations with story understanding, human-robot verbalization, abductive reasoning, and explanatory coherence.

V. TAXONOMY

The taxonomy organizes explanations by whether they justify outputs, reveal internal representations, or are produced by architectures designed to explain themselves.

  • Some methods provide non-representative justifications for emitted choices to support perceived accuracy and reasonableness.
  • Representation-focused methods explain internal network data structures and how intermediate representations support specific choices.
  • Explanation-producing networks are designed to simplify interpretation of opaque subsystems and improve their transparency.
  • The taxonomy distinguishes approaches that may answer different questions despite addressing the same broad interpretability problem.
  • Table I classifies reviewed methods, with processing and explanation-producing roles more populated than the representation role.
  • The taxonomy promotes evaluation across categories and emphasizes processing, representation, and self-explaining architectural focuses.
  • The authors highlight absent standardized evaluation metrics and propose cross-category research, especially involving explanation-producing architectures.

VI. EVALUATION

This section examines how explanation methods are evaluated, focusing on completeness, faithfulness, substitute tasks, human judgments, and representation-based tests.

  • The three explanation focuses do not share common evaluation criteria, and surveyed work uses several evaluation types.
  • Proxy explanations can be evaluated by how closely they approximate the original model, while substitute-task explanations assess related attributes.
  • Human evaluations assess whether explanations help people predict original-model behavior or reveal model biases.
  • Processing-model explanations should be evaluated locally for faithfulness because globally complex networks may still have understandable local behavior.
  • Salience methods can be evaluated for faithfulness using occlusion experiments that measure sensitivity to input regions.
  • Representation-based methods use transfer tasks and human-understandable concepts to characterize explanatory roles.
  • Concept Activation Vectors can test explanatory power by examining whether activations faithfully reveal specific network biases.

C. Explanation-Producing

Explanation-producing systems are evaluated by alignment with human expectations, but judging explanation reasonableness alone cannot separate model failures from explanation-generator failures. The survey finds that explainability approaches remain siloed and advocates diverse, purpose-aligned metrics and cross-category integration.

  • Explanation-producing systems can be evaluated by comparing their outputs with human attention, known latent variables, test sets, or human judgments.The appropriate comparison depends on the explanation type, such as attention, disentangled representations, or human-readable text.
  • Reasonableness-based evaluation can conflate failures in the underlying model with failures in generating its explanation.An explanation may seem unreasonable because the model reasons poorly or because the generator describes it poorly; conversely, a plausible explanation may be unfaithful.
  • Theoretical understanding and explanation of deep-network processing, representations, and system-level behavior have produced encouraging results.
  • Explainability approaches are siloed, with limited work combining technique categories to produce more effective explanations.
  • The survey recommends diverse evaluation metrics aligned with an explanation’s purpose and desired completeness.
Loading 1806.00069v3…