PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Neural Network Interpretability

Yu Zhang, Peter Tiňo, Aleš Leonardis, Ke Tang

arXiv:2012.14261v3cs.LGcs.AI

TL;DR

Deep neural networks’ success has heightened concern about their black-box behavior, creating a need to clarify interpretability and its importance for reliable, ethical, and scientific use. The survey synthesizes neural-network interpretability, defines its explanatory basis, and organizes methods into a three-dimensional taxonomy. It concludes that active intervention remains underexplored and that making networks interpretable without harming performance is an open problem.

  • Problem

    Deep neural networks can behave unexpectedly and remain poorly understood, while interpretability lacks a clear consensus definition despite its relevance to reliability, fairness, and scientific use.

  • Method

    The survey clarifies interpretability and reviews methods using a taxonomy spanning passive versus active approaches, explanation format, and local-to-global focus.

  • Results

    The survey provides a comprehensive review, identifies four common explanation types, and presents a three-dimensional categorization of existing neural-network interpretation approaches.

  • Takeaways & Limitations

    Active intervention and making neural networks interpretable without harming performance remain open research problems.

Abstract

from arXiv · show

Along with the great success of deep neural networks, there is also growing concern about their black-box nature. The interpretability issue affects people's trust on deep learning systems. It is also related to many ethical problems, e.g., algorithmic discrimination. Moreover, interpretability is a desired property for deep networks to become powerful tools in other research fields, e.g., drug discovery and genomics. In this survey, we conduct a comprehensive review of the neural network interpretability research. We first clarify the definition of interpretability as it has been used in many different contexts. Then we elaborate on the importance of interpretability and propose a novel taxonomy organized along three dimensions: type of engagement (passive vs. active interpretation approaches), the type of explanation, and the focus (from local to global interpretability). This taxonomy provides a meaningful 3D view of distribution of papers from the relevant literature as two of the dimensions are not simply categorical but allow ordinal subcategories. Finally, we summarize the existing interpretability evaluation methods and suggest possible research directions inspired by our new taxonomy.

I. INTRODUCTION

Deep neural networks have achieved broad success, but their unexpected behavior and opaque mechanisms motivate clearer interpretability research. This survey clarifies interpretability, its importance, and the forms of explanation used to understand neural networks.

  • Motivation: Deep neural networks achieve strong performance across vision, speech, language, and scientific applications, but their millions of parameters can produce unexpected behavior.Examples include adversarial inputs that change predictions imperceptibly and unrecognizable images classified with 99.99% confidence.
  • Motivation: Interpretability research lacks a consensus definition, with prior studies using the term in different or discordant ways.The survey therefore treats clarification of interpretability as a necessary starting point.
  • Definition: The survey defines interpretability as the ability to provide explanations in understandable terms to a human.Explanations may be logical rules or key elements that support constructing explanations, while understandable terms draw on task-related or common knowledge.
  • Definition: The survey emphasizes explanation form and domain knowledge, arguing that representations grounded in task knowledge can be more understandable and evaluable by domain experts.The paper considers rules, saliency masks, and other explanation forms rather than requiring complete natural-language sentences.
  • Importance and scope: Interpretability is relevant to reliable systems, ethical and legal requirements, and scientific knowledge discovery, while this survey focuses on trained neural networks and their inner workings and input-output mappings.The paper distinguishes this scope from broader questions about general neural-network methodology.

B. The Importance of Interpretability

The survey organizes the importance of interpretability around reliable systems, ethical and legal requirements, and scientific knowledge, while positioning its taxonomy as a clearer framework for prior work.

  • High reliability: Interpretability helps detect potential failures and identify their root causes in prediction systems where errors may have catastrophic consequences.The paper states that interpretability does not itself improve reliability or performance, but supports highly reliable system design.
  • Ethical and legal requirements: Interpretability is needed to address algorithmic discrimination and support ethical and legal requirements surrounding automated decisions.The survey discusses bias in training data, fairness in applications, and the right to human intervention and contestation under GDPR.
  • Scientific knowledge: In scientific fields, interpretability can help reveal knowledge represented by deep networks applied to complex data.The paper connects this motivation to applications including drug discovery, genomics, astronomy, physics, and social science.
  • Related work: The survey reviews prior interpretability summaries and identifies a lack of clear consensus on definitions and comprehensive taxonomies.Earlier work is described as offering basic categorization, enumeration, or narrower method groupings.
  • Contributions: The survey groups interpretability needs into highly reliable systems, ethical or legal requirements, and tools for enhancing knowledge in relevant science fields.This organization expands the discussion beyond examples of dangerous black-box behavior.
  • Contributions: The proposed taxonomy organizes approaches by engagement, explanation format, and interpretability scope from local to global.Its three dimensions provide a schematic 3D view of relationships among diverse approaches and suggest future research directions.

D. Organization of the Survey

The survey organizes interpretability methods along three dimensions: engagement, explanation format, and scope from local to global. It uses these dimensions to map the literature in a 3D view and structure the survey.

  • Dimension 1 — Passive vs. Active Approaches: The first dimension separates passive post-hoc explanations of trained networks from active changes to architecture or training for interpretability.Active interventions commonly use regularization terms.
  • Dimension 2 — Type of Explanations: The second dimension orders explanations by format and increasing explanatory power, including rules, feature attributions, hidden-unit interpretations, and examples or prototypes.Logic rules are treated as more explicit, while saliency maps require additional human interpretation.
  • Dimension 3 — Local to Global Interpretability: The third dimension treats interpretability as a continuum from local explanations of individual predictions through semi-local explanations to global decision logic.Semi-local methods apply explanations to neighborhoods or groups of inputs.
  • Examples Across the Taxonomy: The survey illustrates the taxonomy with examples spanning local rules, global rule sets, hidden-neuron semantics, concept sensitivity, and prototype-based explanations.These examples show how explanation type and scope combine within the taxonomy.
  • Organization of the Survey: The three dimensions support a 3D visualization of existing papers, while Table III provides an overview for navigation through the reviewed approaches.The survey proceeds first by passive versus active methods, then by explanation type and interpretability scope.

III. PASSIVE INTERPRETATION OF TRAINED NETWORKS

Most existing network-interpretation methods are passive: they analyze already trained networks and classify the resulting explanations by type.

  • III. PASSIVE INTERPRETATION OF TRAINED NETWORKS: Most existing network-interpretation methods passively analyze already trained networks.The survey introduces these methods according to the types of explanations they produce.

A. Passive, Rule as Explanation

Passive rule-as-explanation methods translate network behavior into logic rules, with approaches covering individual inputs, neighborhoods, or broader input groups. Their representations range from sparse feature conditions to routing patterns and anchors.

  • Rule Extraction: Rule extraction commonly produces global explanations by deriving a single rule set or decision tree from a trained network.A smaller set of methods produces semi-local rule-form explanations.
  • Rule Form: A rule explanation uses an antecedent P composed of feature conditions and a consequent Q representing the network prediction.Complex models may use propositional, first-order, or fuzzy rules.
  • Local Rule Explanations: Local rule methods identify features that are minimally sufficient or necessary for maintaining or changing a target prediction.They use sparse perturbations to find present and absent feature conditions supporting a class decision.
  • Routing-Path Explanations: Critical data routing paths assign gate weights to network channels, but without human-understandable meanings for critical nodes they remain activation patterns rather than full explanations.The method optimizes non-negative channel gates to identify critical paths for each input.
  • Semi-Local Rules: Anchors provide semi-local if-then rules that are sufficiently precise across groups of similar examples.When an anchor applies, the associated predictions are almost always the same.

2) Passive, Rule as Explanation, Global:

Global passive rule extraction seeks an overall interpretation of a network, either by using network structure and weights or by treating the network as a black box that generates examples.

  • Global Rule Extraction: Global rule extraction methods aim to explain the network’s overall decision logic rather than behavior at a single input.They divide into decompositional approaches using network information and pedagogical approaches treating the network as a black box.
  • Decompositional Approaches: Decompositional approaches generate rules by observing network connections, especially in classic fully connected feedforward networks.Rule extraction searches for attribute combinations that make the output approach the target Boolean value.
  • KT Algorithm: The KT algorithm separates positive and negative attributes, booleanizes neurons, and searches combinations that satisfy a predefined output threshold.It then rewrites layer-wise rules to omit hidden neurons when applied to multilayer networks.
  • M-of-N Rules: M-of-N rules emphasize how many expressions are true rather than the individual expressions themselves, enabling an approximately cubic algorithm through weight clustering and network simplification.The method clusters and reassigns link weights, removes unimportant clusters, and retrains the network.
  • Pedagogical Approaches: Pedagogical approaches learn rules or decision trees from examples generated by the neural network, using traditional rule-learning procedures.Examples include sequential covering, CART, and C4.5.
  • Scope Boundary: Earlier global rule methods were usually applied to small datasets, and a single decision tree is unlikely to faithfully approximate increasingly deep networks.More recent rule-as-explanation methods therefore turn toward local or semi-local interpretability.

B. Passive, Hidden Semantics as Explanation

Hidden-semantics methods explain what neurons or layers represent, often by identifying inputs that maximize their activations or by measuring alignment with human-defined concepts. These approaches provide global interpretability but may require regularization because unconstrained optimized inputs can be unrealistic.

  • Meaning of hidden neurons or layers: Hidden-semantics explanations associate abstract concepts with activations of individual neurons or layers, such as animal parts detected by different neurons.By definition, this explanation type provides global interpretability.
  • Activation maximization: Activation maximization searches for an input that maximizes the activation of a selected neuron, channel, layer, or class.For supervised ImageNet classification, the class score before softmax can be maximized.
  • Activation maximization: Unregularized activation maximization often produces unrealistic and uninterpretable images, so priors or regularizers are used to improve visualization quality.High-frequency noise is a major nuisance, motivating regularization choices such as L2 penalties and blurring.
  • Concept alignment: Network Dissection aligns a thresholded, upscaled kernel activation mask with pixel-wise concept masks to quantify kernel–concept correspondence.The alignment uses the intersection and union of the two masks.
  • Meaning of hidden neurons or layers: For NLP, individual neurons can be ranked by linear-model weights and interpreted through salient inputs or highly activating words.The method builds a linear model between hidden neurons and network output before investigating the top-ranked neurons.

C. Passive, Attribution as Explanation

Attribution methods assign signed importance scores to input features or construct local surrogate explanations for individual predictions. The survey distinguishes gradient-related and model-agnostic approaches while emphasizing saturation, implementation dependence, and the gap between plausible saliency maps and faithful explanations.

  • Attribution concept: Attribution assigns each input feature a signed, real-valued importance score reflecting its impact on a prediction.For images, these scores are commonly represented as saliency maps.
  • Gradient-related and backpropagation methods: Gradient-related methods use gradients or backpropagated attribution signals to explain individual classification decisions.A gradient indicates the direction and rate of fastest increase in the loss or class-related output.
  • Gradient-related and backpropagation methods: Plain gradients can fail under saturation, producing zero gradients when the model output is locally constant; reference-based methods such as DeepLIFT address this with a neutral input.The reference input may be a blank image or zero vector, depending on the task.
  • Gradient-related and backpropagation methods: Discrete-gradient methods can depend on network implementation, whereas integrated gradients were proposed to address this issue for functionally equivalent networks.Implementation dependence means equivalent networks can receive different attributions.
  • Evaluation and limitations: Attribution methods can yield plausible saliency maps that remain different from real explanations and can be manipulated by adversarial inputs.Validity requirements include sensitivity, implementation invariance, and input invariance.
  • Model-agnostic attribution: LIME fits an interpretable local model to perturbed samples around an input, balancing fidelity to the network with a complexity penalty.Its intermediate representation can encode interpretable units such as image patches or words.
  • Model-agnostic attribution: Shapley-value attribution assigns each feature a payoff based on its marginal contribution across coalitions, but exact computation has exponential complexity.Sampling over permutations and inputs provides an approximation.

2) Passive, Attribution as Explanation, Global:

Global attribution summarizes patterns across individual explanations or measures how a concept affects model outputs in general. Methods include clustering local attributions, building multilevel explanation trees, and using concept activation vectors.

  • Global attribution: Global attribution can be obtained by clustering individual attributions and summarizing groups of prediction strategies.Related approaches can organize explanations into multilevel trees or combine local attributions from multiple networks.
  • Concept-based attribution: TCAV measures a model’s sensitivity to a user-defined concept represented by a direction in hidden-layer space.The score averages output sensitivity to an infinitesimal movement along the concept vector.
  • Concept-based attribution: Because TCAV explains how a concept affects the output in general, it provides global interpretability.Concepts may be manually selected or discovered by clustering input segments.

D. Passive, Explanation by Example

Explanation-by-example methods support predictions by returning similar, counter, or representative examples. They can operate locally for one input or globally through representative samples across one or more classes.

  • Explanation by example: Explanation-by-example methods return other examples that support or counter a new input’s prediction.A basic strategy selects examples most similar in latent representation.
  • Explanation by example: Example-based explanations are local when they explain one input but can become global when they select representative samples within or across classes.The scope depends on whether the examples represent an individual case or broader class behavior.

1) Passive, Explanation by Example, Local:

Influence-based methods explain an individual prediction by identifying training examples that most affect its loss, revealing which examples support or oppose the prediction.

  • Influence methods estimate how changing a training example affects a model’s prediction on an unseen test example.They first approximate the resulting parameter change, then compute its effect on the test-point loss.

IV. ACTIVE INTERPRETABILITY INTERVENTION DURING TRAINING

Active interpretation methods impose interpretability constraints during training, using regularization or architectural mechanisms to shape global, local, population-level, or semantic explanations.

  • ACTIVE INTERPRETABILITY INTERVENTION DURING TRAINING: Active methods add interpretability restrictions during training, commonly through a regularization term Ω(θ) added to the loss.The weights θ are optimized jointly with the interpretability objective.
  • ACTIVE INTERPRETABILITY INTERVENTION DURING TRAINING: Tree regularization favors networks that shallow decision trees can approximate, enabling global or semi-local interpretability.Regional tree regularization extends the approach in a semi-local way.
  • ACTIVE INTERPRETABILITY INTERVENTION DURING TRAINING: A feature-map regularizer encourages high-layer convolutional filters to represent single, disentangled concepts rather than mixtures of patterns.The method masks feature maps with position-specific templates and adds an extra loss during backpropagation.
  • ACTIVE INTERPRETABILITY INTERVENTION DURING TRAINING: ExpO regularizes local attributions for high fidelity and stability, while DAPr jointly trains a rough feature-importance prior with the prediction model.These methods target local attribution quality rather than decision-tree approximability.
  • ACTIVE INTERPRETABILITY INTERVENTION DURING TRAINING: Dual-net learns population-level feature importance by jointly training a selector network and an operator network.After training, the selector ranks features for an input population.

D. Active, Explanations by Prototypes (global)

Prototype-based active interpretation links predictions to learned examples, while evaluation and taxonomy discussions emphasize task-dependent measurement, explanation complexity, and the continuum from local to global understanding.

  • D. Active, Explanations by Prototypes (global): Prototype layers make predictions according to proximity between encoded inputs and learned prototypes, with regularizers encouraging prototypes to correspond to encoded inputs.The prototype network also uses cross-entropy and reconstruction losses.
  • D. Active, Explanations by Prototypes (global): Interpretability evaluation is difficult because end tasks diverge and may require domain expertise.Application-grounded, human-grounded, and functionally-grounded approaches measure task utility, human judgments, or proxy qualities such as sparsity.
  • D. Active, Explanations by Prototypes (global): Rule-based explanations are often evaluated by extracted-model size, whereas attribution explanations can be assessed through performance degradation after masking salient inputs.Retrained-model evaluation can use a fixed uninformative mask instead of noise.
  • D. Active, Explanations by Prototypes (global): Passive methods are broadly applicable and often efficient, while active methods can incorporate preferred explanations but require compatibility between networks and interpretation methods.Passive methods have difficulty incorporating domain knowledge or priors; active methods intervene during optimization.
  • D. Active, Explanations by Prototypes (global): Local explanations suit individual predictions, whereas global explanations can reveal general knowledge in fields such as genomics and astronomy.The survey treats local and global interpretability as a continuum that can include multilevel explanations.
  • D. Active, Explanations by Prototypes (global): The survey organizes interpretability methods across engagement, explanation type, and interpretability focus, including ordinal positions such as semi-local.It presents this taxonomy as a coherent overview and visualizes approaches in a three-dimensional space.
  • D. Active, Explanations by Prototypes (global): Actively making networks interpretable without harming performance remains an open problem, although recent work is addressing it.The survey identifies active intervention as underexplored and notes that learned representations are not necessarily human-interpretable.
  • D. Active, Explanations by Prototypes (global): Domain-related terms in explanations remain simple, motivating better incorporation of domain knowledge into neural networks.The survey frames task-specific concepts as a continuing research direction.
Loading 2012.14261v3…