Source-linked AI summary
On Interpretability of Artificial Neural Networks: A Survey
Fenglei Fan, Jinjun Xiong, Mengzhou Li, Ge Wang
TL;DR
Deep neural networks are powerful but difficult to explain, creating a barrier to their use in mission-critical applications. This survey develops a taxonomy and systematically reviews interpretability methods, applications, and future directions. It covers model explanations, mathematical analyses, medical applications, and connections with rule systems and brain science.
Problem
Deep neural networks operate as black boxes, making their mechanisms difficult to explain and limiting their acceptance in mission-critical applications, especially healthcare.
Method
The paper systematically surveys deep-neural-network interpretability using a comprehensive taxonomy covering methods, applications, examples, and future research directions.
Results
The review presents a holistic landscape of interpretability research, including medical applications, mathematical and physical analyses, representative methods, and open-source implementations.
Takeaways & Limitations
Interpretability research spans model understanding, quality assessment, medical use, and prospective integration with rule systems and brain science.
Abstract
from arXiv · showhide
Deep learning as represented by the artificial deep neural networks (DNNs) has achieved great success in many important areas that deal with text, images, videos, graphs, and so on. However, the black-box nature of DNNs has become one of the primary obstacles for their wide acceptance in mission-critical applications such as medical diagnosis and therapy. Due to the huge potential of deep learning, interpreting neural networks has recently attracted much research attention. In this paper, based on our comprehensive taxonomy, we systematically review recent studies in understanding the mechanism of neural networks, describe applications of interpretability especially in medicine, and discuss future directions of interpretability research, such as in relation to fuzzy logic and brain science.
I. INTRODUCTION
Deep neural networks achieve strong performance across major domains but remain difficult to explain, limiting their acceptance in mission-critical applications. This survey addresses the gap through a taxonomy, systematic review, medical applications, and perspectives on interpretability.
- Motivation: Deep learning performs well across text, images, videos, and graphs, yet its underlying mechanisms and behaviors are difficult to explain.The resulting questions include how predictions are made, why features are favored, and how performance might be improved.
- Motivation: Interpretability is especially important in healthcare because it relates to robustness, vulnerability analysis, accuracy, reliability, and ethical accountability.The survey also connects interpretability with patient trust and responsible use of deep learning.
- Motivation: Limited interpretability is a barrier to adopting deep learning in mission-critical applications, while regulations may grant affected individuals a right to explanations.The authors argue that rapid developments in the field require an up-to-date review.
- Scope and contributions: The review separates post-hoc explanations from ad-hoc interpretable model construction and focuses specifically on deep neural networks.It additionally covers generative models, mathematical and physical analyses, medical applications, examples, and publicly available code.
- Scope and contributions: The survey organizes its discussion around what interpretability means, why it is difficult, and how to build a good interpretation method.It adopts existing discussion for the first question, complements prior work on the second, and provides its own perspectives on the third.
A. What Does Interpretability Mean?
Interpretability lacks a universally agreed meaning, and the survey frames it as human ability to understand and reason about a model at multiple levels. These levels include whole-model simulation, component analysis, and training-process transparency.
- Interpretability meaning: Interpretability generally refers to the extent of human ability to understand and reason about a model.The lack of consensus on its exact meaning contributes to the diversity of current interpretation methods.
- Interpretability levels: Simulatability concerns understanding the entire model through a unified theoretical framework.The survey gives radial basis function networks expressed as regularized interpolation solutions as an example.
- Interpretability levels: Decomposability analyzes a model through components such as neurons, layers, and blocks.This modularized analysis treats complicated systems as combinations of functionalized modules.
- Interpretability levels: Algorithmic transparency concerns understanding a model’s training process and dynamics.Highly non-convex objectives and non-unique deep-model solutions make this form of understanding difficult.
B. Why Is Interpretability Difficult?
The survey attributes difficulty in interpreting neural networks to limits in human expertise, commercial incentives for opacity, wild real-world data, and algorithmic complexity. These barriers constrain both model understanding and interpretation construction.
- Sources of difficulty: Human expertise can be insufficient for intricate problems that even professionals cannot adequately comprehend.The survey illustrates this barrier with neural-network prediction of pseudo-random sequential digits.
- Sources of difficulty: Commercial incentives encourage companies to preserve black-box models for profit and to protect proprietary work from reverse engineering.Opacity can allow customers to receive satisfactory results while limiting competitors’ ability to copy intellectual property.
- Sources of difficulty: Heterogeneous, inconsistent, high-dimensional real-world data hamper model accuracy, interpretability construction, and reasoning.The survey notes that even MNIST requires mapping 784 input variables to one of ten digits.
- Sources of difficulty: Large-scale nonlinear operations, many trainable parameters, recursion, and chaotic behavior complicate interpretation of deep models.Tiny changes in initial inputs can produce large outcome differences in chaotic systems.
C. How to Build a Good Interpretation Method?
The survey proposes evaluating interpretation methods through five rules-of-thumb while reviewing diverse approaches and future directions. It emphasizes exactness, consistency, completeness, universality, and reward, alongside connections to medicine, rule systems, and brain science.
- C. How to Build a Good Interpretation Method?: The survey proposes five criteria for interpretation quality: exactness, consistency, completeness, universality, and reward.These rules-of-thumb are described as fine-grained and focused on interpretation-method characteristics.
- C. How to Build a Good Interpretation Method?: Exactness concerns interpretation accuracy, with quantitative analysis generally preferred over qualitative description.The criterion asks whether an explanation is limited to qualitative description or includes quantitative analysis.
- C. How to Build a Good Interpretation Method?: Consistency requires non-contradictory explanations across similar samples and agreement with the authentic model’s predictions.Proxy methods are evaluated by how closely they replicate the original model.
- C. How to Build a Good Interpretation Method?: Completeness concerns effectiveness across the maximal number of data instances and data types.Universality asks whether one interpreter can handle many model types, although model variability makes this challenging.
- C. How to Build a Good Interpretation Method?: Reward concerns gains from understanding neural networks, including practitioner and user trust and insights into network design and training.The survey frames understanding as relevant to both neural-network research and applications.
- C. How to Build a Good Interpretation Method?: The review contributes a taxonomy, educational illustrations with examples and code, and future directions involving rule systems, brain science, and medical interpretability.Its conclusion calls for a unified and accountable interpretation framework and reports open-sourced implementations.
II. A SURVEY ON INTERPRETATION METHODS
The review searches broadly across major scholarly databases to cover important interpretability papers, documenting exponential growth in the field.
- The authors search Web of Science for four interpretability-related terms from 2000 to 2019 and supplement it with Google Scholar, PubMed, and IEEE Xplore.The Web of Science search was conducted on September 22, 2020, without limiting the review to a narrow subset of papers.
- The number of interpretability articles shows an exponential trend over the reviewed period.
A. Taxonomy Definition
The taxonomy separates post-hoc analysis of existing models from ad-hoc construction of interpretable models, while organizing diverse techniques and their limitations. It also adds advanced mathematical/physical analysis as a distinct class and acknowledges that methods may span multiple categories.
- Taxonomy Definition: The taxonomy first divides methods into post-hoc interpretability analysis and ad-hoc interpretable modeling.Post-hoc methods explain existing models, whereas ad-hoc methods construct interpretable ones.
- Post-hoc Interpretability Analysis: Post-hoc analysis includes feature analysis, model inspection, saliency, proxy, mathematical/physical, case-based, and text-based explanation methods.
- Post-hoc Interpretability Analysis: Post-hoc methods avoid directly compromising predictive performance, but their explanations are usually not completely faithful to the original model.
- Post-hoc Interpretability Analysis: Feature analysis offers qualitative insights but lacks a rigorous unified understanding, whereas model inspection extracts structural and parametric information with analytical tools.Model inspection can use data-routing paths to support faithful compression of a model into a compact one.
- Post-hoc Interpretability Analysis: Saliency methods identify input attributes relevant to predictions, but random tests found that some maps are independent of both the model and the data.Such maps can resemble outputs from edge detectors, undermining their ability to identify prediction-relevant attributes.
- Post-hoc Interpretability Analysis: Proxy methods approximate complex black-box models with simpler interpretable models, while case-based methods provide representative examples but may function mainly as sanity checks.Proxy construction incurs extra cost, and case-based explanations reveal limited information about a network’s inner workings.
- Post-hoc Interpretability Analysis: Text-based explanations are useful for image-language tasks but require a language module and therefore are not general for every deep learning model.The paper gives generating a diagnostic report from an X-ray radiograph as an example.
- Ad-hoc Interpretable Modeling: Ad-hoc modeling includes interpretable representations and model renovation through regularization or purpose-designed network components.Interpretable representations use properties such as sparsity, decomposability, and monotonicity, while renovation inserts interpretable machinery into networks.
B. Post-hoc Interpretability Analysis Feature Analysis
Feature analysis examines neural networks through learned features, attribution methods, data-processing paths, and interpretable proxies. The reviewed methods include saliency, statistical dependence, rule extraction, distillation, and case-based explanations, while also exposing limitations in robustness, faithfulness, and theoretical understanding.
- Feature Analysis: Neuron activations and learned representations reveal how networks respond to different inputs and whether independently initialized models learn similar features.Neuron-level inspection compares activation values across layers, inputs, and initializations.
- Feature Analysis: Control gates expose important data-processing paths, while concept activation vectors quantitatively measure associations between learned representations and human-defined concepts.The control-gate formulation uses an l1 norm to encourage sparse gates.
- Feature Analysis: PDP fixes selected variables and averages predictions over changing complementary variables, whereas ICE varies the selected variable for individual instances while fixing the others.These model-agnostic tools visualize dependence between responsible variables and predictions.
- Feature Analysis: Gradient-based saliency methods assign input importance, with Integrated Gradient and Deep Taylor producing sharper and less noisy maps than raw gradients and SmoothGrad in a LeNet-5-like evaluation.Saliency methods also motivate requirements such as sensitivity and implementation invariance.
- Feature Analysis: Proxy methods translate neural networks into more interpretable forms through rule extraction, knowledge distillation, and local behavioral approximation.Rule extraction discretizes hidden activations and clusters inputs, while distillation uses informative soft labels from complex models.
- Feature Analysis: Case-based explanations identify influential or harmful examples and construct counterfactuals by finding minimal changes that produce a different prediction.Influence functions can identify harmful training images, while counterfactual explanations seek a closest possible case with a different output.
C. Ad-hoc Interpretable Modeling Interpretable Representation
Ad-hoc interpretable modeling designs representations and network components whose structure makes learned concepts, decisions, or computations more understandable. Approaches include semantic latent codes, constrained representations, explicit decision boundaries, and rule-compatible architectures.
- Interpretable Representation: Interpretability-oriented regularization can promote decomposability, monotonicity, non-negativity, sparsity, or human-in-the-loop priors.These constraints target the learned representation rather than only conventional overfitting control.
- Interpretable Representation: InfoGAN maximizes mutual information between latent codes and observations, encouraging each noise dimension to encode a semantic concept.Its discrete categorical and continuous style codes include controls for localized parts and digit rotation.
- Interpretable Representation: Non-negative weights and sparse embeddings improve interpretability by reducing cancellation, aliasing, or overlap among represented features.The cited examples apply non-negativity to network weights and k-sparsity to word embeddings.
- Model Renovation: Piecewise-linear networks expose decision boundaries and permit closed-form prediction solutions, while Soft-AE corresponds directly to a learned cascaded wavelet system.Soft-AE uses adaptable soft-thresholding in encoding layers and linear activations in decoding layers.
- Model Renovation: Interpretable architectures can expose concepts, relevance weights, hierarchical tasks, or rule-based decisions through modular network structures.Examples include additive concept-based models, deep supervision, rule-set/black-box hybrids, and FA-RNNs transformable into regular expressions.
III. INTERPRETABILITY IN MEDICINE
Interpretability methods in medicine are applied across imaging and clinical prediction to inspect features, localize relevant regions, generate explanations, and assess whether models rely on medically meaningful evidence. The surveyed examples include post-hoc analyses and models designed with interpretable intermediate representations.
- Motivation: Medical interpretability is motivated by the high stakes of clinical decisions, where correct predictions alone are insufficient.The section frames interpretability as relevant to evaluating model behavior in medicine.
- Post-hoc Interpretability Analysis: Activation maps, CAM, integrated gradients, DeepLIFT, LRP, and attention maps were used to identify image regions or features relevant to clinical predictions.Applications span skin lesions, chest radiographs, CT scans, EEG, haemorrhage detection, and ICU mortality prediction.
- Post-hoc Interpretability Analysis: Radiologists strongly agreed in all cases that lung-cancer prediction saliency maps focused on nodules.The maps were derived with integrated gradients and assessed by experienced radiologists.
- Post-hoc Interpretability Analysis: 78.1% of bleeding points were detected in attention maps for acute intracranial haemorrhage detection and classification.Localization accuracy was evaluated by measuring overlap between bleeding points and attention maps.
- Post-hoc Interpretability Analysis: A gradient-boosting-tree surrogate maintained the deep models’ mortality and ventilator-free-days performance on a Pediatric ICU dataset.The tree was trained using predictions from three deep models as labels, providing an interpretable model for electronic health-record prediction.
- Ad-hoc Interpretable Modeling: Interpretable medical models also used hierarchical semantic tasks, multi-scale feature arms, text explanations, and shape-aware activation maps.These designs exposed semantic characteristics, hierarchical morphology, doctor-preferred descriptions, or subtle irregular nodule features.
IV. PERSPECTIVE
The paper proposes future interpretability directions linking neural networks with fuzzy logic, neuroscience, biologically plausible learning, and architecture design. It also identifies task coverage and expert collaboration as practical priorities.
- Fuzzy Logic and Deep Learning: Combining fuzzy logic’s interpretability and accountability with neural networks’ knowledge-acquisition ability is proposed as a complementary direction.The paper cites existing combinations including ANFIS, generic fuzzy perceptrons, and RBF networks.
- Fuzzy Logic and Deep Learning: RBF networks are presented as a potential vehicle for encoding fuzzy rules because they are functionally equivalent to fuzzy inference systems under mild conditions and are universal approximators.The paper states that rule generation and fuzzy-rule representation are more straightforward in adaptable RBF networks than in multilayer perceptrons.
- Convergence of Neuroscience and Deep Learning: Neuroscience may inform interpretability because biological and artificial neural networks are deeply connected.The paper proposes biologically plausible cost functions, architectures, and optimization algorithms as related directions.
- Convergence of Neuroscience and Deep Learning: Backpropagation differs from biological learning because biological synapses update locally, whereas deep-network connections are tuned through non-local backpropagation.The paper also highlights neuromodulators as a missing mechanism in deep networks.
- Future Medical Applications: Interpretability research in medicine should expand beyond classification to segmentation, registration, reconstruction, and other radiological tasks.The paper also calls for collaboration among medical doctors, technical experts, and theoretical researchers to assess and apply interpretability methods.
V. CONCLUSION
The survey presents a taxonomy-based landscape of neural-network interpretability, reviews representative methods and medical applications, and discusses existing limitations and future directions.
- Conclusion: The survey reviews key ideas, implications, and limitations of interpretability studies using its proposed taxonomy.It also illustrates representative interpretation methods and emphasizes applications in medicine.