Source-linked AI summary
Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyond
Xuhong Li, Haoyi Xiong, Xingjian Li, Xuanyu Wu, Xiao Zhang, Ji Liu, Jiang Bian, Dejing Dou
TL;DR
Deep models are difficult to understand because their overparameterized black-box behavior complicates interpretation and interpretability assessment. This survey clarifies these concepts, proposes a taxonomy of interpretation algorithms, and reviews evaluation methods and related connections. It organizes recent algorithms and evaluation approaches while identifying challenges in measuring trustworthiness and model interpretability.
Problem
Deep models’ black-box predictions are difficult to understand, while interpretation and interpretability remain confused and their quantitative evaluation lacks sufficient metrics or ground truth.
Method
The paper surveys interpretation algorithms through a three-dimensional taxonomy and reviews evaluations of algorithm trustworthiness, model interpretability, and related connections.
Results
The survey categorizes recent interpretation algorithms and comprehensively reviews metrics and approaches for evaluating their trustworthiness and model interpretability.
Takeaways & Limitations
Trustworthy interpretation algorithms support meaningful evaluation and quantitative comparison of interpretation methods and model interpretability where suitable evaluation approaches exist.
Takeaways & Limitations
Quantitative evaluation remains constrained by the lack of a proper trustworthiness definition and well-defined metrics, while model interpretability lacks scalable ground truth.
Abstract
from arXiv · showhide
Deep neural networks have been well-known for their superb handling of various machine learning and artificial intelligence tasks. However, due to their over-parameterized black-box nature, it is often difficult to understand the prediction results of deep models. In recent years, many interpretation tools have been proposed to explain or reveal how deep models make decisions. In this paper, we review this line of research and try to make a comprehensive survey. Specifically, we first introduce and clarify two basic concepts -- interpretations and interpretability -- that people usually get confused about. To address the research efforts in interpretations, we elaborate the designs of a number of interpretation algorithms, from different perspectives, by proposing a new taxonomy. Then, to understand the interpretation results, we also survey the performance metrics for evaluating interpretation algorithms. Further, we summarize the current works in evaluating models' interpretability using "trustworthy" interpretation algorithms. Finally, we review and discuss the connections between deep models' interpretations and other factors, such as adversarial robustness and learning from interpretations, and we introduce several open-source libraries for interpretation algorithms and evaluation approaches.
1 Introduction
Deep models perform strongly across many tasks, but their overparameterized, nonlinear black-box nature makes predictions difficult to understand and raises trust concerns in high-stakes applications. This survey clarifies core concepts, organizes interpretation algorithms through a three-dimensional taxonomy, and reviews evaluation approaches for algorithm trustworthiness and model interpretability.
- Motivation: Deep models achieve strong performance across varied tasks, yet their overparameterization and hierarchical nonlinearity make predictions difficult to understand.The paper links limited interpretability to trust concerns in high-stakes applications.
- Core concepts: Interpretations explain model decisions through outputs such as discriminative features or training-sample contributions, whereas interpretability is an intrinsic model property concerning human understandability.Trustworthy interpretation algorithms can support comparisons of interpretability across models when human-labeled results are available.
- Interpretation algorithms: Interpretation algorithms may highlight important inputs, inspect internal features or counterfactuals, and analyze training-data contributions, learning difficulty, or mislabeled samples.The reviewed designs use gradients, perturbations, proxy models, visualization, counterfactual examples, and data analysis.
- Taxonomy: The survey categorizes interpretation algorithms along interpretation representation, target-model type, and the algorithm’s relation to the deep model.The taxonomy includes feature importance or sample influence, differentiable or architecture-specific models, and closed-form or compositional relations.
- Evaluation: Trustworthiness requires interpretations to reliably and faithfully reveal arbitrary deep models’ behaviors rather than produce explanations irrelevant to the model.The paper distinguishes trustworthiness of interpretation algorithms from interpretability of models.
- Evaluation: Trustworthiness evaluations use perturbations or proxy models, while model-interpretability evaluations use expert ground truths or cross-model explanations.Both evaluation areas remain challenging: trustworthiness lacks well-defined metrics, and interpretability lacks scalable ground truth.
2 Main Concepts: Interpretations and Interpretability
The paper distinguishes interpretations, interpretation algorithms, and model interpretability, then frames trustworthy explanations as a prerequisite for assessing interpretability. It organizes interpretation algorithms by the information they expose and emphasizes that human understandability remains subjective.
- Conceptual distinctions: Interpretations are explanations or attributions produced by algorithms to reveal how deep models make decisions, whereas interpretability is an intrinsic model property.Examples include discriminative feature indications and training-sample contributions.
- Interpretation algorithms: Interpretation algorithms expose different information, including training-sample difficulty, feature importance, intermediate representations, and decision-making processes.The survey reviews algorithms for analyzing model inputs, internals, and training data.
- Trustworthiness: An interpretation algorithm must be trustworthy by properly revealing the model’s underlying rationale rather than producing explanations independent of the model.The paper discusses parameter-randomization sanity checks in which some methods remain unchanged despite significant model changes.
- Trustworthiness: Self-interpretable models combine a model with an intrinsic interpretation algorithm, and interpretations used directly for decisions are considered trustworthy.The paper lists Capsule Models, Neural Additive Models, and CALM as examples.
- Model interpretability: Model interpretability concerns how easily humans can understand and trace a model’s decisions, while its measurement is difficult because understandability is subjective.The paper notes that few metrics currently quantify interpretability.
- Desiderata: The survey separates trustworthiness from human-centered properties such as plausibility and satisfaction, while relating reliability, robustness, fidelity, and faithfulness to trustworthiness.It treats fairness, transferability, and privacy as standards constraining models rather than properties of interpretation algorithms.
3 Interpretation Algorithms: Taxonomy, Algorithm Designs, and Miscellaneous
The survey organizes interpretation algorithms along three orthogonal dimensions: interpretation representation, target-model type, and algorithm–model relation. It reviews representative designs, compares their distribution, and identifies gaps and application-specific limitations.
- Categorization and Discussion: The survey presents its taxonomy and algorithm categorization as a framework for organizing the interpretation literature and guiding future extensions.Tables 1 and 2 provide compact categorizations and publication listings.
- Taxonomy: The taxonomy categorizes algorithms by interpretation representation, target-model type, and relation between the algorithm and deep model.These dimensions support comparison across feature, model-response, rationale-process, and dataset interpretations; model-agnostic, differentiable, and specific-model targets; and closed-form, composition, dependence, and proxy relations.
- Taxonomy: Representation categories include feature importance, model response, model rationale process, and dataset influence on optimization.Feature methods estimate contributions, response methods probe model behavior with generated examples, rationale methods indicate decision paths, and dataset methods assess training-sample effects.
- Taxonomy: The relation dimension distinguishes closed-form formulas, interpretable components learned with models, post-training operations, and proxy models.Closed-form methods derive formulas from the target model; composition methods act as interpretable model components; dependence methods add operations after training; proxy methods learn or derive explanatory substitutes.
- Algorithm Designs: Reviewed designs include LIME’s locally faithful interpretable surrogate, LRP’s recursive relevance propagation, CAM’s important CNN regions, and global feature-importance analysis.LIME fits perturbed nearby samples with an interpretable model, LRP propagates relevance from outputs toward input pixels, and CAM identifies influential image regions from late CNN operations.
- Categorization and Discussion: The categorization contains many feature, proxy, and dependence methods but few rationale, closed-form, and model-agnostic composition methods.The authors associate these patterns with the difficulty of analyzing or approximating complex deep-network rationales and closed forms, and identify blank categories as possible future directions.
- Categorization and Discussion: The survey notes that adversarial attacks can target interpretations as well as trained models, leaving further investigation open.Improving input–output interpretations is also identified as a future perspective.
- Miscellaneous: Interpretation requirements vary by application: DRL may need internal rationale explanations, while medical deployment remains limited by interpretability.For safety-critical DRL, input–output explanations may not persuade consumers; in medical applications, interpretation tools are applicable and their trustworthiness can be evaluated, but clinic deployment remains limited.
4 Trustworthiness Evaluations of Interpretation Algorithms and Model Interpretability Evaluations
The paper distinguishes trustworthiness evaluations of interpretation algorithms from evaluations of model interpretability, using the former as a basis for the latter. It reviews perturbation, retraining, randomization, synthetic-data, trojaning, metric-based, and human-centered approaches.
- Evaluation framework: Trustworthiness evaluates whether interpretation results reliably and faithfully reveal arbitrary deep-model behavior rather than producing irrelevant or human-preferred explanations.
- Trustworthiness evaluations: Perturbation-based evaluations test whether removing the most salient features first causes rapid performance decay, commonly summarized by MoRF or LeRF area under the curve.
- Trustworthiness evaluations: Retraining after removing important features separates degradation caused by distribution shift from degradation caused by removing genuinely informative features, though it adds substantial computation.
- Trustworthiness evaluations: Randomizing model parameters can expose saliency methods that identify similar regions despite altered model weights, indicating that they do not depend on the model.
- Trustworthiness evaluations: BAM and trojaning-based evaluations create controlled or contaminated datasets and models whose ground-truth feature importance or trigger reliance can test interpretation algorithms.
- Model interpretability evaluations: Model interpretability methods compare interpretation results with semantic items, cross-model explanations, out-of-distribution behavior, or users’ understanding of model decisions.
- Evaluation framework: Assessing trustworthiness remains challenging for most interpretation algorithms, and adversarial attacks can target interpretations as well as trained models.
5 Impact beyond Interpretations
The survey connects interpretations with adversarial robustness and with training strategies that use interpretation results. These connections suggest that interpretability and model performance can be improved simultaneously, while several questions remain open.
- Interpretability and adversarial robustness: Adversarially trained models show more interpretable input gradients, and theoretical analysis links increased robustness with better alignment between inputs and their gradients.
- Interpretability and adversarial robustness: Adversarially robust models rely on fewer texture features and are more shape-biased, which the survey describes as more consistent with human interpretation.
- Open directions: The survey leaves further investigation of attacks on interpretations and connections between interpretation-related terms and other deep-model properties for future work.
- Learning from interpretations: Interpretation results can guide data augmentation and regularization strategies, including saliency-based improvements to Mixup and regularizers encouraging alignment.
- Learning from interpretations: The surveyed works support the view that interpretability and model performance are not contradictory measures and can be improved simultaneously.
6 Open-Source Libraries for Deep Learning Interpretation
The paper introduces open-source libraries that simplify practical use of deep-learning interpretation algorithms. These libraries target mainstream frameworks and cover popular interpretation methods.
- TF-Explainer, Captum, and InterpretDL implement popular interpretation algorithms for TensorFlow, PyTorch, and PaddlePaddle, respectively.
- Other listed resources include interpretml and AIX360 for machine learning, while LIT is intended for NLP models.
7 Discussions and Conclusions
The paper surveys interpretation research by clarifying core concepts, organizing algorithms, evaluating trustworthiness and interpretability, and connecting interpretations to other deep-learning factors.
- The paper clarifies interpretation algorithms, model interpretability, and their connection through the notion of trustworthy interpretations.This distinction frames how interpretations explain model decisions and how interpretability is evaluated.
- A new taxonomy categorizes recent interpretation algorithms from different perspectives and supports clearer comparison and future research.The taxonomy organizes algorithm designs according to multiple perspectives rather than a single principle.
- Performance metrics are surveyed to evaluate interpretation trustworthiness and quantitatively compare algorithms instead of relying on tenuous descriptions or qualitative visualizations.These metrics are intended to support more appropriate use of interpretation results and evidence-based algorithm comparisons.
- Current methods for evaluating model interpretability are summarized, but relatively few methods exist beyond those aligned with popular network architectures.Designing additional interpretability evaluation methods is identified as an important research direction.
- The review connects deep-model interpretations with adversarial robustness and learning from interpretations, and notes the availability of open-source libraries.These connections provide further ways to observe and analyze deep models, while libraries simplify practical use and future research.