Source-linked AI summary
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
Bas H. M. van der Velden, Hugo J. Kuijf, Kenneth G. A. Gilhuijs, Max A. Viergever
TL;DR
Deep learning decisions are difficult to comprehend and may be biased, motivating explainability in medical image analysis. This survey systematically reviews XAI papers, introduces classification criteria, and adopts a holistic scope that includes explanations, critiques, and evaluation methods.
Problem
Deep learning models are often treated as black boxes because their decision processes are difficult to fully comprehend, while concerns exist that they may be biased.
Method
The survey uses a systemic review procedure to provide a comprehensive overview of XAI papers in medical image analysis and classify them through a holistic approach.
Results
The survey covers XAI methods including class activation mapping, layer-wise relevance propagation, and trainable attention, with reported applications and performance findings.
Takeaways & Limitations
A holistic account of XAI in medical image analysis includes non-visual explanations, critiques of XAI, and methods for evaluating XAI.
Takeaways & Limitations
Model-based explanation through enforced sparsity or simulatability is infeasible for the deep neural networks considered by the survey.
Abstract
from arXiv · showhide
With an increase in deep learning-based methods, the call for explainability of such methods grows, especially in high-stakes decision making areas such as medical image analysis. This survey presents an overview of eXplainable Artificial Intelligence (XAI) used in deep learning-based medical image analysis. A framework of XAI criteria is introduced to classify deep learning-based medical image analysis methods. Papers on XAI techniques in medical image analysis are then surveyed and categorized according to the framework and according to anatomical location. The paper concludes with an outlook of future opportunities for XAI in medical image analysis.
1. Introduction
Deep learning advances medical image analysis but is difficult to understand and may conceal bias, making explainability especially important in high-stakes medical decisions. This survey systematically reviews deep learning-based XAI methods, introduces a classification framework, and organizes the literature by explanation method and anatomical location.
- Motivation: Deep learning is difficult to comprehend because neural networks contain many layers linked by nonlinear, intertwined relations.The learned features range from low-level properties such as edges to higher-level structures such as cancer borders.
- Motivation: Black-box decisions may contain unnoticed bias, a concern with far-reaching implications in medical applications.Medical experts have specifically raised concerns about deep learning’s black-box nature, while medical decisions often carry high stakes.
- Motivation: XAI approaches are used to better understand deep-learning systems and provide insight into medical-image-analysis algorithms.The survey uses XAI as the term for explainable or interpretable deep learning approaches.
- Scope and method: The survey provides a comprehensive overview focused specifically on deep learning-based XAI in medical image analysis.Its paper-inclusion strategy followed a systematic review procedure, discussions with colleagues, and snowballing of references.
- Organization: The authors introduce an XAI framework and classify surveyed techniques by explanation method and anatomical location.The survey also discusses adaptations of computer-vision techniques using medical-imaging domain knowledge.
- Contribution: Compared with earlier reviews, the survey takes a holistic approach covering non-visual explanations, XAI critiques, evaluation methods, systematic surveying, and future opportunities.It positions itself as extending reviews focused mainly on computer-vision techniques or visual explanations.
2. Explainable Artificial Intelligence (XAI) framework
The framework classifies XAI techniques along three criteria: whether explanations are model-based or post hoc, model-specific or model-agnostic, and global or local. These distinctions organize how explanations relate to the model, model choice, and explanation scope.
- XAI techniques are classified by model-based versus post hoc, model-specific versus model-agnostic, and global versus local explanations.The framework adapts distinctions from prior XAI surveys and organizes techniques discussed in the paper.
- Model-based versus post hoc explanation: Model-based explanation uses models designed to be understandable, whereas post hoc explanation analyzes a trained neural network after it has been built.Deep neural networks are generally difficult to make sparse or human-simulatable, motivating post hoc approaches in this survey.
- Model-based versus post hoc explanation: Deep neural networks typically contain thousands to millions of weights, making sparsity and human simulation of the entire decision process infeasible.The survey therefore focuses on XAI methods that explain deep-learning models rather than enforcing model-based interpretability.
- Model-specific versus model-agnostic explanation: Model-specific methods use attributes of particular model classes, while model-agnostic methods operate only on model inputs and outputs.Model-agnostic methods can perturb inputs and inspect output changes to identify regions driving a prediction, but model-specific methods may restrict model choice.
- Global versus local explanation: Global explanations describe relationships learned across a dataset, whereas local explanations account for a single input or output.A saliency map identifying the MRI region contributing to a tumor prediction is an example of a local explanation.
3. XAI in medical image analysis
The survey organizes XAI in medical image analysis into visual, textual, and example-based methods, classified by explanation scope and model dependence. It reviews these techniques and their medical-imaging applications, with visual explanations most common.
- The framework categorizes methods as visual, textual, or example-based, and distinguishes model-based from post hoc, model-specific from model-agnostic, and global from local explanations.
- Visual explanation, or saliency mapping, highlights image regions important for a model decision and is the most common XAI form in medical image analysis.
- CAM produces local, model-specific, post hoc explanations by weighting visual patterns across spatial locations, while Grad-CAM generalizes CAM beyond architectures requiring global average pooling.
- Multi-scale CAM variants provide higher-resolution maps and improve localization of small retinal structures, brain tumors, and interpapillary capillary patterns.
- LRP backpropagates output relevance through network layers under a conservation law, and one medical-imaging comparison found it more specific than guided backpropagation for Alzheimer’s regions.
- SHAP assigns each feature a marginal contribution to the model output, but Shapley-value computation is resource-intensive because it assesses many permutations.
Q. Li et al. (2019) MRI CAM
The supplied material lists MRI and other medical-imaging studies using multiple XAI techniques, including CAM, Grad-CAM, guided backpropagation, attention, and perturbation-based methods.
- The listed studies pair MRI and other imaging modalities with CAM, Grad-CAM, guided backpropagation, backpropagation, trainable attention, prediction difference analysis, and occlusion sensitivity.
- The supplied entries also include CT, ultrasound, histology, endoscopy, and X-ray applications alongside MRI studies.
H. Xu et al. (2019) MRI LRP
The supplied material enumerates XAI studies across anatomical locations and imaging modalities, including saliency-map, textual, and example-based explanation approaches.
- Across these anatomical areas, the entries include CAM, Grad-CAM, LRP, SHAP, LIME, occlusion sensitivity, multiple-instance learning, and trainable attention.
- The listed studies span cardiovascular, chest, breast, eye, gastrointestinal, dental, skin, thyroid, prostate, musculoskeletal, and female reproductive imaging.
- Textual explanation adds descriptions ranging from simple characteristics such as “spiculated mass” to entire medical reports.
3.2.1. Image captioning
Image captioning generates textual explanations from medical images using image and text encoders, with later approaches linking generated language to visual attention or saliency.
- Image-captioning systems commonly encode images with convolutional neural networks and text with recurrent networks such as LSTMs.
- These systems train on human-generated sentences and evaluate generated descriptions with BLEU, which measures word N-gram precision against reference sentences.
- For chest X-rays, Singh et al. used GloVe and radiology-specific RadGloVe embeddings to train an LSTM captioning model.
- Using both RadGloVe and GloVe achieved higher generated-radiology-report performance than using GloVe alone.
- Captioning with visual explanation adds visual word constraints or attention mechanisms, linking textual explanations to saliency regions in mammography, histology, and chest X-rays.
3.2.3. Testing with Concept Activation Vectors (TCAV)
TCAV explains neural-network decisions through human-understandable concepts, using user-defined examples to quantify concept sensitivity. Medical applications relate concepts to disease, biomarkers, and continuous clinical measures.
- Medical applications: Concept-based explanations distinguished healthy and cancerous breast-histopathology regions through differences in the clinically used concept “nuclei area.”Both regions scored high on “contrast,” while nuclei area differed between them.
- TCAV: TCAV provides global explanations in terms of human-understandable concepts.It is applied post hoc using concept examples and random non-concept examples.
- TCAV: TCAV quantifies a trained model’s sensitivity to concepts such as stripes or spiculated masses using concept activation vectors.Test-case responses to the vectors measure sensitivity to each concept.
- Medical applications: TCAV related microaneurysm annotations to diabetic retinopathy in fundus imaging, demonstrating feasibility in a medical image-processing example.
- Medical applications: In cardiac cine-MRI, TCAV linked clinically known biomarkers to cardiac disease and enabled reconstructions with low peak ejection rate.The reconstruction added the concept activation vector to the latent space.
- Extensions: Regression concept vectors extend binary TCAV by modeling continuous-valued concepts such as tumor size.
3.3.2. Influence functions
Influence functions explain model decisions by estimating how training examples affect predictions. In medical imaging, they provided global links between liver-lesion classifications and radiological characteristics.
- Influence functions: Influence functions approximate how removing or changing a training input would affect a model’s decision.The approximation avoids the expense of direct perturbation.
- Medical application: For multiphase liver-lesion MRI, influence functions associated classifications with radiological characteristics in a global explanation.
- Medical application: The class “benign cyst” was most often associated with the radiological finding “thin-walled mass.”
- Medical application: Outputting both class labels and corresponding radiological characteristics was described as potentially enhancing user trust.
4. Discussion
The survey finds that medical-image XAI is dominated by accessible post hoc, local explanations, while evaluation remains difficult and methodological concerns persist. It highlights broader explanation combinations and future directions including biological and causal explanations.
- Survey trends: The survey discusses 223 papers on XAI for deep learning in medical image analysis.
- Survey trends: Most surveyed methods use post hoc explanations, which are more accessible than model-based XAI because they operate after neural-network training.
- Survey trends: Most papers provide local rather than global explanations, while both model-specific and model-agnostic methods are represented.
- Survey trends: Most papers focus on chest or brain imaging and on X-ray or MRI modalities.
- Evaluation: Explanation quality is harder to define and evaluate than standard medical-image metrics such as accuracy, Dice coefficient, or ROC analysis.The survey describes application-grounded, human-grounded, and functionally-grounded evaluation approaches.
- Evaluation: Application-grounded evaluation directly tests the intended objective but is costly, whereas human-grounded evaluation is cheaper but assesses explanation quality through a proxy.
- Evaluation: Randomization tests found that guided backpropagation and guided Grad-CAM produced similar maps despite randomized parameters and labels, warranting caution in visualization.
- Outlook: The survey reports increasing use of holistic approaches that combine multiple forms of explanation and identifies biological and causal XAI as future directions.
5. Conclusion
The paper surveys 223 papers on XAI for deep-learning-based medical image analysis, organizing them by an XAI framework, anatomical location, and imaging technique. It also addresses XAI evaluation, critiques, and future perspectives.
- 223 papers on XAI for deep-learning-based medical image analysis were surveyed.
- The surveyed papers were classified using an XAI framework and categorized by anatomical location and imaging technique.
- The paper discusses how XAI can be evaluated and reviews current critiques of XAI.
- The paper presents future perspectives for XAI in medical image analysis.
6. Additional information
The work reports partial funding from the Dutch Cancer Society through grant 10755.
- The work was partially funded by the Dutch Cancer Society through grant 10755.
7. Appendix
The appendix describes the literature-search strategy and screening process used for the survey, including database querying, active-learning assistance, and snowballing. It also records the publication types and time boundary included.
- The search used a query combining explainability terms with medical-imaging terms in SCOPUS.
- Search results were analyzed with an active-learning toolbox that ranked papers by relevance and updated its sorting from user input.
- The researchers supplemented database searching with colleague discussions and snowballing through cited and citing papers.
- Titles and abstracts were screened, with paper content reviewed when inclusion was uncertain.
- For multiple publications on the same subject by the same authors, the journal or most recent conference publication was selected.
- The survey included peer-reviewed journal papers and conference proceedings published up to October 2020.