Explainable artificial intelligence (XAI) in deep learning-based medical image analysis

1. Introduction

• Deep learning replaces handcrafted image features with features learned by neural networks, whose many nonlinear, intertwined layers make their decisions difficult to comprehend.The paper describes this opacity as the “black box” problem.Source: §1 · p. 3
• Medical experts have raised concerns about opaque deep-learning decisions (Jia et al. (2020)), while GDPR Article 15 requires patients’ right to meaningful information about how decisions are rendered.These concerns arise in a high-stakes medical decision-making context.Source: §1 · pp. 3–4
• The survey comprehensively reviews deep-learning-based XAI papers in medical image analysis, using a systematic review, colleague discussion, and snowballing to identify articles.Its search included papers referenced by included works and papers that cited them.Source: §1 · p. 4
• The survey introduces an XAI framework, characterizes reviewed papers with it, and groups applications by explanation method and anatomical location.It discusses early adopters when multiple papers use the same technique and summarizes the remaining papers in tables.Source: §1 · p. 4
• The survey describes whether and how computer-vision techniques have been adapted for medical image analysis and extends Huff et al. (2021)'s review with a holistic approach to non-visual explanations, XAI critiques, and evaluation methods.It systematically surveys papers reflecting the current status of XAI in medical imaging.Source: §1 · p. 4

2. Explainable Artificial Intelligence (XAI) framework

• The framework distinguishes XAI methods along three axes: model-based versus post hoc, model-specific versus model-agnostic, and global versus local explanations.It adapts criteria from Adadi and Berrada (2018) and Murdoch et al. (2019).Source: §2 · p. 6

Model-based versus post hoc explanation

• Model-based explanations make the model interpretable, whereas post hoc explanations analyze a neural network after it has been trained.The survey uses “model” and “neural network” interchangeably for deep learning.Source: Model-based versus post hoc explanation · p. 7, §2.1.2 · p. 8

2.1.1. Model-based explanation

• Model-based explanation through sparsity or simulatability is infeasible for deep networks with thousands to millions of weights.Such networks are neither sparse nor suited to internal simulation by a person.Source: §2.1.1 · p. 7

2.1.2. Post hoc explanation

• Post hoc methods inspect learned features, feature importance, and feature interactions, or produce visual explanations such as saliency maps.Examples include methods cited by Abbasi-Asl and Yu (2017), Olden et al. (2004), Tsang et al. (2018), Selvaraju et al. (2017), Simonyan et al. (2013), Springenberg et al. (2014), Zeiler and Fergus (2014), and Zhou et al. (2016).Source: §2.1.2 · p. 8

2.2.1. Model-specific explanation

• Model-specific explanations use attributes of particular model classes, potentially excluding a model that might better fit the input-output relationship.Model-based explanation is model-specific, but model-specific explanation need not be model-based; some CNN saliency methods are post hoc and model-specific.Source: Model-specific versus model-agnostic… · p. 8, §2.2.1 · p. 8

2.2.2. Model-agnostic explanation

• Model-agnostic explanations operate independently of model choice by perturbing inputs and observing output changes to identify influential regions.They are naturally post hoc.Source: §2.2.2 · p. 9

2.3.1. Global explanation

• Explanation scope is global when it describes a model across a dataset, and local when it explains a single output.Global explanations can show dataset-level feature importance or learned filters and their relevance to the task.Source: Scope of explanation · p. 9, §2.3.1 · p. 9

2.3.2. Local explanation

• A local explanation can use an MRI saliency map to pinpoint the image region that drove a tumor classification for one person.This differs from explaining the relationship between blood pressure and cardiac risk across a dataset.Source: §2.3.2 · p. 10

3. XAI in medical image analysis

• The survey groups explanations as visual, textual, or example-based and classifies methods by model-based versus post hoc, model-specific versus model-agnostic, and global versus local criteria.Visual explanations, also called saliency mapping, are the most common form; most saliency mapping techniques use backpropagation, while some use perturbation-based or multiple-instance-learning approaches.Source: §3 · pp. 11, 13

3.1.1.1. (Guided) backpropagation and deconvolution

• Early backpropagation and deconvolution methods visualized output partial derivatives at pixel level and provided local, post hoc, CNN-specific explanations.De Vos et al. (2019) used deconvolution to visualize the basis for coronary artery calcium estimates in cardiac or chest CT slices.Source: §3.1.1.1 · p. 14

3.1.1.2. Class Activation Mapping (CAM)

• CAM uses global average pooling over the last convolutional feature maps to form a weighted linear sum of visual patterns across spatial locations.CAM provides local, model-specific, post hoc explanations; multi-scale CAMs were proposed because medical images often contain information at multiple scales.Source: §3.1.1.2 · pp. 14–15
• Higher-resolution activation maps were better at identifying small structures on retinal fundus images, while high-resolution CAMs accurately localized brain tumors on MRI and highlighted small interpapillary capillary-loop patterns in endoscopy images.Source: §3.1.1.2 · p. 15

3.1.1.3. Gradient-weighted Class Activation Mapping (Grad-CAM)

• Grad-CAM generalizes CAM to any CNN without requiring global average pooling, while guided Grad-CAM multiplies guided backpropagation and Grad-CAM element-wise.Medical imaging uses included highlighting areas informing decisions about metastatic tissue, small-bowel enteropathies, and brain tumors.Source: §3.1.1.3 · p. 15

3.1.1.4. Layer-wise relevance propagation (LRP)

• LRP iteratively backpropagates a network output score, assigning relevance to input neurons while conserving the source neuron's total relevance.Böhle et al. (2019) used LRP to identify regions responsible for Alzheimer’s disease from brain MR images.Source: §3.1.1.4 · p. 16
• LRP saliency maps were more specific than guided backpropagation maps in identifying regions known for Alzheimer’s disease.Böhle et al. (2019) compared the maps while identifying regions responsible for Alzheimer’s disease from brain MR images.Source: §3.1.1.4 · p. 16

3.1.1.6. Trainable attention

• Grid attention demonstrated high performance for segmentation and localization when attention gates were added to a UNET and a VGG variant.Schlemper et al. (2019) used grid attention to capture anatomical information in medical images; attention coefficients explained which image areas the network focused on.Source: §3.1.1.6 · p. 17

3.1.2.1. Occlusion sensitivity

• Zeiler and Fergus (2014) used occlusion sensitivity to show which image regions mattered for classification; occluding a dog’s face changed its correct breed prediction to “tennis ball.”Source: §3.1.2.1 · p. 17

3.1.2.2. Local Interpretable Model-agnostic Explanations (LIME)

• LIME approximates a complex model locally with a simpler model learned from perturbed inputs, weights examples by similarity to the original, and uses superpixels to identify important image regions.Malhi et al. (2019) used LIME to identify bloody regions in gastral endoscopy images.Source: §3.1.2.2 · p. 18

3.1.2.3. Meaningful perturbation

• Meaningful perturbation tests prediction changes using plausible local alterations such as constant values, noise, or blurring, but these proposed perturbations were considered unsuitable for medical images.Uzunova et al. (2019) noted that constant-value replacement is implausible and medical images naturally tend to be noisy and blurry.Source: §3.1.2.3 · p. 18
• A VAE replacing pathological regions with healthy-tissue equivalents localized pathology in OCT and brain MRI, with better localization than simple blurring or constant-value perturbations.The examples included intraretinal and subretinal fluid, pigment epithelium detachments, and stroke lesions.Source: §3.1.2.3 · pp. 18–19
• Inpainting healthy-tissue equivalents outperformed backpropagation and Grad-CAM for pinpointing breast masses and tuberculosis on chest X-rays, measured by Hausdorff distance between thresholded heatmaps and pixel-level ground truth.Source: §3.1.2.3 · p. 19

3.1.2.4. Prediction difference analysis

• Prediction difference analysis assigns pixel relevance by measuring prediction changes when pixels are treated as unknown, and conditional sampling and connected-pixel patches extend the analysis.Applications included explaining HIV-versus-control brain MRI classification and distinguishing Alzheimer’s disease patients from normal controls using multiscale superpixel or supervoxel maps.Source: §3.1.2.4 · p. 19

3.1.3. Multiple instance learning-based approaches

• Multiple instance learning supports visualizing explanations by treating images as bags of unlabeled patches; each image bag is labeled while its patch instances are not; Schwab et al. (2020) localized critical findings in chest X-rays.Source: §3.1.3 · p. 20

• Trainable attention was reported for MRI by Lian et al. (2019) and H. Xu et al. (2019).Source: Q. Li et al. (2019) MRI CAM · p. 22
• Grad-CAM studies covered MRI (Liao et al., 2020; Natekar et al., 2020; Pereira et al., 2018; Pominova et al., 2018; Windisch et al., 2020) and ultrasound (B. Xie et al., 2020).Source: Q. Li et al. (2019) MRI CAM · pp. 21–22
• MRI studies used prediction difference analysis and occlusion sensitivity (Shahamat and Saniee Abadeh, 2020).Source: Q. Li et al. (2019) MRI CAM · p. 22
• CAM was used for MRI in Ng et al. (2018) and Shinde et al. (2019a, 2019b).Source: Q. Li et al. (2019) MRI CAM · p. 22
• Other entries include Grad-CAM for histology (Z. Tang et al., 2019) and multiple instance learning for CT.Source: Q. Li et al. (2019) MRI CAM · p. 22
• Backpropagation methods included guided backpropagation for MRI (X Wang et al., 2020) and backpropagation for MRI.Source: Q. Li et al. (2019) MRI CAM · p. 22

• Candemir et al. (2020) used Grad-CAM for cardiovascular CT.Source: H. Xu et al. (2019) MRI LRP · p. 22
• Ausawalaithong et al. (2018) used CAM for chest X-rays, and M. Ahmad et al. (2019) used CAM for fundus photography.Source: H. Xu et al. (2019) MRI LRP · pp. 22, 24
• Vila-Blanco et al. (2020) used Grad-CAM for dental X-rays, and X. Chen et al. (2019) used Grad-CAM for gastrointestinal CT.Source: H. Xu et al. (2019) MRI LRP · pp. 23–24
• Kermany et al. (2018) used occlusion sensitivity across multiple modalities.Source: H. Xu et al. (2019) MRI LRP · p. 25
• Schlemper et al. (2019) used trainable attention for CT.Source: H. Xu et al. (2019) MRI LRP · p. 23

3.2.1. Image captioning

• The framework coupled a convolutional neural network for image encoding with an LSTM for textual encoding; human-generated sentences served as training ground truth, and BLEU evaluated word n-gram precision between generated and reference sentences.Source: §3.2.1 · p. 26
• Singh et al. (2019) achieved higher radiology-report performance using RadGloVe with GloVe than using GloVe alone.The survey describes evaluation with BLEU and its variants.Source: §3.2.1 · p. 26

3.2.2. Image captioning with visual explanation

• Zhang et al. (2017a) used dual text-and-image attention to link histology captions with visual attention maps.The approach facilitates high-level interactions between image and text predictions.Source: §3.2.2 · p. 27
• X. Wang et al. (2018) linked different parts of chest X-ray explanations to saliency regions corresponding to multiple radiological findings.Lee et al. (2019a) added a visual-word constraint to breast-mammography text generation, improving explanations and linking reports to saliency maps.Source: §3.2.2 · p. 27

3.2.3. Testing with Concept Activation Vectors (TCAV)

• TCAV offers post-hoc global explanations in human-understandable concepts by measuring model sensitivity to user-defined concepts, including microaneurysms in diabetic-retinopathy fundus images.Clough et al. (2019) used TCAV to relate clinically known biomarkers to cardiac disease in cine-MRI.Source: §3.2.3 · pp. 27–28
• Graziani et al. (2020) extended binary TCAV with regression concept vectors for continuous measures, such as tumor size, and contrasted healthy with cancerous histology using nuclei area.The method explained why regions were classified differently even when both scored high on contrast.Source: §3.2.3 · p. 28

3.2.4. Other textual explanation techniques

• Shen et al. (2019) jointly classified lung-nodule malignancy and five radiologist-assessed textual characteristics, providing interpretable descriptions without significantly outperforming a normal CNN.The approach used a hierarchical semantic CNN on CT images.Source: §3.2.4 · p. 28

3.3.1. Triplet network

• Triplet networks learn embeddings for example retrieval: Peng et al. (2019) searched for nearest colorectal histology cases to explain a decision.Three shared-parameter networks compare samples using latent-space distances, and inspection of neighbors provides similar examples.Source: §3.3.1 · pp. 30–31
• Yan et al. (2018) learned lesion embeddings using anatomy and relative body coordinates, achieving good separation and test-set example retrieval; Codella et al. (2018) combined triplet loss with global average pooling to provide activation maps.Source: §3.3.1 · p. 31

3.3.2. Influence functions

• Wei Koh and Liang (2017) approximated effects of training-input removal or changes with influence functions; C. J. Wang et al. (2019) linked liver-lesion classifications to characteristics such as benign cysts and thin-walled masses.The liver MRI analysis provided a global account of network behavior and associated classifications with radiological findings.Source: §3.3.2 · pp. 31–32

3.3.3. Prototypes

• Uehara et al. (2019) used learned class prototypes to show which histology-image regions supported cancer or non-cancer predictions and how similar they were to training examples.Prototype-based explanations represented an actual computation of the model through weighted similarities to typical class examples.Source: §3.3.3 · p. 32

3.3.4. Examples from the latent space

• Sarhan et al. (2019) used a disentangled residual adversarial VAE to improve reconstruction fidelity and produce more human-interpretable lesion size, eccentricity, and skin-color concepts than a regular VAE.The approach traversed the latent space to examine reconstructions and underlying generative characteristics.Source: §3.3.4 · pp. 32–33
• Biffi et al. (2020) presented an explainable anatomical shape-analysis framework using a ladder VAE with 2D or 3D latent spaces for direct visualization of shape differences in hypertrophic cardiomyopathy versus healthy controls on cardiac MRI and Alzheimer’s disease versus healthy controls on brain MRI.Source: §3.3.4 · p. 33
• Silva et al. (2018) retrieved similar and dissimilar cases and extracted meta-feature rules, proposing completeness, correctness, and compactness as criteria for rule validity.Nearest neighbors from the same and other classes served as the most similar and dissimilar cases, respectively.Source: §3.3.4 · p. 33
• Silva et al. (2020) found saliency-map fine-tuning produced more consistent chest-X-ray examples than a baseline CNN without saliency maps.They searched for nearest neighbors between test-image embeddings and a curated image catalogue.Source: §3.3.4 · pp. 33–34
• Table 4 sorts papers by anatomical location and identifies the example-based technique used or the technique on which each method is based.CT means computed tomography, and MRI means magnetic resonance imaging.Source: Table 4 · p. 35, §3.3.4 · p. 35
• P. Yang et al. (2020) is listed for histology with latent-space examples and visual explanation.Source: §3.3.4 · p. 35

4. Discussion

• Most surveyed methods were post hoc and local, included model-specific and model-agnostic approaches, and focused on chest or brain imaging using X-ray or MRI.Post hoc explanations are produced after model training, while local explanations apply to individual cases rather than datasets.Source: §4 · p. 36, Figure 3 · p. 37

4.2.1. Application-grounded evaluation

• Application-grounded evaluation tests explanations with domain experts in real applications, directly assessing the system’s objective but at considerable cost.For example, radiologists can judge whether example-based explanations are useful based on their clinical experience.Source: §4.2.1 · p. 38

4.2.2. Human-grounded evaluation

• Human-grounded evaluation uses simpler experiments with laypeople, costing less than expert evaluation while yielding general quality judgments that remain proxies for actual explanation quality.One example is crowdsourced assessment of saliency maps.Source: §4.2.2 · p. 38

4.2.3. Functionally-grounded evaluation

• Functionally-grounded evaluation uses non-human proxies and can be inexpensive, but medical-image annotations may be resource-intensive unless suitable annotations already exist.Existing curated annotations can enable repeated automatic evaluation, including during explanation-method development.Source: §4.2.3 · pp. 38–39

Critique on XAI

• Rudin (2019) advised caution with black-box explanations for high-stakes decisions; explanations may be unfaithful or lack enough detail, while backpropagation and guided Grad-CAM showed similar saliency maps in both tests.Source: Critique on XAI · pp. 39–40
• Layer-wise relevance propagation and guided backpropagation produced the most coherent attribution maps, not fully in line with Adebayo et al. (2018); more research on this topic in medical image analysis is desired.The discussion notes an increasing number of papers with a holistic approach combining multiple forms of explanation.Source: Critique on XAI · p. 40
• The authors propose biological explanation as a future direction, including pathway analysis linking imaging phenotypes with biological processes.The cited examples of predicting molecular subtypes from imaging use biological targets for training; pathway analysis in the reverse direction could provide biological explanations.Source: Critique on XAI · p. 41
• Causal reasoning could help assess bias and generalizability in medical-image explanations, while the required sample size for XAI and deep learning remains unsettled.The discussion notes that even prototype methods may be sensitive to training-population differences and suggests incorporating causal analyses into image explanations.Source: Critique on XAI · p. 41
• The cost of medical-imaging datasets in money, time, and patient burden motivates guidelines for minimum sample sizes required by different XAI techniques.Source: Critique on XAI · p. 41

Limitations

• The survey cannot guarantee full coverage because XAI often supports methods and may go unmentioned in paper titles or bodies (Rudin (2019)); its appendix provides the search strategy for transparency.Source: Limitations · p. 42

5. Conclusion

• The paper surveyed 223 papers on XAI in deep-learning-based medical image analysis.The papers were classified using an XAI framework and categorized by anatomical location and imaging technique.Source: §5 · p. 42
• The survey classified papers according to an XAI framework and categorized them by anatomical location and imaging technique.Source: §5 · p. 42
• The paper discussed how to evaluate XAI, current critiques of XAI, and future perspectives for XAI in medical image analysis.Source: §5 · p. 42

6. Additional information

• The work was partially funded by the Dutch Cancer Society (KWF), grant 10755.Source: §6 · p. 43

7. Appendix

• The survey searched SCOPUS with a query combining explainability and interpretability terms with medical-imaging terms, then ranked results using an active-learning review toolbox.The toolbox updated its relevance ordering with user input.Source: §7 · p. 44
• The authors supplemented the search with colleague discussions and snowballing, screening titles and abstracts and browsing full content when inclusion was uncertain.Snowballing examined papers cited by included studies and papers that cited them.Source: §7 · p. 44
• The survey included peer-reviewed journal papers and conference proceedings through October 2020, preferring journal publications or the most recent conference paper for repeated same-author work on the same subject.Source: §7 · p. 44