Source-linked AI summary
Explainable deep learning models in medical image analysis
Amitojdeep Singh, Sourya Sengupta, Vasudevan Lakshminarayanan
TL;DR
Deep learning performs effectively in medical diagnosis, but its black-box nature has restricted clinical use and created a need for clinically meaningful explanations. This paper reviews explainable deep learning applications across medical imaging, covering attribution and domain-specific approaches, evaluation challenges, and clinical deployment considerations. It concludes that explainability methods can reveal relevant features, support expert decision-making, and require careful evaluation and human oversight.
Problem
Deep learning’s black-box nature has restricted clinical deployment despite strong medical diagnostic performance, motivating explainability research for medical imaging.
Method
The paper reviews explainable deep learning studies in medical imaging, comparing attribution methods with architecture- or domain-specific techniques and their applications.
Results
Attribution visualizations often showed models learning relevant features, while expert evaluation and clinical integration revealed task-specific benefits and trade-offs.
Takeaways & Limitations
Clinical deployment is likely to begin with human-in-the-loop systems that keep clinical experts in control while explainability provides additional guidance and safety support.
Takeaways & Limitations
The review’s non-attribution-method coverage is not exhaustive, and attribution evaluation remains difficult because model errors and explanation errors can be hard to disentangle.
Abstract
from arXiv · showhide
Deep learning methods have been very effective for a variety of medical diagnostic tasks and has even beaten human experts on some of those. However, the black-box nature of the algorithms has restricted clinical use. Recent explainability studies aim to show the features that influence the decision of a model the most. The majority of literature reviews of this area have focused on taxonomy, ethics, and the need for explanations. A review of the current applications of explainable deep learning for different medical imaging tasks is presented here. The various approaches, challenges for clinical deployment, and the areas requiring further research are discussed here from a practical standpoint of a deep learning researcher designing a system for the clinical end-users.
1. Introduction
Deep learning supports increasingly effective medical diagnosis, but its black-box nature and limited knowledge representation have slowed clinical deployment. This review organizes explainability approaches and examines their medical-imaging applications, practical challenges, and future directions.
- Motivation: Deep learning has achieved strong results across medical imaging tasks, yet black-box behavior and computational costs have limited significant clinical deployment.The paper contrasts deep models with simpler self-explanatory methods that lack sufficient complexity for many medical tasks.
- Motivation: Clinical diagnosis systems require transparency, understandability, and explainability to support trust, regulatory retraceability, and responsible use.The discussion connects explainability with safe, ethical, fair, and trustworthy deployment.
- Concepts: Interpretation maps abstract outputs to domain examples, whereas explanation identifies domain features such as image pixels contributing to a model decision.The paper also places model uncertainty alongside explainability because classifiers may return narrow-margin predictions without expressing uncertainty.
- Review scope: The review surveys explainability studies in medical imaging, compares attribution methods, reviews modality-specific applications, and summarizes trends and future possibilities.Its organization separates attribution-based methods from other explainability techniques and examines their clinical applications.
- Taxonomy: XAI taxonomies overlap because methods can be classified simultaneously according to different characteristics and logical intuitions.The paper discusses model-specific versus model-agnostic, local versus global, pre-model, in-model, post-model, surrogate, and visualization perspectives.
- Taxonomy: Local methods explain individual outcomes, while global methods characterize model behavior using broader knowledge of the model, training, and associated data.Feature importance is presented as an example of a global method.
3. Explainability methods - attribution based
Attribution-based explainability methods estimate how input features contribute to a target output and visualize those contributions as attribution maps. The section contrasts perturbation and backpropagation approaches, including their computational and evaluation trade-offs.
- Attribution maps: Attribution methods assign contribution or relevance values to input features relative to a target neuron, commonly the correct-class output.Arranging these values in the input shape produces attribution maps, with positive and negative evidence often shown in red and blue.
- Perturbation-based methods: Perturbation methods alter, mask, or remove input features, rerun the model, and rank features by the resulting output change.They are model agnostic and can reveal reliance on irrelevant or spurious features.
- Perturbation-based methods: Occlusion is computationally expensive because testing individual pixels requires repeated forward passes and can take several hours per image.Using patches such as 5x5 or 10x10 reduces the number of perturbations depending on target features and available resources.
- Perturbation-based methods: Shapley value sampling approximates feature contributions through repeated network evaluations and is slower than other attribution approaches.Its cost scales with the number of samples multiplied by the number of features.
- Backpropagation-based methods: Backpropagation methods estimate all feature attributions with a single forward and backward pass, making them faster but weakening the direct relationship with output variation.Some methods require repeated passes, but their runtime remains independent of the number of input features.
- Evaluation: Completeness requires attributions to sum to the target output minus the baseline output; integrated gradients, DeepTaylor, and Deep SHAP satisfy it, while DeepLIFT’s rescale rule does not.Sensitivity-n compares attribution sums with target-output variation using Pearson’s correlation coefficient, with occlusion reported to have a higher PCC than other methods.
- Evaluation: Attribution evaluation must distinguish model errors from explanation errors and balance behavioral fidelity against similarity to human expert expectations.Expert ratings can introduce observer bias, but may help develop explanations more likely to be trusted by end users.
4. Applications
The review organizes explainability applications in medical imaging into attribution-based methods and novel, architecture- or domain-specific techniques. It discusses methods alongside the medical imaging applications where they are used.
- Application categories: Applications are categorized into pre-existing attribution-based methods and other often specific explainability methods.The review discusses each category according to both explainability method and medical imaging application.
- Attribution-based applications: Attribution methods are widely used because researchers can add post-model analysis without making the neural network inherently explainable.This permits either pre-existing deep learning models or custom architectures while preserving task-specific performance choices.
- Attribution-based applications: Post-model attributions can reveal relevant or spurious learned features and inform adjustments to model architecture and hyperparameters.The reviewed applications span modalities including brain MRI, retinal imaging, and breast imaging.
Brain imaging
A brain MRI Alzheimer’s classification study quantitatively compared the robustness and repeatability of several attribution methods. Its findings indicated unreliable feature focus in some attribution maps.
- Attribution robustness: The study compared Gradient × input, GBP, LRP, and occlusion for Alzheimer’s classification using brain MRI.It quantitatively assessed robustness using the L2 norm between average attribution maps from multiple runs of the same model.
- Attribution robustness: GBP maps sometimes focused on non-tumor regions and spurious patterns, indicating unreliable features.The reported focus was associated with errors in the attribution analysis.
Retinal imaging
An integrated-gradients assistance system produced heatmaps alongside predictions for diabetic retinopathy grading. Assistance improved grading accuracy and confidence with experience, but accuracy decreased for patients without diabetic retinopathy.
- Diabetic retinopathy grading: Integrated-gradients heatmaps paired with model predictions increased ophthalmologists’ diabetic retinopathy grading accuracy over unassisted experts or model predictions alone.The system was evaluated as an assistance tool for grading.
- Diabetic retinopathy grading: Grading time initially increased with assistance, then decreased as users gained experience while grading confidence increased.The confidence increase was especially apparent when both predictions and heatmaps were used.
- Diabetic retinopathy grading: Accuracy decreased for patients without diabetic retinopathy when model assistance was used.
Breast imaging
IG and SmoothGrad visualizations showed that breast MRI CNNs learned clinically relevant spatial and dynamic features, while also exposing preprocessing artifacts.
- IG and SmoothGrad revealed relevant features in both spatial and dynamic domains for estrogen receptor classification from breast MRI.
- The visualizations exposed irrelevant features caused by preprocessing artifacts, prompting changes to preprocessing.
- CNNs learned mass edges, which were identified as the main clinical criteria, while remaining sensitive to context.
Skin imaging
Attribution maps showed that high-performing melanoma classifiers could still focus on irrelevant diagnostic features, and similar accuracy did not imply similar explanations.
- GradCAM and Kernel SHAP comparisons across 30 melanoma CNNs showed occasional attention to irrelevant diagnostic features, even in highly accurate models.
- Models with similar accuracy produced different explanations, indicating that neural network architectures learned different features.
- Attribution maps from both methods highlighted differences among the models’ learned features.
CT imaging
A DeepDreams-inspired attribution method explained liver CT tumor segmentation by probing feature sensitivity, distinguishing clinically relevant behavior in networks trained on real versus synthetic tumors.
- The method explained liver CT tumor segmentation by maximizing a target neuron’s activation through gradient ascent.
- Networks trained on real tumors were more sensitive to clinically relevant features than networks trained on synthetic tumors.
- The network responded to intensity and sphericity, consistent with domain knowledge.
X-ray imaging
Explainability methods for medical imaging range from attribution maps to attention, concept vectors, retrieval, language, rules, and intrinsically interpretable latent spaces. Across applications, they help inspect learned features, but often require domain-specific design and remain incomplete or architecture-dependent.
- X-ray imaging: GSInquire produced COVID-19 chest X-ray heatmaps and outperformed SHAP and Expected Gradients on impact score and impact coverage.
- Attribution methods: Attribution methods revealed relevant features in most reviewed cases, while spurious features were flagged and used to adjust model training.
- Non-attribution methods: Non-attribution approaches used attention maps, concept vectors, similar images, text justifications, expert rules, generative modeling, and hybrid machine-learning methods.
- Attention-based methods: Attention mechanisms visualized diagnostic processes by identifying predominant and discriminatory features linking medical images to diagnostic reports.
- Concept vectors: TCAV explained predictions through human-understandable concepts, while RCV represented continuous concepts in model activation space.
- Scope and limitations: Non-attribution methods can provide more domain-specific explanations but generally require greater design effort and are often post-hoc for specific architectures or domains.
5. Discussion
The discussion highlights that explainability must address both model reliability and clinical usability. Future systems should combine uncertainty, expert feedback, and human oversight while supporting richer multimodal decisions.
- Scope of explanations: Most explainability methods target local decisions for individual examples, while concept vectors offer a more global, class-level view.Global explanations are otherwise often approximated by averaging local highlights.
- Reliability concerns: A model can reach the right decision for the wrong reason, creating a reliability concern for real-world deployment.The discussion emphasizes examining which features drive apparently correct predictions.
- Research needs: Explainability research still needs methods that are more holistic and interwoven with uncertainty estimation.The paper also calls for studies measuring effects on clinical experts’ decision time and accuracy.
- Clinical deployment: Clinical applications are initially likely to use human-in-the-loop systems that retain expert control and responsibility for final decisions.The paper compares this arrangement with adaptive cruise control and lane-keeping assistance, which reduce workload while adding a safety net.
- Future directions: Future systems may attribute decisions across medical images and patient records to better reflect clinicians’ multimodal diagnostic workflow.The paper suggests this could potentially improve accuracy and make explanations more comprehensive.
Acronyms
The paper uses acronyms spanning explainable AI, medical imaging, diagnostic methods, neural networks, and clinical data sources.
- Explainability and diagnostics: XAI denotes Explainable AI, while AI denotes artificial intelligence and CAD denotes computer-aided diagnostics.
- Deep learning methods: CNN, DNN, and CAM denote convolutional neural network, deep neural networks, and class activation maps, respectively.
- Imaging and records: CT, fMRI, and EHR denote computerized tomography, functional magnetic resonance imaging, and electronic healthcare record, respectively.
- Explainability techniques: TCAV denotes Testing Concept Activation Vectors, and UBS denotes Uniform unit Ball surface Sampling.