PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable Deep Learning Methods in Medical Image Classification: A Survey

Cristiano Patrício, João C. Neves, Luís F. Teixeira

arXiv:2205.04766v3eess.IVcs.AIcs.CVcs.LG

TL;DR

Limited interpretability has hindered clinical adoption of accurate deep-learning medical-imaging systems. This survey synthesizes XAI methods, datasets, evaluation metrics, and report-generation comparisons, finding differing metric-specific strengths among methods and persistent open challenges.

  • Problem

    Accurate deep-learning medical-imaging systems remain seldom adopted clinically because their decision processes are difficult to interpret.

  • Method

    The paper surveys visual, textual, example-based, and concept-based XAI methods, datasets, explanation-quality metrics, and report-generation approaches for medical imaging.

  • Results

    Report-generation methods show metric-specific strengths, with Selivanov et al. leading several BLEU and ROUGE-L measures while Jing et al. leads BLEU-3, BLEU-4, and METEOR.

  • Takeaways & Limitations

    The survey identifies open challenges and future research directions aimed at improving explanation trustworthiness and supporting clinical adoption of medical-imaging systems.

  • Takeaways & Limitations

    Concept-based explanations require manual concept-example annotations, which may be impractical for some medical-imaging modalities and require clinician involvement.

Abstract

from arXiv · show

The remarkable success of deep learning has prompted interest in its application to medical imaging diagnosis. Even though state-of-the-art deep learning models have achieved human-level accuracy on the classification of different types of medical data, these models are hardly adopted in clinical workflows, mainly due to their lack of interpretability. The black-box-ness of deep learning models has raised the need for devising strategies to explain the decision process of these models, leading to the creation of the topic of eXplainable Artificial Intelligence (XAI). In this context, we provide a thorough survey of XAI applied to medical imaging diagnosis, including visual, textual, example-based and concept-based explanation methods. Moreover, this work reviews the existing medical imaging datasets and the existing metrics for evaluating the quality of the explanations. In addition, we include a performance comparison among a set of report generation-based methods. Finally, the major challenges in applying XAI to medical imaging and the future research directions on the topic are also discussed.

1 INTRODUCTION

Deep learning has achieved human-level accuracy in medical image classification, yet its limited interpretability has hindered clinical adoption. This survey reviews explainable deep learning methods, datasets, and interpretable approaches for medical imaging diagnosis.

  • Motivation: Medical image analysis has reached human-level classification accuracy, but automated medical imaging remains seldom adopted in clinical practice.
  • Motivation: Interpretability research is important before deep learning methods can be deployed clinically, with recent work primarily analyzing pre-built models’ decision processes indirectly.
  • Survey scope: The survey comprehensively reviews recent advances in explainable deep learning applied to medical diagnosis.
  • Contributions: Its contributions include reviewing prior surveys, cataloguing medical-imaging interpretability datasets, and covering post-hoc and inherently interpretable approaches.

2 RELATED SURVEYS

Existing surveys cover medical-imaging interpretability through varied taxonomies, modalities, explanation types, and evaluation strategies, but leave gaps in dataset coverage, inherently interpretable models, and performance comparisons. This survey addresses these gaps by comprehensively reviewing explanation modalities and processes, datasets, and explainable approaches.

  • Existing survey coverage: Existing surveys classify medical-imaging interpretability methods by perceptive versus mathematical interpretability, attribution versus non-attribution, or explanation type.These reviews span different image modalities and include methods for analyzing feature contributions and generated explanation quality.
  • Existing survey coverage: Specialized surveys examine XAI for digital pathology, cancer detection in MRI, and automatic medical-image report or caption generation.Their analyses consider explanation criteria, evaluation strategies, datasets, model designs, and language or captioning metrics.
  • This survey’s contribution: This survey is the first to comprehensively review medical-imaging XAI across explanation modalities and processes while analyzing relevant datasets and their use in explainable approaches.It also includes a comparative analysis of existing surveys and addresses the field-wide coverage gaps identified above.
  • Identified gaps: Most prior surveys incompletely cover important medical-imaging datasets, particular interpretability aspects, inherently interpretable models, or performance comparisons.The reviewed literature often focuses on visual or textual approaches, while human evaluation of explanations also remains limited.
  • This survey’s contribution: Concept-based learning is highlighted as an under-exploited inherently interpretable approach that explains diagnoses through physician-aligned high-level concepts and supports physician-AI intervention.The survey treats concept-based methods as a contemporary trend within its proposed XAI taxonomy.

3 BACKGROUND IN XAI

XAI methods are organized by model dependence, explanation scope, and timing, while also differing in whether they provide visual, textual, example-based, or concept-based explanations. The section highlights post-hoc methods such as LIME, SHAP, and saliency techniques, alongside their computational and reliability trade-offs.

  • XAI taxonomy: XAI methods are categorized as model-agnostic or model-specific, global or local, and post-hoc or intrinsic.Model-agnostic methods explain arbitrary architectures, whereas model-specific methods require internal model information; global explanations address general data patterns, while local explanations target individual samples.
  • Explanation modality: Explanation modalities include saliency maps, semantic text descriptions, similar examples, and high-level concepts.These modalities correspond to feature attribution, text, examples, and concepts, respectively.
  • Perturbation-based methods: LIME is a post-hoc, model-agnostic method that locally approximates any black-box model with an interpretable surrogate around an individual prediction.For images, LIME randomly masks superpixels and uses the resulting perturbations to identify regions with the largest positive contribution.
  • Perturbation-based methods: SHAP explains predictions by estimating each feature value’s average contribution across all combinations of features, drawing on Shapley values from cooperative game theory.For stroke-risk prediction, explanations express the contributions of features such as age, gender, and Body Mass Index.
  • Saliency methods: Saliency methods provide pixel-importance maps at lower computational cost than perturbation-based methods, but gradient-based explanations can be ambiguous, unreliable, and sensitive to adversarial perturbations.Grad-CAM generalizes CAM using target-class gradients, while Integrated Gradients interpolates between a baseline and the original image to attribute prediction changes.

4 DATASETS

The survey reviews 25 publicly available medical imaging datasets spanning multiple image types, with chest X-ray datasets especially prominent. It also highlights dataset-scale and annotation-quality trade-offs, including risks from automated labeling and the value of expert annotations.

  • Dataset overview: The review covers 25 publicly available medical imaging datasets grouped by image type.Table 3 summarizes their main characteristics.
  • Annotation quality: VinDr-CXR addresses automated-labeling concerns by providing radiologist-level annotations in both training and test sets.Most chest X-ray datasets instead rely heavily on automatic rule-based labelers such as the CheXpert NLP tool.
  • Dataset annotations: Several datasets provide clinically meaningful annotations, such as bounding boxes, severity grades, lesion segmentation, and paired radiology reports.VinDr-SpinalXR includes bounding boxes for abnormal findings, Knee Osteoarthritis provides grades from 0 to 4, and COV-CTR pairs CT scans with Chinese and English reports.
  • Imaging modalities: The collection spans diverse applications, including dermatology, microscopy, retinal imaging, COVID-19 CT, and multimodal medical image databases.Examples include ISIC 2020, BreakHis, APTOS, COV-CTR, PEIR, and ROCO.
  • Chest X-ray datasets: Chest X-ray is the most popular modality by both dataset count and scale, supporting model training from scratch but creating concerns about label reliability.Some labels are automatically extracted and may be unverified or mislabeled.

5 XAI METHODS IN MEDICAL DIAGNOSIS · 5.1 Explanation by Feature Attribution · 5.2 Explanation by Text

The survey organizes medical-diagnosis XAI into feature attribution and textual explanation methods, covering perturbation, saliency, captioning, and concept-based approaches. It also highlights interpretability limitations, including ambiguous visual explanations and constraints of recurrent models for long reports.

  • 5 XAI METHODS IN MEDICAL DIAGNOSIS: The survey categorizes medical-diagnosis XAI methods to improve transparency and trustworthiness in clinical deployment, where uninformed decisions can endanger patients.
  • 5.1 Explanation by Feature Attribution: Feature attribution methods quantify each input feature’s contribution through importance scores or saliency maps, divided into perturbation-based and saliency methods.
  • 5.1.1 Perturbation-based methods.: Perturbation methods modify inputs to observe prediction changes; examples include LIME and SHAP, while RISE uses random masking to create pixel-level saliency maps.
  • 5.1.1 Perturbation-based methods.: Saliency methods assign pixel relevance to a target class using techniques such as CAM, Grad-CAM, DeepLIFT, and Integrated Gradients.
  • 5.1.2 Saliency Methods.: Feature-attribution explanations can be ambiguous and may require preprocessing to produce plausible visualizations, motivating exploration of textual explanations.
  • 5.2 Explanation by Text: Textual explanations are organized into image captioning, captioning with visual explanation, and concept attribution, often combining CNN image embeddings with recurrent language models and attention.
  • 5.2.1 Image Captioning.: Captioning approaches evolved from hierarchical LSTM and attention-based systems to retrieval-generation agents and Transformer decoders with memory modules for radiology reports.
  • 5.2.2 Concept Attribution.: Concept attribution learns human-defined concepts from CNN activations, with CAVs and TCAV providing human-understandable and quantifiable concept importance; unlike RNN text generation, it offers more objective interpretation.

5.3 Explanation by Examples

Example-based explanations clarify model decisions by retrieving similar cases, generating counterfactual inputs, or learning representative prototypes. These approaches can improve human interpretability, but their reliability depends on robustness and the availability and diversity of representative examples.

  • Explanation by Examples: Example-based explanation methods select similar examples to explain model decisions and comprise Case-Based Reasoning, Counterfactual Explanations, and Prototypes.These strategies also mirror how clinicians commonly explain the rationale behind decisions.
  • Case-Based Reasoning and Content-Based Image Retrieval: Case-Based Reasoning and Content-Based Image Retrieval search databases for visually similar entries to a query image, commonly using feature-vector distances to rank retrieved cases.The surveyed applications include clinical diagnosis, breast-mass classification, skin-cancer diagnosis, and visual similarity-based saliency maps.
  • Counterfactual Explanations: Counterfactual explanations perturb inputs in a controlled way to reverse a model prediction, producing post-hoc explanations that do not require access to model internals.Generative models can create image sequences or contrastive images that expose biomarkers or changes associated with a prediction.
  • Counterfactual Explanations: Diffusion probabilistic models are an under-explored future direction for medical-imaging counterfactual explanations because they can handle missing data and distributional shifts.The passage identifies these capabilities as benefits of using diffusion models for counterfactual generation.
  • Prototypes: Prototype-based models provide inherently interpretable explanations, but ProtoPNet can be vulnerable to adversarial and compression noise, which is consequential in healthcare.Example- and prototype-based systems also depend on having enough diverse, class-representative examples or prototypes.

5.4 Explanation by Concepts

Concept-based methods derive predictions through human-specified or automatically discovered concepts, making explanations interpretable by design. Approaches include concept bottleneck models, concept whitening, unsupervised concept discovery, visual concept mining, and ante-hoc concept encoders, but manual concept annotation remains a major limitation.

  • Concept Bottleneck Models: Concept Bottleneck Models predict human-specified attributes before using them to infer final classifications.This intermediate concept layer supports explanations through the concepts used for prediction.
  • Concept Bottleneck Models: Joint training was identified as the preferred Concept Bottleneck Model setup because it minimizes a weighted sum of classification and concept losses.Concept predictions can also be intervened upon to change the final output.
  • Concept Whitening: Concept Whitening replaces Batch Normalization so latent-space points can be interpreted through known concepts and their spatial localization.The approach decomposes the decision process into human-interpretable concepts with visual localization.
  • Automatic Concept Discovery: ACE discovers high-level pseudo-concepts without supervision by segmenting images, clustering latent-space segments, and retaining important concepts using TCAV scores.Visual Concept Mining extends this rationale to explain an infectious keratitis classifier with human-interpretable concepts.
  • Advantages and Limitations: Concept-based explanations are interpretable by design because final predictions derive from learned concepts, but manual concept annotation is their major limitation.The survey characterizes these methods as under-explored in medical imaging but promising for human-understandable explanations.

5.5 Other Approaches

Alternative interpretability strategies include Bayesian neural networks for prediction-uncertainty estimation and adversarial training for improving generated explanations. Bayesian methods support uncertainty-aware clinical decisions, while adversarially trained models can produce more coherent explanations but remain under-explored and sensitive to training noise.

  • Bayesian neural networks quantify uncertainty associated with model predictions, addressing the deterministic parameters and bias-related overestimation of conventional CNNs.
  • Monte Carlo dropout obtains a probability distribution over model predictions for uncertainty estimation.
  • Adversarial training augments training examples with adversarial perturbations, increasing model robustness to potential malicious examples.
  • Margeloiu et al. applied PGD-based adversarial training to CNNs for skin-cancer diagnosis on HAM10000 to improve interpretability.
  • Adversarially trained models produced explanations that appeared more coherent and sharper, while the approach remains under-explored and sensitive to training noise.The reported method was not ready for real-world deployment because saliency methods were sensitive to training noise.

6 EVALUATION METRICS

The survey separates explanation evaluation into visual and textual metrics. Visual methods include perturbation-, retraining-, region-, and Bayesian-consistency measures, while textual explanations are commonly assessed with NLP metrics such as BLEU, ROUGE-L, METEOR, and CIDEr.

  • Evaluation metric categories: Evaluation metrics are divided into measures for visual explanations and measures for textual explanations.The distinction reflects the explanation modality being evaluated.
  • Visual explanation metrics: Clinician and radiologist assessment is common but time-consuming and dependent on evaluator experience.This motivates objective metrics for explanation quality.
  • Visual explanation metrics: AOPC evaluates heatmaps by iteratively removing image regions according to their relevance and measuring the resulting model behavior.Deletion and insertion instead track class-probability degradation after removing salient pixels and increase after inserting them.
  • Visual explanation metrics: ROAR addresses replacement artefacts by retraining the model after removing important features and measuring accuracy degradation.The metric was proposed because modification-based deletion and insertion may not reflect the model’s actual reasoning.
  • Visual explanation metrics: POMPOM measures meaningful pixels outside the region of interest, while Activation Precision measures relevant information from the region used for classification.A Bayesian direction would sample explanations from posterior models and assess whether they consistently identify the same ROI.
  • Textual explanation metrics: The most-used textual explanation metrics are BLEU, ROUGE-L, METEOR, and CIDEr.BLEU compares modified n-gram precision; ROUGE-L measures longest-common-subsequence precision and recall; METEOR applies a chunk penalty; and CIDEr uses TF-IDF-weighted n-gram similarity.

7 PERFORMANCE COMPARISON

The comparison finds no universally best approach because methods use different evaluation metrics and data partitions. Among report-generation methods, Selivanov et al. and ICT show metric-specific advantages, while transformer models with feature-capturing mechanisms improve MIMIC-CXR report generation.

  • Comparison limitations: Different evaluation metrics often make direct performance comparisons between competing explanation methods infeasible.The review therefore compares only selected methods.
  • Findings+impression reports: Selivanov et al. achieved the strongest BLUE-1, BLEU-2, and ROUGE-L results among methods using findings+impression report sections.The approach combines preprocessing and squeezing of clinical records with Show-Attend-Tell and GPT-3.
  • Findings-only reports: ICT outperformed other findings-only methods on BLEU-2, BLEU-3, and METEOR by integrating inter-intra report features into report generation.Its transformer-based architecture uses auxiliary feature-capture and report-calibration modules.
  • MIMIC-CXR comparisons: TieNet and Liu et al. used distinct MIMIC-CXR data partitions, making their results unsuitable for direct comparison with most other methods.Their results are included primarily for reference.
  • MIMIC-CXR comparisons: Transformer-based models with mechanisms for capturing complex, relevant features improved generated-report performance on MIMIC-CXR.The survey identifies this pattern across several reviewed methods.

8 GENERAL DISCUSSION

The discussion identifies unresolved challenges in trustworthy medical-imaging XAI, especially unreliable saliency maps, limited interpretability-compliant datasets, and clinical deployment constraints. It highlights inherently interpretable, textual, example-based, uncertainty-aware, transformer-based, and concept-based approaches as directions for future research.

  • Interpretability challenges: Saliency maps from methods such as Grad-CAM, Integrated Gradients, and LRP remain the dominant interpretation approach, but can be unreliable, fragile, and focused on irrelevant regions.Different classes may also receive very similar explanations, and some explanations may be unusable.
  • Alternative explanation methods: Inherently interpretable models are a promising but largely unexplored direction for medical imaging, while textual, multimodal, example-based, prototype-based, and concept-based explanations offer alternative modalities.Textual explanations may be preferred because they are inherently understandable by humans, and some example-based approaches have shown promising classifier-comparable performance.
  • Future model directions: Bayesian Neural Networks could help estimate or quantify prediction uncertainty, although few studies have tested their effectiveness.Vision transformers are another future direction, offering medical-classification accuracy comparable to CNNs and built-in saliency maps.
  • Dataset limitations: Public medical-image-captioning datasets are limited and concentrated on anatomical regions such as the chest, leaving areas such as the breast and brain underrepresented.XAI datasets also require costly, time-consuming annotations for concepts and textual descriptions, reducing class representativeness.
  • Future research directions: Future work can use concept-based methods to bridge free-text report generation and clinician-supported reporting, while dataset augmentation can target under-represented samples and misclassification-causing concepts.BRACE and utility-based selection are described as approaches for identifying and adding informative samples to the training set.

9 CONCLUSIONS

The paper reviews advances in explainable deep learning for medical imaging diagnosis, compares existing surveys, and introduces prominent XAI methods to support discussion of recent developments.

  • The paper reviews advances in explainable deep learning applied to medical imaging diagnosis.
  • It compares existing surveys, highlighting their major conclusions and weaknesses.
  • It briefly describes prominent XAI methods to provide foundational concepts for discussing recent medical-imaging advances.

A APPENDIX · A.1 Intepretability Frameworks

The appendix surveys publicly available interpretability frameworks implementing classical XAI methods. It highlights LRP Toolbox, DeepExplain, TorchRay, and Captum, including their supported platforms and explanation algorithms.

  • A.1 Intepretability Frameworks: LRP Toolbox, launched in 2016, implements Layer-wise Relevance Propagation for artificial neural networks in Matlab and Python.An extension also supports the Caffe Deep Learning framework.
  • A.1 Intepretability Frameworks: DeepExplain is a framework implementing perturbation-based and gradient-based attribution methods.
  • A.1 Intepretability Frameworks: TorchRay implements Gradient, Guided-BackProp, Grad-CAM, and RISE visualization methods for PyTorch.It is research-oriented and provides code for reproducing results from several papers.
  • A.1 Intepretability Frameworks: Captum is a PyTorch library providing state-of-the-art algorithms for model interpretability and understanding.It includes general-purpose implementations of Integrated Gradients and SmoothGrad.
  • A.1 Intepretability Frameworks: The appendix presents a table of publicly available interpretability frameworks.
  • A.1 Intepretability Frameworks: The framework discussion appears in the September 2023 publication record.

A.2 Methods

The methods are summarized by interpretability method, image modality, and dataset. A dash indicates that explanations are provided only through text sentences.

  • Methods: Table 5 categorizes XAI methods by the interpretability method employed, image modality, and dataset.The table provides a structured overview of the surveyed methods across these dimensions.
  • Methods: A dash denotes that an explanation is provided only through text sentences.This notation distinguishes text-only explanations from other explanation formats represented in the table.
  • Methods: The method summary combines interpretability categories with the medical imaging modality and dataset used.These dimensions organize the comparison of surveyed XAI methods.
Loading 2205.04766v3…