PaperlayerView exact revision on arXiv

Source-linked AI summary

Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods

Zohaib Salahuddin, Henry C Woodruff, Avishek Chatterjee, Philippe Lambin

arXiv:2111.02398v1eess.IVcs.AIcs.CVcs.LG

TL;DR

Deep neural networks remain difficult to understand in medical imaging, despite their clinical potential. This narrative review categorizes nine interpretability-method types, examines explanation evaluation, and discusses limitations, guidelines, and future directions. It concludes that quantitative and clinician-grounded evaluation is necessary for robust, trustworthy explanations.

  • Problem

    Deep neural networks’ vague decision-making processes impede their incorporation into clinical workflows, creating a need for transparent and trustworthy explanations.

  • Method

    The review synthesizes interpretability methods for medical image analysis by discussing their technical details, limitations, applications, and qualitative and quantitative evaluation approaches.

  • Results

    The review identifies nine categories of interpretability methods and reports progress in evaluating the explanations they generate.

  • Takeaways & Limitations

    Quantitative evaluation and application-grounded assessment with medical doctors are needed to validate explanation utility, robustness, and potential bias.

  • Takeaways & Limitations

    Prototype-based models may fail to capture class-discriminative features when their fixed-size comparison patches are too limited.

Abstract

from arXiv · show

Artificial Intelligence has emerged as a useful aid in numerous clinical applications for diagnosis and treatment decisions. Deep neural networks have shown same or better performance than clinicians in many tasks owing to the rapid increase in the available data and computational power. In order to conform to the principles of trustworthy AI, it is essential that the AI system be transparent, robust, fair and ensure accountability. Current deep neural solutions are referred to as black-boxes due to a lack of understanding of the specifics concerning the decision making process. Therefore, there is a need to ensure interpretability of deep neural networks before they can be incorporated in the routine clinical workflow. In this narrative review, we utilized systematic keyword searches and domain expertise to identify nine different types of interpretability methods that have been used for understanding deep learning models for medical image analysis applications based on the type of generated explanations and technical similarities. Furthermore, we report the progress made towards evaluating the explanations produced by various interpretability methods. Finally we discuss limitations, provide guidelines for using interpretability methods and future directions concerning the interpretability of deep neural networks for medical imaging analysis.

1. Introduction

Medical imaging data support diagnosis and treatment but are increasingly difficult for clinicians to analyze, while opaque deep-learning decisions impede clinical adoption. The review defines interpretability, surveys methods and evaluation strategies, and addresses their clinical integration.

  • Medical imaging provides patient-specific information for diagnosis, treatment planning, and follow-up, but growing data volumes increase clinicians’ analysis burden.
  • Deep-learning models achieve strong performance across medical imaging tasks, yet their opaque internal decision processes hinder translation into clinical practice.
  • Explanations can reveal faulty algorithmic processes, limitations, biases, and potentially important imaging information that might otherwise go unnoticed.
  • Interpretability explains a model’s decision-making process in a way understandable to end-users and addresses why it makes a particular prediction.
  • The review uses systematic keyword searches and domain expertise to examine available methods, their grouping, advantages, limitations, and trustworthy-explanation evaluation.

2. Interpretability Methods

The review groups medical-imaging interpretability approaches by explanation type and technical similarity. Design-time and post-hoc methods include concept-based models that express predictions through clinically meaningful attributes.

  • The review identifies nine interpretability method types for medical image analysis based on generated explanations and technical similarities.
  • Concept Learning Models: Concept Learning Models first predict high-level clinical concepts and then use those concepts, alone or with deep features, to predict the final label.
  • Concept Learning Models: X-Caps predicts lung-nodule malignancy from visual attributes such as sphericity, margin, and texture, while HSCNN uses five semantic characteristics with low-level features.
  • Concept Learning Models: HSCNN achieved better results than some non-explainable 3D CNNs, while X-Caps approached the performance of 3D non-explainable CNNs for malignancy prediction.

2.2. Case-Based Models

Case-based models explain predictions by comparing image features with class-discriminative prototypes and combining similarity scores. Their interpretability can coexist with competitive performance, but prototype similarity has important robustness and semantic limitations.

  • Case-based models compare extracted image features with class-discriminative prototypes, making predictions from weighted prototype-similarity scores.
  • ProtoPNet uses a feature extractor, prototype layer, and fully connected layer to learn prototypes and convert their similarity scores into predictions.
  • ProtoPNet’s interpretability does not come at a performance cost compared with black-box deep-learning models.
  • Fixed-size prototype patches may miss discriminative features, and latent-space similarity can be corrupted by noise or compression artifacts or fail to match human-interpretable similarity.

2.3. Counterfactual Explanation

Counterfactual explanations generate minimally modified images that change a classifier’s prediction, helping identify influential disease regions and the changes associated with an alternative class. Implementations use GANs, autoencoders, or image-to-image translation, with generated-image quality and training stability as constraints.

  • A counterfactual explanation applies minimal perturbations to an image to maximize prediction change and switch the classifier’s predicted class.
  • Counterfactual images can identify diseased areas and indicate changes associated with switching the classifier’s prediction.
  • GANs and autoencoder latent-space perturbations are used to synthesize counterfactual images for medical-imaging explanations.
  • GAN-based approaches can be difficult to train because of loss-function instability and hyperparameter sensitivity, while autoencoder-generated image resolution may be limited.
  • StyleGAN2 and CycleGAN counterfactual methods were validated on osteoarthritis, histology, chest X-ray, and other medical-imaging prediction tasks.

2.4. Concept Attribution

Concept attribution explains deep neural network decisions using high-level image concepts, including continuous radiomics features, and quantifies their influence through directional analysis.

  • Concept attribution: TCAV quantifies how a high-level image feature influences a model’s classification decision using a concept activation vector and directional derivative.A linear classifier separates concept examples from random examples, defining the concept activation vector relative to its classification boundary.
  • Automated concepts: Automated Concept-based Explanation reduces the need for human concept labels by extracting image segments at multiple resolutions and comparing activation-space distances.This addresses the difficulty of creating labeled datasets for medical imaging concepts.
  • Continuous concepts: Radiomics features provide human-understandable continuous-valued concepts for explaining medical imaging models.Radiomics converts medical images into quantitative features that can support clinical decision-making.
  • Continuous concepts: Regression Concept Vectors extend TCAV by estimating radiomics features from CNN activations with a least-squares linear regression model.The influence of increasing radiomics concepts is assessed during testing using directional derivatives and quantified with Br and TCAV scores.
  • Robustness analysis: Gradient-based radiomic perturbations test segmentation-network robustness by changing input images along steepest-slope directions and tracking radiomic feature variations.A network is considered robust to a feature when it is not sensitive to that feature’s variations.

2.5. Language Description

Deep neural networks can accompany predictions with learned textual or emergent symbolic explanations, although natural-language justification models face training challenges.

  • Language Description: Deep neural networks can generate textual explanations alongside predictions as supervised natural-language justifications or unsupervised emergent-language justifications.These two forms differ in whether the explanatory language is explicitly learned from labeled descriptions or emerges without direct supervision.
  • Natural-language justifications: Natural-language justification models can be trained using structured diagnostic reports paired with medical images.The complexity of natural language and limited availability of structured medical reports make these models challenging to train.
  • Emergent language: Emergent symbolic language can provide interpretability by producing symbolic expressions that can be traced back to the input.A proposed interpretable segmentation framework uses a U-Net-like architecture with Sender and Receiver LSTM agents.
  • Visualization: Low-dimensional visualization with t-SNE can reveal dataset bias in a CNN caused by differences in image acquisition.The visualization projects a fully connected layer to make acquisition-related structure inspectable.

2.6. Latent Space Interpretation

Latent-space interpretation uses compressed or disentangled representations to expose learned anatomical and semantic factors, visualize classifier behavior, and inspect data structure, while generated-image quality can limit interpretation.

  • Latent-space foundations: A CNN latent space is a compressed representation of the input image, and disentanglement models salient variation factors independently.Autoencoders create this representation by encoding high-dimensional images into a bottleneck and decoding them.
  • Disentangled representations: Secondary classifiers can incorporate clinical knowledge into VAE latent spaces while reserving part of the space for unknown contributing factors.This approach was used for cardiac resynchronisation therapy response prediction; related work produced domain-agnostic, anatomy-preserving representations for liver segmentation.
  • Concept visualization: Concept-aligned latent-space axes can show how predefined concepts are learned, while interpolation along concept directions can visualize classifier behavior.These approaches were applied to skin-lesion diagnosis and other imaging classification tasks.
  • Limitations: Generated-image quality may be insufficient to observe changes when a VAE latent space is interpolated along a concept activation vector.This limits the interpretability of the resulting visualizations.
  • Feature-space visualization: PCA projects high-dimensional CNN features into a lower-dimensional space while preserving distances, enabling visualization of clusters and data structure.In brain-ventricle segmentation, projected latent representations clustered input images with certain characteristic features and supported Dice-score estimation.

2.7. Attribution Map

Attribution-map methods explain deep neural network predictions by highlighting influential image regions, using either gradients, relevance propagation, perturbations, attention, or training constraints. These approaches differ in computational requirements, localization detail, clinical meaningfulness, and whether they provide local or broader interpretability.

  • Overview: Attribution maps highlight image regions relevant to a model’s prediction, but heatmaps alone do not show how those regions contribute.This limitation motivates methods that provide more structured or clinically meaningful explanations.
  • Layerwise Relevance Propagation: Layerwise Relevance Propagation decomposes nonlinear classifiers pixel by pixel by propagating relevance scores backward from the output through successive layers.LRP preserves total relevance from the classification output to the input layer.
  • Class Activation Maps: Class Activation Maps localize class-specific regions through weighted sums of final convolutional activations, but require architectural changes that exclude fully connected layers.Multi-Layer Class Activation Maps extend this approach to different CNN layers.
  • Grad-CAM: Grad-CAM uses class gradients to weight feature-map activations without retraining or changing the architecture, then upsamples the result to input-image size.Its last-layer outputs lack fine-grained detail; Guided Grad-CAM restores detail, while Grad-CAM++ improves localization and handles multiple occurrences.
  • Integrated Gradients and Perturbation Methods: Integrated Gradients aggregates gradients between an input and a near-zero baseline, whereas occlusion and other perturbation methods infer importance by altering image regions and observing prediction changes.Perturbation methods can be computationally expensive and may alter regions that lack clinically understandable meaning, motivating guideline-based or generative replacements.
  • Other Methods: Saliency, attention, concept-based, and constrained-training methods provide alternative explanations, including gradient-based maps, clinically interpretable concepts, and attribution priors that promote smoothness or sparsity.Reported applications include improved localization, performance, clinical coherence, and reduced reliance on confounding variables, depending on the method.

2.8. Anatomical Prior

Anatomical priors make deep neural networks more interpretable by embedding task-specific structural information into the prediction workflow.

  • Anatomical Prior: Structural knowledge can guide models to identify clinically meaningful anatomical features before prediction.For midline-shift prediction, the approach first estimates the brain midline and then predicts shift from the estimated curve.
  • Anatomical Prior: Region-of-interest segmentation can restrict learning to specified areas and potentially enhance interpretability.

2.9. Internal Network Representation

Internal network representation methods visualize or quantify features learned inside convolutional networks to clarify their decision-relevant patterns.

  • Internal Network Representation: Activation Maximization transforms a randomly initialized image to reveal patterns that trigger a selected neuron.The network weights and desired output remain fixed during this transformation.
  • Internal Network Representation: Network Dissection measures alignment between neuron activations and annotated visual concepts using intersection over union.It has been used to study information organization in a brain-tumor segmentation model.
  • Internal Network Representation: Feature visualizations are useful for inspecting learned filters, but medical-image structures and patterns are often difficult to interpret.

3. Evaluation of Interpretability Methods

Evaluating interpretability methods requires clinical, quantitative, and robustness assessments because explanations are application-dependent and can be subjective.

  • Evaluation of Interpretability Methods: Explanation quality should be assessed with both functionality-grounded metrics and application-grounded evaluation involving medical doctors.The System Causability Scale is identified as one measure for medical-domain explanation quality.
  • Evaluation of Interpretability Methods: Clinical evaluation should test whether explanations help clinicians without introducing class-specific bias or reducing performance.
  • Evaluation of Interpretability Methods: Small input perturbations can radically change DeepLIFT and Integrated Gradient attribution maps even when images are visually indistinguishable and similarly labeled.This motivates robustness testing against perturbations and noise.
  • Evaluation of Interpretability Methods: Qualitative inspection alone cannot establish that explanations are robust, sensitive to model and data, and consistent.Quantitative metrics are needed, although no single quantitative metric applies to every interpretability method.
  • Evaluation of Interpretability Methods: Attribution maps may highlight correct image regions despite wrong predictions and can therefore create false confidence in clinical deployment.Randomization tests found differing faithfulness across methods: Guided BackProp and Guided Grad-CAM failed both tests, while Gradient and Grad-CAM passed both.
  • Evaluation of Interpretability Methods: Most attribution maps were unable to localize modality-specific key features.

4. Discussion

The review identifies transparency as essential for clinical use, summarizes limitations of post-hoc explanations, and highlights inherently interpretable design as a promising direction.

  • Discussion: The review identifies nine categories of interpretability methods and emphasizes evaluation as necessary for validating explanation utility.It also notes that segmentation is especially difficult because the relevance of a single pixel is limited.
  • Discussion: Post-hoc explanations are approximations that may misrepresent model behavior, complicate troubleshooting, and fail to ensure complete accountability.The review recommends careful evaluation before clinical workflow integration.
  • Discussion: Interpretability can be incorporated during network design, including through domain expertise, additional data, and clinically meaningful concepts.Examples include multimodal data and annotated clinical concepts for diagnosis.
  • Discussion: Concept-learning and case-based models can be interpretable by design while achieving performance comparable to black-box CNNs in medical imaging.
  • Discussion: Interpretability methods can reveal candidate imaging biomarkers, including nuclear and stromal morphology and lymphocyte infiltration for estrogen-receptor prediction.
  • Discussion: Future directions include combining images, text, and genomics with interactive explainability solutions using knowledge bases and graph neural networks.

5. Conclusion

Clinical adoption of deep neural networks for medical imaging requires greater transparency and careful evaluation of interpretability methods. The review emphasizes that inherently interpretable models can match black-box performance, while post-hoc explanations require caution.

  • Interpretability methods address the vague understanding of deep neural networks’ decision-making processes that impedes clinical incorporation.
  • Quantitative, qualitative, and doctor-involved application-grounded evaluations are necessary to assess explanation robustness, utility, and potential bias.Medical-doctor evaluations can detect bias associated with over-diagnosis or under-diagnosis.
  • Imaging biomarkers can validate interpretability methods, while these methods may also help discover new imaging biomarkers.
  • Case-based and concept-learning models have achieved performance at par with black-box networks while remaining inherently interpretable.
  • Post-hoc interpretability methods should be used carefully because their approximations of model behavior can create false confidence in AI solutions.
Loading 2111.02398v1…