Source-linked AI summary
A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
Erico Tjoa, Cuntai Guan
TL;DR
High-stakes medical decisions require interpretable machine-learning predictions, but deep-learning mechanisms and interpretability standards remain limited. The paper surveys and categorizes interpretability methods, applies the categorization to medical research, and discusses practical risks and challenges. It concludes that methods span perceptive explanations and mathematically structured analyses, while medical use must account for risks, responsibilities, and constrained interpretations.
Problem
Medical machine-learning decisions can affect lives, while deep-learning mechanisms remain poorly understood and interpretability criteria lack uniform adoption.
Method
The paper surveys interpretability research, categorizes methods, and applies the same categorization to medical interpretability.
Results
The survey distinguishes perceptive explanations from mathematical-structure approaches and reviews their medical applications, challenges, and evaluation perspectives.
Takeaways & Limitations
Clinicians and practitioners can use the categorization to approach interpretable algorithms with caution while considering medical risks and responsibilities.
Takeaways & Limitations
Explanations can be manipulated, and interpretations based on incomplete constraints may overemphasize selected constraints.
Abstract
from arXiv · showhide
Recently, artificial intelligence and machine learning in general have demonstrated remarkable performances in many tasks, from image processing to natural language processing, especially with the advent of deep learning. Along with research progress, they have encroached upon many different fields and disciplines. Some of them require high level of accountability and thus transparency, for example the medical sector. Explanations for machine decisions and predictions are thus needed to justify their reliability. This requires greater interpretability, which often means we need to understand the mechanism underlying the algorithms. Unfortunately, the blackbox nature of the deep learning is still unresolved, and many machine decisions are still poorly understood. We provide a review on interpretabilities suggested by different research works and categorize them. The different categories show different dimensions in interpretability research, from approaches that provide "obviously" interpretable information to the studies of complex patterns. By applying the same categorization to interpretability in medical research, it is hoped that (1) clinicians and practitioners can subsequently approach these methods with caution, (2) insights into interpretability will be born with more considerations for medical practices, and (3) initiatives to push forward data-based, mathematically- and technically-grounded medical education are encouraged.
I. INTRODUCTION
Machine learning’s growing use in high-stakes settings, especially medicine, makes accountability and interpretability pressing concerns. The survey reviews interpretability research, identifies inconsistent assessment practices, and categorizes methods for medical applications.
- Medical machine-learning errors can affect human lives, making early disease detection and reliable decisions especially consequential.
- Interpretability and explainability have become pressing issues because accountability is unclear when models fail or succeed unexpectedly.
- Interpretability assessment lacks uniform adoption across research, with related concepts defined inconsistently across diverse machine-learning implementations.
- The survey categorizes general interpretability research, applies the categories to medicine, and considers clinicians’ and practitioners’ perspectives.
- The authors include challenges and future prospects, discuss accountability and interpretability in ethics and law, and indicate applicability to artificial neural networks.
II. TYPES OF INTERPRETABILITY
The paper organizes interpretability research into distinct categories while acknowledging that no widely adopted standard exists. It contrasts immediately perceptible explanations with outputs requiring mathematical or cognitive processing.
- No widely adopted standard defines machine-learning interpretability, and different studies use different justifiable criteria.
- Perceptive interpretability produces information usually considered immediately interpretable, whereas mathematical-structure methods require another cognitive-processing layer.
- The survey categorizes interpretability methods and lists non-exhaustive examples, including mathematical formalization, visual explanations, and performance-oriented approaches.
A. Perceptive Interpretability
Perceptive interpretability explains model decisions through humanly perceivable information such as importance scores, heatmaps, reconstructions, optimized inputs, and verbal outputs. The section also highlights that these signals may not expose the underlying mechanisms reliably.
- A.1) Saliency: Saliency methods assign values to input components to indicate their importance for a model decision, using forms such as probabilities, super-pixels, or heatmaps.
- A.1) Saliency: Decomposition methods selectively process signals propagated through a model to produce interpretable information, often as heatmaps.
- A.1) Saliency: CAM forms heatmaps from pixel-wise activations, while LRP decomposes importance scores so layer-wise scores sum to the model output.
- Challenges and Future Prospects: Perceptive explanations can remain difficult to validate because visual heatmaps may highlight irrelevant regions and intermediate mechanisms are often not scrutinized.
- A.2) Signal Method: Signal methods transform neuron activations into interpretable forms, including reconstructed images from feature maps and inversions of internal representations.
- Challenges and Future Prospects: Textual explanations may describe objects without explaining the black-box mechanisms generating them, while recurrent-network signals and embeddings remain insufficiently probed.
B. Interpretability via Mathematical Structure
Interpretability via mathematical structure uses models, feature transformations, correlations, sensitivity, and optimization to expose how machine-learning systems represent information. The survey presents these approaches as useful but constrained by mathematical complexity, locality, and the gap between formal patterns and human intuition.
- Pre-defined Models: Pre-defined models embed interpretation through explicit terms and parameters chosen from hypotheses, existing knowledge, or empirical evidence.Linear models make variable contributions directly readable, while neural networks can obscure input-variable meaning.
- Feature Extraction: Feature extraction transforms and selects data, predictions, signals, or model parameters to reveal useful patterns, often requiring mathematical knowledge to interpret them.TCAV separates concepts with a hyperplane, while subspace methods identify directions containing substantial network information.
- Linearity: Linearity-based methods represent predictions through feature contributions or probe neural-network layers, while generalized additive models extend this framework with feature-specific functions and interactions.GAMs use a link function and component functions whose implementations depend on the task.
- Challenges and Future Prospects: The survey identifies unresolved boundaries: mathematical patterns may remain difficult to connect with human intuition, and interpretability claims can be local rather than global.TCAV may be global within each concept label while remaining local relative to the full dataset.
- Sensitivity: Sensitivity methods use gradients, perturbations, and locality to assess how representations or predictions change when inputs are modified.LIME seeks a locally faithful interpretable model while minimizing its complexity; local instability reduces reliability.
C. Other Perspectives to Interpretability
The paper evaluates interpretability through application-based, human-based, and functionally grounded approaches, while also examining invariance properties and data-driven criteria. It emphasizes that interpretability assessments may remain incomplete or non-unified across tasks.
- Data-driven Interpretability: Interpretability criteria may remain incomplete because test cases and justifications for proxy choices cannot cover every scenario.Collective agreement may strengthen support for criteria, but conflicting factions could prevent unique convergence.
- Invariances: Implementation invariance requires functionally equivalent networks to assign identical input-component importance despite differing implementations.For attribution methods, the paper expresses this as equal attribution values for corresponding input dimensions.
- Invariances: Translation-based input invariance is desirable for saliency methods, but the surveyed methods do not all satisfy it.Gradients/sensitivity-based and signal methods are reported as invariant, whereas integrated gradients is not.
- Evaluation of Interpretability: Application-grounded evaluation tests whether explanations improve another person's performance on the same real-world task.The example involves a model providing an explanation that helps a human perform diagnosis or another end-task better.
- Evaluation of Interpretability: Human-grounded evaluation uses real people on simplified or partial tasks when the full application is difficult to evaluate.The explanation itself is assessed in a generic valuation rather than through the complete end-task.
- Evaluation of Interpretability: Functionally grounded evaluation uses predefined proxy metrics such as sparsity, Dice coefficients, attribution values, or dimensionality-reduction components.The suitable metric depends strongly on the task and the relationship between objects and features.
III. XAI IN MEDICAL FIELD
In medical AI, interpretability involves risks and responsibilities because machine-supported decisions can affect human lives. The paper applies its general categorization to medical research but finds too little evidence for conclusive trends and calls for more comparative studies.
- Motivation: U-Net demonstrates that successful deep-learning medical segmentation can remain difficult to interpret because the neural network is still a black box.The paper presents U-Net as a successful model whose interpretability remains limited.
- Motivation: Medical interpretability extends beyond intellectual curiosity because risks and responsibilities accompany decisions that may affect human lives.The paper connects unaccountable machine decisions with ethical concerns and possible catastrophic exploitation.
- Categorization: Table III categorizes medical interpretability work by how methods are incorporated and by the organs targeted by diseases.The categories distinguish direct applications, methodological improvements, and comparisons across organs such as the brain and skin.
- Findings: The survey refrains from presenting a conclusive trend because the medical interpretability literature lacks a substantial number of significant studies.Its overview instead identifies a need for more informative comparisons of existing methods.
- Future Prospects: The paper recommends more studies comparing existing medical interpretability methods and clarifying how they contribute to interpretability.This recommendation is framed as a potential benefit for the XAI research community.
A. Perceptive Interpretability
Perceptive interpretability presents humanly accessible explanations such as saliency maps, verbal justifications, and visualizations in medical applications. The paper reports broad use of these methods but highlights insufficient evaluation and practical limitations before deployment.
- Challenges: Medical imaging complicates interpretability because data may be scarce, administratively difficult to obtain, high-dimensional, and computationally demanding.The paper notes that medical images can include 3D, multimodal, and time-evolving 4D volumes.
- Challenges: Ground-truth medical images may be incorrect or difficult to interpret because biological components and preprocessing transformations require specialized knowledge.The paper gives ADC and isotropic DWI as derivative modalities and notes preprocessing such as skull stripping.
- Saliency: Medical saliency research applies methods such as GradCAM, CAM, Guided Grad-CAM, feature occlusion, integrated gradients, and SmoothGrad to visualize model decisions.Examples span pleural effusion, brain-tumor grading, Alzheimer’s disease pathologies, and other medical classifications.
- Saliency: Comparisons of medical saliency methods have concluded that heatmaps require significant improvement before practical deployment.The paper cites comparisons involving multiple visualizations and tests of GradCAM and SHAP on dermoscopy images.
- Saliency and Verbal Explanations: Perceptive explanations include saliency values over input components and verbal statements such as asthma → lower risk for pneumonia mortality.Other medical systems provide textual justifications or decision rules for stroke and breast-mass prediction.
- Verbal Explanations: Rule-based explanations can encode clinically misleading associations when relevant variables or pathways are omitted from the model.The asthma–pneumonia example could be made more coherent by representing the ICU admission variable explicitly.
- Challenges: Medical saliency maps often lack sufficient evaluation of whether their utilities match clinicians’ practical needs.For lesion detection, the relevant explanation may concern more than simply highlighting the lesion, such as distinguishing haemorrhage types.
B. Interpretability via Mathematical Structure
Mathematical-structure approaches interpret medical ML by embedding domain knowledge, extracting features, analyzing correlations, clustering representations, or measuring sensitivity. The survey also highlights trade-offs: simplification can aid interpretation, but constrained models and neural mechanisms may remain only partially understood.
- B.1) Pre-defined Model: Pre-defined models provide a generic sense of how variables affect outputs by simplifying empirically observed system mechanisms.A cerebral-blood-flow kinetic model is incorporated into a fully convolutional network, partially linking denoising to CBF while leaving the network’s understanding unresolved.
- B.1) Pre-defined Model: Linearity is treated as inherently simple and therefore obviously interpretable in clinical-variable, metabolite-signal, and brain-state models.Deep learning is also used for PET pharmacokinetic modelling and MRS parameter fitting, with outputs interpreted relative to established models or signals.
- B.2) Feature extraction: Feature-extraction methods obtain interpretable representations through separable transformations, correlation-based graphs, and related measurements.The survey groups these methods around extracting features and evaluating them with measures such as correlations.
- B.2) Feature extraction: Correlation, dimensionality reduction, clustering, and sensitivity methods expose structure in medical data through transformed features, latent spaces, maps, or task-specific metrics.Examples include DWT-based EEG processing with neural classification, an F-SVD pipeline achieving around 98% classification accuracy, t-SNE on genetic data, and metrics such as Br and UBS.
- Challenges and Future Prospects: Medical interpretability remains fragmented and experimental, with an unresolved trade-off between ease of use, mathematical interpretability, and contribution to the subject matter.The survey identifies model simplification as a constraint and proposes augmenting models with components that might yield interpretable new insights.
C. Other Perspectives
Data-driven interpretability can compare a new medical case with similar database cases and visually present the similarity evidence to support user judgment.
- Data-driven: Case-Based Reasoning interprets medical evaluations by comparing a query case with similar existing database cases and visually presenting similarity proxies.Users can use these visual measures to accept or reject the algorithm’s suggested decision.
D. Risk of Machine Interpretation in Medical Field
Logical statements may appear interpretable while still lacking the intermediate reasoning needed to make their medical meaning clear. The survey also reports that several explanation methods may fail to work reliably on simple models.
- Risk of Machine Interpretation: The statement “has asthma→lower risk” can seem strange without clarification of the intermediate reasoning behind the patient-risk relationship.A human may infer that more aggressive treatment explains the association, but such humanly inferable reasoning cannot always be assumed.
- Risk of Machine Interpretation: LRP, deconvolution, and guided backpropagation are reported not to work for simple linear models, raising questions about their reliability.
IV. CONCLUSION
The conclusion presents interpretability as a cross-domain survey framework applied to medical ML, while emphasizing risks from manipulated explanations, incomplete constraints, noisy data, and unresolved black-box behavior. It therefore treats explanations as support for, not replacements for, clinical judgment and calls for broader evaluation and oversight.
- IV. CONCLUSION: The survey categorizes general ML interpretability methods, applies the categories to medicine, and discusses mathematical, visual, performance-related, and future-prospect dimensions.Figure 6 summarizes the reported challenges and prospects.
- IV. CONCLUSION: Adversarially altered inputs, arbitrary explanation manipulation, and noise can make medical explanations unreliable even when images appear unchanged.
- IV. CONCLUSION: Interpretations based on incomplete biological constraints may overemphasize the selected constraint because other mechanisms and kinetic-model simplifications remain unrepresented.
- IV. CONCLUSION: Noisy medical training labels can produce garbage-in-garbage-out outcomes because professional ground truths are not always absolutely correct.
- IV. CONCLUSION: Clinical deployment should retain human supervision because systematic methods for probing and correcting wrong explanations are not yet robust.The paper recommends treating interpretability information as complementary support in medical practice.
- IV. CONCLUSION: Researchers should compare interpretability methods fairly, display poor results, and evaluate utility rather than emphasizing only favorable performance metrics.The conclusion also notes that a unified notion of interpretability remains elusive and that black-box decisions carry exploitable risks.
- IV. CONCLUSION: A shift away from purely algorithm-centric interpretability may be needed, although authoritative deployment standards could also stifle research progress.