Source-linked AI summary
A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?
Subrato Bharati, M. Rubaiyat Hossain Mondal, Prajoy Podder
TL;DR
Healthcare AI is increasingly used, but opaque decisions create challenges for clinical adoption, trust, and accountability. This review systematically analyzes healthcare XAI literature using PRISMA-based methods, covering why, how, and when XAI is used and how explanations relate to trustworthy AI. It synthesizes XAI methods, evaluation dimensions, applications, and continuing research challenges.
Problem
Opaque and insufficiently transparent healthcare AI decisions complicate clinical implementation and raise concerns about trust, accountability, fairness, privacy, and safety.
Method
The paper conducts a PRISMA-based review of healthcare XAI literature and organizes the synthesis around XAI motivations, methods, timing, applications, evaluation, and trustworthiness.
Results
The review synthesizes healthcare XAI categories, applications, evaluation measures, and research trends while proposing a framework for choosing among classes of XAI approaches.
Takeaways & Limitations
The review concludes that explainability contributes to trustworthy healthcare AI and that explainable models may be favored over post-hoc explanations when developing such systems.
Takeaways & Limitations
Healthcare XAI methods can be computationally difficult to scale, and evidence remains limited on how people interpret and apply explanations in useful systems.
Abstract
from arXiv · showhide
Artificial intelligence (AI) models are increasingly finding applications in the field of medicine. Concerns have been raised about the explainability of the decisions that are made by these AI models. In this article, we give a systematic analysis of explainable artificial intelligence (XAI), with a primary focus on models that are currently being used in the field of healthcare. The literature search is conducted following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) standards for relevant work published from 1 January 2012 to 02 February 2022. The review analyzes the prevailing trends in XAI and lays out the major directions in which research is headed. We investigate the why, how, and when of the uses of these XAI models and their implications. We present a comprehensive examination of XAI methodologies as well as an explanation of how a trustworthy AI can be derived from describing AI models for healthcare fields. The discussion of this work will contribute to the formalization of the XAI field.
I. INTRODUCTION
This review frames healthcare XAI around the need to make increasingly capable medical AI understandable, transparent, and suitable for clinical use. It uses a PRISMA-based literature search to examine why, how, and when XAI is used in healthcare.
- Motivation: AI can support diagnosis, prevention, and tailored treatment, but healthcare deployment remains limited despite these capabilities.The review links limited deployment to concerns surrounding medical AI and its practical use.
- Motivation: Medical AI systems can match or exceed experts experimentally yet produce high false-positive rates in real-world clinical settings.Related concerns include prejudice, security, privacy, transparency, trust, fairness, informativeness, transferability, and causality.
- Conceptual framing: XAI and interpretable AI are distinguished in the literature: XAI emphasizes why a decision was made, whereas interpretability emphasizes how results or decisions can be understood.The review notes that some studies nevertheless use the terms interchangeably.
- Review method: The review follows PRISMA and PRISMA-ScR guidance to identify relevant literature published from 1 January 2012 to 2 February 2022.The search covers healthcare and medicine applications of XAI.
- Review method: 17,300 publications were identified, 16,850 were manually excluded, and 450 papers were evaluated for healthcare or medical-treatment relevance.The exclusions removed documents that did not match the review’s primary research area.
- Review aims: The paper organizes its contribution around why healthcare needs XAI, how XAI should be implemented, and when it is most appropriate.It also focuses on challenges in the emerging healthcare XAI field.
II. BACKGROUND
Healthcare AI spans diagnosis, imaging, prediction, treatment, monitoring, and robotic applications, but its clinical use is constrained by transparency, accountability, and the difficulty of verifying model reasoning. The background introduces visual and other XAI approaches for addressing these concerns.
- Healthcare applications: Healthcare AI applications include diagnosis, image classification, biomarker identification, genomic interpretation, monitoring, robotic surgery, wearables, and lifelogging.These applications use healthcare data to support automated and more personalized medical services.
- Clinical limitations: A major dermatology limitation is that AI cannot perform the tailored assessment provided by doctors.The review also stresses the need to verify that high-performing models recognize clinically correct image regions rather than insignificant findings.
- XAI approaches: Attention maps, class activation maps, salience maps, and occlusion maps are common visualization methods for explaining image-based medical models.Localization and segmentation are comparatively easier to understand because their outputs are images.
- Clinical risks: AI-assisted medical decisions require transparency and accountability because clinical interpretations can involve life-or-death risks and obligations.The review describes allowing opaque algorithms to make such decisions without sufficient transparency as irresponsible.
- XAI approaches: The review synthesizes healthcare XAI categories including dimension reduction, feature selection, attention mechanisms, knowledge distillation, and surrogate representations.It situates these categories within an overview of XAI systems and prior healthcare applications.
III. REASONS FOR USING XAI
The review argues that explainability is central to the practical acceptability of healthcare AI because clinical users must understand how medical models produce results. It evaluates explanation methods through qualities such as clarity, simplicity, fidelity, soundness, and completeness.
- Rationale for XAI: Deep-learning models can achieve high diagnostic accuracy, but their lack of transparency makes clinical implementation and result explanation difficult.The review reports that experts have considered explainability especially important for practical clinical use.
- Rationale for XAI: Explainability is presented as necessary for the acceptability of AI applications in medicine, where understanding model operation precedes adoption.The review connects this need to the practical use of AI in diagnostics and surgery.
- Explanation evaluation: XAI methods are assessed using interpretability qualities such as simplicity and clarity, together with fidelity qualities such as soundness and completeness.The review discusses example-based, attribution-based, and model-based explanations.
- Model-based explanations: Model-based descriptions satisfy completeness because they provide enough information to calculate an outcome for a given input.Global explanations are more likely to meet clarity, while local explanations can vary substantially across similar cases.
- Model-based explanations: Model interpretability and complexity are often estimated using model size, including feature counts, rule counts, nonzero weights, tree depth, and relationship complexity.These measures provide structural indicators of how complicated a model is.
B. Analyzing explanations based on attribution
Attribution-based explanations describe model contributions but lack a complete evaluation framework. Example-based explanation assessment is also constrained by limited tools and unmet conditions.
- Attribution-based explanations: Attribution-based descriptions provide only a limited account of model behavior and do not satisfy completeness requirements.No metrics were available to measure explanation completeness, although soundness can often be quantified and effectiveness empirically evaluated.
- Attribution-based explanations: Attribution techniques can be evaluated empirically, including against ground truth created for particular cases.
- Example-based explanations: Example-based evaluation remains insufficient because few tools identify influential cases and prototypes, and several assessment conditions remain unmet.Global examples may be clear and case-based explanations efficient when cases are understandable, but no assessment techniques were found for the other conditions.
V. APPLICATIONS SCENARIOS
Healthcare XAI applications use dimensionality reduction, knowledge distillation, feature importance, and interpretable surrogate models to clarify complex medical predictions. These approaches support applications ranging from pharmacological analysis to ICU outcomes and disease-risk prediction.
- Applications of XAI: Dimensionality reduction methods such as Laplacian Eigenmaps, ICA, and PCA are used to depict essential aspects of AI models.Reduced input dimensions can also support pharmacological side-effect estimation with multilabel and k-nearest-neighbor approaches.
- Applications of XAI: Knowledge distillation extracts information from complex models and simplifies it into student models that are easier to understand.Dimension reduction, tree regularization, and model compression may be combined in this process.
- Applications of XAI: Distilled information has been used to build interpretable ICU outcome models for outcomes such as death or ventilator use.
- Applications of XAI: Interpretable approaches include pneumonia-mortality descriptions, Bayesian rule lists for stroke prediction, and model-induced rules visualizing complex-algorithm probabilities.
- Applications of XAI: Feature importance methods describe extracted-feature properties, feature importance, and relationships between features and predicted outcomes.One critical-care application used feature weights to identify the top 10 extractable characteristics for mortality prediction.
D. Attention mechanism in XAI
Healthcare XAI combines attention, saliency, local surrogate, attribution, concept-based, and feature-based methods across clinical imaging and prediction tasks. These techniques improve interpretability or identify influential regions and features, but attention-based explanations can overwhelm users and lack precise answers.
- Attention mechanisms: Global and local attention use all words for context, while self-attention uses multiple mechanisms to locate relationships between words.
- Attention mechanisms: Attention mechanisms can improve interpretability and visualization by showing which ICU input characteristics have greater impact on clinical-event predictions.
- Attention mechanisms: Attention-based approaches identify regions of increased concern rather than providing precise responses to clinical end users.Information overload and warning fatigue are reported drawbacks that may make extensive displays undesirable.
- Other XAI applications: LIME explains any classifier by learning a locally faithful, interpretable surrogate based on linear models in an instance’s neighborhood.
- Other XAI applications: Attribution and saliency methods are applied to medical images to localize disease areas, highlight influential pixels, and support image grading or segmentation.Examples include LRP for Alzheimer’s regions, Grad-CAM and LIME for chest radiographs, CAM for prostate histology, and saliency propagation for vitiligo lesions.
- Other XAI applications: Concept-based methods connect model behavior with clinical biomarkers or continuous measurements, while SHAP-based approaches explain feature contributions in prediction tasks.TCAV identifies biomarkers linked to cardiac disease; regression concept vectors extend TCAV to continuous-valued measurements, and SHAP supports feature selection and model interpretation.
VI. DISCUSSION ON DIFFERENT ASPECTS OF XAI
The review connects explainability with trustworthy healthcare AI while emphasizing that medical interpretability carries risks and responsibilities because clinical decisions can involve death.
- VI. DISCUSSION ON DIFFERENT ASPECTS OF XAI: The review examines how explainability contributes to constructing trustworthy AI and offers a framework for choosing among XAI approach classes.It also provides definitions and contributes quantitative assessment measures for XAI.
- VI. DISCUSSION ON DIFFERENT ASPECTS OF XAI: Medical interpretability involves risks and duties beyond those considered in other fields because medical decisions may carry a risk of death.
A. Developing XAI to create trustworthy AI
Developing trustworthy healthcare AI requires attention to how explanations communicate model reasoning and how their quality is assessed.
- A. Developing XAI to create trustworthy AI: Explanation design should specify what is communicated and provide developers with step-by-step guidance and concrete diagrams.
- A. Developing XAI to create trustworthy AI: Misleading post-hoc explanations may be addressed through techniques that provide argumentation evidence for their assertions.
- A. Developing XAI to create trustworthy AI: Further work is needed to understand explainable-model performance in healthcare and develop rule-based or GAM-based approaches.
B. Complementary approaches to the development of trustworthy healthcare AI
Explainability alone may not establish trust in healthcare AI, so trustworthy development also requires attention to data quality, testing, and governance.
- B. Complementary approaches to the development of trustworthy healthcare AI: Explanations are not always essential or adequate to build faith in AI, because perceived capability, control, and predictability also matter.
- B. Complementary approaches to the development of trustworthy healthcare AI: Real-world healthcare data may be biased, inaccurate, or incomplete, making data quality and acquisition as important to model constraints as explainability.
- B. Complementary approaches to the development of trustworthy healthcare AI: Comprehensive testing, external validation, stability, fairness, privacy, and governing law are complementary considerations for trustworthy healthcare AI.
C. Trends in Existing Research
Existing XAI research spans multiple explanation categories and continues to address practical limitations in computation, scalability, human use, data standardization, and hybrid methods.
- C. Trends in Existing Research: XAI explanations are categorized as post-hoc or model-specific, with gradient-based, perturbation-based, and rule-based attribution among the described methods.
- C. Trends in Existing Research: Several XAI techniques are computationally difficult and may not scale to large datasets or complex models.
- C. Trends in Existing Research: Research still lacks sufficient information about how humans interpret and apply XAI explanations, while medical data standardization remains challenging.
- C. Trends in Existing Research: Recent work explores federated learning, hybrid XAI approaches, and intersections between quantum computing and XAI for healthcare.
D. Challenges and Future Prospects
Healthcare interpretability research remains fragmented, with unresolved opportunities beyond accuracy-focused feature selection and extraction. The review therefore evaluates recent work, identifies gaps, and recommends directions for developing XAI.
- Challenges: Existing interpretable techniques remain fragmented and experimental, while medical machine learning is still described as being in its infancy.
- Future prospects: Interpretability research may still contain substantial undiscovered opportunities despite emphasis on accuracy and performance in feature selection and extraction.
- Future prospects: The review summarizes the state of the art, evaluates research quality, identifies gaps, and recommends future studies focused on healthcare XAI.It discusses why XAI matters, when it is suitable, and how it can be implemented.
APPENDIX
The appendix lists the search terms used for explainable or interpretable AI in medical and healthcare contexts and provides contributor information.
- Search terms: The search query includes “Explainable AI,” “Interpretable AI,” “Explainable Artificial Intelligence,” “Interpretable Artificial Intelligence,” and “XAI.”It combines these terms with “medical” or “healthcare.”
- Contributor information: Subrato Bharati is affiliated with BUET and Concordia University and has authored or co-authored more than 50 publications.The listed publication venues include IEEE, Elsevier, Springer, and others.
- Contributor information: M. Rubaiyat Hossain Mondal earned his PhD from Monash University in 2014 and has worked across periods beginning in 2005 and continuing from 2014 onward.The passage identifies the Department of Electrical and Computer Systems Engineering as his doctoral department.