PaperlayerView exact revision on arXiv

Source-linked AI summary

Designing Interpretable ML System to Enhance Trust in Healthcare: A Systematic Review to Proposed Responsible Clinician-AI-Collaboration Framework

Elham Nasarian, Roohallah Alizadehsani, U. Rajendra Acharya, Kwok-Leung Tsui

arXiv:2311.11055v2cs.AIcs.HCcs.LG

TL;DR

Healthcare AI can produce decisions that are difficult for clinicians to understand, creating a need for interpretable systems. This systematic review synthesizes interpretability approaches and finds that post-processing methods and black-box models predominate, while applications to wearables and generative AI remain limited.

  • Problem

    Complex AI algorithms make clinical decisions difficult to understand, while black-box outputs challenge trust and adoption in healthcare.

    Source: §1 · p. 2, 1.1.The general concept of… · p. 3

  • Method

    The paper conducts a systematic review following PRISMA criteria, including defined search, selection, and data-extraction procedures.

    Source: §2 · p. 10

  • Results

    Post-processing methods predominate in healthcare explainability, black-box models are more common than white-box models, and XAI use remains limited for wearables, biosensors, LLMs, and generative AI.

    Source: §7 · p. 33

  • Takeaways & Limitations

    The review proposes interpretable, trustworthy healthcare AI through explicit design goals, diverse end-user engagement, interdisciplinary collaboration, and attention to fairness, privacy, accountability, sustainability, and robustness.

    Source: §7 · p. 34, §9 · p. 42

    Explainability alone does not guarantee correct decisions, user trust, or clinical justification, although it can support model auditing and bias detection.

    Source: §9 · p. 42

Abstract

from arXiv · show

This paper explores the significant impact of AI-based medical devices, including wearables, telemedicine, large language models, and digital twins, on clinical decision support systems. It emphasizes the importance of producing outcomes that are not only accurate but also interpretable and understandable to clinicians, addressing the risk that lack of interpretability poses in terms of mistrust and reluctance to adopt these technologies in healthcare. The paper reviews interpretable AI processes, methods, applications, and the challenges of implementation in healthcare, focusing on quality control to facilitate responsible communication between AI systems and clinicians. It breaks down the interpretability process into data pre-processing, model selection, and post-processing, aiming to foster a comprehensive understanding of the crucial role of a robust interpretability approach in healthcare and to guide future research in this area. with insights for creating responsible clinician-AI tools for healthcare, as well as to offer a deeper understanding of the challenges they might face. Our research questions, eligibility criteria and primary goals were identified using Preferred Reporting Items for Systematic reviews and Meta-Analyses guideline and PICO method; PubMed, Scopus and Web of Science databases were systematically searched using sensitive and specific search strings. In the end, 52 publications were selected for data extraction which included 8 existing reviews and 44 related experimental studies. The paper offers general concepts of interpretable AI in healthcare and discuss three-levels interpretability process. Additionally, it provides a comprehensive discussion of evaluating robust interpretability AI in healthcare. Moreover, this survey introduces a step-by-step roadmap for implementing responsible AI in healthcare.

1. Introduction

The introduction frames healthcare AI as increasingly capable but difficult to understand, creating mistrust and motivating interpretable, explainable systems that balance predictive performance with transparency. It positions this review as a systematic examination of interpretability across the clinical decision-support pipeline, including methods, applications, evaluation challenges, and implementation guidance.

1. Introduction

  • Healthcare AI increasingly relies on complex deep models that can extract clinical patterns effectively but create a Black-Box challenge because their decisions become difficult to understand.This complexity arises as model layers, features, and hyperparameters increase, making DNN outputs harder to understand and trust than classical ML outputs.Source: §1 · p. 2

1.1.The general concept of interpretability in healthcare

  • Interpretability helps clinicians and other users understand predictions, identify errors and bias, assess reliability, and build confidence in AI decision-making.Black-Box opacity is especially consequential in healthcare, where users need understandable results for clinical decisions and patient care.Source: 1.1.The general concept of… · p. 3

1.2.The Importance of transparency in healthcare

  • Healthcare AI requires transparency because inaccurate or opaque decisions can harm patients, motivating white-box, gray-box, or explainable black-box approaches to support clinician-AI communication.White-box models prioritize understandability, black-box models prioritize predictive accuracy, and gray-box models seek a middle ground [22].Source: 1.2.The Importance of transparency in… · p. 4

1.3. Balancing accuracy and interpretability in healthcare systems

  • Interpretability and accuracy can conflict, so healthcare systems should address interpretability throughout data collection, modeling, and decision-making rather than treating explanations as an afterthought.The paper emphasizes that no general scientific balance between accuracy and interpretability has been established, while explanations can improve trust, clinician-AI communication, model comparison, and system updating.Source: §1.3 · pp. 5–6

1.4.Role of explainability in digital health interventions (DHIs)

  • XAI is intended to make digital health interventions more communicable and transparent for patients, families, professionals, managers, and data services, while addressing privacy, security, inclusion, diversity, and transparency challenges,.The paper applies this perspective to EHRs, telemedicine platforms, and clinical decision support systems.Source: 1.4.Role of explainability in digital… · pp. 6–7

1.5.Current approaches of interpretability in health application

  • Interpretability may be inherent in models such as DT and LR or added post hoc, but saliency maps, feature visualizations, LIME, and SHAP can leave explanations subjective, incomplete, or vulnerable to confirmation bias.The section therefore highlights a preference for inherently interpretable models in high-stakes decisions while recognizing that complex medical data often requires post-processing explanations.Source: 1.5.Current approaches of… · pp. 7–8

1.6. Motivations and aims of this review

  • This review addresses the lack of a whole-pipeline account of healthcare interpretability by examining pre-processing, interpretable modeling, and post-processing, alongside applications, implementation challenges, evaluation, and a step-by-step roadmap.It also stresses that explanations lack consistent performance guarantees and should complement rigorous safety, effectiveness, bias, and outcome evaluation rather than replace it.Source: §1.6 · pp. 9–10

2. Systematic review method

The review followed PRISMA and PICO-guided procedures, using structured searches, independent screening, quality appraisal, and standardized data extraction. It ultimately included 74 studies and positioned its contribution as comprehensive coverage of interpretability, evaluation, implementation, and clinician-AI communication.

2. Systematic review method

  • The review followed PRISMA criteria, incorporating a defined protocol for searching, study selection, and data extraction.Source: §2 · p. 10

2.1.Search strategy and selection criteria

  • Searches used Boolean strings across PubMed, Scopus, and Web of Science, guided by nine conceptual frameworks covering remote monitoring, healthcare technologies, XAI, and IML,.The final search was conducted on June 16, 2023, after preliminary keyword-based evidence mining.Source: 2.1.Search strategy and selection… · pp. 11–12
  • Independent reviewers screened records through EndNote21 and Covidence, removing duplicates and resolving disagreements by consensus.Studies uncertain at title-and-abstract screening proceeded to full-text review.Source: 2.1.Search strategy and selection… · p. 12

2.2.Quality appraisal, risk of bias assessment and search for grey literature

2.3.Data analysis and search results

  • 74 studies were selected from 2241 records: 10 reviews and 64 experimental publications, after duplicate removal, title-and-abstract screening, full-text exclusion, and critical appraisal.The included studies were analyzed and narratively summarized using descriptive statistics in tables or graphs,.Source: 2.3.Data analysis and search results · p. 13

2.4.Data mapping of included studies

3. Proposed interpretability process for intelligent health system · 3.1.Data pre-processing interpretability and its health application

The paper proposes a three-level interpretability process spanning data pre-processing, model selection, and post-modeling, with explanations tailored to different healthcare end-users. Its pre-processing level improves understanding of datasets and features through data-specific analysis, standardization, summarization, augmentation, and explainable feature engineering.

3. Proposed interpretability process for intelligent health system

  • The proposed healthcare interpretability framework integrates pre-processing, model selection, and post-modeling explanations to tailor understanding to developers, clinicians, patients, and other end-users.It aims to make explainable-system design more cost-effective and explanations more focused and understandable than generalized explanations.Source: §3 · p. 16, Figure 9 · p. 17

3.1.Data pre-processing interpretability and its health application

  • Pre-processing interpretability describes datasets and features while supporting cleaning, normalization, balancing, exploratory analysis, feature engineering, dataset standardization, summarization, and knowledge-graph use.Because training data substantially influence model decisions, understanding and preparing inputs is presented as necessary for explainable, efficient, and robust AI systems.Source: 3.1.Data pre-processing… · p. 18

3.1.1. Considering diverse types of datasets

  • Interpretability methods should be tailored to data characteristics, including dimensionality, structure, modality, and evolving formats such as IoT sensor and streaming data, to support effective analysis and interpretation.For medical notes, NLP methods can extract meaningful information from textual data, while clinical tabular data may use feature-importance analysis and decision-tree visualization [46].Source: §3.1.1 · pp. 18–19

3.1.2. Exploratory data analysis (EDA)

  • EDA identifies dataset statistics, missing samples, and potential bias, while visualization, PCA, t-SNE, and UMAP support interpretation of complex or high-dimensional medical data [58] [59].PCA is described as suitable for mostly linear structure, whereas t-SNE is preferred otherwise but may be slow on large datasets; UMAP may be more efficient.Source: §3.1.2 · p. 20

3.1.3. Dataset description standardization

  • Dataset-description standardization improves communication about bias, missing values, provenance, collection, legal considerations, and ethics through tools such as Datasheets for Datasets and Data Declarations for NLP.Applications include noise removal, image resizing, CutMix, MixUp, and sensor-based feature extraction in healthcare studies [62].Source: §3.1.3 · pp. 20–21

3.1.4. Dataset summarizing methods

  • Dataset summarization preserves representative samples or weighted smaller datasets to make case-based explanations practical when storing the full training set is costly, using methods such as prototype selection and data squashing.A cardiovascular-diagnosis example segmented PPG signals into 30-second intervals and normalized each chunk [68].Source: §3.1.4 · p. 21

3.1.6. Explainable feature engineering

  • Explainable feature engineering uses domain-specific or model-based approaches to identify relevant, understandable attributes, including an Attention ResNet18 method for asthma prediction and pre-modeling analysis for post-stroke stress and anxiety diagnosis [74].The Attention ResNet18 workflow included pre-processing, network training, deep-feature extraction, feature prioritization, and classification.Source: §3.1.6 · pp. 21–22

3.2.Interpretable process of model selection and its health application · 3.3.Post-processing interpretability and its health application

Model selection interpretability spans white-box, hybrid, joint, and architecturally adjusted models, balancing explainability against predictive accuracy in healthcare. Post-processing methods explain completed models through attribution, visualization, examples, game theory, knowledge extraction, and neural interpretation.

3.2.Interpretable process of model selection and its health application

  • White-box, hybrid, joint, and architecturally adjusted models provide healthcare interpretability, but model selection still requires balancing simplicity, explainability, accuracy, and computational demands.Decision trees, logistic regression, hybrid neural models, explanation-integrated training, and modified CNN or vision-language architectures illustrate these approaches.Source: 3.2.Interpretable process of model… · p. 22, §3.2.1 · p. 23, §3.2.2 · p. 23

3.2.1. Inherently interpretable machine learning models (White-Box)

  • White-box models such as decision trees, K-nearest neighbors, Bayesian rule lists, and regularized logistic regression support understandable clinical predictions, though high-dimensional data can reduce explainability.Applications include dementia, COVID-19, metabolic disease, stroke, and cardiovascular disease detection.Source: §3.2.1 · p. 23

3.2.2. Hybrid interpretable methods

  • Hybrid methods combine interpretable white-box components with black-box models to improve performance and explanation, including cardiac MRI diagnosis, acute coronary syndrome risk prediction, and early stroke detection.Examples include DkNN, SENN, CEN, BagNets, neural-symbolic models, and combinations such as logistic regression with ANN.Source: §3.2.2 · pp. 23–24

3.2.3. Joint prediction and interpretable methods

  • Joint methods generate predictions and explanations together, using explanation-augmented training, multimodal explanations, semantic attention, neural additive models, and attention-based recurrent networks in medical applications.Applications include medical image segmentation, heart-attack prediction, and mortality prediction in MIMIC-III.Source: §3.2.3 · p. 24

3.2.4. Interpretability through architectural adjustments

  • Architectural adjustments can improve both understanding and accuracy, as shown by an explainable multi-frequency CNN for EEG abnormality prediction and a pre-trained vision-language model for brain tumor diagnosis.The cited CNN produced more understandable information and higher accuracy.Source: §3.2.4 · p. 24

3.3.Post-processing interpretability and its health application

  • Post-processing interpretability explains already-built models through six categories: attribution, visualization, example-based, game theory, knowledge extraction, and neural methods.Most XAI research attention has focused on post-modeling explainability, which aims to make pre-developed methods understandable to users.Source: 3.3.Post-processing interpretability… · p. 24

3.3.1. Attribution methods · 3.3.2. Visualization methods

  • Attribution and visualization methods expose feature or pixel contributions through techniques such as LIME, Grad-CAM, SHAP, PDP, ALE, ICE, feature importance, and surrogate models for healthcare prediction.Applications include chronic-impairment activity recognition, asthma detection, hepatitis and liver conditions, and hypertension prediction.Source: §3.3.1 · pp. 24–25, §3.3.2 · p. 25

3.3.3. Example-based explanation methods · 3.3.4. Game theory methods

  • Example-based explanations use prototypes, criticisms, counterfactuals, and adversarial examples, while SHAP assigns feature contributions to individual predictions for personalized healthcare decision support.SHAP applications include stress forecasting in pregnant mothers, amiodarone-induced thyroid dysfunction risk, and heart-rate-variability mental-stress detection.Source: §3.3.3 · p. 26, §3.3.4 · p. 26

4. Application of interpretability in AI-based medical devices

AI-based medical devices support prediction, diagnosis, treatment, and disease management, but interpretability is needed to address trust, communication, and inclusion concerns. The section reviews XAI applications in wearable devices and medical LLMs, emphasizing transparency, oversight, and responsible deployment.

4. Application of interpretability in AI-based medical devices

  • AI-based medical devices enhance prediction, diagnosis, treatment, and disease management, with potential to expand healthcare access and reduce global disparities.Source: §4 · p. 27
  • End-users remain concerned about the trustworthiness of IoT sensors, wearables, telemedicine, medical LLMs, and digital care twins, requiring better interpretability and clinician-AI communication.Source: §4 · p. 27

4.1.Interpretability applications in IoT-based biosensors and wearable medical devices

  • XAI applied to IoT biosensors and wearable trackers improves transparency, helps users understand model insights, and supports more informed healthcare decisions.Table 3 summarizes selected wearable-device applications, while the broader IoMT domain extends beyond those examples.Source: 4.1.Interpretability applications in… · p. 28, Table 3 · p. 28

4.2.Interpretability applications in ChatGPT and medical LLMs

  • Medical LLMs such as GPT-4 and Bard could support documentation and patient inquiries, but healthcare integration requires safety, ethics, privacy, accuracy, and human oversight.Without responsible implementation, generative systems may produce misinformation or inaccurate content that compromises patient care and trust.Source: 4.2.Interpretability applications in… · p. 29
  • ChatGPT’s opacity, potential errors, misinformation, and training-data biases require clinicians to review and validate suggestions before clinical use.Only a limited number of medical LLM studies have applied XAI, and detailed method descriptions are deferred to existing literature.Source: 4.2.Interpretability applications in… · p. 29, Table 4 · p. 29

5. Quality evaluation and improvement of the interpretability process

The framework evaluates interpretability by examining whether explanations are comprehensible and useful to healthcare end-users, especially those involved in clinical and health-system decisions. Its quality assessment spans technical, human-centered, ethical, regulatory, and clinical dimensions, supported by Table 5 and Figure 13.

Quality evaluation

  • Evaluation prioritizes explanation comprehensibility for patients, doctors, nurses, caregivers, health system managers, and other domain experts as practical decision-makers.The assessment is intended to gauge the XAI system’s utility and efficacy in real-world healthcare and medicine.Source: §5 · p. 29

Quality dimensions

  • The quality-control framework covers 12 dimensions: accuracy, reliability, robustness, interpretability, usability, human-AI interaction, ethics, feedback responsiveness, regulatory compliance, clinical validation, transparency, and scalability.Table 5 provides a comprehensive description of these quality-control and improvement dimensions.Source: §5 · p. 30, Table 5 · p. 30

Quality visualization

  • Figure 13 maps interpretability quality in human-AI interaction through attributes including scalability, reliability, robustness, usability, and transparency.The figure provides a visual representation of explainability-process quality levels.Source: Figure 13 · p. 30, §5 · p. 30

6. Step-by-step roadmap to implement responsible clinician-AI-collaboration framework

The roadmap proposes a responsible clinician-AI-collaboration framework centered on a Health-to-Data Center and a three-stage interpretability process. It connects explainable pre-processing, model processing, post-processing, and quality assessment to end-user needs and trustworthy healthcare AI.

Health-to-Data Center

  • The framework establishes a Health-to-Data Center with four coordinated bodies: a Clinician-AI Communication Desk, Data Scientist Help Desk, Quality Assessment Board, and Ethical Review Board.These groups support step-by-step AI decision-making and implementation in intelligent health systems.Source: §6 · p. 31

Interpretability workflow

  • Figure 14 depicts the workflow from the Data to Health Center through explainable data pre-processing, model processing, and post-processing to quality assessment and end-user interaction.The framework tailors machine-learning decisions to specific end-user needs.Source: §6 · p. 31, Figure 14 · p. 32

Interpretability and evaluation

  • The three-level interpretability approach tailors explanations to patients, clinicians, health managers, or developers by considering questions at each stage of data pre-processing, model selection, and post-processing.The process is followed by quality-assessment tools that evaluate machine-learning decisions and the robustness of the XAI system.Source: §6 · p. 32
  • Robustness evaluation assesses model resilience and stability against adversarial attacks, data perturbations, and variations in real-world scenarios.Quality tools also evaluate decision accuracy and efficacy under diverse conditions to support reliability and trust in healthcare AI.Source: §6 · p. 32

7. Discussion

The review finds growing use of AI-based medical devices but persistent gaps in interpretable applications for wearables and LLMs. It emphasizes post-hoc explainability, human oversight, and clinician–data scientist–ethics collaboration to improve trust and safety.

Explainability approaches

  • Post-hoc explainability predominates in healthcare applications, with SHAP and LIME among the most frequently used methods despite continued reliance on black-box models.Post-hoc methods exceed 100% of cases, indicating that some studies used multiple methods.Source: §7 · pp. 33, 35, Figure 15 · p. 35

Research gaps

  • Only a small fraction of wearable-health-tracker studies used XAI, while research applying XAI to healthcare LLMs and generative AI remains limited.The review identifies both areas as priorities for further research on trustworthy and interpretable healthcare systems.Source: §7 · pp. 33–34

Wearable devices

  • Integrating XAI into IoT medical sensors and wearable trackers could improve transparency, help users understand model insights, and support more informed healthcare decisions.The review notes that Table 3 presents only selected examples of XAI applications in wearable medical devices.Source: §7 · p. 33

Large language models

  • Healthcare LLMs require robust regulation, ethical guidelines, human oversight, and professional validation because opacity, bias, misinformation, and errors can compromise diagnoses, treatment recommendations, privacy, safety, and trust.The review specifically warns that ChatGPT suggestions should be thoroughly reviewed before entering clinical decision-making, from pre-processing through post-processing.Source: §7 · p. 34

Responsible collaboration

  • The proposed data-to-health center would connect clinicians, data scientists, and ethical and quality-control boards to strengthen interpretable processes and trustworthy clinical decision support.The framework aims to provide understandable results for specific end-users and ensure robust health-system performance.Source: §7 · pp. 33–34

Reviewed technologies

  • Figure 15 reports that 80% of reviewed tools were AI tools, compared with 15% wearable medical devices and 5% LLMs such as ChatGPT.The distribution suggests that wearables and LLMs remain less represented in the reviewed studies.Source: §7 · pp. 34–35, Figure 15 · p. 35

8. Open challenges and future insights for XAI in health systems

XAI in healthcare still faces unresolved challenges involving explanation quality, user tailoring, performance–interpretability trade-offs, ethics, evaluation, uncertainty, and continuous improvement. The proposed direction is a structured, user-centered, domain-specific approach that supports trustworthy and safer clinical AI.

8.1.Misuse of post-processing explainers in health application

  • Post-hoc explainers may produce uncertain or manipulable explanations that cannot reliably satisfy patient-level decision-support and legal transparency requirements.The paper calls for open discussion of their limitations, especially where explanations may serve conflicting interests in clinical care.Source: 8.1.Misuse of post-processing… · pp. 35–36

8.2.Designing XAI framework for specific health application

  • The “Data to Health Center” roadmap prioritizes end-users, application-specific requirements, and iterative refinement of performance and interpretability with healthcare professionals.Its goal is to make AI decisions understandable, assessable, and acceptable as clinical and research needs evolve.Source: 8.2.Designing XAI framework for… · p. 36

8.3. Trade-off between interpretability and accuracy

  • Healthcare XAI must balance human-understandable explanations with accurate fitting of complex data, while transparently communicating misclassification risks across the full data-to-decision pipeline.Models are commonly characterized as Black-Box, Grey-Box, or White-Box according to their transparency.Source: §8.3 · pp. 36–37

8.4.Ethical guidelines (Inclusion and diversity)

  • Ethical XAI requires domain-specific safeguards for fairness, bias, privacy, security, patient safety, and gender equity, supported by interdisciplinary and legal frameworks.Guidelines from the IEEE Global Initiative and EU Ethics Guidelines for Trustworthy AI illustrate efforts to make deployment fair, secure, privacy-preserving, and safety-conscious.Source: 8.4.Ethical guidelines (Inclusion and… · p. 37

8.5.User-Centered AI system

  • User-centered XAI requires validating explanations with patients and clinicians, whose preferences differ from technical users, and collaborating with clinicians to improve models.Clinicians and patients may prefer concise, visual, intuitive, socially acceptable, and contrastive explanations rather than mathematical presentations.Source: 8.5.User-Centered AI system · pp. 37–38

8.6.Evaluation metrics (Quality control tools)

  • Evaluation should use goal-specific metrics and feedback on trust, comprehensibility, usefulness, and adequacy, aligning iterative design with healthcare needs [147].Surveys, interviews, and targeted feedback can differentiate trust and explanation-quality dimensions rather than applying one uniform quality-control measure.Source: 8.6.Evaluation metrics (Quality… · pp. 38–39

8.7.Cognitive and usability engineering

  • XAI systems should integrate cognitive and usability engineering to reduce cognitive load, align with clinical workflows, improve accessibility, and support safer decision-making.This includes clearer information presentation, usable health-information systems, and support for complex tasks such as medical-image interpretation.Source: 8.7.Cognitive and usability engineering · p. 39

8.8.Uncertainty quantification; 8.9.Explainable casual inference techniques; 8.10. Kaizen and continuous improvement; 8.11. XAI for wearable health trackers and medical LLMs

  • Trustworthy healthcare AI requires uncertainty quantification, causal reasoning, multidisciplinary collaboration, and continuous Kaizen-style improvement to make systems safer, more reliable, interpretable, and aligned with clinical and ethical standards.Causal models articulate relationships between actions, potential outcomes, and associated risks, while Kaizen improves transparency, robustness, efficiency, and interpretability.Source: 8.8.Uncertainty quantification · pp. 39–40, 8.9.Explainable casual inference… · p. 40, §8.10 · pp. 40–41

9. Conclusion

The study positions trustworthy, interpretable AI as important for healthcare while concluding that explainability alone cannot guarantee correct decisions, trust, or clinical justification. It recommends using explanations primarily for development and auditing, alongside empirically verified performance, cautious adoption, interdisciplinary design, and interactive feedback.

  • Explainability methods do not reliably guarantee individual-decision correctness, user trust, or clinical justification, although they support model diagnosis, auditing, performance improvement, and bias detection.Source: §9 · p. 42
  • For now, developers and auditors should use explanations as instruments, while clinicians remain cautious and regulators deliberate before mandating explanations for clinical AI.Without major advances in explainable AI, systems may need to be trusted partly through empirically verified performance despite their black-box character.Source: §9 · p. 42
  • The proposed trustworthy-AI roadmap combines staged interpretability with fairness, privacy, accountability, sustainability, and robustness, while tailoring explanations to diverse healthcare users.It covers data pre-processing, model selection, and post-processing, and calls for clear design goals, diverse end-users, and interdisciplinary collaboration.Source: §9 · p. 42
  • Real-world deployment, alignment between legal requirements and technological progress, and interactive explanation-feedback systems remain necessary to build trust and enable AI adoption.Current methods may not achieve fully explainable AI, so systems that explain decisions and request feedback can reassure end-users and decision-makers.Source: §9 · p. 42

Appendix 1. Abbreviations · Appendix 2. Criteria for study selection using PRISMA 2020 and PICO.

The appendices define the study-selection framework using PRISMA 2020 and PICO, covering eligible populations, interventions, study designs, comparisons, findings, and publication dates. Appendix 1 is identified as an abbreviations section, but no abbreviation content is provided here.

Appendix 2. Criteria for study selection using PRISMA 2020 and PICO.

Appendix 3. Results of the structured searching.

The structured search combined healthcare technology terms with explainable and interpretable AI concepts across three documented query formulations, yielding 2241 papers in the final combined search.

Search concepts

  • The search covered healthcare technologies including IoMT, telehealth, digital health, smart and wearable devices, digital twins, and medical large language models.These technology terms were combined with terminology for explainable, interpretable, transparent, and reliable AI.Source: Appendix 3. Results of the structured… · p. 44

Search formulations

  • The appendix documented equivalent search concepts in distinct database-query syntaxes while retaining the same technology and interpretable-AI scope.The formulations used field-tagged terms, brace-delimited terms, and quoted Boolean terms.Source: Appendix 3. Results of the structured… · p. 44

Final structured search

Appendix 4. Summary of important surveys on IML and XAI in healthcare and medicine.

This appendix surveys important IML and XAI reviews in healthcare and medicine, spanning general healthcare applications and specialized fields such as precision oncology, ECG-based heart disease diagnosis, drug discovery, cardiology, and electronic health records.

Precision oncology

Cardiology and ECG

Drug discovery and EHRs

General healthcare XAI

  • The listed reviews cover XAI methods and applications across healthcare, including a systematic review of the last decade (2011–2022) [f98952].The appendix also identifies reviews addressing interpretability, evaluation, usability, reliability, and implementation in healthcare.Source: Appendix 4. Summary of important… · p. 45

Appendix 5. Compact summary of existing papers on interpretable intelligent health systems.

This appendix summarizes interpretable intelligent health-system studies across diverse disease, physiological monitoring, medical imaging, wearable, and clinical decision-support applications. It highlights the use of transparent models and XAI methods to support prediction, explanation, clinical decision-making, understanding, trust, usability, and user acceptance.

Prediction and decision support

  • The cited studies applied LIME or SHAP-based explainability to activity recognition, prenatal stress prediction, thyroid dysfunction risk assessment, and post-stroke adverse mental-outcome prediction,.Source: Appendix 5. Compact summary of… · pp. 46–47

Imaging and physiological signals

  • Image-based medical models improved readers’ confidence and accuracy in breast-cancer diagnosis, outperformed EfficientNet-B0 for dental-caries prediction, and achieved high performance across metrics for myocarditis diagnosis [97],, [62].Source: Appendix 5. Compact summary of… · pp. 46–47
  • Medical-imaging and physiological-signal studies used Grad-CAM, attention, learned kernels, and interpretable EEG systems to deliver accurate, reliable, or trusted outputs for asthma, Alzheimer’s disease, brain injury, and signal assessment.Reported outcomes included high accuracy, precise Alzheimer’s detection, increased end-user trust, and reliable low-power signal assessment.Source: Appendix 5. Compact summary of… · p. 46

Monitoring and risk stratification

  • Interpretable systems also supported wearable monitoring, gestational-diabetes prediction, mortality or readmission detection, and genetic-risk stratification through individualized tracking, early detection, and understandable explanations, [119], [173], [175].The summarized studies emphasized faster diagnosis, cost-effective prediction, and decision-relevant rule or Shapley-based explanations.Source: Appendix 5. Compact summary of… · p. 47

Evaluation and usability

  • Evaluation-focused studies examined explanation usability, consistency, fairness, missing-data handling, and subgroup feature interpretation, while tools such as SUS, SCS, and rule extraction supported deliverable explanations and reduced potential harms, [171],,.The appendix also reports model-agnostic interpretation identifying the most predictive findings for metabolic syndrome.Source: Appendix 5. Compact summary of… · p. 47
  • Interpretable systems addressed dementia, ovarian cancer, respiratory abnormalities, breast lesions, and pulmonary function using decision trees, SHAP, random forests, and evolutionary methods.Reported outcomes included correct dementia prediction, greater reliability and transparency for ovarian-cancer models, and enhanced pulmonary-function decision support.Source: Appendix 5. Compact summary of… · p. 47
  • Wearable and sensor-based systems supported COVID-19 detection, human-activity recognition, diabetes detection, and behavioral monitoring, with explainable models applicable across settings.The studies used wearable sensors, smartwatch data, LIME, SHAP, discriminant analysis, and long short-term memory models.Source: Appendix 5. Compact summary of… · p. 48
  • Interpretable deep-learning and feature-selection pipelines improved explanation of imaging and clinical prediction models, including mammography, small image datasets, and structured clinical records.Reported approaches included case-based deep learning, automated feature selection, ensemble trees, and structured explainability pipelines.Source: Appendix 5. Compact summary of… · p. 48
  • Clinical decision-support applications included septic-shock diagnosis, ischemic-stroke outcome prediction, hepatitis deterioration, and acute-coronary-syndrome risk prediction.These systems used real-time clinical and administrative data, global and local interpretability, SHAP, LIME, partial-dependence plots, and multiple predictive models.Source: Appendix 5. Compact summary of… · p. 48
  • Later studies linked explanation techniques with clearer decisions, higher interpretability, better AI acceptance, and stronger explanations in survival prediction, glioma classification, and computer-aided diagnosis.Methods included recurrent neural networks with visualization, SHAP, LIME, transparent feature extraction, and Ada-WHIPS.Source: Appendix 5. Compact summary of… · p. 48

Related papers

Loading 2311.11055v2…