PaperlayerView exact revision on arXiv

Source-linked AI summary

Designing Interpretable ML System to Enhance Trust in Healthcare: A Systematic Review to Proposed Responsible Clinician-AI-Collaboration Framework

Elham Nasarian, Roohallah Alizadehsani, U. Rajendra Acharya, Kwok-Leung Tsui

arXiv:2311.11055v2cs.AIcs.HCcs.LG

TL;DR

Healthcare AI needs outputs that clinicians and other stakeholders can understand, yet complex models and explanation methods may not reliably reveal decision processes. This paper systematically reviews interpretable ML and XAI methods, applications, and implementation challenges across healthcare workflows. It finds that post-processing methods such as SHAP and LIME are widely used, while XAI remains limited for wearables and LLMs, motivating a responsible implementation roadmap.

  • Problem

    Complex healthcare models can be difficult to interpret, while explanation methods may rely on human judgment and introduce an interpretability gap.

  • Method

    The paper reviews healthcare interpretability across data pre-processing, model selection, and post-processing, including methods, applications, quality assessment, and implementation practices.

  • Results

    Post-processing methods predominate, with SHAP and LIME among the most frequently used XAI methods; XAI applications remain limited for wearable devices and healthcare LLMs.

  • Takeaways & Limitations

    Responsible clinical AI implementation should combine interpretability with clear design goals, diverse end-user engagement, interdisciplinary collaboration, and quality-control considerations.

  • Takeaways & Limitations

    Explainability alone does not guarantee correctness, trust, or justification of individual clinical recommendations and can still support auditing and bias detection.

Abstract

from arXiv · show

This paper explores the significant impact of AI-based medical devices, including wearables, telemedicine, large language models, and digital twins, on clinical decision support systems. It emphasizes the importance of producing outcomes that are not only accurate but also interpretable and understandable to clinicians, addressing the risk that lack of interpretability poses in terms of mistrust and reluctance to adopt these technologies in healthcare. The paper reviews interpretable AI processes, methods, applications, and the challenges of implementation in healthcare, focusing on quality control to facilitate responsible communication between AI systems and clinicians. It breaks down the interpretability process into data pre-processing, model selection, and post-processing, aiming to foster a comprehensive understanding of the crucial role of a robust interpretability approach in healthcare and to guide future research in this area. with insights for creating responsible clinician-AI tools for healthcare, as well as to offer a deeper understanding of the challenges they might face. Our research questions, eligibility criteria and primary goals were identified using Preferred Reporting Items for Systematic reviews and Meta-Analyses guideline and PICO method; PubMed, Scopus and Web of Science databases were systematically searched using sensitive and specific search strings. In the end, 52 publications were selected for data extraction which included 8 existing reviews and 44 related experimental studies. The paper offers general concepts of interpretable AI in healthcare and discuss three-levels interpretability process. Additionally, it provides a comprehensive discussion of evaluating robust interpretability AI in healthcare. Moreover, this survey introduces a step-by-step roadmap for implementing responsible AI in healthcare.

1. Introduction

Healthcare AI can support complex clinical decisions, but increasing model complexity makes outputs harder to understand and trust. Interpretability methods therefore address transparency, clinician communication, and the limitations of explanations in high-stakes care.

  • AI is increasingly used across healthcare, including disease prediction, surgery, drug discovery, wearables, and clinical decision support.
  • Deep neural networks are harder to understand and trust than classical models because sophisticated training creates complex decision processes.
  • Interpretability helps users understand predictions, investigate errors, build trust, and identify bias or other model problems.
  • White-box, gray-box, and black-box models represent differing trade-offs among transparency, explainability, complexity, and predictive accuracy.White-box models are more transparent, whereas black-box models may offer higher accuracy but less understandable decisions.
  • Current explanation methods can leave an interpretability gap because humans must infer whether an explanation reflects the model’s actual reasoning.This concern applies to methods such as feature visualization, LIME, and SHAP, whose outputs may require subjective interpretation.
  • This review examines interpretability across clinical decision-support stages and addresses the need to evaluate explanations rather than assume transparency guarantees validity.It organizes interpretability across pre-processing, modeling, and post-processing while reviewing related methods and applications.

2. Systematic review method

The review followed PRISMA-informed procedures, structured searches across three databases, independent screening, quality appraisal, and standardized data extraction. It synthesized the included evidence to examine interpretability across healthcare applications and identify gaps in end-to-end implementation frameworks.

  • Protocol and search strategy: The review protocol incorporated PRISMA criteria, search techniques, selection standards, and data-extraction procedures.
  • Protocol and search strategy: Searches across PubMed, Scopus, and Web of Science used conceptual frameworks and targeted terms related to explainable AI, interpretable ML, wearables, and telehealth.The structured search yielded 2,241 imported studies before duplicate removal and screening.
  • Study selection: Independent reviewers screened titles and abstracts, advanced uncertain records to full-text review, and resolved discrepancies through consensus.EndNote21 was used for duplicate removal before records were imported into Covidence.
  • Quality appraisal: MetaQAT assessed study quality and risk of bias after full-text review, excluding studies with substantial risk of bias.The appraisal was intended to retain studies with robust methodological quality and reduce potentially misleading evidence.
  • Data extraction and synthesis: 74 papers were selected for data extraction after duplicate removal, title-and-abstract screening, full-text review, and critical appraisal.The final set comprised 10 review papers and 64 experimental publications, and data were mapped using standardized extraction tools.
  • Review scope and gap: The review addresses a reported gap in examining the full interpretability process from preprocessing through post-modeling explainability alongside evaluation and implementation frameworks.Existing surveys were described as frequently focusing on specific applications rather than the complete process across healthcare systems.

3. Proposed interpretability process for intelligent health system

The proposed interpretability process spans data pre-processing, model selection, and post-processing to make clinical AI explanations clearer and more useful for different end-users. It emphasizes data-type-specific analysis, dataset understanding, summarization, and augmentation before modeling.

  • Framework: The framework organizes healthcare interpretability across pre-processing, model selection, and post-processing.This three-level structure is presented as a comprehensive process for understanding AI decisions throughout clinical system development.
  • Pre-processing interpretability: Pre-processing interpretability examines dataset characteristics and features because training data substantially influences AI decision-making.The process includes exploratory analysis, data-type-specific methods, and techniques for understanding structured, unstructured, and emerging data.
  • Data-specific methods: Medical data types require tailored interpretability methods, including NLP for notes, feature analysis for tabular data, heatmaps for images, and waveform analysis for signals.The paper links these methods to distinct challenges such as ambiguity, high dimensionality, noise, and artifacts.
  • Exploratory data analysis: Exploratory data analysis identifies dimensionality, summary statistics, missing samples, and potential dataset bias such as class imbalance.Visualization choices should reflect the dataset, application, and statistical characteristics being communicated.
  • Dataset summarization and augmentation: Dataset summarization retains representative samples to reduce storage demands, while data augmentation generates synthetic samples when limited data increases overfitting risk.Examples include prototype selection, data squashing, normalization, and SMOTE-based minority-class oversampling.

3.2.Interpretable process of model selection and its health application

The model-selection level addresses interpretability during model development, combining inherently interpretable, hybrid, joint, architectural, attribution, visualization, and example-based approaches. It also recognizes that complex models require additional techniques to expose their decision processes.

  • Interpretable modeling: Interpretable model selection is necessary because accurate models may remain difficult to understand after data preparation.The paper contrasts white-box, hybrid, joint, and modified-architecture approaches for improving model transparency.
  • White-box methods: White-box methods such as decision trees and logistic regression provide inherently interpretable alternatives for healthcare prediction tasks.The reviewed applications include dementia prediction and COVID-19 detection using tree-based approaches.
  • Hybrid methods: Hybrid methods combine inherently interpretable techniques with sophisticated black-box models to seek both performance and explainability.Examples include deep k-nearest neighbors, self-explaining neural networks, and related combined designs.
  • Joint methods: Joint methods generate predictions and explanations together by incorporating explanatory information into model training.The TED framework combines outputs and explanations during training and produces user-oriented outcomes.
  • Post-modeling explanations: Post-modeling techniques use attribution, visualization, and examples to explain complex model decisions through feature relevance, representations, or similar cases.The review covers DTD, perturbation, backpropagation, Deep Lift, PDP, ALE, ICE, prototypes, criticisms, and counterfactuals.

4. Application of interpretability in AI-based medical devices

Interpretability is applied across AI-based medical devices, wearable sensors, and medical language models to address transparency and trust concerns in healthcare use. The applications span diverse medical conditions and require safeguards for high-stakes generative systems.

  • Medical devices: AI-based medical devices include technologies used for medical purposes that may complement existing devices or function independently.The paper discusses applications involving sensors, wearables, telemedicine, medical LLMs, and digital care technologies.
  • Diagnostic applications: XAI applications in medical diagnostics cover conditions including COVID-19, Alzheimer’s disease, cancer, and other diseases.Figure 12 presents the distribution of current applications across these diagnostic areas.
  • Wearables and sensors: Integrating XAI with IoT medical sensors and wearable trackers can improve transparency and help users understand device-generated insights.The review summarizes existing wearable medical-device applications and connects interpretability with more informed healthcare decision-making.
  • Medical LLMs: Medical LLM deployment requires attention to safety, ethics, privacy, regulation, and human oversight because generated content may be inaccurate or misleading.The paper identifies these requirements as especially important in patient-care settings.
  • LLM limitations: ChatGPT-like systems may lack transparency and contain errors, misinformation, or training-data biases that can affect diagnostic or treatment recommendations.The paper states that medical professionals should review and validate such suggestions before use.

5. Quality evaluation and improvement of the interpretability process

The proposed quality evaluation process focuses on whether explanations are comprehensible and useful to healthcare stakeholders in real-world settings. It assesses interpretability alongside technical, human, ethical, regulatory, and clinical dimensions.

  • Evaluation focus: Evaluation prioritizes explanation comprehensibility for patients, doctors, nurses, caregivers, managers, and other healthcare experts.These stakeholders are treated as important end-users and decision-makers for assessing practical utility.
  • Quality dimensions: The quality-control framework covers accuracy, reliability, robustness, interpretability, usability, human-AI interaction, ethics, feedback responsiveness, compliance, clinical evidence, transparency, and scalability.These dimensions are summarized in Table 5 as criteria for evaluating and improving XAI systems.
  • Human-AI interaction: Quality assessment must consider both system properties and the human-AI interaction in which explanations are interpreted and used.Figure 13 maps scalability, reliability, robustness, usability, and transparency in relation to human-AI interaction.

6. Step-by-step roadmap to implement responsible clinician-AI-collaboration framework

The proposed responsible clinician-AI-collaboration framework connects data, explainable processing, quality assessment, and end-user interaction through a staged implementation roadmap.

  • Implementation involves a Health-to-Data Center with clinician-AI communication, data-science support, quality assessment, and ethical components.
  • The framework flows from a Data to Health Center through explainable data pre-processing, model processing, and post-processing before quality assessment and end-user interaction.
  • Interpretability questions should be considered across pre-processing, model selection, and post-processing for patients, clinicians, health managers, and developers.
  • Quality assessment tools should be developed after the explainability process to evaluate AI systems in healthcare.

7. Discussion

The discussion identifies uneven adoption of interpretable AI across healthcare applications, with post-hoc methods predominating and wearable, biosensor, LLM, and generative-AI applications remaining comparatively underexplored.

  • SHAP and LIME are among the most frequently used XAI methods, while black-box models are more common than white-box methods in clinical decision support.
  • Only a small fraction of wearable-health-tracker studies use XAI, and research on XAI for LLMs and generative AI in healthcare remains limited.
  • Integrating XAI into IoT medical sensors and wearable trackers is presented as a way to improve transparency and interpretability for end-users managing healthcare data.
  • LLM deployment in healthcare requires attention to safety, ethics, privacy, human oversight, and responsible implementation because inaccurate or misleading content could compromise care and trust.
  • The review emphasizes interpretable processes, collaboration among clinicians, data scientists, and oversight boards, and quality control for trustworthy clinical decision support.
  • AI tools comprise 80% of reviewed medical-device studies, compared with 15% for wearable medical devices and 5% for LLMs such as ChatGPT.
  • Post-hoc explainability is used in over 100% of cases, while processing is less prevalent and pre-processing is least used.The reported value above 100% indicates that some studies applied more than one post-hoc method.

8. Open challenges and future insights for XAI in health systems

Open challenges in healthcare XAI include selecting and evaluating explanations, tailoring them to users and applications, balancing performance with interpretability, and ensuring safe, inclusive implementation. The paper therefore emphasizes user-centered collaboration, cognitive and usability engineering, uncertainty communication, and continuous improvement.

  • Open challenges: Healthcare XAI still lacks settled answers about effective explanation methods, evaluation quality, user tailoring, performance–interpretability balance, and continuous improvement.These questions frame the section’s open research and implementation challenges.
  • Open challenges: Post-hoc explainers may produce untrustworthy results, so relying on them alone may not satisfy healthcare’s legal and patient-level decision-support requirements.The paper questions whether current post-processing methods can provide sufficiently reliable explanations.
  • Implementation roadmap: The proposed Data to Health Center roadmap prioritizes end-users, application-specific requirements, and an ongoing balance between model performance and understandable transparency.The roadmap links tailoring and balancing to trust and assessment of AI decisions.
  • User-centered design: Responsible implementation requires collaboration between ML experts and clinicians, because clinicians interpret outputs and identify opportunities for model refinement.The paper also notes that explanation preferences differ between technical specialists, clinicians, and patients.
  • Cognitive and usability engineering: Cognitive and usability engineering should align XAI systems with users’ cognitive workflows, making them more accessible, intuitive, safe, and effective.The approach addresses cognitive demands on healthcare professionals and patients during clinical use.
  • Continuous improvement: Continuous improvement, uncertainty quantification, and attention to data noise are presented as mechanisms for making healthcare AI more reliable, interpretable, adaptable, and safer.The discussion connects iterative refinement with changing healthcare needs while stressing that data noise can compromise automated diagnosis.

9. Conclusion

The conclusion presents explainability as important but insufficient for guaranteeing correct patient-level decisions, trust, or clinical justification. It advocates a broader trustworthy-AI roadmap involving structured interpretability, diverse end-users, interdisciplinary collaboration, and interactive feedback while acknowledging that current methods may remain incomplete.

  • Conclusion: Explainability alone may not guarantee correct individual patient decisions, foster trust, or justify clinical use of AI recommendations.The conclusion cautions against relying exclusively on explainability for patient-level decisions.
  • Conclusion: Current explainability methods remain an ongoing challenge, and healthcare professionals should exercise caution when interpreting their outputs.The paper suggests that, absent major advances, empirically verified performance may remain important for trust.
  • Contribution: The study organizes trustworthy XAI across data pre-processing, model selection, and post-processing while proposing a roadmap based on clear goals, diverse end-users, and interdisciplinary collaboration.The roadmap is presented as part of the paper’s contribution to responsible clinical AI implementation.
  • Future direction: Interactive systems that provide explanations and solicit feedback are proposed as a way to address the gap between technical progress, legal requirements, and practical trustworthiness.The conclusion states that current methodologies may not suffice for fully explainable AI systems.

Appendix 2. Criteria for study selection using PRISMA 2020 and PICO.

The study-selection criteria targeted peer-reviewed, open-access research on XAI and interpretable machine learning in AI-based medical devices and clinical decision support systems. Eligible studies reported interpretability-related findings, while duplicates, reviews, and studies focused on classical ML tools were excluded.

  • Population: The population comprised AI-based medical devices and clinical decision support systems.This defines the healthcare technology scope of the review.
  • Study design: Eligible designs were peer-reviewed, open-access journal and conference articles, while review protocols and review-of-reviews studies were excluded.The criteria distinguish primary eligible publications from excluded review formats.
  • Intervention: The intervention criterion required studies examining XAI or interpretable ML in medical devices or clinical decision support systems to improve quality and interpretability.The intervention was defined around interpretability in intelligent health systems.
  • Outcomes: Studies had to report results involving an IML or XAI domain as a main finding or one of the main findings.This criterion specified the required outcome focus.
  • Time limit: All publication dates were accepted within the stated 2012–2023 range, with duplicate studies excluded.The criteria specify both the date range and duplicate-handling rule.

Appendix 3. Results of the structured searching.

The structured search combined terms for connected and digital health technologies with terms for explainable, interpretable, transparent, and reliable AI. The final combined search returned 2,241 papers.

  • Search scope: The search covered Internet of Medical Things, telehealth, remote and digital health, smart and wearable devices, digital twins, and medical large language models.These technology terms were combined in the database-specific search strings.
  • Search scope: The search strings paired healthcare-technology terms with explainable artificial intelligence, interpretable machine learning, transparent machine learning, or reliable AI.The strings operationalized the review’s interpretability focus across databases.
  • Search results: 2,241 papers were returned by the final search combining the three component searches.The appendix reports this count for Final (#1) AND (#2) AND (#3).

Appendix 4. Summary of important surveys on IML and XAI in healthcare and medicine.

The surveyed literature covers interpretable and explainable AI in healthcare across clinical domains, methods, and review designs. This review positions its contribution around a robust interpretability approach addressing processes, evaluation, and implementation.

  • Contribution of the present review: The paper’s review focuses on IML and XAI processes in healthcare, evaluation, implementation, and a proposed framework.Its stated contribution is a robust interpretability approach intended to enhance trustworthy AI in healthcare.
  • Healthcare implementation focus: Prior reviews address interpretability and explainability in healthcare applications, methods, and stakeholder-oriented guidance.One cited review specifically covers performance, interpretability, explainability, guidelines, and healthcare stakeholders for AI-based medical devices.
  • Review designs: The reviewed literature includes systematic, narrative, scoping, and brief review types.The survey table explicitly distinguishes these review methodologies across the cited studies.
  • Scope of prior surveys: Existing reviews examine explainable AI methods and applications across healthcare and medicine.Topics include healthcare applications, precision oncology, ECG-based heart disease diagnosis, drug discovery, cardiology, and biomedical or medical data.

Appendix 5. Compact summary of existing papers on interpretable intelligent health systems.

The compiled studies apply interpretable machine learning and explainable AI to diverse health problems, using model-based, post-hoc, visualization, and rule-based approaches. Reported outcomes include clearer explanations, improved prediction or recognition, personalized risk assessment, and increased trust in selected settings.

  • Risk prediction and decision support: Interpretable systems were developed for stress, heart disease, diabetes, hypertension, mental health, and other clinical decision-support tasks.Methods included post-hoc interpretation, SHAP, Markov decision processes, contextual prompting, and logic learning machines.
  • Communication and usability: Several studies used visual or interactive explanations to communicate model outputs to clinicians and users.Examples include SHAP visualizations for physicians, interfaces for interpretable decisions, and tools designed to provide understandable or usable outcomes.
  • Imaging and signal analysis: Interpretable models were applied to medical imaging and physiological signals, including brain tumors, breast cancer, asthma, Alzheimer’s disease, dental caries, myocarditis, and EEG abnormalities.Reported approaches included vision-language models, Grad-CAM, LIME, SHAP, attention-based networks, and multiple feature-importance methods.
  • Reported outcomes: Reported benefits included increased end-user trust, higher confidence and accuracy, precise or explainable detection, and reliable signal assessment.These outcomes were reported in studies of ICU EEG prediction, mammography, Alzheimer’s detection, asthma detection, and low-power PPG devices.
  • Personalized and early-risk applications: Other studies used interpretable methods for personalized prediction, early intervention, mortality or readmission detection, genetic-risk stratification, and prevention of unnecessary harms.Examples include post-stroke mental outcomes, gestational diabetes, parental stress, unplanned mortality or readmission, primary biliary cholangitis, and real-time prediction.
Loading 2311.11055v2…