PaperlayerView exact revision on arXiv

Source-linked AI summary

Why we do need Explainable AI for Healthcare

Giovanni Cinà, Tabea Röber, Rob Goedhart, Ilker Birbil

arXiv:2206.15363v1cs.HCcs.AIcs.LG

TL;DR

Healthcare XAI faces doubts about reliability and its place in guidelines and standards, while evaluating explanations is difficult without objective ground truth. This Viewpoint reviews those criticisms and concludes they do not warrant abandoning XAI research; it argues that XAI remains important for oversight and human interaction with medical AI.

  • Problem

    There is no consensus on XAI reliability, and researchers question whether XAI should be included in guidelines and standards or deployed at all.

    Source: §1 · p. 2

  • Method

    The Viewpoint collects, reviews, and appraises criticisms of XAI, then discusses their counterpoints.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    The authors conclude that the criticisms offer no decisive reason to abandon XAI research, particularly in medicine, where greater oversight is preferable.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors argue that XAI research should be pursued because clinicians should remain able to mediate between algorithms, scientific protocols, and patients’ human experience.

    Source: §4 · p. 9

    XAI evaluation is difficult because objective ground truth is lacking, and evidence-based assessment can involve trials that poorly reflect practice or use non-representative populations.

    Source: §1 · p. 2, Evidence-based assessment is the… · p. 6

Abstract

from arXiv · show

The recent spike in certified Artificial Intelligence (AI) tools for healthcare has renewed the debate around adoption of this technology. One thread of such debate concerns Explainable AI and its promise to render AI devices more transparent and trustworthy. A few voices active in the medical AI space have expressed concerns on the reliability of Explainable AI techniques, questioning their use and inclusion in guidelines and standards. Revisiting such criticisms, this article offers a balanced and comprehensive perspective on the utility of Explainable AI, focusing on the specificity of clinical applications of AI and placing them in the context of healthcare interventions. Against its detractors and despite valid concerns, we argue that the Explainable AI research program is still central to human-machine interaction and ultimately our main tool against loss of control, a danger that cannot be prevented by rigorous clinical validation alone.

1 Introduction

Healthcare XAI is debated because its reliability is contested and its explanations can be difficult to evaluate. This Viewpoint reviews the criticisms and concludes they do not decisively justify abandoning XAI research, particularly where greater oversight is preferable.

1 Introduction

  • There is no consensus on XAI reliability, and researchers question its inclusion in guidelines and standards or deployment at all.8,9Source: §1 · p. 2
  • Machine-learning opacity reflects both models’ capacity to handle many factors and the sub-symbolic character of statistical learning.Source: §1 · p. 2
  • XAI includes local or global, model-specific or model-agnostic techniques, as well as post-hoc explanations and inherently explainable models.10,11,12Local methods explain individual outputs; global methods explain models at an aggregate level.Source: §1 · p. 2
  • To address black-box opacity, XAI techniques can add layers of complexity.Source: §1 · p. 2
  • XAI techniques can produce inaccurate or deceiving explanations, and explainability can trade off against other goals such as high performance.Source: §1 · p. 2
  • XAI evaluation is difficult because there is no objective ground truth.Source: §1 · p. 2
  • Healthcare’s heterogeneous XAI methods and the literature’s doubts and limitations make the technology’s role difficult to settle.Source: §1 · p. 2
  • The Viewpoint reviews criticisms of XAI and concludes they provide no decisive argument to abandon research for better techniques, especially in medicine, where more oversight is preferable.Source: §1 · p. 2

2 Criticisms to Explainable AI

The section introduces the article’s review of criticisms and counterpoints about XAI.

2 Criticisms to Explainable AI

  • The article gathers and discusses criticisms of XAI in no particular order, with each criticism expanded alongside its counterpoint.Source: §2 · p. 2
  • Figure 1 visually represents the criticisms and their main counterpoints.Source: §2 · p. 2
  • The review is organized around doubts raised about XAI’s usefulness and efficacy.Source: §2 · p. 2

4. XAI may work on an aggregate level, but not on an individual level.

The section heading frames evidence-based assessment, rather than XAI, as the proper way to evaluate medical interventions.

4. XAI may work on an aggregate level, but not on an individual level.

  • The section frames evidence-based assessment as the proper evaluation approach for medical interventions, including AI interventions, rather than XAI.Source: §4 · p. 3

7. XAI is not suitable for all users.

The article reviews concerns about XAI’s reliability and use, but argues that these concerns do not justify abandoning it in healthcare. Its role depends on the data, users, and clinical application, and it should complement rather than replace evidence-based evaluation.

XAI techniques may be incorrect or unreliable.

  • XAI methods can be fooled by adversarial attacks or noise, and approximations of black-box models may not reflect the models’ actual behavior.The authors argue that awareness and countermeasures can preserve usefulness; attacks on SHAP suggest excluding out-of-distribution samples from approximate calculations.Source: XAI techniques may be incorrect or… · pp. 3–4

Human users may fall prey to biases when interacting with XAI tools.

  • Users may misjudge explanations through confirmation bias or misuse, so the authors favor proper training rather than abandoning XAI.They note that medical-device user training is regulated in Europe through the Medical Device Regulation and ISO standards.Source: Human users may fall prey to biases… · p. 4

There are no objective measures of explainability.

  • Because explainability has no obvious ground truth, assessing explanation quality is difficult and requires further work and experimentation.The authors point to user studies and interdisciplinary research as existing approaches to understanding what explanations are useful.Source: There are no objective measures of… · p. 4

XAI may work on an aggregate level, but not on an individual level.

  • Individual-level explanations may be hard to judge, but semantically clear EHR features such as BMI and glucose can make them more clinically assessable than image pixels.The authors suggest users can check prediction reasons and compare them with other clinical information to monitor possible problems.Source: XAI may work on an aggregate level,… · pp. 4–5

Evidence-based assessment is the proper way to evaluate medical interventions, not XAI.

  • XAI should not replace evidence-based validation; the authors argue that both can coexist and that explainability’s priority depends on the application.They also describe explainability as a developer tool that can identify clinically implausible models before testing and may improve or expedite RCTs and evidence-based medicine.Source: Evidence-based assessment is the… · pp. 5–6

In the medical domain, we cannot trade performance for explainability.

  • Some XAI techniques or simpler interpretable models can reduce performance, so the authors say the trade-off should be decided case by case.They argue that simpler algorithms with feature engineering or reduction can perform well, and cite work aiming to match complex models’ performance.Source: In the medical domain, we cannot trade… · pp. 6–7

7. XAI is not suitable for all users.

  • Different professionals need different forms of XAI; the authors argue that robust methods, training, and expectation management can help address misuse by non-expert users.They point to research on interactive machine learning for user-targeted explanations.Source: XAI is not suitable for all users · p. 7

3 Should we downsize our efforts in XAI for healthcare?

XAI is not a substitute for rigorous evaluation, but it can provide additional oversight for healthcare AI. Whether and how to use it depends on the case, given AI’s potential risks and rapid adoption.

3 Should we downsize our efforts in XAI for healthcare?

  • XAI is not a silver bullet or a substitute for rigorous evaluation of model performance.The authors argue that human oversight remains important as AI may be deployed rapidly at scale.Source: §3 · p. 8
  • Guidelines could allow XAI without requiring a specific technique, while requiring practitioners to explain their choice and understand potential failure modes.The authors note that XAI may not yet be mature enough for hard requirements in guidelines.Source: §3 · p. 8
  • Explainability and performance are competing objectives, so the appropriate trade-off depends on the particular case.The authors do not argue that explainability must be required for every healthcare intervention.Source: §3 · p. 8
  • AI may be biased against subgroups, lose reliability when input data changes, or rely on spurious correlations, making higher oversight preferable.The authors connect these risks with AI’s potential for rapid and impactful adoption.Source: §3 · p. 8

4 Conclusion

The authors conclude that XAI research for healthcare should be stepped up, as the field is growing and techniques can be honed more quickly. They argue that clinicians should remain able to engage with AI rather than become mere executors of proven interventions.

4 Conclusion

  • Rapid growth in XAI research means new techniques are tested extensively, shortening the time to a time-tested approach compared with the past.Source: §4 · pp. 8–9
  • The authors conclude that efforts in healthcare XAI research should be stepped up because the research program is promising and addresses a core issue.They argue that growing research may shorten the time needed for techniques to become time-tested.Source: §4 · pp. 8–9
  • The authors argue that clinicians should remain mediators between algorithms, scientific protocols, and patients’ human experience.If clinical users cannot interact with machines and become mere executors, care could become de-humanizing.Source: §4 · p. 9

Related papers

Loading 2206.15363v1…