PaperlayerView exact revision on arXiv

Source-linked AI summary

Assessing the communication gap between AI models and healthcare professionals: explainability, utility and trust in AI-driven clinical decision-making

Oskar Wysocki, Jessica Katharine Davies, Markel Vigo, Anne Caroline Armstrong, Dónal Landers, Rebecca Lee, André Freitas

arXiv:2204.05030v3cs.AIcs.HCcs.LG

TL;DR

This paper addresses the limited clinical investigation of explanation utility and suitability for ML decision support. It evaluates explanations with healthcare professionals using a pragmatic framework and a within-subject CORONET study. Explanations showed mixed effects, including possible benefits for ambiguous decisions and reduced automation bias, alongside adverse effects and limited improvement in critical model understanding.

  • Problem

    The perceived utility and suitability of ML explanations had not been systematically investigated in clinical contexts, although clinicians need to understand model strengths and limitations.

    Source: §1 · pp. 1–2

  • Method

    The study evaluated explanation utility through a pragmatic framework and a within-subject experiment with 23 healthcare professionals making 10 admission decisions with CORONET, first without and then with explanations.

    Source: §1 · p. 2

  • Results

    Explanations had mixed effects: 17–35% of participants showed an undesirable effect, 57% did not improve critical understanding, and explanations were associated with reflection on disagreement and better handling of ambiguous cases.

    Source: §5 · p. 16

  • Takeaways & Limitations

    Pragmatic evaluation of user-model interaction is presented as important to deployment of healthcare recommendation tools, alongside clinical performance, clinicians’ initial attitudes, and biases.

    Source: §5 · p. 16

    The authors note that understanding the model and building trust may require substantial time, which the study design did not sufficiently incentivize.

    Source: §4.1 · p. 16

Abstract

from arXiv · show

This paper contributes with a pragmatic evaluation framework for explainable Machine Learning (ML) models for clinical decision support. The study revealed a more nuanced role for ML explanation models, when these are pragmatically embedded in the clinical context. Despite the general positive attitude of healthcare professionals (HCPs) towards explanations as a safety and trust mechanism, for a significant set of participants there were negative effects associated with confirmation bias, accentuating model over-reliance and increased effort to interact with the model. Also, contradicting one of its main intended functions, standard explanatory models showed limited ability to support a critical understanding of the limitations of the model. However, we found new significant positive effects which repositions the role of explanations within a clinical context: these include reduction of automation bias, addressing ambiguous clinical cases (cases where HCPs were not certain about their decision) and support of less experienced HCPs in the acquisition of new domain knowledge.

1 Introduction

Clinical ML may bring real-world evidence into decisions, but effective use depends on clinicians understanding a model’s strengths and limitations. This study addresses the limited investigation of explanation utility in clinical settings by evaluating explainability with healthcare professionals in a realistic decision-support scenario.

1 Introduction

  • Effective communication requires healthcare professionals to represent the model faithfully and critically assess its strengths and limitations.Source: §1 · p. 1
  • The clinical utility and suitability of ML explanations had not been systematically investigated in clinical contexts.Source: §1 · p. 2
  • The experiment aimed to approximate real-world clinical decisions involving multiple factors and model recommendations, unlike studies using simplified tasks.Source: §1 · p. 2
  • The study evaluated current ML explanation modalities with 23 healthcare professionals across 10 cancer-and-COVID-19 admission scenarios, comparing score-only with score-plus-explanation support.Source: §1 · p. 2

2 Methods

The methods frame explanation evaluation around clinicians’ mental models and perceived clinical utility, then apply that framework to CORONET using a within-subject study with healthcare professionals.

2.1 Framework for pragmatic evaluation of the model’s explanation

  • The framework evaluates interpretation, trust, satisfaction, model understanding, bias, usability, and clinical performance as contributors to perceived pragmatic utility.Utility is assessed by whether healthcare professionals would use the model in clinical practice.Source: §2.1 · pp. 2–3
  • The study used a recommendation-then-explanation comparison, but did not directly measure model performance’s contribution to clinicians’ attitudes or provide an explicit performance metric.Source: §2.1 · p. 3

2.2 Clinical setting & supporting model

  • CORONET predicts a 0–3 severity score and maps thresholds to admission recommendations, but its scope is COVID-19 severity rather than other clinical needs.The model’s four severity outcomes range from discharge to admission with oxygen requirement and death.Source: §2.2 · pp. 3–4
  • The explanation combines a cohort scatterplot that reveals recommendation errors with patient-specific SHAP bars showing feature contributions and their direction.Source: §2.2 · p. 4

2.3 Study design

  • Each participant first assessed CORONET’s recommendation and score, then received the same type of output with an explanation in a within-subject design.Source: §2.3 · p. 4, Figure 4 · p. 7
  • The questionnaire presented five score-only cases followed by five explained cases, assessed usefulness after each set, and tracked time spent on decisions.Source: §2.3 · pp. 4, 7
  • Ten clinician-reviewed cases included two intentional wrong-discharge scenarios involving oncological emergencies outside CORONET’s COVID-19 scope.These cases tested whether clinicians would override the model when other admission reasons were present.Source: §2.3 · pp. 4–5

2.4 Selection of participants

  • Participants were clinically active professionals working with cancer patients presenting with COVID-19, recruited through NHS trusts, clinician networks, and social media.Source: §2.4 · p. 7

3 Results

Among 23 participating HCPs, most wanted explanations of model recommendations, and uncertainty information was considered even more important. Participants considered knowing the model’s mathematics less important than explanations and uncertainty.

3.1 Participants

  • Twenty-three healthcare professionals with varied experience and expertise participated; median knowledge of managing patients with COVID-19 was five on a one-to-seven scale, evaluated via direct question.Source: §3.1 · p. 7
  • 91% (21/23) considered an explanation of an individual recommendation important, and 87% (20/23) wanted to know which features contributed to it.Source: §3.2 · p. 7
  • 96% (22/23) of HCPs at least slightly agreed that knowing recommendation uncertainty was important, the study’s highest ‘strongly agree’ proportion.43% (10/23) strongly agreed.Source: §3.2 · p. 8
  • HCPs rated knowing the model’s mathematical framework significantly less important than local or global explanations and model uncertainty (p < 0.001).Among HCPs uninterested in the mathematics, 73% wanted an explanation and uncertainty information.Source: §3.2 · p. 8

3.3 Visual explanations were easy to interpret

HCPs generally found CORONET’s visual outputs interpretable, but adding explanations did not significantly change overall attitudes toward the model. Responses also varied with participants’ expectations and perceptions of the score-only output.

3.3 Visual explanations were easy to interpret

  • 83% (19/23) found the colour bar easy to interpret, while the score-only colour scale convinced 48% (11/23) to accept or reject a recommendation.Source: §3.3 · pp. 8–9
  • HCPs showed no significant differences in ease of interpretation or persuasive power across the colour bar, scatterplot, and contribution plot (p > 0.05).Perceptions of the visual outputs were also not significantly correlated with task knowledge or expectations for ML-based decision support.Source: §3.3 · p. 9

3.4 Explanations did not significantly impact on the decision-making

  • Adding explanations did not significantly change HCPs’ satisfaction, trust, or understanding of how the model produced recommendations.A slight positive change in perceived help when less confident was not significant (p = 0.056).Source: §3.4 · p. 9
  • In the score-only condition, the lower the HCPs’ expertise, the more helpful the tool appeared; stronger desire to know contributing features was associated with lower perceived helpfulness.The associations covered making safe decisions (r = −0.653; p = 0.001) and less-confident cases (r = −0.553, p = 0.006).Source: §3.4 · p. 9
  • In the score-only condition, being convinced by the colour bar was correlated with satisfaction (r = 0.484, p = 0.019) and reassurance (r = 0.527, p = 0.01).Source: §3.4 · p. 9

3.5 Model explanations may have an adverse effect on HCPs

Responses to explanations were mixed: some HCPs reported greater satisfaction, helpfulness, or reassurance, while others reported declines. Explanations often did not improve understanding of when the model was wrong, and shifts in that understanding tracked shifts in reassurance.

3.5 Model explanations may have an adverse effect on HCPs

  • 34% of HCPs were more satisfied with explanations, while 23% reported decreased satisfaction.Higher satisfaction was weakly correlated with ease of interpreting the contribution plot (r = 0.436, p = 0.038), but not with being convinced by it (p = 0.24).Source: §3.5 · p. 9
  • Explanations did not change understanding of when and why the model makes wrong recommendations for 57% (13/23) of HCPs, and understanding decreased for 17%.These outcomes were not attributed to time spent or diagram interpretability (p > 0.05).Source: §3.5 · p. 10
  • Changes in understanding of wrong recommendations were associated with changes in the tool’s reassuring utility (r = 0.762, p < 0.001, padj = 0.003), the strongest correlation between changes.Source: §3.5 · p. 10
  • 52% of positive changes were reported for help in less-confident cases, although 22% found explanations less helpful.For making safe clinical decisions, 26% found explanations more helpful and 26% found them less helpful; 22% felt more reassured while 35% felt less reassured.Source: §3.5 · p. 10

3.6 Over-reliance on the model’s recommendation leads to wrong decisions

HCP decisions were least concordant with the approved correct action in cases where CORONET recommended the wrong action, with the reported examples showing HCPs following its incorrect discharge recommendation.

3.6 Over-reliance on the model’s recommendation leads to wrong decisions

  • There were only three cases in which all HCPs made the correct decision, while six cases had concordance above 80%.Source: §3.6 · p. 10
  • In Daniel’s wrong-discharge case, 65% (15/23) chose discharge as CORONET recommended; in Christine’s case, 48% (11/23) did so.The authors interpret the low concordance in these cases as evidence of over-reliance on the model, for both score-only and explanation outputs.Source: §3.6 · p. 10
  • The lowest concordance with the correct decision occurred in cases where CORONET recommended the wrong action.The figure identifies Daniel and Christine as the cases with incorrect discharge recommendations.Source: §3.6 · p. 10, Figure 8 · p. 11

3.7 Quicker decisions due to over-reliance on explanations

Explanations shortened decisions when healthcare professionals agreed with CORONET, while disagreement took longer; overall feedback was positive, but explanations did not improve perceived clinical utility.

3.7 Quicker decisions due to over-reliance on explanations

  • 38s versus 48s: aligned decisions were quicker with CS+Exp than CS, while disagreement decisions took 84s versus 61s, a nonsignificant difference.Within CS+Exp, aligned decisions were also faster than opposed decisions (38s versus 84s, p < 0.001).Source: §3.7 · p. 11

3.8 Positive feedback, but no attribution to the explanatory model

  • The CORONET model and interface were considered easy to use, and most respondents would recommend the tool to colleagues.General feedback about CORONET was positive.Source: §3.8 · p. 11
  • Explanations did not significantly change whether participants considered CORONET suitable for clinical practice (p > 0.05).The comparison of clinical utility likewise showed no significant change (p > 0.05).Source: §3.8 · p. 11

3.9 Respondents’ engagement

Participants spent less time overall on cases with explanations in just over half of cases, but total decision time did not differ significantly between conditions.

3.9 Respondents’ engagement

  • Actual questionnaire completion averaged ≈19 minutes (median 16), versus an estimated ≈30 minutes.The reported average and median completion times reflect the online questionnaire.Source: §3.9 · p. 11
  • Time anomalies in some sections were manually curated and were considered likely consequences of interruptions inherent to the clinical setting.Respondents could be distracted by more urgent matters.Source: §3.9 · p. 12
  • 56% (13/23) of HCPs spent less time deciding with CS+Exp, but total decision time did not significantly differ between CS and CS+Exp (p = 0.846).The comparison used a Wilcoxon signed-rank test.Source: §3.9 · p. 12

4 Discussion: Pragmatic clinical embedding of explainable ML models

The discussion argues that explanations need pragmatic clinical evaluation: they may support uncertain decisions and learning, but can also burden users, reinforce prior beliefs, and leave model limitations unclear.

4 Discussion: Pragmatic clinical embedding of explainable ML models

  • CORONET’s local feature explanations support modular, relevant interpretation, but its Random Forest prevents users from simulating the formal decision process or seeing behavior across the feature space.The model uses more than 100 decision trees, and the user-facing study omitted dependency plots that had been examined during model derivation.Source: §4 · pp. 12–13

Characterising HCPs’ attitude on the explanations

  • HCPs most strongly valued knowing model uncertainty; more than 87% considered feature contributions important, while an intuition about mathematical principles was significantly less important.Differences among expectations for uncertainty, feature contribution, and individual recommendations were not significant (p > 0.05).Source: Characterising HCPs’ attitude on the… · p. 13

Unmet need to build pragmatically grounded explanatory models

Ambiguity on the utility and perceived value of explanations

  • Explanations did not improve critical understanding for 57% of participants, and many reported unchanged or worsened understanding, satisfaction, or perceived safety and support in ambiguous cases.The local feature bars vary by case, while the tool focuses on COVID-19 and a finite predictor set rather than underlying biological processes; model limitations often remained unidentified.Source: Ambiguity on the utility and perceived… · p. 14

Explainable models help HCPs to address ambiguous cases

  • Explanations showed their strongest positive changes in cases where HCPs were less confident, and may offer less experienced clinicians new evidence or domain knowledge.Less experienced HCPs tended to trust the model more; explanations may guide users who lack consolidated prior knowledge.Source: Explainable models help HCPs to… · p. 15

Models and explanations can increase confirmation bias

Evidence of explanations reducing automation bias

4.1 Assumptions & limitations of this study

  • The study’s online, uncontrolled setting, short participation for some HCPs, simulated cases, and two-oncologist definition of correct actions constrain interpretation and generalizability.Simulated cases cannot reproduce clinical observations; the authors call for supervised studies with longer participation.Source: §4.1 · p. 16

5 Conclusions

The paper presents a pragmatic framework for evaluating explainable clinical decision support and finds a nuanced mix of benefits and risks. It calls for further evaluation in complex workflows and collaboration with domain experts.

5 Conclusions

  • The paper proposes pragmatic evaluation of explainable ML for clinical decision support, based on a within-subject study of 23 HCPs comparing explainable and black-box models.The framework examines user-model interaction and clinical usefulness, while the small, heterogeneous participant group limits statistically robust conclusions.Source: §5 · p. 16
  • Explanations may reduce automation bias and help with ambiguous cases and knowledge acquisition, but can also increase confirmation bias and over-reliance, while failing to improve critical understanding for 57%.Undesirable effects were reported for 17–35% of participants and may relate to increased cognitive effort.Source: §5 · p. 16
  • Further work should pragmatically evaluate explanations in complex clinical workflows, co-develop them with domain experts, and examine their interaction with other model-safety mechanisms.Source: §5 · p. 16

Funding

The project received funding from the EU Horizon 2020 programme, The Christie Charitable fund, and the National Institute for Health Research.

Funding

  • Funding came from Horizon 2020 grant 965397, The Christie Charitable fund (1049751), and support for Dr Rebecca Lee from the National Institute for Health Research.The passage identifies funding for the project and CORONET’s development, as well as Dr Lee’s support.Source: Funding · p. 17

Supplementary Information

The supplementary material lists the study questions, clinician feedback, interface design information, and additional tables and figures. It reports no significant changes in several output ratings or total decision time when explanations were added.

List of questions

  • Questions covered participant demographics, expectations about model outputs, interpretation of visualizations, and attitudes toward CORONET.The attitude questions included perceived safety, understanding of incorrect recommendations, confidence, reassurance, and uncertain decisions.Source: List of questions · p. 22

Feedback from Healthcare Professionals

  • HCP feedback described CORONET as easy to use and potentially useful for less-obvious admit/discharge choices, while some questioned its clinical fit and individualization.One clinician noted that a score could add confidence without replacing their judgment; others called the tool formulaic or insufficiently individualized.Source: Feedback from Healthcare Professionals · pp. 22–23
  • Suggested improvements included EPR integration, clearer risk and action categories, and accounting for oxygen needs and cancer-treatment factors.Clinicians also raised legal implications and guideline inclusion as considerations for future use.Source: Feedback from Healthcare Professionals · p. 23

User interface design

  • CORONET’s web interface was developed with usability and universal design principles to meet WCAG 2.1 Level AA.The passage points to additional online accessibility details.Source: User interface design · p. 23

Supplementary Tables

  • Supplementary tables report no significant differences in visual interpretation ratings or convincing capacity, and no significant correlations with competence or expectations.Expectations about mathematics were significantly lower than expectations in other areas; expectations about features, individual recommendations, and uncertainty did not significantly differ.Source: Table S.2 · p. 24, Table S.3 · p. 25, Table S.4 · p. 25
  • Responses to questions A–G showed no statistically significant pairwise change between score-only and score-plus-explanation conditions; only one correlation remained after Sidak correction.These results are summarized in Tables S.6 and S.7.Source: Table S.6 · p. 26, Table S.7 · p. 26

Supplementary Figures

  • Supplementary figures show questions A–G were asked after score-only and score-plus-explanation cases, alongside correlations in score-only responses.The correlations associate perceived helpfulness with expertise, need for feature information, and reactions to the colour bar.Source: Figure S.1 · p. 27, Figure S.2 · p. 28, Figure S.3 · p. 28
  • Total decision time did not significantly differ between conditions (p=0.846); other figures show clinician-level time changes and interface warnings and a calculation popup.Figure S.5 stratifies case-level time by concordance between clinicians’ final decisions and the model recommendation.Source: Figure S.6 · p. 29, Figure S.5 · p. 29, Figure S.7 · p. 30
Loading 2204.05030v3…