PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable Artificial Intelligence for Human Decision-Support System in Medical Domain

Samanta Knapič, Avleen Malhi, Rohit Saluja, Kary Främling

arXiv:2105.02357v1cs.AIcs.CVcs.HCcs.LG

TL;DR

Medical image analysis uses CNNs to support lesion detection, but their decisions can be difficult for people to understand. The paper compares LIME, SHAP, and CIU explanations on VCE images in human studies, where CIU provided the strongest decision support among the three methods.

  • Problem

    CNN decisions can be difficult to explain, although people need to understand machine decisions in medical settings to validate outcomes and support trust.

    Source: §1 · pp. 1–2

  • Method

    The study applied LIME, SHAP, and CIU visual explanations to VCE images and evaluated them in three human decision-support studies.

    Source: §1 · p. 2, §4.2 · p. 5, §5.1 · p. 6

  • Results

    CIU provided greater decision support and user satisfaction than LIME and SHAP, and users with CIU better recognized incorrect explanations than users with LIME or SHAP.

    Source: §8 · p. 18, §9 · p. 19

  • Takeaways & Limitations

    The authors conclude that CIU showed the highest decision-support and performance, while the methods may be generalized to other medical datasets after implementation improvements.

    Source: §9 · p. 19

    The study used a single medical image dataset and lay-person human evaluation rather than real-world evaluation of domain experts’ tasks.

    Source: §8.1 · p. 19

Abstract

from arXiv · show

In the present paper we present the potential of Explainable Artificial Intelligence methods for decision-support in medical image analysis scenarios. With three types of explainable methods applied to the same medical image data set our aim was to improve the comprehensibility of the decisions provided by the Convolutional Neural Network (CNN). The visual explanations were provided on in-vivo gastral images obtained from a Video capsule endoscopy (VCE), with the goal of increasing the health professionals' trust in the black box predictions. We implemented two post-hoc interpretable machine learning methods LIME and SHAP and the alternative explanation approach CIU, centered on the Contextual Value and Utility (CIU). The produced explanations were evaluated using human evaluation. We conducted three user studies based on the explanations provided by LIME, SHAP and CIU. Users from different non-medical backgrounds carried out a series of tests in the web-based survey setting and stated their experience and understanding of the given explanations. Three user groups (n=20, 20, 20) with three distinct forms of explanations were quantitatively analyzed. We have found that, as hypothesized, the CIU explainable method performed better than both LIME and SHAP methods in terms of increasing support for human decision-making as well as being more transparent and thus understandable to users. Additionally, CIU outperformed LIME and SHAP by generating explanations more rapidly. Our findings suggest that there are notable differences in human decision-making between various explanation support settings. In line with that, we present three potential explainable methods that can with future improvements in implementation be generalized on different medical data sets and can provide great decision-support for medical experts.

1. Introduction

Medical image review is time-consuming, while CNN-based lesion detection can automate analysis but remains difficult to explain. This study compares three visual explanation methods on VCE images through human decision-support studies.

1. Introduction

  • CNNs were reported to achieve higher accuracy than standard approaches, while their black-box nature limits access to the explanatory structures behind predictions.The passage describes automatic feature representation and cites for the accuracy comparison.Source: §1 · pp. 1–2
  • CNN-based lesion detection can automate analysis and reduce manual effort, but these models lack explicit explanations of their decisions.Doctors otherwise inspect thousands of images from a single procedure, requiring prolonged attention.Source: §1 · pp. 1–2
  • Medical professionals need understandable, re-traceable explanations of machine decisions to support trust and diagnosis in complex medical settings.The paper argues that explanations can help clarify black-box decisions for people using dispersed and heterogeneous data.Source: §1 · p. 2
  • Because clinicians need to validate AI decisions, the study examines whether LIME, SHAP, and CIU explanations improve comprehension and trust in CNN predictions on VCE images.The authors evaluated the three explanation types with human decision-support studies.Source: §1 · p. 2

2. Literature review

The literature review surveys explainability research and its attention to human interpretation, then situates the paper among medical XAI work. It notes prior studies of explanation methods, human decision-making, and explainability in clinical applications.

2. Literature review

  • The review describes a gap between machine-learning systems’ limited ability to explain decisions and research that has focused mainly on technical users.It surveys XAI approaches including SHAP, LIME, and CIU.Source: §2 · p. 2

2.1. Explainable Artificial Intelligence in machine learning

  • Research on explainable deep learning spans visual explanations, implementation directions, and human-subject studies, with prior work emphasizing careful choice of interpretation criteria.The cited human studies assessed how algorithmic explanations affect decision-making and simulatability across domains and techniques.Source: §2.1 · pp. 2–3

2.2. Explainable Artificial Intelligence in the medical field

  • Prior medical XAI work describes explainability as supporting transparent, understandable decisions for practitioners and patients, and examines its role in clinical and diagnostic settings.The review also discusses LIME’s classifier explanations and work on trust in AI-based computer vision.Source: §2.2 · p. 3

3. Background

The background frames transparency as a concern for machine decisions and distinguishes interpretability from explainability. It introduces post-hoc XAI and contrasts LIME and SHAP with CIU’s context-dependent feature assessments.

3. Background

  • GDPR gives users a right to be informed about machine-generated decisions, and affected people may seek reasons for those outcomes.The passage presents this as part of a broader movement toward transparency requirements.Source: §3 · p. 3

3.1. Black box predictions

  • Transparency concerns whether a model is understandable on its own, while interpretability and explainability concern human understanding of meaning and internal decision processes.The paper cautions that interpretability does not necessarily imply explainability, or vice versa.Source: §3.1 · pp. 3–4

3.2. Explainable Artificial Intelligence (XAI)

  • Post-hoc explanations provide instance-specific information about black-box decisions, with XAI aiming to support human understanding while maintaining prediction accuracy.The paper also links explanations to evaluating model strengths and limitations.Source: §3.2 · p. 4

3.3. LIME, SHAP and CIU

  • LIME approximates a prediction locally through input perturbations, while SHAP distributes prediction value among features according to their contributions.The review describes SHAP values as supporting global and instance-level interpretation.Source: §3.3 · p. 4
  • LIME and SHAP approximate local black-box behavior with linear models, whereas CIU makes feature importance context-dependent and separates importance from utility.CIU’s Contextual Importance estimates feature importance in context; Contextual Utility estimates how favorable a feature value is for an output class.Source: §3.3 · pp. 4–5

4. XAI Methods

The section presents three approaches for explaining CNN predictions: post-hoc LIME and SHAP, and the contextual alternative CIU. They differ in how they derive explanations, from local surrogate fitting and Shapley contributions to context-dependent importance and utility.

4.1. LIME

  • LIME fits a local surrogate to approximate black-box behavior near the instance being explained, balancing fidelity against complexity and sampling neighboring data.The surrogate can be a linear regression or decision tree.Source: §4.1 · p. 5

4.2. SHAP

  • SHAP explains a prediction by averaging each feature’s marginal contribution across possible coalitions, comparing black-box predictions with and without that feature.The Kernel SHAP implementation is model-agnostic and estimates SHAP values.Source: §4.2 · pp. 5–6

4.3. CIU

  • CIU explains outputs through contextual feature importance and utility, without surrogate models or linearity assumptions; feature relevance can vary with context.Contextual Importance estimates feature importance in the current context, while Contextual Utility estimates whether a feature value favors a given output class.Source: §4.3 · p. 6

5. Methodology

The methodology applies a CNN and three explanation methods to VCE images, then evaluates how explanation-supported predictions affect human decision-making.

5. Methodology

  • The study pipelines image preprocessing, CNN prediction, LIME/SHAP/CIU explanation generation, and user-study assessment of decision-making.The methods are evaluated by comparing human decision-making with and without explanation support.Source: §5 · p. 6

5.1. Image data set

  • The data come from Video Capsule Endoscopy images of the gastrointestinal tract, using 3,295 images with physician-approved annotations.The Red Lesion Endoscopy set includes bleeding and non-bleeding images; the annotations mark bleeding areas.Source: §5.1 · pp. 6–7

5.2. Implementation of the black box model

  • The CNN classifies images as bleeding or normal and achieved 98.58% validation accuracy on the randomly split data.The model was trained for 50 epochs with batch size 16; the reported training and validation split contains 2,941 and 354 images.Source: §5.2 · p. 7

5.3. Explainability

  • LIME, SHAP, and CIU explanations highlight image features relevant to the CNN’s bleeding or non-bleeding predictions.The explanations were checked against physician-confirmed annotated masks to ensure they at least partly marked correct areas.Source: §5.3 · p. 9

5.3.1. LIME explanations

  • LIME uses yellow boundaries to mark image regions contributing positively to the predicted bleeding or non-bleeding class.Explanations used 2,500 neighborhood samples and up to 10 features.Source: §5.3.1 · p. 9, Figure 6 · p. 10

5.3.2. SHAP explanations

  • Kernel SHAP explains CNN image predictions on superpixel-segmented images, using green for support and red for contradiction to each class.SHAP explanations used 2,500 samples.Source: §5.3.2 · p. 10, Figure 7 · p. 10

5.3.3. CIU explanations

  • CIU explanations mark image areas contributing to the predicted bleeding or non-bleeding class.Explanations used a 0.01 threshold and 50 superpixels.Source: §5.3.3 · p. 10

6. Human evaluation user study

The study compares decision-making with and without LIME, SHAP, or CIU explanations, and assesses participants’ understanding of those explanations. It uses web-based tests with non-medical participants.

6. Human evaluation user study

  • Three user studies assessed how LIME, SHAP, and CIU explanation support affected decision-making, user understanding, satisfaction, and recognition of explanation correctness.Each study compared no-explanation and explanation-supported conditions.Source: §6 · pp. 10–11, §6.2 · p. 11

6.1. Data collection

  • The 60 participants were non-medical users, with 20 assigned to each method’s study group.Participants completed web-based surveys; the sample came from a university academic environment and predominantly had STEM backgrounds.Source: §6.1 · p. 11

6.2. Study description and design of the user study

  • Participants learned bleeding versus normal classification, made decisions first without and then with explanations, rated satisfaction, and judged explanation correctness.The study tested decision accuracy against no-explanation conditions and compared groups’ ability to identify incorrect explanations; participants saw five incorrect explanations among twelve cases.Source: §6.2 · pp. 13–14

7. Analyses of the results

Across the user studies, CIU support was associated with more correct decisions and higher satisfaction than LIME or SHAP on several comparisons, although many differences were not statistically significant. CIU explanations also took less time to generate, and users requested more precise visual areas, text explanations, and interaction options.

7.1. Performance of LIME, SHAP and CIU

  • 8.5 seconds per image for CIU, versus 9.8 for SHAP and 11.4 for LIME; the methods ran on different systems.For 28 images, CIU took 4 minutes; LIME took about 5 minutes 20 seconds and SHAP about 4 minutes 35 seconds.Source: Table 4 · p. 14, §7.1 · p. 14

7.2. Quantitative analyses of explanations

  • The quantitative analyses compared users' decisions across explanation and no-explanation conditions and between method groups, using unequal-variance t-tests at α = 0.05.The researchers also compared group means and medians and conducted exploratory statistics.Source: §7.2 · pp. 14–15

7.2.1. Analyses of human decision-making from the three users groups with different explanation support methods

  • LIME users improved slightly with explanations (p = 0.738), while SHAP users did better without explanations (p = 0.464); neither difference was significant.CIU users also answered more questions correctly with explanation support than without it, but that difference was not significant (p = 0.158).Source: §7.2.1 · pp. 15–16
  • SHAP users reported greater satisfaction than LIME users (p = 0.0135) and took longer to complete the study (p = 0.011); LIME users answered more questions correctly, nonsignificantly.SHAP users' reported understanding was not significantly higher than LIME users' (p = 0.999).Source: §7.2.1 · p. 16
  • CIU users answered more questions correctly than LIME and SHAP users, though not significantly; satisfaction was significantly higher than with LIME (p = 0.000).CIU users also reported higher—but not statistically significant—understanding and satisfaction than SHAP users.Source: Table 9 · p. 16, §7.2.1 · p. 16, Table 10 · p. 17

7.2.2. Correlation analyses

  • Correlations varied by condition: LIME decision counts correlated with age and satisfaction, SHAP counts with satisfaction and understanding, and CIU showed no significant correlations.In the no-explanation condition, no significant correlations were found for LIME; SHAP decision counts correlated with less completion time.Source: §7.2.2 · pp. 17–18

7.3. Qualitative analyses of explanations

  • Users rated CIU explanations most highly, but requested more precise image areas, supplementary text, and options to interact for deeper information.The evaluation also found that most participants said they could understand the provided explanations.Source: §7.3 · p. 18

8. Discussion

The discussion reports that CIU support was associated with better decision-making and explanation understanding than LIME or SHAP on several measures, though not every comparison was significant. The study's evaluation was limited to one medical image dataset and lay participants, and the authors propose broader datasets and expert, real-world evaluations.

8. Discussion

  • Across the three explanation-support groups, the authors conclude that CIU provided greater help with correct decisions and higher satisfaction than LIME and SHAP.They interpret this pattern as indicating that CIU explanations appeared clearer to users.Source: §8 · p. 18
  • CIU users were better at distinguishing correct from incorrect explanations than LIME users, and somewhat better than SHAP users; they also completed the study faster than SHAP users.The authors suggest lower explanation complexity and more accurate depiction of significant image areas as possible reasons for CIU users' understanding.Source: §8 · p. 18

8.1. Limitations

  • Evaluation covered one medical image dataset and lay participants, not real-world tasks performed by domain experts.The authors identify application-based evaluation with diagnostic physicians as a potential extension.Source: §8.1 · p. 19

8.2. Future work

  • The authors propose testing the explanations on different medical datasets, larger participant samples, medical experts, and real-world scenarios.They note that a larger sample could yield more statistically meaningful hypothesis outcomes.Source: §8.2 · p. 19

9. Conclusion

CIU showed the highest decision-support and performance among the explanation settings, while explanations increased users’ correct decisions in two of three studies. The authors propose further evaluation with domain experts and future work toward broader medical applications.

9. Conclusion

  • CIU showed the highest decision-support and performance, and explanations increased users’ correct decisions in two of three studies compared with no explanation.The evaluation used lay participants; the authors propose application-grounded evaluation with domain experts.Source: §9 · p. 19
  • The authors propose evaluating explanations with domain experts performing tasks specific to their use for a clearer assessment.The present evaluation used lay participants.Source: §9 · p. 19
  • With future implementation improvements, the methods may be generalized to other medical datasets and image-processing settings for decision support.The authors also identify testing and improvement with medical professionals as steps toward broader applicability.Source: §9 · p. 20
Loading 2105.02357v1…