PaperlayerView exact revision on arXiv

Source-linked AI summary

Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysis

Weina Jin, Xiaoxiao Li, Mostafa Fatehi, Ghassan Hamarneh

arXiv:2202.10553v3cs.LGcs.AIcs.CVeess.IV

TL;DR

Clinical XAI lacks specific, canonical criteria for ensuring that explanations are both technically sound and clinically useful. The paper proposes five Clinical XAI Guidelines, introduces multi-modal explanation evaluation with new metrics, and systematically assesses 16 heatmap methods. The methods failed G3 Truthfulness and G4 Informative plausibility, demonstrating the guidelines’ use in evaluating clinically oriented XAI.

  • Problem

    Clinical XAI lacks canonical evaluation criteria identifying which explanation objectives and metrics are suitable for clinical applications.

  • Method

    The paper proposes five Clinical XAI Guidelines and evaluates 16 post-hoc heatmap algorithms on two multi-modal medical imaging tasks using metrics including MSFI.

  • Results

    The evaluated heatmap methods failed G3 Truthfulness and G4 Informative plausibility and were therefore not suitable for clinical use.

  • Takeaways & Limitations

    The Clinical XAI Guidelines can support the design and evaluation of clinically oriented XAI in real-world clinical tasks.

  • Takeaways & Limitations

    The guidelines address general clinical requirements, while task-dependent requirements such as data privacy protection require additional criteria.

Abstract

from arXiv · show

Explainable artificial intelligence (XAI) is essential for enabling clinical users to get informed decision support from AI and comply with evidence-based medical practice. Applying XAI in clinical settings requires proper evaluation criteria to ensure the explanation technique is both technically sound and clinically useful, but specific support is lacking to achieve this goal. To bridge the research gap, we propose the Clinical XAI Guidelines that consist of five criteria a clinical XAI needs to be optimized for. The guidelines recommend choosing an explanation form based on Guideline 1 (G1) Understandability and G2 Clinical relevance. For the chosen explanation form, its specific XAI technique should be optimized for G3 Truthfulness, G4 Informative plausibility, and G5 Computational efficiency. Following the guidelines, we conducted a systematic evaluation on a novel problem of multi-modal medical image explanation with two clinical tasks, and proposed new evaluation metrics accordingly. Sixteen commonly-used heatmap XAI techniques were evaluated and found to be insufficient for clinical use due to their failure in G3 and G4. Our evaluation demonstrated the use of Clinical XAI Guidelines to support the design and evaluation of clinically viable XAI.

1. Introduction

Clinical XAI lacks canonical criteria for judging whether explanations are both technically sound and clinically useful. The paper proposes Clinical XAI Guidelines and applies them to evaluate heatmap methods for medical-image explanation.

  • Motivation: Existing XAI evaluation literature offers many objectives and metrics, but no canonical criterion establishes which explanations are suitable for clinical applications.The gap is especially relevant because clinical users rely on explanations for informed decision-making and evidence-based practice.
  • Clinical XAI Guidelines: The Clinical XAI Guidelines provide five criteria for clinical XAI, combining clinical requirements with technical evaluation objectives.G1 Understandability and G2 Clinical relevance guide explanation-form selection, while G3 Truthfulness, G4 Informative plausibility, and G5 Computational efficiency guide technique design.
  • Findings: The evaluated heatmap methods met G1, partially met G2, and failed G3 and G4, indicating inadequacy for clinical use.The results are presented as an application of the guidelines to clinically oriented XAI evaluation.
  • Evaluation: The study systematically evaluates multiple feature-attribution-map algorithms on two medical imaging tasks to assess their adherence to the guidelines.The evaluation focuses on 16 commonly used heatmap methods.
  • Evaluation: The paper formulates multi-modal medical image explanation as a clinically important extension of single-modal explanation and proposes MSFI to quantify modality-specific feature importance.MSFI is intended to automate physicians’ assessment of explanation plausibility.

2. Clinical Explainable AI Guidelines

The Clinical XAI Guidelines define five evaluation objectives for clinically usable explanations, combining clinical usability, technical evaluation, and operational considerations. They prioritize understandability and clinical relevance when selecting an explanation form, then truthfulness, informative plausibility, and computational efficiency when optimizing the technique.

  • Guidelines: The guidelines are a checklist of five evaluation objectives for optimizing clinical XAI techniques.They categorize the objectives into clinical usability, evaluation, and operation.
  • Clinical usability: G1 Understandability requires explanations that clinical users can interpret without technical knowledge of machine learning, AI, or programming.
  • Clinical usability: G2 Clinical relevance requires explanations to provide evidence that physicians can incorporate into image interpretation and clinical reasoning.The relevant information should correspond to feature extraction and reasoning on extracted features.
  • Evaluation: G3 Truthfulness requires explanations to reflect the model decision process before G4 Informative plausibility is assessed.Truthfulness can be evaluated through feature removal, synthetic datasets, and correlations between decision quality and plausibility measures.
  • Evaluation: Violating truthfulness may cause clinicians to reject correct AI suggestions or accept wrong predictions supported by seemingly plausible explanations.
  • Evaluation and operation: G4 Informative plausibility assesses whether explanations help users identify potential AI decision flaws or biases, while G5 requires computational efficiency for time-sensitive clinical workflows.Plausibility quantification can make validation automatic, reproducible, standardized, and computationally efficient, but it is not itself the evaluation end goal.

3. Evaluation problem setup

The evaluation targets the novel problem of explaining multi-modal medical images across glioma grading and knee lesion identification. It uses multiple MRI modalities, clinical requirements for modality-specific explanations, and post-hoc feature attribution methods.

  • Problem setup: The study evaluates multi-modal medical image explanation, a generalized form of single-modal explanation involving multiple signals from the same anatomy or lesion.Doctors compare and combine modality-specific information for diagnosis and differential diagnosis.
  • Model settings: The evaluation covers input-level fusion for brain tumor grading and feature-level fusion for knee lesion identification.
  • Clinical tasks: The two clinical tasks are glioma grading on brain MRI and meniscus tear identification on knee MRI.Gliomas are classified as lower-grade or high-grade, while knee MRI identifies meniscus tear versus intact tissue.
  • Datasets: The brain MRI data include four modalities, and the knee MRI data include coronal, sagittal, and axial views with modality-specific weighting.
  • Explanation methods: The study uses post-hoc feature attribution maps, excluding ante-hoc methods because their explanations are entangled with specialized model architectures.Sixteen post-hoc XAI algorithms are evaluated, including gradient-based methods.

4. Evaluation method

The evaluation follows the Clinical XAI Guidelines to assess whether heatmap explanations are suitable for clinical use. Researchers selected heatmaps through clinical feedback, tested their clinical relevance with neurosurgeons, and assessed truthfulness with perturbation, modality, and synthetic-data analyses.

  • Evaluation design: The systematic evaluation follows the Clinical XAI Guidelines so that its results indicate heatmap suitability for clinical settings.
  • G1 Understandability: Clinical collaborators selected feature attribution maps as the explanation form after commenting on understandable explanation formats.
  • G2 Clinical relevance: A prototype embedded AI predictions and heatmap explanations in a neurosurgeon user study to assess clinical relevance.Physicians assessed explanations in a clinical usage scenario.

4.3. Evaluating G3: Truthfulness

The truthfulness evaluation tests whether heatmaps identify features and modalities that drive model predictions. It combines cumulative feature removal with ∆AUPC, modality-importance correlation using Shapley values, and synthetic data with controlled ground-truth rationale.

  • Cumulative feature removal: Cumulative feature removal ranks heatmap features from most to least important, replaces them with constants, and measures the resulting model-performance curve.Accuracy is used for glioma grading and AUC for knee lesion identification.
  • Cumulative feature removal: ∆AUPC(H) = AUPC(Hb) − AUPC(H) compares an XAI method’s perturbation curve with a random-permutation baseline.A larger ∆AUPC indicates better identification of important predictive features relative to the random baseline.
  • Modality importance: MI correlation compares heatmap modality scores with ground-truth modality importance computed using Shapley values.Estimated modality importance sums positive heatmap values within each modality, and agreement is measured with Kendall’s Tau-b.
  • Synthetic evaluation: The synthetic experiment controls feature localization and modality importance so explanation agreement can be measured against known model-decision rationale.Tumor synthesis controls shapes and masks, while modality alignment controls the predictive importance of T1C and FLAIR features.
  • Synthetic evaluation: In the TIC-versus-FLAIR construction, AccT1C = 0.99 and AccFLAIR = 0 establish ground-truth modality importance of 1 for T1C and 0 for the remaining modalities.

4.4. Evaluating G4: Informative plausibility

The paper introduces MSFI to quantify informative plausibility for multi-modal explanations by combining localized feature agreement with modality prioritization. This automates and standardizes plausibility assessment against clinical prior knowledge.

  • Quantifying plausibility: FP measures the portion of highlighted heatmap regions that overlaps human-annotated feature masks.The indicator function selects heatmap values inside spatial regions where the annotated mask is positive.
  • Quantifying plausibility: MSFI combines feature localization with modality prioritization by weighting modality-specific heatmap agreement with normalized modality importance.For each modality, it measures heatmap values inside the annotated feature mask and weights them by modality importance normalized to [0, 1].
  • Quantifying plausibility: A higher MSFI score indicates greater agreement with clinical prior knowledge about important modalities and localized features.MSFI is normalized to [0, 1] and generalizes feature portion from single-modality to multi-modality images.
  • Quantifying plausibility: Automated plausibility metrics avoid manual physician assessment and allow multiple XAI methods to be evaluated using one annotated dataset.The paper motivates this approach partly because manual ratings may be susceptible to cognitive bias.
  • Quantifying plausibility: Plausibility annotations can use segmentation masks, bounding boxes, or landmarks because MSFI and FP penalize regions outside the annotation rather than requiring exact boundaries.The evaluation used tumor segmentation masks for glioma and bounding boxes for knee images.
  • Validating clinical utility: The evaluation links plausibility to clinical utility by testing its relationship with prediction probability and prediction correctness.Decision quality is assessed using soft output probability and hard-threshold prediction correctness on two classification tasks.

4.5. Evaluating G5: Computational efficiency

The evaluation records the time required to generate each heatmap as a measure of computational efficiency.

  • Computational efficiency: Heatmap generation time was recorded using a computer with 1 GTX Quadro 24 GB GPU and 8 CPU cores, plus a similarly configured computing cluster.The measurement setup provides the hardware context for evaluating computational efficiency.

5. Evaluation result

The evaluation found that heatmap explanations only partially met clinical relevance and generally failed truthfulness and informative plausibility requirements across the two tasks. Computational efficiency varied substantially, with some gradient methods near real time and some sampling-based methods taking minutes.

  • G1 Understandability and G2 Clinical relevance: Heatmaps were understandable to physicians but only partially clinically relevant because highlighted regions lacked pathological descriptions and feature reasoning.Physicians were perplexed when highlighted areas did not align with prior knowledge.
  • G3 Truthfulness: On the glioma task, Guided BackProp, Guided GradCAM, Lime, Shapley Value Sampling, and Smooth Grad achieved average ∆AUPC scores around 0,5, whereas all knee-task methods scored around 0 against the random baseline.The glioma top-ranked methods were relatively stable across models, but knee-task methods performed poorly.
  • G3 Truthfulness: On synthetic glioma data, no method exceeded an average MSFI score of 0.5, and Guided BackProp alone significantly outperformed the other methods.The synthetic-data MSFI ranking did not significantly correlate with either MI correlation or ∆AUPC on real-patient data.
  • G3 Truthfulness: Guided BackProp was the only method outperforming others across feature-level and modality-level truthfulness evaluations on glioma, although top-method performance remained around 0.5.Method rankings did not significantly generalize between glioma and knee tasks.
  • G4 Informative plausibility: MSFI correlated more strongly with physicians’ heatmap-quality ratings than FP, but low inter-rater agreement indicated that physicians’ ratings were subjective.MSFI had Pearson’s r = 0.59, while FP had r = 0.57; Krippendorff’s Alpha was 0.528 and Fleiss’ kappa was 0.009.
  • G4 Informative plausibility: The examined methods failed G4 Informative plausibility on both tasks, despite some glioma associations with prediction probability and limited class-conditioned significance.On knee, all methods had similar MSFI distributions for right and wrong predictions; glioma results were confounded by LGG and HGG distributions.
  • G5 Computational efficiency: Some gradient-based methods generated near-real-time explanations, whereas methods such as Lime or Shapley Value Sampling required 7–30 minutes.Depending on use case and parameter settings, long wait times may prevent clinical use.

6. Discussion

The evaluation found that existing heatmap methods were understandable but insufficiently clinically useful because they failed key truthfulness and informative-plausibility requirements. The discussion positions the guidelines as a staged process: select an explanation form clinically, then optimize its technique technically.

  • Evaluation findings: Heatmaps were easily understandable but only partially clinically relevant because they lacked descriptions of feature pathology.
  • Evaluation findings: None of the examined heatmap methods fulfilled G3 Truthfulness across both glioma and knee tasks.Truthfulness was assessed at both modality and feature levels.
  • Evaluation findings: G4 plausibility could correlate with model prediction probability on the glioma task, but decision-quality information remained the more clinically significant target.
  • Interpretation: Post-hoc heatmaps may reveal information already known to the model, such as prediction labels or probabilities, rather than prediction correctness, quality, or reliability.
  • Guideline implications: The guidelines distinguish clinical selection of an explanation form through G1 and G2 from technique optimization through G3, G4, and G5.
  • Guideline implications: Designers should assess truthfulness across multiple models, then test informative plausibility and computational efficiency for clinically viable deployment.

7. Limitations and future work

The paper identifies scope boundaries and methodological limitations in the guidelines’ evaluation methods. It also points toward automated, standardized evaluation pipelines as future work.

  • Scope: The guidelines cover general clinical requirements, while task-dependent requirements such as data privacy remain add-on criteria.
  • Evaluation limitations: The authors do not claim that their G3 and G4 evaluation methods are the best general methods.
  • G3 Truthfulness: Cumulative feature removal assumes feature independence, and modality-level correlation is too coarse for medical image analysis settings.
  • G4 Informative plausibility: G4 statistical testing may require enough wrongly predicted cases for statistical power, which can be difficult with accurate models and small test sets.
  • Future work: Future work may develop novel evaluation methods and automated, end-to-end, standardized pipelines for clinical XAI.

8. Conclusion

The paper proposes Clinical XAI Guidelines that combine clinical requirements with technical objectives for designing and evaluating clinical XAI. Applying them to multimodal medical-image explanation showed that the evaluated heatmap methods failed G3 and G4 and were not suitable for clinical use.

  • Conclusion: The guidelines define G1 Understandability and G2 Clinical relevance for selecting explanation forms, and G3–G5 for optimizing their techniques.
  • Conclusion: The evaluation addressed multimodal medical-image explanation in brain-tumor grading and knee-lesion identification tasks.
  • Conclusion: The study proposed MSFI to support multimodal explanation evaluation while reducing reliance on physicians’ manual plausibility assessment.
  • Conclusion: The evaluated heatmap methods failed G3 and G4, so they were not suitable for clinical use in the studied tasks.

Clinical Explainable AI Guidelines (Full Version)

The full guidelines are a five-criterion checklist organized around clinical usability, evaluation, and operation. They use clinical-user requirements to select understandable and relevant explanation forms, then evaluate their content for truthfulness, plausibility, and efficiency.

  • Framework: The guidelines are a five-criterion checklist developed from clinical and technical perspectives to guide clinically oriented XAI.
  • Guideline 1: Understandability: G1 Understandability requires explanation forms and contexts that clinical users can interpret without machine-learning, AI, or programming knowledge.
  • Guideline 1: Understandability: Physicians considered heatmaps understandable, while other candidate forms included examples, scoring, rules, and contextual information.
  • Assessment: Understandability can be assessed through AI-knowledge review, low-fidelity physician usability studies, or conversations with clinical users.
  • Guideline 2: Clinical relevance: Clinical relevance requires explanations to support physicians’ image interpretation, including feature localization, pathology identification, and reasoning over extracted features.
  • Guideline 2: Clinical relevance: Heatmaps may be clinically incomplete because they localize features without describing their pathology or supporting reasoning about those features.
  • Guideline 4: Informative plausibility: Plausibility can be quantified by comparing explanation measures with clinical assessments or decision-quality indicators, while truthfulness remains the optimization target.
  • Guideline 5: Computational efficiency: Gradient-based methods can generate near-real-time explanations up to 10 seconds, whereas Shapley Value Sampling took about 30 minutes for one heatmap.
Loading 2202.10553v3…