Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond

Guang Yang; Qinghao Ye; Jun Xia

Source-linked AI summary

Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond

Guang Yang, Qinghao Ye, Jun Xia

arXiv:2102.01998v1Feb 3, 2021cs.AIcs.CVcs.ITcs.LG

TL;DR

Medical AI increasingly uses powerful deep learning models whose opaque decisions can hinder understanding and clinical adoption. This paper reviews healthcare XAI, presents multi-modal and multi-centre data-fusion solutions in two clinical showcases, and reports quantitative and qualitative evidence supporting their efficacy. The authors conclude that these approaches may apply across a broader range of clinical questions.

Problem
Opaque deep learning decisions make it difficult to verify what clinical data drive predictions and can hinder confidence in medical AI.
Method
The paper surveys healthcare XAI and introduces multi-modal, multi-centre data-fusion solutions validated in two clinical showcases using interpretable diagnosis and segmentation analyses.
Results
The proposed method achieved the best performance among compared state-of-the-art methods, including at least 4.4% higher Dice scores on mixed datasets and 7.2% higher patient-level AUC than VB-Net.
Takeaways & Limitations
The review and showcases support applying XAI solutions to a broader range of clinical questions.
Takeaways & Limitations
Surrogate explanations must accurately represent the black-box model and remain sufficiently comprehensible for clinicians.

Abstract

from arXiv · show

Explainable Artificial Intelligence (XAI) is an emerging research topic of machine learning aimed at unboxing how AI systems' black-box choices are made. This research field inspects the measures and models involved in decision-making and seeks solutions to explain them explicitly. Many of the machine learning algorithms can not manifest how and why a decision has been cast. This is particularly true of the most popular deep neural network approaches currently in use. Consequently, our confidence in AI systems can be hindered by the lack of explainability in these black-box models. The XAI becomes more and more crucial for deep learning powered applications, especially for medical and healthcare studies, although in general these deep neural networks can return an arresting dividend in performance. The insufficient explainability and transparency in most existing AI systems can be one of the major reasons that successful implementation and integration of AI tools into routine clinical practice are uncommon. In this study, we first surveyed the current progress of XAI and in particular its advances in healthcare applications. We then introduced our solutions for XAI leveraging multi-modal and multi-centre data fusion, and subsequently validated in two showcases following real clinical scenarios. Comprehensive quantitative and qualitative analyses can prove the efficacy of our proposed XAI solutions, from which we can envisage successful applications in a broader range of clinical questions.

1. Introduction

AI is a broad field spanning intelligent machines, machine learning, deep learning, and related applications. Its growth has been driven by expanding data, computing capacity, and increasingly sophisticated learning systems.

AI development has accelerated through explosive data growth and rapid advances in computing hardware.
AI is an umbrella term covering machine intelligence, computer vision, robotics, natural language processing, machine learning, and deep learning.
AI systems are increasingly used in healthcare, drug development, agriculture, climate, manufacturing, transport, and sensitive policy domains.
Machine learning uses data examples to identify characteristics and improve how computers obtain predefined target outputs.
Supervised, unsupervised, reinforcement, semi-supervised, and weakly supervised learning differ according to labels, clustering objectives, experience-based rewards, or indirect labels.

1.3. Boom of Deep Learning

Deep learning uses multilayer neural networks and has transformed machine learning through strong performance on large, complex, and unstructured data. Its use requires substantial computational resources and does not provide a universal solution to every problem.

Deep learning comprises algorithms that implement neural networks with deep layers rather than a single algorithm.
Deep learning performs well on vast quantities of complex or unstructured data and often outperforms other algorithms in image detection, language processing, and voice recognition.
Deep neural networks require computer clusters and modern computing methods such as GPUs for successful training.
Successful application depends on the exact problem and available knowledge, and hybrid solutions may combine multiple algorithms.

1.4. Stunt by the Black-box and Promotion of the Explainable AI

Deep learning can deliver high performance while remaining difficult to understand, creating a need for explainable AI. XAI combines model-specific and model-agnostic strategies to expose decision rationale and support more transparent AI systems.

Deep learning tools can produce reliable outcomes while remaining opaque, making their behaviour difficult for experts to understand.
Trustable AI includes Valid, Responsible, Privacy-Preserving, and Explainable AI, while GDPR highlights fairness, privacy, transparency, and explainability.
XAI adds insight into how AI makes decisions and predictions, including system strengths, drawbacks, and possible future behaviour.
XAI methods may be model-specific or model-agnostic, and local or global, with intrinsic or post-hoc explanations.
Attention mechanisms can identify image fragments linked to generated clinical terms, providing instance-level explanations for medical image descriptions.
Surrogate models approximate black-box systems with interpretable structures, but explanations must remain accurate and sufficiently simple for clinicians to understand.
LIME explains a complex model near one instance by fitting a weighted linear model to perturbed local data.

2. Related Studies in AI for Healthcare and XAI for Healthcare

AI has expanded across healthcare, but integrating heterogeneous clinical data and interpreting complex models remain important barriers. XAI methods address black-box reasoning through visual explanations, attention, distillation, and surrogate representations, each with practical limitations.

AI in Healthcare: AI supports healthcare applications including clinical decision support, diagnosis, prognosis, treatment planning, and patient management.
AI in Healthcare: Heterogeneous data, cross-site bias, costly gold-standard labels, and limited prospective evaluation constrain real-world clinical adoption.
AI in Healthcare: Deep learning performs well on defined tasks but remains less straightforward for diagnostics and treatment planning requiring integration across modalities.
XAI in Healthcare: Black-box models make it difficult to verify whether clinically appropriate image regions drive predictions, motivating occlusion, salience, class activation, and attention maps.
XAI in Healthcare: Attention methods highlight concerning areas without directly advising responses, which can create information overload, warning fatigue, or ignored critical regions.
XAI in Healthcare: Surrogate models and mimic learning provide interpretable approximations, but explanations require faithful approximation and sufficient clinician comprehension.

3.1. Problem Formulation

The paper formulates XAI applications around common medical image-analysis problems and organizes relevant methods and clinical applications. Its showcases cover classification and segmentation using multimodal imaging datasets.

3.1. Problem Formulation: The study demonstrates XAI techniques for classification and segmentation, two widely discussed problems in medical image analysis and digital healthcare.
3.1. Problem Formulation: The paper summarizes XAI methods in digital healthcare by category, including dimension reduction and other method families listed in the table.
3.1. Problem Formulation: Figure 7 tracks yearly XAI and medical-XAI publications and reports percentages for two research categories using Scopus data retrieved January 8, 2021.

3.2. XAI for Classification

The classification solution combines explainable activation maps with slice integration and noisy-label correction for multicentre COVID-19 CT data. It addresses domain shift, patient-level aggregation, and image-level label noise while exposing regions supporting predictions.

Multicentre CT data differ across hospitals because scanning protocols and parameters create distinct distributions and visually different images.
Explainable Diagnosis Module (EDM): CAM computes class scores from weighted sums of backbone feature maps and produces class-specific localisation maps.F k denotes the k-th feature map, while W fc contains the class weights.
Explainable Diagnosis Module (EDM): The EDM replaces the fully connected layer with a 1×1 convolutional layer, enabling activation maps to be learned adaptively during training.The resulting maps indicate important CT regions, locate infected parts, and provide explanations for predictions.
Slice Integration Module (SIM): The SIM divides each patient’s CT volume into sections and aggregates section probabilities to estimate patient-level infection probability.It uses a Multiple Instance Learning framework and assumes consecutive lesions with consistent lesion-position distributions.
Slice Integration Module (SIM): The k-max selection method reduces the effect of slices containing few infections when computing section probabilities.Section probabilities are derived from the highest-scoring images for each class.
Noisy Correction Module (NCM): The NCM models a noise transaction distribution to transform true-label probabilities into noisy-label probabilities for image-level learning.The estimated distribution is combined with classification losses to form the total loss.

3.3. XAI for Segmentation

The segmentation solution targets hydrocephalus ventricle segmentation across thick- and thin-slice multimodal MRI using thick-slice annotations and unlabelled thin-slice images. Multimodal training aligns the settings, while latent-space analysis links samples to prediction quality.

The thickness-agnostic model supports both thick- and thin-slice MRI while requiring annotations only for thick-slice images.This addresses the labour cost of thin-slice annotation and poor generalisation from thick- to thin-slice images.
The model uses unlabelled thin-slice images to reduce the performance gap between thick- and thin-slice images while enabling post-hoc XAI.
The segmentation network adopts a U-Net-inspired encoder-decoder architecture with a ResNet-50 encoder pretrained on ImageNet.Sub-pixel convolution operators transform feature-map tensors during decoding.
Multimodal Training: Multimodal training jointly optimises thick-slice supervised loss and thin-slice objectives for both image types.β weights the supervised and thin-slice losses, while ps and pt are the corresponding segmentation probability maps.
Multimodal Training: The thin-slice objective promotes distribution alignment and reduces prediction uncertainty by measuring distance from the uniform distribution with an f-divergence.
Multimodal Training: Pearson χ2-divergence replaces KL-divergence to mitigate gradient imbalance during optimisation, with constant gradient slope after the replacement.
A multilayer perceptron fits decomposed latent-space samples and their Dice scores to estimate prediction quality in regions without available data.This analysis exposes relationships between sample characteristics and prediction power.

3.4. Implementation Details

The experiments use a common pretrained backbone and specified optimisation settings for the classification and segmentation tasks.

Both tasks use an ImageNet-pretrained ResNet-50 backbone; classification images are resized to 224 × 224.
Classification: Classification uses λ = 1 × 10^-4, dropout rate 0.7, L2 weight decay 1×10^-5, section length ls = 16, and k = 8.
Segmentation: Segmentation uses β = 1 × 10^-2, Adam optimisation with learning rate 1 × 10^-3, 4,000 iterations, and batch size 8.Experiments ran on four NVIDIA RTX GPUs using PyTorch 1.5.

4.1. Showcase I: Classification for COVID-19

The COVID-19 classification showcase evaluates cross-centre CT learning under heterogeneous acquisition conditions and noisy image-level labels. The proposed method combines section-based modelling with explainability analyses, achieving strong classification performance and clinically relevant visual explanations.

Data and setup: 380 COVID-19-positive and 424 COVID-19-negative CT volumes were collected from four Chinese hospitals for cross-centre training.The CC-CCII dataset was used for independent testing.
Data and setup: Different CT scanning protocols and parameters create distribution shifts across hospitals, causing performance to degrade when models transfer between centres.The collected data also contain class-imbalanced centres, complicating discriminative feature learning.
Quantitative Results: The proposed method achieved the best performance among the compared state-of-the-art COVID-19 CT classification methods.The comparison was performed on the CC-CCII dataset using patient-level and image-level settings.
Quantitative Results: 7.2% higher AUC than VB-Net was obtained at the patient level, while specificity exceeded other methods by at least 40% with high sensitivity.The authors attribute the AUC improvement to modelling section information in CT volumes through SIM.
Quantitative Results: 95.53% patient-level AUC and 86.06% image-level AUC were achieved, whereas image-label models had AUCs close to 50%.The results support the authors’ assumption that many slices in mild COVID-19 cases are disease-free.
Qualitative Results: CAMs, LIME, and Kernel SHAP highlighted infected lesions or positively contributing super-pixels for individual COVID-19 predictions.The EDM also extracted accurate lesion bounding boxes from salient CAM regions, supporting its proposed auxiliary diagnostic use.

4.2. Showcase II: Segmentation for Hydrocephalus

The hydrocephalus showcase addresses segmentation across thick- and thin-slice CT and MRI data using multimodal distribution alignment. The proposed method improves mixed-dataset segmentation and provides feature-space analyses linking prediction quality to image characteristics.

Data and setup: The cohort comprised 20 normal elderly people, 20 patients with cerebral atrophy, 64 with normal pressure hydrocephalus, and 51 with acquired hydrocephalus.Head CT scans were acquired using two Siemens CT instruments, alongside MRI examinations.
Data and setup: The study used thick-slice images with annotations together with unlabelled thin-slice images across CT and MRI modalities.Table 3 summarises the numbers of thick- and thin-slice images.
Quantitative Results: At least 4.4% higher Dice scores were achieved than competing state-of-the-art methods on mixed thick- and thin-slice datasets.The largest improvement occurred on thin-slice MRI and CT images, where the method aligned distributions and reduced uncertainty.
Quantitative Results: Training only on thick-slice images produced poor thin-slice performance, while the proposed loss terms improved cross-slice generalisation.The effectiveness of LS and LT was investigated in Table 5.
Qualitative Results: PCA projection with smoothed Dice scores showed that samples with multiple large ventricles were predicted well, whereas small ventricle regions were difficult.The projection clustered images with similar characteristics together.
Qualitative Results: On thin-slice MRI, the proposed method and U-Net++ segmented four ventricles, with the proposed method predicting the third ventricle more completely.On thin-slice CT, performance varied with image contrast, but the proposed method predicted most ventricle regions in high-contrast cases.
Qualitative Results: The proposed model could not predict the connection region between the third and fourth ventricles because the area was too small to distinguish.This is the authors’ stated limitation of the qualitative segmentation results.

4.3. Discussions

The paper’s showcases illustrate XAI for medical classification and segmentation, while the discussion emphasizes combining perceptive and mathematical evidence and improving explanations for human users.

Open challenges: Surrogate white-box models may remain too complicated or abstract for humans to understand, limiting their practical explainability.
Showcases: The showcases examine classification and segmentation models using LIME, Kernel SHAP, T-SNE, and CAMs for sensitivity and decomposition analyses.LIME and Kernel SHAP analyze individual samples through super-pixels, while T-SNE and CAMs support decomposition-oriented interpretation.
Showcases: LIME and Kernel SHAP provide local views of how explanations affect final predictions for individual diagnostic samples.
Showcases: T-SNE links decomposed image-feature distributions with prediction performance and reveals clustered characteristics of raw inputs.These patterns help identify weaknesses in black-box segmentation models and reasons for their predictions.
Interpretability: Perceptive and mathematical interpretability are complementary, so the paper argues that multiple methods should be applied simultaneously for reliable interpretation.Mathematical evidence can complement visual evidence when visual evidence is unhelpful or erroneous.
Open challenges: Current XAI explanations often target expert intuition, provide low-level feature attributions, and offer poor abstraction for lay users and clinical human-machine interaction.The discussion calls for user-focused explanations, meta-explanations connecting heatmaps with anatomical or pathological regions, and broader multimodal information fusion.

5. Conclusion

The paper concludes that the convergence of clinical databases, deep learning, open-source software, and expanding computing resources is reshaping medical AI. It reports specialized clinical successes in which AI can assess some patient prognoses more effectively than doctors and support medical decision-making.

Large annotated clinical databases, deep learning, open-source software, and increasing computing capacity and cloud storage have driven rapid growth in medical AI.
AI systems have achieved specialized success in clinical activities where they can assess patient prognosis more effectively than doctors and support medical decision-making.