PaperlayerView exact revision on arXiv

Source-linked AI summary

SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosis

Yuqing Yang, Alexander Schmatz, Zhaozhao Ma, Changkyu Choi, Robert Jenssen, Shujian Yu

arXiv:2609.05174v1cs.CVcs.LG

TL;DR

Medical diagnosis needs explanations that remain faithful in multimodal settings, where existing approaches are mostly post-hoc and unimodal. SMILE jointly learns modality-specific informative features and predictive representations through an information bottleneck framework, achieving strong diagnostic performance and transparent feature relevance across heterogeneous medical datasets.

  • Problem

    Multimodal medical diagnosis lacks principled frameworks that jointly optimize prediction and explanation, while existing methods are largely post-hoc and unimodal.

  • Method

    SMILE jointly learns modality-specific explainers and predictive representations within an information bottleneck framework, using encoders to make mutual-information optimization tractable.

  • Results

    SMILE consistently achieves state-of-the-art diagnostic performance and produces instance-wise, modality-specific explanations validated quantitatively and qualitatively.

  • Takeaways & Limitations

    The learned explanations provide transparent, modality-aware feature relevance while selected features remain intrinsic to the diagnostic decision process.

  • Takeaways & Limitations

    SMILE assumes sufficiently expressive encoders and leaves high-order cross-modal interaction terms implicit.

Abstract

from arXiv · show

Explainability is increasingly seen as a crucial requirement in AI-based medical diagnosis, particularly in safety-critical clinical decision-making. Most existing explainability methods in healthcare operate in a post-hoc manner and are predominantly designed for unimodal data, which limits their applicability in increasingly prevalent multimodal diagnostic settings. This paper addresses the problem of self-explainable multimodal diagnosis by formulating it within the information bottleneck (IB) framework. We propose a unified learning paradigm that jointly optimizes predictive performance and modality-specific explainability by identifying the most informative elements inside each modality that contribute to diagnostic decisions. To enable tractable and stable optimization, we employ a matrix-based Renyi's $α$-order entropy functional under the assumption of sufficiently expressive encoders. Extensive experiments on representative medical datasets spanning heterogeneous modalities demonstrate that the proposed method consistently achieves strong diagnostic performance, including an absolute accuracy improvement of 9.1 percentage points on the iCTCF dataset. Moreover, the learned explanations provide transparent and modality-aware insights into feature relevance, thereby improving both the explainability and generalization.

1 Introduction

Medical diagnosis increasingly uses complex multimodal data, but existing explainability methods remain largely post-hoc and unimodal. SMILE addresses this gap by jointly learning diagnostic representations and modality-specific explanations within an information bottleneck framework.

  • Black-box medical models can undermine clinicians’ confidence in automated diagnostic decisions, motivating explainable AI for safety-critical care.
  • Post-hoc methods such as Grad-CAM, LIME, and SHAP identify influential features but are predominantly designed for unimodal data and may limit explanation faithfulness.
  • Heterogeneous modalities provide complementary diagnostic information, while their interactions make individual feature and modality contributions difficult to interpret.
  • SMILE formulates self-explainable multimodal diagnosis as an information bottleneck problem, learning compact modality-specific feature subsets that retain diagnostic information and discard redundancy.
  • SMILE jointly learns predictive representations and explanations, making selected features intrinsic to the decision process rather than post-hoc analyses.
  • Experiments on representative multimodal medical datasets report state-of-the-art diagnostic performance and validated instance-wise, modality-specific explanations.

2 Related Work

Prior work covers post-hoc, modality-level, and information-bottleneck-based explanations, but multimodal medical diagnosis still lacks a generic self-explainable feature-level framework. SMILE targets this gap with modality-aware explanations integrated into prediction.

  • Explainability for Medical Diagnosis: Medical XAI includes saliency-based and perturbation-based post-hoc methods, with saliency approaches commonly visualizing clinically relevant image regions.
  • Explainability for Medical Diagnosis: Multimodal medical explanations are predominantly post-hoc, often disease-specific, and frequently extend classic methods from unimodal settings.
  • Explainability for Medical Diagnosis: Modality- and interaction-level methods quantify which modalities or modality interactions drive predictions but provide coarser explanations than feature-level analyses.
  • Explainability for Medical Diagnosis: SMILE instead targets modality-aware, feature-level self-explainability in a generic framework applicable in principle across data modalities and disease types.
  • Information Bottleneck in Deep Learning: The information bottleneck identifies a compact variable that preserves predictive information about Y while restricting information retained from X.
  • Information Bottleneck in Deep Learning: SMILE’s modality-specific explainers select top-k informative features, whose selected representations are used directly for prediction to align explanations with decisions.
  • Information Bottleneck in Deep Learning: Unlike related IB attribution methods designed as post-hoc tools for specific vision-language architectures, SMILE addresses self-explainable clinical diagnosis.

3 Methodology

SMILE jointly learns modality-specific explanations and predictive representations by selecting compact, informative input subsets within an information bottleneck framework. Its objective combines predictive information, sparsity, and compression, with tractable latent-space estimation and differentiable top-k selection.

  • Information bottleneck objective: The multimodal objective combines predictive information with an ℓ0 sparsity constraint and an information bottleneck compression term.The fusion network combines encoded selected inputs across modalities for prediction.
  • Self-explainable multimodal model: SMILE learns modality-specific binary explainers that identify informative input elements and jointly trains them with the predictive model.Selected elements can be modality-dependent and sample-dependent, covering features such as image patches, graph components, or structured variables.
  • Information bottleneck objective: β = 0 makes the information bottleneck objective reduce to INVASE as a special case.This establishes a direct relationship between SMILE’s single-modality objective and the related instance-wise feature-selection formulation.
  • Information bottleneck objective: Mutual-information compression complements ℓ0 sparsity by regulating preserved information rather than only counting selected features.In deterministic masking, I(X; ˜X) = H(˜X), so compression encourages compact selected representations.
  • Tractable optimization: The method estimates otherwise infeasible mutual information in modality-specific latent spaces under a sufficient-encoder assumption.The ℓ0 penalty is implemented as an explicit top-k cardinality constraint in the tractable formulation.
  • Tractable optimization: Gumbel-Softmax provides differentiable subset selection, while a continuous relaxation represents the top-k informative elements.Selection networks assign importance scores, and the resulting explanation mask is used to identify informative features.
  • Generalization analysis: Theoretical analysis states that the compression term does not necessarily impose an intrinsic accuracy penalty and can reduce generalization error.This connects the information bottleneck regularizer to SMILE’s generalization analysis.

4 Experiments and Results

SMILE is evaluated across heterogeneous multimodal medical datasets using modality-specific selection networks and empirically chosen feature budgets. Across quantitative and qualitative analyses, it delivers strong predictive results while identifying clinically or biologically relevant, modality-specific features and regions.

  • Experimental Setup: The evaluation covers multi-omics, imaging-plus-clinical, ophthalmic imaging, and neuroimaging datasets with modality-specific selection networks.Image explanations operate at the patch level, while structured-data selection networks accommodate heterogeneous modalities.
  • Quantitative Analysis: SMILE achieves the best BRCA ACC, WeightedF1, and MacroF1 at 87.3%, 87.8%, and 84.6%, respectively.On ROSMAP, SMILE leads ACC at 85.1% and F1 at 85.6%, while DMIB has the best AUC at 91.6% versus SMILE’s 90.3%.
  • Quantitative Analysis: 9.1 percentage points: iCTCF accuracy rises from 83.3% to 92.4% over HUST-19.SMILE also attains AUC of 95.6%, JW0.5 of 84.3%, and JW0.6 of 82.2%.
  • Quantitative Analysis: SMILE improves Glaucoma Grading Cohen’s Kappa to 69.2% from 65.4% with DuelRes and to 72.4% from 70.1% with Res-DEN.The experiments replace classifier encoders with pre-trained models and report better stability.
  • Quantitative Analysis: REST-meta-MDD accuracy reaches 67.4%, exceeding MoNIG by 3.8 percentage points, while MCC reaches 35.0% versus 27.0%.SMILE’s F1-score is 69.7%, near MIB’s best-baseline value of 70.1%.
  • Qualitative Analysis: Qualitative explanations identify disease-relevant biomarkers, lung findings, clinical variables, image features, and anatomically coherent neuroimaging regions across modalities.Examples include breast-cancer and Alzheimer’s biomarkers, COVID-19 consolidation and GGOs, glaucoma features, and sparse REST-meta-MDD selections.
  • Ablation Study: Compared with post-hoc LIME and SHAP applied to the same late-fusion encoders, SMILE’s built-in explainer is evaluated for fidelity and stability alongside multimodal interactions.The ablation results report that SMILE captures more decision-relevant information and is more stable than random masks, whose performance gains have higher variance.

5 Conclusion and Future Work

SMILE combines modality-specific explainers, encoders, and a fused classifier within an information-bottleneck framework, achieving strong diagnostic performance and faithful explanations across heterogeneous multimodal datasets. Future work targets the sufficient-encoder assumption and explicit modeling of cross-modal synergy.

  • Conclusion: SMILE combines modality-specific explainers, latent-representation encoders, and a fused classifier under the information-bottleneck principle.The framework jointly selects informative modality-specific features and performs multimodal diagnosis.
  • Conclusion: Across five heterogeneous multimodal medical datasets, SMILE exceeds state-of-the-art diagnostic performance while producing faithful modality-specific explanations.The conclusion reports both predictive and explanatory benefits across the evaluation suite.
  • Conclusion: The IB compression term does not necessarily impose an accuracy penalty and can improve the generalization bound by controlling retained information.This theoretical claim is stated for the proposed framework rather than as a universal guarantee.
  • Future Work: The sufficient-encoder assumption simplifies optimization but remains over-optimistic, making estimation of I(X; ˜X) without it an open problem.The authors identify this assumption as a limitation of the current analysis.
  • Future Work: SMILE leaves cross-modal structure implicit and future work will explicitly model high-order modality interactions such as synergy.The current method assumes that all modalities may contribute while focusing on modality-specific informative features.

A Proof to Proposition 1

The proof relates the proposed single-modality information-bottleneck objective to INVASE and shows that INVASE is recovered when the compression term is removed. It uses KL-divergence decomposition, the Markov property, and fixed mutual information to establish the connection.

  • The proposed IB objective contains INVASE as a special case when β = 0, because INVASE lacks a compression term.
  • The proof uses the Markov chain ˜X ← X → Y, implying P(Y |X) = P(Y |X, ˜X).
  • Minimizing the expected KL divergence is equivalent to maximizing I(Y ; ˜X), because I(X; Y ) is fixed by the training data.

B Proof to Proposition 4

The proof applies information-theoretic generalization bounds to the multimodal architecture. It models modalities and their encoded representations as information sources transmitted through decoupled channels, yielding an upper bound on generalization error.

  • The generalization analysis starts from a high-probability bound over training data drawn from p(x, y), using a bounded per-sample loss.
  • The resulting representation after the first l layers is denoted Z_t^l and enters the generalization-bound analysis.
  • The multimodal inputs are treated as a joint information source transmitted through a parameterized channel to ¯Z and then to Y.
  • The encoded representations are treated as a joint source Z, and the data processing inequality provides an upper bound on generalization error.
  • The proof assumes modality-specific channels are decoupled, so the conditional distribution of Z given X factorizes across modalities.

C.0.1 BRCA

The BRCA implementation uses modality-specific encoders and selectors to retain 30 features from each omics modality. Their 500-dimensional representations are concatenated and classified into five PAM50 subtypes.

  • Encoders: The mRNA branch maps selected 1000-dimensional inputs to 500-dimensional latent representations using projection, ReLU, and dropout with p = 0.5.
  • Feature selection: Each modality uses a selector that produces feature logits and retains k = 30 features through the SMILE top-k operation.
  • Encoders: The DNA-methylation branch uses the same encoder design, mapping selected 1000-dimensional inputs to 500-dimensional representations with dropout p = 0.5.
  • Encoders: The miRNA branch maps selected 503-dimensional inputs to 500-dimensional representations using a fully connected projection, ReLU, and dropout.
  • Fusion and training: The three modality representations are concatenated into a 1500-dimensional vector and classified into five PAM50 subtype labels.
  • Fusion and training: Training runs for 2000 epochs with Adam, initial learning rate 10^-4, weight decay 10^-4, scheduled decay, and β = 0.08.

C.0.2 ROSMAP

The section specifies modality-specific selectors and encoders for omics, clinical, HRCT, fundus, and OCT data, followed by multimodal classifier heads. Differentiable top-k selection identifies informative features or image patches before fusion.

  • Omics modalities: The mRNA, methylation, and miRNA encoders project selected 200-dimensional vectors into 300-dimensional latent representations.The mRNA and miRNA branches use fully connected projections with ReLU and dropout, while methylation uses the same fully connected encoder.
  • Omics modalities: SMILE retains 30 features for each mRNA, DNA-methylation, and miRNA modality through differentiable top-k selection.Each omics selector outputs feature logits before retaining the top k = 30 features.
  • Clinical and HRCT modalities: Clinical and HRCT representations are each 1024-dimensional and are fused into a 2048-dimensional vector for binary diagnosis.The HRCT selector retains 60 patches from a 10 × 10 grid before encoding.
  • Glaucoma modalities: SMILE selects 100 patches from both fundus and OCT inputs before encoding them into 1000- and 512-dimensional modality representations.The fundus selector operates on a 32 × 32 grid, while the OCT selector operates on a 32 × 32 patch-logit grid.
  • Glaucoma modalities: The fundus and OCT embeddings are concatenated into a 1512-dimensional vector and mapped to three diagnostic classes.The classifier uses a 1512 →756 →3 architecture with ReLU activation before the output layer.

C.0.5 REST-meta-MDD

The REST-meta-MDD configuration combines functional-connectivity graphs with volumetric sMRI representations. Differentiable sparse selection identifies informative edges and subvolumes before the two modalities are fused for MDD diagnosis.

  • rs-fMRI: rs-fMRI preprocessing uses AAL-116 ROI-averaged BOLD signals with motion, normalization, filtering, and nuisance-regression steps.The temporal band-pass range is 0.01 to 0.10 Hz, and nuisance regressors include motion, global signal, white matter, cerebrospinal fluid, and drift terms.
  • rs-fMRI: The rs-fMRI graph encoder produces a 1024-dimensional embedding through GCN processing, normalization, and second-order pooling.Node features follow 116 →116 →96 →64, and a bilinear outer product is flattened into the graph representation.
  • rs-fMRI: The subgraph selector converts admissible edge logits into a sparse differentiable edge mask using soft top-k selection.Edge embeddings are formed from concatenated node pairs and processed by an MLP before masking with the shared adjacency.
  • sMRI: sMRI inputs are z-normalized GMV maps, and the encoder reduces them to a 7 × 9 × 7 feature grid with 64 channels.The five-layer 3D CNN processes maps of spatial size 121 × 145 × 121.
  • sMRI: Soft top-k selection retains 25 sMRI subvolumes from 441 candidates while annealing the budget and temperature during warm-up.The selection budget decreases from 441 to 25 and the temperature from 1.0 to 0.05 over the first 200 epochs.
  • Multimodal fusion: The 1024-dimensional rs-fMRI and 64-dimensional sMRI embeddings form a 1088-dimensional representation for MDD diagnosis.Training uses encoder warm-up, early stopping, and 10 repeated runs.

D Additional Visualization Results

Additional visualizations show how SMILE exposes modality-specific evidence for clinical interpretation. iCTCF visualizations highlight HRCT patches and summarize selected clinical-variable importance, while glaucoma visualizations highlight fundus and OCT regions.

  • iCTCF: iCTCF visualizations highlight selected HRCT patches and report normalized importance for selected clinical variables.Representative severe and mild cases support inspection of modality-specific evidence across prognostic outcomes.
Loading 2609.05174v1…