Source-linked AI summary
Modality-aware Mutual Learning for Multi-modal Medical Image Segmentation
Yao Zhang, Jiawei Yang, Jiang Tian, Zhongchao Shi, Cheng Zhong, Yang Zhang, Zhiqiang He
TL;DR
Multi-modal liver tumor segmentation must integrate image information effectively and accommodate missing modalities. MAML uses collaboratively learning modality-specific models with adaptive feature exchange, achieving 81.25% Dice per case and demonstrating missing-modality handling on liver and BRATS 2018 datasets.
Problem
Multi-modal segmentation needs effective integration of modality information and a way to handle missing modalities, a scenario common in practice.
Source: §1 · p. 2Method
MAML combines modality-specific models that learn mutually and use a modality-aware module to adaptively aggregate features across modalities.
Source: §1 · p. 3, §2.3 · p. 5Results
81.25% Dice per case was reported for liver tumor segmentation, and MAML handled a one-modality input scenario on clinical and BRATS 2018 datasets.
Source: §1 · p. 4Takeaways & Limitations
The authors conclude that MAML supports multimodal segmentation and missing-modality handling, which they identify as valuable in clinical practice.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Liver cancer is one of the most common cancers worldwide. Due to inconspicuous texture changes of liver tumor, contrast-enhanced computed tomography (CT) imaging is effective for the diagnosis of liver cancer. In this paper, we focus on improving automated liver tumor segmentation by integrating multi-modal CT images. To this end, we propose a novel mutual learning (ML) strategy for effective and robust multi-modal liver tumor segmentation. Different from existing multi-modal methods that fuse information from different modalities by a single model, with ML, an ensemble of modality-specific models learn collaboratively and teach each other to distill both the characteristics and the commonality between high-level representations of different modalities. The proposed ML not only enables the superiority for multi-modal learning but can also handle missing modalities by transferring knowledge from existing modalities to missing ones. Additionally, we present a modality-aware (MA) module, where the modality-specific models are interconnected and calibrated with attention weights for adaptive information exchange. The proposed modality-aware mutual learning (MAML) method achieves promising results for liver tumor segmentation on a large-scale clinical dataset. Moreover, we show the efficacy and robustness of MAML for handling missing modalities on both the liver tumor and public brain tumor (BRATS 2018) datasets. Our code is available at https://github.com/YaoZhang93/MAML.
1 Introduction
The introduction motivates multi-modal liver tumor segmentation that can use complementary CT phases and tolerate missing modalities, then presents MAML as a collaborative, modality-aware approach.
- Tumor textures can be inconspicuous on CT, while contrast-enhanced arterial- and venous-phase images provide complementary information for diagnosis.The introduction identifies effective integration of multi-modal information and handling missing modalities as two major challenges for FCN-based segmentation.Source: §1 · p. 2
- Existing early-, middle-, late-, and ultimate-fusion methods,,, combine modality features straightforwardly, neglecting their diverse contributions.The described multi-stream approaches require registered multi-modal inputs.Source: §1 · p. 2
- Missing-modality approaches include synthesis [16] and modality-invariant features, but synthesis is computationally heavy and invariant methods can fail when most modalities are absent.KD-Net transfers knowledge from a multi-modal network to a mono-modal student, but its one-way transfer requires an additional teacher and a student for each missing modality.Source: §1 · pp. 2–3
- MAML uses modality-specific models and a modality-aware module to produce pixelwise weights and aggregate features, while mutual learning coordinates the models.Intra-modality losses encourage discriminative modality-specific features, while joint losses encourage models to learn shared high-level commonality; this interactive transfer avoids a separate teacher model.Source: §1 · p. 3, Figure 1 · p. 3
- 81.25% Dice per case was reported for liver tumor segmentation, and MAML significantly outperformed other advanced multi-modal works on the 654-volume clinical dataset.Source: §1 · pp. 3–4
2 Method
MAML uses arterial- and venous-specific models that learn collaboratively, while a modality-aware module adaptively exchanges information between phases. The framework can extend to additional modalities.
2.1 Modality-specific Model
- Each phase is processed by an nnUNet model that extracts high-level semantic embeddings from the CT volume for modality-specific segmentation.The arterial and venous embeddings have 32 channels and retain the input volume’s spatial dimensions.Source: §2.1 · p. 4
2.2 Modality-Aware Module
- The modality-aware module uses phase-specific attention to weight modality features, combining their discriminative information while reducing redundant noise in fused features.It forms fused features from the phase outputs, generates an attention map for each phase using its features and the fused features, and combines the weighted phase features. Two cascaded convolutional layers model correlations between dual-modality information and phase features; the resulting attention maps also visualize each phase’s contribution.Source: §2.2 · pp. 4–5
2.3 Mutual Learning Strategy
- Mutual learning makes the arterial and venous models reciprocal teachers and students, using intra-phase and joint losses to learn phase-specific features and shared high-level information.The losses combine cross-entropy and Dice segmentation loss, with their weighting factor λ set to 0.5. The approach is designed to handle missing modalities without modification, allowing each model to learn from the other and support segmentation when a modality is absent.Source: §2.3 · pp. 5–6
3 Experiments and Results
MAML improves multi-modal liver tumor segmentation by combining modality-aware fusion with mutual learning, while offering interpretable attention and knowledge transfer when only one modality is available.
Datasets and Evaluation Metrics
- Experiments used 654 arterial- and venous-phase CT volumes and BRATS 2018 MRI, evaluating segmentation with Dice and ASSD.The BRATS 2018 missing-modality comparison used 3-fold cross-validation with T1ce alone as input.Source: §3 · pp. 6, 7, 9
Handling Missing Modalities
- MAML significantly outperforms single-phase nnUNet and narrows the arterial–venous performance gap; on BRATS 2018 T1ce input, it excels over KD-Net, U-HVED, and HeMIS.The current framework supports either the full modality set or only one modality, not arbitrary numbers of missing modalities.Source: Table 3 · p. 8, §3 · pp. 8–9
Effectiveness of Multi-modal Modeling
- MA improves Dice and ASSD over MS+Ensemble, and adding ML further boosts performance; MAML also shows outstanding results against nnUNet and OctopusNet.The clinical-dataset ablation compares MS+Ensemble, MS+MA, and MAML.Source: §3 · p. 8
Interpretable Fusion
- The attention maps show venous focus on tumor edges, pseudo capsule, and bleeding regions, while arterial attention focuses on the tumor body.These visualizations are presented as evidence that MA captures medical-imaging knowledge for interpretable fusion.Source: Figure 2 · p. 7, §3 · p. 8
4 Conclusion
MAML combines modality-specific models that learn complementary information through mutual learning with an adaptive, explainable modality-aware module. The paper reports improved multimodal liver tumor segmentation and effective handling of missing modalities.
- MAML substantially improves multimodal segmentation performance and effectively handles missing modalities, which the authors describe as valuable in clinical practice.Source: §4 · p. 9
- Mutual learning enables an ensemble of modality-specific models to learn complementary information collaboratively.Source: §4 · p. 9
- The modality-aware module operates adaptively and explainably to improve multimodal liver tumor segmentation.Source: §4 · p. 9