Source-linked AI summary
Modality-aware Mutual Learning for Multi-modal Medical Image Segmentation
Yao Zhang, Jiawei Yang, Jiang Tian, Zhongchao Shi, Cheng Zhong, Yang Zhang, Zhiqiang He
TL;DR
Automated liver tumor segmentation needs to integrate multi-modal CT information while addressing missing modalities. MAML uses modality-specific models with modality-aware adaptive fusion and collaborative mutual learning, achieving promising segmentation performance and robustness when only one modality is available. Its current implementation, however, supports either all modalities or a single modality, not arbitrary numbers of missing modalities.
Problem
Multi-modal medical image segmentation must effectively integrate modality information and handle missing modalities, which are common in practice.
Method
MAML combines modality-specific segmentation models with a modality-aware attention module and mutual learning using intra-modality and joint losses.
Results
MAML achieves 81.25% Dice per case for liver tumor segmentation and demonstrates effectiveness and robustness with only one modality available.
Takeaways & Limitations
MAML collaboratively transfers modality-specific and shared information to support multi-modal segmentation and missing-modality handling without modifying the models.
Takeaways & Limitations
The current implementation accepts either the full modality set or only one modality, rather than an arbitrary number of missing modalities.
Abstract
from arXiv · showhide
Liver cancer is one of the most common cancers worldwide. Due to inconspicuous texture changes of liver tumor, contrast-enhanced computed tomography (CT) imaging is effective for the diagnosis of liver cancer. In this paper, we focus on improving automated liver tumor segmentation by integrating multi-modal CT images. To this end, we propose a novel mutual learning (ML) strategy for effective and robust multi-modal liver tumor segmentation. Different from existing multi-modal methods that fuse information from different modalities by a single model, with ML, an ensemble of modality-specific models learn collaboratively and teach each other to distill both the characteristics and the commonality between high-level representations of different modalities. The proposed ML not only enables the superiority for multi-modal learning but can also handle missing modalities by transferring knowledge from existing modalities to missing ones. Additionally, we present a modality-aware (MA) module, where the modality-specific models are interconnected and calibrated with attention weights for adaptive information exchange. The proposed modality-aware mutual learning (MAML) method achieves promising results for liver tumor segmentation on a large-scale clinical dataset. Moreover, we show the efficacy and robustness of MAML for handling missing modalities on both the liver tumor and public brain tumor (BRATS 2018) datasets. Our code is available at https://github.com/YaoZhang93/MAML.
1 Introduction
Multi-modal liver tumor segmentation must integrate complementary CT information while remaining robust when modalities are unavailable. MAML addresses these challenges with modality-specific models, modality-aware fusion, and mutual learning, achieving 81.25% Dice per case and robustness with only one modality available.
- Problem: Multi-modal FCN segmentation primarily requires effective information integration and reliable handling of missing modalities in clinical practice.These challenges motivate the proposed framework.
- Related work: Existing multi-modal approaches commonly fuse modality-specific streams through early-, middle-, or late-fusion architectures after image registration.The related strategies process modalities jointly at the input, share a decoder, or maintain separate encoder-decoder streams before fusion.
- Related work: Missing-modality methods include modality synthesis, modality-invariant features, and knowledge distillation, but synthesis is computationally heavy and prior invariant methods can fail when most modalities are missing.KD-Net additionally requires a separate student for each missing modality and a teacher for one-way transfer.
- Proposed method: MAML uses modality-specific models, a modality-aware module for adaptive feature aggregation, and mutual learning through intra-modality and joint losses.The modality-aware module produces pixel-wise weight maps, while mutual learning enables interactive knowledge transfer without an additional teacher model.
- Results: 81.25% Dice per case was achieved for liver tumor segmentation, with effectiveness and robustness also demonstrated when only one modality was available.The latter evaluation covered both the clinical liver tumor dataset and the public BRATS 2018 dataset.
2 Method
MAML combines modality-specific models, modality-aware attention, and mutual learning to integrate arterial and venous CT information while supporting missing-modality segmentation.
- Modality-Specific Model: MAML uses a common FCN for each modality, with nnUNet extracting high-level semantic embeddings from separate arterial and venous CT volumes.The modality-specific outputs have the same spatial shape as the input image and contain 32 channels.
- Modality-Aware Module: The modality-aware module concatenates dual-modality and phase-specific features, generates attention maps, and adaptively aggregates them into fused representations.Attention weights are applied element-wise to modality-specific features before weighted summation.
- Modality-Aware Module: Attention maps selectively emphasize phase characteristics and provide a visual interpretation of each modality’s contribution to liver tumor segmentation.The maps indicate how much attention is assigned to features from each phase.
- Mutual Learning Strategy: MAML uses modality-specific models that collaboratively learn from arterial and venous phases through mutual teacher-student interactions.Each stream learns phase-specific features while also receiving information from the other modality.
- Mutual Learning Strategy: The objective combines intra-phase and joint segmentation losses, using Cross-Entropy and Dice losses with λ empirically set to 0.5.The combined segmentation loss addresses imbalanced tumor distributions.
- Mutual Learning Strategy: Mutual learning supports both multi-modal segmentation and missing-modality handling without architectural modification by transferring information among modality-specific models.The collaboration combines modality-specific characteristics with shared information across modalities.
3 Experiments and Results
Experiments evaluate MAML on clinical contrast-enhanced CT and brain-tumor missing-modality settings, comparing modality fusion, qualitative detection, and missing-modality robustness.
- Datasets and Evaluation Metrics: The experiments use 654 paired arterial and venous contrast-enhanced CT volumes from Chinese PLA General Hospital.
- Attention maps capture enhanced tumor regions as well as bleeding parts and pseudo capsules.
- MAML detects tumors in qualitative cases where single-phase methods fail on either arterial or venous images.Tumors are marked in red and highlighted with yellow arrows.
- Effectiveness of Multi-modal Modeling: MAML outperforms the MS+Ensemble baseline and further improves performance when mutual learning is added to modality-aware aggregation.The comparison uses Dice and ASSD on the clinical liver-tumor dataset and also reports outstanding performance against nnUNet and OctopusNet.
- Handling Missing Modalities: On BRATS 2018 with only T1ce input, MAML outperforms KD-Net, U-HVED, and HeMIS for missing-modality handling.The comparison uses Dice, with comparator results taken directly from prior work.
4 Conclusion
The paper concludes that MAML supports effective and robust multi-modal segmentation by combining collaborative modality-specific learning with adaptive, explainable modality interaction.
- MAML enables effective and robust multi-modal segmentation through collaborative learning among modality-specific models.
- Mutual learning captures complementary information, while modality-aware aggregation adaptively and explainably improves multi-modal liver tumor segmentation.
- The method is reported to handle missing modalities effectively, a capability described as valuable in clinical practice.