Source-linked AI summary
Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalization
Yunpeng Zhao, Cheng Chen, Qing You Pang, Quanzheng Li, Carol Tang, Beng-Ti Ang, Yueming Jin
TL;DR
Missing modalities can occur during both training and inference, while many approaches assume complete training data. The paper proposes reconstruction and model personalization for varied missing-modality settings, reporting superior results on two brain tumor segmentation benchmarks, including with only 1% full-modality training data.
Problem
Many missing-modality approaches focus on inference and assume complete training data, despite modality-incomplete samples occurring in practical settings and missing inputs affecting both phases.
Source: §1 · pp. 2–3Method
A multimodal masked autoencoder with distribution approximation reconstructs missing modalities, and reconstructed information guides learning while a CLIP-driven hyper-network personalizes parameters by missing-modality scenario.
Source: §1 · p. 3Results
The method reports superior performance on two brain tumor segmentation benchmarks across all-stage missing-modality settings, including superiority to methods with 100% full-modality training when using only 1%.
Source: §1 · p. 4, §5 · p. 15Takeaways & Limitations
The proposed model is presented as accommodating varying complete/incomplete training-data ratios and different missing-modality scenarios at testing.
Source: §1 · p. 3The distribution-approximation rationale assumes a segmentation model with favorable generalization that has its lowest segmentation loss on full-modality inputs.
Source: §3.1 · p. 6
Abstract
from arXiv · showhide
Addressing missing modalities presents a critical challenge in multimodal learning. Current approaches focus on developing models that can handle modality-incomplete inputs during inference, assuming that the full set of modalities are available for all the data during training. This reliance on full-modality data for training limits the use of abundant modality-incomplete samples that are often encountered in practical settings. In this paper, we propose a robust universal model with modality reconstruction and model personalization, which can effectively tackle the missing modality at both training and testing stages. Our method leverages a multimodal masked autoencoder to reconstruct the missing modality and masked patches simultaneously, incorporating an innovative distribution approximation mechanism to fully utilize both modality-complete and modality-incomplete data. The reconstructed modalities then contributes to our designed data-model co-distillation scheme to guide the model learning in the presence of missing modalities. Moreover, we propose a CLIP-driven hyper-network to personalize partial model parameters, enabling the model to adapt to each distinct missing modality scenario. Our method has been extensively validated on two brain tumor segmentation benchmarks. Experimental results demonstrate the promising performance of our method, which consistently exceeds previous state-of-the-art approaches under the all-stage missing modality settings with different missing ratios. Code will be available.
1 Introduction
The paper targets modality-incomplete brain tumor segmentation during both training and inference, proposing reconstruction and model personalization for varied missing-modality scenarios. It reports superior results on two benchmarks, including when training data contain only 1% full-modality samples.
1 Introduction
- Missing MRI modalities are common in clinical scenarios, while conventional multimodal segmentation methods require complete inputs and struggle to use incomplete data during training or maintain inference performance.The cited causes include artifacts, acquisition protocols, contrast-agent allergies, and economic considerations.Source: §1 · p. 2
- The all-stage setting must address information loss in both phases and heterogeneous distributions across missing-modality combinations, including 15 combinations for four modalities.The paper notes that forcing diverse distributions through shared parameters can be suboptimal.Source: §1 · p. 3
- Distribution approximation lets a multimodal masked autoencoder reconstruct missing modalities and masked patches using complete and incomplete training data, while data-model co-distillation uses reconstructions to guide learning.The approach is designed to exploit inter-modal correlations and reconstructed full-modality information.Source: §1 · pp. 3–4
- A CLIP-driven hyper-network uses textual modality prompts and visual embeddings to personalize partial model parameters for distinct missing-modality scenarios.The personalized parameters are prediction-level parameters.Source: §1 · pp. 3–4
- The authors describe their work as the first to tackle missing modalities during both training and testing.Source: §1 · p. 3
- With only 1% full modality in training data, the model achieves superior results to other methods with a 100% ratio.The reported evaluation covers two brain tumor segmentation benchmarks under all-stage missing-modality settings.Source: §1 · p. 4
2 Related Work
The section contrasts missing-modality methods that require complete training samples with an incomplete-training approach whose flexibility is constrained.
2 Related Work
- Knowledge-distillation, shared-latent-space, and generative methods all require every training sample to be modality-complete, so they fail to address missing modalities during training.Source: §2 · p. 4
- Konwer et al. use both complete and incomplete training data, but their meta-learning framework does not permit varying proportions of full-modality data.The method can become ineffective when the full-modality ratio is high.Source: §2 · p. 4
- When missing-modality data greatly outnumber full-modality data (|Df| ≪ |Dm|), the cited method’s performance declines significantly.Source: §2 · p. 5
3 Methodology
The method combines partial-modality reconstruction, teacher–student co-distillation, and CLIP-conditioned personalization to train a universal segmenter for missing modalities during training and inference.
3 Methodology
- A multimodal MAE is pretrained on modality-dropped, patch-masked inputs, then its weights initialize the segmentation model used for all-stage missing-modality training and inference.At inference, the segmentation model with the CLIP-driven hyper-network takes modality-incomplete inputs.Source: §3 · p. 5, §3.1 · p. 5
3.1 Modality Reconstruction via Distribution Approximation
- Segmentation-loss guidance supplies auxiliary supervision for reconstructed modalities without ground truth and steers pretraining toward tumor-aware multimodal representations.The method aims to avoid over-reconstructing irrelevant background voxels.Source: §3.1 · pp. 6–7
3.2 Data-model Co-distillation with Reconstructed Modality
- A learnable recovery module fills missing training modalities for a frozen teacher, while the incomplete-input student receives distillation from the teacher’s representations.A data-refining loss also updates the recovery process while the teacher remains frozen.Source: §3.2 · pp. 7–8
3.3 CLIP-driven Hyper-network for Model Personalization
- CLIP embeddings of available modalities and pooled visual features condition a hyper-network that personalizes prediction parameters; an all-ones code locks the hyper-teacher to full-modality mode.The CLIP text encoder can model semantic correlations between MRI modalities, unlike the implicit orthogonality of binary modality codes.Source: §3.3 · pp. 8–9
3.4 Overall Objective for Fine-tuning
- The overall objective jointly optimizes segmentation, data refining, and knowledge distillation losses, with α and β set to 1.0 and 0.1, respectively.The segmentation loss uses random modality dropout and the modality code for the CLIP-driven hyper-network.Source: §3.4 · p. 9
4 Experiment
Experiments evaluate the method on BraTS2018 and BraTS2020 across varying full-modality training ratios and missing-modality inference scenarios. Comparisons and ablations examine its performance and the contributions of its components.
4.1 Datasets and Metrics
- BraTS2018 and BraTS2020 provide 285 and 369 cases, respectively, with four MRI sequences and three tumor-region labels; evaluation uses DSC and HD95.The datasets use the splits 199:29:57 and 219:50:100 for training, validation, and testing, respectively.Source: §4.1 · pp. 9–10
4.2 Implementation Details
- Training uses a 3D U-Net backbone, randomly selected full-modality samples, and remaining samples with one to three modalities discarded.The reported setup uses two RTX A5000 GPUs, 128 × 128 × 128 crops, and retains only the segmentation model for inference across 15 scenarios.Source: §4.2 · pp. 10–11
4.3 Comparison with State-of-the-arts
- The study compares RFNet, M3AE, and Meta-learning using a shared 3D U-Net backbone across 1%, 10%, and 50% full-modality training ratios.Inference is evaluated under 15 modality-input scenarios; the authors note that Meta-learning is implemented from its paper because its code is unavailable.Source: §4.3 · p. 11
- On BraTS2018 and BraTS2020, the proposed approach outperforms state-of-the-art methods under any full-modality (FM) training ratio; comparisons report DSC across 15 input-modality scenarios.Source: Table 1 · p. 10, Table 2 · p. 11, §4.3 · p. 12
- The method trained with 1% full-modality data achieves results comparable to other methods, and surpasses them with 100% full-modality training.This comparison is on BraTS2018 against methods trained with 100% full-modality data.Source: §4.3 · p. 12, Table 3 · p. 12
4.4 Ablation Studies
- Ablations report gains over baseline of 1.43% from modality-incomplete pre-training, 2.31% with the CLIP-driven hyper-network, and 2.49% with data-model co-distillation.The authors attribute the pre-training gain to auxiliary supervision from distribution approximation.Source: §4.4 · p. 13, Table 4 · p. 13
- The co-distillation analysis finds that transferring recovered-modality knowledge helps when missing modalities are synthesized, refined during distillation, and supplied through a full-modality hyper-teacher.The hyper-network ablation reports that frozen CLIP text embeddings improve all three classes, raising average DSC from 73.82% to 74.32%; reconstruction visualizations also show clearer tumor areas and more recovered anatomical detail with auxiliary supervision.Source: Table 5 · p. 13, §4.4 · pp. 14–15
5 Conclusion
The paper presents a universal model for missing modalities during both training and inference. It combines distribution-approximation pre-training, data-model co-distillation, and a CLIP-driven hyper-network, and reports superior performance on two brain tumor segmentation benchmarks.
5 Conclusion
- The universal model addresses missing modalities in both training and inference, with superior performance reported across two brain tumor segmentation benchmarks and varying missing ratios.The conclusion describes the setting as missing modalities across both phases.Source: §5 · p. 15
- Distribution approximation pre-trains the model for modality reconstruction, after which data-model co-distillation uses knowledge from the reconstructed full modality.The conclusion describes the CLIP-driven hyper-network as a means to alleviate distribution heterogeneity.Source: §5 · p. 15
- The conclusion presents robust modality reconstruction and model personalization as components of the universal model.The model is evaluated on two brain tumor segmentation benchmarks.Source: §5 · p. 15