Source-linked AI summary
Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling
Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, Gustavo Carneiro
TL;DR
Missing modalities challenge multi-modal systems because available methods may cover only evaluation or specific missing-modality settings and tasks. ShaSpec learns shared and specific features using distribution alignment, domain classification, and residual fusion, and supports classification and segmentation. It achieves state-of-the-art performance across computer-vision classification and medical-imaging segmentation benchmarks, with BraTS2018 gains exceeding 3%, 5%, and 3% for enhancing tumour, tumour core, and whole tumour.
Problem
Multi-modal methods often require complete modalities during training and evaluation, while existing missing-modality approaches may be limited to specific settings or tasks.
Method
ShaSpec learns shared and specific features through distribution alignment and domain classification, then uses residual feature fusion for prediction.
Results
ShaSpec achieves state-of-the-art performance across computer-vision classification and medical-imaging segmentation benchmarks, including BraTS2018 gains exceeding 3%, 5%, and 3% for three tumour metrics.
Takeaways & Limitations
ShaSpec provides one simple approach for missing-modality training and evaluation across dedicated and non-dedicated settings and classification and segmentation tasks.
Takeaways & Limitations
Robust-Mseg performance was unstable across missing-modality scenarios, and its implementation could not handle missing data during training.
Abstract
from arXiv · showhide
The missing modality issue is critical but non-trivial to be solved by multi-modal models. Current methods aiming to handle the missing modality problem in multi-modal tasks, either deal with missing modalities only during evaluation or train separate models to handle specific missing modality settings. In addition, these models are designed for specific tasks, so for example, classification models are not easily adapted to segmentation tasks and vice versa. In this paper, we propose the Shared-Specific Feature Modelling (ShaSpec) method that is considerably simpler and more effective than competing approaches that address the issues above. ShaSpec is designed to take advantage of all available input modalities during training and evaluation by learning shared and specific features to better represent the input data. This is achieved from a strategy that relies on auxiliary tasks based on distribution alignment and domain classification, in addition to a residual feature fusion procedure. Also, the design simplicity of ShaSpec enables its easy adaptation to multiple tasks, such as classification and segmentation. Experiments are conducted on both medical image segmentation and computer vision classification, with results indicating that ShaSpec outperforms competing methods by a large margin. For instance, on BraTS2018, ShaSpec improves the SOTA by more than 3% for enhancing tumour, 5% for tumour core and 3% for whole tumour. The code repository address is https://github.com/billhhh/ShaSpec/.
1. Introduction
ShaSpec addresses missing modalities during both training and evaluation while supporting dedicated and non-dedicated training. Its simple shared-specific feature design is adaptable to classification and segmentation, and it achieves state-of-the-art results across benchmarks.
- Missing modalities limit real-world multi-modal learning because existing methods usually require complete modality sets during training and evaluation.
- ShaSpec handles missing modalities in training and testing, including dedicated and non-dedicated training settings.Non-dedicated training uses one model for different missing-modality combinations.
- ShaSpec models and fuses shared and specific features to address missing modalities across training and evaluation.
- ShaSpec is designed for easy adaptation to both classification and segmentation tasks.
- More than 3% improvement was achieved for enhancing tumour, 5% for tumour core, and 3% for whole tumour on BraTS2018 versus competing approaches.
2. Related Work
Related work addresses missing modalities through reconstruction, unified representations, and specialized architectures, but often remains task-specific or complex. ShaSpec instead learns shared and specific features with auxiliary objectives and supports multiple missing-modality settings and tasks.
- Multi-modal Learning Models: Real-world multi-modal systems may lack subsets of modalities during both training and evaluation.
- Multi-modal Learning Models: ShaSpec learns shared features through multi-modal distribution alignment and specific features through modality classification rather than reconstruction.The method does not use a generative model for image reconstruction, simplifying training.
- Addressing Missing Modality in Multi-modal Learning: Existing missing-modality methods commonly reconstruct missing modalities or features, or introduce sophisticated architectures.
- Addressing Missing Modality in Multi-modal Learning: ShaSpec uses shared and specific encoders followed by residual fusion with a linear projection to produce fused features for decoding.
- Addressing Missing Modality in Multi-modal Learning: Prior approaches are often developed specifically for either computer-vision classification or medical-image segmentation.
- Addressing Missing Modality in Multi-modal Learning: ShaSpec can handle missing modalities in training and testing, dedicated and non-dedicated training, classification, and segmentation.
3. Methodology
ShaSpec models shared and modality-specific features, fuses them residually, and uses auxiliary objectives to support missing-modality training and evaluation across classification and segmentation.
- Overall Architecture: The architecture uses a shared encoder, modality-specific encoders, a projection layer, and a decoder for segmentation outputs.For classification, fused features are instead passed to fully connected layers.
- Evaluation with Full and Missing Modalities: When a modality is missing, ShaSpec generates its fused feature from the available modalities and adapts the process for multiple missing modalities.The output is then produced from the resulting embeddings.
- Overall Architecture: Residual fusion concatenates shared and specific features, projects them, and adds the projection output to the shared features.This produces a semantically rich modality embedding before decoding.
- Overall Architecture: Shared features capture modality-consistent information, while specific features represent modality-heterogeneous information.The distinction supports representation learning when modalities are incomplete.
- Auxiliary Objectives: Training jointly optimizes the main task with domain classification for specific features and distribution alignment for shared features.The main task may use cross-entropy for classification or Dice loss for segmentation.
- Auxiliary Objectives: For missing-modality training, losses for the missing modality's features are omitted from the auxiliary objectives, allowing the framework to handle missing modalities during training and evaluation.This applies to the framework's missing-modality optimization procedure.
4. Experiments
Experiments evaluate ShaSpec on BraTS2018 segmentation and Audiovision-MNIST classification under missing-modality settings, including dedicated and non-dedicated training. ShaSpec generally outperforms competing methods, with strong gains on both benchmarks.
- Datasets and setup: BraTS2018 evaluates missing-modality brain-tumour segmentation using Dice scores for enhancing tumour, tumour core, and whole tumour.The experiments include dedicated and non-dedicated training settings.
- Datasets and setup: Audiovision-MNIST evaluates missing-modality classification using image and audio data, with accuracy measured across available audio rates.The dataset contains 1,500 paired audio-image samples and uses a 70%/30% training-evaluation split.
- Segmentation results: ShaSpec achieves the best or second-best result in 45 of 48 non-dedicated BraTS2018 comparisons across missing-modality combinations and tumour types.The comparison includes U-HeMIS, U-HVED, Robust-MSeg, and mmFormer.
- Segmentation results: 8.47%, 6.63%, and 5.92% are ShaSpec’s gains over mmFormer for enhancing tumour, tumour core, and whole tumour when only T1 is available.With T1, T1c, and T2 available, the corresponding gains are 1.62%, 3.79%, and 3.76%.
- Classification results: 93.33% accuracy is achieved by ShaSpec at a 5% available audio rate, compared with 92.89% for the second-best model on Audiovision-MNIST.ShaSpec remains the best-performing model as the available audio rate increases.
- Classification results: ShaSpec remains superior to SMIL across all tested image and audio availability settings in the two-modality evaluation.The tested availability rates are 15%, 20%, 50%, 70%, and 100%.
- Analyses: L1 distribution-alignment loss performs best among the tested DAO objectives, while α = 0.1 and β = 0.02 generally produce the best results.Setting α = β = 1 causes a significant performance drop, whereas small auxiliary-loss weights avoid interfering with main-task optimisation.
5. Conclusion
The conclusion presents ShaSpec as a simple method for missing-modal multi-modal learning across training and evaluation, dedicated and non-dedicated settings, and classification and segmentation. It reports strong performance across tasks and identifies broader testing on other tasks and datasets as future work.
- ShaSpec addresses missing modalities in training and testing for dedicated and non-dedicated training across segmentation and classification tasks.
- The authors attribute ShaSpec’s performance to semantically rich shared and specific features related to the main task.A t-SNE visualisation is used to further examine the shared and specific feature spaces.
- Future work will test ShaSpec on other tasks, such as regression, and on additional datasets.