Source-linked AI summary
Hetero-modal learning and corruption-resistant hetero-modal inference for joint segmentation of white matter hyperintensities and ischaemic stroke lesions in MRI
Jesse Phitidis, William N. Whiteley, Joanna M. Wardlaw, Miguel O. Bernabeu, Francesco Dalla Serra, Maria del C. Valdés Hernández
TL;DR
Joint WMH and ISL segmentation is challenged by visually confounding lesions and datasets with incomplete labels or MRI sequences. The paper evaluates hetero-modal learning and introduces MMAR with corruption augmentation; hetero-modal models generally outperform modality specialists in data-constrained settings, and MMAR addresses severely corrupted inputs.
Problem
Joint WMH and ISL segmentation is difficult to support because datasets may lack labels for both pathologies or complete MRI sequences.
Source: §1 · p. 2Method
The paper evaluates hetero-modal learning and introduces a Multimodal Attention Router with corruption augmentation to route modality features and address degraded inputs.
Source: §1 · p. 4, §3.4.1 · p. 9Results
Hetero-modal models generally outperformed modality-specialist baselines, including under a split with only 10% full-modality training data, and generalized to unseen modality combinations.
Source: §1 · p. 5, §5 · p. 29Takeaways & Limitations
The MMAR-based model can treat severely degraded scan regions as missing and preserve segmentation accuracy under extreme noise, bias fields, and false-present modalities.
Source: §5 · p. 29Comparative results may be sensitive to training-data volume and modality distribution, so extrapolation to other distributions warrants caution.
Source: §5 · p. 29
Abstract
from arXiv · showhide
White matter hyperintensities (WMH) and ischaemic stroke lesions (ISL) are visually confounding, co-occurring pathologies that require large, diverse datasets for robust deep learning segmentation. However, assembling such datasets is hindered by cohort samples that lack reference segmentations from both features and complete sets of MRI structural sequences (i.e., "modalities"). To maximise data utility, we investigate hetero-modal learning using a dataset of 206 vascular disease patients across four MRI sequences (T1-weighted, T2-weighted, fluid-attenuated inversion recovery, and diffusion-weighted imaging) with expert annotations of both WMH and ISL. We demonstrate that hetero-modal learning outperforms models trained using a single imaging modality in scenarios with substantial missing data, including a split where only 10% of the training data contains all four modalities while the remainder is uni-modal, and a split relying on a single shared "anchor" modality with zero overlap between the remaining modalities. Furthermore, models trained under this second split successfully perform inference on unseen combinations of modalities. Yet, while standard hetero-modal networks handle missing sequences, clinical deployment introduces the additional challenge of silent data degradation - where modalities are present but severely corrupted. To bridge this gap, we introduce the Multimodal Attention Router (MMAR) block. Our experiments demonstrate that, when trained with a "corruption augmentation" strategy, the MMAR effectively dynamically weights the encoded features of each modality, maintaining strong performance during hetero-modal inference even in the presence of unflagged catastrophically corrupted modalities.
1. Introduction
Joint segmentation of visually similar WMH and ISL is difficult to support with datasets that provide both lesion labels and complete MRI sequences. This paper evaluates hetero-modal learning for that setting and introduces a router and training strategy for corrupted inputs.
1. Introduction
- WMH and ISL often look similar across MRI sequences, while available datasets may lack labels for both pathologies or complete, pristine sequences.DWI can sometimes distinguish ISL, typically when it is acute, but is not always a reliable identifier.Source: §1 · p. 2
- Hetero-modal learning trains from heterogeneous subsets of available MRI sequence types, unlike standard models that require a fixed input set.The paper focuses on this missing-sequence challenge rather than partially labelled datasets.Source: §1 · p. 3
- Existing datasets do not provide joint WMH and ISL ground-truth labels, limiting evaluation of segmentation when these visually similar lesions co-occur.The paper uses a privately held dataset with expert annotations for both pathologies and four MRI sequences.Source: §1 · p. 4
- The Multimodal Attention Router dynamically routes modality features, and corruption augmentation is used to address inputs that are present but severely degraded.The authors report robustness to catastrophic noise, severe bias fields, and modalities incorrectly identified as present.Source: §1 · pp. 4–5
- The paper establishes a joint hetero-modal WMH–ISL benchmark and reports better performance than fixed-modality baselines when only 10% of training data has all modalities.It also reports inference on unseen modality combinations.Source: §1 · p. 5
2. Experimental setup
The experiments use a 206-patient, four-sequence dataset and construct training splits with complete, mostly uni-modal, or anchor-overlapping sequence availability. Checkpoint selection varies by evaluation setting.
2.1. Dataset and processing
- The dataset contains 206 patients with complete T1w, T2w, FLAIR, and DWI scans and WMH and ISL masks; preprocessing included skull-stripping, bias-field correction, and resampling.The data were partitioned into 103 training, 41 validation, and 62 test subjects.Source: §2.1 · pp. 5–6
2.2. Hetero-modal training splits
- Three training distributions were derived to test whether hetero-modal learning can use incomplete scans rather than discard those subjects.The distributions vary from all subjects having all sequences to limited overlap across sequences.Source: §2.2 · p. 6, Figure 2 · p. 7
2.2.1. Training split 1
- In training split 1, all sequences are available for every training subject, and the split also supplies corruption-augmentation experiments.Figure 2 depicts the split distributions; this setup is the complete-modality condition.Source: §2.2.1 · pp. 6–7, Figure 2 · p. 7
2.2.2. Training split 2
- In split 2, only 10% of subjects have all sequences, while each remaining subject contributes one sequence to test learning with highly uni-modal data.The design tests whether additional disjointed training data can benefit target-sequence prediction beyond what a specialist model can access.Source: §2.2.2 · p. 7, Figure 2 · p. 7
2.2.3. Training split 3
- In split 3, every subject has T1w and one additional sequence, with no overlap among the other sequence types, enabling tests on unseen combinations.T1w is designated the anchor sequence; examples of unseen combinations include FLAIR and DWI.Source: §2.2.3 · p. 7, Figure 2 · p. 7
2.3. Checkpoint selection
- For each inference modality setup, hetero-modal models were evaluated using the checkpoint with the highest validation DSC, yielding 15 checkpoints per training run.Corruption-augmentation tests instead used final checkpoints because corrupted modalities were assumed not to be identifiable in advance.Source: §2.3 · p. 8
3. Methods
The methods section compares two established hetero-modal baselines and modality specialists with the paper’s MMAR architecture, which targets robustness to corrupted modalities.
3. Methods
- The study evaluates a multi-channel U-Net and U-HeMIS as hetero-modal baselines, alongside the proposed Multimodal Attention Router.The MMAR is designed to enhance robustness against corrupted modalities.Source: §3 · p. 8
- The hetero-modal methods are compared against modality-specialist baselines.The section introduces the architectures and training methodologies used in the comparison.Source: §3 · p. 8
- The authors report that code is available in the MultiModal GitHub repository.The repository address is given in the paper.Source: §3 · p. 8
3.1. Modality specialist baselines
Modality-specialist baselines use a fixed modality combination and therefore cannot generally use the full multi-modal training data in incomplete-data splits.
3.1. Modality specialist baselines
- Specialist models are trained and evaluated on a fixed, unique modality combination, limiting their access to the full data in splits 2 and 3 except for T1w in split 3.The T1w specialist is the exception because T1w is present for all subjects in split 3.Source: §3.1 · p. 8
- To reduce overfitting on their simpler tasks, specialist baselines use 16, 32, 64, 128, 256, and 320 channels per resolution level.These configurations total approximately 16.7M parameters.Source: §3.1 · p. 8
- The specialists serve as comparative benchmarks for the hetero-modal approaches.Each specialist is restricted to its own fixed modality combination.Source: §3.1 · p. 8
3.2. Multi-channel U-Net
The multi-channel U-Net is a straightforward hetero-modal baseline that handles missing modalities by zero-imputation.
3.3. Multi-channel U-Net
- Missing modalities are passed as zero-valued channels, requiring the network to handle extreme distribution shifts in its first layers.The U-Net was configured with 91M parameters to match the capacity of the other hetero-modal methods.Source: §3.2 · p. 9
3.3. U-HeMIS
U-HeMIS independently encodes each modality and aggregates available modalities’ features at every U-Net resolution level.
3.3. U-HeMIS
- At each resolution level, U-HeMIS concatenates the mean and variance of available-modality features and sends them to a shared convolutional decoder.Each independent encoder uses 16, 32, 64, 128, 256, and 512 channels across the resolution levels.Source: §3.3 · p. 9
3.4. Multimodal Attention Router (MMAR)
The Multimodal Attention Router (MMAR) replaces statistical feature aggregation with learnable routing across spatial and channel dimensions. It processes available encoded modality features and weights them to produce a fused representation.
3.4.1. Overview
- MMAR dynamically routes features across spatial and channel dimensions, amplifying important modality features and suppressing irrelevant ones, including features from corrupted modalities.The block is motivated by corrupted modalities that are present but not marked as missing, and is inspired by U-HeMIS and SFusion.Source: §3.4.1 · p. 9, §3.4 · p. 9
3.4.2. MMAR block
- MMAR takes encoded features from available modality encoders as input, excluding missing modalities.The model uses a multi-encoder, shared-decoder architecture, with MMAR blocks fusing encoder features before decoding.Source: §3.4.2 · p. 9, Figure 3 · p. 10
- Available features are normalised, patchified, and processed by windowed self-attention blocks before being reshaped and fused using softmax-generated modality attention maps.The attention maps weight the original features through element-wise multiplication to produce a fused representation.Source: Figure 3 · p. 10, §3.4.2 · p. 11
3.5. Implementation and training details
The implementation uses a shared 3D U-Net framework, modality dropout, and standard image augmentations, with additional corruption augmentation in selected experiments. MMAR training also includes deep supervision and router-specific loss and stability adjustments.
3.5.1. 3D U-Net architecture
- The common 3D U-Net backbone uses two Conv-Norm-Act blocks per level across six levels, with residual encoder connections and transposed-convolution decoder upsampling.The architecture is based on nnU-Net.Source: §3.5.1 · p. 12
3.5.2. Modality dropout
- Modality dropout randomly removes some available modalities from each training sample, including samples that already lack modalities.The number dropped is sampled from zero to one fewer than the sample’s available modality count.Source: §3.5.2 · p. 12
3.5.3. Standard data augmentation
- Training applies standard spatial and intensity augmentations, including affine transforms, noise, blur, contrast and gamma adjustment, low-resolution simulation, flipping, and MRI artefacts.The listed artefacts are motion, ghosting, spike, and bias field.Source: §3.5.3 · p. 12
3.5.4. Corruption augmentation
- For corruption experiments, selected scans initially chosen for modality dropout were instead corrupted with 50% probability using severe noise, contrast, blur, and artefact transforms.Corruptions were applied to entire scans or random patches; Figure 4 illustrates the training augmentations across T1w, T2w, FLAIR, and DWI.Source: §3.5.4 · p. 13, Figure 4 · p. 14
3.5.5. Loss functions, optimisation, and schedule
- All models train for 2000 epochs with AdamW, Dice plus cross-entropy loss, and deep supervision over the four highest-resolution decoder outputs.MMAR additionally scales loss by the fraction of modalities available per sample to improve training stability on splits 2 and 3; the authors hypothesise this reduces gradient interference, and report that the scaling harms validation metrics for the multi-channel U-Net and U-HeMIS.Source: §3.5.5 · p. 13, §3.5.6 · p. 15
3.5.6. MMAR-specific training details
- MMAR training adds an auxiliary loss that penalises assigning high attention weights to corrupted modalities.The auxiliary loss is computed over corrupted modalities at each MMAR block level; its inclusion assumes training scans are not already corrupted.Source: §3.5.6 · pp. 13, 15
3.6. Evaluation
Evaluation used the Dice Similarity Coefficient (DSC) and simulated severe test-time corruption, including scans that appear present but contain almost no image data.
3.6.1. Metric
- DSC was used to evaluate segmentation, with Y and predicted masks denoting the ground truth and model output, respectively.Source: §3.6.1 · p. 15
3.6.2. Test time corruption
- Test-time simulations used Gaussian noise (std=10), bias-field artefacts (degree 2; coefficients [0.7, 0.9]), and false-present scans zeroed except for one corner voxel.The corruption parameters differed substantially from the training augmentations to reduce evaluation bias.Source: §3.6.2 · p. 16
- Figure 5 shows corruption transformations on a validation FLAIR scan; the bias-field example is rescaled, with WMH in red and ISL in blue.Source: Figure 5 · p. 16
4. Results and discussion
Hetero-modal models benefit from incomplete-modality training and MMAR performs strongly on unseen modality combinations. With corruption augmentation, MMAR also maintains performance when modalities are severely corrupted but not flagged as missing.
4.1. Modality specialists win when trained on fully multi-modal data
- With all subjects possessing all modalities during training, specialist U-Nets achieved the strongest mean macro-average performance and significantly improved over the multi-channel U-Net; U-HeMIS or MMAR led 6 of 15 inference combinations without significant gains over specialists.Source: §4.1 · p. 16
4.2. Hetero-modal models win when trained on highly uni-modal data
- With highly uni-modal training, hetero-modal models had significantly better mean macro-average results than specialists, though the multi-channel U-Net generally outperformed U-HeMIS and MMAR without significant differences.Across almost all modality setups, hetero-modal approaches beat specialists, with exceptions for MMAR on T1w and U-HeMIS on DWI; the multi-channel U-Net had the strongest ISL DSC, while MMAR had the best WMH DSC.Source: §4.2 · p. 18, Table 2 · p. 19
4.3. Hetero-modal models perform well on unseen modality combinations
- MMAR performed best on 6 of 8 novel modality combinations; its mean macro-average score significantly exceeded U-HeMIS but not the multi-channel U-Net.The highest test-set macro-average DSC was 54.52% for U-HeMIS using T1w and FLAIR; MMAR achieved 54.33% and 54.04% on two novel combinations.Source: Table 3 · p. 20, §4.3 · pp. 20–21
4.4. MMAR outperforms established hetero-modal models on corrupted data
- Under false-present, Gaussian-noise, and severe-bias-field corruption, MMAR consistently outperformed the multi-channel U-Net and U-HeMIS for both lesion types, with minimal degradation from explicitly missing inputs.The router loss reduces gradient flow to encoders processing corrupted images.Source: §4.4 · p. 21, Table 5 · p. 23, Table 6 · p. 24
4.5. MMAR: attention router analysis
- MMAR assigns negligible weight to globally corrupted FLAIR, producing attention maps similar to missing FLAIR, while bias-field corruption prompts spatially selective weighting.Router weighting also reflects the training distribution: FLAIR dominates levels 0–2 in splits 1 and 2, whereas T1w is prioritized in split 3.Source: §4.5 · p. 26, Figure 6 · p. 27, Figure 7 · p. 28
5. Conclusion
Hetero-modal models generally outperform modality-specialist baselines in data-constrained joint WMH and ISL segmentation, while also generalizing to unseen modality combinations. MMAR with corruption augmentation addresses unflagged corrupted inputs, but the evaluation has important data and testing limitations.
5. Conclusion
- Hetero-modal models generally outperform modality specialist baselines in data-constrained joint WMH and ISL segmentation, maximizing the utility of incomplete scans.The study establishes a benchmark for jointly segmenting co-occurring WMH and ISL.Source: §5 · p. 29
- Models trained on disjoint datasets sharing one anchor modality generalized to novel modality combinations, including inference without FLAIR.Source: §5 · p. 29
- With corruption augmentation, MMAR spatially routes features and treats severely degraded regions as missing, preserving segmentation accuracy under extreme corruptions.The reported corruptions include extreme noise, bias fields, and false-present modalities.Source: §5 · p. 29
- Evaluation used a single random data split and one training run per model; results may be sensitive to training-data volume and modality distribution.Source: §5 · p. 29
- Corruption testing used transformed scans because truly corrupted scans were unavailable; evaluation also used DSC alone and foundational baselines, leaving systematic benchmarking for future work.Source: §5 · p. 30