Source-linked AI summary
When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation
Yuchen Pei, Xiaoyu Hu, Yixiong Zou, Dingwen Hu, Hui Chu, Yutao Ma, Shijun Qiu, Gang Li
TL;DR
Multi-modal segmentation can degrade when spatially aligned inputs differ in resolution because corrupted features interfere with modality selection. CoReFuse-Med suppresses corruption during feature transmission and rebalances high-level modality contributions, improving accuracy and stability across the evaluated datasets and settings.
Problem
Spatially aligned modalities with resolution-quality mismatch can make fusion underperform single-modality baselines because degraded modalities affect predictions despite weak training updates.
Method
CoReFuse-Med suppresses resampling-induced feature corruption before fusion and calibrates modality contributions during deeper semantic reasoning.
Results
CoReFuse-Med improves accuracy and stability across EPVS, BraTS, and WMH under modality-quality discrepancies, with auxiliary validation under noise.
Takeaways & Limitations
Reliable modality handling is important when resolution degradation severely reduces task-relevant evidence.
Takeaways & Limitations
The study primarily addresses spatially aligned resolution degradation rather than complete modality absence, while noise is only an auxiliary setting.
Abstract
from arXiv · showhide
Multi-modal medical image segmentation leverages complementary diagnostic information, yet fusion can underperform single-modality baselines when spatially aligned inputs differ in quality. Here, "corruption" primarily denotes resolution-induced degradation rather than misalignment or complete modality absence, while synthetic noise is evaluated only as an auxiliary setting. We identify a critical optimization-inference inconsistency: degraded modalities can receive weak training updates yet substantially affect predictions, indicating active interference with fusion. We attribute this failure to resampling-induced feature corruption and optimization bias, where noisy features propagate through skip connections and encourage unreliable modality selection. We therefore propose CoReFuse-Med, a Corruption-aware Rebalanced Fusion framework that suppresses corruption during feature transmission and rebalances modality contributions during high-level fusion. Experiments on EPVS, BraTS, and WMH, including multiple Z-axis slice-retention ratios and an auxiliary noise test, demonstrate improved accuracy and robustness under modality-quality discrepancies. Our code is available at https://github.com/lrever/CoReFuse.
1 Introduction
Multi-modal fusion can fail when aligned modalities differ in resolution and quality, because degraded features interfere with optimization and prediction. CoReFuse-Med addresses this by suppressing corruption before fusion and rebalancing modality contributions during high-level reasoning.
- 1 Introduction: Aligned modality-quality mismatch can make multi-modal fusion underperform single-modality baselines despite complementary imaging information.The identified failure concerns resolution-induced degradation rather than spatial misalignment or complete modality absence.
- 1 Introduction: Degraded modalities may receive weak gradient updates yet substantially influence predictions, revealing an optimization–inference inconsistency.This indicates that such modalities are neither simply ignored nor reliably exploited during fusion.
- 1 Introduction: Resolution discrepancies introduce resampling-induced noise that propagates through skip connections, while optimization favors cleaner modalities and suppresses ambiguous informative signals.Their interaction produces unstable cross-modal reasoning and biased modality selection.
- 1 Introduction: CoReFuse-Med separates fusion into corruption mitigation before fusion and modality-contribution rebalancing during high-level reasoning.The design targets reliable representations and more effective use of complementary information under severe resolution discrepancies.
- 1 Introduction: Experiments across EPVS, BraTS, and WMH with multiple corruption settings show that CoReFuse-Med achieves robust performance under modality-quality discrepancies.The stated contribution includes both identifying the failure mode and demonstrating consistent robustness across datasets and settings.
2 Related Work
Prior multi-modal fusion work addresses feature interaction, modality imbalance, and explicit heterogeneity such as misalignment or missing inputs. These approaches generally assume that available modality features are reliable, leaving resolution-induced corruption insufficiently addressed.
- 2 Related Work: Existing fusion methods enhance feature interaction or use resolution-robust operators but generally assume clean, compatible inputs.Under cross-resolution conditions, such methods can amplify noise-contaminated signals rather than suppressing them.
- 2 Related Work: Modality-imbalance methods address unequal contributions through prototype guidance, self-distillation, augmentation, or optimization scheduling.These approaches typically treat discrepancies as data-availability or optimization issues.
- 2 Related Work: The paper identifies a gap in methods that account for unreliable features created by resolution discrepancies before modality rebalancing.Its distinction is between under-optimized modalities and modalities whose representations are themselves corrupted.
3 Problem Analysis
The analysis shows that resolution-induced corruption changes shallow feature quality and propagates through skip connections, while gradient contributions can diverge from actual predictive impact. These findings motivate a two-stage approach that limits corruption before decoding and calibrates modality contributions in deeper semantic spaces.
- 3.1 An Unexpected Finding: Fusion Failure under Aligned Quality Mismatch: Fusing all three EPVS modalities yields DSC 0.695, 1.14% lower than T1 alone at 0.703 under severe resolution mismatch.The result demonstrates that adding aligned modalities is not automatically beneficial when their quality differs substantially.
- 3.2 Hidden Noise in Cross-resolution Multi-modal Data: T1 has 3.4× higher F-CNR than FLAIR, while low-resolution FLAIR and T2 show elevated background variation associated with resampling artifacts.The F-CNR analysis characterizes separability between lesion signals and background noise in latent features.
- 3.2 Hidden Noise in Cross-resolution Multi-modal Data: Resampling-induced corruption is structurally introduced at early stages and propagates through skip connections into deeper layers.This corruption affects feature representations rather than behaving as purely stochastic noise.
- 3.3 Fusion Network Takes a Greedy Shortcut: In 3D U-Net, T1 contributes 77.4% of gradients, yet masking T1 or FLAIR each causes a DSC drop of 70.2%, exposing optimization–prediction misalignment.Swin-UNETR shows the issue beyond 3D U-Net, with masking drops of 69.6% for T1 versus 35.0% for FLAIR despite more balanced gradients.
- 3.3 Fusion Network Takes a Greedy Shortcut: BraTS retains useful evidence across modalities, whereas severely degraded EPVS T2 and FLAIR retain limited incremental utility; CoReFuse-Med therefore prevents contamination without reconstructing absent information.The contrast shows that fusion is constrained by both the fusion mechanism and the available task-relevant evidence.
- 3.3 Fusion Network Takes a Greedy Shortcut: The proposed solution first limits corruption before shared decoding and then calibrates modality contributions in deeper semantic spaces.IMSH addresses shallow corruption, while MBCF calibrates deeper modality interactions and reduces dominance by cleaner modalities.
4 Methodology
CoReFuse-Med disentangles multimodal fusion into corruption suppression in shallow layers and modality-bias correction in deeper semantic spaces. It combines the lightweight LACE backbone with IMSH and MBCF to support reliable fusion under resolution discrepancies.
- CoReFuse-Med separates fusion into shallow feature purification and deep modality-bias correction to address artifacts and semantic conflicts.IMSH isolates shared structures from resampling artifacts, while MBCF calibrates modality interactions during deeper reasoning.
- 4.2 Lightweight Axial-Context Encoder: LACE uses DoubleConv for local details and factorized axial strip convolutions for direction-aware context in anisotropic MRI volumes.Selective channel attention balances local and axial representations, preserving structural continuity with an ultralightweight profile.
- 4.3 Inter-Modal Spatial-Scale Harmonization: IMSH mitigates cross-modal contamination by separating structurally consistent bases from detail-sensitive residuals instead of entangling both contexts during fusion.The design supports stable shallow fusion with a lightweight computational profile.
- 4.3 Inter-Modal Spatial-Scale Harmonization: IMSH decomposes each modality into shared macroscopic bases and detail residuals, then competes over bases and gates residuals to suppress resampling noise.Spatial filtering and shared gating purify shallow representations before deeper semantic interaction while preserving reliable boundary structures.
- 4.4 Modality Bias Calibration Fusion: MBCF applies joint channel calibration, spatial gating, and symmetric cross-modal attention to skip-connected deep features, reducing dominance by high-SNR modalities.Its shared consensus query enables balanced aggregation without assigning a single modality the query role.
5 Experiments
Experiments evaluate CoReFuse-Med across clinical anisotropy, simulated slice degradation, information loss, and auxiliary noise, comparing general-purpose and specialized fusion models. Across EPVS, BraTS, and WMH, the framework improves segmentation quality, boundary preservation, robustness across degradation levels, and computational efficiency.
- Quantitative Results and Analysis: Specialized MMFormer and HNoSegXS degrade markedly under severe resolution mismatch, with EPVS DSC values below 0.55 and larger HD95.CoReFuse-Med records HD95 values of 7.35 on EPVS and 4.43 on BraTS.
- Quantitative Results and Analysis: On BraTS, CoReFuse-Med reaches DSC 0.8528 and HD95 4.43±3.07, improving overlap and boundary localization over MedNeXt under modality degradation.MedNeXt has HD95 7.59±11.29, indicating larger boundary variance across cases.
- Quantitative Results and Analysis: Modality calibration is task-conditioned: CoReFuse-Med suppresses degraded EPVS T2/FLAIR pathways but preserves more balanced contributions on BraTS, where modalities retain useful tumor cues.In EPVS, 3D U-Net loses 70.2% DSC after masking FLAIR despite a 13.1% gradient contribution.
- Quantitative Results and Analysis: On WMH, CoReFuse-Med obtains HD95 6.22 and IAVD 0.171, compared with 8.59 and 0.216 for MMFormer and 6.30 and 0.193 for sysu_media.The results demonstrate boundary preservation under information loss.
- Quantitative Results and Analysis: CoReFuse-Med reduces parameters by 95.7% and GFLOPs by 69.1% relative to Swin-UNETR while matching sysu_media’s F1 score of 0.76 with one model.The model uses 2.665M parameters and 102.397 GFLOPs.
- Quantitative Results and Analysis: CoReFuse-Med achieves the best DSC and HD95 at both 20% and 30% Z-axis slice retention, demonstrating robustness across degradation levels.Each model was retrained separately for the corresponding retention ratio; synthetic noise was evaluated only as an auxiliary setting.
- Qualitative Results: Qualitatively, Ours (Med) better preserves irregular edema boundaries and internal tumor structures, while competing models oversmooth, fragment, or miss degraded regions.The improvement is most visible around ambiguous transitions and small internal structures vulnerable to resampling artifacts.
- Ablation Study: On EPVS, the complete ablated model achieves DSC 0.7322 and HD95 7.35, while IMSH and MBCF provide larger recall gains through artifact suppression and modality calibration.Combining LACE with either fusion module further reduces boundary errors.
6 Limitations and Future Work
The study focuses on spatially aligned resolution degradation rather than complete modality absence, with noise evaluated only as an auxiliary setting. Future work will address dynamically missing modalities and a broader range of acquisition artifacts.
- Limitations and Future Work: The study primarily addresses spatially aligned resolution degradation, not complete modality absence, and evaluates noise only as an auxiliary setting.The in-house EPVS dataset also cannot be released because of clinical privacy constraints.
- Limitations and Future Work: Future work will extend CoReFuse-Med to dynamically missing modalities and a broader range of acquisition artifacts.
7 Conclusion
CoReFuse-Med addresses spatially aligned modality-quality mismatch by suppressing resampling-induced corruption and calibrating modality contributions. Across EPVS, BraTS, and WMH, it improves segmentation accuracy and stability, highlighting the importance of modality reliability when task-relevant information is degraded.
- Conclusion: CoReFuse-Med improves accuracy and stability across EPVS, BraTS, and WMH under spatially aligned resolution degradation.The study also includes auxiliary validation under noise.
- Conclusion: The results highlight modality reliability as important for multi-modal segmentation when task-relevant information is severely degraded.