Source-linked AI summary
ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalities
Yixin Wang, Yang Zhang, Yang Liu, Zihao Lin, Jiang Tian, Cheng Zhong, Zhongchao Shi, Jianping Fan, Zhiqiang He
TL;DR
Brain-tumor segmentation relies on complementary MRI modalities, yet clinical scans may be missing modalities and unified models often perform worse when multiple modalities are absent. ACN couples full- and missing-modality learning with adversarial alignment and modality-mutual-information transfer. On BraTS2018, it outperforms existing methods across missing-modality situations, while requiring dedicated models that may add training cost.
Problem
Clinical MRI scans may be incomplete, and existing unified models perform much worse when more than one modality is missing.
Source: §1 · p. 2Method
ACN trains related models for missing-modality situations and couples multimodal and unimodal paths using adversarial learning and modality-mutual-information knowledge transfer.
Source: §1 · pp. 2–3, §2.1 · p. 3Results
ACN outperforms existing methods on BraTS2018 in all missing-modality situations.
Source: §4 · p. 9Takeaways & Limitations
The authors describe the inference improvement without extra inference cost as valuable for clinical application.
Source: §4 · p. 9Training dedicated models for each missing-modality situation may bring training cost.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Accurate segmentation of brain tumors from magnetic resonance imaging (MRI) is clinically relevant in diagnoses, prognoses and surgery treatment, which requires multiple modalities to provide complementary morphological and physiopathologic information. However, missing modality commonly occurs due to image corruption, artifacts, different acquisition protocols or allergies to certain contrast agents in clinical practice. Though existing efforts demonstrate the possibility of a unified model for all missing situations, most of them perform poorly when more than one modality is missing. In this paper, we propose a novel Adversarial Co-training Network (ACN) to solve this issue, in which a series of independent yet related models are trained dedicated to each missing situation with significantly better results. Specifically, ACN adopts a novel co-training network, which enables a coupled learning process for both full modality and missing modality to supplement each other's domain and feature representations, and more importantly, to recover the `missing' information of absent modalities. Then, two unsupervised modules, i.e., entropy and knowledge adversarial learning modules are proposed to minimize the domain gap while enhancing prediction reliability and encouraging the alignment of latent representations, respectively. We also adapt modality-mutual information knowledge transfer learning to ACN to retain the rich mutual information among modalities. Extensive experiments on BraTS2018 dataset show that our proposed method significantly outperforms all state-of-the-art methods under any missing situation.
1 Introduction
Brain-tumor segmentation benefits from complementary MRI modalities, but clinical scans may be incomplete and existing unified models perform worse when multiple modalities are missing. ACN couples full- and missing-modality learning and combines adversarial learning with modality-mutual-information transfer.
1 Introduction
- Clinical MRI modalities may be unusable or missing because of corruption, artifacts, incorrect machine settings, contrast-agent allergies, or limited scan time.The available scan may therefore include only a subset of the full treatment modalities.Source: §1 · p. 2
- Existing unified models for missing-modality situations perform much worse when more than one modality is missing, motivating dedicated models for each situation.The introduction also notes that single-modality dedicated models can be biased and may miss full-modality domain knowledge and multilevel features.Source: §1 · p. 2
- ACN couples full-modality and missing-modality learning to supplement domain and feature representations and recover information from absent modalities.Its multimodal path derives rich modality information, while its unimodal path generates modality-specific representations.Source: §1 · pp. 2–3
- Entropy adversarial learning bridges the multimodal–unimodal domain gap, knowledge adversarial learning aligns latent representations, and modality-mutual-information transfer recovers relevant knowledge across feature levels.The paper reports that ACN outperforms state-of-the-art methods on a popular brain-tumor dataset.Source: §1 · p. 3
2 Method
ACN couples a multimodal path trained on complete MRI inputs with a unimodal path trained on available modalities, using consistency learning and three complementary modules. These components align predictions and representations, transfer mutual information, and guide the overall co-training procedure.
2.1 ACN: Adversarial Co-training Network
- The multimodal path processes all four MRI modalities while the unimodal path processes available modalities; consistency learning minimizes the KL divergence between their output distributions.The paths share a U-Net architecture and are trained independently and simultaneously.Source: §2.1 · p. 3
2.2 Entropy Adversarial Learning
- Entropy adversarial learning matches the unimodal and multimodal paths’ entropy-map distributions to address their domain gap and prediction-confidence differences.The method describes unimodal predictions as tending toward high entropy and multimodal predictions toward low entropy.Source: §2.2 · pp. 4–5
2.3 Knowledge Adversarial Learning
- Knowledge adversarial learning uses a generator and discriminator to soft-align high-level representations, encouraging the unimodal path to learn missing knowledge from the full-modality path.This is proposed instead of simply minimizing KL divergence between bottleneck features, which could disturb unimodal learning in deep layers.Source: §2.3 · p. 5
2.4 Modality-mutual Information Knowledge Transfer Learning
- Modality-mutual information transfer retains task-relevant information between paired multimodal and unimodal representations from encoder layers.The method estimates mutual information through variational information maximization.Source: §2.4 · pp. 5–6
2.5 Overall Loss and Training Procedure
- The overall objective combines the co-training losses, with trade-off parameters selected using the setting where only T1ce is available.The reported parameter values are λmulti=0.2, λuni=0.8, λ0=0.001, λ1=0.0002, and λ2=0.5.Source: §2.5 · p. 6
3 Experimental Results
On BraTS2018, ACN is evaluated against catch-all and dedicated methods, and its modules are examined in T1ce-only segmentation. The experiments use DSC and HD95 to assess performance on nested tumor subregions.
3.1 Experimental Setup
- The BraTS2018 training set contains 285 multi-contrast scans, and evaluation uses DSC and HD95 on whole tumor, tumor core, and enhancing tumor.Higher DSC and lower HD95 indicate better segmentation; scans are split into training and validation sets at a 2:1 ratio.Source: §3.1 · p. 7
3.2 Results and Analysis
- ACN significantly outperforms HeMIS and U-HVED across all 15 missing-modality combinations, with DSC above 40% versus about 10%–20% in hard cases.The hard cases include missing T1ce, which severely reduces ET and TC scores, and missing Flair, which drops WT scores.Source: §3.2 · p. 7, Table 1 · p. 8
- The KD-Net comparison is restricted to cases with only one available modality, and Table 2 reports DSC for that setting.The supplied results passage identifies the comparison and its scope but does not state its outcome.Source: §3.2 · p. 7, Table 2 · p. 8
- Adding EnA, KnA, and MMI improves the T1ce-only unimodal path; ACN also produces better segmentation with reduced uncertainty than the path without its modules.The ablation evaluates the modules incrementally, with Table 3 reporting DSC and Fig. 3 showing qualitative results and entropy maps.Source: Figure 3 · p. 8, Ablation Study · p. 9, Table 3 · p. 9
4 Conclusion
The conclusion presents ACN as a missing-modality brain-tumor segmentation method that outperforms existing methods across missing situations on BraTS2018. It also notes a trade-off: dedicated models cost more to train, but not more to run at inference.
4 Conclusion
- ACN outperforms existing methods by a considerable margin across all missing situations on the multimodal BraTS2018 dataset.The conclusion describes ACN as a “One stop shop” approach that trains dedicated models for each missing situation.Source: §4 · p. 9
- Two unsupervised adversarial modules align domain and feature distributions between full- and missing-modality paths.A modality-mutual information module transfers knowledge to recover missing information.Source: §4 · p. 9
- Dedicated models may increase training cost, while inference brings large improvement without extra cost.The authors identify this trade-off as valuable for clinical application and suggest generalization to other incomplete-data domains.Source: §4 · p. 9