Source-linked AI summary
Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation
Yingda Xia, Dong Yang, Zhiding Yu, Fengze Liu, Jinzheng Cai, Lequan Yu, Zhuotun Zhu, Daguang Xu, Alan Yuille, Holger Roth
TL;DR
Deep medical image segmentation is limited by the cost of obtaining expert annotations, while semi-supervised learning and domain adaptation can use unlabeled data. UMCT addresses these settings with uncertainty-aware multi-view co-training for volumetric images, and experiments report strong performance across pancreas, multi-organ, and adaptation tasks, including adaptation without source data.
Problem
Expert annotation is costly for volumetric medical image segmentation, motivating methods that use unlabeled data across semi-supervised and domain-adaptation settings.
Method
UMCT generates rotated and permuted 3D views, trains a network on each view, and uses uncertainty-weighted multi-view consistency to produce pseudo-label supervision.
Results
UMCT outperformed other approaches on NIH pancreas segmentation, improved multi-organ segmentation, and achieved good unsupervised adaptation performance even without source-domain data.
Takeaways & Limitations
The framework supports unified use of unlabeled data for semi-supervised segmentation, domain adaptation, and adaptation when source data is inaccessible.
Takeaways & Limitations
The evaluated domain shift concerns CT datasets differing in sources and pathological or healthy organ status, rather than cross-modality changes.
Abstract
from arXiv · showhide
Although having achieved great success in medical image segmentation, deep learning-based approaches usually require large amounts of well-annotated data, which can be extremely expensive in the field of medical image analysis. Unlabeled data, on the other hand, is much easier to acquire. Semi-supervised learning and unsupervised domain adaptation both take the advantage of unlabeled data, and they are closely related to each other. In this paper, we propose uncertainty-aware multi-view co-training (UMCT), a unified framework that addresses these two tasks for volumetric medical image segmentation. Our framework is capable of efficiently utilizing unlabeled data for better performance. We firstly rotate and permute the 3D volumes into multiple views and train a 3D deep network on each view. We then apply co-training by enforcing multi-view consistency on unlabeled data, where an uncertainty estimation of each view is utilized to achieve accurate labeling. Experiments on the NIH pancreas segmentation dataset and a multi-organ segmentation dataset show state-of-the-art performance of the proposed framework on semi-supervised medical image segmentation. Under unsupervised domain adaptation settings, we validate the effectiveness of this work by adapting our multi-organ segmentation model to two pathological organs from the Medical Segmentation Decathlon Datasets. Additionally, we show that our UMCT-DA model can even effectively handle the challenging situation where labeled source data is inaccessible, demonstrating strong potentials for real-world applications.
1 Introduction
UMCT targets the high annotation cost of volumetric medical image segmentation by using unlabeled data for semi-supervised learning and unsupervised domain adaptation. It combines transformed 3D views, co-training, and uncertainty-weighted pseudo-label fusion.
- Motivation: Medical volumetric segmentation requires expert-annotated labels, making large-scale labeled datasets costly to acquire.The task labels tissues and organs in CTs and MRIs and supports biomedical image analysis and surgical planning.
- Problem setting: SSL assumes labeled and unlabeled data from the same distribution, whereas UDA assumes a larger distribution shift between them.Both settings use labeled and unlabeled subsets, but their distributional assumptions differ.
- Method: UMCT creates multiple 3D views through rotations and permutations, then trains a deep network on each view.The transformed views are processed by networks with asymmetrical 3D kernels.
- Method: On unlabeled data, UMCT enforces consistency across view predictions and uses uncertainty-derived confidence scores to weight pseudo-label fusion.Predictions are inverse-transformed before confidence-weighted labels supervise the networks.
- Evaluation: The framework extends from conference-level pancreas experiments to multi-organ segmentation and unsupervised adaptation across datasets.The paper also considers adaptation when source-domain data is inaccessible.
2 Related Work
Prior work applies semi-supervised strategies such as self-training, co-training, and graph-based methods to medical image analysis and domain adaptation. Deep co-training extends multi-view co-training to 2D networks using adversarial examples.
- Semi-supervised learning: Semi-supervised deep learning methods often add regularization terms that make networks resistant to specific noise.These approaches have been applied successfully to image recognition.
- Medical image analysis: Medical semi-supervised methods include self-training with teacher-student models, co-training with handcrafted features, and graph-based optimization.A deep self-training framework also uses conditional random field refinements.
- Multi-view co-training: Deep co-training extends co-training to 2D deep networks and multiple views by using adversarial examples to encourage view differences.The approach is presented as a way to boost performance through differentiated views.
3 Problem Definitions
The paper distinguishes SSL, UDA, and UDA without source data by their data distributions and source-data availability. It seeks one approach capable of addressing all three volumetric segmentation settings.
- Semi-supervised learning: In SSL, labeled and unlabeled samples form one available dataset, with voxel-wise 3D masks for labeled volumetric images.The labeled set is S, the unlabeled set is U, and D = S ∪ U.
- Unsupervised domain adaptation: UDA uses labeled source data and unlabeled target data whose distributions differ while the segmentation task remains identical.Domain gaps may reflect modalities, imaging protocols, image quality, or patient populations.
- Relationship among settings: SSL methods can also address UDA because the distinction between the settings lies in data distributions.The paper frames the two tasks as closely related under this difference.
- UDA without source domain: UDA without source data assumes a pretrained source model and unlabeled target data, with the original source-domain data unavailable.This setting is identified as an important and relatively underinvestigated medical-imaging challenge.
- Unified objective: The proposed unified approach is intended to solve SSL, UDA, and UDA without source-domain data.This goal is stated as the paper’s overarching problem formulation.
4 Uncertainty-aware Multi-view Co-training
UMCT uses multiple transformed views of 3D medical volumes and uncertainty-aware co-training to learn from unlabeled data for semi-supervised segmentation and domain adaptation. The framework aligns view predictions, fuses them into pseudo labels, and can adapt a source-pretrained model even when source data is unavailable.
- Multi-view co-training: Co-training enforces consensus across view predictions on unlabeled volumes by using predictions from each view as pseudo labels for the others.The combined objective adds a weighted co-training loss to the supervised segmentation loss.
- 3D multi-view generation: UMCT generates multiple 3D views through rotations or coordinate permutations and trains a separate network on each view.Predictions are mapped back through inverse transformations before comparison.
- Uncertainty estimation: Epistemic uncertainty is estimated from voxel-wise variance across predictions produced by different dropout configurations of a Bayesian deep-network branch.The method uses sampled outputs from K random dropout configurations to estimate uncertainty.
- Uncertainty-weighted label fusion: Confidence scores derived from inverse uncertainty weight predictions from different views when assigning pseudo labels to each view.The pseudo label for one view is computed from predictions from all other views.
- Domain adaptation: UMCT applies the same unlabeled-data co-training framework to UDA, where labeled source and unlabeled target domains share the segmentation task.For source-data-inaccessible UDA, UMCT-DA fine-tunes a source-pretrained model by iteratively refining pseudo labels.
- Evaluation and inference: Multi-view networks improve performance in both single-view and multi-view ensemble settings, while sliding-window testing produces final predictions at the original image resolution.Reported testing time ranges from 1 minute to 5 minutes per case depending on input-volume size.
5 Experiments
Experiments evaluate UMCT across semi-supervised pancreas and multi-organ segmentation, then test domain adaptation to pathological MSD organs, including settings without source data. The approach improves segmentation performance, data utilization, and adaptation results across these settings.
- Data utilization efficiency: DSC 80.35% with 20% labeled data exceeded DSC 78.95% from 60% supervised training, saving approximately 70% of labeling effort.When labeled data exceeds 80%, simple supervised training suffices according to the reported utilization analysis.
- Ablation studies: Uncertainty-weighted label fusion improved three-view and six-view co-training performance, while performance also became better with more views.ULF is not applicable to two-view training because only one prediction is available as a pseudo-label for the other view.
- Experimental settings: UMCT was evaluated on NIH pancreas and multi-organ datasets under multiple labeled-data ratios, followed by adaptation to MSD liver and pancreas domains.The experiments include semi-supervised learning, unsupervised domain adaptation, and adaptation without accessible source-domain data.
- Multi-organ segmentation: UMCT improved consistently over almost every organ under each labeled-unlabeled ratio in the multi-organ experiments.The authors report that these results demonstrate the approach can handle complex multi-organ settings.
- Unsupervised domain adaptation: Standard UDA improved DSC by 1.12% on liver and 4.70% on pancreas over source-only testing, and UMCT-DA significantly outperformed the reported baselines.Without source-domain data, UMCT-DA achieved comparable results using only the co-training loss on unlabeled target data.
6 Discussions
The framework demonstrated robust multi-view co-training across fully supervised and domain adaptation settings. Its evaluated domain shift was limited to CT-source and pathological-status differences, leaving larger shifts for future work.
- 6 Discussions: 3-view co-training ranked NVDLMED 3rd in the first phase and 2nd in final validation of the Medical Segmentation Decathlon Challenge.The framework was applied simultaneously to ten segmentation tasks using axial, coronal, and sagittal views.
- 6 Discussions: The fixed multi-view framework showed effectiveness and robustness without the winning team’s heavy model selection and cross-validation ensemble.The comparison concerns fully supervised challenge performance, although the approach was not originally targeted at improving fully supervised training.
- 6 Discussions: Direct TCIA-to-MSD transfer reduced average Dice from 95% to 92% for liver and from 81% to 70% for pancreas.The paper characterizes this CT-dataset and pathological/healthy-status shift as smaller than cross-modality shifts but still unacceptable for potential clinical applications.
- 6 Discussions: The study does not investigate larger domain shifts such as CT-to-MRI adaptation, contrast changes, or resolution differences.These settings remain identified as active research topics.
7 Summary & Conclusion
UMCT combines 3D multi-view analysis, uncertainty estimation, and consistency enforcement to use unlabeled data for semi-supervised segmentation and domain adaptation. It improved segmentation across pancreas, multi-organ, and pathological-organ settings, including adaptation without source-domain data, while fixed views and other domain shifts remain future directions.
- 7 Summary & Conclusion: UMCT extends dual-view and deep co-training to 3D volumes by analyzing multiple viewpoints, estimating uncertainty, and enforcing consistency on unlabeled data.The framework is aimed at semi-supervised learning and domain adaptation.
- 7 Summary & Conclusion: UMCT outperformed other approaches by a large margin on the NIH pancreas dataset and significantly improved each organ on multi-organ datasets.These findings concern the paper’s semi-supervised segmentation evaluations.
- 7 Summary & Conclusion: UMCT-DA achieved good performance adapting multi-organ segmentation to pathological MSD pancreas and liver, even without source-domain data.The result supports the model’s applicability to the source-inaccessible domain adaptation setting.
- 7 Summary & Conclusion: Future work will examine additional and random views, and domain shifts involving modality changes and contrast variants.The current co-training views are fixed and predefined.