Source-linked AI summary
3D Semi-Supervised Learning with Uncertainty-Aware Multi-View Co-Training
Yingda Xia, Fengze Liu, Dong Yang, Jinzheng Cai, Lequan Yu, Zhuotun Zhu, Daguang Xu, Alan Yuille, Holger Roth
TL;DR
3D medical-image segmentation needs methods that learn effectively when expert-labeled data is scarce but unlabeled data is abundant. UMCT creates complementary transformed views, uses asymmetric 3D kernels and Bayesian uncertainty to generate weighted pseudo labels, and performs strongly across semi-supervised and fully supervised benchmarks.
Problem
3D medical-image annotation is expensive and expert-intensive, motivating semi-supervised learning that can use abundant unlabeled data.
Method
UMCT combines transformed multi-view 3D co-training, asymmetric 3D kernels, and Bayesian uncertainty-weighted label fusion for reliable pseudo labels.
Results
UMCT outperforms other semi-supervised methods on NIH pancreas and LiTS segmentation, while its fully supervised variant reaches state-of-the-art LiTS performance and second place in the MSD challenge.
Takeaways & Limitations
Multi-view consistency and uncertainty-based confidence estimation support 3D semi-supervised training, while the framework also remains effective with full supervision.
Abstract
from arXiv · showhide
While making a tremendous impact in various fields, deep neural networks usually require large amounts of labeled data for training which are expensive to collect in many applications, especially in the medical domain. Unlabeled data, on the other hand, is much more abundant. Semi-supervised learning techniques, such as co-training, could provide a powerful tool to leverage unlabeled data. In this paper, we propose a novel framework, uncertainty-aware multi-view co-training (UMCT), to address semi-supervised learning on 3D data, such as volumetric data from medical imaging. In our work, co-training is achieved by exploiting multi-viewpoint consistency of 3D data. We generate different views by rotating or permuting the 3D data and utilize asymmetrical 3D kernels to encourage diversified features in different sub-networks. In addition, we propose an uncertainty-weighted label fusion mechanism to estimate the reliability of each view's prediction with Bayesian deep learning. As one view requires the supervision from other views in co-training, our self-adaptive approach computes a confidence score for the prediction of each unlabeled sample in order to assign a reliable pseudo label. Thus, our approach can take advantage of unlabeled data during training. We show the effectiveness of our proposed semi-supervised method on several public datasets from medical image segmentation tasks (NIH pancreas & LiTS liver tumor dataset). Meanwhile, a fully-supervised method based on our approach achieved state-of-the-art performances on both the LiTS liver tumor segmentation and the Medical Segmentation Decathlon (MSD) challenge, demonstrating the robustness and value of our framework, even when fully supervised training is feasible.
1. Introduction
UMCT extends co-training to 3D data by creating complementary views through spatial transforms and asymmetric 3D kernels, then weighting cross-view pseudo labels by Bayesian uncertainty. It targets costly 3D medical-image labeling and outperforms other semi-supervised methods, while its fully supervised version achieves strong benchmark results.
- 1. Introduction: UMCT addresses costly 3D medical-image annotation by leveraging unlabeled data for semi-supervised segmentation.The motivation centers on expert-intensive labeling of volumetric medical images.
- 1. Introduction: The framework creates complementary views through rotations or permutations of 3D data and asymmetric 3D kernels across multiple networks.Data-level transforms provide viewpoint differences, while asymmetric kernels encourage complementary feature representations.
- 1. Introduction: Bayesian uncertainty produces view confidence scores that weight predictions during label fusion, yielding pseudo labels for unlabeled-data supervision.The confidence-weighted fusion module is designed to retain more reliable predictions from different views.
- 1. Introduction: The method outperforms other semi-supervised approaches on NIH pancreas and LiTS liver tumor segmentation datasets.The supplied result passage reports a large margin but provides no numerical values.
- 1. Introduction: The fully supervised variant achieves state-of-the-art performance on LiTS liver tumor segmentation and second place in the Medical Segmentation Decathlon challenge.These results were obtained without complicated data augmentation or model ensembles.
2. Related Work
Prior work spans semi-supervised learning, medical-image analysis, uncertainty estimation, and hybrid 2D/3D networks. UMCT builds on these directions by combining multi-view co-training with uncertainty-based confidence estimation for 3D data.
- 2. Related Work: Semi-supervised learning commonly uses unlabeled data with regularization, self-training, co-training, or graph-based methods.The related work describes these approaches across general and medical-image settings.
- 2. Related Work: Deep co-training extends co-training to multiple 2D networks by using adversarial examples to encourage view differences.This provides a precedent for constructing complementary views when only one data source exists.
- 2. Related Work: Bayesian deep networks provide a basis for estimating uncertainty, which UMCT uses to assess view confidence when training data lacks annotations.The paper emphasizes uncertainty estimation because most training data in its setting is unannotated.
- 2. Related Work: Hybrid 2D/3D methods combine 2D pretrained weights or probability and feature maps with 3D modeling to exploit complementary strengths.The related work contrasts 2D initialization and established architectures with 3D spatial modeling.
3. Uncertainty-aware Multi-view Co-training
UMCT performs 3D semi-supervised co-training by creating complementary views and weighting their pseudo-labels according to Bayesian uncertainty. The framework combines transformed inputs, asymmetric 3D models, and uncertainty-weighted fusion to exploit unlabeled data.
- Overall Framework: UMCT trains multiple 3D views on rotated or permuted inputs, using inverse transformations to align their prediction maps for co-training.Each view has a separate model, and predictions are transformed back before supervised or cross-view learning.
- Overall Framework: The co-training objective encourages consensus among view predictions while using fused pseudo-labels to supervise models on unlabeled data.The combined loss adds a weighted co-training term to the supervised loss, with λcot controlling its contribution.
- Encouraging View Differences: Asymmetric n × n × 1 kernels initialized with 2D weights create feature-level view differences while retaining full 3D fine-tuning.The design is intended to produce complementary representations and reduce the risk of collapsed networks during co-training.
- Reliable Pseudo-Labels: The uncertainty-weighted label fusion module estimates each view’s epistemic uncertainty with Bayesian dropout and converts confidence into pseudo-label weights.The implementation samples K = 10 outputs per view, aggregates voxel-wise uncertainty over the volume, and uses reciprocal uncertainty as confidence.
- Implementation Details: Training uses an encoder–decoder 3D ResNet-18 variant, with inflated and asymmetric convolutions, skip connections, separate labeled-view pretraining, and co-training fine-tuning.The co-training loss is applied to unlabeled data in the semi-supervised setting, while segmentation loss trains on labeled data.
4. Experiments
UMCT was evaluated on NIH pancreas and LiTS liver tumor segmentation under semi-supervised and fully supervised settings, with comparisons to established methods and ablations of its components. The experiments report improved data efficiency, strong lesion segmentation gains, and competitive challenge performance.
- 4.1.1 NIH Pancreas Segmentation Dataset: UMCT outperformed other semi-supervised methods on NIH pancreas segmentation, achieving about a 4% gain with 10% labeled data while retaining gains at 20%.The comparison used average single-view DSC scores with matched backbone implementations.
- 4.1.1 NIH Pancreas Segmentation Dataset: 20% labeled UMCT reached DSC 80.35% versus 78.95% for 60% supervised training, indicating approximately 70% labeling-effort savings.The data-utilization plot compares UMCT with the fully supervised 3D ResNet-18 baseline across labeled-data proportions.
- 4.1.2 LiTS Liver Tumor Segmentation Challenge: On LiTS, UMCT produced its largest improvements for challenging lesion segmentation, while liver segmentation gains were limited because single-view performance was already high.The study reports both single-view and multi-view ensemble results for 10% and 20% labeled data.
- 4.2. Application to Fully Supervised Settings: A fully supervised 3-view UMCT method achieved state-of-the-art tumor DSC on LiTS and second place in the Medical Segmentation Decathlon final testing phase.The challenge results are summarized in Tables 3 and 4.
- 4.3. Ablation Studies: Ablations found that the asymmetric 3D ResNet-18 design outperformed V-Net by more than 1%, while uncertainty-weighted fusion improved performance with three and six views.Multi-view co-training also improved both individual models and the ensemble in fully supervised NIH experiments.
5. Conclusion
UMCT extends co-training to 3D through multi-view data transformations, asymmetric 3D kernels, and uncertainty-weighted label fusion. It improves semi-supervised segmentation and also performs strongly when co-training is applied in fully supervised settings.
- UMCT extends dual-view and deep co-training from 2D images to multi-view 3D training with data-level view differences.
- Asymmetrical 3D kernels initialized from 2D pre-trained models introduce feature-level differences across views.
- Bayesian uncertainty estimation enables uncertainty-weighted label fusion that assigns larger weights to more confident view predictions.
- On NIH pancreas and LiTS liver tumor datasets, UMCT outperformed other approaches by a large margin on the NIH dataset.
- Applying co-training objectives to labeled data also produced promising results, supporting multi-view co-training on 2D-initialized networks.