Semi-Supervised Multi-Organ Segmentation via Deep Multi-Planar Co-Training

Yuyin Zhou; Yan Wang; Peng Tang; Song Bai; Wei Shen; Elliot K. Fishman; Alan L. Yuille

Source-linked AI summary

Semi-Supervised Multi-Organ Segmentation via Deep Multi-Planar Co-Training

Yuyin Zhou, Yan Wang, Peng Tang, Song Bai, Wei Shen, Elliot K. Fishman, Alan L. Yuille

arXiv:1804.02586v3Apr 7, 2018cs.CV

TL;DR

Voxel-wise annotation for abdominal multi-organ CT segmentation is costly and difficult, while large unlabeled volumes are easier to obtain. DMPCT uses co-training across axial, coronal, and sagittal planes with multi-planar fusion to refine pseudo-labels, and it outperforms fully supervised and single-planar baselines, especially with limited annotations.

Problem
Voxel-wise annotation of abdominal CT volumes is difficult, expensive, and time-consuming, whereas large unlabeled CT datasets are easier to obtain.
Method
DMPCT jointly trains multi-planar teacher and student networks and uses fusion across axial, coronal, and sagittal predictions to generate more reliable pseudo-labels.
Results
DMPCT outperforms the fully supervised approach by 4.23% and also outperforms the single-planar semi-supervised method, with larger gains for difficult organs.
Takeaways & Limitations
DMPCT can leverage unlabeled data for abdominal multi-organ segmentation and performs especially well when labeled data are limited.
Takeaways & Limitations
The paper does not analyze how different fusion rules affect performance and notes that better fusion could improve results.

Abstract

from arXiv · show

In multi-organ segmentation of abdominal CT scans, most existing fully supervised deep learning algorithms require lots of voxel-wise annotations, which are usually difficult, expensive, and slow to obtain. In comparison, massive unlabeled 3D CT volumes are usually easily accessible. Current mainstream works to address the semi-supervised biomedical image segmentation problem are mostly graph-based. By contrast, deep network based semi-supervised learning methods have not drawn much attention in this field. In this work, we propose Deep Multi-Planar Co-Training (DMPCT), whose contributions can be divided into two folds: 1) The deep model is learned in a co-training style which can mine consensus information from multiple planes like the sagittal, coronal, and axial planes; 2) Multi-planar fusion is applied to generate more reliable pseudo-labels, which alleviates the errors occurring in the pseudo-labels and thus can help to train better segmentation networks. Experiments are done on our newly collected large dataset with 100 unlabeled cases as well as 210 labeled cases where 16 anatomical structures are manually annotated by four radiologists and confirmed by a senior expert. The results suggest that DMPCT significantly outperforms the fully supervised method by more than 4% especially when only a small set of annotations is used.

1. Introduction

Abdominal multi-organ segmentation is clinically important but difficult and costly to annotate, motivating semi-supervised methods that exploit unlabeled CT volumes. DMPCT addresses this gap by combining multi-planar co-training with fusion-based pseudo-label refinement.

1. Introduction: Abdominal organ segmentation is challenging because CT images have low contrast and substantial anatomical shape variability.The paper frames this difficulty in contrast with segmentation of structures such as the brain or heart.
1. Introduction: Semi-supervised learning is motivated by the high cost and time required for radiologists to produce and verify voxel-wise annotations in 3D volumes.Large unannotated CT datasets are described as easier to obtain than fully labeled datasets.
1. Introduction: Earlier biomedical semi-supervised segmentation methods were mainly graph-based, while deep network-based approaches had received limited attention.Existing deep approaches could reinforce errors when iteratively updating pseudo-labels from incorrect initial predictions.
1. Introduction: DMPCT combines axial, coronal, and sagittal networks through co-training and multi-planar fusion to generate more reliable pseudo-labels.The framework uses a teacher model, fusion module, and student model to exploit unlabeled 3D CT data.
1. Introduction: 4.23% improvement over the fully supervised method was observed on a newly collected large dataset.The paper reports this improvement while evaluating DMPCT on abdominal CT multi-organ segmentation.

2. Related Work

Prior work includes fully supervised CNN-based segmentation and several semi-supervised paradigms, including self-training, co-training, multi-view learning, and graph-based methods. DMPCT embeds the multi-view structure of 3D medical data into co-training.

Fully-supervised multi-organ segmentation: CNN-based multi-organ segmentation methods are generally more efficient and accurate than multi-atlas approaches, with 2D CNNs requiring less time than sliding-window 3D CNNs.The comparison concerns computational efficiency and training strategy across CNN-based approaches.
Semi-supervised learning: Semi-supervised learning commonly uses self-training, co-training, multi-view learning, or graph-based methods.These paradigms represent the main technique categories reviewed by the paper.
Semi-supervised learning: Self-training can reinforce classification mistakes because a model is retrained using its own predictions on unlabeled data.This failure mode motivates approaches that obtain information from multiple learners or views.
Semi-supervised learning: Co-training uses multiple feature sets describing the same data and can yield more accurate and robust results than self-training.Its standard formulation assumes each feature subset is sufficient to train a good classifier.
2. Related Work: DMPCT embeds the multi-view property of 3D medical data into a co-training framework.The method uses multiple planes as complementary views of the same CT volume.

3. Deep Multi-Planar Co-Training

DMPCT trains plane-specific teacher models, fuses their predictions into pseudo-labels for unlabeled CT volumes, and iteratively retrains student models on labeled and pseudo-labeled data. Multi-planar fusion combines sagittal, coronal, and axial predictions using agreement and confidence to produce improved 3D estimations.

3.1. Teacher Model: Three 2D-FCN models independently segment sagittal, coronal, and axial slices derived from each labeled 3D volume.The models are trained separately for the three planes using the labeled dataset.
3.2. Multi-Planar Fusion Module: Multi-planar fusion reconstructs 3D predictions from plane-specific slices, exploiting complementary information across views.Predictions are reconstructed into volumes before fusion, which the method describes as more accurate and robust than any single-plane estimate.
3.2. Multi-Planar Fusion Module: Majority voting selects labels when planes agree, while the maximum-confidence prediction resolves voxel-level disagreements.The resulting fused pseudo-label is generated from hard predictions across sagittal, coronal, and axial planes.
3.2. Multi-Planar Fusion Module: Fusion improves both over- and under-estimation, producing a better outcome than predictions from individual planes.The paper illustrates this comparison using reconstructed predictions and their fusion output.
3.3. Student Model: The student model is trained on the union of manually labeled and automatically labeled data, with pseudo-labeling and retraining repeated for T iterations.The overall procedure begins with supervised teacher training and alternates student optimization with pseudo-label generation.
3. Deep Multi-Planar Co-Training: DMPCT iteratively alternates pseudo-label generation and student training after initializing models on labeled data.The teacher is trained on manually labeled volumes; pseudo-labels are generated for unlabeled volumes, and the student is trained on the combined dataset.

4.1. Dataset and Evaluation

The study evaluates DMPCT on labeled and unlabeled abdominal CT datasets covering 16 anatomical structures, using average DSC and standard deviation across test cases.

Dataset: The fully labeled dataset contains 210 portal-venous-phase abdominal CT images, split into 50 training, 30 validation, and 80 testing patients.Four radiologists segmented 16 structures per case, with confirmation by an independent senior expert.
Dataset: The unlabeled dataset comprises 100 cases acquired from a local hospital.
Evaluation: Evaluation reports each organ’s average Dice-Sørensen Coefficient with its standard deviation over testing cases.DSC measures similarity between predicted and ground-truth voxel sets.

4.2. Implementation Details

Implementation uses multi-window CT inputs, PascalVOC-pretrained FCN parameters, and fixed teacher–student training schedules with validation-based hyperparameter tuning.

Training: The learning rate is 10^-9, and the teacher and student models are trained for 80,000 and 160,000 iterations, respectively.
Training: Validation data tunes hyperparameters, while the EM-like iteration count is set to T = 2 because validation performance saturates.
Input and initialization: Three Hounsfield-unit windows are used as input channels, with slice intensities rescaled to [0.0, 1.0].The windows are [−125, 275], [−160, 240], and [−1000, 1000].
Input and initialization: The network parameters θ are initialized from an FCN-8s model pretrained on PascalVOC image segmentation.

4.3. Comparison with the Baseline

DMPCT outperforms fully supervised FCN and single-planar semi-supervised baselines, with larger gains for difficult organs and improved average DSC as unlabeled data increases.

Overall comparison: Average DSC rises from 73.71% to 77.94% as unlabeled data increases from 0 to 100 cases with 50 labeled cases.The corresponding standard deviation decreases from 9.97% to 8.51%.
Overall comparison: DMPCT outperforms both fully supervised FCN and single-planar semi-supervised SPSL across the evaluated labeled-plus-unlabeled settings.Both baselines apply individual planes separately before multi-planar fusion.
Difficult organs: DMPCT yields larger improvements for difficult small or geometrically complex organs, including Colon, Duodenum, and Small bowel.Reported gains range from 5.48% to 9.89% across the listed difficult organs.

4.4. Discussion

Experiments show DMPCT benefits from unlabeled data, multi-planar fusion, and cross-dataset evaluation, while outperforming supervised and alternative semi-supervised baselines under limited annotation.

4.4.1 Amount of labeled data: DMPCT outperformed SPSL in every tested labeled/unlabeled-data setting, demonstrating the usefulness of multi-planar fusion within co-training.The performance advantage became less prominent when the labeled set was enlarged to 100 cases.
4.4.2 Comparison with 3D network-based self-training: 80.75% versus 78.87% and 72.75%: DMPCT achieved the highest average DSC in a representative multi-plane comparison, ahead of SPSL and FCN.The comparison used 50 labeled cases for all methods and 100 unlabeled cases for SPSL and DMPCT.
4.4.2 Comparison with 3D network-based self-training: 69.66% to 65.21%: adding 100 unlabeled cases reduced patch-based 3D UNet’s mean DSC, consistent with pseudo-label errors being reinforced during student training.The 3D UNet results used 50 labeled cases, and the authors attribute the degradation to inadequate teacher-model training.
4.4.3 Comparison with traditional co-training: 76.52%: confidence-sample selection slightly exceeded SPSL but remained below DMPCT and required more than 50 hours to converge.The method selected the 5,000 highest-confidence samples per iteration under a 50-labeled, 50-unlabeled setting.
4.4.4 Cross dataset generalization: 59.23 ± 22.20% to 67.38 ± 19.64%: DMPCT improved mean DSC and reduced variability across 10 organs on a public dataset without retraining.On the NIH pancreas dataset, DMPCT achieved 66.16% versus 58.73% for FCN.

5. Conclusion

The paper concludes that DMPCT uses multi-planar co-training and fusion to improve semi-supervised abdominal CT segmentation, particularly when labeled data are scarce, with potential clinical utility.

5. Conclusion: DMPCT outperformed fully supervised learning and single-planar semi-supervised learning, and benefited especially from additional unlabeled data when labeled data were limited.The conclusion attributes the framework’s approach to incorporating multi-planar information during training and iterative pseudo-label updating.
5. Conclusion: DMPCT may assist radiologists because annotating multiple organs in 3D volumes requires substantial labor.The framework is reported to work across multiple complex anatomical structures within efficient computational time.
5. Conclusion: The authors leave higher accuracy from a more powerful backbone or advanced fusion algorithm as future work.This identifies prospective extensions rather than a demonstrated result of the current framework.