Source-linked AI summary
Deep Co-Training for Semi-Supervised Image Segmentation
Jizong Peng, Guillermo Estrada, Marco Pedersoli, Christian Desrosiers
TL;DR
Semi-supervised segmentation seeks to use unlabeled images when expert pixel annotations are scarce and costly. The paper introduces an ensemble-based deep adversarial co-training method in which models exchange information while adversarial examples preserve diversity, achieving strong results on two medical-image datasets.
Problem
Expert pixel annotations for medical images are difficult and time-consuming to obtain, motivating segmentation methods that use unlabeled images alongside labeled data.
Method
The method jointly trains an ensemble of segmentation networks on different labeled subsets and shared unlabeled images, using adversarial examples to preserve diversity.
Results
The method outperforms recent approaches including Pseudo Label, VAT, and Mean Teacher on two public benchmark datasets.
Takeaways & Limitations
The experiments demonstrate the usefulness of deep co-training for single-image segmentation and show benefits from combining ensemble agreement and diversity losses.
Takeaways & Limitations
Training and testing multiple segmentation networks increases computational requirements, while balancing the three loss terms requires tuning that can affect performance.
Abstract
from arXiv · showhide
In this paper, we aim to improve the performance of semantic image segmentation in a semi-supervised setting in which training is effectuated with a reduced set of annotated images and additional non-annotated images. We present a method based on an ensemble of deep segmentation models. Each model is trained on a subset of the annotated data, and uses the non-annotated images to exchange information with the other models, similar to co-training. Even if each model learns on the same non-annotated images, diversity is preserved with the use of adversarial samples. Our results show that this ability to simultaneously train models, which exchange knowledge while preserving diversity, leads to state-of-the-art results on two challenging medical image datasets.
1. Introduction
Semantic segmentation is important across computer vision, especially medical imaging, but obtaining sufficient expert pixel annotations is difficult. The paper extends deep co-training to segmentation, using adversarial training and experiments to study its potential and design choices.
- Semantic segmentation assigns a category label to every pixel and supports applications including medical image assessment and treatment planning.
- Deep segmentation models require large annotated datasets, but medical images are difficult and time-consuming for highly trained experts to label.For infant brain segmentation, annotating one subject may take a radiologist up to a week.
- Semi-supervised segmentation leverages unlabeled images alongside labeled images, avoiding the need for annotations tailored to every individual image.
- Deep co-training had previously been applied only to classification, whereas segmentation has a more complex structured output space.
- The paper presents deep adversarial co-training for semantic segmentation and identifies it as the first co-training method proposed for single-image semantic segmentation.
- The experiments examine prediction diversity, classifier count, agreement–diversity trade-offs, and adversarial-example generation across different image-segmentation tasks.
2. Related work
Related work spans semi-supervised learning methods that propagate information from labeled to unlabeled images, including pseudo-labeling, representation learning, and adversarial approaches. The paper frames deep co-training as a collaborative ensemble method for structured segmentation outputs.
- Pseudo-labeling uses the current model’s most likely labels as ground truth for unlabeled data.
- Other semi-supervised approaches use unlabeled samples through unsupervised representations learned by autoencoders or variational autoencoders.
- Semi-supervised learning methods for segmentation propagate labels from training samples to unlabeled images, but segmentation requires handling structured outputs.
- GAN-based methods for semi-supervised segmentation train a discriminator to distinguish segmentations of labeled and unlabeled images.
- Adversarial examples differ from GANs because adversarial training generates subtly modified samples that can fool a learned model.
3. Methodology
Deep Co-Training Segmentation trains multiple segmentation networks collaboratively with labeled and unlabeled images. Supervised, agreement, and diversity losses exchange information while preserving differences between models, whose outputs are combined at test time.
- Proposed approach: The method trains an ensemble of k segmentation networks collaboratively and combines their outputs to predict labels for new images.The paper focuses on a dual-view setting with k = 2 while describing an extension to multiple views.
- Training objective: The objective combines supervised, co-training agreement, and diversity losses as L(θ; D) = Lsup(θ; S) + λcot Lcot(θ; U) + λdiv Ldiv(θ; D).The three terms respectively use labeled data, unlabeled-data agreement, and diversity-promoting examples.
- Supervised loss: Each model receives a potentially different labeled subset, and supervised loss aggregates separate pixel-wise losses for the models.The implementation uses pixel-wise cross-entropy, while the framework allows other segmentation losses.
- Ensemble agreement loss: Agreement loss minimizes disagreement on unlabeled images using Jensen-Shannon divergence between each model’s prediction and the mean prediction.Because JSD is symmetric, predictions from all models are treated equally when disagreement is minimized.
- Diversity loss: Adversarial examples from labeled and unlabeled images teach other models, preserving output diversity when ordinary training images constrain agreement or ground-truth consistency.The diversity loss encourages robustness to another model’s adversarial examples and discourages identical decision boundaries.
- Testing process: At test time, hard voting selects majority pixel labels, whereas soft voting averages pixel-wise class probabilities across models.The trained models are fed the same unlabeled image before their outputs are combined.
4. Experiments and results
Experiments on three medical segmentation datasets evaluate DCT-Seg across labeled-data regimes, ensemble sizes, modalities, resolutions, and diversity-loss behavior. DCT-Seg generally improves segmentation, especially when labels are scarce, while soft voting consistently benefits ensemble predictions.
- Evaluation datasets and metrics: Experiments use ACDC, SCGM, and Spleen medical image segmentation datasets, with DSC and HD as evaluation metrics.The Spleen experiments also test 256×256 and 512×512 input resolutions.
- Evaluation protocol: Soft voting improves accuracy over individual-model predictions for both DSC and HD across all cases.The study compares DCT-Seg with Pseudo Label, VAT, Mean Teacher, Independent, JSD, and full supervision baselines.
- ACDC dataset: 5.63%: DCT-Seg’s overall mean DSC improvement over Independent on ACDC with 20% labeled training images.DCT-Seg also improves mean DSC by 3.16% over Mean Teacher and remains 2.55% below full supervision; its HD outperforms all three baselines, while diversity adds little over JSD for HD.
- ACDC dataset: A small DSC increase occurs from two to three views, but a fourth view provides no further improvement for deep co-training.Increasing views does not significantly improve individually trained models, suggesting limited views capture the relevant variability effectively.
- ACDC dataset: 7.00% and 7.06%: deep co-training’s largest DSC improvements over separate training occur at labeled ratios la = 0.05 and la = 0.1.Mean DSC increases sharply as the labeled-image ratio increases, but co-training remains better than separate training at every tested ratio.
- SCGM dataset: 12%: DCT-Seg’s mean DSC exceeds the best baseline, Pseudo Label, on the SCGM dataset.Deep co-training increases DSC scores of Independent and JSD by nearly 25%, indicating a larger benefit when supervised training is limited.
- Spleen dataset: 3-4%: soft-voting DCT-Seg’s mean DSC boost over Mean Teacher on Spleen images is consistent at both tested resolutions.DCT-Seg’s advantage persists across CT data and 256×256 or 512×512 inputs, although visual differences are most pronounced for small foreground regions.
- Impact of diversity loss: A diversity-loss-only model rapidly approaches a reference model, but its accuracy gap grows with adversarial-noise magnitude ϵ.For the Myo class, the reported gaps are 1.43% at ϵ = 0.01, 1.38% at ϵ = 0.001, and 0.02% at ϵ = 0.0001.
5. Discussion and conclusion
The paper applies deep co-training to single-image segmentation and reports benefits from combining ensemble agreement with diversity. It also identifies computational, tuning, and validation boundaries for the method.
- Deep co-training was applied to single-image segmentation and evaluated on two public benchmark datasets.
- Training multiple segmentation networks simultaneously increases computational requirements and restricts the number of possible views.
- Balancing supervised, co-training, and diversity losses requires tuning because the terms can compete during training.
- The method was tested on three medical image segmentation problems using MRI and CT images, while broader validation remains future work.