Source-linked AI summary
Adversarial Learning for Semi-Supervised Semantic Segmentation
Wei-Chih Hung, Yi-Hsuan Tsai, Yan-Ting Liou, Yen-Yu Lin, Ming-Hsuan Yang
TL;DR
Pixel-level annotation is costly, and existing adversarial segmentation methods do not fully exploit unlabeled images. The paper introduces a fully convolutional discriminator that aligns predicted probability maps with ground-truth structure and identifies trustworthy unlabeled regions. Experiments on PASCAL VOC 2012 and Cityscapes validate the framework's effectiveness.
Problem
Pixel-level annotations are expensive, while prior adversarial segmentation methods do not tackle semi-supervised learning with unlabeled images.
Method
A fully convolutional discriminator compares predicted probability maps with ground-truth label maps and supplies adversarial and confidence-based supervision for labeled and unlabeled images.
Results
Experiments on PASCAL VOC 2012 and Cityscapes validate the effectiveness of the proposed adversarial semi-supervised segmentation framework.
Takeaways & Limitations
The framework leverages unlabeled images and discriminator responses to improve segmentation without additional computation during inference.
Takeaways & Limitations
For unlabeled data, adversarial training requires a smaller λadv because adversarial loss can over-correct predictions without cross-entropy supervision.
Abstract
from arXiv · showhide
We propose a method for semi-supervised semantic segmentation using an adversarial network. While most existing discriminators are trained to classify input images as real or fake on the image level, we design a discriminator in a fully convolutional manner to differentiate the predicted probability maps from the ground truth segmentation distribution with the consideration of the spatial resolution. We show that the proposed discriminator can be used to improve semantic segmentation accuracy by coupling the adversarial loss with the standard cross entropy loss of the proposed model. In addition, the fully convolutional discriminator enables semi-supervised learning through discovering the trustworthy regions in predicted results of unlabeled images, thereby providing additional supervisory signals. In contrast to existing methods that utilize weakly-labeled images, our method leverages unlabeled images to enhance the segmentation model. Experimental results on the PASCAL VOC 2012 and Cityscapes datasets demonstrate the effectiveness of the proposed algorithm.
1 Introduction
The paper addresses the high cost of pixel-level annotation by introducing adversarial semi-supervised semantic segmentation that uses labeled and unlabeled images. A fully convolutional discriminator supplies structural and confidence-based supervision without additional inference computation.
- Motivation: Pixel-level semantic segmentation annotations are expensive and time-consuming, motivating semi-supervised alternatives beyond image-, box-, or point-level supervision.The paper positions unlabeled-image training as an alternative to methods requiring additional weak annotations.
- Adversarial framework: A fully convolutional discriminator distinguishes ground-truth label maps from predicted probability maps and encourages high-order structural agreement through adversarial loss.The discriminator replaces the need for a separate post-processing module during testing.
- Semi-supervised learning: For unlabeled images, discriminator confidence maps identify trustworthy regions that guide masked cross-entropy training in a self-taught manner.The confidence maps indicate which prediction regions are close to the ground-truth label distribution.
- Contributions: The framework combines adversarial learning with unlabeled-image supervision and improves segmentation without adding inference-time computation.The contributions include using images without annotations and discriminator responses as supervisory signals.
Related Work
Prior semantic-segmentation methods reduce annotation demands through weak labels or adversarial techniques, but the proposed method trains with entirely unlabeled images. Its discriminator learns which predictions are trustworthy instead of relying on hand-designed refinement constraints.
- Semantic segmentation: Weakly supervised methods reduce annotation effort but often struggle to recover detailed boundary information from image-, box-, or point-level signals.Semi-supervised approaches commonly combine fully annotated data with weakly labeled samples.
- Proposed distinction: The proposed algorithm leverages unlabeled images by using discriminator outputs as supervisory signals that compensate for absent image annotations.This distinguishes it from methods using weakly labeled images.
- Proposed distinction: Unlike hand-designed constraints that refine prediction maps before training, the method learns the self-taught selection criterion through its adversarial network.The discriminator selects regions whose prediction distributions are close to ground-truth distributions.
- Generative adversarial networks: Earlier adversarial segmentation work did not achieve substantial baseline improvement and did not address the semi-supervised setting.Other GAN-based approaches generated adversarial examples whose similarity to real images was uncertain.
3 Algorithm Overview
The algorithm couples a semantic-segmentation network with a fully convolutional discriminator. The discriminator evaluates spatial probability maps, while labeled and unlabeled images provide different training signals for the segmentation network.
- Segmentation network: The segmentation network maps an H ×W ×3 image to class-probability maps of size H ×W ×C.The framework can use segmentation architectures such as FCN, DeepLab, or DilatedNet.
- Discriminator network: The fully convolutional discriminator takes predicted or ground-truth class-probability maps and outputs spatial probability maps of size H ×W ×1.Each output pixel represents whether the corresponding input region comes from ground truth or the segmentation network.
- Training procedure: The discriminator accepts either segmentation predictions or one-hot encoded ground truth as its two possible input distributions.Its confidence output provides spatial rather than image-level discrimination.
- Training procedure: With labeled images, the segmentation network uses cross-entropy and adversarial losses, while the discriminator is trained only on labeled data.Unlabeled images instead train the segmentation network through the proposed semi-supervised procedure.
4 Semi-Supervised Training with Adversarial Network
The method combines a fully convolutional discriminator with segmentation losses and uses its confidence maps to supervise trustworthy regions in unlabeled images.
- Network architecture: The discriminator takes segmentation probability maps or one-hot ground-truth maps and outputs a spatial confidence map at each image location.It receives H×W×C maps and produces an H×W×1 confidence map.
- Discriminator training: The discriminator is trained with spatial cross-entropy to distinguish predictions from ground-truth labels.Predicted maps are assigned discriminator label 0, while ground-truth maps receive label 1.
- Segmentation training: The segmentation network minimizes cross-entropy, adversarial, and semi-supervised losses weighted by λadv and λsemi.The losses are Lce, Ladv, and Lsemi, respectively.
- Unlabeled data: For unlabeled data, adversarial training remains applicable, but λadv must be smaller because adversarial loss can over-correct predictions without cross-entropy supervision.Unlabeled images lack ground-truth annotations, so Lce is not applied.
- Unlabeled data: The discriminator confidence map identifies trustworthy regions whose predictions become masked cross-entropy supervision for self-taught learning.The confidence map is thresholded, and the resulting self-taught target and indicator are treated as constants during training.
5 Experimental Results
Experiments on PASCAL VOC 2012 and Cityscapes evaluate the adversarial and semi-supervised components, showing consistent gains and a trade-off between confidence thresholds and selected pixels.
- Dataset and evaluation: The models are evaluated on PASCAL VOC 2012 and Cityscapes under varying labeled-data settings, including 1/8, 1/4, and 1/2 labeled data.PASCAL VOC 2012 uses 10,582 training images and a 1,449-image validation set; Cityscapes uses 19 annotated classes.
- PASCAL VOC 2012: The adversarial loss improves PASCAL VOC 2012 performance by 1.6% to 2.8% across different amounts of training data.
- Cityscapes: On Cityscapes, adversarial loss yields a 0.5% to 1.9% gain over the baseline, while combining adversarial and semi-supervised learning yields a 1.6% to 3.3% gain.The adversarial loss is described as encouraging the segmentation network to learn structural information from the ground-truth distribution.
- Hyper-parameter analysis: The best reported hyper-parameter setting achieves 69.5% mean IU with a 1.9% gain when λsemi is 0.1.The model performs well for Tsemi values from 0.1 to 0.3, with the best results at Tsemi = 0.2.
- Hyper-parameter analysis: Higher Tsemi selects fewer pixels but produces more confident and accurate pixel predictions, revealing a confidence-versus-coverage trade-off.When Tsemi = 0, all pixel predictions are trusted, which degrades performance.
- Ablation study: Removing the fully convolutional discriminator reduces performance by 1.0% with all data and 0.9% with one-eighth of the data.
6 Conclusions
The paper proposes adversarial learning for semi-supervised semantic segmentation, using a discriminator with labeled and unlabeled data. Experiments on PASCAL VOC 2012 and Cityscapes validate the method's effectiveness.
- Conclusion: The discriminator enhances the segmentation network with labeled and unlabeled data through adversarial loss and confidence-map supervision.For labeled data, adversarial loss learns higher-order structural information; for unlabeled data, discriminator confidence maps provide self-taught signals.
- Conclusion: The proposed approach improves segmentation without requiring additional computation during inference and can benefit from images without annotations.
- Conclusion: Experiments on PASCAL VOC 2012 and Cityscapes validate the effectiveness of the proposed algorithm.
A Pixel Accuracy in Semi-Supervised Learning
The Cityscapes analysis examines how the confidence threshold affects selected-pixel accuracy and coverage, alongside hyper-parameter robustness and training settings.
- Threshold analysis: Higher Tsemi produces more confident discriminator outputs and more accurate pixel predictions, but selects fewer pixels for back-propagation.The threshold therefore exposes a direct accuracy–supervision-coverage trade-off.
- Threshold analysis: Table 7 reports average segmentation accuracy against the number of selected pixels for different Tsemi values on Cityscapes.
- Hyper-parameter analysis: Table 8 analyzes λsemi, Tsemi, and λadv, reporting that the adversarial learning is robust to different λadv values.
- Implementation: Training parameters for both datasets are provided in Table 9, using a PyTorch implementation with planned public release of code and models.
D Additional Qualitative Results
Additional qualitative comparisons use half the training data on PASCAL VOC and Cityscapes, showing improved segmentation quality from adversarial learning and semi-supervised training.
- Qualitative comparisons: Figures 4–5 compare models on PASCAL VOC using half the training data, while Figure 6 compares models on Cityscapes under the same data setting.
- Qualitative comparisons: The qualitative results show that both adversarial learning and the semi-supervised training scheme improve segmentation quality.