Source-linked AI summary
Constructing Self-motivated Pyramid Curriculums for Cross-Domain Semantic Segmentation: A Non-Adversarial Approach
Qing Lian, Fengmao Lv, Lixin Duan, Boqing Gong
TL;DR
Semantic segmentation models trained on synthetic images suffer from a source–target visual mismatch when applied to real images. PyCDA connects curriculum adaptation with self-training through a self-motivated pyramid of target-domain properties, and reports best or state-of-the-art performance on GTAV-to-Cityscapes and SYNTHIA-to-Cityscapes adaptation without extra discriminators or minmax optimization.
Problem
Synthetic-to-real semantic segmentation is limited by visual mismatch between synthetic source domains and real target domains, causing performance degradation.
Method
PyCDA combines curriculum adaptation and self-training by deriving target-image, region, and pixel properties from the segmentation network and enforcing them through a pyramid curriculum.
Results
PyCDA outperforms existing methods in both GTAV-to-Cityscapes and SYNTHIA-to-Cityscapes settings, including state-of-the-art adversarial approaches.
Takeaways & Limitations
The approach provides a lighter-weight, easier-to-optimize alternative that does not require extra discriminator networks or minmax problems.
Takeaways & Limitations
Evaluation follows standard GTAV-to-Cityscapes and SYNTHIA-to-Cityscapes benchmarks, using the Cityscapes validation set because official test labels are unavailable.
Abstract
from arXiv · showhide
We propose a new approach, called self-motivated pyramid curriculum domain adaptation (PyCDA), to facilitate the adaptation of semantic segmentation neural networks from synthetic source domains to real target domains. Our approach draws on an insight connecting two existing works: curriculum domain adaptation and self-training. Inspired by the former, PyCDA constructs a pyramid curriculum which contains various properties about the target domain. Those properties are mainly about the desired label distributions over the target domain images, image regions, and pixels. By enforcing the segmentation neural network to observe those properties, we can improve the network's generalization capability to the target domain. Motivated by the self-training, we infer this pyramid of properties by resorting to the semantic segmentation network itself. Unlike prior work, we do not need to maintain any additional models (e.g., logistic regression or discriminator networks) or to solve minmax problems which are often difficult to optimize. We report state-of-the-art results for the adaptation from both GTAV and SYNTHIA to Cityscapes, two popular settings in unsupervised domain adaptation for semantic segmentation.
1. Introduction
Synthetic imagery reduces annotation burden for semantic segmentation but creates a visual mismatch that degrades transfer to real images. PyCDA connects curriculum adaptation and self-training to construct a self-motivated pyramid curriculum without extra discriminators or minmax optimization.
- Motivation: Synthetic datasets can reduce the heavy annotation burden of training semantic segmentation networks.GTAV-based imagery was labeled in about 49 hours, or roughly 7 seconds per image.
- Motivation: The visual mismatch between synthetic source domains and real target domains causes significant performance degradation when transferring segmentation models.
- Approach: PyCDA connects curriculum domain adaptation and self-training to construct a new curriculum for cross-domain semantic segmentation.
- Contribution: PyCDA achieves results on par with or better than adversarial adaptation methods while avoiding extra discriminator networks and minmax optimization.It also outperforms the original curriculum adaptation and self-training methods individually.
- Approach: The method combines pseudo labels with a pyramid of target-image regions, whose label distributions are inferred from the segmentation network itself.The pyramid includes full images, regions, and pixels; properties are inferred from pixelwise target predictions and used in a backpropagated loss.
2. Related Work
Prior semantic segmentation methods use deep networks and multiscale context, while domain adaptation addresses distribution mismatches through curriculum-based and adversarial strategies. PyCDA is presented against this background as a curriculum-style adaptation method.
- Semantic segmentation: Semantic segmentation assigns a label to every pixel and commonly uses deep networks with dilated convolutions and multiscale context.Some methods extend dilated convolution into pyramids or resize features at multiple scales.
- Domain adaptation: Domain adaptation addresses performance drops caused when training and test data come from mismatched distributions.
- Domain adaptation for semantic segmentation: Domain adaptation for semantic segmentation includes curriculum strategies that solve easier target-domain tasks before using them to regularize segmentation.Self-training is viewed in this work as a curriculum-style domain adaptation method.
- Domain adaptation for semantic segmentation: Adversarial methods align domains through features, structured outputs, spatial information, class boundaries, or outlier handling.
3. Approach
PyCDA connects curriculum domain adaptation with self-training to build a target-domain pyramid of label properties, inferred by the segmentation network and enforced during training. The pyramid combines full-image distributions, region-level labels, and pixel pseudo-labels while replacing costly superpixels with GPU-friendly squares.
- Self-motivated pyramid CDA (PyCDA): The objective combines source pixel-wise cross-entropy with target-image, squared-region, and pseudo-labeled-pixel losses.The trade-off parameters are set to λ1 = 1 and λ2 = 0.5 in the experiments.
- CDA vs. ST: PyCDA unifies curriculum domain adaptation and self-training by combining their target-domain property sets.The resulting design uses curriculum-style label distributions and self-training-style pseudo-labels within one objective.
- Self-motivated pyramid CDA (PyCDA): PyCDA infers target-domain properties from the segmentation network’s predictions instead of using additional logistic regression or SVM models.The network estimates distributions over squared regions and full images during each training iteration.
- Self-motivated pyramid CDA (PyCDA): The target-image pyramid has pixels at the bottom, small regions in the middle, and the full image at the top.Pixel labels are pseudo-labels, region labels are one-hot vectors, and the full image uses a label distribution.
- Self-motivated pyramid CDA (PyCDA): PyCDA replaces non-overlapping superpixels with overlapped squares to reduce computation and enable efficient GPU processing.Average pooling supplies square-level predictions, which are thresholded into labels and converted to one-hot distributions.
- Self-motivated pyramid CDA (PyCDA): Pixel pseudo-labels are assigned by selecting the highest-probability class and retaining it when its probability exceeds 0.5.Pixels below the threshold receive null labels and are excluded from the bottom pyramid layer.
4. Experiments
Experiments evaluate PyCDA for unsupervised adaptation from GTAV and SYNTHIA to Cityscapes using standard benchmarks and IoU-based metrics. PyCDA outperforms prior methods across both settings, while ablations support combining curriculum adaptation with self-training and using pixel squares.
- Experimental setup: Experiments use GTAV-to-Cityscapes and SYNTHIA-to-Cityscapes as standard unsupervised domain adaptation benchmarks.Cityscapes supplies real target images, while GTAV and SYNTHIA provide synthetic source images.
- Experimental setup: IoU is the evaluation metric, with mIoU reported as the mean across classes.For each class, IoU uses true-positive, false-positive, and false-negative pixel counts.
- Results on GTAV to Cityscapes: PyCDA achieves the best reported mIoU for adaptation from GTAV to Cityscapes and outperforms existing state-of-the-art methods.Compared methods include adversarial and distribution-matching approaches; several prior methods use more target images and no separate validation set.
- Results on GTAV to Cityscapes: PyCDA is particularly effective on dominant classes and improves over CBST on small-object classes.The dominant classes include road, building, vegetation, and car, while the small-object examples include rider, wall, and fence.
- Results on SYNTHIA to Cityscapes: PyCDA again outperforms existing state-of-the-art methods by a large margin when adapting from SYNTHIA to Cityscapes with different backbones.The reported setting uses FCN8s with VGG-16 and PSP-Net with ResNet-101.
- Ablation study: Connecting curriculum domain adaptation with self-training at either pixel or pixel-square levels outperforms either method individually, while using both levels further boosts mIoU.The full PyCDA pyramid combines pixel and pixel-square levels with the top image-level layer.
- Ablation study: Pixel squares achieve comparable mIoU to superpixels while avoiding superpixel generation overhead of about 3.6s per image.The replacement is intended to reduce computation cost.
- Qualitatively comparing GTAV and SYNTHIA: GTAV-based PyCDA produces better qualitative results than SYNTHIA-based PyCDA, especially for the road class, whose IoU is 90.5%.The authors attribute this observation to GTAV's greater visual similarity to real self-driving scenes in appearance and spatial layout.
5. Conclusion
The conclusion presents PyCDA as a self-motivated pyramid curriculum for pixel-level semantic segmentation domain adaptation. It combines image-, region-, and pixel-level properties derived from the segmentation network and is effective on two synthetic-to-real benchmarks.
- Conclusion: PyCDA connects self-training and curriculum domain adaptation for pixel-level semantic segmentation.The method is presented as a new perspective on cross-domain adaptation.
- Conclusion: The curriculum uses pixel squares at multiple sizes, with the full image as the top layer and pixels as the bottom layer.Label distributions are derived from the same network in the previous training iteration.
- Conclusion: The pyramid preserves and captures local information for objects appearing at different scales.This is the stated role of using pixel squares at multiple sizes.
- Conclusion: Experiments on GTAV to Cityscapes and SYNTHIA to Cityscapes demonstrate PyCDA's effectiveness against other state-of-the-art methods.The conclusion summarizes results across both benchmark settings.
1. Number of middle layers
The experiments vary the number of middle layers in the PyCDA pyramid for GTAV-to-Cityscapes adaptation. Middle layers improve overall performance, with results remaining relatively consistent across layer counts unless pixel squares become too large.
- Table 1 reports mIoUs% for GTAV-to-Cityscapes adaptation with different numbers of inserted middle layers.
- Middle layers improve the overall performance of PyCDA.
- Results remain relatively consistent across different numbers of middle layers.
- Pixel squares larger than 128 × 128 can harm both accuracy and training speed.