Source-linked AI summary
Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
Yang Zhang, Philip David, Boqing Gong
TL;DR
Semantic segmentation requires costly annotated imagery, while synthetic data creates a domain mismatch with real urban scenes. The paper learns global and local target label distributions as easy curriculum tasks, then regularizes a source-trained segmentation network with them. The method outperforms competing simulated-to-real adaptation methods and complements global and local distribution cues.
Problem
Urban semantic segmentation needs labor-intensive annotated data, while synthetic imagery introduces a mismatch with real target images.
Method
The method first estimates global image and local landmark-superpixel label distributions, then uses them to regularize target-domain predictions of a source-trained segmentation network.
Results
The method outperforms several competing methods for adapting simulated urban-scene images to real photos.
Takeaways & Limitations
Global image distributions indicate how predictions should update, while landmark-superpixel distributions indicate where, and their combination achieves the best target-domain segmentation results.
Takeaways & Limitations
Using all target-image superpixels can over-regularize the network, especially when their label distributions are inaccurate.
Abstract
from arXiv · showhide
During the last half decade, convolutional neural networks (CNNs) have triumphed over semantic segmentation, which is one of the core tasks in many applications such as autonomous driving. However, to train CNNs requires a considerable amount of data, which is difficult to collect and laborious to annotate. Recent advances in computer graphics make it possible to train CNNs on photo-realistic synthetic imagery with computer-generated annotations. Despite this, the domain mismatch between the real images and the synthetic data cripples the models' performance. Hence, we propose a curriculum-style learning approach to minimize the domain gap in urban scenery semantic segmentation. The curriculum domain adaptation solves easy tasks first to infer necessary properties about the target domain; in particular, the first task is to learn global label distributions over images and local distributions over landmark superpixels. These are easy to estimate because images of urban scenes have strong idiosyncrasies (e.g., the size and spatial relations of buildings, streets, cars, etc.). We then train a segmentation network while regularizing its predictions in the target domain to follow those inferred properties. In experiments, our method outperforms the baselines on two datasets and two backbone networks. We also report extensive ablation studies about our approach.
1. Introduction
The paper addresses domain adaptation for urban-scene semantic segmentation, where synthetic training imagery differs from real target images. It proposes learning easy target-domain label-distribution properties before regularizing a segmentation network.
- Motivation: Semantic segmentation needs massive annotated imagery, but collecting diverse urban scenes and annotating them is laborious.Cityscapes annotation and quality control take more than 1.5 hours per image.
- Motivation: Synthetic imagery reduces annotation effort but introduces substantial visual and statistical mismatch with real images.The paper contrasts simulated SYNTHIA imagery with real Cityscapes imagery and motivates domain adaptation to bridge the gap.
- Challenge: Feature-space adaptation is less suitable here because semantic segmentation is structured prediction with an exponentially large label space.The paper also notes that matching feature representations without structured-label considerations can suppress discriminative cues.
- Approach: Urban scenes provide domain-robust idiosyncrasies, while structured outputs enable posterior regularization instead of relying on a shared prediction function.Examples include the sizes and spatial relations of buildings, streets, and cars.
- Approach: The curriculum first estimates global image-level and local landmark-superpixel label distributions, then uses them to regularize target predictions.Image distributions inform how predictions should update, while superpixel distributions indicate where updates should occur.
- Approach: The method learns the segmentation network from labeled source data while checking that target predictions remain consistent with inferred target-domain properties.It does not alter the segmentation architecture or require intermediate-layer constraints.
2. Related work
Prior work largely addresses domain adaptation for classification, regression, and detection, while this paper focuses on transferring synthetic urban-scene data to real-image semantic segmentation. It positions curriculum adaptation as a distinct alternative for this structured task.
- Domain adaptation: Domain adaptation addresses discrepancies between training and test distributions, which conventional machine learning assumes are identical.The related work surveys domain adaptation across classification, regression, and deep neural networks.
- Synthetic-to-real transfer: Synthetic data has been used for generic object detection, pedestrian detection, and improving simulated imagery with real data.These studies span transfers from synthetic images to real environments and the reverse direction.
- Semantic segmentation: Semantic segmentation assigns an object label to every image pixel, and contemporary high-performing methods are predominantly CNN-based.Earlier approaches relied on manually designed local image features.
- Data and annotation: Segmentation datasets require substantial annotation labor, motivating virtual imagery with automatically generated labels.Synthetic-image annotation through a computer game averaged 7 seconds, compared with much longer manual segmentation times.
- Domain adaptation for segmentation: The closest prior synthetic-to-real segmentation work regularizes intermediate layers and network outputs, whereas this paper solves easy target-property tasks first.The proposed curriculum then uses the learned target-domain knowledge to regularize network predictions.
3. Approach
The approach adapts segmentation by first estimating target-domain label distributions, then regularizing a segmentation network to match these properties while retaining source-domain supervision.
- 3.1. Domain adaptation using the target properties: Target properties are represented as category-occupancy distributions over whole images or selected superpixels, computed from annotations or network predictions.For an image, each category's value is its proportion of pixels assigned that category.
- 3.1. Domain adaptation using the target properties: Training minimizes cross entropy between estimated and predicted target-property distributions alongside pixel-wise cross entropy on labeled source images.The parameter γ balances source supervision against target-property regularization.
- 3.2.1 Global label distributions of images: Because target annotations are unavailable, the method estimates image-level distributions from labeled source data using logistic regression, nearest-neighbor transfer, source means, or uniform predictions.Image features for these estimators are extracted with Inception-ResNet-v2.
- 3.2.1 Global label distributions of images: The source-only baseline often confuses streets with sidewalks or cars, producing highly disproportionate target-label assignments that the curriculum regularization is designed to rectify.The method uses global distributions to guide how predictions update and local distributions to guide where they update.
- 3.2.2 Local label distributions of landmark superpixels: Global image distributions discourage disproportionate target segmentation but lack spatial constraints, so landmark-superpixel distributions provide local guidance.Using every superpixel can overconstrain the network when distributions are inaccurate; the method instead selects landmark superpixels using linear SVMs.
- 3.1. Domain adaptation using the target properties: Curriculum domain adaptation separates easy target-property inference from the hard task of pixel-level semantic segmentation.The inferred properties constrain target predictions during network training.
4. Experiments
Experiments evaluate adaptation from synthetic urban imagery to real Cityscapes images, showing that inferred label distributions improve segmentation beyond source-only training and that global and landmark-superpixel cues complement each other.
- Datasets and evaluation: Cross-dataset nearest neighbors are visually very different, illustrating the substantial mismatch between the synthetic and real domains.Figure 2 pairs Cityscapes images with nearest SYNTHIA neighbors retrieved using Inception-Resnetv2 features.
- Datasets and evaluation: The experiments adapt segmentation from synthetic SYNTHIA images to real Cityscapes images using shared urban-scene classes.Cityscapes supplies the target domain, while SYNTHIA supplies the source domain; 16 common classes are used.
- Results of inferring global label distributions: The source-only baseline produces severely disproportionate target-domain label predictions, whereas nearest-neighbor and logistic-regression estimates perform best for global label distributions.The comparison uses χ2 distance, where smaller values are better; logistic regression is used in subsequent experiments.
- Comparison results: Selected landmark superpixels achieve more than 88% classification accuracy, compared with 71% for all target-domain superpixels.The selected landmarks comprise 60% of the superpixels.
- Comparison results: All tested adaptation methods significantly outperform NoAdapt, and global image-level distributions correct obvious road-versus-sidewalk prediction errors.The image-level method provides useful supervision despite operating at a higher level than pixel-wise segmentation.
- Comparison results: Landmark-superpixel distributions raise Ours (SP) to 28.1% mean IoU, while regularizing every superpixel yields little improvement when estimates are inaccurate.Superpixel methods are strong on large regions but miss small objects; image-level distributions perform better on small objects, so the cues complement each other.
- Comparison results: The method gains 7% over the authors’ baseline, compared with a 3% gain reported for the prior method under its experimental setup.The paper notes possible implementation or setup differences when comparing the baselines.
5. Conclusion
The paper concludes that curriculum domain adaptation can transfer knowledge from simulated images to real urban scenes by learning target-domain label distributions before pixel-wise segmentation. Its reported experiments show that this strategy outperforms competing methods, while future work will infer additional target properties.
- Conclusion: The approach first estimates global image and local landmark-superpixel label distributions, then uses them to regularize target-domain segmentation predictions.These distribution-estimation tasks are treated as easier than pixel-wise label assignment.
- Conclusion: The method outperforms several competing approaches for adapting from simulated images to real photos of urban traffic scenes.
- Conclusion: Future work will explore additional target properties that can be inferred to enrich the curriculum domain adaptation framework.