Source-linked AI summary
Effective Use of Synthetic Data for Urban Scene Semantic Segmentation
Fatemeh Sadat Saleh, Mohammad Sadegh Aliakbarian, Mathieu Salzmann, Lars Petersson, Jose M. Alvarez
TL;DR
Synthetic-only training is attractive because pixel-level real-image annotation is costly, but domain shift limits performance on real images and existing adaptation methods require real images during training. The paper handles background classes with semantic segmentation and foreground classes with detection, then combines their predictions; experiments on Cityscapes and CamVid show improved segmentation without real images during training.
Problem
Synthetic-only semantic segmentation performs poorly on real images, while existing domain adaptation methods require access to real images during training.
Method
The method combines DeepLab predictions for background classes with Mask R-CNN foreground predictions, using synthetic data and automatically generated VEIS instance annotations.
Results
Experiments on Cityscapes and CamVid show higher real-image segmentation accuracy than standard synthetic-only training and state-of-the-art domain adaptation without seeing real images during training.
Takeaways & Limitations
Handling foreground classes through detection allows synthetic data with realistic object shapes but non-photorealistic textures to improve semantic segmentation.
Takeaways & Limitations
The approach assumes that foreground objects provide a reliable shape-based signal for distinguishing them from background.
Abstract
from arXiv · showhide
Training a deep network to perform semantic segmentation requires large amounts of labeled data. To alleviate the manual effort of annotating real images, researchers have investigated the use of synthetic data, which can be labeled automatically. Unfortunately, a network trained on synthetic data performs relatively poorly on real images. While this can be addressed by domain adaptation, existing methods all require having access to real images during training. In this paper, we introduce a drastically different way to handle synthetic images that does not require seeing any real images at training time. Our approach builds on the observation that foreground and background classes are not affected in the same manner by the domain shift, and thus should be treated differently. In particular, the former should be handled in a detection-based manner to better account for the fact that, while their texture in synthetic images is not photo-realistic, their shape looks natural. Our experiments evidence the effectiveness of our approach on Cityscapes and CamVid with models trained on synthetic data only.
1 Introduction
Synthetic data reduces the cost of obtaining pixel-level annotations, but domain shift makes models trained only on synthetic images perform poorly on real images. The paper addresses this by treating background and foreground classes differently, combining semantic segmentation for backgrounds with detection for foregrounds.
- Pixel-level labeling is expensive, taking about 90 minutes per Cityscapes image on average.
- Synthetic-only semantic segmentation performs poorly on real images, while domain adaptation requires access to real images during training.
- Background textures are relatively realistic in synthetic images, whereas foreground textures differ more but their shapes remain natural.
- The paper therefore uses semantic segmentation for background classes and object detection for foreground classes, exploiting texture realism and object shape respectively.
- On foreground classes, Mask R-CNN outperforms DeepLab on all Cityscapes classes except motorcycle when trained on synthetic data.
- The combined framework improves real-image segmentation on Cityscapes and CamVid, outperforming standard synthetic-only training and domain adaptation without seeing real images during training.
2 Related work
Prior work uses weak annotations or synthetic datasets to reduce segmentation-labeling costs, but synthetic-to-real domain shift remains problematic. This paper proposes an orthogonal class-specific strategy that uses segmentation for backgrounds and detection for foregrounds.
- Weakly supervised methods reduce annotation costs but often focus on foreground objects and collapse background classes into one entity.
- Synthetic datasets provide fully supervised annotations, yet models trained on synthetic images still suffer from domain shift when applied to real images.
- The paper distinguishes foreground and background classes because their perceptual mismatches between synthetic and real images differ.
- The proposed strategy uses semantic segmentation for realistic background textures and detection for foreground objects whose shapes remain realistic despite less natural textures.
3 Method
The method trains separate synthetic-data components for background and foreground classes, then fuses their predictions into a semantic segmentation map. It can additionally use unlabeled real images by turning initial predictions into pseudo-ground truth.
- The method targets driving scenes but is presented as applicable to other semantic segmentation problems, especially where synthetic urban data is available.
- Foreground classes: Mask R-CNN is trained on VEIS instance-level annotations to handle foreground objects using shape-oriented detection.
- Background classes: DeepLab is trained on GTA5 pixel annotations to handle background classes, whose synthetic textures are relatively photo-realistic.
- Prediction fusion: Pixels left unassigned by Mask R-CNN are filled with the highest-probability DeepLab label to produce the final segmentation map.
- Foreground classes: Predicted foreground instances are sorted by confidence and overlapping pixels are removed from lower-priority segments.
- Unsupervised real images: With unlabeled real images, initial predictions become pseudo-ground-truth labels for training a DeepLab model on real images, further boosting accuracy.
4 The VEIS Environment and Dataset
VEIS is a Unity3D-based environment and dataset designed to provide instance-level annotations for urban foreground objects. Although its images are less photo-realistic, realistic object shapes and varied poses make them useful within the paper’s detection-based framework.
- Environment: VEIS addresses missing instance-level annotations in existing synthetic urban-scene datasets such as GTA5, SYNTHIA, and VIPER.GTA5 lacks instance-level annotations, while SYNTHIA and VIPER do not provide them for all required classes.
- Environment: The Unity3D environment lets researchers manually construct urban scenes and add freely available 3D foreground objects.Synthetic images and instance-level semantic segmentation maps are rendered simultaneously in real time.
- Environment: VEIS provides virtually unlimited annotated images after one day of development effort.The paper describes this as enabling broad coverage of object classes used in CamVid and Cityscapes.
- Dataset: VEIS images are less photo-realistic, so they provide little benefit when used alone for semantic segmentation on real images.Within the proposed framework, however, realistic foreground shape and broad shape-pose variation make VEIS useful.
- Dataset: The dataset contains 61,305 frames from multi-class complex scenes and single-class simple scenes.The two scene types contribute 30,180 and 31,125 frames, respectively.
5 Experiments
The experiments evaluate synthetic-data training on real Cityscapes and CamVid images, compare against supervised, weakly supervised, and domain-adaptation baselines, and test the shape-versus-texture motivation. Across datasets, the proposed foreground detection and background segmentation framework improves over synthetic-only segmentation and can outperform some methods using real annotations or adaptation data.
- Setup: Models are trained with synthetic GTA5 and VEIS data and evaluated on real Cityscapes and CamVid images.The study also reports fully supervised SYNTHIA and VIPER models and uses DeepLab and Mask R-CNN components.
- Methods: The evaluated methods include synthetic-only baselines, GTA5+VEIS training, pseudo-label variants, and the proposed detection-based foreground model.The proposed method combines a detection-based model for foreground classes with DeepLab for background classes.
- Cityscapes: On Cityscapes, combining GTA5 and VEIS within the proposed framework yields a significant improvement, especially for foreground classes.The paper attributes this to detection focusing on natural object shape rather than photo-realistic texture.
- Cityscapes: On Cityscapes, the proposed method outperforms the compared weakly supervised and domain adaptation baselines without seeing Cityscapes images during training.Using unlabeled Cityscapes images for pseudo-labeling improves performance further.
- CamVid: On CamVid, the proposed method outperforms the weakly supervised method and synthetic-trained DeepLab, and surpasses some fully supervised methods using annotated CamVid images.The paper reports the same overall trend across these comparisons.
- Shape versus Texture: Shape is more robust to the synthetic-to-real shift than texture for foreground objects.A silhouette classifier reached 70.0% accuracy, versus 95.1% with textured objects; textured training also produced a larger real–synthetic accuracy gap.
6 Conclusion
The approach leverages synthetic urban-scene data by handling foreground classes through detection, outperforming standard synthetic-only segmentation and domain-adaptation techniques.
- Foreground classes are handled in a detection-based manner to leverage synthetic training data for urban-scene semantic segmentation.
- The approach outperforms standard semantic segmentation trained on synthetic data and state-of-the-art domain adaptation techniques.
- The framework is orthogonal to domain adaptation, enabling future investigation of combining both approaches.