Learning Semantic Segmentation from Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach

Yuhua Chen; Wen Li; Xiaoran Chen; Luc Van Gool

Source-linked AI summary

Learning Semantic Segmentation from Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach

Yuhua Chen, Wen Li, Xiaoran Chen, Luc Van Gool

arXiv:1812.05040v2Dec 12, 2018cs.CV

TL;DR

Synthetic-to-real semantic segmentation is hindered by the domain gap and the difficulty of collecting and densely annotating diverse urban-scene images. GIO-Ada uses geometric information for input- and output-level adaptation, and improves performance across synthetic-to-real evaluations, including a +16% mIoU gain over the non-adaptive baseline on Virtual KITTI→KITTI.

Problem
The domain gap limits semantic segmentation models trained on synthetic data from generalizing to real-world applications, while collecting and densely annotating diverse urban-scene images is difficult.
Method
GIO-Ada uses semantic and depth maps to guide synthetic-to-real image transformation, jointly predicts depth and semantics, and adversarially aligns their joint output space.
Results
The method significantly improves performance over the non-adaptive baseline and methods without geometric information; on Virtual KITTI→KITTI, it improves mIoU by +16% over the non-adaptive baseline.
Takeaways & Limitations
Geometric information from synthetic data is effective for cross-domain semantic segmentation, supporting adaptation at both the input and output levels.
Takeaways & Limitations
The protocol assumes labeled synthetic images with semantic and depth annotations, but only unlabelled images are available in the target domain.

Abstract

from arXiv · show

Recently, increasing attention has been drawn to training semantic segmentation models using synthetic data and computer-generated annotation. However, domain gap remains a major barrier and prevents models learned from synthetic data from generalizing well to real-world applications. In this work, we take the advantage of additional geometric information from synthetic data, a powerful yet largely neglected cue, to bridge the domain gap. Such geometric information can be generated easily from synthetic data, and is proven to be closely coupled with semantic information. With the geometric information, we propose a model to reduce domain shift on two levels: on the input level, we augment the traditional image translation network with the additional geometric information to translate synthetic images into realistic styles; on the output level, we build a task network which simultaneously performs depth estimation and semantic segmentation on the synthetic data. Meanwhile, we encourage the network to preserve correlation between depth and semantics by adversarial training on the output space. We then validate our method on two pairs of synthetic to real dataset: Virtual KITTI to KITTI, and SYNTHIA to Cityscapes, where we achieve a significant performance gain compared to the non-adapt baseline and methods using only semantic label. This demonstrates the usefulness of geometric information from synthetic data for cross-domain semantic segmentation.

1. Introduction

Synthetic data reduces the cost of obtaining dense segmentation annotations, but appearance differences cause models trained synthetically to perform poorly in real environments. GIO-Ada uses geometric information to adapt both image inputs and joint depth-semantic outputs, improving cross-domain segmentation.

Motivation: Dense urban-scene annotation is expensive because collecting diverse images and labeling every pixel are labor-intensive.The challenge spans varied weather, cities, and other conditions.
Motivation: Synthetic images provide inexpensive per-pixel annotations, but their appearance mismatch with real scenes causes a significant performance drop after transfer.Computer graphics enables images and annotations from virtual 3D environments.
Motivation: Synthetic environments offer geometric cues such as surface normals, optical flow, and depth at lower cost than equivalent real-world information.Prior methods typically use only synthetic images and semantic labels.
Motivation: Geometry and semantics are strongly correlated in urban scenes, while this relationship suffers less domain shift than image appearance.Examples include flat roads, distant skies, and vertical poles across synthetic and real data.
Approach and validation: GIO-Ada adapts at input and output levels by using depth and semantics for realistic image translation, joint depth-semantic prediction, and adversarial output alignment.The method is evaluated on Virtual KITTI→KITTI and SYNTHIA→Cityscapes, with significant gains over non-adaptive and non-geometric methods.

2. Related Works

Prior work addresses segmentation, domain shift, and joint depth-semantic learning separately. GIO-Ada differs by using depth-semantic correlation specifically to perform domain adaptation.

Semantic Segmentation: Modern semantic-segmentation methods commonly use fully convolutional networks modified for pixel-wise prediction.Examples include DilatedNet, DeepLab, and PSP-Net.
Synthetic Data: Synthetic datasets such as GTAV, SYNTHIA, and Virtual KITTI offer a lower-cost alternative to expensive pixel-wise annotation.These datasets are generated using computer-graphics techniques.
Domain Adaptation: Domain adaptation seeks to mitigate performance loss caused by distribution mismatch between training and test data.The literature includes both conventional and CNN-based approaches.
Cross-Domain Segmentation: Cross-domain segmentation methods address domain shift through feature alignment, curriculum style learning, cycle consistency, and output-space alignment.These approaches form the closest related work to the paper's setting.
Geometry-Guided Segmentation: Joint depth estimation and semantic segmentation exploit the strong correlation between these two scene-understanding tasks.Prior techniques include hierarchical CRFs, uncertainty modeling, cross-modality influence, distillation, recursive estimation, and task attention.
Paper Positioning: GIO-Ada differs by leveraging depth-semantic correlation for domain adaptation rather than only for mutually beneficial multi-task learning.Multi-task learning produces multiple outputs from a single network.

3. Geometrically Guided Input-Output Adaptation

GIO-Ada uses synthetic depth and semantic information to adapt across domains at both the image-input and task-output levels. Its jointly trained networks transform synthetic images, predict depth and semantics, and align outputs between synthetic and real domains.

The unsupervised adaptation setting uses synthetic images with semantic and depth labels as source data, while real images form the unlabeled target domain.
Input-Level Adaptation: GIO-Ada performs input-level adaptation by feeding synthetic images, depth maps, and semantic labels to an image transform network that produces realistic-looking images while preserving task information.The discriminator distinguishes transformed synthetic images from real images, while task-network gradients guide preservation of useful information.
Input-Level Adaptation: Depth is included because geometric information is inexpensive in synthetic environments and is highly correlated with semantic information, but is often discarded by pixel-level adaptation methods.
Output-Level Adaptation: The task network shares a feature extractor and uses separate heads to jointly predict semantic segmentation and depth for transformed synthetic or real images.Source-domain supervision uses cross-entropy for segmentation and an ℓ1 loss for depth estimation.
Output-Level Adaptation: Output-level adaptation uses a discriminator to distinguish joint depth-and-semantic predictions from transformed synthetic images and real images, aligning outputs while retaining task correlation.The combined input- and output-level modules are trained jointly with adversarial optimization.
During testing, only the task network processes real images; the image-transform network and discriminators are removed.

4. Experiments

Experiments evaluate GIO-Ada across synthetic-to-real urban-scene settings and ablations of its input- and output-level adaptation modules. Results show that geometric information improves translation consistency and segmentation performance, with complementary gains across both adaptation levels.

Experiment Settings: Experiments use Virtual KITTI→KITTI and SYNTHIA→Cityscapes for synthetic-to-real semantic segmentation evaluation.Virtual KITTI and SYNTHIA provide publicly available depth information; KITTI and Cityscapes serve as real target datasets.
Virtual KITTI→KITTI: +16% mIoU over the non-adaptive baseline is achieved on Virtual KITTI→KITTI, while input- and output-level adaptation contribute +8.4% and +12.5%, respectively.The two adaptation levels are reported as complementary when integrated.
Qualitative Analysis: Qualitative results show that geometric cues improve segmentation of geometrically structured objects and preserve semantic-geometric consistency during image translation.Compared with CycleGAN, the proposed model avoids examples such as buildings or trees appearing in the sky and poles turning into trees.
Input-Level Ablation: Adding depth and semantic labels to input-level translation yields +6.7% and +7.2% individually, while combining them produces +8.4% over the baseline.CycleGAN improves the baseline by +2.3%, whereas task-network guidance achieves +6%.
Output-Level Ablation: Joint depth-and-segmentation output alignment achieves +12.5%, exceeding separate semantic and depth alignment, which reaches +8.8%.The results support modeling correlation between the two output tasks rather than optimizing them independently.
SYNTHIA→Cityscapes: On SYNTHIA→Cityscapes, input-level, output-level, and integrated adaptation improve the baseline by +10.5%, +12.0%, and +14.0%, respectively.The model also outperforms competing methods by a healthy margin in this setting.

5. Conclusion

GIO-Ada leverages geometric information from synthetic data at both input and output levels to address cross-domain semantic segmentation. The paper validates this approach on two dataset pairs and illustrates qualitative improvements over non-adaptive results and CycleGAN.

5. Conclusion: GIO-Ada uses depth and semantic annotations to guide input-level image transformation, then applies adversarial training to a joint depth-semantic output space.These two adaptation levels target domain shift in raw pixels and task outputs, respectively.
5. Conclusion: The method was experimentally validated on two synthetic-to-real dataset pairs for cross-domain semantic segmentation.The supplied conclusion states that the results demonstrate GIO-Ada’s effectiveness using geometric information from virtual data.
5. Conclusion: Figure 3 compares input images, non-adaptive segmentation results, and results produced by GIO-Ada on KITTI.The label maps use Cityscapes’ color encoding scheme.
5. Conclusion: Figure 4 compares synthetic input images with CycleGAN translations and GIO-Ada translations during input-level adaptation.The figure presents the three conditions from left to right: input synthetic image, CycleGAN, and the proposed method.