Label-Driven Reconstruction for Domain Adaptation in Semantic Segmentation

1 Introduction

• Pixel-wise annotations are labor-intensive, and models trained on synthetic data can degrade when applied directly to real images because of domain shift.The introduction identifies appearance and spatial-layout differences as examples of domain shift.Source: §1 · p. 1
• Source-to-target translation can introduce bias, especially when the source dataset is larger, while feature-space alignment alone does not ensure joint-distribution consistency.The introduction links these limitations to false positives and false negatives in target predictions, particularly with layout or structure discrepancies.Source: §1 · p. 2
• Target-to-source translation is proposed to reduce translation bias and computation relative to source-to-target translation.The framework can train segmentation using original source images and their corresponding ground-truth labels.Source: §1 · pp. 2–3
• The framework reports new state-of-the-art performance for synthetic-to-real semantic segmentation, evaluated on GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes.The experiments also include ablations of framework components.Source: §1 · p. 3
• A label-driven reconstruction module reconstructs both source and target images from predicted labels to encourage semantic consistency.The paper describes reconstructing directly from label space rather than feature space, with the aim of bringing same-category cross-domain features closer.Source: §1 · p. 3

2 Related Work

• Synthetic-data segmentation models can degrade on real datasets because of domain shift, while supervised approaches depend on costly human-annotated masks.Weakly supervised annotations and densely labeled synthetic datasets are described as alternatives to pixel-level human labeling.Source: §2 · p. 4
• Unsupervised domain adaptation addresses domain discrepancy when target-domain labels are unavailable.The passage distinguishes supervised, semi-supervised, and unsupervised adaptation by target-label availability.Source: §2 · p. 4
• Source-to-target translation is used for pixel-level adaptation, but the larger source dataset can make translated images biased; target-to-source translation is proposed as a more efficient alternative that retains original source images and labels.The paper positions this direction as addressing translation uncertainty while reducing pixel-level domain discrepancy.Source: §2 · p. 4
• Structured-output adaptation can provide weak supervision to target images, but may yield meaningless predictions when source and target layouts differ greatly.The proposed reconstruction network is presented as a way to enforce semantic consistency between target images and predicted labels.Source: §2 · pp. 4–5
• Reconstruction-based adaptation often reconstructs images from feature representations, whereas this paper reconstructs source and target images from predicted labels.Prior examples include approaches based on feature representations and disentangled structure and texture representations.Source: §2 · p. 5

3.1 Overview

• Target images are translated into source-like images, while original source images and translated target images are both segmented and reconstructed from their predicted labels.The framework uses a cycle-reconstruction loss to measure reconstruction error and guide segmentation predictions.Source: §3.1 · p. 6, Figure 3 · p. 7

3.2 Target-to-source Translation

• Target-to-source translation maps target images toward the source-image distribution and is more computationally efficient than source-to-target translation when the source set is much larger.The approach also retains full supervision from original source images and their labels while reducing bias in translated images.Source: §3.2 · p. 6

3.3 Semantic Segmentation

• A shared segmentation network predicts labels for source and source-like images, with adversarial alignment applied to their segmentation outputs as domain-invariant features.The segmentation objective uses source labels and pseudo-labels for translated target images.Source: §3.3 · pp. 6–7

3.4 Image Reconstruction from the Label Space

• The reconstruction network maps predicted label representations back to images, and a cycle-reconstruction loss matches image and reconstruction features to encourage semantic consistency.The loss combines VGG perceptual and discriminator feature-matching terms; the paper states that it brings translated-target features closer to source features with the same label.Source: §3.4 · pp. 7, 9
• Hard conversion of predicted scores to categorical labels is non-differentiable, so the method uses a continuous, differentiable Boltzmann-probability approximation for label-space reconstruction.The temperature parameter τ controls this approximation.Source: §3.4 · p. 8

4.4 GTA5 →Cityscapes

• On GTA5→Cityscapes, VGG16 results are best among existing state-of-the-art methods, while ResNet101 remains below two newer models; gains over source-only are 12.9% (ResNet101) and 25.7% (VGG16).Source: Table 1 · p. 10, §4.4 · p. 12

4.1 Datasets

• GTA5 (24,966 labeled images) and SYNTHIA (9,400) serve as sources; Cityscapes training is unlabeled and its 500 labeled validation images are test data.Source: §4.1 · pp. 9–10

4.2 Network Architecture

• FCN-VGG16 and DeepLab-ResNet101 are baselines; both share a five-layer discriminator, while reconstruction uses nine ResNet blocks and three multi-scale PatchGAN discriminators.Source: §4.2 · pp. 10–11

4.5 SYNTHIA →Cityscapes

• On SYNTHIA→Cityscapes, average performance exceeds competitors, including on both backbones; ResNet101 gains are 6.4% over ASN and 1.7% over BDL.Source: Table 2 · p. 11, §4.5 · p. 13

4.3 Implementation Details

• Training uses batch size one; translation is trained first, reconstruction is pretrained on source images paired with labels, and optimizer and schedule choices differ by backbone.Source: §4.3 · pp. 11–12

4.6 Ablation Study

• τ = 0.001 achieves the best performance and is used throughout to approximate semantic label maps.Source: Table 4 · p. 13, §4.6 · p. 14
• Across GTA5 and SYNTHIA, target-to-source translation improves mIoU over source-to-target; adding reconstruction yields further gains, including 1.3% on GTA5 and 1.1% and 1.0% on SYNTHIA.Source: Table 3 · p. 12, §4.6 · pp. 13–14
• Label-space reconstruction benefits semantic consistency over feature-space reconstruction; VGG perceptual and discriminator feature-matching losses play complementary roles.Source: §4.6 · p. 14, Table 5 · p. 14, Table 6 · p. 14

• Target-to-source translation reduces image-translation bias and computational cost compared with source-to-target translation.Source: §5 · p. 14
• Reconstructing source and target images from label space is intended to bring cross-domain features of the same category closer.Source: §5 · p. 14
• Experiments show significant improvement when adapting from GTA5 and SYNTHIA to Cityscapes.Source: §5 · p. 14

1.1 Target-to-source Translation

• Target-to-source translation maps target images into the source domain, reducing translation bias and computation while allowing segmentation training on original source images and labels.Source: §1.1 · p. 19

1.2 Network Architecture

• The reconstruction model uses three identical 70×70 PatchGAN discriminators, each with five convolution layers.Source: §1.2 · p. 19

1.3 Implementation Details

• The learning rate is linearly decayed to zero with step size 100, and momentum {0.5, 0.999} is used.Source: §1.3 · p. 19

1.4 mIoU Gap

• The comparison measures mIoU gap to the fully supervised Oracle; on GTA5, the method trails with ResNet101 but outperforms them by a large margin with VGG16.Source: Table 1 · p. 20, §1.4 · p. 20

1.5 Qualitative Comparison

• Qualitative figures compare target images and ground truth with source-to-target baseline and target-to-source/reconstruction predictions; the paper reports component contributions to overall improvement.Source: §1.5 · pp. 20–21