Source-linked AI summary
Learning to Adapt Structured Output Space for Semantic Segmentation
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, Manmohan Chandraker
TL;DR
Pixel-level supervision is costly, and segmentation models may not generalize across unseen domains. The paper uses adversarial learning on structured segmentation outputs, with discriminators at multiple feature levels, and reports favorable results across synthetic-to-real and cross-city settings.
Problem
Semantic segmentation needs costly pixel-level ground truth and can generalize poorly when source and target image domains differ.
Method
The method adapts semantic segmentation through adversarial learning in output space and extends it with multi-level discriminators operating at different feature levels.
Results
The method performs favorably against baseline and state-of-the-art methods across synthetic-to-real and cross-city domain adaptation settings.
Takeaways & Limitations
Structured output-space adaptation can align scene layout and local context between source and target images while improving adaptation performance.
Abstract
from arXiv · showhide
Convolutional neural network-based approaches for semantic segmentation rely on supervision with pixel-level ground truth, but may not generalize well to unseen image domains. As the labeling process is tedious and labor intensive, developing algorithms that can adapt source ground truth labels to the target domain is of great interest. In this paper, we propose an adversarial learning method for domain adaptation in the context of semantic segmentation. Considering semantic segmentations as structured outputs that contain spatial similarities between the source and target domains, we adopt adversarial learning in the output space. To further enhance the adapted model, we construct a multi-level adversarial network to effectively perform output space domain adaptation at different feature levels. Extensive experiments and ablation study are conducted under various domain adaptation settings, including synthetic-to-real and cross-city scenarios. We show that the proposed method performs favorably against the state-of-the-art methods in terms of accuracy and visual quality.
1. Introduction
Semantic segmentation models require costly pixel-level annotation and can fail across domain gaps. The paper addresses this by adapting structured segmentation outputs adversarially, including at multiple feature levels.
- Motivation: CNN segmentation models may generalize poorly when training and test images differ in appearance, scene, weather, or lighting.Such domain gaps arise across cities and within the same city.
- Motivation: Re-annotating pixel-level ground truths for different scenarios incurs prohibitively high labor cost.
- Motivation: Feature adaptation is challenging for segmentation because high-dimensional features must encode appearance, shape, and context.
- Method: The proposed method uses adversarial learning in output space to align predicted label distributions between source and target domains.A segmentation model predicts outputs while a discriminator distinguishes source from target segmentation outputs.
- Method and Contributions: Multi-level adversarial learning adapts features at different levels of the segmentation model and improves performance.The approach is evaluated in synthetic-to-real and other domain adaptation settings.
2. Related Work
Prior work addresses domain shift through feature alignment, synthetic data, and adversarial or label-transfer methods. This paper instead exploits the structured spatial and local information in pixel-level predictions for output-space adaptation.
- Semantic Segmentation: Semantic segmentation methods build on fully convolutional networks, context information, and enlarged receptive fields.
- Semantic Segmentation: Synthetic datasets provide cheaper pixel-level annotations but models trained solely on them do not generalize well to real-world data.
- Domain Adaptation: Image-classification domain adaptation commonly aligns feature distributions between source and target images.
- Domain Adaptation: Semantic-segmentation adaptation has used fully convolutional feature adversarial learning, class-wise adversarial learning, and label transfer.
- Novelty: Feature-space adaptation remains challenging for pixel-level tasks, motivating adversarial alignment in the structured output space.Pixel-level predictions contain spatial and local information that can be used for adaptation.
3. Algorithmic Overview
The algorithm trains a segmentation network on labeled source images and uses discriminators to make unlabeled target predictions resemble source predictions. Its objective combines source segmentation loss with target adversarial adaptation.
- Architecture: The adaptation system consists of a segmentation network G and level-specific discriminators D_i.The index i identifies a discriminator level in the multi-level architecture.
- Data Flow: Source images with annotations optimize the segmentation network, while target images produce unlabeled softmax predictions.
- Data Flow: Discriminators receive source and target segmentation predictions and distinguish their domains.
- Objective: The objective combines source cross-entropy segmentation loss with adversarial loss that adapts target predictions toward source-prediction distributions.The adversarial-loss weight λ_adv balances the two losses.
4. Output Space Adaptation
The method adapts semantic segmentation across domains by adversarially aligning predicted outputs rather than only high-dimensional features. Multi-level adversarial learning extends this alignment to predictions from different feature levels.
- Output Space Adaptation: Output-space adaptation aligns predicted label distributions between source and target domains using adversarial learning.A discriminator distinguishes source from target segmentation outputs while the segmentation model is trained adversarially.
- Single-level Adversarial Learning: Segmentation training uses source ground-truth cross-entropy together with an adversarial objective on target predictions.The target prediction is trained to be considered a source prediction by the discriminator.
- Multi-level Adversarial Learning: Multi-level adaptation adds adversarial modules at lower feature levels because output-space back-propagation may not adequately adapt low-level features.Each selected feature level produces a segmentation output before its individual discriminator applies adversarial learning.
- Multi-level Adversarial Learning: The multi-level objective combines losses across feature levels while minimizing source segmentation loss and making target predictions resemble source predictions.The level index identifies the feature level used to predict each segmentation output.
5. Network Architecture and Training
The architecture combines a fully convolutional discriminator with single- or multi-level segmentation outputs and is trained jointly on source and target images. Experiments compare output-space adaptation settings on GTA5-to-Cityscapes using VGG-16 and ResNet-101 models.
- Network Architecture: The discriminator uses five fully convolutional layers to retain spatial information while distinguishing source and target outputs.The layers use 4 × 4 kernels, stride 2, and channels {64, 128, 256, 512, 1}.
- Experiments: Table 1 compares VGG-16 output-space adaptation with prior methods and evaluates ResNet-101 ablations across adaptation components.The listed settings include feature adaptation and single- or multi-level output-space adversarial learning.
- Network Architecture: The multi-level model adds a conv4 feature branch with an ASPP auxiliary classifier and a second discriminator.Two levels are used to balance efficiency and accuracy.
- Network Training: Joint training forwards source images for segmentation loss and source-target outputs for discriminator and adversarial losses.The single-level and multi-level models are trained in one stage.
- Experiments: Table 2 measures the performance gap between adapted and fully supervised oracle models for VGG-based methods and the proposed ResNet model.The comparison examines how closely adapted models approach fully supervised performance.
6. Experimental Results
Experiments evaluate output-space and multi-level adversarial adaptation across synthetic-to-real and cross-city settings. The method generally outperforms feature-space adaptation and improves results over baseline models.
- Experimental settings: Experiments cover GTA5-to-Cityscapes, SYNTHIA-to-Cityscapes, and Cross-City adaptation settings.The evaluations include comparisons with state-of-the-art methods, ablations, and oracle-gap analysis.
- Parameter analysis: Output-space adaptation tolerates a wider range of λadv values, whereas feature-space adaptation is more sensitive to this parameter.The authors note that overly large λadv can propagate incorrect gradients.
- Output-space adaptation: Output-space adaptation performs better than feature-space adaptation in the single-level comparisons.The authors attribute this to matching structured segmentation outputs rather than high-dimensional feature distributions.
- Multi-level adaptation: Multi-level adversarial adaptation improves segmentation accuracy over baseline, feature-space, and single-level models.The multi-level strategy jointly adapts outputs predicted from different feature levels.
- Oracle-gap analysis: The method achieves the smallest mean IoU gap to the fully supervised oracle on SYNTHIA-to-Cityscapes and minimizes that gap below 30%.The comparison uses adapted and oracle results under different baseline models.
- Cross-City adaptation: On Cross-City, the final multi-level model shows consistent improvement across different cities.The setting uses Cityscapes as source data and evaluates adaptation to Rio, Rome, Tokyo, and Taipei.
7. Concluding Remarks
The paper treats segmentation as a structured output and adapts source models through output-space adversarial learning. A multi-level adversarial network further adapts features at different levels and performs favorably against baseline and state-of-the-art methods.
- Concluding Remarks: Segmentation outputs share structural similarities across source and target domains, motivating adversarial adaptation in the output space.The approach targets pixel-level semantic prediction under domain shift.
- Concluding Remarks: A multi-level adversarial network performs output-space domain adaptation at different feature levels.The paper presents this as an enhancement to the adapted model.
- Concluding Remarks: Experimental results favor the proposed method against numerous baseline models and state-of-the-art algorithms.The experiments include GTA5, SYNTHIA, Synscapes, and Cross-City settings.
A. Least Squares Objective
The study replaces the vanilla GAN objective with a least-squares loss and evaluates its effect within the single-level adaptation network. Across GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes settings, LS-GAN achieves higher mean IoU than vanilla GAN.
- The least-squares loss is adopted to improve GAN training stability and generate higher-quality results.The paper motivates this objective using prior claims about more stable training and higher-quality outputs.
- The discriminator labels target samples with z = 0 and source samples with z = 1.These labels define the source-versus-target distinction used in discriminator training.
- Both GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes experiments report higher mean IoU with LS-GAN than with vanilla GAN.The comparison uses a single-level adaptation network and a ResNet-101 backbone.
B. Synscapes
Synscapes is used as a photorealistic synthetic source domain for adaptation to Cityscapes. Because this pairing has a smaller domain gap, the non-adapted model already reaches 45.3% mean IoU, while output-space adaptation further improves performance competitively.
- Synscapes is a photorealistic synthetic street-scene dataset with 25,000 RGB images at 1440 × 720 resolution.Its annotations follow the Cityscapes convention with 19 categories, and the entire dataset is used as the source domain.
- 45.3% mean IoU is achieved without adaptation when transferring from Synscapes to Cityscapes.The paper attributes this relatively strong non-adapted result to the smaller domain gap compared with GTA5 or SYNTHIA.
- Output-space adaptation further improves Synscapes-to-Cityscapes results with both vanilla GAN and LS-GAN objectives.Both objectives are reported to perform competitively in this setting.