PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly Supervised Adversarial Domain Adaptation for Semantic Segmentation in Urban Scenes

Qi Wang, Junyu Gao, Xuelong Li

arXiv:1904.09092v1cs.CV

TL;DR

Synthetic-to-real urban segmentation is difficult because pixel-level annotation is costly and domain differences cause models trained on synthetic images to perform poorly on real scenes. The paper proposes weakly supervised adversarial adaptation with a detection-segmentation model and pixel- and object-level domain classifiers. It reports a new mIoU record for the same cross-domain urban-scene segmentation problem.

  • Problem

    Synthetic data reduces annotation effort, but differences from real scenes limit segmentation performance when models transfer from synthetic source data to real target scenes.

  • Method

    The method combines an asymmetric detection-and-segmentation model with pixel-level and object-level domain classifiers trained through adversarial optimization.

  • Results

    The proposed method yields a new record of mIoU accuracy for cross-domain fully urban scene labeling.

  • Takeaways & Limitations

    The approach reduces the domain gap by learning domain-invariant features through pixel- and object-level adversarial training.

Abstract

from arXiv · show

Semantic segmentation, a pixel-level vision task, is developed rapidly by using convolutional neural networks (CNNs). Training CNNs requires a large amount of labeled data, but manually annotating data is difficult. For emancipating manpower, in recent years, some synthetic datasets are released. However, they are still different from real scenes, which causes that training a model on the synthetic data (source domain) cannot achieve a good performance on real urban scenes (target domain). In this paper, we propose a weakly supervised adversarial domain adaptation to improve the segmentation performance from synthetic data to real scenes, which consists of three deep neural networks. To be specific, a detection and segmentation ("DS" for short) model focuses on detecting objects and predicting segmentation map; a pixel-level domain classifier ("PDC" for short) tries to distinguish the image features from which domains; an object-level domain classifier ("ODC" for short) discriminates the objects from which domains and predicts the objects classes. PDC and ODC are treated as the discriminators, and DS is considered as the generator. By adversarial learning, DS is supposed to learn domain-invariant features. In experiments, our proposed method yields the new record of mIoU metric in the same problem.

I. INTRODUCTION

The paper addresses semantic segmentation from synthetic to real urban scenes, where pixel-level annotation is costly and domain differences degrade transfer. It proposes weakly supervised adversarial adaptation using detection, segmentation, and domain-classification components.

  • Semantic segmentation requires predicting a label for every pixel in fully labeled urban scenes.
  • Pixel-wise urban datasets contain no more than 10,000 images, and this amount is insufficient for some practical applications.
  • Weakly supervised methods using image-level or object-level labels mainly address salient foreground objects in simple scenes, not full urban scenes with occlusion and background objects.
  • Synthetic and real images differ in textures, architectural styles, and road materials, causing models trained on synthetic images to perform poorly on real scenes.
  • The proposed setting uses pixel- and object-level labels in the source domain but only object-level labels in the target domain.
  • The method combines a detection-and-segmentation model with pixel-level and object-level domain classifiers trained adversarially.
  • The proposed method reports a new mIoU record for cross-domain fully urban scene labeling.

II. RELATED WORK

Related work spans fully and weakly supervised segmentation, general domain adaptation, and domain adaptation for semantic segmentation. The paper motivates structured object-level features and hierarchical adversarial adaptation for synthetic-to-real urban scenes.

  • Semantic segmentation: FCN-based semantic segmentation methods include recurrent conditional random fields, encoder-decoder architectures, dilated convolutions, and pyramid pooling.
  • Semantic segmentation: Weakly supervised segmentation methods reduce annotation costs by learning from image-level or bounding-box labels.
  • Domain adaptation: Domain adaptation methods minimize domain gaps through adversarial training, reconstruction, or Maximum Mean Discrepancy.
  • Proposed framework: The proposed framework alternately adversarially optimizes DS, PDC, and ODC, using source images for pixel- and object-level training and target images for object-level training.
  • Domain adaptation for semantic segmentation: Prior segmentation adaptation methods reduce domain gaps at pixel, image, or superpixel levels but do not focus on structured object-level features.
  • Problem formulation: The adaptation goal is to reduce the domain gap between a synthetic source domain and a real-world target domain while predicting target pixel-wise score maps.

A. Weak supervision for segmentation

The weak-supervision design combines detection and segmentation in an asymmetric two-stream model. Object-level and contextual features supplement local pixel features, while source and target domains receive different training supervision.

  • FCN-based methods emphasize local features such as texture and color, which differ substantially between domains.
  • The asymmetric DS model feeds source images through both streams but uses target images only in the detection stream during training.
  • At testing, images enter only DS’s segmentation stream, which predicts the pixel-wise score map.
  • DS combines FCN-8s and SSD-512 with shared early convolutional layers.
  • SSD-512 learns structured object features and contextual features from large unstructured objects such as roads, skies, and buildings.
  • The DS loss combines supervised pixel-wise cross-entropy with MultiBox detection losses for source and target object annotations.

B. Adversarial domain adaptation

The approach uses adversarial domain classifiers to make DS learn features that support segmentation while becoming domain-invariant at pixel and object levels. PDC distinguishes source from target pixels, and its adversarial objective is optimized alternately with DS to reduce training oscillation.

  • Adversarial framework: Adversarial learning treats PDC as a discriminator and DS as the generator, encouraging DS to learn domain-invariant features.The discriminator distinguishes domains while DS is trained to confuse it.
  • Pixel-level adaptation: PDC distinguishes source and target domains from segmentation-stream feature maps using a two-channel score map at the original image resolution.Its inputs come from the segmentation network, and soft-max produces per-pixel domain confidence scores.
  • Pixel-level adaptation: The PDC inverse loss guides DS toward domain confusion rather than domain discrimination.The inverse objective is defined after the standard PDC loss and is used to train DS.
  • Optimization: A domain confusion objective replaces direct optimization of the inverse formulation because jointly optimizing the two objectives is prone to oscillation.The practical training objective is formed from the PDC loss and its inverse.
  • Optimization: PDC and DS parameters are updated alternately: first PDC is optimized with DS fixed, then DS is optimized with PDC fixed.This alternating procedure minimizes the respective objectives for the two models.

D. Object-level adaptation

Object-level adaptation extends domain classification to object features, jointly encoding object category and domain. ROI pooling extracts object representations, while the classifier uses a 2×N output to separate source and target class scores.

  • D. Object-level adaptation: ODC distinguishes each object’s category and domain, providing object-level guidance for learning discriminative features.Unlike traditional domain classifiers, it predicts both the object class and its domain.
  • D. Object-level adaptation: ROI pooling extracts object features from a 32×32 feature map because higher-layer spatial outputs are too small for accurate object-feature extraction.The ROI locations are provided by ground truth, and the selected feature map has H × W = 32 × 32.
  • D. Object-level adaptation: ODC maps each pooled object feature to a 2×N confidence vector whose halves represent N classes in the source and target domains.The first N entries score source-domain classes, while the remaining N entries score target-domain classes.
  • D. Object-level adaptation: ODC represents object labels as 2N-dimensional one-hot vectors, assigning class c to different domain-specific positions.Source objects use the first domain block, while target objects use the second block.
  • D. Object-level adaptation: The ODC objective is designed to classify object features jointly by category and domain.The formulation follows the domain-specific one-hot label construction.

S and OODC

The object-level discriminator uses inverse labels to guide SSD-512 toward domain-invariant object features. Full-model training alternates discriminator updates with DS updates for adversarial adaptation.

  • S and OODC: ODC inverse labels guide SSD-512 to learn domain-invariant object features.The inverse ODC loss is computed from domain-swapped labels.
  • S and OODC: The object-level domain confusion objective is used to avoid oscillation during adversarial optimization.It parallels the domain-confusion strategy introduced for pixel-level adaptation.
  • S and OODC: The full model jointly trains DS, PDC, and ODC with separate objectives for the two discriminators and the DS generator.The objectives are specified for the complete three-network system.
  • S and OODC: Training alternates between updating PDC and ODC with DS fixed and updating DS with both discriminators fixed.The two discriminators are optimized simultaneously in the first step.

E. Network Architecture

The architecture combines SSD-512-based detection and FCN-8s-based segmentation within DS, while PDC and ODC receive segmentation and ROI-pooled features. The section also introduces the experimental comparison setup.

  • E. Network Architecture: DS attaches SSD-512 to VGG-19’s conv4 4∗ layer, using a 512-channel feature map at 1/16 input resolution.The detection stream is connected at the specified VGG-19 layer.
  • E. Network Architecture: FCN-8s combines conv3 4∗, conv4 4∗, and conv5 4∗ outputs to predict the final segmentation map.These layers provide the multiscale inputs to the segmentation stream.
  • E. Network Architecture: PDC receives features from the conv5 4∗ layer, whereas ODC receives ROI-pooled features.The two discriminators therefore operate on segmentation-stream and object-level representations, respectively.
  • E. Network Architecture: The experiments report implementation details, model results, and comparisons with existing methods for the same cross-domain segmentation problem.The passage introduces the experimental-results section without providing numerical outcomes.

A. Datasets

Experiments use GTA5 and SYNTHIA as synthetic source domains and Cityscapes as the real target domain, with comparisons against prior adaptation methods and stepwise variants.

  • A. Datasets: GTA5 and SYNTHIA provide synthetic source data, while Cityscapes supplies real urban-scene validation data.The experiments use GTA5 and SYNTHIA as source domains and test all models on the Cityscapes validation set.
  • A. Datasets: GTA5 contains 24,996 scenes from the Grand Theft Auto V game, with annotations compatible with Cityscapes and CamVid.
  • A. Datasets: SYNTHIA-RAND-CITYSCAPES contributes 9,400 images across 13 classes for the synthetic-to-Cityscapes adaptation setting.
  • A. Datasets: Bounding-box labels for DS and ODC are generated because the three datasets do not provide object-level annotations directly.Background-object boxes come from pixel-wise ground truth, while some occluded foreground boxes require detection-based generation.
  • A. Datasets: The study compares direct DS training, pixel-level adaptation with PDC, full adaptation with ODC, and a ResNet-152-initialized variant against FCN Wld and CDA.

C. GTA5 →Cityscapes

For GTA5-to-Cityscapes adaptation, hierarchical adversarial training improves segmentation over prior methods, with the ResNet-152 variant achieving the highest reported mean IoU.

  • C. GTA5 →Cityscapes: ODC contributes a 3.1% improvement over DS+PDC, compared with PDC’s 2.3% improvement over DS.
  • C. GTA5 →Cityscapes: Visualizations compare input images, ground truth, and predictions from DS, DS+PDC, Full, and Full† on Cityscapes validation examples.The reported examples show fewer mistakes after PDC and more elaborate pedestrian segmentation after object-level adversarial learning.

D. SYNTHIA →Cityscapes

For SYNTHIA-to-Cityscapes adaptation, the proposed method obtains the best reported performance, while bounding-box-based detection and hierarchical adversarial learning outperform single-FCN alternatives.

  • D. SYNTHIA →Cityscapes: 35.7% mean IoU is the best reported performance for SYNTHIA-to-Cityscapes adaptation.
  • D. SYNTHIA →Cityscapes: Full improves over CDA’s 29.0% by 6.7 percentage points, or 23.1% relative mean IoU.
  • D. SYNTHIA →Cityscapes: Sky IoU increases from approximately 6% to approximately 53% for the no-adaptation comparisons.
  • D. SYNTHIA →Cityscapes: ODC contributes 4.3% versus PDC’s 2.9%, while Full† slightly decreases performance from 35.7% to 35.2%.The text attributes the reduction to the larger domain gap between SYNTHIA and Cityscapes.
  • D. SYNTHIA →Cityscapes: Bounding-box labels are converted into overlapping multi-label coarse maps, with single-label predictions selected from the resulting outputs.The source domain uses single-label predictions, while the target domain uses multi-label predictions in the last two FCN experiments.
  • D. SYNTHIA →Cityscapes: DS and Full outperform single-FCN baselines because detection supplies structured inter- and intra-object features and Full adds hierarchical adversarial learning.

F. DS vs. Mask RCNN

The paper compares DS with Mask RCNN as two multi-task detection-and-segmentation architectures. Mask RCNN achieves higher segmentation performance, while DS is faster at test time because detection is auxiliary rather than required.

  • Runtime trade-off: DS is faster during testing because it does not require object detection, while Mask RCNN must first detect bounding boxes.Detection remains auxiliary for DS during inference but is essential for Mask RCNN.
  • Performance comparison: Mask RCNN outperforms DS in the no-adaptation experiments on synthetic-to-real data.Both models are trained on synthetic data and tested on real data.
  • Architectural differences: DS uses an asymmetric architecture in which detection and segmentation share only backbone features, whereas Mask RCNN uses a sequential architecture.Mask RCNN’s segmentation module directly exploits detection features; DS’s modules do not share task-specific features.
Loading 1904.09092v1…