PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully Convolutional Adaptation Networks for Semantic Segmentation

Yiheng Zhang, Zhaofan Qiu, Ting Yao, Dong Liu, Tao Mei

arXiv:1804.08286v1cs.CV

TL;DR

Expensive pixel-level annotation motivates synthetic training data, but domain shift limits transfer from game images to real scenes. FCAN combines pixel-space appearance adaptation with adversarial representation adaptation for semantic segmentation, achieving strong unsupervised transfer results, including 47.53% mIoU on BDDS with an ensemble.

  • Problem

    Pixel-level annotation is expensive, while models trained on synthetic images may generalize poorly to real images because of domain shift.

  • Method

    FCAN combines Appearance Adaptation Networks, which transfer domain style in pixel space, with Representation Adaptation Networks, which learn domain-invariant representations adversarially.

  • Results

    47.53% mIoU was achieved on BDDS by the FCAN(MS+EN) ensemble in an unsupervised setting.

  • Takeaways & Limitations

    Appearance-level and representation-level adaptation are complementary, with their combination producing increasingly accurate cross-domain segmentation results.

Abstract

from arXiv · show

The recent advances in deep neural networks have convincingly demonstrated high capability in learning vision models on large datasets. Nevertheless, collecting expert labeled datasets especially with pixel-level annotations is an extremely expensive process. An appealing alternative is to render synthetic data (e.g., computer games) and generate ground truth automatically. However, simply applying the models learnt on synthetic images may lead to high generalization error on real images due to domain shift. In this paper, we facilitate this issue from the perspectives of both visual appearance-level and representation-level domain adaptation. The former adapts source-domain images to appear as if drawn from the "style" in the target domain and the latter attempts to learn domain-invariant representations. Specifically, we present Fully Convolutional Adaptation Networks (FCAN), a novel deep architecture for semantic segmentation which combines Appearance Adaptation Networks (AAN) and Representation Adaptation Networks (RAN). AAN learns a transformation from one domain to the other in the pixel space and RAN is optimized in an adversarial learning manner to maximally fool the domain discriminator with the learnt source and target representations. Extensive experiments are conducted on the transfer from GTA5 (game videos) to Cityscapes (urban street scenes) on semantic segmentation and our proposal achieves superior results when comparing to state-of-the-art unsupervised adaptation techniques. More remarkably, we obtain a new record: mIoU of 47.5% on BDDS (drive-cam videos) in an unsupervised setting.

1. Introduction

Pixel-level labeling is costly, motivating synthetic training data, but domain shift can make models trained on games perform poorly on real scenes. The paper proposes FCAN to address this through appearance- and representation-level adaptation.

  • Pixel-level annotation remains expensive and time-consuming, motivating automatically labeled synthetic data from computer games.
  • Models trained on synthetic game images may fail to segment semantic categories correctly in real street-view videos because of domain shift.
  • Unsupervised domain adaptation uses labeled source examples and many unlabeled target examples to reduce prediction error on target data.
  • Appearance adaptation transfers source-image content toward target-domain style, while representation adaptation learns domain-invariant features through adversarial domain-discriminator training.
  • FCAN is proposed for semantic segmentation by combining Appearance Adaptation Networks with Representation Adaptation Networks.

2. Related Work

Prior work covers semantic segmentation architectures, weak supervision, and deep domain adaptation, while unsupervised cross-domain segmentation remains relatively underexplored. FCAN differs from FCNWild by addressing domain shift at both appearance and representation levels.

  • Semantic segmentation: Semantic segmentation predicts pixel-level semantic labels and has advanced through multi-scale features, context preservation, and post-processing methods.
  • Semantic segmentation: Because pixel-level annotations are expensive, prior segmentation work also explores weak supervision using bounding boxes, image tags, and auxiliary categories.
  • Deep domain adaptation: Deep domain adaptation transfers models from labeled source domains to target domains through unsupervised, supervised, or semi-supervised settings.
  • Deep domain adaptation: Unsupervised adaptation for semantic segmentation is seldom investigated, with FCNWild identified as the closest related approach.
  • Deep domain adaptation: FCAN addresses domain shift through both visual appearance-level and representation-level adaptation, unlike FCNWild's fully convolutional adversarial training alone.

3. Fully Convolutional Adaptation Networks (FCAN) for Semantic Segmentation

FCAN combines pixel-space appearance adaptation with adversarial representation adaptation to reduce domain shift in semantic segmentation. AAN preserves source semantics while importing target-domain style, whereas RAN learns representations that confuse a domain discriminator.

  • Framework overview: FCAN combines Appearance Adaptation Networks (AAN) and Representation Adaptation Networks (RAN) for unsupervised semantic-segmentation domain adaptation.AAN addresses visual appearance, while RAN addresses representation-level invariance.
  • Representation Adaptation Networks (RAN): RAN learns domain-invariant representations adversarially by training a discriminator to classify the domain of image regions associated with feature-map spatial units.The adversarial objective guides representation learning across source and target domains.
  • Appearance Adaptation Networks (AAN): AAN recombines source-image content with target-domain style so adapted images retain source semantics while appearing target-like.It begins from white noise and iteratively produces an adaptive image using source content and target-image style.
  • Appearance Adaptation Networks (AAN): AAN preserves source content by minimizing weighted feature-map distances between the source image and adaptive image across selected CNN layers.Different convolutional layers represent different semantic levels and receive different weights.
  • Appearance Adaptation Networks (AAN): AAN represents target style through correlations between CNN response maps and matches the adaptive image's correlations to target-domain statistics.These correlations are treated as spatially agnostic style measurements.
  • Representation Adaptation Networks (RAN): RAN uses multi-scale representations through parallel dilated convolutions implemented by an extended Atrous Spatial Pyramid Pooling strategy.The strategy accounts for objects of varying sizes without relying on multiple resized input images.

4. Implementation

The implementation uses ResNet-based convolutional networks, weighted multi-layer features for appearance adaptation, and dilated multi-scale branches for segmentation and adversarial learning. Training proceeds through source-domain pre-training followed by joint fine-tuning with segmentation and adaptation objectives.

  • Appearance Adaptation: AAN uses five ResNet-50 convolutional layers—conv1, res2c, res3d, res4f, and res5c—to represent different feature scales.These layers are selected because their representations generally have high capability at their respective scales.
  • Appearance Adaptation: The AAN style term is treated as a small appearance adjustment so semantic content remains precise for pixel-level segmentation.The content-style tradeoff is controlled by α, with style empirically treated as a delta-like adjustment.
  • Representation Adaptation: The FCN uses a dilated ResNet-101 backbone, feeds res5c features into segmentation and adversarial branches, and adds Pyramid Pooling to the supervised branch.The adversarial branch uses k = 4 parallel dilated convolutional layers to produce multiple feature maps.
  • Training: RAN is pretrained on the source domain with segmentation loss, then jointly fine-tuned using segmentation and adaptation objectives with mini-batch SGD.The initial learning rate is 0.0025, momentum is 0.9, weight decay is 0.0005, batch size is 6, and maximum training length is 30k iterations.

5. Experiments

Experiments evaluate FCAN for unsupervised transfer from GTA5 to Cityscapes and BDDS, testing appearance adaptation, representation adaptation, component contributions, and comparisons with prior methods. Across these evaluations, adaptation improves segmentation, with strongest reported results from FCAN variants and multi-scale or ensemble settings.

  • Datasets and metrics: Experiments transfer semantic segmentation from GTA5 game videos to Cityscapes urban street scenes, using IoU per category and mean IoU across categories.GTA5 provides automatically generated pixel-level ground truth, while Cityscapes is used as the target domain.
  • Evaluation of AAN: 46.21% mIoU is achieved when source images receive AAN followed by RAN, outperforming settings without AAN.Applying AAN to source, target, or both domains improves direct FCN segmentation, and RAN further improves each setting.
  • Evaluation of AAN: AAN preserves semantic content while transferring visual appearance: GTA5 images become bleaker and less saturated, whereas Cityscapes images become brighter and more saturated.These examples demonstrate appearance-level transfer between the two domains.
  • Ablation study: 46.6% mIoU is reached by the full FCAN configuration after ABN, adversarial designs, ASPP, and AAN are added.ABN raises mIoU from 29.15% to 35.51%; ADA, Conv, and ASPP contribute 5.78%, 1.88%, and 1.64%, respectively, while AAN adds 1.79%.
  • Comparisons with state-of-the-art: 47.75% mIoU is obtained with the multi-scale FCAN setting on Cityscapes, which achieves the best performance in 17 of 19 categories.The comparison evaluates FCAN against state-of-the-art unsupervised domain adaptation methods.
  • Results on BDDS: 47.53% mIoU is achieved on BDDS by the FCAN(MS+EN) ensemble, compared with 43.35% for FCAN and 45.47% for FCAN(MS).The ensemble fuses models derived from ResNet-101, ResNet-152, and SENet.

6. Conclusion

FCAN addresses semantic-segmentation domain adaptation through complementary visual appearance-level and representation-level adaptation. Experiments validate the approach on GTA5-to-Cityscapes transfer and report state-of-the-art performance on GTA5-to-BDDS transfer.

  • FCAN combines Appearance Adaptation Networks and Representation Adaptation Networks for semantic-segmentation domain adaptation.AAN transfers domain style to produce invariant appearances, while RAN guides representation learning adversarially toward domain-invariant representations.
  • Experiments on transfer from GTA5 game videos to Cityscapes urban street-view scenes validate the proposal and its analysis.
  • The method achieves new state-of-the-art performance when transferring game videos to BDDS drive-cam videos.
Loading 1804.08286v1…