PaperlayerView exact revision on arXiv

Source-linked AI summary

ROAD: Reality Oriented Adaptation for Semantic Segmentation of Urban Scenes

Yuhua Chen, Wen Li, Luc Van Gool

arXiv:1711.11556v2cs.CV

TL;DR

Synthetic-to-real domain differences can make segmentation models overfit synthetic styles and generalize poorly to real urban scenes. ROAD-Net combines target guided distillation with spatial-aware adaptation, achieving reported gains on Cityscapes experiments, including 39.4% mean IoU for GTAV-to-Cityscapes adaptation.

  • Problem

    Synthetic images reduce annotation costs but differ from real images in style and distribution, causing segmentation performance to drop on real urban scenes.

  • Method

    ROAD-Net uses target guided distillation on real images and spatial-aware adaptation based on urban-scene layout, integrated with conventional segmentation networks.

  • Results

    39.4% mean IoU is reported for GTAV-to-Cityscapes adaptation, and the method outperforms competing methods in additional SYNTHIA-to-Cityscapes experiments.

  • Takeaways & Limitations

    The two modules improve ROAD-Net's reported generalizability when adapting semantic segmentation models from synthetic to real urban scenes.

  • Takeaways & Limitations

    Improvement is not significant for some categories, such as poles and traffic signs, whose sparse pixels make results less stable.

Abstract

from arXiv · show

Exploiting synthetic data to learn deep models has attracted increasing attention in recent years. However, the intrinsic domain difference between synthetic and real images usually causes a significant performance drop when applying the learned model to real world scenarios. This is mainly due to two reasons: 1) the model overfits to synthetic images, making the convolutional filters incompetent to extract informative representation for real images; 2) there is a distribution difference between synthetic and real data, which is also known as the domain adaptation problem. To this end, we propose a new reality oriented adaptation approach for urban scene semantic segmentation by learning from synthetic data. First, we propose a target guided distillation approach to learn the real image style, which is achieved by training the segmentation model to imitate a pretrained real style model using real images. Second, we further take advantage of the intrinsic spatial structure presented in urban scene images, and propose a spatial-aware adaptation scheme to effectively align the distribution of two domains. These two modules can be readily integrated with existing state-of-the-art semantic segmentation networks to improve their generalizability when adapting from synthetic to real urban scenes. We evaluate the proposed method on Cityscapes dataset by adapting from GTAV and SYNTHIA datasets, where the results demonstrate the effectiveness of our method.

1. Introduction

ROAD-Net addresses synthetic-to-real urban-scene segmentation by targeting both synthetic-style overfitting and cross-domain distribution mismatch. It combines target guided distillation with spatial-aware adaptation and reports 39.4% mean IoU on Cityscapes when adapting from GTAV.

  • Motivation: Synthetic urban-scene data lowers pixel-level annotation costs but still causes significant performance drops on real images because of domain differences.The identified problems are synthetic-style overfitting in convolutional filters and mismatch between synthetic and real data distributions.
  • Approach: ROAD-Net extends conventional segmentation networks with target guided distillation for real style orientation and spatial-aware adaptation for real distribution orientation.The two modules are designed to address the representation and distribution problems respectively.
  • Approach: Target guided distillation trains the segmentation model on real images to imitate a pretrained real style model, encouraging real-image filters and discriminative outputs.This uses a distillation objective rather than relying only on synthetic inputs.
  • Approach: Spatial-aware adaptation exploits intrinsic urban-scene geometry to ease alignment between domains under substantial spatial visual variance.Objects in central regions are typically smaller than objects in outer regions, complicating direct distribution alignment.
  • Results: 39.4% mean IoU is reported on Cityscapes when adapting from GTAV, while experiments using SYNTHIA also outperform competing methods.The modules can be integrated with existing state-of-the-art semantic segmentation networks.

2. Related Works

The related work situates ROAD-Net within CNN-based semantic segmentation, domain adaptation, synthetic-data learning, and knowledge distillation. Prior work addresses feature alignment and synthetic-data transfer, while ROAD-Net applies distillation to real-style filter learning for segmentation.

  • Semantic Segmentation: CNN-based semantic segmentation replaced manually designed features with learned representations and includes architectures using per-pixel classification and dilated convolutions.The related work traces this progression from FCN-style segmentation toward larger receptive fields.
  • Domain Adaptation: Domain adaptation addresses performance degradation when training and test data do not follow the same distribution.The standard machine-learning i.i.d. assumption does not always hold in real-world scenarios.
  • Domain Adaptation for Semantic Segmentation: Semantic-segmentation domain adaptation includes adversarial feature alignment, curriculum-learning approaches, and other methods for handling domain shift.The cited literature focuses largely on aligning features across domains.
  • Learning Using Synthetic Data: Synthetic-data research has trained detectors on virtual images and pursued domain-invariant detection across synthetic and alternative domains.These studies primarily concern object detection rather than the urban-scene segmentation setting addressed here.
  • Other Related Works: ROAD-Net is inspired by model distillation and scene-understanding research but uses distillation to learn real-style convolutional filters for semantic segmentation.This distinguishes its use of knowledge transfer from earlier compression and cross-modality applications.

3. Reality Oriented Adaptation Networks

ROAD-Net addresses synthetic-to-real urban-scene segmentation through target guided distillation and spatial-aware adaptation. The modules orient representations toward real image style and align domain distributions using spatial structure.

  • Target Guided Distillation: ROAD-Net uses target guided distillation to reduce convolutional filters’ overfitting to synthetic image style.The segmentation model uses target real images to imitate a pretrained real style model.
  • Target Guided Distillation: Target guided distillation feeds unlabeled real images to both the segmentation and frozen pretrained models, matching their corresponding feature-map activations.The pretrained model shares the segmentation backbone’s structure, and the loss uses Euclidean distance over activations.
  • Target Guided Distillation: Freezing convolutional layers can limit segmentation capacity, whereas target guided distillation allows all weights to adapt while softly imitating the ImageNet model.The paper reports that freezing layers prevents corruption to some extent but remains inferior to target guided distillation.
  • Spatial-Aware Adaptation: Spatial-aware adaptation divides scene images into regions and aligns source and target pixel-level features from corresponding spatial regions.This exploits spatial patterns such as smaller central objects and larger outer-region objects, while allowing traditional domain-alignment losses within each region.
  • Network Overview: ROAD-Net combines segmentation, distillation, and spatial-aware losses and can be integrated with conventional networks including DilatedNet, DeepLab, and PSPNet.The joint objective minimizes the segmentation loss together with the two adaptation modules.

4. Experimental Results

Experiments evaluate ROAD-Net for unsupervised adaptation from synthetic GTAV or SYNTHIA images to real Cityscapes scenes. The two modules improve segmentation across backbones, partitions, and source datasets, while gains vary by category.

  • Experiment Setup: The protocol uses labeled synthetic source images and unlabeled Cityscapes target images, evaluating on Cityscapes validation data.GTAV provides 25,000 synthetic images; Cityscapes provides 2,975 training and 500 validation images across 19 categories.
  • Experiment Setup: ROAD-Net combines target guided distillation and spatial-aware adaptation, with DeepLab V2 and PSPNet used as base segmentation networks.DeepLab V2 uses VGG-16, whereas PSPNet uses ResNet-101.
  • Experimental Results: 35.9% mean IoU is achieved by ROAD-Net with DeepLab, compared with 21.9% for NonAdapt; the two modules individually reach 31.1% and 32.6%.With PSPNet, ROAD-Net reaches 39.4% mean IoU, and qualitative results show more reasonable spatial layouts.
  • Limitations: Improvements are not significant for some categories, such as pole and traffic sign, which contain relatively few pixels and therefore have less stable results.Cityscapes validation contains 1.8M traffic-light pixels versus 345M road pixels.
  • Real Style Orientation: Target guided distillation achieves the best improvement among the compared alternatives, including frozen layers and source distillation.All alternatives outperform NonAdapt, while target guided distillation uses real images during distillation to reduce overfitting to synthetic style.
  • Real Distribution Orientation: 3 × 3 spatial partitioning gives the best results, while every spatial partition outperforms the 1 × 1 global-alignment baseline.The 2 × 1 partition benefits from vertical semantic structure, whereas 2 × 2 adds little improvement because urban scenes are generally vertically symmetric.
  • Cross-Dataset Comparison: ROAD-Net achieves 35.9% mean IoU against state-of-the-art methods on GTAV-to-Cityscapes adaptation and outperforms competing methods on SYNTHIA-to-Cityscapes adaptation.For GTAV adaptation, the reported improvements are +7.0% and +8.8% over the compared methods.

5. Conclusion

ROAD-Net addresses synthetic-to-real urban scene segmentation with target guided distillation and spatial-aware adaptation. Evaluations on Cityscapes using GTAV and SYNTHIA as source domains demonstrate its effectiveness.

  • ROAD-Net is a model for urban scene semantic segmentation learned from synthetic data.
  • Target guided distillation adapts style from real images by imitating a pretrained network, while spatial-aware adaptation reduces distribution mismatch using layout information.
  • The two modules can integrate with different semantic segmentation networks to improve generalizability when applying them to a new domain.
  • Experiments on Cityscapes use GTAV and SYNTHIA as source domains, and benchmark results demonstrate ROAD-Net's effectiveness.
Loading 1711.11556v2…