PaperlayerView exact revision on arXiv

Source-linked AI summary

No More Discrimination: Cross City Adaptation of Road Scene Segmenters

Yi-Hsin Chen, Wei-Yu Chen, Yu-Ting Chen, Bo-Cheng Tsai, Yu-Chiang Frank Wang, Min Sun

arXiv:1704.08509v1cs.CVcs.AI

TL;DR

Pre-trained road scene segmenters lose accuracy on unseen cities because of dataset bias, while city-specific annotation is costly. The paper adapts them without target annotations by combining Street View temporal static-object priors with global and class-specific adversarial learning. The method improves semantic segmentation across multiple cities and compares favorably with annotated-data approaches.

  • Problem

    Pre-trained road scene segmenters degrade on unseen cities because of dataset bias, while collecting large annotated datasets for each city is costly and time-consuming.

  • Method

    The method adapts a pre-trained segmenter using unlabeled target-city Street View images, temporal static-object priors, soft pseudo labels, and joint global and class-specific domain adversarial learning.

  • Results

    The method improves semantic segmentation across multiple cities and performs favorably against state-of-the-art approaches requiring annotated training data.

  • Takeaways & Limitations

    Cross-city road scene segmenters can be adapted without user annotation or interaction by leveraging temporal Street View information and joint global/class-wise alignment.

  • Takeaways & Limitations

    Static-object priors cannot be extracted from Cityscapes because it lacks temporally paired images.

Abstract

from arXiv · show

Despite the recent success of deep-learning based semantic segmentation, deploying a pre-trained road scene segmenter to a city whose images are not presented in the training set would not achieve satisfactory performance due to dataset biases. Instead of collecting a large number of annotated images of each city of interest to train or refine the segmenter, we propose an unsupervised learning approach to adapt road scene segmenters across different cities. By utilizing Google Street View and its time-machine feature, we can collect unannotated images for each road scene at different times, so that the associated static-object priors can be extracted accordingly. By advancing a joint global and class-specific domain adversarial learning framework, adaptation of pre-trained segmenters to that city can be achieved without the need of any user annotation or interaction. We show that our method improves the performance of semantic segmentation in multiple cities across continents, while it performs favorably against state-of-the-art approaches requiring annotated training data.

1. Introduction

Pre-trained road scene segmenters degrade on unseen cities because of dataset bias, while collecting city-specific annotations is costly. The paper proposes unsupervised cross-city adaptation using temporal Street View information, static-object priors, soft pseudo labels, and joint global/class-wise adversarial learning.

  • Deploying a pre-trained segmenter on a previously unseen city degrades performance because of dataset biases.
  • 25-30% mIoU drop was observed when a Cityscapes-trained segmenter was applied to annotated road scenes from Rome, Rio, Taipei, and Tokyo.The pilot experiment demonstrates the severity of cross-city dataset bias.
  • Collecting large city-specific annotated datasets is time-consuming and expensive; pixel labeling one Cityscapes image takes 90 minutes on average.Existing alternatives using 3D, rendered, or weak labels still require human annotation during data collection.
  • The proposed framework adapts a pre-trained segmenter using only unlabeled target-city images collected through Google Street View’s time-machine feature.Images from the same or nearby locations across different times provide information for extracting static-object priors without human interaction or annotation.
  • The method performs joint global and class-wise adaptation with soft pseudo labels and static-object priors.These components are integrated through an adversarial learning framework to reduce cross-city domain discrimination.

2. Related Work

Related work spans CNN-based semantic segmentation, road-scene datasets, weakly supervised labeling, and domain adaptation. Existing domain-adaptation approaches include discrepancy-based, consistency-based, GAN-based, and adversarial methods, providing context for the paper’s cross-city segmentation goal.

  • CNN-based semantic segmentation produces dense pixel-wise labels, with later methods improving resolution or preserving contextual information using deconvolution and CRF components.
  • Road-scene segmentation datasets use large-scale natural images, transferred 3D labels, or computer-generated images to obtain training supervision.Computer-generated approaches may require computationally expensive construction of CG worlds.
  • Weakly supervised methods reduce labeling requirements using point labels, image-level labels, object-size constraints, object locations, boundaries, or free-form squiggles.These approaches still provide forms of human supervision during data collection or training.
  • Deep domain-adaptation methods align domains using MMD, CMD, cyclic or structured consistency, and related discrepancy objectives.
  • GAN-based approaches adapt domains by generating corresponding instances, while DANN uses adversarial training to suppress domain biases.Adversarial frameworks have also been extended to semantic segmentation.

3. Dataset

The dataset uses Google Street View to collect geographically diverse road scenes and temporally paired images from four cities. It provides unlabeled image pairs for adaptation and separately annotated test images for evaluation.

  • The dataset contains road scenes from Rome, Rio, Tokyo, and Taipei sampled at diverse locations to capture significant visual variation.
  • Google Street View time-machine features provide image pairs from the same locations at different times, enabling static-object prior extraction.
  • 1600 image pairs, totaling 3200 high-quality images, were collected at 1600 locations per city.The images have a resolution of 647 × 1280 pixels.
  • Each city contributes 100 test image pairs with pixel-level ground-truth labels annotated by 15 image-processing experts.The annotations cover 13 major classes, including road, building, vegetation, sky, person, car, bus, motorcycle, and bicycle.
  • Unlike typical datasets focused on nearby locations, this dataset combines cross-city annotated road scenes with temporally paired images for additional learning and processing.

4. Our Method

The method adapts pre-trained road-scene segmenters across cities using unlabeled target images, jointly aligning global and class-wise domains. Google Street View time-machine imagery supplies static-object priors, while pseudo labels support class-specific alignment.

  • Overall Framework: The framework adapts pre-trained segmenters across cities using only unlabeled target-domain images and no user annotation.Source images and labels are available, whereas the target city contributes images only.
  • Global Domain Alignment: Global domain alignment uses adversarial learning to reduce overall appearance differences between source and target cities.A domain discriminator distinguishes source from target feature-map grids, while the feature extractor is updated toward a common feature space.
  • Class-wise Domain Alignment: Class-wise alignment extends adversarial learning to object classes rather than relying only on similar class composition across cities.The method uses class-specific soft labels and pseudo labels to align class-related information between domains.
  • Class-wise Domain Alignment: Grid-level soft labels are computed from the proportion of pixels belonging to each class, with target labels obtained from predicted pixel-wise soft pseudo labels.Each feature-map grid corresponds to multiple image pixels, so pixel probabilities are aggregated over the pixels associated with that grid.
  • Harvesting Static-Object Prior: Static-object priors are extracted from same-location images captured at different times using dense matching and superpixel segmentation.Matched superpixels containing more than k matched pixels are treated as static-object priors, with k fixed to 3.
  • Harvesting Static-Object Prior: The static-object prior refines target soft pseudo labels by suppressing probabilities for non-static interpretations.This provides temporal information for target-domain images without human annotation or interaction.

5. Experiments

Experiments show substantial cross-city performance degradation, while the proposed unsupervised adaptation improves segmentation across city and synthetic-to-real shifts without requiring target annotations.

  • 5.3. Cross-City Adaptation: The unsupervised method achieved comparable performance to fully supervised methods in most Cityscapes-to-Our Dataset cases.The fully supervised baseline serves as an adaptation upper bound.
  • 5.3. Cross-City Adaptation: Global alignment contributed 2.6% mIoU gain on average over four cities, with further contributions from class-wise alignment and static-object priors.The experiment separately evaluates global alignment, class-wise alignment, and the full method against pre-trained and fully supervised baselines.
  • 5.3. Cross-City Adaptation: Figure 5 highlights regions with improved segmentation after adaptation across paired before-and-after rows.The first and third rows show pre-adaptation results, while the second and fourth rows show post-adaptation results.
  • 5.4. Synthetic to Real Adaptation: In SYNTHIA-to-Cityscapes adaptation, global and class-wise alignment achieved 3.1% and 1.9% mIoU gains, respectively, without static-object priors.Cityscapes lacks temporally paired images needed to extract those priors.

6. Conclusion

The paper presents an unsupervised domain-adaptation framework for road-scene segmentation that jointly aligns global and class-wise features and uses temporal static-object priors. It evaluates the method across domain shifts and introduces a paired, annotated multi-city dataset.

  • The framework alleviates cross-domain discrimination in road-scene images across different cities.
  • Domain adversarial learning performs joint global and class-wise alignment using soft labels from source and target domains.
  • Static-object priors are retrieved from images through natural synchronization of static objects over time.
  • The dataset contains road-scene images from four cities, quality annotations, and paired images with temporal information.

A. Visualize GA, CA and Static-Object prior

The visualizations show how global and class-wise alignment reduce source–target feature gaps and how temporal matching identifies static-object regions for adaptation.

  • T-SNE Visualization: With the pre-trained model, t-SNE features show an obvious shift between source and target domains.
  • T-SNE Visualization: Global alignment brings same-label clusters closer, but a domain gap remains after this step.
  • T-SNE Visualization: Adding class-wise alignment nearly eliminates the remaining gap between domains.
  • Harvesting Static-Object Prior: Most regions identified by the static-object prior method belong to static objects.
  • Harvesting Static-Object Prior: The static-object prior pipeline combines dense matching and superpixel segmentation to extract red static-object regions from image pairs across time.

B. Dataset

The dataset is designed for cross-city adaptation with diverse unlabeled temporal image pairs and separately annotated images demonstrating label quality.

  • The dataset is presented through examples intended to demonstrate its usefulness for road-scene segmenter adaptation.
  • Unlabeled Image Pairs: Unlabeled image pairs were collected across cities with diverse visual appearances and contain temporal information for unsupervised adaptation.
  • Labeled Image: Additional annotated images demonstrate the dataset’s label quality.

C. Synthetic to Real Adaptation

The paper illustrates synthetic-to-real adaptation from SYNTHIA to Cityscapes by comparing segmentation results before and after adaptation. The dataset examples include unlabeled image pairs, labeled images, and the corresponding adaptation output.

  • The dataset includes unlabeled image pairs collected from different cities and locations.
  • The dataset also provides high-quality labeled images from different cities.
  • Figure 10 compares SYNTHIA-to-Cityscapes segmentation results before and after adaptation.
Loading 1704.08509v1…