PaperlayerView exact revision on arXiv

Source-linked AI summary

Bidirectional Learning for Domain Adaptation of Semantic Segmentation

Yunsheng Li, Lu Yuan, Nuno Vasconcelos

arXiv:1904.10620v1cs.CV

TL;DR

Dense pixel-level labels are expensive, while synthetic-to-real domain mismatch limits semantic segmentation performance without target labels. The paper introduces a bidirectional framework that alternately improves image translation and segmentation adaptation, using self-supervision and semantic consistency. Experiments report state-of-the-art performance across multiple adaptation tasks and network backbones, with large margins over other methods.

  • Problem

    Unsupervised semantic segmentation adaptation must transfer knowledge from labeled synthetic images to unlabeled real images despite visual domain mismatch and costly target annotation.

  • Method

    A closed-loop bidirectional framework alternately learns image translation and segmentation adaptation, combining self-supervised target pseudo-labeling with a semantic-consistency perceptual loss.

  • Results

    State-of-the-art performance is reported for multiple adaptation tasks and different networks, outperforming other methods by a large margin.

  • Takeaways & Limitations

    Bidirectional training improves real-domain segmentation performance by allowing the translation and segmentation adaptation models to promote each other.

Abstract

from arXiv · show

Domain adaptation for semantic image segmentation is very necessary since manually labeling large datasets with pixel-level labels is expensive and time consuming. Existing domain adaptation techniques either work on limited datasets, or yield not so good performance compared with supervised learning. In this paper, we propose a novel bidirectional learning framework for domain adaptation of segmentation. Using the bidirectional learning, the image translation model and the segmentation adaptation model can be learned alternatively and promote to each other. Furthermore, we propose a self-supervised learning algorithm to learn a better segmentation adaptation model and in return improve the image translation model. Experiments show that our method is superior to the state-of-the-art methods in domain adaptation of segmentation with a big margin. The source code is available at https://github.com/liyunsheng13/BDL.

1. Introduction

The paper frames unsupervised segmentation adaptation as necessary because dense target labels are costly and synthetic-to-real domain mismatch harms performance. It proposes a closed-loop framework in which translation and segmentation models alternatively improve one another through self-supervision and semantic consistency.

  • Motivation: Dense pixel-level annotation is costly, motivating synthetic labeled data for semantic segmentation.Computer-generated annotations provide an alternative, but synthetic images may not transfer well to real images.
  • Problem setting: Unsupervised adaptation targets semantic segmentation from labeled synthetic source images to unlabeled real target images.The task focuses on adapting large rendered datasets such as SYNTHIA or GTA5 to Cityscapes without target labels.
  • Existing limitation: Earlier pipelines translate source images first and then adapt segmentation features, but segmentation quality depends heavily on translation quality.Sequential processing can leave the segmentation model vulnerable when image-to-image translation fails.
  • Proposed framework: The proposed bidirectional system forms a closed loop that alternatively learns the image translation and segmentation adaptation models.The two directions are translation-to-segmentation and segmentation-to-translation, with both models promoting one another.
  • Self-supervised adaptation: Self-supervised learning uses high-confidence predicted labels from unlabeled real images to incrementally align source and target features.Low-confidence predictions are excluded, and improved segmentation feeds back into the translation model.
  • Segmentation-to-translation: A new perceptual loss supervises image translation by enforcing semantic consistency between images and their translated versions.The updated segmentation adaptation model provides the supervision for the backward direction.
  • Results: The method achieves state-of-the-art performance on SYNTHIA/GTA5-to-Cityscapes adaptation and outperforms other methods by a large margin.The framework is also reported to generalize across different backbone networks.

2. Related Work

Related work addresses domain adaptation through distribution matching, adversarial feature alignment, image translation, and self-training. The paper distinguishes its approach by jointly and iteratively coupling translation with segmentation adaptation rather than relying on a fixed sequential pipeline.

  • Distribution and feature alignment: Domain adaptation methods commonly reduce distribution discrepancy using statistical moments or adversarial feature alignment.Moment matching includes mean and covariance, while adversarial methods train features to fool a discriminator.
  • Distribution and feature alignment: Statistical matching can be insufficient when feature distributions are not Gaussian, limiting alignment based only on mean and covariance.
  • Synthetic data: Semantic segmentation adaptation increasingly uses automatically labeled virtual data because collecting dense labels for real images is labor-intensive.GTA5 and SYNTHIA are cited as popular synthetic city-street datasets.
  • Segmentation adaptation: Feature-level segmentation adaptation methods use global, local, multilevel, or class-specific alignment strategies.Examples include global and local alignment, multiple discriminators, and separate foreground-background treatment.
  • Translation-based adaptation: Image-to-image translation reduces visual domain discrepancy before feature alignment, as in Cycada and DCAN.These methods achieve state-of-the-art performance but remain dependent on translation quality.
  • Positioning: The paper addresses sequential pipeline failure by introducing bidirectional learning between image translation and segmentation adaptation.Unlike related approaches, the two modules iteratively provide feedback to each other.
  • Positioning: Related segmentation self-training and segmentation-assisted translation methods differ because they omit either target-domain adaptation or image translation.The paper combines these directions for unsupervised synthetic-to-real segmentation adaptation.
  • Bidirectional learning: Bidirectional learning has also been applied to language translation and image generation, where it can improve performance over one-direction learning.

3. Method

The method jointly adapts image translation and semantic segmentation through alternating bidirectional updates, with self-supervised target pseudo-labels improving the segmentation model and feeding back into translation.

  • 3. Method: Unsupervised adaptation uses source images S with labels YS and unlabeled target images T to train segmentation for T.The framework retains separate image-translation and segmentation-adaptation subnetworks.
  • 3.1. Bidirectional Learning: The training process alternates updates to F and M in a closed loop, progressively reducing the domain gap and allowing each model to promote the other.The system comprises image translation and segmentation adaptation components, with the training procedure described by Algorithm 1.
  • 3.1. Bidirectional Learning: The forward direction trains F, translates source images into S′, and trains M using S′ with YS together with target images T.Because F does not change source labels, S′ retains YS for supervised segmentation loss; adversarial loss aligns source and target feature representations.
  • 3.1. Bidirectional Learning: The backward direction uses the updated segmentation model M to compute perceptual loss for improving the image-translation model F.GAN and reconstruction losses are added, and the losses are computed symmetrically for S → T and T → S.
  • 3.2. Self-supervised Learning for Improving M: Self-supervised learning selects high-confidence target pixels as pseudo-labels and uses them with segmentation loss, while excluding low-confidence predictions.The pseudo-labeled subset T_ssl may initially be empty and can grow as M produces more confident predictions; this SSL is combined with adversarial learning.
  • 3.3. Network and Loss Function: The architecture combines a segmentation model, an image-translation model, adversarial discriminators, reconstruction losses, and perceptual losses that preserve semantic consistency.The perceptual loss connects translation and segmentation by requiring corresponding original, translated, and reconstructed images to retain semantic consistency.

4. Discussion

The experiments show that bidirectional learning improves segmentation adaptation, while self-supervised learning further strengthens the model and supports later iterations. Threshold and iteration analyses identify 0.9 and N=2 as effective choices for the reported experiments.

  • Bidirectional Learning without SSL: M(0) is the baseline lower bound, while combining translation and segmentation adaptation yields about 1.6% further improvement after one iteration.M(1) and M(0)(F(1)) each improve by more than 7% over M(0); M(1)(F(1)) adds about 1.6% further improvement.
  • Bidirectional Learning with SSL: Self-supervised learning improves mIoU by 4.5% for M(2)(F(1)).The analysis attributes this improvement to retaining well-aligned source-target data while further aligning the remaining data through adversarial learning.
  • Bidirectional Learning with SSL: 48.5 mIoU is reached after SSL in the second iteration, despite an initial drop from 47.2 to 44.3.The SSL-enhanced result outperforms the first-iteration result reported in the ablation.
  • Hyperparameter Learning: A 0.9 threshold is selected because lower thresholds introduce incorrect predictions, whereas higher thresholds make SSL more sensitive to usable pixel quantity.Soft thresholding performs worse, probably because labeling noise remains insufficiently alleviated by lower weights.
  • Hyperparameter Learning: N=2 is selected because mIoU and the SSL pixel ratio nearly stop increasing at N=2 or 3.The iteration number is chosen from the point at which the predicted-label pixel ratio stabilizes alongside segmentation adaptation learning.

5. Experiments

Experiments evaluate the method on synthetic-to-real adaptation using GTA5 or SYNTHIA as source data and Cityscapes as target data, with multiple segmentation backbones. Results compare against state-of-the-art methods and quantify remaining distance from supervised upper bounds.

  • Comparison Setup: The study compares the proposed method with state-of-the-art domain-adaptation methods across GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes tasks.The SYNTHIA evaluation reports results for both 13-category and 16-category settings where applicable.
  • Network Architecture: DeepLab V2 with ResNet101 and FCN-8s with VGG16 serve as the segmentation models, initialized from ImageNet-pretrained networks.The segmentation adaptation discriminator uses five convolutional layers.
  • Dataset: Experiments use GTA5 and SYNTHIA as source datasets and Cityscapes as the target dataset for synthetic-to-real adaptation.GTA5 contains 24,966 images, and the evaluation uses common categories with Cityscapes.
  • Results: 6% improvement over Cycada and DCAN is achieved with ResNet101, while self-training with VGG16 yields 10.4% improvement in the reported comparison.The results also state that feature-level alignment alone remains about 5% worse than the proposed results.
  • Results: At least 4% improvement over most compared methods is achieved on SYNTHIA-to-Cityscapes despite its larger domain gap and incomplete category overlap.Road, sidewalk, and car are reported as more than 10% worse in this setting, affecting SSL confidence.
  • Performance Gap to Upper Bound: A performance gap of at least 16.6 remains between the proposed method and supervised upper-bound results, leaving substantial room for improvement.The reported upper bounds include 65.1 and 60.3 for GTA5-to-Cityscapes with ResNet101 and VGG16, respectively.

6. Conclusion

The paper concludes that bidirectional learning with self-supervised learning improves segmentation performance on real data. Experiments report state-of-the-art results across multiple tasks and network architectures.

  • 6. Conclusion: The proposed method combines bidirectional learning with self-supervised learning for segmentation adaptation.The conclusion identifies this combination as the paper's central method.
  • 6. Conclusion: Extensive experiments show improved segmentation performance for real datasets when the model is trained bidirectionally.The conclusion directly attributes the reported improvement to bidirectional training.
  • 6. Conclusion: The method achieves state-of-the-art results across multiple tasks and different network architectures.This conclusion summarizes the reported experimental scope rather than a single benchmark.
Loading 1904.10620v1…