Source-linked AI summary
Differential Treatment for Stuff and Things: A Simple Unsupervised Domain Adaptation Method for Semantic Segmentation
Zhonghao Wang, Mo Yu, Yunchao Wei, Rogerio Feris, Jinjun Xiong, Wen-mei Hwu, Thomas S. Huang, Humphrey Shi
TL;DR
Unsupervised semantic segmentation must bridge synthetic-source and real-target domains despite costly target annotation and unstable global adversarial alignment. The paper proposes SIM, which differentially matches stuff classes and individual thing instances, combines it with self-supervised learning, and reports new state-of-the-art results on two adaptation tasks.
Problem
Unsupervised adaptation must address domain shift between annotated synthetic source data and unlabeled real target data, while global adversarial alignment lacks differential treatment for stuff and thing features.
Method
SIM aligns target stuff representations with source class features and target thing instances with their most similar source instances, using explicit distance minimization and high-confidence self-supervised predictions.
Results
The combined SIM and self-training approach achieves new state-of-the-art segmentation performance on GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes.
Takeaways & Limitations
Differential treatment of stuff and thing features, combined with self-training, supports more accurate adaptation and stabilizes GAN training at longer iterations.
Abstract
from arXiv · showhide
We consider the problem of unsupervised domain adaptation for semantic segmentation by easing the domain shift between the source domain (synthetic data) and the target domain (real data) in this work. State-of-the-art approaches prove that performing semantic-level alignment is helpful in tackling the domain shift issue. Based on the observation that stuff categories usually share similar appearances across images of different domains while things (i.e. object instances) have much larger differences, we propose to improve the semantic-level alignment with different strategies for stuff regions and for things: 1) for the stuff categories, we generate feature representation for each class and conduct the alignment operation from the target domain to the source domain; 2) for the thing categories, we generate feature representation for each individual instance and encourage the instance in the target domain to align with the most similar one in the source domain. In this way, the individual differences within thing categories will also be considered to alleviate over-alignment. In addition to our proposed method, we further reveal the reason why the current adversarial loss is often unstable in minimizing the distribution discrepancy and show that our method can help ease this issue by minimizing the most similar stuff and instance features between the source and the target domains. We conduct extensive experiments in two unsupervised domain adaptation tasks, i.e. GTA5 to Cityscapes and SYNTHIA to Cityscapes, and achieve the new state-of-the-art segmentation accuracy.
1. Introduction
The paper targets domain shift in unsupervised semantic segmentation by differentially aligning stuff regions and thing instances, stabilizing adversarial training, and adding self-supervised learning. It evaluates the approach on GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes, achieving new state-of-the-art performance on both tasks.
- Motivation: Pixel-level annotation is costly, motivating adaptation from annotated synthetic source data to unlabeled real target data.Annotating one Cityscapes image takes 1.5 hours on average, totaling 7,500 hours for 5,000 images.
- Motivation: Global adversarial alignment is suboptimal because stuff regions and thing instances differ in appearance variance and require different treatments.The paper also observes that the global adversarial signal can be weak and unstable for segmentation.
- Stuff Instance Matching: SIM aligns stuff using global class features while matching each target thing instance to its most similar source instance.This design accounts for diverse appearances among instances and avoids globally over-aligning thing features.
- Stuff Instance Matching: SIM applies an L1 loss to minimize distances between target features and their closest source stuff or instance counterparts.The stated goal is a more accurate adaptation direction than rough distribution matching from adversarial cross-entropy alone.
- Self-supervised learning: Self-supervised learning uses high-confidence predicted segmentations to train the model and enhance alignment for stuff and thing categories.The method combines this strategy with SIM for target-domain adaptation.
- Results: The method achieves new state-of-the-art performance on GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes adaptation tasks.The paper also reports that SIM addresses performance drops at longer training iterations with few additional computations.
2. Related works
Related work organizes semantic-segmentation domain adaptation around image-level, feature-level, and label-level transfer. These approaches respectively modify image appearance, align extracted feature distributions, or generate target pseudo-labels without human annotation.
- Overview: Semantic-segmentation domain adaptation commonly transfers knowledge from synthetic source datasets to real target datasets by aligning feature distributions.The task is harder than classification because it involves pixel-level classification and structured contextual semantic adaptation.
- Image-level transferring: Image-level transfer changes color, illumination, or other stylization factors to make source and target images visually similar.The related methods include transferring appearances between domains or to a neutral domain.
- Feature-level transferring: Feature-level transfer matches extracted source and target feature distributions to reduce domain-shift discrepancies affecting target performance.Representative approaches use GAN structures, image reconstruction, output-space adaptation, or channel-wise feature alignment.
- Label-level transferring: Label-level transfer assigns pseudo-labels to target images using source-domain knowledge, enabling self-supervised adaptation without human target annotations.Examples include class-balanced self-training and joint self-learning with image transfer.
3. Background
The paper frames unsupervised semantic-segmentation domain adaptation as learning from labeled source images and unlabeled target images, with feature alignment used to reduce domain shift.
- Unsupervised adaptation trains on source images with pixel-level annotations and predicts pixel-level labels for unlabeled target images.
- A feature extractor and classification head produce semantic predictions, supervised by cross-entropy loss on the annotated source domain.
- The generator G combines the feature extractor F and classification head C, while discriminator D distinguishes the generated output’s domain.
- Adversarial optimization aims to reduce the feature-distribution discrepancy between source and target domains.
4. Proposed Methods
The proposed SIM framework reduces intra-class domain shift by matching stuff classes globally and thing instances to similar source counterparts, then adds confident target pseudo-labels for self-supervision.
- Stuff and instance matching (SIM): SIM gives stuff regions and thing instances different alignment strategies to reduce intra-class domain shift.Stuff uses class-level representations, whereas things use instance-level representations.
- Stuff and instance matching (SIM): Stuff representations average features belonging to the same background semantic class across image width and height.
- Stuff and instance matching (SIM): Target stuff representations are generated from predicted label maps and matched to the closest intra-class source stuff representation.
- Stuff and instance matching (SIM): Foreground instance masks use disconnected regions within each foreground class to create instance-level feature representations without instance annotations.
- Stuff and instance matching (SIM): Target instance features are pulled toward the closest intra-class source instance feature sample.
- Self-supervised learning with SIM: The self-supervised stage assigns pseudo-labels to high-confidence target pixels and adds target-domain segmentation loss during retraining.
- Self-supervised learning with SIM: Training proceeds in two steps: adversarially train without self-supervision, then reinitialize and retrain using pseudo-labels and augmented SIM losses.
5. Implementation
The implementation uses convolutional segmentation and discriminator networks with specified optimization settings, and reports comparisons for GTA5-to-Cityscapes adaptation.
- Segmentation Network: The segmentation network uses an ImageNet-pretrained ResNet-101 backbone with five convolutional stages and dilated convolutions.
- Discriminator: The discriminator has five convolutional layers with channels {64, 128, 256, 512, 1} and no batch normalization.
- Evaluation: Table 1 compares state-of-the-art results for adapting GTA5 to Cityscapes.
- Optimization: SGD with Nesterov momentum 0.9 and weight decay 5×10−4 optimizes the segmentation network.
- Optimization: The initial segmentation learning rate is 2.5×10−4 and polynomially decays with power 0.9.
6. Experiments
Experiments evaluate the method on GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes adaptation, finding state-of-the-art performance with both ResNet-101 and VGG16 backbones. Ablations show progressive gains from adversarial adaptation, image transfer, SIM, and self-supervised learning.
- Experimental setup: Cityscapes evaluation uses 500 validation images with 19 labels, while GTA5 provides 24,966 synthetic images sharing all 19 evaluation classes.
- GTA5 to Cityscapes: The method achieves state-of-the-art performance on GTA5-to-Cityscapes adaptation with both ResNet-101 and VGG16 backbones.
- GTA5 to Cityscapes: On GTA5-to-Cityscapes, mIoU rises from 36.6 with source-only training to 41.4 with adversarial adaptation and 44.9 after image transfer.
- GTA5 to Cityscapes: Adding SIM and self-supervised learning further raises GTA5-to-Cityscapes mIoU to 49.2, with pseudo-label confidence thresholds selected per class.
- Hyperparameters: Hyperparameter experiments find best performance at λci = 0.01 and w = 50 stored semantic feature samples.
- SYNTHIA to Cityscapes: For SYNTHIA-to-Cityscapes, mIoU improves from 38.6 with source-only training to 46.0 with adversarial adaptation and transferred images, 47.1 with SIM, and 52.1 after self-supervised retraining.
7. Conclusions
The paper proposes SIM for synthetic-to-real semantic-segmentation adaptation by treating stuff regions and thing instances differently and matching target features to closest source counterparts. Combined with self-training, SIM achieves new state-of-the-art performance.
- SIM differentiates adaptation for stuff regions and thing instances according to their different appearance variance.
- SIM explicitly minimizes distances to the closest source-domain stuff and instance features, helping stabilize GAN training at longer iterations.
- Combining SIM with self-training achieves new state-of-the-art performance for synthetic-to-real semantic-segmentation adaptation.