Source-linked AI summary
FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation
Judy Hoffman, Dequan Wang, Fisher Yu, Trevor Darrell
TL;DR
Semantic segmentation models can degrade under domain shifts, while pixel-level target annotations are costly and segmentation adaptation has received limited attention. The paper introduces an unsupervised framework combining global adversarial alignment with category-specific constrained adaptation, and reports improved target performance across synthetic, seasonal, and city shifts without target annotations.
Problem
Semantic segmentation models may perform poorly on new domains, while pixel-wise annotations are expensive and prior domain adaptation research has focused mainly on classification.
Method
The method uses fully convolutional domain-adversarial training for global alignment and constrained multiple-instance learning to transfer category layout from labeled source data to unlabeled targets.
Results
The approach improves target semantic segmentation across synthetic-to-real, cross-season, and cross-city shifts, including a 4.4% raw mIoU gain from domain adversarial training for GTA5→Cityscapes.
Takeaways & Limitations
The results demonstrate the importance of domain adaptation for pixel-level dense prediction and provide a new large-scale real-city driving dataset.
Takeaways & Limitations
The learning scenario assumes strong supervision in the source domain but no supervision in the target domain.
Abstract
from arXiv · showhide
Fully convolutional models for dense prediction have proven successful for a wide range of visual tasks. Such models perform well in a supervised setting, but performance can be surprisingly poor under domain shifts that appear mild to a human observer. For example, training on one city and testing on another in a different geographic region and/or weather condition may result in significantly degraded performance due to pixel-level distribution shift. In this paper, we introduce the first domain adaptive semantic segmentation method, proposing an unsupervised adversarial approach to pixel prediction problems. Our method consists of both global and category specific adaptation techniques. Global domain alignment is performed using a novel semantic segmentation network with fully convolutional domain adversarial learning. This initially adapted space then enables category specific adaptation through a generalization of constrained weak learning, with explicit transfer of the spatial layout from the source to the target domains. Our approach outperforms baselines across different settings on multiple large-scale datasets, including adapting across various real city environments, different synthetic sub-domains, from simulated to real environments, and on a novel large-scale dash-cam dataset.
1. Introduction
Semantic segmentation models face substantial challenges when transferred to new visual domains, where appearance and class-distribution shifts combine with costly pixel-level annotation. The paper proposes unsupervised global and category-specific adaptation and evaluates it across several domain shifts.
- Semantic segmentation supports applications including robotic navigation, self-driving cars, and mapping the natural world.
- New domains introduce appearance, lighting, pose, and class-prevalence shifts that can be especially pronounced for pixel-level localization.Street-scene object distributions may vary across cities and driving routes.
- Pixel-wise annotations are expensive and tedious, motivating transfer of information between related settings.
- The method combines global convolutional domain-adversarial alignment with category-specific constrained multiple-instance learning that transfers source spatial-layout information without target annotations.
- The approach is evaluated on synthetic-to-real, cross-season, and cross-city shifts using GTA5, SYNTHIA, Cityscapes, and the BDDS drive-cam dataset.The authors also introduce BDDS and report improved target segmentation performance without target annotations.
2. Related Work
Prior semantic segmentation work largely assumes supervised or weakly labeled settings, while domain adaptation has focused mainly on image classification. The paper addresses this gap with segmentation-specific adaptation under strong source supervision and no target supervision.
- Semantic Segmentation: Semantic segmentation models commonly use convolutional networks, including FCNs, to map RGB inputs directly to semantic pixel spaces.
- Semantic Segmentation: Weakly supervised segmentation methods use image-level class tags and multiple-instance learning to reinforce confident pixel predictions.
- Semantic Segmentation: The paper instead assumes strong pixel-level supervision in the source domain and no annotations in the target domain.
- Domain Adaptation: Domain adaptation research has primarily targeted image classification through feature confusion, distribution alignment, and related representation-learning approaches.
- Domain Adaptation: Earlier detection-to-segmentation work evaluated FCNs but did not propose a segmentation-specific adaptation method.
3. Fully Convolutional Adaptation Models
The method adapts fully convolutional semantic segmentation models to unlabeled target domains by jointly addressing global and category-specific distribution shifts. It combines pixel-level adversarial alignment with constrained multiple-instance learning that transfers source scene-layout statistics to target images.
- The framework assumes shared source–target label spaces and adapts a source-supervised FCN to an unlabeled target domain.The source model must perform better than chance on the target domain.
- Joint Objective: The joint objective combines supervised source segmentation, global domain alignment, and category-specific target adaptation.Source segmentation remains optimized to avoid diverging too far from the effective source solution.
- Global Domain Alignment: Global adaptation alternates domain-classifier training with representation updates that minimize source–target distances over receptive-field regions.Using spatial units as instances preserves information relevant to pixel prediction better than aligning whole-image representations.
- Category Specific Adaptation: Category-specific adaptation applies constrained multiple-instance learning to target images after global adversarial alignment.The approach generalizes constrained weak learning to modify category-specific network parameters without target annotations.
- Category Specific Adaptation: Source per-class pixel-percentage histograms define target lower, average, and upper size statistics, transferring spatial-layout information across domains.The constraints reflect class-specific scene structure, such as roads occupying more pixels than street signs.
4. Experiments
The experiments evaluate unsupervised adaptation across synthetic-to-real, seasonal, and cross-city shifts using multiple datasets and ablations. Adaptation improves segmentation without target annotations, with gains varying by shift and component.
- Experimental Setup: The evaluation covers cities→cities, season→season, and synthetic→real adaptation across four datasets.Experiments use Cityscapes, SYNTHIA, GTA5, and BDDS, with IoU-based evaluation and source dilation models as baselines.
- Large Shift: Synthetic to Real Adaptation: 4.4% raw and ∼20% relative percentage mIoU improvement comes from domain adversarial training for GTA5→Cityscapes adaptation.Multiple instance loss contributes an additional 1.6% raw and ∼6% relative percentage mIoU improvement.
- Medium Shift: Cross Seasons Adaptation: ∼3 percentage mIoU improvement is obtained on average for season→season adaptation, with higher mIoU for 12/13 object categories.Car shows no improvement, while road is among the categories with some of the largest gains in the fall-to-winter shift.
- Small Shift: Cross City Adaptation: 3.6 percentage mIoU improvement comes mainly from global alignment in Cityscapes cross-city adaptation.Category-specific alignment provides noticeable gains mainly for traffic light, rider, and train.
- BDDS Adaptation: Cityscapes-trained models show noisy or incorrect-context segmentations on BDDS, while adaptation usually produces cleaner qualitative results.The BDDS experiment uses ∼60,000 unlabeled San Francisco images and postpones extensive quantitative evaluation until annotations are available.
5. Conclusion
The paper presents an unsupervised fully convolutional framework for semantic-segmentation domain adaptation. It combines global domain alignment with class-aware spatial-layout transfer and demonstrates effectiveness across city, season, and synthetic-to-real shifts while introducing BDDS.
- Conclusion: The framework uses fully convolutional domain adversarial training for global alignment and class-aware constrained multiple instance loss for spatial-layout transfer.The method targets semantic segmentation without requiring target-domain annotations.
- Conclusion: The experiments demonstrate effectiveness across different cities, seasons, and synthetic-to-real domain shifts.The paper also introduces a large-scale real-city driving image dataset, BDDS.