Source-linked AI summary
DCAN: Dual Channel-wise Alignment Networks for Unsupervised Scene Adaptation
Zuxuan Wu, Xintong Han, Yen-Liang Lin, Mustafa Gkhan Uzunbas, Tom Goldstein, Ser Nam Lim, Larry S. Davis
TL;DR
Synthetic segmentation data offer inexpensive labels, but transfer to realistic target images is hindered by domain discrepancies and limited generalization. DCAN addresses this with channel-wise feature alignment in both an image generator and segmentation network, using unlabeled target-image statistics. Experiments on synthetic-to-real urban-scene adaptation report effectiveness over state-of-the-art methods and compatibility with modern segmentation networks.
Problem
Synthetic images provide readily available labels, but domain discrepancies and limited generalization hinder semantic-segmentation transfer to novel realistic data.
Method
DCAN uses target-image channel-wise feature statistics to generate target-like source images and normalize high-level segmentation features through end-to-end alignment.
Results
DCAN outperforms state-of-the-art methods with clear margins in transfers from Synthia and Gta5 to Cityscapes and is compatible with several modern segmentation networks.
Takeaways & Limitations
Channel-wise alignment provides a lightweight, easy-to-train alternative to adversarial distribution alignment for unsupervised scene adaptation.
Abstract
from arXiv · showhide
Harvesting dense pixel-level annotations to train deep neural networks for semantic segmentation is extremely expensive and unwieldy at scale. While learning from synthetic data where labels are readily available sounds promising, performance degrades significantly when testing on novel realistic data due to domain discrepancies. We present Dual Channel-wise Alignment Networks (DCAN), a simple yet effective approach to reduce domain shift at both pixel-level and feature-level. Exploring statistics in each channel of CNN feature maps, our framework performs channel-wise feature alignment, which preserves spatial structures and semantic information, in both an image generator and a segmentation network. In particular, given an image from the source domain and unlabeled samples from the target domain, the generator synthesizes new images on-the-fly to resemble samples from the target domain in appearance and the segmentation network further refines high-level features before predicting semantic maps, both of which leverage feature statistics of sampled images from the target domain. Unlike much recent and concurrent work relying on adversarial training, our framework is lightweight and easy to train. Extensive experiments on adapting models trained on synthetic segmentation benchmarks to real urban scenes demonstrate the effectiveness of the proposed framework.
1 Introduction
DCAN addresses unsupervised scene adaptation by aligning channel-wise feature statistics at both pixel-level and feature-level. It uses target images to synthesize target-like source images and refine segmentation features while preserving spatial and semantic information.
- Motivation: Synthetic segmentation data provide labels cheaply, but realism gaps and model overfitting limit transfer to real images.Dense pixel-level annotations are expensive, while synthetic images differ from real-image distributions and trained networks may generalize poorly.
- Motivation: Unsupervised domain adaptation seeks to bridge training and testing distributions without using target-domain labels.For segmentation, aligning high-dimensional feature maps is especially challenging compared with classification features.
- Method: DCAN performs channel-wise alignment in an image generator and a segmentation network to reduce low-level and high-level domain differences.The approach normalizes source features using target-image statistics while preserving spatial structures and independently processing channel-encoded semantic information.
- Method: The generator maps source images to multiple target-style representations with unchanged content, guided by randomly selected unlabeled target images.The synthesized images resemble target samples and are subsequently used with target images for segmentation-network feature alignment.
- Contribution: DCAN is trained end-to-end to synthesize images and normalize segmentation features on-the-fly, without relying on adversarial distribution alignment.The authors report extensive transfers from Synthia and Gta5 to Cityscapes, with compatibility across modern segmentation networks and clear margins over state-of-the-art methods.
2 Related Work
Prior domain-adaptation work mainly aligns features or stylizes pixels, with much of the literature focused on classification. DCAN combines channel-wise alignment in a generator and segmentation network, avoiding adversarial training while refining target-domain transfer.
- Unsupervised Domain Adaptation: Domain-adaptation methods commonly align source and target features explicitly through statistics or implicitly through domain-invariance objectives.Examples include MMD, covariance matching, reversed gradients, domain confusion, and adversarial losses.
- Unsupervised Domain Adaptation: Pixel-level adaptation removes low-level differences by stylizing source images to resemble target images, whereas segmentation has received less attention than classification.The related work describes color and texture transfer as the main pixel-level objective.
- DCAN Framework: DCAN's framework contains an image generator and a segmentation network, applying channel-wise alignment in both stages.The generator reduces low-level appearance differences before segmentation, while the segmentation network refines features used to produce prediction maps.
- DCAN Framework: Unlike GAN-based approaches, DCAN uses channel-wise feature alignment to synthesize target-like images and normalize source-image feature maps for segmentation.The framework is presented as lightweight and easier to train than approaches relying extensively on adversarial training.
3 Approach
DCAN performs dual channel-wise alignment using unlabeled target images: an image generator transfers target style to source content, while a segmentation network further aligns high-level features before prediction.
- Framework overview: DCAN randomly pairs each labeled source image with an unlabeled target image to guide alignment during training.The target image provides style information for synthesizing source-content images and calibrating segmentation features.
- Channel-wise feature alignment: Channel-wise alignment matches per-channel feature statistics while preserving spatial locations and independently processing channel-encoded semantic information.The operation uses channel means and variances across spatial dimensions and can be inserted into different network layers.
- Image generator: The image generator uses an encoder, channel-wise alignment, and decoder to synthesize target-styled images that retain the source image’s content.Its objective combines content preservation with style matching based on feature statistics and Gram matrices.
- Segmentation network: The segmentation network applies a second alignment stage to synthesized images, reducing remaining high-level mismatches before an encoder-decoder predicts pixel-level labels.This stage calibrates features to the sampled target style while retaining spatial information needed for segmentation.
- Inference: At test time, alignment in the segmentation network is disabled, and the trained network directly predicts semantic maps for novel target-domain images.Training has already matched feature statistics across domains, enabling direct application to target inputs.
- Optimization: Stochastic target sampling replaces exhaustive source-target pairing, avoiding cost that grows linearly with target-set size while serving as regularization for target-domain generalization.The sampled expectation introduces randomized target styles during end-to-end optimization.
4 Experiments
Experiments on synthetic-to-real urban-scene adaptation show that DCAN improves source-only segmentation across datasets and architectures, while ablations attribute gains to resolution, end-to-end synthesis, and channel-wise feature alignment.
- 4.2 Main Results: DCAN improves the source-only baseline by around 8–9 absolute percentage points across three network architectures and both adaptation datasets.The evaluated settings are Synthia → Cityscapes and Gta5 → Cityscapes.
- 4.2 Main Results: With PSPNet, DCAN reaches 41.7% mIoU from Gta5 and 38.4% from Synthia on Cityscapes.The reported relative mIoU gains over corresponding baselines are 8.4% and 9.5%, respectively.
- 4.3 Discussions: At 512×1024 resolution with FCN8s-VGG16, DCAN raises mIoU from 25.9 to 35.4, a 9.5-point gain.The authors report significant gains at all tested resolutions and better adaptation with larger resolution.
- 4.3 Discussions: DCAN outperforms CycleGAN and UNIT with and without segmentation-network feature alignment, and improves two-stage training by 1.7% mIoU.The end-to-end synthesis process is guided by gradients from the final segmentation task.
- 4.3 Discussions: DCAN generates target-style images with fewer artifacts than CycleGAN and UNIT, while instance-level stochastic translation covers styles sampled from the target set.The comparison contrasts DCAN’s instance-level sampling with fixed dataset-level mappings learned by CycleGAN and UNIT.
- 4.3 Discussions: DCAN outperforms ADDA by 1.4% and converges faster, while MMD and CORAL perform worse than the source-only methods.Alignment after Conv3 performs best, and preserving detailed spatial information is favored over alignment at Conv6 or Conv7.
5 Conclusion
DCAN reduces domain shift at both pixel and feature levels for unsupervised scene adaptation, using channel-wise alignment in image generation and segmentation.
- DCAN uses channel-wise feature alignment in both an image generator and segmentation network to reduce domain shift at pixel and feature levels.The framework synthesizes target-like samples while normalizing source-image feature maps.