Source-linked AI summary
CrDoCo: Pixel-level Domain Transfer with Cross-Domain Consistency
Yun-Chun Chen, Yen-Yu Lin, Ming-Hsuan Yang, Jia-Bin Huang
TL;DR
Dense prediction adaptation is hindered by expensive pixel-level labeling and domain shifts that limit generalization, while distribution alignment does not explicitly constrain target predictions. CrDoCo combines pixel-level adversarial adaptation, image translation, and domain-specific task networks with a cross-domain consistency loss. The paper reports consistent improvements and favorable or state-of-the-art performance across diverse unsupervised domain adaptation tasks, while noting that training is memory-intensive.
Problem
Expensive pixel-level annotation and source-target domain shifts make it difficult for dense prediction models trained on labeled data to generalize to unlabeled target datasets.
Method
CrDoCo combines image-to-image translation, two domain-specific task networks, pixel-level adversarial adaptation, and a cross-domain consistency loss on task predictions.
Results
CrDoCo consistently improves performance across a wide variety of unsupervised domain adaptation tasks and achieves favorable or state-of-the-art results against existing methods.
Takeaways & Limitations
Cross-domain prediction consistency provides additional supervisory signals for adapting dense prediction models across domains and tasks.
Takeaways & Limitations
Training is memory-intensive because multiple networks are involved simultaneously.
Abstract
from arXiv · showhide
Unsupervised domain adaptation algorithms aim to transfer the knowledge learned from one domain to another (e.g., synthetic to real images). The adapted representations often do not capture pixel-level domain shifts that are crucial for dense prediction tasks (e.g., semantic segmentation). In this paper, we present a novel pixel-wise adversarial domain adaptation algorithm. By leveraging image-to-image translation methods for data augmentation, our key insight is that while the translated images between domains may differ in styles, their predictions for the task should be consistent. We exploit this property and introduce a cross-domain consistency loss that enforces our adapted model to produce consistent predictions. Through extensive experimental results, we show that our method compares favorably against the state-of-the-art on a wide variety of unsupervised domain adaptation tasks.
1. Introduction
CrDoCo addresses poor cross-domain generalization in dense prediction by combining pixel-level adversarial adaptation with a cross-domain consistency loss. The method uses translated images to enforce matching task predictions and reports applicability across multiple unsupervised adaptation tasks.
- Dense prediction requires large pixel-labeled datasets, but annotation is expensive and labor intensive.
- Domain shift between labeled source and unlabeled target datasets causes learned models to generalize poorly to new datasets.
- Aligning source and target marginal distributions does not explicitly constrain target predictions, so performance may remain unsatisfactory.
- CrDoCo combines an image-to-image translation network with domain-specific task networks and enforces consistent predictions through a cross-domain consistency loss.Translated images may differ in style while retaining the same task predictions, providing additional supervisory signals during training.
- Extensive experiments demonstrate state-of-the-art performance against existing unsupervised domain adaptation techniques across a wide range of dense prediction tasks.
2. Related Work
Prior work adapts domains at the feature level, pixel level, or both, but distribution alignment alone does not ensure consistent target predictions. CrDoCo builds on these approaches by using image translation and explicitly coupling domain-specific task networks through prediction consistency.
- Unsupervised domain adaptation methods include feature-level distribution alignment using correlation distance, maximum mean discrepancy, or adversarial learning.
- Semantic segmentation methods align feature activations at multiple levels and also use class-wise adversarial, self-paced, curriculum, and progressive adaptation strategies.
- Joint feature-level and pixel-level methods use image-to-image translation to adapt images before feature-level alignment in classification, segmentation, and depth prediction.
- CrDoCo differs by explicitly penalizing inconsistent predictions between two domain-specific task networks rather than only aligning marginal distributions.
- Cycle consistency has been used for unpaired image translation, semantic matching, motion analysis, and correspondence learning.
- The proposed consistency loss extends beyond synthetic-to-real adaptation to real-to-real adaptation tasks.
3. Method
CrDoCo combines image translation with domain-specific task networks and trains them using task, adversarial, reconstruction, and cross-domain consistency losses. The consistency loss couples predictions across domains, including unlabeled target images, while task-dependent losses support segmentation, depth, and optical flow.
- Network overview: CrDoCo uses image-to-image translation networks and separate source- and target-domain task networks for dense prediction.Images are translated between domains and passed to the corresponding task networks.
- Objective function: The training objective combines task, consistency, and reconstruction losses with image- and feature-level adversarial losses weighted by hyperparameters.Image- and feature-level adversarial terms align distributions within corresponding domains, whereas consistency aligns predictions across domains.
- Cross-domain consistency loss: Cross-domain consistency couples the two task networks by requiring consistent predictions for target images and their translated source-domain versions.This supplies supervisory signals for unlabeled target images.
- Task-specific losses: The consistency loss is task-dependent: bidirectional KL divergence is used for semantic segmentation, ℓ1 loss for depth, and endpoint error for optical flow.The task loss likewise uses cross-entropy for segmentation, ℓ1 loss for depth, and endpoint error for optical flow.
- Task supervision: Source labels supervise both original and translated source images because translation is assumed to preserve their ground-truth task labels.The source task loss trains the two domain-specific task networks using labeled image-label pairs.
- Other losses: Image translation is regularized by reconstruction, requiring reverse translation to recover the original source or target image.The reconstruction loss uses the ℓ1 norm.
4. Experimental Results
Experiments evaluate CrDoCo across synthetic-to-real and real-to-real semantic segmentation, depth prediction, and optical flow adaptation. The method generally performs favorably against existing methods, while cross-domain consistency improves predictions across tasks and settings.
- Semantic segmentation: CrDoCo is evaluated on synthetic-to-real and real-to-real semantic segmentation settings, including GTA5/SYNTHIA to Cityscapes and Cityscapes to different cities.The evaluation uses Cityscapes validation data for synthetic-to-real segmentation and four target cities for real-to-real adaptation.
- Semantic segmentation: 4.9% higher mean IoU than the previous best competitor is achieved with DRN-26, while the gain is 1.0% with FCN8s-VGG16.These comparisons are reported for GTA5-to-Cityscapes semantic segmentation.
- Semantic segmentation: 5.7% higher mean IoU and 3.4% higher pixel accuracy result when cross-domain consistency is added with DRN-26.The paper identifies the cross-domain consistency loss Lconsis as critical for the improved performance.
- Semantic segmentation: State-of-the-art performance is achieved on SYNTHIA-to-Cityscapes segmentation with both DRN-26 and FCN8s backbones.The cross-domain consistency loss also produces sizable improvement in this setting.
- Semantic segmentation: State-of-the-art performance is achieved on two of four Cross-City target cities, while cross-domain consistency consistently improves results on all four cities.The comparison uses the same ResNet-50 feature-backbone setting for the reported AdaptSegNet results.
- Single-view depth estimation: State-of-the-art performance is reported across all adopted depth-prediction metrics for SUNCG-to-NYUDv2 adaptation, with consistent improvement from Lconsis.The metrics include RMSE, RMSE log., Sq. Rel., Abs. Rel., and thresholded accuracy.
- Optical flow estimation: 1.76 lower average endpoint error on KITTI 2012 and 10.6% higher F1-all on KITTI 2015 are obtained when incorporating Lconsis for optical flow estimation.The model is trained on clean MPI Sintel and evaluated on KITTI 2012 and KITTI 2015 without KITTI fine-tuning.
- Limitations: Training is memory-intensive because multiple networks are involved simultaneously.Suggested mitigations include partial sharing between task networks and shared encoders in the image translation network.
5. Conclusions
CrDoCo uses a cross-domain consistency loss to improve pixel-level unsupervised domain adaptation for dense prediction tasks, with applicability demonstrated across diverse tasks.
- CrDoCo introduces a cross-domain consistency loss for pixel-level unsupervised domain adaptation in dense prediction tasks.
- The proposed loss consistently improves performance across a wide range of tasks.
- Extensive experiments demonstrate that the method applies to a wide variety of tasks.