Source-linked AI summary
Domain Adaptation for Structured Output via Discriminative Patch Representations
Yi-Hsuan Tsai, Kihyuk Sohn, Samuel Schulter, Manmohan Chandraker
TL;DR
Models trained on labeled source domains may not generalize to unlabeled target domains, while semantic segmentation requires expensive per-pixel annotations. The paper learns discriminative source patch representations by clustering annotation-derived patch modes, then adversarially aligns target patches in that clustered space. Across synthetic-to-real and cross-city segmentation benchmarks, the approach consistently improves results and complements existing adaptation techniques.
Problem
Semantic-segmentation models may fail to generalize across domains, while target-domain per-pixel annotation is prohibitively expensive.
Method
The method clusters annotation-based source patches into modes, learns clustered-space patch representations, and adversarially aligns target patch distributions with source distributions.
Results
The approach consistently improves semantic-segmentation results across synthetic-to-real and cross-city settings and complements existing domain-adaptation techniques.
Takeaways & Limitations
Patch-level alignment provides a favorable structured-output adaptation strategy across numerous semantic-segmentation benchmarks and can be integrated with other adaptation methods.
Takeaways & Limitations
The formulation assumes labeled source data with per-pixel semantic categories and unlabeled target images.
Abstract
from arXiv · showhide
Predicting structured outputs such as semantic segmentation relies on expensive per-pixel annotations to learn supervised models like convolutional neural networks. However, models trained on one data domain may not generalize well to other domains without annotations for model finetuning. To avoid the labor-intensive process of annotation, we develop a domain adaptation method to adapt the source data to the unlabeled target domain. We propose to learn discriminative feature representations of patches in the source domain by discovering multiple modes of patch-wise output distribution through the construction of a clustered space. With such representations as guidance, we use an adversarial learning scheme to push the feature representations of target patches in the clustered space closer to the distributions of source patches. In addition, we show that our framework is complementary to existing domain adaptation techniques and achieves consistent improvements on semantic segmentation. Extensive ablations and results are demonstrated on numerous benchmark datasets with various settings, such as synthetic-to-real and cross-city scenarios.
1. Introduction
Unsupervised domain adaptation addresses poor cross-domain generalization without requiring target-domain annotations, a need that is especially acute for semantic segmentation. The paper proposes patch-level adversarial alignment that discovers source patch modes and aligns target representations in the resulting clustered space.
- Motivation: Unsupervised domain adaptation uses unlabeled target data to address performance gaps caused by mismatch between labeled source and target domains.The motivation applies broadly to computer vision, while target annotation is avoided.
- Motivation: Semantic segmentation makes domain adaptation especially important because per-pixel annotation is prohibitively expensive.
- Problem: Convolutional discriminators may learn low-level domain differences instead of capturing multiple modes and high-level patterns in structured outputs.
- Method: The method discovers source patch modes with K-means over annotation-based patch representations, then performs patch-level alignment in a K-dimensional clustered space.The two stages are patch mode discovery and patch alignment.
- Results: Experiments cover synthetic-to-real and cross-city semantic segmentation, with ablations and combinations showing consistent improvements and state-of-the-art performance.The approach is combined with output-space adaptation, pixel-level adaptation, and pseudo-label retraining.
2. Related Work
Prior work adapts domains through feature, output, pixel-level, and pseudo-label approaches, while disentangled representations typically rely on predefined factors within a single domain. This paper instead uses source label distributions to learn discriminative patch representations for domain adaptation.
- UDA for Image Classification: Image-classification UDA commonly aligns source and target feature distributions using adversarial learning or Maximum Mean Discrepancy.
- UDA for Semantic Segmentation: Semantic-segmentation adaptation includes global feature alignment and source-derived category-specific constraints for road-scene images.
- UDA for Semantic Segmentation: Segmentation methods also align global structure, synthesize target samples, or retrain with pseudo ground truth.
- Learning Disentangled Representations: Disentangled-representation methods learn interpretable latent spaces from predefined factors such as transformations, rotation, labels, or attributes.
- Patch-Level Alignment: The proposed clustered-space approach represents multiple source patch modes before adversarially aligning target patch distributions, unlike direct category-distribution alignment.
- Learning Disentangled Representations: Unlike prior disentangled-representation work focused on a single domain, this method uses available label distributions as a factor without predefining factors.
3. Domain Adaptation for Structured Output
The framework discovers semantic modes of source patches, learns discriminative clustered representations, and adversarially aligns target patches to the source distribution. Training combines source supervision, patch-mode classification, and adversarial alignment while retaining the main segmentation objective.
- Patch Mode Discovery: Source annotation patches are represented by spatial label histograms and clustered with K-means to discover semantic patch modes.A 2-by-2 grid produces a 2 × 2 × C histogram vector for each sampled patch, which is assigned one of K cluster indices.
- Patch Mode Discovery: The categorization module H predicts each patch’s cluster membership from the segmentation output, producing K-dimensional softmax representations.Each spatial location in Fs corresponds to an input patch, and H is trained with cross-entropy against Γ(Ys).
- Patch-level Alignment: Target patch representations are aligned adversarially to source patches in the clustered K-dimensional space rather than directly in the output space.The discriminator classifies whether F comes from the source or target domain, while target features are encouraged toward the source distribution.
- Network Optimization: The learning objective combines supervised segmentation loss, patch classification loss, and adversarial loss in an alternating GAN-style optimization.The discriminator is updated separately, followed by updates to G and H with the discriminator fixed.
- Implementation Details: Updating H back-propagates into G, and H is required only during training, leaving inference runtime unaffected relative to output-space adaptation.The baseline uses DeepLab-v2 with ResNet-101, while H produces K-channel patch features and D operates on K-dimensional vectors.
- Implementation Details: The experiments fix λd = 0.01, λadv = 0.0005, and K = 50, with 10K iterations of source-only training before all losses are enabled.These settings are used across the experiments to avoid initially noisy predictions.
4. Experimental Results
Experiments evaluate patch-level alignment through ablations, visualizations, integrations with existing adaptation methods, and synthetic-to-real and cross-city benchmarks. The method improves segmentation consistently, while clustered-space design choices affect performance and training difficulty.
- Ablation Study and Analysis: The ablation study evaluates the proposed loss functions and patch-level alignment on GTA5-to-Cityscapes with a ResNet-101 network.The study examines different loss components and design choices in the framework.
- Ablation Study and Analysis: 38.8% versus 36.6% mIoU results when adding discriminative patch representations without alignment, while patch-level adversarial alignment adds 4.7% mIoU.The representation alone improves performance, and the final adversarial alignment provides a further gain.
- Ablation Study and Analysis: 1.9% lower performance without the clustered loss, compared with the method’s 41.3% result, demonstrating the importance of learning the clustered space.The clustered space supplies a signal for separating and aligning different patch modes across domains.
- Ablation Study and Analysis: Clustered-space adaptation embeds source and target patch features into overlapping groups, with example patches showing high cross-domain similarity.Figure 4 compares the method with a baseline without patch-level adaptation using t-SNE and clustered-space patch examples.
- Improvement on Domain Adaptation Methods: Patch-level alignment consistently improves output-space, pixel-level, and pseudo-label adaptation, with GTA5-to-Cityscapes gains of 1.8% to 2.7% in mIoU.The approach is presented as complementary to multiple domain adaptation methods.
- Comparisons with State-of-the-art Methods: 72.0% mean IoU on Cityscapes-to-Oxford RobotCar improves the no-adaptation model by 10.1% and exceeds output-space adaptation by 2.5%.The cross-city setting uses sunny and rainy weather conditions and averages performance over nine categories.
5. Conclusions
The paper presents patch-level alignment for domain adaptation of structured outputs, learning discriminative patch representations through a clustered source space and adversarially aligning target patches. The method complements existing adaptation approaches and performs favorably across synthetic-to-real and cross-city semantic-segmentation settings.
- Patch-level alignment learns discriminative patch representations by constructing a clustered space of source patches and adversarially bringing target patch distributions closer to source distributions.
- The framework is complementary to various domain adaptation approaches and provides additional improvement.
- Extensive ablations and experiments evaluate the method under synthetic-to-real and cross-city semantic-segmentation challenges.
- The method performs favorably against existing algorithms on semantic segmentation.
A. Training Details
Training uses one source image and one target image per iteration, with image and patch sizes specified separately for training and testing. Images preserve aspect ratio without cropping and are down-sampled to the listed sizes.
- Each training iteration randomly samples one source image and one target image, giving a batch size of 1.
- Image and patch sizes differ between training and testing according to the sizes listed in Table 6.
- Images maintain their aspect ratio without cropping and are then down-sampled to the tabled size.
B. Relation to Entropy Minimization
Entropy minimization can replace patch-level adversarial loss as a target-feature regularizer, but the reported result is lower than the proposed source-guided adversarial alignment.
- Entropy minimization can replace the patch-level adversarial loss to push target feature representations toward one of the source clusters.
- 41.9% IoU with entropy regularization is lower than the proposed patch-level adversarial alignment at 43.2%.
- The proposed model uses annotated source labels to learn discriminative target-patch representations by moving them toward the source distribution in clustered space.
C. More Ablation Study on Clustered Space
An ablation study compares the learned H module with direct category-histogram alignment on GTA5-to-Cityscapes, finding the direct implementation performs slightly worse.
- On GTA5-to-Cityscapes, directly aligning category histograms from segmentation outputs achieves 0.7% lower IoU than the proposed method.
- The proposed method achieves 41.3% IoU in this comparison.
- The H module uses learnable parameters to estimate K-means memberships, whereas direct histograms rely solely on updating the segmentation network G.
D. More Details on Pseudo Label Re-training
The method reuses the semi-supervised implementation of [19] by treating target samples as unlabeled data and retaining its output-space discriminator and loss.
E. Result of Cityscapes-to-Oxford
The Cityscapes-to-Oxford RobotCar experiment evaluates the proposed method against no adaptation and output-space adaptation, with additional qualitative comparisons provided.
- The experiment adapts Cityscapes under sunny conditions to Oxford RobotCar rainy scenes.
- The proposed method is compared with a model without adaptation and the output-space adaptation approach [40].
- Figures 7 and 8 provide qualitative results for the Cityscapes-to-Oxford RobotCar setting.
F. Qualitative Comparisons
Qualitative comparisons cover Cityscapes-to-Oxford RobotCar, GTA5-to-Cityscapes, and SYNTHIA-to-Cityscapes settings. The proposed approach is reported to produce more detailed, less noisy segmentation outputs in the latter comparisons.
- Qualitative Comparisons: In GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes comparisons, the proposed approach often yields more detailed segmentation outputs and less noisy regions.
- Qualitative Comparisons: Figures 9–11 compare results without adaptation, output-space adaptation [40], and the proposed method for target images.
- Qualitative Comparisons: Table 6 reports image and patch sizes used for training and testing.
- Qualitative Comparisons: Table 7 reports results for adapting Cityscapes to Oxford RobotCar.
- Qualitative Comparisons: Figures 7 and 8 sequentially show video images and their adapted segmentations for Cityscapes-to-Oxford RobotCar.