Source-linked AI summary
Source-Free Domain Adaptation for Semantic Segmentation
Yuang Liu, Wei Zhang, Jun Wang
TL;DR
Semantic-segmentation UDA typically requires private or unavailable labeled source datasets, making source-free adaptation necessary. SFDA combines source-knowledge transfer from a trained source model with target-domain self-supervision, and performs competitively with source-driven UDA methods across benchmark segmentation settings.
Problem
Existing semantic-segmentation UDA methods require access to labeled source datasets, although such data may be private and unavailable alongside trained source models.
Method
SFDA uses knowledge transfer from a fixed source model, synthesized source-like samples, dual attention distillation, and intra-domain patch-level self-supervision on unlabeled target data.
Results
SFDA achieves competitive results with state-of-the-art source-driven UDA approaches in synthetic-to-real and cross-city segmentation scenarios under the source-free setting.
Takeaways & Limitations
Source-free adaptation can preserve source-domain knowledge while exploiting target-domain information without source data or target labels.
Abstract
from arXiv · showhide
Unsupervised Domain Adaptation (UDA) can tackle the challenge that convolutional neural network(CNN)-based approaches for semantic segmentation heavily rely on the pixel-level annotated data, which is labor-intensive. However, existing UDA approaches in this regard inevitably require the full access to source datasets to reduce the gap between the source and target domains during model adaptation, which are impractical in the real scenarios where the source datasets are private, and thus cannot be released along with the well-trained source models. To cope with this issue, we propose a source-free domain adaptation framework for semantic segmentation, namely SFDA, in which only a well-trained source model and an unlabeled target domain dataset are available for adaptation. SFDA not only enables to recover and preserve the source domain knowledge from the source model via knowledge transfer during model adaptation, but also distills valuable information from the target domain for self-supervised learning. The pixel- and patch-level optimization objectives tailored for semantic segmentation are seamlessly integrated in the framework. The extensive experimental results on numerous benchmark datasets highlight the effectiveness of our framework against the existing UDA approaches relying on source data.
1. Introduction
Semantic segmentation requires expensive dense annotation, while existing UDA methods assume access to labeled source data. SFDA addresses source-free adaptation by transferring source knowledge and exploiting target-domain information, achieving competitive results in synthetic-to-real and cross-city settings.
- Dense semantic-segmentation annotation is expensive, with a Cityscapes image taking about 90 minutes to annotate.
- Existing UDA methods require both well-trained source models and labeled source datasets to preserve source knowledge and reduce cross-domain discrepancy.
- Source-free segmentation adaptation is challenging because pixel-level pseudo-labels contain multiple classes with diverse distributions, unlike image-level classification labels.
- SFDA combines knowledge transfer with model adaptation using only a well-trained source model and an unlabeled target dataset.
- A dual attention distillation mechanism transfers contextual segmentation information, while intra-domain patch-level self-supervision exploits target pseudo-label knowledge.
- SFDA achieves competitive results with state-of-the-art source-driven UDA approaches under source-free conditions across synthetic-to-real and cross-city scenarios.
2. Related Work
Prior segmentation UDA methods reduce domain discrepancy through adversarial learning, image translation, or target pseudo-label self-supervision, but assume access to densely annotated source data.
- Segmentation UDA methods use adversarial learning, source-conditioned image translation, or target pseudo-label self-supervision to reduce cross-domain discrepancy.
- Existing segmentation UDA approaches assume that densely annotated source datasets remain available during adaptation.
- The paper identifies source-free unsupervised domain adaptation for image segmentation as an unaddressed problem in prior work.
3. Methodology
The proposed training procedure estimates the unavailable source domain by synthesizing fake samples with a generator, enabling source-free framework training.
- Because the source domain is unavailable, a generator synthesizes fake samples to estimate it during training.
3.1. Notations and Motivation
Source-driven UDA combines supervised source segmentation training with target pseudo-label self-supervision. SFDA instead estimates source data, transfers knowledge from a fixed source model, and uses patch-level target information.
- Notations and Motivation: Source-driven UDA assumes an annotated source dataset, an unlabeled target dataset, and a well-trained source model; the target model shares source-model parameters while processing target data.
- Notations and Motivation: Its objective combines supervised segmentation loss for source knowledge preservation with target self-supervision based on pseudo-labels.
- Notations and Motivation: In source-free adaptation, missing annotated source data prevents the supervised source-knowledge-preservation process, so the source domain must be estimated.
- Notations and Motivation: SFDA uses a fixed copy of the source model and a generator that synthesizes fake samples for transferring retained source knowledge to the adapting model.
- Notations and Motivation: An intra-domain patch-level self-supervision module uses patch-level pseudo-label information to improve target-data utilization.
3.2. Source-Free Domain Knowledge Transfer
Source-free knowledge transfer estimates source-domain information with generated samples, then distills output and contextual knowledge between fixed and trainable models. The framework uses batch-normalization statistics, adversarial discrepancy objectives, and dual attention to preserve semantic relationships during adaptation.
- Source Domain Estimation: A generator synthesizes fake source samples from Gaussian noise to estimate the unavailable source domain.The generated samples support transfer of source knowledge from a fixed source model to the target model.
- Source Domain Estimation: Batch-normalization statistics constrain synthetic samples to match feature distributions stored in the source model.The constraint compares batch-wise means and variances at each layer with the corresponding source-domain BN parameters.
- Knowledge Transfer Objectives: Semantic-aware adversarial knowledge transfer uses output-space and feature-level discrepancies between fixed, source, and target models.The output discrepancy is formulated as mean absolute error, while contextual relationships are transferred with dual attention distillation.
- Dual Attention Module: The dual attention module captures spatial and channel-based long-range dependencies for distillation.Spatial and channel attention maps are computed from reshaped features, transformed to a common shape using the original feature, and concatenated.
- Knowledge Transfer Objectives: The generator maximizes source-target model discrepancy while minimizing target-domain attention discrepancy, with α, β, and τ balancing the losses.The target model also minimizes output and attention discrepancies with the source model while learning from target pseudo-labels.
3.3. Self-supervised Model Adaptation
The module uses entropy-ranked target patches to exploit reliable pseudo-label information and reduce uncertainty during source-free model adaptation. It separates easy and hard patches, then uses adversarial learning to narrow their gap.
- The method exploits correct target-domain patch information to expand supervision and alleviate uncertainty in entire pseudo-labels.
- Each target image is divided into K × K position-based patches, whose prediction maps provide patch-level probabilities for entropy estimation.Patch height and width are H/K and W/K, respectively.
- Patches with lower entropy are treated as easier and separated from harder patches through entropy ranking at corresponding positions or classes.In an even-sized batch, half the patches form each group.
- A discriminator distinguishes easy and hard patches, while the target model learns to fool it from the hard-patch side to reduce their gap.
- The training objective extends the existing objective with IPSM adversarial loss and self-supervision loss for the shared target and source models.The adversarial-loss weight is controlled by γ.
4. Experiments
SFDA is evaluated for synthetic-to-real and cross-city semantic segmentation using only source models and unlabeled target data. It achieves competitive or superior performance against source-driven UDA methods, with ablations supporting its knowledge-transfer and patch-level self-supervision components.
- Datasets: Experiments cover synthetic-to-real adaptation from GTA5 or SYNTHIA to Cityscapes and cross-city adaptation from Cityscapes to NTHU.NTHU contains Rio, Rome, Tokyo, and Taipei; Cityscapes is the target in synthetic-to-real experiments.
- Synthetic-to-Real Adaptation: On GTA5→Cityscapes, SFDA outperforms MinEnt without source data, while IPSM helps correct errors in some regions through self-supervision.The full method is compared with an SFDA variant without IPSM.
- Synthetic-to-Real Adaptation: On SYNTHIA→Cityscapes, SFDA with IPSM outperforms source-driven UDA methods and performs competitively on small objects including traffic lights, traffic signs, and motorbikes.Results are reported using IoU and mIoU for 16-class and 13-class segmentation.
- Cross-City Adaptation: On Cityscapes→NTHU, SFDA with IPSM achieves competitive performance across four city datasets compared with the best UDA method, MaxSquare.SFKT alone transfers valuable source knowledge to the target, though it performs below source-only evaluation.
- Ablation Study: Ablations show that DAD losses are more effective than BNS loss, while combining them further improves performance.The study evaluates key SFKT components in Table 4.
- Ablation Study: Fake samples from knowledge transfer resemble source-domain representations and outputs in convolutional networks, making them central to transferring source knowledge.Without BNS, generated samples do not preserve the original source semantic distribution; without DAD, refined segmentation and small-object attention are limited.
- Parameter Analysis: IPSM performs best with 3 to 5 patches, whereas too few or too many patches are unsuitable.K=1 denotes training without IPSM.
5. Conclusion
The paper concludes that SFDA provides a source-free adaptation framework for semantic segmentation by transferring source knowledge and exploiting target pseudo-labels. Experiments and ablations show favorable performance against source-driven UDA methods, but high-resolution segmentation remains unsupported.
- Conclusion: SFDA preserves source-domain knowledge through dual attention distillation and uses intra-domain patch-level self-supervision on target pseudo-labels.The framework combines pixel-level knowledge transfer with patch-level target-domain learning.
- Conclusion: Experiments and ablation studies across different segmentation tasks show favorable performance against existing source-driven UDA methods.
- Conclusion: SFDA does not support high-resolution image segmentation because of limitations in generative fake-sample synthesis.The authors identify this as future work.