Source-linked AI summary
Generalize then Adapt: Source-Free Domain Adaptive Semantic Segmentation
Jogendra Nath Kundu, Akshay Kulkarni, Amit Singh, Varun Jampani, R. Venkatesh Babu
TL;DR
Source-free adaptation for segmentation is needed when source and target data cannot be accessed together, and domain shifts challenge models trained on a single distribution. The paper combines virtual multi-source vendor training with client-side pseudo-label self-training, reporting state-of-the-art results on two benchmarks, including comparisons with non-source-free methods.
Problem
Typical domain adaptation assumes concurrent access to labeled source and unlabeled target data, although data-sharing restrictions and distribution shifts complicate deployment.
Source: §1 · p. 1Method
The vendor simulates multiple labeled domains with augmentations and trains multi-head SoMAN, while the client uses selected predictions and cPAE for pseudo-label self-training.
Source: §1 · pp. 1–2, §3.2.1 · p. 6Results
Takeaways & Limitations
The framework balances generalization and specificity, and category-shift scenarios are identified as a direction for future research.
Source: §5 · p. 8Some alternatives require K heads, while leave-r-out solutions have higher computational cost.
Source: §3.1.1 · p. 4
Abstract
from arXiv · showhide
Unsupervised domain adaptation (DA) has gained substantial interest in semantic segmentation. However, almost all prior arts assume concurrent access to both labeled source and unlabeled target, making them unsuitable for scenarios demanding source-free adaptation. In this work, we enable source-free DA by partitioning the task into two: a) source-only domain generalization and b) source-free target adaptation. Towards the former, we provide theoretical insights to develop a multi-head framework trained with a virtually extended multi-source dataset, aiming to balance generalization and specificity. Towards the latter, we utilize the multi-head framework to extract reliable target pseudo-labels for self-training. Additionally, we introduce a novel conditional prior-enforcing auto-encoder that discourages spatial irregularities, thereby enhancing the pseudo-label quality. Experiments on the standard GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes benchmarks show our superiority even against the non-source-free prior-arts. Further, we show our compatibility with online adaptation enabling deployment in a sequentially changing environment.
1. Introduction
The introduction frames source-free adaptation as a response to domain shift and restrictions on sharing source and target data. It proposes vendor-side training across virtual domains and client-side self-training, reporting state-of-the-art results on two segmentation benchmarks.
1. Introduction
- Source-free domain adaptation addresses settings where source and target data cannot be shared, while domain shifts can undermine segmentation models that rely on training-domain correlations.Source: §1 · p. 1
- Augmentation groups perturb task-irrelevant image factors to construct virtual labeled source domains from a single labeled domain.Source: §1 · p. 2
- SoMAN uses a multi-head design to balance domain-generic and domain-specific learning, with theoretical analysis favoring a combination of domain-generic and leave-one-out configurations.Source: §1 · p. 2
- At the client, the method selects SoMAN predictions and uses a conditional prior-enforcing auto-encoder to improve pseudo-labels for source-free self-training.Source: §1 · p. 2
- The source-free framework achieves state-of-the-art results on GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes, even against non-source-free prior work.Source: §1 · p. 2
2. Related Work
Prior segmentation adaptation methods often require concurrent source-target access, while source-free work includes several target-side approaches. The paper also situates its approach among self-training and domain-generalization methods.
2. Related Work
- Feature-space and image-space adaptation methods commonly require source-target co-existence; the cited source-free work instead includes entropy minimization, distillation, self-training, and other target-side strategies.Source: §2 · p. 2
- Prior self-training methods use confident target predictions or other techniques to improve pseudo-labels, and most retain source knowledge using labeled source data.Source: §2 · p. 3
- Unlike multi-source approaches that use several labeled datasets, this work generates virtual domains with augmentations in a single-source setting.Source: §2 · p. 3
3. Approach
The approach prepares a source-free model through vendor-side multi-source learning, then adapts it on unlabeled target data using selected pseudo-labels. Its components balance domain-generic and domain-specific support while using a conditional prior model to refine predictions.
3.1. Vendor-side Strategy
- The vendor prepares an immutable hypothesis support set without target information, enabling adaptation for multiple clients without sharing data.The proposed setup lets the vendor prepare one source model for future clients, reducing compute and storage needs for both parties.Source: §3.1 · p. 3
3.1.1 Multi-source representation learning
- DE++ combines ERM with source-specific experts, while LO++ combines ERM with leave-one-domain-out supports to cover different target scenarios.Both configurations have K + 1 hypothesis supports; the paper favors LO++ across a wider range of practical target scenarios.Source: §3.1.1 · pp. 3–4
- LO++ and DE++ have target risk no greater than ERM, while LO++ supports a wider range of practical target scenarios than DE++.DE++ is particularly useful when the target resembles a source domain; LO++ is favored across a wider range of unique target scenarios.Source: §3.1.1 · p. 4, Figure 2 · p. 4
3.1.2 Preparing virtual multi-source domains
- Virtual augmentation-groups create labeled source domains by varying non-causal image factors while preserving class factors, and a risk-gap threshold filters candidate augmentations.The paper cautions that too many groups increase computational cost and can diminish LO++'s advantage.Source: §3.1.2 · p. 4
3.1.3 Vendor-side architecture and training
- SoMAN shares a CNN backbone between a global ERM head and leave-one-out heads, trained jointly to retain domain-generic and domain-sensitive cues.The global head uses all augmented datasets, while each leave-one-out head trains on the datasets excluding its corresponding group.Source: §3.1.3 · p. 5
3.1.4 Conditional prior-enforcing autoencoder (cPAE)
- The cPAE refines predicted segmentation maps using domain-generic features and is trained as a denoising autoencoder to discourage structurally irregular predictions.Noisy maps come from leave-one-out heads, and the cPAE output is trained toward the source label distribution.Source: §3.1.4 · p. 5
3.2. Client-side Strategy
- On the client side, SoMAN heads and cPAE produce pseudo-labels, while updating only a handful of later backbone weights preserves vendor-side task knowledge.The frozen heads constrain optimization to the vendor-initialized hypothesis subspace.Source: §3.2 · pp. 5–6
3.2.1 Pseudo-label extraction via cPAE
- The client selects the head with lowest average target-training self-entropy, retains the most confident third of pixel predictions per class, and self-trains for three rounds.Unselected pixels receive an “unknown” class excluded from training; the default test-time model uses the optimal head with the shared backbone, while cPAE is used for pseudo-label extraction.Source: §3.2.1 · p. 6, §3.2.2 · p. 6, §3.2.3 · p. 6
4. Experiments
The experiments evaluate the approach on two synthetic-to-real benchmarks and report state-of-the-art performance across benchmarks, ablations, cross-dataset generalization, and online adaptation. They also test two segmentation architectures and five augmentation groups.
4.1. Experimental Settings
- The evaluation uses GTA5→Cityscapes and SYNTHIA→Cityscapes, five selected augmentation groups, and DeepLabv2-ResNet101 and FCN8s-VGG16 architectures.GTA5→Cityscapes reports mIoU over 19 classes; SYNTHIA→Cityscapes reports mIoU over 13 and 16 classes.Source: §4.1 · p. 6
4.2.1 Comparison with prior arts.
- The method achieves state-of-the-art performance across all benchmarks.The comparison covers prior domain-adaptation works and vendor-side domain-generalization methods.Source: Table 1 · p. 7, Table 3 · p. 7, Table 4 · p. 7
- Against source-free prior arts, the approach exceeds one method by 8.1% and concurrent source-free methods by approximately 12%.The comparisons span GTA5→Cityscapes and SYNTHIA→Cityscapes results.Source: §4.2.1 · p. 7
- The proposed method's gains depend on more than augmented data: it still outperforms prior methods trained with the same augmentation groups.The authors attribute additional gains to the multi-head leave-one-out SoMAN framework and cPAE.Source: §4.2.1 · p. 7
4.2.2 Ablation study
- LO++ improves 3.6% over DE++; ERM improves 3.2% over the baseline, while DE++ improves 0.4%.Vendor-side ablations use consistent client-side training for a fair comparison.Source: §4.2.2 · pp. 7–8
- cPAE for pseudo-label extraction boosts performance by 1.8%, and using cPAE for inference adds a further 0.8–0.9%.The qualitative evaluation reports that cPAE regularizes predictions and addresses merged-region and split-region problems.Source: §4.2.2 · p. 8, Figure 4 · p. 8
4.2.3 Analyzing cross-dataset generalization
- The approach achieves superior average generalization on Foggy-Cityscapes and NTHU-Cross-City, even without concurrent access to Cityscapes samples.The comparison considers performance before and after self-training on Cityscapes; concurrent access helps characterize the domain gap.Source: §4.2.3 · p. 8, Table 5 · p. 8
4.2.4 Compatibility to online domain adaptation
- The client-side training is compatible with online adaptation to secondary Foggy-Cityscapes and NTHU-Cross-City domains.The adapted SoMAN retains task-specific knowledge through its frozen multi-head parameters while adapting to unlabeled samples from new environments.Source: §4.2.4 · p. 8
5. Conclusion
The conclusion presents source-free semantic segmentation adaptation as a vendor-side generalization stage followed by client-side self-training. SoMAN balances generalization and specificity, while cPAE improves prediction denoising and pseudo-label quality.
5. Conclusion
- The paper proposes source-free semantic segmentation adaptation for settings where source and target data are not concurrently accessible.The vendor uses systematically selected augmentation groups with SoMAN; cPAE denoises segmentation predictions to improve pseudo-labels for client-side self-training.Source: §5 · p. 8
- Extending the framework to domain-adaptation scenarios with category shift is identified as future research.Source: §5 · p. 8
Supplementary Material
The supplementary material collects theoretical insights, implementation details, additional analyses, and performance evaluations. It also summarizes the paper's notation groups.
Supplementary Material
- The authors report releasing code and trained network weights for reproducibility.Source: Supplementary Material · p. 9
- The supplement includes extended theoretical insights and criteria for selecting augmentation groups.It also includes notation and empirical evaluation of Result 1.Source: Supplementary Material · p. 9
- Implementation materials cover experimental settings, vendor-side and client-side training, and optimization details.Source: Supplementary Material · p. 9
- Additional analyses cover augmentation selection, cPAE impact, time complexity, and quantitative results.Source: Supplementary Material · p. 9
A. Notations
- The notation summary groups terms under distributions, datasets, networks, samples/outputs, and theory.Source: A. Notations · p. 9
B.1. Discussion on Result 1
The discussion shows that the DE++ and LO++ hypothesis supports include the ERM-level target-risk solution, and motivates selecting augmentations that reduce reliance on the original domain. It also outlines the architectures, datasets, and training used in the experiments.
B.1. Discussion on Result 1
- DE++ and LO++ have optimal target error no greater than ERM, with equality possible when their ERM subspace supplies the optimal hypothesis.The configurations pair an ERM subspace with domain-specific or leave-one-out subspaces.Source: §B.1 · p. 9
B.2. Augmentation selection criteria
- An augmentation is selected when its transformation reduces the original-domain factor, expressed as |γ_i| < 1.A single-source model’s performance on augmented samples serves as a proxy for residual original-domain dependence, but candidates still need manual filtering to avoid altering class-relevant factors.Source: §B.2 · pp. 9–10
C.1. Experimental settings
- DeepLabv2-ResNet101 and FCN8s-VGG16 use shared backbones with output heads, keeping inference parameter counts equal to prior arts.Experiments use GTA5→Cityscapes and SYNTHIA→Cityscapes, with Cityscapes training data for client adaptation and its validation set for testing.Source: §C.1 · p. 10
C.2. Vendor-side training
- SoMAN training samples an augmentation group and trains the global head plus K−1 non-global heads in a leave-one-out manner.The subsequent cPAE training uses noisy segmentation maps from trained non-global heads and source ground-truth maps.Source: §C.2 · p. 10
C.3. Client-side training
Client adaptation identifies a target-suited head, extracts filtered pseudo-labels, and self-trains the shared model. The analysis finds performance saturates at five augmentation groups and weaker augmentations are filtered out.
C.3. Client-side training
- The client selects the head with lowest average target self-entropy, then retains only each class’s top 33% most confident predictions as pseudo-labels.Pixels below their class threshold receive an unknown label and are excluded from the training loss.Source: §C.3 · p. 10
- Self-training updates only a block under the shared backbone and performs three offline rounds; performance does not increase with additional rounds.The trainable block is Layer3 for DeepLabv2-ResNet101 and Conv3+Conv4 for FCN8s-VGG16.Source: §C.3 · pp. 10–11
D.1. Optimal choice of AGs
- The augmentation selection filters out weaker transformations that do not produce enough domain shift.Candidate groups are selected using a standard single-source model’s performance; the figure depicts this selection and the tested augmentation order.Source: §D.1 · p. 11, Figure 5 · p. 11
- Performance saturates at K = 5 augmentation groups, so adding more groups is not expected to yield substantial gains over the added computational burden.The figure plots vendor-side performance as K varies.Source: §D.1 · p. 11, Figure 5 · p. 11
D.2. Empirical evaluation of Result 1
Different SoMAN heads are optimal for different target scenarios, and leave-one-out heads often outperform ERM. The cPAE preserves performance in correctly predicted regions while improving it in erroneous regions.
D.2. Empirical evaluation of Result 1
- Different SoMAN heads are optimal across target scenarios, and at least one leave-one-out head beats ERM in most scenarios.ERM is optimal for Foggy-Cityscapes (0.02), LO-E for Foggy-Cityscapes (0.01) and (0.005), and different heads for different NTHU-Cross-City cities.Source: §D.2 · pp. 11–12
- For each sampled source batch, SoMAN selects an augmentation i1, computes global-head cross-entropy on transformed samples, and computes cross-entropy for each of the K−1 other heads.Source: §D.2 · p. 12
D.3. Impact of cPAE
- The cPAE leaves performance in correctly predicted regions almost unchanged, improves it in incorrectly predicted regions, and gains no significant benefit from repeated inference.The authors therefore use cPAE inference once.Source: §D.3 · pp. 12–13
D.4. Time complexity analysis
Client-side adaptation takes less training time than prior arts while maintaining state-of-the-art adaptation performance. Partial backbone training and the absence of additional discriminator networks are part of the approach.
D.4. Time complexity analysis
- The proposed client-side adaptation takes less training time than prior arts while maintaining state-of-the-art adaptation performance.The paper describes the pipeline as suitable for practical and online adaptation scenarios.Source: §D.4 · p. 13
- Partial shared-backbone training and no additional adversarial discriminator are among the stated features of the client-side adaptation.The comparison reports average time per iteration across the forward pass, backward pass, and optimizer step; FDA and PCEDA take longer per iteration because they train the entire model while using FFT-based augmentation and an image-to-image translation network, respectively.Source: §D.4 · p. 13
- The timing comparison uses batch size 1 without automatic mixed precision on a GTX1080Ti system.The measured interval includes forward pass, backward pass, and optimizer step.Source: §D.4 · p. 13
D.5. Qualitative analysis
Qualitative results show predictions improving from vendor-side to client-side to online-adapted models, while cPAE regularizes merged- and split-region errors. Some merged-region failures remain for smaller classes.
D.5. Qualitative analysis
- Paired samples train cPAE as a denoising autoencoder to encourage structural regularity in segmentation predictions.The figure marks merged-region and split-region problems with yellow and blue circles, respectively.Source: Figure 6 · p. 14
- The GTA5 examples illustrate the diversity of the devised augmentations.Examples of the AGs are shown in Fig. 7.Source: Figure 7 · p. 14, §D.5 · p. 16
- Performance generally improves from vendor-side to client-side to online-adapted models on NTHU-Cross-City and Foggy-Cityscapes.The qualitative evaluation also includes GTA5→Cityscapes.Source: Figure 8 · p. 15, §D.5 · p. 16
- cPAE structurally regularizes predictions and addresses merged- and split-region problems, though some failure cases remain.The qualitative examples also show vendor-side generalization outperforming the baseline but performing worse than the client-side model because of the domain gap.Source: Figure 9 · p. 16
- Table 10 compares adaptation training time with prior arts and indicates whether additional networks are involved.The supplied caption does not report the comparison values.Source: Table 10 · p. 16
- Merged-region failures still occur for smaller-sized classes, and the authors identify more explicit shape priors as a direction for improvement.The failures are shown in Fig. 9.Source: §D.5 · p. 16
D.6. Quantitative analysis
The approach achieves state-of-the-art performance across the reported GTA5→Cityscapes and SYNTHIA→Cityscapes settings, including against non-source-free methods.
D.6. Quantitative analysis
- The approach achieves state-of-the-art performance across all settings, including against non-source-free approaches.The reported quantitative results cover GTA5→Cityscapes and SYNTHIA→Cityscapes.Source: §D.6 · p. 16, Table 11 · p. 17, Table 12 · p. 17
- On GTA5→Cityscapes, the approach has better or competitive performance on minority classes such as motorcycle compared with non-source-free prior arts.Table 11 covers DeepLabv2 ResNet101 and FCN8s VGG-16 architectures.Source: Table 11 · p. 17
- For SYNTHIA→Cityscapes, Table 12 reports mIoU and mIoU* averaged over 16 and 13 categories, respectively.The table covers DeepLabv2 ResNet-101 and FCN8s VGG-16 architectures.Source: Table 12 · p. 17