Self-supervised Augmentation Consistency for Adapting Semantic Segmentation

Nikita Araslanov; Stefan Roth

Source-linked AI summary

Self-supervised Augmentation Consistency for Adapting Semantic Segmentation

Nikita Araslanov, Stefan Roth

arXiv:2105.00097v1Apr 30, 2021cs.CVcs.LG

TL;DR

Semantic-segmentation domain adaptation often uses computationally costly adversarial, style-transfer, or multi-network pipelines. The paper instead uses augmentation consistency with a momentum network and co-evolving pseudo labels, achieving state-of-the-art accuracy while retaining modest training and model complexity.

Problem
UDA for semantic segmentation has become increasingly complex through the use of style transfer, adversarial training, and network ensembles, which can impede reproducibility.
Method
The framework enforces prediction consistency under photometric noise, flipping, and multi-scale cropping using a momentum network and co-evolving pseudo labels trained end-to-end in one round.
Results
The method sets a new state of the art on established semantic-segmentation adaptation benchmarks.
Takeaways & Limitations
Ordinary augmentation techniques and momentum updates can provide accurate adaptation without sacrificing modest training or model complexity.
Takeaways & Limitations
The approach relies on the assumption that semantic predictions are invariant to photometric transforms and equivariant to spatial similarity transforms.

Abstract

from arXiv · show

We propose an approach to domain adaptation for semantic segmentation that is both practical and highly accurate. In contrast to previous work, we abandon the use of computationally involved adversarial objectives, network ensembles and style transfer. Instead, we employ standard data augmentation techniques $-$ photometric noise, flipping and scaling $-$ and ensure consistency of the semantic predictions across these image transformations. We develop this principle in a lightweight self-supervised framework trained on co-evolving pseudo labels without the need for cumbersome extra training rounds. Simple in training from a practitioner's standpoint, our approach is remarkably effective. We achieve significant improvements of the state-of-the-art segmentation accuracy after adaptation, consistent both across different choices of the backbone architecture and adaptation scenarios.

1. Introduction

The paper presents a lightweight UDA framework for semantic segmentation that replaces complex adaptation pipelines with augmentation consistency and co-evolving pseudo labels. It targets state-of-the-art IoU while avoiding substantial training effort.

Motivation: UDA adapts models using unlabelled data from a distribution different from the annotated dataset.Synthetic data is a motivating case because it is easier to annotate than real-world images.
Motivation: Existing semantic-segmentation UDA methods often combine style transfer, adversarial training, or network ensembles, increasing complexity and impeding reproducibility.The paper identifies model complexity as a potential drag on further progress.
Contribution: The proposed framework reaches state-of-the-art segmentation accuracy measured by IoU without substantial training effort.It uses self-training standalone rather than pairing it with adversarial training or network ensembles.
Contribution: Co-evolving pseudo labels enable end-to-end training in one round, avoiding expert intervention between repeated self-training rounds.This preserves a modest training process while updating labels and the model together.
Method: Augmentation consistency enforces consistent semantic predictions under photometric jitter, flipping, and multi-scale cropping.The underlying assumption is photometric invariance and spatial equivariance of pixelwise semantic predictions.
Method: A momentum network, defined as a slowly advancing copy of the original model, supplies stable yet recent targets for self-supervision.The framework also maintains an exponentially moving class prior to adjust confidence thresholds and loss contributions for rare classes.

2. Related Work

Prior segmentation adaptation methods commonly rely on adversarial alignment, style transfer, multiple networks, or repeated pseudo-label training. This work streamlines adaptation through one-round co-evolving pseudo labels and augmentation-based consistency, using multi-scale and flip predictions as supervision.

Adversarial and image-space alignment: Adversarial alignment methods reduce source-target feature discrepancies with discriminators that may operate across scales or use spatial, conditional, or class-specific priors.These methods may also combine adversarial alignment with entropy minimisation or conservative losses.
Adversarial and image-space alignment: Style-transfer approaches align domains in image space but must preserve semantic consistency under appearance changes.They often require concurrent training of independent networks, making them computationally costly and difficult to train.
Relation to this approach: Table 1 compares IoU against VGG-16 and ResNet-101 state-of-the-art baselines while tracking whether methods use multiple training rounds.The proposed framework reportedly outperforms both backbone-specific state-of-the-art references without adversarial training or multiple rounds.
Self-training: Self-training is computationally lighter but commonly uses offline pseudo labels and several training rounds.The paper instead trains with co-evolving pseudo labels in one round.
Self-training: Co-evolving pseudo labels can be unstable, motivating regularisation or fixed representations in related methods.The paper addresses pseudo-label quality through confidence-aware procedures and consistency-based supervision.
Spatial priors: Segmentation adaptation methods characteristically use spatial priors, whose success can depend on semantic-layout similarity across benchmark domains.The paper distinguishes its approach from these prior-based strategies.
Relation to this approach: The framework combines consistency regularisation with self-ensembling while omitting adversarial training and multiple training rounds.It leverages photometric invariance and scale and flip equivariance, averaging predictions across transformations to obtain pseudo supervision.
Relation to this approach: Figure 2’s pipeline feeds multi-scale crops and flips to the momentum network, reprojects predictions, and averages overlaps before selecting confident pseudo masks.The segmentation network additionally receives photometric perturbations, encouraging invariance to appearance changes.

3. Self-Supervised Augmentation Consistency

The framework adapts segmentation through multi-view pseudo-label generation and consistency training, using a momentum network and mechanisms that improve supervision for rare classes.

Self-supervision: Target-image crops and flips are processed by both networks, with momentum predictions inverse-transformed and averaged into a merged semantic map.The merged map supports pseudo-ground-truth extraction, while the segmentation network additionally receives photometric augmentation and the momentum network receives clean inputs.
Self-supervision: The momentum network provides stable, recent training targets while its parameters are periodically updated from the segmentation network.The update pace is controlled by γψ; low values can be unstable, whereas high values may cause premature suboptimal convergence.
Long-tail handling: Moving class priors lower confidence thresholds for long-tail classes, selecting more of their pixels for self-supervision while bounding thresholds for predominant classes.The prior is maintained with an exponential moving average, and β controls the steepness of threshold reduction for rare classes.
Long-tail handling: The pseudo-label loss uses dominant-class thresholding, ignores low-confidence or non-dominant predictions, and applies confidence-regularized focal weighting to emphasize long-tail categories.Higher λ increases relative weighting on long-tail classes, while λ = 0 disables the focal term.

4. Experiments

Experiments evaluate SAC on synthetic-to-real semantic-segmentation adaptation using two backbones and Cityscapes validation. SAC consistently achieves state-of-the-art accuracy while retaining comparatively modest training and model complexity, with ablations identifying augmentation consistency and momentum targets as central components.

Datasets and setup: The experiments use GTA5 and SYNTHIA synthetic scenes for adaptation to Cityscapes, measuring per-class IoU and mean IoU on the validation split.Cityscapes provides 2,975 training and 500 validation images; GTA5 provides 24,966 annotated scenes, and SYNTHIA provides 9,400 compatible images.
Comparison to state of the art: SAC sets a new mIoU state of the art on both GTA5 → Cityscapes and SYNTHIA → Cityscapes across VGG-16 and ResNet-101 settings.The comparison uses single-scale inference, and SAC reaches the top rank consistently across backbone and source-data settings.
Comparison to state of the art: +3.4% and +1.2% over the best published GTA5 → Cityscapes results are achieved with VGG-16 and ResNet-101, respectively.The compared approaches use higher model complexity or multiple training rounds, whereas SAC trains with the target loss in a single pass.
Comparison to state of the art: 7.6% and 1.4% improvements over previous best results are achieved on SYNTHIA → Cityscapes with VGG-16 and ResNet-101, respectively.With ResNet-101, SAC also exceeds PyCDA by 5.9% IoU under full evaluation.
Ablation study: Ablations show that removing augmentation consistency decreases IoU by 8.0%, while removing the momentum network decreases it by 6.4%.Among individual augmentations, removing photometric jitter causes a 3.9% decrease, compared with −2.6% for multi-scale fusion and −0.6% for flipping.

5. Conclusion

The framework adapts semantic segmentation simply and accurately, using ordinary augmentations and momentum updates without sacrificing modest training or model complexity.

The approach combines ordinary augmentation techniques with momentum updates for semantic-segmentation domain adaptation.
The framework achieves state-of-the-art accuracy while maintaining modest training and model complexity.
Its components are not strictly specialised and rely on a relatively weak, broadly applicable assumption.
The paper focuses on semantic segmentation and proposes exploring the techniques for other dense prediction tasks.

– Supplemental Material –

The supplemental material adds implementation, long-tail, hyperparameter, architecture, and evaluation-protocol analyses around the proposed framework.

The appendix provides further training and implementation details for the framework.
It examines long-tail class accuracy before and after adaptation.
It discusses hyperparameter selection and performs a sensitivity analysis.
It evaluates the framework with an FCN8s segmentation architecture.
It discusses limitations of the current evaluation protocol and proposes a revision based on best practices.

B. Further Technical Details

The supplementary technical details specify the augmentation procedures, source-target consistency distinction, implementation note, and typical training schedule.

Photometric noise: Photometric noise uses Gaussian smoothing, greyscaling, and colour jittering, with parameters reused from MoCo-v2.
Constraint-free data augmentation: Source images are randomly scaled before cropping, while semantic consistency is enforced only for target data because source ground truth is available.
Constraint-free data augmentation: Random horizontal flipping is applied to both target and source images.
Photometric noise: The Pillow library converts the Gaussian-blur radius into a box length using L = 3 ∗r2 + 1.
Photometric noise: The appendix includes a Python implementation figure for the photometric noise.
Training schedule: The framework typically converges after 150 − 200K total iterations, corresponding to approximately three days on standard GPUs for VGG16 and ResNet-101.
Table 5 reports per-class IoU (%) on Cityscapes validation for GTA5 →Cityscapes with a VGG-16 backbone.

C.1. A closer look at long-tail adaptation

The long-tail components primarily affect rare categories and complement one another, improving the aggregate mean IoU despite trade-offs for individual classes.

The framework uses class-based thresholding, importance sampling, and focal loss to address long-tail classes.
Ubiquitous classes are hardly affected, while long-tail categories show the primary accuracy changes.
Mean IoU rises from 44.5% with no active component to 46.8% with one, 48.4% with two, and 49.9% with all three.
Focal loss tends to improve wall, fence, and pole, while thresholding benefits traffic light and several rare classes.
Adding importance sampling can reduce train IoU from 29.2% to 17.3% while increasing other classes and mean IoU.
All three components provide the best accuracy trade-off across individual classes and improve mean IoU by 5.4% combined.

C.2. Hyperparameter search and sensitivity

The framework remains effective across choices of ζ and β, although suboptimal settings reduce accuracy. The weakest tested model still substantially improves over the baseline.

The sensitivity study initially explores ζ ∈ (0.7, 0.8) and β ∈ (0.0001, 0.01) using MobileNetV2.Performance is measured by mean IoU on 500 Cityscapes validation images.
A poor setting of ζ = 0.8 and β = 0.01 still improves over the baseline by 8.5% IoU.The authors report this as the weakest model in the sensitivity study.
β = 0.01 causes a more tangible accuracy drop because low-confidence predictions enter the pseudo labels.
Suboptimal hyperparameter choices produce inferior results with a standard deviation of ±1.4% mIoU.
The framework maintains strong accuracy under different settings of ζ and β.The evaluation uses mean IoU on GTA5 →Cityscapes validation data.

C.3. VGG-16 with FCN8s

The section highlights that segmentation architecture can affect observed domain-adaptation accuracy. Comparisons using VGG-16 span FCN8s and DeepLabv2 architectures.

Previous work uses either FCN8s with VGG-16 or DeepLabv2, while this work adopts DeepLabv2 in its main experiments.
The segmentation architecture alone may contribute to accuracy differences between domain-adaptation methods.The authors caution that improvements may otherwise be attributed to the methods themselves.

D. Towards Best-practice Evaluation

The authors argue that the usual evaluation protocol risks using public validation annotations for model selection and reporting. They propose selecting models on Cityscapes val and reporting final accuracy on the regulated Cityscapes test set.

The prevailing protocol selects models using ground truth from 500 Cityscapes train images and reports accuracy on 500 Cityscapes val images.
Cityscapes val annotations are public, making unrestricted consultation inconsistent with holdout-test best practice.Holdout data should be reserved for unbiased final performance assessment.
The proposed protocol trains on Cityscapes train without target labels, uses labeled Cityscapes val for model selection, and reports on Cityscapes test.
Because Cityscapes val contains images from different cities than Cityscapes train, it is more suitable for detecting overfitting to city-specific particularities.
The authors expect Cityscapes test accuracy to be on a par with previously reported Cityscapes val accuracy because the method was not finetuned to Cityscapes val.Table 8 reports per-class IoU on Cityscapes test and gives prior validation results for reference.
Regulated test-set access is intended to improve transparency and fairness in benchmarking.The authors hope future UDA segmentation methods will report Cityscapes test results.