PaperlayerView exact revision on arXiv

Source-linked AI summary

DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation

Lukas Hoyer, Dengxin Dai, Luc Van Gool

arXiv:2111.14887v2cs.CV

TL;DR

Pixel-wise real-image annotation is costly, and prior UDA studies largely rely on outdated architectures without systematically assessing newer networks. The paper benchmarks architectures and proposes DAFormer, combining a Transformer encoder, context-aware fusion decoder, and stabilizing training strategies; it improves state-of-the-art performance by 10.8 mIoU and 5.4 mIoU on two adaptation tasks.

  • Problem

    Prior UDA methods mostly use outdated architectures, while the effect of recent network architectures on UDA semantic segmentation has not been systematically studied.

  • Method

    DAFormer combines a Transformer encoder and context-aware multi-level fusion decoder with Rare Class Sampling, Thing-Class ImageNet Feature Distance, and learning-rate warmup.

  • Results

    10.8 mIoU and 5.4 mIoU are the reported state-of-the-art improvements on GTA→Cityscapes and Synthia→Cityscapes, respectively.

  • Takeaways & Limitations

    DAFormer reveals the potential of Transformers for UDA and enables strong learning of difficult classes including train, bus, and truck.

  • Takeaways & Limitations

    The approach assumes that Transformers and ImageNet-pretrained features provide useful robustness or guidance, while more capable architectures can increase adaptation instability and source-domain overfitting.

Abstract

from arXiv · show

As acquiring pixel-wise annotations of real-world images for semantic segmentation is a costly process, a model can instead be trained with more accessible synthetic data and adapted to real images without requiring their annotations. This process is studied in unsupervised domain adaptation (UDA). Even though a large number of methods propose new adaptation strategies, they are mostly based on outdated network architectures. As the influence of recent network architectures has not been systematically studied, we first benchmark different network architectures for UDA and newly reveal the potential of Transformers for UDA semantic segmentation. Based on the findings, we propose a novel UDA method, DAFormer. The network architecture of DAFormer consists of a Transformer encoder and a multi-level context-aware feature fusion decoder. It is enabled by three simple but crucial training strategies to stabilize the training and to avoid overfitting to the source domain: While (1) Rare Class Sampling on the source domain improves the quality of the pseudo-labels by mitigating the confirmation bias of self-training toward common classes, (2) a Thing-Class ImageNet Feature Distance and (3) a learning rate warmup promote feature transfer from ImageNet pretraining. DAFormer represents a major advance in UDA. It improves the state of the art by 10.8 mIoU for GTA-to-Cityscapes and 5.4 mIoU for Synthia-to-Cityscapes and enables learning even difficult classes such as train, bus, and truck well. The implementation is available at https://github.com/lhoyer/DAFormer.

1. Introduction

Pixel-wise annotation is costly, motivating UDA from synthetic source data to unlabeled real target images. This work rethinks UDA architectures and training, proposing DAFormer to address outdated backbones, domain shift, and source overfitting.

  • 1.5 hours are required to annotate a single Cityscapes image, while adverse-weather images take 3.3 hours.
  • UDA adapts models trained on synthetic source data to real target data without using target labels.Synthetic data are more accessible, but commonly used CNNs generalize poorly across the synthetic-to-real domain shift.
  • Most previous UDA methods use outdated DeepLabV2 or FCN8s architectures, whose 65 mIoU supervised Cityscapes performance trails recent networks reaching 85 mIoU.
  • DAFormer combines a Transformer encoder with context-aware multi-level feature fusion to improve UDA architecture design.The authors identify Transformers as promising because they are more robust than predominant CNNs in this setting.
  • Rare Class Sampling, Thing-Class ImageNet Feature Distance, and learning-rate warmup stabilize training and reduce source-domain overfitting.Rare Class Sampling improves pseudo-label quality by counteracting self-training confirmation bias toward common classes.
  • 10.8 mIoU and 5.4 mIoU are the reported state-of-the-art improvements for GTA→Cityscapes and Synthia→Cityscapes, respectively.DAFormer also improves train, truck, and bus IoU on GTA→Cityscapes from 16 to 65, 49 to 75, and 59 to 78.

2. Related Work

Related work covers CNN-based semantic segmentation, adversarial and self-training UDA, and strategies for long-tailed class distributions. DAFormer extends class-balanced sampling and ImageNet feature distillation to semantic-segmentation UDA, particularly targeting rare classes and Transformer training.

  • Semantic Image Segmentation: Semantic segmentation commonly uses encoder-decoder networks with skip connections, dilated convolutions, resolution preservation, or contextual modules to recover spatial detail.
  • Unsupervised Domain Adaptation (UDA): UDA methods include adversarial distribution alignment at input, feature, output, or patch levels, alongside self-training with pseudo-labels.
  • Long-Tailed Distributions: Long-tailed datasets bias models toward common classes, motivating re-sampling, loss re-weighting, and transfer-learning strategies.
  • Rare Class Sampling: Rare Class Sampling extends class-balanced sampling from classification to semantic segmentation by addressing rare and common classes co-occurring in one sample.The authors report that re-sampling is particularly effective for training Transformers for UDA.
  • Feature Distillation: Thing-Class ImageNet Feature Distance applies ImageNet feature distillation to self-training while restricting the distance to thing-class regions.This restriction reflects that ImageNet predominantly labels thing classes.

3. Methods

The method combines a Transformer-based DAFormer architecture with self-training and three strategies designed to improve adaptation stability, rare-class learning, and transfer from ImageNet pretraining.

  • Self-Training (ST) for UDA: UDA trains on labeled source images and unlabeled target images, using self-training to adapt without target labels.The teacher produces target pseudo-labels and confidence estimates, which additionally train the student network.
  • Training Strategies for UDA: The framework also uses online self-training, exponential-moving-average teacher updates, target augmentations, and learning-rate warmup to stabilize adaptation.The three training strategies address instability and source-domain overfitting in more capable architectures.
  • DAFormer Network Architecture: DAFormer replaces outdated DeepLabV2-based UDA architectures with a Transformer encoder and context-aware multi-level feature fusion.Its encoder uses Mix Transformers with small patches, sequence reduction, and overlapping patch merging to produce multi-level features.
  • Rare Class Sampling: Rare Class Sampling oversamples source images containing infrequent classes to learn them earlier and improve pseudo-label quality.The sampling probability increases as class frequency decreases, while temperature controls how strongly the distribution focuses on rare classes.
  • Thing-Class ImageNet Feature Distance: Thing-Class ImageNet Feature Distance regularizes bottleneck features toward ImageNet features on regions containing object-like classes.The feature-distance loss is restricted to thing-class regions because ImageNet primarily represents objects with well-defined shapes.

4. Experiments

Experiments show that Transformer-based architectures, especially SegFormer and MiT encoders, improve UDA performance, while DAFormer’s decoder and training strategies further strengthen adaptation. Rare Class Sampling, feature distance, and warmup address instability, rare-class learning, and source-domain overfitting.

  • Network architecture: SegFormer increases source-only / UDA / oracle mIoU from 34.3 / 54.2 / 72.1 to 45.6 / 58.2 / 76.4 over the compared architecture.The authors attribute the improved UDA performance partly to SegFormer’s stronger source-only domain generalization.
  • Network architecture: MiT encoders generalize better than CNN encoders, with MiT-B5 achieving the best UDA mIoU and separating semantically similar vehicle classes more effectively.ResNet structures stuff-classes slightly better, whereas MiT is stronger at separating vehicle classes that are difficult to adapt.
  • Learning rate warmup: Learning rate warmup significantly improves UDA and oracle performance by stabilizing early training, with especially important benefits for difficult classes.Its relative benefit is larger for UDA than supervised learning and is observed across architectures.
  • Rare Class Sampling: RCS makes rare-class IoU rise earlier and remain higher across sampling seeds, improving UDA performance by +5.8 mIoU.The largest gains occur for rider, train, motorcycle, and bicycle; RCS also accounts for class co-occurrence unlike class-balanced sampling.
  • DAFormer decoder: DAFormer’s context-aware multi-level decoder improves UDA performance over the SegFormer decoder from 67.0 to 68.3 mIoU.Using context from multiple encoder stages is more effective than restricting context to bottleneck features, whose performance decreases by -1.3 mIoU.

5. Conclusions

DAFormer combines a Transformer encoder, a context-aware fusion decoder, and three training policies tailored for UDA. It improves state-of-the-art performance by 10.8 mIoU on GTA→Cityscapes and 5.4 mIoU on Synthia→Cityscapes.

  • DAFormer uses a Transformer encoder and a context-aware fusion decoder tailored for UDA.
  • Three training policies stabilize and regularize UDA, enabling DAFormer's capabilities.
  • 10.8 mIoU on GTA→Cityscapes and 5.4 mIoU on Synthia→Cityscapes improve state-of-the-art performance.

Supplementary Material

The supplementary material adds implementation details, analyses, ablations, sensitivity studies, and resource comparisons, while the source code supports reproduction. It also defines thing- and stuff-classes for Cityscapes.

  • The supplementary material provides implementation details and additional experimental results and analysis.Its sections cover sampling statistics, training-strategy ablations, parameter sensitivity, self-training, and runtime and memory consumption.
  • The source code for DAFormer and all ablation studies is provided for reproduction.The README includes environment and dataset setup information.
  • Cityscapes thing-classes include traffic objects and people, whereas stuff-classes include surfaces, structures, vegetation, terrain, and sky.

C. Rare Class Sampling Statistics

Rare Class Sampling addresses imbalanced source distributions by sampling images containing rare classes more often, increasing their pixel representation. Its class probabilities are higher for rare classes and near zero for some ubiquitous common classes.

  • RCS increases the proportion of pixels belonging to rare classes in re-sampled images.
  • RCS assigns higher class sampling probabilities to rare classes than to common classes.For the default temperature T = 0.01, the probabilities are defined by Eq. 7.
  • Rare Class Sampling samples images with rare classes more often than random sampling.
  • For common classes such as road and sky, RCS sampling probabilities are very close to zero because these classes occur in almost every image.
  • Table S1 reports the component ablation setup, and Table S2 reports the RCS and FD hyperparameter sensitivity setup.

D. Training Strategies for DeepLabV2

The improved training strategies benefit both DeepLabV2 and SegFormer, but the performance improvement is significantly larger for SegFormer, supporting the importance of architecture for UDA. RCS and FD remain stable across moderate hyperparameter deviations.

  • Learning rate warmup, RCS, FD, and their combination all benefit DeepLabV2 and SegFormer.
  • The performance improvement is significantly larger for SegFormer than for DeepLabV2, supporting the importance of network architecture for UDA.
  • RCS is stable up to a fivefold deviation from its default temperature, while FD is stable up to about a twofold weighting deviation.The defaults are chosen using intuitive strategies based on re-sampled pixels and gradient-magnitude balance.

F. Ablation of Self-Training

The ablation shows that RCS and FD improve source-only generalization and reinforce self-training, while DAFormer adds runtime and memory costs during training.

  • Ablation results: +5.1 mIoU is obtained from RCS and FD without target-domain self-training.The model is trained only on source data in this setting.
  • Ablation results: +8.0 mIoU is obtained when RCS and FD are combined with self-training.The improvement exceeds their source-only benefit, indicating reinforcement of self-training.
  • Runtime and memory: DAFormer uses 9.6 GB GPU memory during training and runs at 0.7 it/s on a single RTX 2080 Ti.It can be trained within 16 hours.
  • Runtime and memory: Compared with DeepLabV2 under the same UDA configuration, DAFormer requires 12% more GPU memory and about 30% more training and inference time.DAFormer is heavier than DeepLabV2 and SegFormer.
  • Runtime and memory: Ablating RCS and FD reduces GPU memory consumption by 54% and training time by 36%, without changing inference throughput.The reductions mainly come from removing the additional ImageNet encoder and feature-distance calculation.

H. Qualitative Analysis

Qualitative comparisons show that DAFormer improves recognition across difficult object and scene classes, while remaining vulnerable to annotation mismatches and visually similar categories.

  • Comparison with ProDA: DAFormer improves recognition of train, bus, truck, car, sidewalk, rider, bicycle, and fence compared with ProDA.The examples span different perspectives, object sizes, and appearances.
  • Comparison with ProDA: DAFormer generally segments fine structures better, especially for small classes such as pole, traffic sign, and traffic light.
  • Domain generalization: SegFormer source-only training generalizes better to the target domain than DeepLabV2 source-only training.
  • Domain generalization: A considerable performance gap remains between source-only SegFormer and DAFormer, showing the importance of target-domain adaptation.
  • Error cases: Typical errors include confusing sidewalk with road, wall with fence, and partly occluded buses, plus misclassifying special vans and people near bikes.

I. Comparison with Previous Methods

Against previous UDA methods, DAFormer achieves stronger overall performance, although its advantage is not uniform across all datasets and classes.

  • Overall comparison: DAFormer outperforms all previous works in mIoU and classwise IoU on GTA→Cityscapes, often by a considerable margin.
  • Overall comparison: On Synthia→Cityscapes, DAFormer outperforms previous works except for the stuff classes road, sidewalk, vegetation, and sky.
  • Fair comparison: DACS reaches 54.2 mIoU on GTA→Cityscapes with the authors’ training configuration including warmup, while DAFormer exceeds it by 14.1 mIoU.
  • Evaluation protocol: DAFormer results are averaged over three training runs and have a standard deviation of 0.5 mIoU on both benchmarks.The authors report this as evidence of stable training.

J.1. Limitations

The limitations concern architecture coverage, Transformer behavior for stuff classes, domain-specific context assumptions, rare target classes, annotation-policy mismatches, and broader application risks.

  • J.1. Limitations: Only a selection of network architectures is evaluated because of computational constraints.Further architectures and larger batch sizes remain unexplored.
  • J.1. Limitations: Transformers may be disadvantageous for stuff classes when texture is important because their shape bias focuses attention on shape rather than texture.The authors leave further investigation to future work.
  • J.1. Limitations: Context-aware fusion assumes context correlations are domain-invariant, but this assumption can break down in special cases from other domains.Misclassifying a cow on a road as a horse is given as an example.
  • J.1. Limitations: If a class is extremely rare in the target dataset, its pseudo-labels may be absent, preventing targeted selection of target samples containing that class.
  • J.1. Limitations: Differences between source and target annotation policies cause errors such as bicycle-wheel segmentation mismatches and unresolved corner cases.Additional annotation information or a few target labels could help address these issues.
  • J.2. Potential Negative Impact: Improved semantic-segmentation adaptability could support autonomous driving but also undesired uses such as surveillance or military UAVs.The paper suggests legal restrictions as a possible countermeasure.
Loading 2111.14887v2…