Source-linked AI summary
HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation
Lukas Hoyer, Dengxin Dai, Luc Van Gool
TL;DR
UDA seeks to adapt semantic-segmentation models from labeled source data to unlabeled target data, but memory constraints and resolution trade-offs limit existing approaches. HRDA combines high-resolution detail crops with low-resolution context crops through learned scale attention, achieving substantial gains on GTA-to-Cityscapes and Synthia-to-Cityscapes.
Problem
UDA semantic segmentation must avoid expensive target-domain pixel annotations, while existing downscaled inputs lose fine details and high-resolution crops limit long-range context.
Method
HRDA combines small high-resolution detail crops and large low-resolution context crops using learned scale attention for memory-efficient UDA training.
Results
73.8 mIoU on GTA→Cityscapes and 65.8 mIoU on Synthia→Cityscapes represent gains of +5.5 mIoU and +4.9 mIoU over previous SOTA, respectively.
Takeaways & Limitations
HRDA combines long-range context with high-resolution segmentation details and achieves consistent significant improvements across UDA strategies.
Takeaways & Limitations
Full-sized high- or multi-resolution UDA inputs remain infeasible under the reported GPU memory constraints, including no HR1.0 result.
Abstract
from arXiv · showhide
Unsupervised domain adaptation (UDA) aims to adapt a model trained on the source domain (e.g. synthetic data) to the target domain (e.g. real-world data) without requiring further annotations on the target domain. This work focuses on UDA for semantic segmentation as real-world pixel-wise annotations are particularly expensive to acquire. As UDA methods for semantic segmentation are usually GPU memory intensive, most previous methods operate only on downscaled images. We question this design as low-resolution predictions often fail to preserve fine details. The alternative of training with random crops of high-resolution images alleviates this problem but falls short in capturing long-range, domain-robust context information. Therefore, we propose HRDA, a multi-resolution training approach for UDA, that combines the strengths of small high-resolution crops to preserve fine segmentation details and large low-resolution crops to capture long-range context dependencies with a learned scale attention, while maintaining a manageable GPU memory footprint. HRDA enables adapting small objects and preserving fine segmentation details. It significantly improves the state-of-the-art performance by 5.5 mIoU for GTA-to-Cityscapes and 4.9 mIoU for Synthia-to-Cityscapes, resulting in unprecedented 73.8 and 65.8 mIoU, respectively. The implementation is available at https://github.com/lhoyer/HRDA.
1 Introduction
UDA reduces the need for expensive target-domain pixel annotations, but GPU-intensive training has led most semantic-segmentation methods to use downscaled inputs. HRDA combines high-resolution detail crops, low-resolution context crops, and learned scale attention to preserve details while capturing context efficiently.
- Pixel-wise semantic-segmentation annotations can take more than one hour per image, motivating UDA from labeled or cheaper source data to unlabeled target images.
- Most UDA segmentation methods use downscaled images because multiple domains, auxiliary networks, and additional losses create substantial GPU memory demands.For Cityscapes, these methods commonly use half resolution, 1024×512 pixels.
- Low-resolution inputs often miss small objects and fine details, whereas high-resolution random crops restrict learning long-range, domain-robust context.The trade-off includes distant traffic lights and pedestrian limbs versus scene layout and context relations.
- HRDA combines a large low-resolution context crop with a small nested high-resolution detail crop and fuses their predictions using input-dependent scale attention.The context crop targets large objects and long-range dependencies; the detail crop targets small objects and segmentation details.
- HRDA gains +5.5 mIoU on GTA→Cityscapes and +4.9 mIoU on Synthia→Cityscapes with DAFormer, reaching 73.8 and 65.8 mIoU, respectively.
2 Related Work
Semantic-segmentation research has expanded receptive fields, integrated context and attention, refined boundaries, and adopted Transformer architectures. Related multi-scale methods aggregate features or combine predictions, while UDA commonly uses adversarial training or self-training.
- Semantic-segmentation architectures have improved receptive fields, context integration, attention mechanisms, boundary refinement, and Transformer-based processing.
- Some architectures aggregate intermediate features at different scales generated from a single-scale input to incorporate context information.
- Multi-scale input inference combines predictions from resized images, but content-independent average or max pooling can be suboptimal.
- Learned attention-weighted prediction fusion and hierarchical attention address limitations of naive multi-scale pooling.
- UDA semantic segmentation methods are commonly grouped into adversarial training, which aligns domain distributions, and self-training.
3 Preliminary
UDA trains a segmentation network with labeled source images and unlabeled target images, adding target-domain objectives such as adversarial or self-training losses. In self-training, a teacher generates confidence-weighted pseudo-labels and is updated by exponential moving average.
- UDA uses labeled source images and labels, while target images lack labels and require an additional target-domain loss.
- Most previous UDA methods downsample images by dataset-specific factors to satisfy GPU memory constraints.
- The overall UDA objective combines supervised source loss and weighted target loss as L = LS + λLT.
- Self-training adapts the network with pseudo-labels for target images generated by a teacher network.
- Pseudo-label quality is weighted by confidence, and the teacher is updated through an exponential moving average of the student weights.
- DAFormer combines self-training with consistency training, a domain-robust Transformer, rare-class sampling, and ImageNet-based feature regularization.
4 Methods
HRDA combines a large low-resolution context crop with a small high-resolution detail crop to learn long-range context while preserving fine segmentation details. Learned scale attention fuses their predictions and supports resolution-specific adaptation within manageable memory use.
- Context and Detail Crop: HRDA combines a large LR context crop for long-range context relations with a small HR detail crop for small objects and fine segmentation details.The context crop supplies broader scene layout, while the detail crop focuses on high-resolution regions that do not require far-reaching context.
- Context and Detail Crop: The context crop is produced by cropping the original HR image and bilinearly downsampling it, while the detail crop is randomly cropped within the context region.Cropping the detail region inside the context region enables later fusion of their predictions.
- Pseudo-Label Generation: For pseudo-label generation, overlapping sliding-window HR detail predictions cover the entire context crop and are fused with the LR prediction using the full scale attention.Overlapping windows provide predictions with different contexts, which are averaged for robustness; the resulting HRDA pseudo-label supports self-training at both resolutions.
- Context and Detail Crop: HRDA uses equal context and detail crop dimensions with context downscale factor s = 2, so the context crop covers 4 times more content at half the resolution.This configuration balances resource requirements between the two resolutions.
- Context and Detail Crop: The shared encoder and semantic decoder process both resolutions, saving memory and increasing robustness to different input resolutions.The context and detail predictions are produced with the same feature encoder and semantic decoder.
- Multi-Resolution Fusion: A learned scale attention weights LR context and HR detail predictions according to their regional trustworthiness, favoring HR for small objects and LR for large regions.LR can suppress overly detailed, domain-specific textures in large stuff regions, whereas HR can preserve discriminative small-object details.
5 Experiments
Experiments evaluate HRDA on Cityscapes adaptation, compare it with prior UDA methods and variants, and analyze how resolution, crop size, and multi-resolution fusion affect performance.
- Comparison with State-of-the-Art UDA Methods: 73.8 and 65.8 mIoU are achieved for GTA→Cityscapes and Synthia→Cityscapes, respectively, surpassing the previous state of the art by +5.5 and +4.9 mIoU.HRDA improves IoU for nearly all classes, with especially strong gains on fine-detail classes and additional benefits for truck, bus, and train.
- HRDA for Different UDA Methods: At least +2.4 mIoU is gained when HRDA is applied to three additional UDA methods, demonstrating compatibility across methods and network architectures.The largest improvement occurs for self-training methods, reaching +5.5 mIoU or more.
- Influence of Resolution and Crop Size on UDA: A 4-times smaller low-resolution crop reduces UDA performance by 39%, compared with 14% for supervised training, showing the importance of context crop size.Larger crops particularly improve difficult-to-adapt classes such as wall, fence, truck, bus, and train.
- Influence of Resolution and Crop Size on UDA: Higher input resolution improves small-class IoU, while low resolution is advantageous for some large classes such as road, sidewalk, and terrain.The observed class-wise pattern supports using different resolutions for different object scales.
- Combining Crops from Multiple Resolutions with HRDA: +3.4 mIoU is obtained by multi-resolution training over both LR-only and HR-only training, while enlarging the context crop adds +5.3 mIoU.The combination outperforms the best single-scale IoU for each class, reflecting complementary LR context and HR detail information.
- Combining Crops from Multiple Resolutions with HRDA: +3.8 mIoU over naive HR0.75 crops is achieved by HRDA, and a matched smaller HRDA configuration remains +1.3 mIoU better with 40% less GPU memory.An upsampled LR detail crop improves over baseline but remains −1.9 mIoU below a real HR detail crop.
6 Conclusions
HRDA combines small high-resolution detail crops with large low-resolution context crops through learned scale attention while maintaining a manageable GPU memory footprint. It achieves 73.8 mIoU on GTA→Cityscapes and 65.8 mIoU on Synthia→Cityscapes.
- HRDA combines small HR detail crops and large LR context crops using learned scale attention for memory-efficient UDA.
- 73.8 mIoU on GTA→Cityscapes and 65.8 mIoU on Synthia→Cityscapes represent gains of +5.5 and +4.9 mIoU over previous SOTA, respectively.
Supplementary Material
The supplementary material provides reproducibility resources and additional analyses supporting evaluation of HRDA.
- The supplementary material includes source code, additional experiments, further baseline comparisons, runtime analysis, extended UDA comparisons, and qualitative prediction analysis.
- The implementation reproducing HRDA and its ablation studies is available on GitHub and is based on DAFormer and mmsegmentation.
C Influence of Detail Loss Weight
HRDA performance is robust across a broad detail-loss-weight range, while context resolution and sliding-window inference affect performance within measurable limits.
- Influence of Detail Loss Weight: Detail loss weights λd between 0.1 and 0.3 consistently provide good UDA performance.Values outside this range shift HRDA's focus excessively toward LR or HR, respectively.
- Context Downscale Factor: A context downscale factor of sc = 2 performs best, while sc = 4 reduces performance by -2.6 mIoU but remains +0.8 mIoU above using only the detail crop.
- Inference Strategy: Overlapping sliding-window inference improves DAFormer by only +0.3 mIoU, while HRDA exceeds DAFormer with OSW by +5.2 mIoU.
E.2 Naive High-Resolution UDA
HRDA outperforms naive high-resolution training across UDA methods and remains stronger than a scale-invariance-loss baseline.
- Naive HR Training: HRDA outperforms naive HR training by +3.6 mIoU for DACS and +3.8 mIoU for DAFormer.
- Scale-Invariance Baseline: DAFormer with scale-invariance loss reaches 68.8 mIoU on GTA→Cityscapes, a +0.5 mIoU gain, while HRDA achieves +5.5 mIoU.
F Training and Inference Time
HRDA trains in 32 hours on a Titan RTX, while inference prioritizes UDA performance over speed. Its inference uses 0.8 img/s, 3.3 TFLOPs, and 9.4 GB GPU memory.
- 32h training time on a Titan RTX is shorter than the several-day training reported for other UDA methods.The comparison concerns methods.
- 0.8 img/s, 3.3 TFLOPs, and 9.4 GB GPU memory characterize HRDA inference.
- HRDA prioritizes UDA performance rather than fast inference, which the authors describe as not an inherent UDA constraint.
G Extended Comparison with Previous UDA Methods
Extended comparisons show HRDADAFormer outperforming previous UDA methods across class-wise IoU and overall mIoU on GTA→Cityscapes and Synthia→Cityscapes. Gains are strongest for fine-detail classes, although some stuff classes remain weaker on Synthia→Cityscapes.
- HRDADAFormer outperforms previous UDA methods by a large margin in class-wise IoU and overall mIoU on both benchmarks.The results in Tables S5 and S6 are averaged over 3 random seeds.
- The largest gains occur for pole, traffic light, traffic sign, person, rider, motorbike, and bike.
- Truck, bus, and train also benefit from HRDA in the extended comparisons.
- On Synthia→Cityscapes, road, sidewalk, fence, and vegetation underperform the respective best comparison method.The authors associate these lower results with limitations inherited from DAFormer and possibly its Transformer encoder's shape bias.
H Further Qualitative Examples
Qualitative examples show HRDA improving recognition and fine segmentation for small and distant objects, while using scale-specific predictions for different region types. Failure cases remain for difficult stuff classes with low UDA performance.
- HRDA better recognizes small classes and preserves finer details, especially for distant poles, traffic lights, and traffic signs.Scale attention indicates HRDA focuses on the HR input in these regions.
- Distant pedestrians, riders, bicycles, and motorcycles receive better recognition and finer segmentation from HRDA.The examples show HR input receiving greater scale-attention weight for these regions.
- HRDA improves recognition of difficult stuff classes such as sidewalk and wall, and differentiates vehicle classes including car, truck, bus, and train.The qualitative examples compare these predictions on GTA→Cityscapes.
- HRDA uses LR input for purposes where low-resolution features are preferred, complementing its HR emphasis on fine-detail regions.
- Failure cases remain for sidewalk, terrain, and fence, which are easy to confuse and have considerable domain gaps.