PaperlayerView exact revision on arXiv

Source-linked AI summary

Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditions

David Bruggemann, Christos Sakaridis, Prune Truong, Luc Van Gool

arXiv:2207.06825v3cs.CV

TL;DR

Adverse-condition semantic segmentation lacks dense pixel-level labels, and clear-weather-trained models are fragile under visual shifts. Refign adds uncertainty-aware alignment and adaptive label correction using corresponding normal-condition images to self-training UDA. It reports state-of-the-art results on multiple adverse-condition benchmarks, including ACDC and Dark Zurich.

  • Problem

    Clear-weather-trained segmentation models are fragile under adverse visual conditions, where dense pixel-level semantic annotations are scarce.

    Source: §1 · p. 1

  • Method

    Refign extends self-training UDA by uncertainty-aware alignment of reference predictions to target predictions and non-parametric adaptive correction of target labels.

    Source: §3.1 · p. 4, §3.2 · p. 5, §5 · p. 8

  • Results

    Refign reports state-of-the-art normal-to-adverse adaptation across multiple driving-scene datasets, including 65.6% mIoU on ACDC and 56.2% mIoU on Dark Zurich with DAFormer.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Refign is a generic add-on to self-training UDA that leverages corresponding reference images to improve target predictions.

    Source: §5 · p. 8

    The approach assumes each target image has a corresponding reference image of the same scene, from a different viewpoint and under better visual conditions.

    Source: §3 · p. 3

Abstract

from arXiv · show

Due to the scarcity of dense pixel-level semantic annotations for images recorded in adverse visual conditions, there has been a keen interest in unsupervised domain adaptation (UDA) for the semantic segmentation of such images. UDA adapts models trained on normal conditions to the target adverse-condition domains. Meanwhile, multiple datasets with driving scenes provide corresponding images of the same scenes across multiple conditions, which can serve as a form of weak supervision for domain adaptation. We propose Refign, a generic extension to self-training-based UDA methods which leverages these cross-domain correspondences. Refign consists of two steps: (1) aligning the normal-condition image to the corresponding adverse-condition image using an uncertainty-aware dense matching network, and (2) refining the adverse prediction with the normal prediction using an adaptive label correction mechanism. We design custom modules to streamline both steps and set the new state of the art for domain-adaptive semantic segmentation on several adverse-condition benchmarks, including ACDC and Dark Zurich. The approach introduces no extra training parameters, minimal computational overhead -- during training only -- and can be used as a drop-in extension to improve any given self-training-based UDA method. Code is available at https://github.com/brdav/refign.

1. Introduction

Semantic segmentation models trained on clear-weather data are fragile under adverse visual conditions, motivating UDA where target labels are unavailable. Refign uses corresponding normal-condition images to align and refine target predictions, addressing noisy self-training labels; with DAFormer it sets state-of-the-art results on ACDC and Dark Zurich.

  • Clear-weather-trained segmentation models are fragile to changes in image quality, illumination, and weather, motivating adaptation to unlabeled target domains.Source: §1 · p. 1
  • Refign first spatially aligns reference predictions to the target, then refines target predictions through adaptive label correction.Source: Figure 1 · p. 1, §1 · p. 2
  • Refign uses same-scene normal-condition reference images, which can be collected by capturing routes twice and matching frames via GPS, to guide predictions for adverse-condition targets.Source: §1 · p. 1
  • Self-training can propagate noisy pseudo-label errors and drift, so Refign uses complementary predictions from two views to modify target labels through adaptive refinement.Source: §1 · pp. 1–2
  • Because alignment can be imperfect from dynamic objects, occlusions, and warp inaccuracies, Refign estimates pixel-level confidence to guide the downstream refinement.Source: §1 · p. 2
  • 65.6% mIoU on ACDC and 56.2% mIoU on Dark Zurich are achieved by adding Refign to DAFormer, setting new state of the art on both benchmarks.Source: §1 · p. 2

2. Related Work

Prior work addresses domain adaptation through adversarial alignment, self-training, synthetic weather, curricula, and other strategies, but fewer methods target normal-to-adverse adaptation. Cross-condition correspondences have enabled weak supervision, while Refign combines dense matching with uncertainty-guided refinement and uses complementary views for label correction.

  • Fewer methods address normal-to-adverse adaptation than synthetic-to-real adaptation, despite its relevance to automated driving in fog, night, rain, and snow.Source: §2 · p. 2
  • Dark Zurich and ACDC cross-condition correspondences enabled weak-supervision approaches, including sparse-match consistency, static-class warping, and two-view-geometry fusion.Source: §2 · p. 2
  • Unlike prior correspondence approaches, Refign directly warps corresponding images with uncertainty-aware dense matching, using warp uncertainty to guide refinement that can include dynamic objects.Source: §2 · p. 2, Figure 2 · p. 3
  • Dense matching methods range from correlation-volume approaches to direct flow regression; PDC-Net also estimates uncertainty but requires sparse SfM matches for training.Source: §2 · p. 2
  • Unlike self-label correction approaches that rely on the learner's predictions, Refign uses complementary information from two views of the same scene to correct labels.Source: §2 · p. 3

3. Refign

Refign extends self-training UDA with corresponding normal-condition reference images to improve adverse-domain predictions. It aligns reference predictions to target images, then adaptively refines target predictions for pseudo-labeling.

3. Refign

  • Refign is a framework-agnostic extension to self-training UDA that uses corresponding normal-condition references to guide adverse-target self-training.The model adapts to the reference domain every second training iteration; the reference is assumed to show the same scene from another viewpoint under better visual conditions.Source: §3 · p. 3

3.1. Alignment

  • Uncertainty-Aware WarpC extends warp consistency with probabilistic flow estimates, aligning reference predictions to targets while estimating pixel-wise warp confidence.The alignment model is trained separately and frozen during segmentation self-training; confidence is based on the probability that the true flow lies within radius r of the estimated flow.Source: §3.1 · p. 4

3.2. Refinement

  • Refign refines target class probabilities by convexly combining them with aligned reference probabilities, using adaptive element-wise weights before pseudo-labeling.The model's normalized target-prediction entropy informs a trust score, allowing reliance on reference guidance to change as confidence develops during training.Source: §3 · p. 3, §3.2 · p. 5
  • For large static classes, Refign increases refinement while a spatial and channel-wise mask restricts mixing to locations where both domains predict those classes.Large static classes are matched more accurately than small static classes; pixel-wise warp confidence also lets the method ignore difficult-to-align dynamic and small static objects.Source: §3.2 · p. 5

4. Experiments

Refign is evaluated across adverse-condition segmentation and geometric matching tasks, with comparisons to existing methods and analyses of its components. On ACDC and Dark Zurich, it reports new state-of-the-art results; further experiments examine semi-supervised adaptation, ablations, and matching quality.

4.1. Setup

  • Experiments use Cityscapes as the source and ACDC, Dark Zurich, RobotCar Correspondence, or CMU Correspondence as target and reference domains, with DACS and DAFormer as base methods.Segmentation is evaluated by mIoU; geometric matching uses PCK-T and uncertainty estimation uses AUSE for AEPE.Source: §4.1 · pp. 5–6

4.2. Comparison to the State of the Art in UDA

  • 65.5% mIoU on ACDC sets a new Cityscapes-to-ACDC domain-adaptation state of the art, while 56.2% mIoU on Dark Zurich sets a new benchmark there.Refign improves DAFormer by 10.1% on ACDC; on Dark Zurich it also improves over the DACS and DAFormer baselines and generalizes to two unseen domains.Source: §4.2 · pp. 6–7, Table 1 · p. 6, Table 2 · p. 6
  • Qualitative ACDC validation comparisons show Refign-DAFormer correcting typical DAFormer misclassifications, including labeling sky as road.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3. Semi-Supervised Domain Adaptation Results

  • On RobotCar and CMU, Refign achieves the best semi-supervised domain-adaptation scores among the compared models, using dense matching rather than the sparse-match pipeline of prior models.The compared prior pipeline uses camera-pose estimation, 3D point-cloud generation and matching, match pruning, and 2D reprojection.Source: Table 3 · p. 6, §4.3 · p. 7

4.4. Ablation Study and Further Analysis

  • Ablations report gains of +21.9% mIoU from alignment, +9.6% from PR, +3.9% from large-static-class mixing, +1.8% from trust-score mixing, and +1.0% from concurrent adaptation to R.PR improves dynamic and small static classes but reduces some large static classes; adding M recovers those large static classes. Trust-score mixing improves 17 of 19 classes. The mean-entropy exponent γ is fairly robust to decreases, and Refign preserves higher prediction diversity than simple averaging.Source: Table 4 · p. 7, Table 5 · p. 7, Figure 6 · p. 7

4.5. Geometric Matching Results

  • On RobotCar and CMU, UAWarpC generalizes more accurately than WarpC and matches or outperforms PDC-Net+ in AUSE.Qualitative visualizations show the alignment module identifying dynamic objects and assigning them low confidence.Source: §4.5 · p. 8, Table 6 · p. 8, Figure 7 · p. 8

5. Conclusion

Refign adds uncertainty-aware alignment and adaptive, non-parametric refinement to self-training-based UDA. Applied to DACS and DAFormer, it reports state-of-the-art normal-to-adverse segmentation results across multiple driving-scene datasets.

  • Refign is a generic add-on to self-training UDA that uses an additional reference image for each target-domain image.Source: §5 · p. 8
  • UAWarpC, a probabilistic extension to WarpC [55], provides uncertainty-aware alignment and reaches state-of-the-art flow accuracy and uncertainty estimation.Source: §5 · p. 8
  • Refign’s second step is non-parametric label correction, and its application to DACS [51] and DAFormer [15] yields state-of-the-art normal-to-adverse results across multiple driving-scene datasets.Source: §5 · pp. 8–9

A. Mathematical Derivations

The derivations describe a Gaussian likelihood model for flow and a confidence map obtained by integrating its density over a circle.

B. Training Details

Both the alignment and segmentation networks were trained with Automatic Mixed Precision on a single consumer RTX 2080 Ti GPU.

B.1. Alignment Network

The alignment network is trained in two stages on image pairs with synthetic transformations, and its flow architecture adds per-pixel uncertainty estimation. The appendix also specifies training settings for the segmentation network.

B.1. Alignment Network

  • Training has two stages: the visibility mask is omitted initially, then activated alongside stronger augmentation in stage two.Transformations include color jitter, blur, homography, TPS, and affine-TPS; local elastic transformations are added in stage two.Source: §B.1 · pp. 11–12
  • MegaDepth training uses about 58,000 sampled pairs from 150 scenes, with 1,800 validation pairs and no SfM ground-truth correspondences for training.The dataset comprises 196 reconstructed scenes from 1,070,468 internet photos.Source: §B.1 · pp. 11–12
  • UAWarpC uses a four-level GLU-Net flow architecture and outputs a single per-pixel log-variance as its uncertainty estimate.The encoder uses frozen ImageNet weights; bilinear upsampling replaces transposed convolutions.Source: §B.1 · p. 12

B.2. Segmentation Network

  • For DAFormer segmentation, inputs are half-resolution on Cityscapes, ACDC, and Dark Zurich; RobotCar and CMU use dataset-specific sizes and 512×512 random crops.Random horizontal flipping is also used; extra coarsely labeled target images receive rotation and color jitter.Source: §B.2 · p. 12
  • Segmentation training uses AdamW with a 6 · 10−4 peak learning rate and batch size 2, for 40k iterations on ACDC/Dark Zurich and 20k on RobotCar/CMU.Coarsely labeled extra target images are used every second iteration on RobotCar and CMU.Source: §B.2 · p. 12

C. Small vs. Large Static Classes

Warped reference predictions are less reliable for small static classes than for large ones, while confidence guidance particularly benefits dynamic and small static classes.

C. Small vs. Large Static Classes

  • A 0.70 Pearson correlation links static-class connected-component size with warped-prediction IoU; poles, traffic lights, and traffic signs are smaller and less accurate.Indiscriminate warping is surprisingly accurate for large static classes.Source: C. Small vs. Large Static Classes · p. 13
  • As the warp-confidence threshold rises, performance improves monotonically for dynamic and small static classes but remains mostly flat for large static classes.The figure reports performance increase by class category as a function of the threshold.Source: C. Small vs. Large Static Classes · p. 13, Figure C-2 · p. 13
  • The appendix includes a Dark Zurich-test comparison under the common input-size protocol and a separate table for models outside that protocol; Refign-HRDA ranks first on public leaderboards.The table captions distinguish the two comparison settings.Source: Table D-2 · p. 13, Table D-1 · p. 13

D. Additional Experimental Results

Additional experiments report Refign results across adverse-condition datasets, foggy-scene benchmarks, and an alternate segmentation architecture. A separate evaluation finds more moderate gains when refinement is applied only at test time.

D. Additional Experimental Results

E. Refign at Test-Time

  • At test time, one refinement iteration on a trained model yields more moderate gains than applying Refign during training.This evaluation uses DAFormer on the ACDC and Dark Zurich validation sets.Source: E. Refign at Test-Time · p. 14, Table E-1 · p. 14

F. Qualitative Results

The appendix provides qualitative ACDC validation examples of segmentation predictions, reference-image warps, and confidence maps, including warp failures.

F. Qualitative Results

Related papers

Loading 2207.06825v3…