PaperlayerView exact revision on arXiv

Source-linked AI summary

Adverse Weather Image Translation with Asymmetric and Uncertainty-aware GAN

Jeong-gi Kwak, Youngsaeng Jin, Yuanming Li, Dongsik Yoon, Donghyeon Kim, Hanseok Ko

arXiv:2112.04283v3cs.CVcs.GR

TL;DR

Adverse-weather translation is difficult because rainy-night images contain artifacts and information imbalance, challenging symmetric domain-transfer models. AU-GAN uses asymmetric feature transfer and matching together with uncertainty-aware cycle consistency; the authors report qualitative and quantitative comparisons with state-of-the-art methods.

  • Problem

    Adverse-weather images contain artifacts and insufficient information, and symmetric cycle-based architectures can be unsuitable when the domains have imbalanced information.

    Source: §1 · p. 2, §3.2 · p. 6

  • Method

    AU-GAN adds a feature transfer network to the adverse-to-standard generator, uses asymmetric feature matching, and models regional uncertainty in cycle consistency.

    Source: §1 · pp. 2–3

  • Results

    The authors report qualitative and quantitative comparisons demonstrating AU-GAN’s superiority over state-of-the-art methods.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The paper presents asymmetric feature transfer and uncertainty-aware cycle consistency as an approach to adverse-weather image translation.

    Source: §5 · p. 10

Abstract

from arXiv · show

Adverse weather image translation belongs to the unsupervised image-to-image (I2I) translation task which aims to transfer adverse condition domain (eg, rainy night) to standard domain (eg, day). It is a challenging task because images from adverse domains have some artifacts and insufficient information. Recently, many studies employing Generative Adversarial Networks (GANs) have achieved notable success in I2I translation but there are still limitations in applying them to adverse weather enhancement. Symmetric architecture based on bidirectional cycle-consistency loss is adopted as a standard framework for unsupervised domain transfer methods. However, it can lead to inferior translation result if the two domains have imbalanced information. To address this issue, we propose a novel GAN model, i.e., AU-GAN, which has an asymmetric architecture for adverse domain translation. We insert a proposed feature transfer network (${T}$-net) in only a normal domain generator (i.e., rainy night-> day) to enhance encoded features of the adverse domain image. In addition, we introduce asymmetric feature matching for disentanglement of encoded features. Finally, we propose uncertainty-aware cycle-consistency loss to address the regional uncertainty of a cyclic reconstructed image. We demonstrate the effectiveness of our method by qualitative and quantitative comparisons with state-of-the-art models. Codes are available at https://github.com/jgkwak95/AU-GAN.

1 Introduction

Adverse weather images challenge domain translation because artifacts and information imbalance make symmetric cycle-based architectures unsuitable. AU-GAN addresses this with one-way feature transfer, asymmetric feature matching, and uncertainty-aware cycle consistency, and is compared with state-of-the-art methods.

1 Introduction

  • Adverse-weather artifacts, blur, and reflections create a domain gap that makes commonly used symmetric CycleGAN-based architectures inappropriate.The motivation notes that performance drops on night images and can be worse in rainy-night conditions, including in applications where reliability matters.Source: §1 · p. 2
  • AU-GAN adds a feature transfer network only to the adverse-to-standard generator to enhance encoded adverse-domain features.The introduction also describes asymmetric feature matching as a way to improve disentanglement without removing local objects.Source: §1 · pp. 2–3
  • Uncertainty-aware cycle consistency weights image regions according to a confidence map to address artifacts that can remain under ordinary cycle consistency.The method is motivated by the possibility that the cycle-consistency constraint leaves artifacts in adverse-domain images.Source: §1 · p. 2
  • The authors report qualitative and quantitative comparisons demonstrating AU-GAN’s superiority over state-of-the-art methods.The introduction lists this comparison among the paper’s contributions.Source: §1 · p. 3

2 Related work

Related work traces unsupervised image translation through cycle consistency and feature disentanglement, then reviews adverse-weather enhancement and uncertainty-aware learning. These lines of work motivate translating rainy-night images and accounting for their regionally varying blur and reflections.

2 Related work

  • CycleGAN uses cycle consistency for unsupervised translation, while later methods use shared latent spaces or disentangled domain-invariant and domain-specific representations.The review also describes multi-domain translation and methods that seek diverse outputs.Source: §2 · p. 3
  • Prior adverse-weather enhancement work includes low-light enhancement, night-to-day translation, and ForkGAN’s rainy-night-to-day translation using a common encoding space.The review connects degraded scene-understanding performance to training data composed mostly of normal-weather images.Source: §2 · p. 3
  • Heteroscedastic aleatoric uncertainty models observation noise that varies by region, which the paper relates to blur and reflections in rainy-night images.The review says this approach is applied to minimize differences between an original image and its cyclic reconstruction.Source: §2 · p. 3

3 Proposed method

AU-GAN uses an asymmetric architecture for adverse-weather translation, with a transfer network only in the adverse-to-standard generator. Its losses address feature disentanglement, regional uncertainty in adverse-domain cycles, and realistic outputs.

3.1 Asymmetric architecture

  • T-net is inserted only in the adverse-to-standard generator to enhance and disentangle encoded features; asymmetric feature matching penalizes differences between input and transferred-image encodings.This addresses the tension between retaining adverse-domain details for reconstruction and excluding them during translation.Source: §3.1 · p. 5

3.2 Uncertainty-aware cyclic loss

  • The adverse-domain cycle loss weights regions by an unsupervised confidence map, reducing the effect of pixel differences in high-uncertainty regions while increasing regularization.The normal-domain cycle retains a pixel-level ℓ1 loss.Source: §3.2 · p. 6

3.3 Model objective

  • LSGAN adversarial losses use domain-specific discriminators to align real and translated-image distributions, alongside the reconstruction, feature-matching, and cyclic terms in the full objective.Hyperparameters λrec, λfeat, and λcyc modulate the relative importance of those terms.Source: §3.3 · p. 6

Experiments

Experiments evaluate AU-GAN on Alderley and BDD100K, comparing image quality, downstream segmentation, component ablations, and predicted uncertainty. The reported results favor AU-GAN in both datasets, while ablations and uncertainty visualizations show where its components matter.

4.1 Experimental setup

  • Experiments use Alderley rainy-night/day images and BDD100K road images, with 256 × 512 inputs and outputs.BDD10K segmentation labels support evaluation with a pretrained segmentation model; training uses Adam with learning rate 0.0002.Source: §4.1 · p. 7

4.2 Qualitative result

  • On Alderley (rainy night→day) and BDD100K (night→day), the paper’s model successfully performs adverse-weather translation, producing visually superior results to other methods in most regions, including dark and blurry areas, while preserving existing objects.Source: §4.2 · p. 7, Figure 2 · p. 8

4.3 Quantitative result

  • In FID, the model outperforms other methods by a large gap on Alderley and BDD100K, comparing 1,000 real day images with 1,000 transformed rainy-night-to-day images from each test set.Source: §4.3 · p. 8
  • AU-GAN further improves semantic segmentation on translated BDD100K night images compared with other methods.ToDayGAN and ForkGAN decrease segmentation performance versus before translation, while CycleGAN and UNIT slightly improve it.Source: §4.3 · pp. 8–9, Table 2 · p. 9

4.4 Ablation study

  • Removing uncertainty-aware loss degrades visual quality more on high-uncertainty Alderley than on BDD100K, while removing T-net degrades FID on both datasets.T-net also improves visual quality and helps preserve existing objects; segmentation performance decreases as proposed elements are removed.Source: §4.4 · p. 9

4.5 Visualization of uncertainty

  • Alderley has higher overall predicted uncertainty than BDD100K, with high-uncertainty regions including glare, wet-road reflections, raindrops, and wiper marks.Yellow or purple map regions indicate high uncertainty; the method learns dataset-specific uncertainty without supervision.Source: §4.5 · pp. 9–10

5 Conclusion

The paper presents AU-GAN for adverse-weather image translation, combining asymmetric feature transfer with uncertainty-aware cycle consistency. Qualitative and quantitative comparisons demonstrate superiority over state-of-the-art methods; future work will extend the method to multimodal translation and attention-based techniques.

5 Conclusion

  • AU-GAN addresses adverse-weather image translation by enhancing adverse-domain encoded features with T-net and estimating a confidence map for uncertainty-based cycle consistency.The paper reports superiority over state-of-the-art methods in qualitative and quantitative studies.Source: §5 · p. 10
  • Future work will extend AU-GAN to multimodal translation and jointly exploit attention-based techniques.Source: §5 · p. 10

Appendix

The appendix documents AU-GAN’s modules, analyzes its uncertainty-aware cycle loss, and adds higher-resolution experiments, failure cases, comparisons, and qualitative results.

6.1 Implementation

  • Implementation details specify shared encoder structures, the A→B-only T-net, an uncertainty-map decoder branch, and multi-scale Patch-GAN discriminators.The encoders use dilated residual blocks; the T-net has four residual blocks.Source: §6.1 · p. 11

6.2 Analysis of uncertainty-aware cyclic loss

  • Without uncertainty-aware cycle loss, cyclic reconstructions retain artifacts or reflections that also appear in day translations; with it, low-confidence regions are removed more clearly.Figure 6 compares model variants with and without the uncertainty-aware loss.Source: §6.2 · pp. 11–12, Figure 6 · p. 13

6.3.1 Experiments on higher resolution

  • On BDD100K at 512×1024, adding one layer to each encoder, decoder, and discriminator did not yield well-converged training, inferior visual quality, and artifacts.The appendix leaves suitable hyperparameters and architecture for high-resolution training to future work.Source: §6.3.1 · pp. 13–14, Figure 7 · p. 13

6.3.2 Failure case

  • Dark buildings or fully dark areas can produce artifacts such as wooded buildings or trees on roads, which the authors attribute to street-tree dataset bias.Road and car regions that commonly appear in the dataset generally translate satisfactorily.Source: §6.3.2 · p. 14

6.5 Extra qualitative results

  • Additional qualitative results show that AU-GAN also performs the opposite day-to-night translation reasonably on BDD100K and Alderley.The paper’s main purpose remains adverse-weather image translation.Source: §6.5 · p. 15, Figure 10 · p. 17

6.4 Additional comparison result and training details

  • Against CycleGAN, UNIT, ToDayGAN, and ForkGAN, AU-GAN delivers superior visual quality while preserving objects in Alderley and BDD100K comparisons.The comparison uses the same datasets and 256 × 512 image size across methods.Source: §6.4 · p. 14, Figure 9 · p. 16

Related papers

Loading 2112.04283v3…