PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual Attention-in-Attention Model for Joint Rain Streak and Raindrop Removal

Kaihao Zhang, Dongxu Li, Wenhan Luo, Wenqi Ren

arXiv:2103.07051v2cs.CV

TL;DR

Most deep deraining methods address rain streaks or raindrops separately, although both can appear together. DAiAM combines branches for the two rain types with attention to heavy and light rain, while D-DAiAM refines deraining through a heavy-to-light scheme. The authors report state-of-the-art results on public rain-streak and raindrop datasets and consistently better results on images containing both types.

  • Problem

    Most deep deraining methods treat rain streaks and raindrops as separate tasks, although both often occur together and differ in how they degrade images.

    Source: §1 · p. 1

  • Method

    DAiAM uses separate branches for rain streaks and raindrops, with heavy- and light-rain attention maps, and D-DAiAM refines outputs using a heavy-to-light scheme.

    Source: §1 · p. 2

  • Results

    The method achieves state-of-the-art performance on public rain-streak and raindrop datasets and consistently better results on images with both rain types.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that the method can remove rain streaks and raindrops simultaneously and performs well in real-world scenarios.

    Source: §5 · p. 11

Abstract

from arXiv · show

Rain streaks and rain drops are two natural phenomena, which degrade image capture in different ways. Currently, most existing deep deraining networks take them as two distinct problems and individually address one, and thus cannot deal adequately with both simultaneously. To address this, we propose a Dual Attention-in-Attention Model (DAiAM) which includes two DAMs for removing both rain streaks and raindrops. Inside the DAM, there are two attentive maps - each of which attends to the heavy and light rainy regions, respectively, to guide the deraining process differently for applicable regions. In addition, to further refine the result, a Differential-driven Dual Attention-in-Attention Model (D-DAiAM) is proposed with a "heavy-to-light" scheme to remove rain via addressing the unsatisfying deraining regions. Extensive experiments on one public raindrop dataset, one public rain streak and our synthesized joint rain streak and raindrop (JRSRD) dataset have demonstrated that the proposed method not only is capable of removing rain streaks and raindrops simultaneously, but also achieves the state-of-the-art performance on both tasks.

1 INTRODUCTION

Rain streaks and raindrops often co-occur, but most deep deraining methods and datasets focus on one type; existing attention methods can also miss light rain when thresholds are unsuitable. The paper proposes DAiAM and D-DAiAM and introduces JRSRD, reporting state-of-the-art results on separate rain datasets and consistently better results on images containing both rain types.

1 INTRODUCTION

  • Rain streaks occlude objects and scenes, whereas raindrops can change their shape; both types often appear together in real-world images.Most deep learning deraining methods and datasets focus on only one type.Source: §1 · p. 1
  • Previous threshold-based attention methods focus on heavy-rain regions and can ignore light rain when the fixed threshold is unsuitable.The paper relates greater rain intensity to a larger pixel difference between rainy and clean images.Source: §1 · pp. 1–2
  • DAiAM uses two DAM branches to address rain streaks and raindrops simultaneously, with heavy- and light-rain attention maps in each branch.The maps attend to different rain intensities, avoiding the problems associated with an unsuitable fixed threshold.Source: §1 · p. 2
  • D-DAiAM applies a heavy-to-light scheme to refine DAiAM outputs and further remove rain of different intensities or types.The differential-driven module uses rainy inputs and DAiAM derained outputs to guide learning of the following DAiAM.Source: §1 · p. 2
  • JRSRD was built to evaluate removal of rain streaks and raindrops together, which often occur simultaneously.The paper presents joint-rain evaluation as necessary for verifying performance in the wild.Source: §1 · p. 2
  • The method achieves state-of-the-art performance on public rain-streak and raindrop datasets and consistently better results on images with both rain types.The introduction also identifies JRSRD as a dataset for evaluating joint rain-streak and raindrop removal.Source: §1 · p. 2

2 RELATED WORKS

The related work reviews prior approaches to rain streaks, raindrops, and attention-based deraining. It highlights that these tasks and datasets are usually separated, despite the real-world co-occurrence of both rain types.

2 RELATED WORKS

  • The paper frames its work as joint removal of rain streaks and raindrops using an attention mechanism.It notes that the two rain phenomena often co-occur, while existing methods and datasets typically address only one.Source: §2 · p. 2, §2.2 · p. 3

2.1 Rain Streak Removal

  • Rain-streak removal work includes hand-crafted image priors, deep-learning methods, video-based approaches, and non-local networks.The reviewed methods include dictionary learning and sparse coding, recurrent networks, and CGAN-based models.Source: §2.1 · p. 2

2.2 Raindrop Removal

  • Raindrop-removal methods include stereo or monocular detection, video cues, and CNNs trained on rainy and sharp image pairs.The review notes that most rain-streak methods are not directly applicable to raindrop removal.Source: §2.2 · pp. 2–3

2.3 Attention Mechanism

  • Attention-based deraining methods use thresholds to classify rainy and non-rainy regions, but unsuitable thresholds can cause errors and restrict their potential.This limitation motivates attention to rain intensity rather than only a binary rain/no-rain distinction.Source: §2.3 · p. 3

3 METHOD

The method builds from a Dual Attention Model (DAM) that separates heavy- and light-rain processing to DAiAM’s joint handling of streaks and drops, then adds staged differential guidance in D-DAiAM.

3.1 Overall Architecture of DAM

  • DAM encodes the rainy image into features and learns separate heavy- and light-rain attention maps; its intermediate outputs and input then feed a global decoder.The encoder E is described as a recurrent network with a CNN layer, three residual blocks, and an LSTM layer.Source: §3.1 · p. 3, §3.2 · p. 4

3.2 Dual Intensity-Aware Maps

  • DAM’s heavy- and light-rain-aware maps are derived from a soft, normalized rainy–clean image difference that represents rain intensity as well as presence.This soft mask avoids the threshold-related effects described for prior attention methods.Source: §3.2 · p. 4

3.3 Attentive Deraining from Regional and Global Levels

  • Two decoders use the respective attention maps to produce intermediate heavy- and light-rain derained images, which are combined with the rainy input for global decoding.The heavy-focused decoder emphasizes heavy-rain regions, while the other targets light-rain regions.Source: §3.3 · p. 4

3.4 Dual Attention-in-Attention Model

  • DAiAM has separate rain-streak and raindrop branches; within each, attention first distinguishes rain type and then targets heavy or light intensity.The branches are fused with a global decoder to produce the final derained image.Source: §3.4 · pp. 5–6

3.5 Differential-Driven DAiAM (D-DAiAM)

  • D-DAiAM uses successive DAiAM stages to transfer heavy rain toward light rain and then no rain, with output differences guiding attention to regions needing further work.The differential-driven module uses differences from the original input and the previous stage’s output to create soft maps for the next stage.Source: §3.5 · pp. 6–7

4 EXPERIMENTS

The experiments compare rain-streak and raindrop removal on public datasets, evaluate joint removal on the synthesized JRSRD dataset, and include ablations, intensity comparisons, and real-world examples. The reported results favor the proposed approach across these settings, with visual comparisons on real-world images.

4 EXPERIMENTS

  • The experiments include public rain-streak and raindrop datasets, a synthesized joint-rain dataset, component ablations, and real-world examples.The experimental plan also compares the separate-task results with state-of-the-art methods.Source: §4 · p. 7

4.1 Implementation Details

  • Training uses mini-batches of four and randomly cropped 112 × 112 patches, with horizontal flips for augmentation.The learning rate starts at 10^-4 and is reduced to 10^-6 after convergence; one differential-driven module is used to reduce training time.Source: §4.1 · p. 7

4.2 Results on Rain Streak Dataset

  • The proposed method outperforms the compared rain-streak methods quantitatively, and its visual result is better than PReNet.The reported model is D-DAM in this rain-streak-only setting; its latent attention map is also described as close to ground truth.Source: §4.2 · p. 8

4.3 Results on Raindrop Dataset

  • On the raindrop dataset, the authors report that their method is more advanced than DID-MDN, DDN, JORDER, and Qian et al. in quantitative and qualitative results.The method is D-DAM in this setting.Source: §4.3 · p. 8

4.4 Results on the Joint Rain Streak and Raindrop Dataset

  • On JRSRD, DAiAM beats the compared CNN-based methods for joint rain-streak and raindrop removal, and exemplar results suggest cleaner images.JRSRD contains both rain types; its test set has 232 pairs.Source: §4.4 · pp. 8–9

4.5 Ablation Study

  • The ablation results show that dual attention improves on single attention, DAiAM outperforms the attention variants, and D-DAiAM improves more than directly connecting two DAiAM models.The comparison also includes D-DAiAM(3), which uses two differential-driven modules.Source: §4.5 · p. 9, Table 7 · p. 10

4.6 Performance of Deraining for Different Rain Intensities

  • As rain intensity increases, all tested methods show more artifacts, while D-DAiAM and D-DAiAM(3) refine removal, especially for heavy rain.The authors also report that streaks and drops can affect each other during attention-map generation, causing erroneous maps.Source: §4.6 · p. 11

4.7 Deployment in Real World

  • D-DAiAM performs better than the compared methods on real-world rainy images, where those methods remove either raindrops or rain streaks.The authors report visual results only and omit quantitative evaluation such as recognition scores or a user study.Source: §4.7 · p. 11

5 CONCLUSION

The conclusion presents DAiAM for simultaneous removal of raindrops and rain streaks, using intensity-aware attention and a differential-driven refinement module. Experiments demonstrate performance against state-of-the-art methods and deraining in real-world scenarios; future work will explore supervision with purely heavy or purely light rain images.

  • DAiAM focuses on raindrops and rain streaks simultaneously through two DAMs that learn heavy- and light-rain-aware maps.Source: §5 · p. 11
  • A differential-driven module is introduced to optimize the deraining process.Source: §5 · p. 11
  • Experiments demonstrate performance against state-of-the-art methods and deraining in real-world scenarios.Source: §5 · p. 11
  • Future work will consider generating purely heavy-rain or purely light-rain images for supervision to help remove rain.Source: §5 · p. 11

Related papers

Loading 2103.07051v2…