Source-linked AI summary
Attentive Generative Adversarial Network for Raindrop Removal from a Single Image
Rui Qian, Robby T. Tan, Wenhan Yang, Jiajun Su, Jiaying Liu
TL;DR
The paper addresses fully automatic single-image removal of raindrops, where affected regions are unknown and much of the background is lost. It uses an attentive GAN that injects attention into both generation and discrimination, and reports stronger quantitative and qualitative performance than the compared methods.
Problem
Single-image raindrop removal must recover backgrounds when occluded regions are unknown and much of their information is lost.
Method
An attentive GAN generates an attention map, uses it with the input in an autoencoder, and guides global and local discrimination.
Results
The method outperforms Eigen13 and Pix2Pix quantitatively and qualitatively, while AA+AD performs better than the tested alternative configurations.
Takeaways & Limitations
Injecting attention into both the generative and discriminative networks is the paper’s main contribution and is reported to work effectively for raindrop removal.
Takeaways & Limitations
For real clean images, the discriminative-network formulation assumes no specific region requires focused attention.
Abstract
from arXiv · showhide
Raindrops adhered to a glass window or camera lens can severely hamper the visibility of a background scene and degrade an image considerably. In this paper, we address the problem by visually removing raindrops, and thus transforming a raindrop degraded image into a clean one. The problem is intractable, since first the regions occluded by raindrops are not given. Second, the information about the background scene of the occluded regions is completely lost for most part. To resolve the problem, we apply an attentive generative network using adversarial training. Our main idea is to inject visual attention into both the generative and discriminative networks. During the training, our visual attention learns about raindrop regions and their surroundings. Hence, by injecting this information, the generative network will pay more attention to the raindrop regions and the surrounding structures, and the discriminative network will be able to assess the local consistency of the restored regions. This injection of visual attention to both generative and discriminative networks is the main contribution of this paper. Our experiments show the effectiveness of our approach, which outperforms the state of the art methods quantitatively and qualitatively.
1. Introduction
The paper targets fully automatic removal of raindrops from single images, where occluded background regions are unknown and largely unrecoverable. It uses attention-guided adversarial generation and discrimination to focus restoration and local realism assessment on raindrop regions.
- Raindrops on windows, windscreens, or lenses degrade background visibility and image quality.
- Single-image raindrop removal is difficult because occluded regions are unspecified and much of their background information is lost.The difficulty increases for large, densely distributed raindrops.
- The attentive-recurrent network produces an attention map that guides the generative network toward raindrop regions.It combines deep residual networks, convolutional LSTM, and convolutional layers.
- An autoencoder receives both the input image and attention map, using multi-scale and perceptual losses to restore contextual and global similarity.
- The discriminative network uses the attention map to assess local consistency despite unknown target regions during testing.It evaluates generated images globally and locally.
- Injecting attention into both generative and discriminative networks is presented as the paper’s main contribution and works effectively in experiments.
2. Related Work
Prior raindrop-removal methods generally require stereo, video, or specialized hardware, while the main single-image baseline struggles with substantial raindrops. The paper instead adapts GAN-based restoration while addressing the absence of known target regions.
- Haze, fog, and rain-streak enhancement methods cannot be directly applied because raindrop image formation and constraints differ.
- Existing raindrop-detection approaches face challenges modeling transparent raindrops with varied shapes and avoiding locally similar false detections.
- Methods based on stereo, video, or optical shutters are not applicable to a single image from a normal camera.
- Eigen et al.’s single-image CNN handles relatively sparse, small droplets and dirt but produces blurry results for large, dense raindrops.
- The method uses a GAN backbone, but unlike inpainting, raindrop target regions are not given for local discrimination.
3. Raindrop Image Formation
The paper models a raindrop-degraded image as a masked background combined with raindrop effects, then seeks to recover the background image. Raindrop regions contain distorted environmental imagery, although peripheral areas may retain background information.
- The image model is I = (1 − M) ⊙ B + R, combining the masked background image with raindrop effects.
- M is a binary mask whose value 1 identifies raindrop pixels, while B denotes the background image and R the raindrop effect.
- The raindrop effect R mixes background information with light reflected by the environment and transmitted through raindrops.
- Because raindrops refract light from the wider environment, their regions can contain imagery different from the background and are mostly blurred.
- The method seeks B from I and creates an attention map guided by the binary mask M during training.Training masks are obtained by subtracting the degraded image from its corresponding clean image and thresholding at 30.
4. Raindrop Removal using Attentive GAN
The attentive GAN combines recurrent visual attention, contextual reconstruction, and attention-guided discrimination to remove raindrops. Attention progressively emphasizes raindrops and surrounding structures for restoration and local assessment.
- Architecture: The generator contains an attentive-recurrent network and a contextual autoencoder, while the discriminator uses convolutional layers guided by the attention map.The attentive-recurrent network identifies regions needing attention; the autoencoder then generates the raindrop-free image.
- Attentive-Recurrent Network: The recurrent attention network uses ResNet features, a convolutional LSTM, and convolutional layers to generate attention maps across time steps.The maps are initialized at 0.5 and concatenated with the input for subsequent recurrent blocks.
- Attentive-Recurrent Network: The attention map is a non-binary 0-to-1 matrix whose larger values indicate greater attention, including raindrops and their surrounding structures.Unlike the binary mask, it can vary within raindrop regions because transparency and contextual relevance differ.
- Attentive-Recurrent Network: The attention loss compares each time-step map with a binary raindrop mask using MSE, with N=4 and θ=0.8 in the described setup.Later attention maps receive larger values as confidence increases; higher N is expected to improve maps but requires more memory.
- Contextual Autoencoder: The contextual autoencoder concatenates the input image with the final attention map and uses skip connections, multi-scale losses, and perceptual loss.Multi-scale outputs target contextual information at different decoder resolutions, while perceptual loss compares pretrained-CNN features.
- Discriminative Network: The attentive discriminator uses the attention map to guide feature weighting toward local target regions when raindrop locations are unavailable during testing.This complements global and local consistency assessment by directing discriminative processing toward attention-indicated regions.
5. Raindrop Dataset
The authors construct paired raindrop-degraded and clean images with identical background scenes because suitable glass-window or lens raindrop data are unavailable. They capture 1119 image pairs under varied scene and raindrop conditions.
- Dataset Construction: The dataset requires image pairs with exactly the same background scene, with one image degraded by raindrops and the other free from raindrops.The authors create their own dataset because no such ground-truth dataset was available for raindrops attached to glass or lenses.
- Dataset Construction: 1119 image pairs were captured using Sony A6000 and Canon EOS 60 cameras across varied background scenes and raindrops.The glass slabs were 3 mm thick, placed 2–5 cm from the camera, and attached to the camera lens.
- Dataset Samples: Figure 5 places raindrop-degraded images on top and their corresponding ground-truth images on the bottom.The paired layout visually demonstrates the degraded-to-clean training correspondence.
6. Experimental Results
The attentive GAN outperforms Eigen13 and Pix2Pix in quantitative and qualitative raindrop-removal evaluations. Ablations and application tests support the value of attention in both networks and show improved object recognition.
- Higher PSNR and SSIM values than Eigen13 and Pix2Pix indicate results more similar to the ground truths.
- AA+AD performs better than A, A+D, and A+AD, providing quantitative evidence that attention benefits both generative and discriminative networks.
- Qualitatively, the complete network removes raindrops more effectively than Eigen13, Pix2Pix, and its partial configurations.
- The attention map increasingly focuses on raindrop regions and relevant structures as recurrent-network time steps increase.
- Compared with Pix2Pix, the method produces fewer artifacts and better restored structures.
7. Conclusion
The paper proposes a single-image attentive GAN for removing severe raindrops. Its attention map guides both generation and discrimination, and the method is presented as the first to handle relatively severe raindrop presence.
- The proposed method removes raindrops from a single image using an attentive-recurrent network and contextual autoencoder within a GAN.
- Injecting the attention map into both generative and discriminative networks is the method’s stated novelty.
- Application results show improved main-object recognition scores and more recognized object labels with the method’s outputs.
- The method is presented as the first to handle relatively severe raindrop presence that state-of-the-art removal methods fail to handle.