Source-linked AI summary
Learning from synthetic photorealistic raindrop for single image raindrop removal
Zhixiang Hao, Shaodi You, Yu Li, Kunming Li, Feng Lu
TL;DR
Adherent raindrops can degrade computer vision in rainy scenes, while realistic paired training data for single-image removal are limited. The paper introduces a photorealistic synthetic dataset and a physics-aware detection-and-removal network; experiments report state-of-the-art performance on synthetic and real-world data.
Problem
Realistic aligned rainy/clear training pairs and pixel-level masks are difficult to obtain, limiting supervised learning for single-image raindrop removal.
Source: §3 · p. 3Method
Results
The authors report that their method outperforms the state of the art and handles both synthetic and real-world data.
Source: §6 · p. 8Takeaways & Limitations
Abstract
from arXiv · showhide
Raindrops adhered to camera lens or windshield are inevitable in rainy scenes and can become an issue for many computer vision systems such as autonomous driving. Because raindrop appearance is affected by too many parameters, therefore it is unlikely to find an effective model based solution. Learning based methods are also problematic, because traditional learning method cannot properly model the complex appearance. Whereas deep learning method lacks sufficiently large and realistic training data. To solve it, in our work, we propose the first photo-realistic dataset of synthetic adherent raindrops for training. The rendering is physics based with consideration of the water dynamic, geometric and photometry. The dataset contains various types of rainy scenes and particularly the rainy driving scenes. Based on the modeling of raindrop imagery, we introduce a detection network which has the awareness of the raindrop refraction as well as its blurring. Based on that, we propose the removal network that can well recover the image structure. Rigorous experiments demonstrate the state-of-the-art performance of our proposed framework.
1. Introduction
Adherent raindrops obstruct, deform, or blur scene regions and can degrade computer-vision systems, while single-image removal methods and realistic training data remain limited. The paper introduces a physics-aware detection-and-removal network and a photorealistic synthetic dataset with pixel-level masks.
1. Introduction
- Adherent raindrops obstruct, deform, or blur parts of background scenes, degrading vision algorithms and motivating automatic single-image removal.The introduction identifies feature detection, tracking, and stereo correspondence among affected tasks.Source: §1 · p. 1
- Existing approaches face limits including reliance on video, oversimplified models, insufficient learning capacity, or small and unrealistic test data.The cited approaches include video-dependent removal, simplified single-image detection, a three-layer network, and a method tested on a small sunny-scene dataset.Source: §1 · pp. 1–2
- The proposed physics-aware, multi-task network detects raindrop locations and shapes, reconstructs affected regions, then smooths the blended image.The design reflects raindrop imagery in which drops are partially blended with, and reflect, the background.Source: §1 · p. 2
- The introduced dataset contains about 30K photo-realistically rendered rainy and clear images using Cityscapes backgrounds, each with 50–70 raindrops sized 0.8–1.5 cm and blur levels of 7–20 pixels.Source: §1 · p. 2
- The authors report that their method outperforms existing methods on existing datasets and real-world rainy images.The introduction states this as a contribution without providing benchmark values here.Source: §1 · p. 2
2. Related Work
Prior work addresses raindrop modeling and removal, but single-image removal remains challenging and research is described as relatively insufficient. Existing methods range from physical models to video-based techniques and learned approaches.
2. Related Work
- Single-image raindrop removal is an ill-posed problem with potential relevance to outdoor vision systems, including surveillance and intelligent vehicles.The section characterizes research on single-image removal as relatively insufficient compared with work on haze or rain streaks.Source: §2 · p. 2
2.1. Adherent Raindrop Modeling
- Raindrop models progress from ray-tracking and geometric representations to methods incorporating liquid dynamics, optics, and surface constraints.Roser et al. use low-dimensional Bézier curves, while You et al. [40] and model optical and physical properties.Source: §2.1 · p. 2
2.2. Raindrop Removal
- Many removal methods rely on stereo or video; single-image detection by Roser and Geiger supports multi-view fusion rather than single-frame restoration.You et al. use video completion and temporal intensity derivatives after detecting raindrop locations.Source: §2.2 · p. 2
- Eigen et al. handle relatively sparse, small drops and dirt but produce blurred outputs and cannot remove dense drops; Qian et al. [24] report sharp results on their test set.Qian et al. use an aligned dataset made with water-sprayed glass and a GAN with attention in both generator and discriminator.Source: §2.2 · p. 2
- Pix2Pix [17] can address the task as image-to-image translation but was not designed specifically for single-image raindrop removal.Source: §2.2 · p. 2
3. Raindrop Imagery Model and Photorealistic Dataset
The paper builds a photorealistic adherent-raindrop dataset by rendering drops on clear backgrounds with a physical refraction model. Its generation process adds blur and provides pixel-level masks, using Cityscapes scenes for outdoor-vision relevance.
3. Raindrop Imagery Model and Photorealistic Dataset
- The dataset addresses the shortage of aligned rainy/clear pairs and pixel-level raindrop masks by rendering drops on Cityscapes backgrounds.The authors describe the dataset as the first photorealistic adherent-raindrop dataset with pixel-level masks in autonomous-driving settings.Source: §3 · p. 3
- Ray tracing leaves pixels unchanged when rays miss drops and refracts rays passing through drops to determine background-plane samples.Total internal reflection is rendered as black, corresponding to dark bands observed at real-world raindrop boundaries; Figure 2 contrasts unobstructed and refracted rays.Source: §3 · p. 3, Figure 2 · p. 3
- A disk blur kernel models focus-related blur over drop regions, and known drop locations provide pixel-level binary masks.The selected blur-kernel diameter is 7–20 pixels; Figure 3 shows clear input, synthetic raindrop image, and ground-truth mask.Source: §3 · p. 3, Figure 3 · p. 3
- Cityscapes supplies urban-street backgrounds, including cloudy-weather scenes, making the dataset more suitable for outdoor vision, especially autonomous driving.The generated set contains about 30,000 training images and 1,525 test images, with 50–70 drops per background.Source: §3 · p. 3
4. End-to-End Raindrop Detection and Removal Network
The method models raindrop degradation with a three-subnetwork end-to-end system that detects raindrops, reconstructs covered regions, and refines the blended image.
4.1. Raindrop Detection Network
- A pixel-level mask network detects raindrops, using dense feature maps to retain location detail and a convolutional residual architecture trained with binary cross-entropy.The network downsamples feature maps to half the input resolution and has five convolution layers and six residual blocks.Source: §4.1 · p. 4
4.2. Raindrop Region Reconstruction Network
- The reconstruction network uses the input image and its edge image to recover blurred raindrop regions, weighting raindrop pixels 20 times more heavily in its loss.The weight is 20 for raindrop pixels and 1 elsewhere.Source: §4.2 · pp. 4–5
4.3. Refine network
- The refine network combines the detected mask and reconstructed regions, then smooths the blend; mask dilation removes raindrop-edge pixels remaining on the road.Its loss mixes SSIM and ℓ1 terms; Figure 5 contrasts blended inputs made with the original and dilated masks.Source: §4.3 · p. 5, Figure 5 · p. 5
4.4. Implementation Details
- Training fits detection and reconstruction separately, then trains only the refine network; training pairs are randomly cropped to 100 × 100 pixels.Each subnetwork is trained for 50 epochs, with a batch size of 32 and Adam optimization.Source: §4.4 · p. 5
5. Experiments
The experiments evaluate raindrop detection and removal on synthetic and real-world images, comparing the proposed method with existing approaches and testing its modules and efficiency.
5. Experiments
- The synthetic test set contains 1,525 rainy/clear image pairs and masks; Eigen, Qian, DID-MDN, and Pix2Pix are compared using PSNR and SSIM across the synthetic and Qian datasets.Existing methods are trained on the proposed synthetic training set for the comparison.Source: §5.1 · p. 6, §5 · p. 6
5.1. Experiments on Synthetic Image
- Detection reaches AP 0.9973 on synthetic images; comparison with other methods is unavailable because they do not predict raindrop masks.The test data include ground-truth binary raindrop masks.Source: §5.1 · p. 6
- On the synthetic dataset, the complete model outperforms other methods by a large margin in both PSNR and SSIM.Table 1 reports quantitative removal results; its last row is the complete model.Source: §5.1 · p. 6, Table 1 · p. 6
- On synthetic examples, the proposed method removes raindrops and avoids artifacts better than Qian and DID-MDN; Eigen and Pix2Pix retain many drops.Qian and DID-MDN also leave artifacts or raindrops in the illustrated results.Source: §5.1 · p. 6, Figure 7 · p. 7
- The complete architecture improves PSNR by 9.4% over 3RN-only; adding RDN and RFN raises PSNR and SSIM, while dilated masks yield the highest scores.The authors report that all proposed modules contribute to removal.Source: §5.1 · p. 6, Table 1 · p. 6
- 69.74 ms per 256 × 512 image for the proposed method, versus 79.1 ms Pix2Pix, 97.16 ms Qian, 107.712 ms Eigen, and 133.64 ms DID-MDN.Times are averages of 20 runs on one NVIDIA TITAN Xp GPU.Source: §5.1 · p. 8
5.2. Experiments on Real-world Image
- On Qian’s real-world dataset, the methods are quantitatively on par with Qian, but similar training and test data limit the challenge and missing masks prevent direct full-model training.The authors instead use roughly estimated masks or train the 3RN network, which does not need masks.Source: §5.2 · p. 8
6. Conclusions
The conclusion presents a realistic masked dataset and a two-stage network for joint adherent-raindrop detection and removal. It reports performance on synthetic and real-world data and proposes further validation on rainy-scene vision systems.
6. Conclusions
- The dataset provides realistic adherent-raindrop images with pixel-level binary masks to address the shortage of paired training data.The authors identify insufficient paired training data as a challenge for supervised deep learning in raindrop removal.Source: §6 · p. 8
- The multi-task network detects raindrop locations and restores drop regions, then blends location clues and refines the resulting image.The conclusion describes these as two stages of the network.Source: §6 · p. 8
- Experiments show the method outperforms the state of the art on synthetic and real-world data; future work will enhance the raindrop model and validate effects on rainy-scene vision systems.The authors specifically name computer vision systems operating in rainy scenes as a future validation target.Source: §6 · p. 8