Source-linked AI summary
Single Image Deraining: A Comprehensive Benchmark Analysis
Siyuan Li, Iago Breno Araujo, Wenqi Ren, Zhangyang Wang, Eric K. Tokuda, Roberto Hirata Junior, Roberto Cesar-Junior, Jiawan Zhang, Xiaojie Guo, Xiaochun Cao
TL;DR
Single-image deraining lacks comprehensive evaluation across diverse rain conditions, realistic data, and task-relevant criteria. This paper introduces MPID and benchmarks representative methods with objective, subjective, and task-driven evaluations, revealing method comparisons and evaluation limitations.
Problem
Existing deraining studies lack large-scale, comprehensive benchmarking across rain types, realistic images, human perception, and computer-vision utility.
Method
The paper constructs MPID and evaluates six representative deraining algorithms using full-reference, no-reference, subjective, and object-detection task-specific criteria.
Results
Experiments reveal differing algorithm performance across rain streak, raindrop, and rain-and-mist conditions, with subjective scores potentially misaligned with no-reference perceptual metrics.
Takeaways & Limitations
MPID supports broader evaluation of deraining methods across synthetic and real-world data, rain models, human judgments, and downstream object detection.
Takeaways & Limitations
Subjective score results were still being strengthened with additional human raters, and no-reference metrics may not align well with human perception.
Abstract
from arXiv · showhide
We present a comprehensive study and evaluation of existing single image deraining algorithms, using a new large-scale benchmark consisting of both synthetic and real-world rainy images.This dataset highlights diverse data sources and image contents, and is divided into three subsets (rain streak, rain drop, rain and mist), each serving different training or evaluation purposes. We further provide a rich variety of criteria for dehazing algorithm evaluation, ranging from full-reference metrics, to no-reference metrics, to subjective evaluation and the novel task-driven evaluation. Experiments on the dataset shed light on the comparisons and limitations of state-of-the-art deraining algorithms, and suggest promising future directions.
1. Introduction
Single-image deraining is an ill-posed visibility-restoration problem whose evaluation has lacked comprehensive, realistic coverage. The paper addresses this gap with MPID, a diverse benchmark and multi-axis evaluation framework spanning rain types, data sources, and downstream tasks.
- Rain degrades scene visibility, motivating deraining for human perception and computer-vision applications such as surveillance and intelligent vehicles.
- Existing deraining research has progressed through image priors and CNN-based models, but lacked a fair comprehensive study of algorithms and performance metrics.
- Rainy Image Formulation Models: Rain degradation includes streaks, raindrops, and mist, while most existing methods assume only one rain model, typically rain streaks.
- Synthetic results often fail to capture real-rain complexity, and PSNR and SSIM may poorly reflect human perception or computer-vision utility.
- Our Contribution: MPID covers rain streak, raindrop, and rain-and-mist images across synthetic and real-world sources, with diverse contents and annotated detection scenarios.
- Our Contribution: The benchmark evaluates deraining with full-reference, no-reference, subjective, and task-specific criteria, including impacts on object detection.
2. Related Work
Prior deraining work spans multi-frame methods, CNN-based rain-streak removal, raindrop removal, and datasets built mainly from synthetic or limited real-world images.
- Multi-frame based approaches: Early deraining methods often required multiple video frames, using temporal information and camera-parameter selection to remove rain streaks.
- CNN-based methods jointly detect and remove rain streaks, estimate rain density, or use visual attention and GANs for raindrop removal.
- Earlier datasets included small photo-realistic sets, 700 training and 100 testing images, and 50 real-world rainy images for qualitative evaluation.
3. New Benchmark: Multi-Purpose Image Deraining (MPID)
MPID is a comprehensive benchmark spanning multiple rain models, data sources, evaluation criteria, and application settings. It combines synthesized training and testing data with real-world testing and task-driven datasets for assessing restoration, perceptual quality, and downstream detection.
- Evaluation platform: MPID evaluates deraining from traditional restoration metrics through no-reference perceptual measures, subjective quality, and task-driven computer-vision performance.Its evaluation framework includes PSNR/SSIM, NIQE, SSEQ, BLIINDS-II, human assessment, and downstream object detection.
- Testing sets: Synthetic testing uses Rain streak (S), Rain drop (S), and Rain and mist (S), containing 200, 149, and 70 clean-rainy pairs, respectively.These sets support restoration evaluation with PSNR and SSIM and complementary no-reference image-quality assessment.
- Testing sets: Three real-world test sets—Rain streak (R), Raindrop (R), and Rain and mist (R)—are collected and inspected to evaluate generalization without readily available clean ground truth.The sets are organized by predefined rain type and collected from Internet images.
- Task-Driven Evaluation Sets: Task-driven evaluation uses RID and RIS, where object bounding boxes support measuring object-detection performance after deraining.RID contains 2,495 real rainy images from high-resolution driving videos, while RID and RIS differ in rain type, image quality, object size, and angle.
4. Experimental Comparison
Experiments across synthetic and real rainy images, subjective judgments, and object detection reveal that deraining performance depends strongly on rain type, evaluation criterion, and downstream task.
- Experimental Setup: Six representative deraining algorithms were evaluated on rain streak, raindrop, and rain-and-mist data, including retrained baselines and a cascaded deraining-dehazing pipeline.The benchmark combines specialized model comparisons with a cascade for the rain-and-mist setting.
- Objective Comparison: DDN led synthetic rain-streak and rain-and-mist results, while DeRaindrop performed best on synthetic raindrops, especially on PSNR and SSIM.The reported synthetic comparisons used PSNR, SSIM, NIQE, SSEQ, and BLIINDS-II.
- Objective Comparison: On real images, JORDER and DDN ranked highest for rain streaks, DeRaindrop dominated raindrops, and CGAN won rain-and-mist comparisons by large margins.The real-image evaluation used NIQE, SSEQ, and BLIINDS-II, and the rain-and-mist result suggests a larger synthetic-to-real domain gap.
- Subjective Comparison: Human preferences differed from objective metrics: CGAN was preferred for rain streaks, GMM for raindrops, and DID-MDN for rain and mist.The subjective scores are ranks rather than absolute values; the survey reported good consensus despite relatively few raters.
- Task-driven Comparison: Object detection results were broadly aligned across RID and RIS, but almost all deraining algorithms reduced detection performance relative to rainy inputs for three detectors.YOLO-V3 achieved the best detection performance independently of the deraining algorithm, while FRCNN on RID was a less reliable indicator because of domain mismatch.
- Conclusions and Future Work: The study concludes that no single deraining algorithm is best across rain types or metrics, motivating mixtures of experts, scene-specific models, and combinations of priors with data-driven methods.The authors also encourage algorithms designed around high-level vision tasks because existing methods did not directly improve detection.