PaperlayerView exact revision on arXiv

Source-linked AI summary

Rain rendering for evaluating and improving robustness to bad weather

Maxime Tremblay, Shirsendu Sukanta Halder, Raoul de Charette, Jean-François Lalonde

arXiv:2009.03683v1cs.CV

TL;DR

Rain disrupts image formation, but existing datasets do not support systematic evaluation at controlled rainfall rates. The paper introduces physics-based, data-driven, and hybrid rain augmentation for real image datasets, finding substantial degradation across vision tasks and improved robustness after finetuning.

  • Problem

    Existing datasets lack calibrated rainfall rates, making the impact of specified rain levels on computer vision performance difficult to measure.

  • Method

    The paper augments KITTI, Cityscapes, and nuScenes with controllable synthetic rain using physics-based rendering, image-to-image translation, and their combination.

  • Results

    Rain reduces performance across tasks, with drops of 15% mAP for object detection, 60% AP for semantic segmentation, and a 6-fold increase in depth estimation error.

  • Takeaways & Limitations

    Rain-augmented datasets support systematic robustness evaluation and finetuning for improved performance on real rainy images.

  • Takeaways & Limitations

    The physical pipeline’s lighting estimation can differ from ground-truth illumination, especially in non-radially symmetric scenes.

Abstract

from arXiv · show

Rain fills the atmosphere with water particles, which breaks the common assumption that light travels unaltered from the scene to the camera. While it is well-known that rain affects computer vision algorithms, quantifying its impact is difficult. In this context, we present a rain rendering pipeline that enables the systematic evaluation of common computer vision algorithms to controlled amounts of rain. We present three different ways to add synthetic rain to existing images datasets: completely physic-based; completely data-driven; and a combination of both. The physic-based rain augmentation combines a physical particle simulator and accurate rain photometric modeling. We validate our rendering methods with a user study, demonstrating our rain is judged as much as 73% more realistic than the state-of-theart. Using our generated rain-augmented KITTI, Cityscapes, and nuScenes datasets, we conduct a thorough evaluation of object detection, semantic segmentation, and depth estimation algorithms and show that their performance decreases in degraded weather, on the order of 15% for object detection, 60% for semantic segmentation, and 6-fold increase in depth estimation error. Finetuning on our augmented synthetic data results in improvements of 21% on object detection, 37% on semantic segmentation, and 8% on depth estimation.

1 Introduction

Rain introduces atmospheric artifacts that challenge vision algorithms, while existing datasets lack calibrated rainfall rates for measuring the effect. The paper addresses this gap with controllable rain augmentation and evaluates robustness across multiple tasks.

  • 1 Introduction: Existing weather datasets rarely provide precise rainfall information, making real-world evaluation at a specified rate impractical or prohibitively expensive.Available bad-weather datasets typically contain only high-level rain labels rather than rainfall amounts.
  • 1 Introduction: The paper proposes physics-based and combined GAN-plus-physics rain rendering methods for realistically augmenting existing image databases.The combined method adds scene wetness through unpaired image translation before compositing physics-based rain.
  • 1 Introduction: The physics-based method controls rainfall from 5 mm/hour to 200+ mm/hour, producing calibrated datasets for systematic robustness evaluation.This range spans very light rain to very heavy storms.
  • 1 Introduction: 15% mAP, 60% AP, and a 6-fold increase in depth estimation error quantify rain-related degradation across object detection, semantic segmentation, and depth estimation.The evaluation covers 13 popular algorithms on rain-augmented KITTI, Cityscapes, and nuScenes datasets.
  • 1 Introduction: The framework extends prior work with learning-based translation, a deeper user study, depth evaluation, deraining comparisons, and broader real-image improvement analysis.The authors also report that the augmented database can be used to finetune algorithms for real rainy conditions.

2 Related work

Prior work spans physical rain modeling, rain removal, unpaired image translation, and weather datasets. This paper combines these strands to augment real images rather than relying only on virtual environments or sparsely labeled weather data.

  • 2 Related work: Prior rain-synthesis research models streak appearance using image databases, frequency methods, ray tracing, and related physical approaches.These models provide foundations for realistic rain rendering.
  • 2 Related work: Rain-removal methods range from photometric and frequency-space techniques to sparse coding, priors, and deep networks.This literature addresses the adverse effects of rain by attempting to remove or attenuate it.
  • 2 Related work: Unpaired image-translation methods learn weather transformations across datasets without paired clear-rain images, using shared latent spaces, cycle consistency, or semantic consistency.These approaches can produce qualitative weather changes but do not by themselves provide physically calibrated rainfall rates.
  • 2 Related work: Existing weather databases include rain but generally lack systematic rainfall-rate labels and dense scene annotations.The paper addresses this limitation by augmenting real image datasets with controlled synthetic rain.
  • 2 Related work: The proposed physics-based rendering uses particle simulation with depth and illumination estimation to add controllable rain to clear images.This connects physical rain modeling with augmentation of real datasets rather than purely virtual scenes.

3 Rain Augmentation

The paper combines physics-based rendering and learning-based translation to synthesize controllable, realistic rain in existing images. Its physics-based pipeline models attenuation, individual streaks, photometry, compositing, and camera exposure, while validation identifies lighting-estimation limitations.

  • Physics-based rendering: Physics-based rendering combines depth-based fog-like attenuation, simulated rain streaks, and compositing to create controllable rainfall rates.The pipeline first generates attenuation, then renders drops individually, and finally adjusts global luminosity.
  • Fog-like rain: Fog-like rain models subpixel drops as volumetric extinction and airlight scattering, using rainfall rate and pixel depth to determine attenuation.Most drops are treated as fog-like rain because they project to less than one pixel; only a small fraction are individually rendered.
  • Rain streaks: Individual streaks are generated from a particle simulator, matched to database appearances, warped to simulated dynamics, and blurred for camera depth of field.The simulator accounts for camera calibration and outputs visible drops in world and image space.
  • Drop photometry: Drop photometry integrates estimated environmental lighting over a 165° field of view, accounting for refraction and reflection.The method estimates an environment map from the image and computes the relevant field-of-view intersection for each drop.
  • Image compositing: The renderer restores mean rainy-image radiance to model camera exposure changes during cloudy rainfall.It preserves the relation ¯I = ¯Irain after rain rendering.
  • Photometric validation: Estimated lighting produces visually similar rain to ground-truth illumination, but can misestimate radiance when scenes are not radially symmetric or the sky is poorly represented.The limitation is especially visible for streaks against a darker sky.

3.2 Image-to-image translation (GAN)

The paper uses unpaired image translation to capture rainy-scene appearance that physics-based rendering omits. Its GAN+PBR hybrid first translates clear images into rainy scenes, then composites physically rendered streaks.

  • Motivation: Physics-based rendering produces streaks and fog-like effects but omits wetness, reflections, clouds, and overall rainy-scene appearance.These visual characteristics provide strong signals for adversarial learning.
  • Image-to-image translation: CycleGAN learns a clear-to-rain mapping from unpaired clear and rainy images.The model uses the 256 × 256 CycleGAN architecture with 448 × 256 input images.
  • Hybrid rendering: The GAN+PBR hybrid first translates a clear image with CycleGAN and then composites physically rendered rain streaks onto it.The translated image’s darker irradiance and environment map keep fog-like rain and streak appearance coherent with the scene.
  • Hybrid rendering: The hybrid renderer upsamples GAN output for full-resolution PBR processing before downsampling the completed rainy image.This preserves the rendering treatment for streaks smaller than one pixel, which are otherwise represented as fog-like rain.

4 Validating rain appearance

The study compares physics-based, GAN-based, and combined rain renderings with real photographs and prior methods, finding complementary strengths in physical accuracy, wetness, and controllability. User studies show all proposed approaches are judged more realistic than previous techniques, while GAN+PBR balances rain characteristics with controllable intensity.

  • 4.1 Qualitative evaluation: PBR reproduces sparse, background-dependent streaks, depth-dependent veiling, and defocus, but lacks visible wetness.Its streak orientation is consistent with camera motion and its photometry with background and depth.
  • 4.1 Qualitative evaluation: GAN renders convincing wetness and global illumination but introduces geometrically incorrect reflections, whole-image blur, and no control over rain amount.The blur arises because the GAN cannot disentangle scene content from lens drops in rainy training images.
  • 4.2 User study: GAN+PBR combines controllable streaks with learned wetness, offering a trade-off that balances the visual characteristics associated with real rain.Users identified falling rain and wetness as the main characteristics in real rain; GAN+PBR balances both.
  • 4.2 User study: All proposed rain augmentation approaches are judged more realistic than previous methods, with MOS values of 0.44 for PBR, 0.68 for GAN, and 0.52 for GAN+PBR.Real photographs score 0.77, while prior methods score 0.30, 0.23, and 0.08.
  • 4.2 User study: The second user study evaluates which visual qualities convey rain realism, emphasizing falling rain and wetness as complementary cues.GAN excels at wetness but lacks falling-rain appearance, whereas PBR shows the opposite pattern.

5 Evaluating the impact of real rain

The authors evaluate object detection, semantic segmentation, and depth estimation on weather-validated clear and rainy nuScenes images. Real rain degrades all three tasks, but the setup cannot relate performance changes to a measured rainfall intensity.

  • 5 Evaluating the impact of real rain: The evaluation uses nuScenes-clear and nuScenes-rain subsets cross-validated against a historical weather database to reduce label noise.The resulting sets contain 24,134 clear images and 6,028 rainy images.
  • 5 Evaluating the impact of real rain: Object detection, semantic segmentation, and depth estimation are evaluated with YOLOv2, PSPNet, and Monodepth2 respectively on the weather-split data.Clear images are divided into train/test subsets, while rainy images provide a separate test condition.
  • 5 Evaluating the impact of real rain: Real rain reduces object-detection mAP from 32.53% to 16.30%, segmentation AP from 40.8% to 18.7%, and increases depth square relative error from 2.96% to 3.53%.These comparisons use clear and rainy nuScenes test subsets, with segmentation labels annotated for 25 images.
  • 5 Evaluating the impact of real rain: The real-image experiment demonstrates degradation under rain but cannot measure how performance varies with rainfall rate because capture-time intensity is unavailable.Accurate rainfall-rate measurement would be required to establish that relationship.

6 Evaluating the impact of synthetic rain

Controlled synthetic rain augmentation enables performance evaluation across rainfall rates on KITTI, Cityscapes, and nuScenes. Increasing rain substantially degrades detection, segmentation, and depth estimation, while GAN-based compositing produces additional task-dependent effects beyond PBR alone.

  • 6.1 Evaluating PBR rain augmentation: The pipeline augments KITTI, Cityscapes, and nuScenes-clear with calibrated PBR rain rates R = {0, 5, 25, 50, 100, 200} mm/hr.PBR and GAN+PBR permit performance measurement under controlled rain, while GAN and GAN+PBR are applied only to nuScenes-clear.
  • 6.2 Evaluating PBR rain augmentation: Under 200 mm/hr PBR rain, detection drops at least 12%, segmentation drops almost 60%, and depth error reaches 3x the clear-image value.Detection losses reach 25–30% for R-FCN, SSD, and MX-RCNN; segmentation losses reach 85% for ESPNet.
  • 6.2 Evaluating PBR rain augmentation: Detection becomes increasingly unreliable as rain obscures objects, especially distant small objects, while Faster R-CNN and DSOD are comparatively robust to fog and rain.The qualitative effect is highly variable because object occlusion differs across images.
  • 6.2 Evaluating PBR rain augmentation: Segmentation methods break down under heavy fog and strong rain, with algorithms changing relative rank as rainfall increases.ESPNet falls from the top three in clear weather to last in 200 mm/hr storm conditions after an 85% relative drop.
  • 6.3 Evaluating GAN and GAN+PBR rain augmentations: GAN-augmented detection performs similarly to PBR at 100 mm/hr, whereas GAN+PBR causes steeper deterioration and depth error 80–100% worse than PBR at matched intensity.The authors attribute the differing curves to cumulative wetness and streak domain shifts, potentially producing a non-linear effect.

7 Improving the robustness to rain

The rain-rendering pipeline is used to evaluate and improve vision-model robustness across synthetic and real rainy images. Finetuning generally improves robustness, while de-raining offers task-dependent trade-offs.

  • Synthetic rain: Object detection, semantic segmentation, and depth estimation improve after finetuning on synthetic rain, including robustness to unseen 200mm/hr rain.The authors attribute this to models learning stronger relevant features; the synthetic evaluation uses separate 1000-image train and test subsets with rain up to 200mm/hr.
  • Evaluation setup: The hybrid GAN+PBR figures compare clear, GAN-augmented, and three GAN+PBR images, while Fig. 17 contrasts untuned and finetuned performance across nuScenes and Cityscapes.Fig. 17 reports lower rain-related performance decreases for finetuned models.
  • Synthetic rain: At 100mm/hr, hybrid GAN+PBR finetuning yields 21.7% detection performance versus 7.5% untuned and 5.7% depth error versus 8.9% untuned.For PBR, YOLOv2 declines from 34.5% to 31.0%, while the untuned model declines from 34.6% to 20.4%; PSPNet reaches 54.0% versus 52.0% untuned at 100mm/hr.
  • Real rain: On real rainy nuScenes images, finetuning improves object detection by 20.7% for PBR and 21.0% for GAN+PBR, segmentation by 36.9%, and depth estimation by 8.2% for GAN+PBR.In clear weather, finetuned models perform on par with or sometimes better than untuned models.
  • Real rain: PBR finetuning underperforms for depth estimation because Monodepth2’s reprojection loss is incompatible with rain streaks that do not reproject across consecutive frames.The authors suggest GAN-based finetuning avoids this issue partly because its training images resemble the test set more closely.
  • De-raining comparison: Compared with de-raining followed by untuned models, rain-aware models perform better for object detection, whereas de-raining performs better for depth estimation on synthetic and real images.De-raining may outperform rain-aware algorithms but requires an additional processing task, potentially limiting practical applications.

8 Discussion

The discussion presents intensity-controlled physical rain augmentation as a basis for systematic multi-task evaluation, while identifying rendering, GAN, and robustness limitations. Synthetic-rain finetuning helps on real rain, but improvements remain modest.

  • Discussion: The intensity-controlled physical framework enables systematic study of rain’s impact on object detection, semantic segmentation, and depth estimation.The paper presents this as its central framework for augmenting existing image databases with realistic rain.
  • PBR limitations: PBR lighting approximations can misestimate scene radiance when the sky is insufficiently or excessively visible, especially against darker skies.The method also sharply distinguishes fog-like rain from individually rendered streaks, while rendering all drops as streaks would be prohibitively expensive.
  • PBR limitations: PBR streaks are added irrespective of scene contents, and its computational cost increases with rainfall intensity, though offline execution avoids a benchmarking downside.Depth estimates could mask streaks behind objects and below the ground plane.
  • GAN limitations: CycleGAN lacks guaranteed temporal smoothness, has an image-resolution limit, and struggles to generate rain across datasets because it does not disentangle rain from scene appearance.The authors characterize this as strong domain dependence and mention possible super-resolution or large-scale GAN alternatives.
  • Future work: Finetuning on synthetic rain improves performance on real rainy images, but the gains remain modest and further work is needed for truly robust rainy-condition algorithms.The authors frame this as a limitation and future research direction.

A Field of view of a drop in a sphere

The appendix estimates a moving raindrop’s field of view by approximating its exposure-time position, constructing a viewing cone, and projecting that cone onto a constant-radius environment sphere.

  • Drop geometry: The drop’s exposure-time field of view is approximated at the midpoint X = (X0 + X1)/2 of its shutter-opening and shutter-closing positions.The constant-FOV assumption is considered acceptable for the short 2ms KITTI and 5ms Cityscapes exposures.
  • Viewing cone: A plane orthogonal to the viewing direction is constructed through the midpoint, then a vector is rotated by θ/2 to form the drop’s viewing cone.The cone uses θ ≈ 165°, with θ/2 because the cone is symmetric around the viewing direction.
  • Environment-map projection: The cone is sampled with 20 radially equidistant vectors, intersected with a 10m camera-centered sphere, and converted to spherical coordinates for environment-map projection.The resulting discrete contour is polygon-filled to obtain the drop FOV used for rain-streak photometry.
  • Photometric use: The projected drop FOV is used to compute each rain streak’s photometry, including its radiance and chromaticity.The construction assumes a constant scene distance and approximates location-dependent environment maps as negligible for camera-near drops.

B Compositing a rain streak with different exposure time

The method adjusts rain-streak appearance for differing exposure times before alpha-compositing the rain layer with the background image.

  • Exposure-time differences must be accounted for when using a streak appearance database to add rain to existing images.The original exposure time cannot be exactly recomputed from the available calibration data.
  • The drop diameter and simulation settings are used to compute the reference residence time τ0.
  • The simulator provides each drop’s pixel residence time τ1, while τ is treated as constant across pixels within a streak.The constant-τ assumption is supported by the negligible changes across pixels for a given streak.
  • After normalization, each streak’s alpha is scaled using τ1 and the target exposure time T before compositing.
  • The final rainy image is formed by alpha blending the background image Ibg with the rain layer Ir.

C Experiments data splits

Table 3 documents the data splits used across the paper’s experimental steps.

  • Table 3 contains the detailed data splits for the paper’s various experimental steps.
  • The split information covers multiple experimental contexts and datasets.
  • The table provides split minutiae rather than reporting experimental outcomes.
Loading 2009.03683v1…