PaperlayerView exact revision on arXiv

Source-linked AI summary

RainyGS: Efficient Rain Synthesis with Physically-Based Gaussian Splatting

Qiyu Dai, Xingyu Ni, Qianfan Shen, Wenzheng Chen, Baoquan Chen, Mengyu Chu

arXiv:2503.21442v2cs.GRcs.CV

TL;DR

RainyGS addresses the challenge of adding physically accurate, dynamic rain to in-the-wild scenes, where scene modeling and physics-based simulation each have limitations. It combines physics-based rain models with 3D Gaussian Splatting, reporting controllable synthesis above 30 fps and advantages over 2D editing and video-generation baselines. The authors conclude that it supports interactive rain generation in complex, open-world environments.

  • Problem

    Adding physically accurate, dynamic rain to in-the-wild scenes is challenging because streaks, water accumulation, reflections, and refractions must evolve together and be simulated efficiently.

    Source: §1 · pp. 1–2

  • Method

    RainyGS reconstructs scene appearance and geometry from multi-view images, then integrates physics-based rain effects with 3D Gaussian Splatting rendering.

    Source: §1 · p. 2

  • Results

    RainyGS renders at over 30 fps with user-controlled intensity and reports better simulation performance, fidelity preservation, and control than 2D editing and video-generation baselines across outdoor and driving scenes.

    Source: §1 · p. 2, §4.3 · p. 8

  • Takeaways & Limitations

    The authors conclude that RainyGS supports interactive generation of realistic rain phenomena in complex, in-the-wild environments.

    Source: §5 · p. 8

    Shallow-water dynamics are less accurate for deep waves and unsuitable for multi-depth water effects, while image-space rendering cannot reflect or refract occluded objects.

    Source: §5 · p. 8

Abstract

from arXiv · show

We consider the problem of adding dynamic rain effects to in-the-wild scenes in a physically-correct manner. Recent advances in scene modeling have made significant progress, with NeRF and 3DGS techniques emerging as powerful tools for reconstructing complex scenes. However, while effective for novel view synthesis, these methods typically struggle with challenging scene editing tasks, such as physics-based rain simulation. In contrast, traditional physics-based simulations can generate realistic rain effects, such as raindrops and splashes, but they often rely on skilled artists to carefully set up high-fidelity scenes. This process lacks flexibility and scalability, limiting its applicability to broader, open-world environments. In this work, we introduce RainyGS, a novel approach that leverages the strengths of both physics-based modeling and 3DGS to generate photorealistic, dynamic rain effects in open-world scenes with physical accuracy. At the core of our method is the integration of physically-based raindrop and shallow water simulation techniques within the fast 3DGS rendering framework, enabling realistic and efficient simulations of raindrop behavior, splashes, and reflections. Our method supports synthesizing rain effects at over 30 fps, offering users flexible control over rain intensity -- from light drizzles to heavy downpours. We demonstrate that RainyGS performs effectively for both real-world outdoor scenes and large-scale driving scenarios, delivering more photorealistic and physically-accurate rain effects compared to state-of-the-art methods. Project page can be found at https://pku-vcl-geometry.github.io/RainyGS/

1. Introduction

RainyGS targets physically accurate, dynamic rain synthesis in open-world scenes, where concurrent streaks, water accumulation, reflections, and refractions make efficient simulation challenging. It combines physics-based rain modeling with 3DGS for controllable, photorealistic effects.

1. Introduction

  • Rain synthesis remains challenging because streaks, surface accumulation, and reflection and refraction effects must evolve together and be simulated efficiently.Scene-editing methods often lack physics-based rain simulation, while traditional physics-based approaches can require artists to manually set up high-fidelity scenes.Source: §1 · p. 2
  • RainyGS reconstructs scene appearance and geometry from multi-view images, then adds streaks, ground water, reflections, refractions, and specular rendering through physics-based models.The framework combines physics-based rain simulation with the efficiency of Gaussian Splatting rendering.Source: §1 · p. 2
  • Height-field shallow-water simulation models surface water dynamics efficiently, while screen-space ray tracing renders reflections of visible scene parts in the rasterization pipeline.The authors describe shallow-water assumptions as particularly suited to rain scenarios and more accurate than position-based dynamics.Source: §1 · p. 2
  • RainyGS renders at over 30 fps and offers intensity control from drizzle to downpour, with reported advantages over 2D editing and video-generation baselines across outdoor and driving scenes.The evaluation includes MipNeRF360, Tanks and Temples, and Waymo driving scenarios.Source: §1 · p. 2

2. Related Work

Related work spans scene representations, weather simulation, and methods that combine physical simulation with neural scene modeling. RainyGS builds on these lines to synthesize dynamic rain in Gaussian Splatting scenes.

3D Scene Modeling

  • NeRF and 3D Gaussian Splatting methods are mostly designed for novel view synthesis [4] [5] [6] [23] [30], while dynamic rainy scenes are more challenging than regular 4D scenes due to flying streaks and waving reflections.Source: §2 · p. 2

Weather Simulation

  • Graphics-based weather methods simulate effects such as capillary waves, smoke, and wind-driven snow, while image- and video-based methods can produce realistic single images but lack view-consistent climate simulation.RainyGS uses shallow-water simulation while retaining scene-modeling properties of 3D representations.Source: §2 · p. 3

Physics-Based Scene Editing

  • Prior physics-based scene-editing work combines neural scene models with physical simulation, including static weather and Gaussian-based solid or fluid effects; RainyGS applies this combination to dynamic rain.ClimateNeRF is described as limited to static weather conditions, while Gaussian Splashing combines solid and fluid animations with 3D Gaussian Splatting.Source: §2 · p. 3

3. Methods

RainyGS reconstructs scene geometry and appearance, then uses auxiliary maps and physics-based simulation to add dynamic water, reflections, refractions, and rain. Its methods combine height-map shallow-water simulation with rasterized water and raindrop effects.

3.1. Preliminary: 3D Gaussian Splatting

  • PGSR provides scene geometry and appearance, while 3DGS represents scenes with anisotropic Gaussian ellipsoids and uses a tile-based rasterizer for rendering.Vanilla 3DGS geometry is often compromised by free movement of Gaussian points, so the method adopts PGSR for high-precision geometry and high-fidelity rendering.Source: §3.1 · pp. 3–4

3.2. Auxiliary Map Extraction

  • A top-down orthographic render produces the height map for simulation, while appearance, depth, and normal maps support rain rendering.PGSR is used for geometry and appearance, with monocular normal priors supervising rendered normal maps; the authors note that PGSR sacrifices albedo, slightly reducing relighting effects.Source: §3.2 · p. 4

3.3. Rain Simulation on Height Maps

  • Shallow-water equations simulate accumulated rainwater on a 2D height map, bypassing hidden 3DGS interior geometry for efficiency.Raindrops disappear on ground contact and add their volume to the height field; drops between the ground and an occlusion height disappear without adding water.Source: §3.3 · pp. 4–5

3.4. Reflection-Aware Water Rasterization

  • Water rasterization uses screen-space ray tracing for visible-surface specular reflections, while highlights use a directional sun light and Blinn–Phong shading.The method also approximates shallow-water refraction by distorting the source image using screen-projected surface normals and water depth.Source: §3.4 · p. 6
  • Rain streaks are represented by connected Gaussian ellipsoids, and splashes by a Gaussian centered at impact, with scale determined by raindrop volume.Streak colors are adjusted according to normalized base-image brightness; the final rendering blends water effects and raindrops using screen-space depth information.Source: §3.4 · pp. 5–6

4. Experiments

RainyGS is evaluated against video-based and text-driven rain-synthesis methods, on in-the-wild datasets, and in applications including rainy-scene modeling and driving environments. The reported results emphasize realistic, physically accurate rain, intensity control, and responsive performance.

4. Experiments

  • The experiments assess high-fidelity results, user control, interactive performance, rendering ablations, computational efficiency, and potential downstream use in autonomous-driving training scenes.The evaluation overview also identifies comparisons with video-based rain synthesis and text-driven 3D editing methods.Source: §4 · p. 6

4.1. Evaluation Details

  • The evaluation compares RainyGS on Garden, Treehill, Bicycle, Family, and Truck scenes with Rain Motion, Runway Gen-3 Alpha, and Instruct-GS2GS.Rain Motion is described as generating temporally consistent rain streaks; Runway Gen-3 Alpha as a video-to-video generation model; and Instruct-GS2GS as a text-driven 3DGS editing method.Source: §4.1 · p. 6

4.2. Comparison with Rain Synthesis Baselines

  • RainyGS preserves scene geometry and produces physically accurate rain dynamics and realistic rendering, while the compared methods show shortcomings in consistency, rain realism, or dynamic editing.Across views, Runway-V2V fails to maintain 3D consistency, while Rain Motion and Instruct-GS2GS do not produce realistic rain streaks and puddles; across time, Rain Motion produces random droplets without hydrops and Runway-V2V is nearly static.Source: §4.2 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7

4.3. Qualitative Results

  • On MipNeRF360 and Tanks and Temples scenes, RainyGS provides high visual fidelity, physical accuracy, and real-time performance, with user-controlled drizzle, moderate rain, and downpour levels.Figure 6 presents the three intensity levels from top to bottom.Source: §4.3 · p. 7, Figure 6 · p. 8
  • RainyGS adds only a small time and memory cost over PGSR and runs at over 30 fps, while being significantly faster than the other rain-synthesis methods in Table 1.Table 1 compares the Garden scene with original PGSR, PGSR using 3D ray tracing for water rendering, and Runway-V2V; the text also reports reduced computation time compared with PGSR using ray tracing and video-based methods.Source: §4.3 · pp. 7–8, Table 1 · p. 8

4.4. Applications

  • RainyGS can synthesize rainy conditions in large-scale driving scenes and is presented as a way to create challenging rainy environments for autonomous-driving safety.The paper also describes rainy-scene modeling as an application, while reporting degraded 3DGS reconstruction quality on simulated rainy images, including poor backgrounds and erroneous rain lines.Source: Figure 8 · p. 8, Figure 7 · p. 8, §4.4 · p. 8

5. Conclusions

RainyGS integrates shallow-water dynamics and screen-space rendering into 3DGS for interactive rain synthesis in complex in-the-wild environments. Its limitations include deep-water interactions, image-space occlusion, and dependence on accurate scene geometry.

5. Conclusions

  • Shallow-water dynamics are less accurate for deep waves and unsuitable for complex, multi-depth water effects such as interactions with submerged surfaces.Source: §5 · p. 8
  • Image-space rendering cannot refract or reflect occluded objects, while limited input views can produce imperfect geometry and water patches.The paper notes that adding more water can cover ground irregularities and still yield realistic rainy effects.Source: §5 · p. 8
  • RainyGS integrates shallow-water dynamics and screen-space rendering into 3DGS to generate realistic rain interactively in complex, in-the-wild environments.The conclusion says users need not worry about scene geometry, lighting conditions, or other physical parameters.Source: §5 · p. 8

RainyGS: Efficient Rain Synthesis with Physically-Based Gaussian Splatting (Supplementary Material)

The supplementary material details scene modeling, auxiliary-map extraction, height-map rain simulation, screen-space reflection, and experimental analyses. Its normal-prior subsection describes how monocular normal estimates improve PGSR and downstream maps.

RainyGS: Efficient Rain Synthesis with Physically-Based Gaussian Splatting (Supplementary Material)

A.1 Normal Prior Supervision

  • PGSR can show artifacts in texture-less regions and transparent or reflective areas such as floors, walls, and car windows.Source: §A.1 · p. 11
  • Normal priors from a pretrained monocular model improve PGSR’s rendering fidelity and geometric accuracy, yielding higher-quality depth, normal, and height maps for rain simulation.Source: §A.1 · p. 11

A.2 Comparison of Scene Modeling Methods

The scene-modeling comparison favors PGSR for quality and efficiency, while the supplementary sections describe auxiliary-map representations and the staged height-map simulation.

A.2 Comparison of Scene Modeling Methods

  • PGSR provides higher rendering quality than GShader+GOF with comparable lighting effects, while using one efficient model rather than two separately trained models.The comparison also notes that PGSR sacrifices PBR properties, though environment maps can still be replaced during Water Rasterization.Source: §A.2 · p. 11, Figure 10 · p. 12

B DETAILS OF AUXILIARY MAP PREPARATION

C DETAILS OF RAIN SIMULATION ON HEIGHT MAPS

D DETAILS OF SCREEN-SPACE REFLECTION

The reflection section describes screen-space reflection as a lower-cost alternative to ray tracing and outlines its ray-marching pipeline. It reports comparable reflection quality with substantial speed and memory gains.

D DETAILS OF SCREEN-SPACE REFLECTION

  • Screen-space reflection approximates reflective surfaces using screen-space depth and color, limiting computation to the visible scene instead of full ray tracing.Source: D DETAILS OF SCREEN-SPACE REFLECTION · p. 13
  • The SSR pipeline casts rays using surface normals and view directions, marches projected rays with DDA, recovers camera-space points, and detects collisions using a depth-thickness test.On collision it fetches the pixel color; when the ray exits screen space, it falls back to the environment map.Source: D DETAILS OF SCREEN-SPACE REFLECTION · pp. 13–14
  • 60× speedup and 1.65× reduction in peak memory usage accompany reflection quality comparable to ray tracing.The comparison attributes the gains to SSR’s limited screen-space pixel stepping instead of ray tracing’s BVH traversal and Gaussian reordering.Source: Figure 11 · p. 14, D DETAILS OF SCREEN-SPACE REFLECTION · p. 14
  • Across viewpoints at one timestep, RainyGS maintains 3D consistency and realistic streaks and water accumulation; across timesteps, it produces evolving hydrops and puddles.The cited comparisons contrast failures in Runway-V2V, Rain Motion, and Instruct-GS2GS.Source: Figure 12 · p. 15, Figure 13 · p. 15

E.1 More Results of Comparison with Video-Based Rain Synthesis

RainyGS is reported to outperform video-based and 3DGS editing baselines in realistic, physically plausible rain while maintaining multi-view consistency and coherent dynamics. A user study also reports consistent preferences for RainyGS in image and video comparisons.

E.1 More Results of Comparison with Video-Based Rain Synthesis

  • The evaluation covers Garden, Treehill, and Bicycle from MipNeRF360, Family and Truck from Tanks and Temples, and three video-based or 3DGS-editing baselines.The baselines are Rain Motion, Runway-V2V, and Instruct-GS2GS.Source: §E.1 · pp. 15, 19
  • Across Treehill, Truck, and Family, RainyGS preserves 3D consistency and renders convincing rain streaks and water accumulation from multiple viewpoints.The corresponding comparisons show RainyGS across viewpoints at the same timestep.Source: Figure 14 · p. 16, Figure 16 · p. 17, Figure 18 · p. 18
  • Across Treehill, Truck, and Family, RainyGS produces dynamic hydrops and puddles over time, unlike the random droplets or static and implausible effects of baselines.The comparisons show results from the same viewpoint at different timesteps.Source: Figure 15 · p. 16, Figure 17 · p. 17, Figure 19 · p. 18
  • Runway-V2V alters scene appearance and geometry and produces oversaturated colors or physically unrealistic ripples and reflections.Rain Motion lacks 3D multiview consistency, while Instruct-GS2GS lacks advanced rain phenomena and dynamic edits.Source: §E.1 · p. 19
  • RainyGS consistently outperforms all baselines in realistic, physically plausible rain, with multi-view consistency and temporally coherent dynamics.Reported effects include fluid dynamics, accurate reflections, and refractions.Source: §E.1 · p. 19

E.2 User Study

  • Users consistently preferred RainyGS over competing methods in both image and video pairwise comparisons.The study involved 54 participants judging 30 randomly selected examples; Table 2 reports preference percentages.Source: Table 2 · p. 20, §E.2 · p. 20

E.3 Additional Results of Precise User Control

RainyGS supports precise, physics-based control of rain intensity, water level, and scene brightness. The examples show these parameters varying over time from the same scene and camera viewpoint.

E.3 Additional Results of Precise User Control

  • RainyGS provides precise, physics-based control over rain intensity, water level, and scene brightness.These controls allow adjustment for different simulation needs.Source: §E.3 · p. 20
  • The Garden progression illustrates dynamic changes in rain intensity, gradual water accumulation, and changing lighting conditions.The sequence is rendered consistently from the same scene and camera viewpoint.Source: §E.3 · p. 20
  • The progression keeps the scene and camera viewpoint fixed while showing changes over time.Source: §E.3 · p. 20

E.4 More Results and Analysis of Performance

Performance statistics report rendering at around 30 FPS, with simulation and data transfer accounting for the most time. Memory use is described as roughly linear in the number of Gaussians.

E.4 More Results and Analysis of Performance

  • Rendering runs at around 30 FPS on an Nvidia GeForce RTX 3090 with 24 GB of memory.The reported statistics cover results from the main text and Figures 12–19.Source: §E.4 · p. 20
  • Simulation and data transfer take the most time, but can be precomputed.Simulation time includes transferring data from height maps to Gaussians.Source: §E.4 · p. 20
  • Memory usage is roughly linear in the number of Gaussians and is described as fitting personal-computer capacity.Source: §E.4 · p. 20

E.5 Additional Results of Downstream Tasks

RainyGS transforms continuous autonomous-driving scenes, including Waymo scenes, into rainy conditions. The resulting scenes are proposed for perception robustness testing and adverse-weather data generation.

E.5 Additional Results of Downstream Tasks

  • RainyGS efficiently transforms continuous driving scenarios, such as Waymo scenes, into rainy weather conditions.Figure 21 presents rain synthesis for Waymo scenes.Source: §E.5 · p. 20, Figure 21 · p. 21
  • The rainy driving scenes can facilitate robustness testing of detection algorithms.Source: §E.5 · p. 20
  • The method offers a way to simulate adverse weather and generate synthetic data for autonomous-driving research.Source: §E.5 · p. 20, Figure 21 · p. 21

Related papers

Loading 2503.21442v2…