PaperlayerView exact revision on arXiv

Source-linked AI summary

EnvGS: Modeling View-Dependent Appearance with Environment Gaussian

Tao Xie, Xi Chen, Zhen Xu, Yiman Xie, Yudong Jin, Yujun Shen, Sida Peng, Hujun Bao, Xiaowei Zhou

arXiv:2412.15215v2cs.CV

TL;DR

Complex reflections remain difficult to reconstruct because environment maps miss near-field effects and high-frequency details. EnvGS adds explicit environment Gaussians to base Gaussians and renders them with a GPU-accelerated differentiable ray tracer. Experiments show superior reflection quality and state-of-the-art real-time novel view synthesis across real and synthetic datasets.

  • Problem

    Environment-map methods struggle with high-frequency reflection details and cannot accurately model near-field reflections because they assume distant lighting.

    Source: §1 · p. 2

  • Method

    EnvGS represents reflections with environment Gaussian primitives combined with base Gaussian primitives and uses a differentiable GPU ray-tracing renderer for joint optimization.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    EnvGS achieves state-of-the-art real-time novel view synthesis and considerably surpasses existing real-time methods, especially for complex reflections in real-world scenes.

    Source: §1 · p. 2, §5.3 · p. 7

  • Takeaways & Limitations

    Explicit environment Gaussians provide detailed modeling of near-field and high-frequency reflections while maintaining real-time rendering performance.

    Source: §1 · p. 2, §6 · p. 8

    EnvGS has difficulty with transparent and refractive materials because it addresses only reflection direction.

    Source: §6 · p. 8

Abstract

from arXiv · show

Reconstructing complex reflections in real-world scenes from 2D images is essential for achieving photorealistic novel view synthesis. Existing methods that utilize environment maps to model reflections from distant lighting often struggle with high-frequency reflection details and fail to account for near-field reflections. In this work, we introduce EnvGS, a novel approach that employs a set of Gaussian primitives as an explicit 3D representation for capturing reflections of environments. These environment Gaussian primitives are incorporated with base Gaussian primitives to model the appearance of the whole scene. To efficiently render these environment Gaussian primitives, we developed a ray-tracing-based renderer that leverages the GPU's RT core for fast rendering. This allows us to jointly optimize our model for high-quality reconstruction while maintaining real-time rendering speeds. Results from multiple real-world and synthetic datasets demonstrate that our method produces significantly more detailed reflections, achieving the best rendering quality in real-time novel view synthesis. The code is available at https://zju3dv.github.io/envgs.

1. Introduction

EnvGS addresses the difficulty of reconstructing high-frequency and near-field reflections by representing environmental reflections with Gaussian primitives and rendering them through a differentiable ray tracer. Across real and synthetic datasets, it achieves state-of-the-art real-time novel view synthesis and surpasses existing real-time methods.

Motivation

  • Environment maps struggle with near-field reflections because they assume distant lighting and lack sufficient capacity for high-frequency reflection details.These limitations remain even when environment maps are combined with shading functions and spherical harmonics.Source: §1 · p. 2

Approach

  • EnvGS represents scene geometry and base appearance with base Gaussians, while environment Gaussians explicitly capture reflection colors along reflected viewing directions.The renderer blends the base color with the reflected environment color to produce the final image.Source: §1 · p. 2
  • A differentiable CUDA and OptiX ray-tracing renderer builds a BVH, gathers ordered Gaussian intersections in chunks, and integrates them through volume rendering for real-time optimization and rendering.This renderer enables joint optimization of the environment and base Gaussians.Source: §1 · p. 2

Results

  • EnvGS achieves state-of-the-art real-time novel view synthesis and considerably surpasses existing real-time methods, particularly for complex reflections in real-world scenes.The evaluation uses multiple real and synthetic datasets.Source: §1 · p. 2

Contributions

  • The method is presented as a real-time representation for photorealistic specular reflections that handles both near-field and high-frequency reflection details.The authors describe it as the first method to achieve real-time photorealistic specular-reflection synthesis in real-world scenes.Source: §1 · p. 2

2. Related Work

Prior work improves view-dependent rendering with NeRF- and Gaussian-based representations, but existing approaches trade reflection fidelity, lighting scope, scene boundedness, or real-time speed. EnvGS uses environment Gaussians with a Gaussian tracer to support detailed near-field, high-frequency, and distant reflections in unbounded scenes.

NeRF-based Methods

  • NeRF-based methods improve reflection quality with reflected directions, refined normals, or directional encodings, but NeRF-Casting requires multiple MLP queries per ray and is unsuitable for real-time rendering.NeRF-Casting handles near- and distant-lighting conditions but requires substantial training time.Source: §2 · p. 2

Gaussian Splatting

  • 3DGS-based methods achieve real-time rendering, yet spherical-harmonic view dependence often produces blurry reflections and environment maps miss near-field lighting and high-frequency details.Other extensions include bounded-scene tensorial factorization or neural rendering components.Source: §2 · p. 3

EnvGS

  • EnvGS adopts 2DGS for scene representation and models environmental illumination with environment Gaussians rendered by a Gaussian tracer.The resulting representation supports high-frequency, near-field, and distant lighting in unbounded scenes while maintaining real-time performance.Source: §2 · p. 3

3. Preliminary

2DGS provides an explicit, surface-aligned Gaussian representation with efficient rasterized rendering and well-defined normals. However, its spherical-harmonic appearance model cannot adequately represent strong view-dependent effects such as specular reflections.

2DGS Representation

  • 2DGS represents scenes with scaled 2D Gaussian primitives defined in local tangent planes and rendered through screen-space rasterization.Each primitive is parameterized by a transformation matrix H.Source: §3 · p. 3
  • The transformation matrix encodes each Gaussian’s center, two principal tangential vectors, and their scaling factors.The symbols p_k, (t_u, t_v), and (s_u, s_v) denote these quantities, respectively.Source: §3 · p. 3

Rendering

  • 2DGS determines ray-primitive intersections and integrates the resulting Gaussian contributions with volume rendering to obtain each pixel color.The Gaussian opacity contribution is formed from the Gaussian evaluation and accumulated transmittance.Source: §3 · p. 3

Advantages and Limitation

  • Compared with 3D Gaussians, 2D Gaussians avoid multi-view depth inconsistency and provide well-defined normals useful for surface reconstruction and reflection calculations.These surface properties motivate using geometry-aligned 2D Gaussians as EnvGS’s base representation.Source: §3 · p. 3
  • The limited capacity of spherical harmonics prevents 2DGS from modeling strong view-dependent effects such as specular reflections, producing poor rendering and foggy geometry.EnvGS addresses this limitation in the following section by adding a reflection representation.Source: §3 · p. 3

4. Method

EnvGS models scene appearance with separate base and environment Gaussian primitives, then combines them through reflection-aware rendering. Its differentiable ray tracer and joint optimization target detailed near- and far-field reflections while retaining real-time operation.

4.1. Reflective Scenes Modeling

  • EnvGS uses base Gaussians for geometry and appearance and environment Gaussians for reflections, addressing the limited expressiveness of spherical harmonics for complex view-dependent effects.The two representations are blended through a per-base-Gaussian weight.Source: §4.1 · pp. 3–4
  • Base Gaussian splatting estimates surface position, normals, base color, and blending weights; reflection rays query environment Gaussians for reflection color before final blending.The output color is c = (1 - β) · c_base + β · c_ref.Source: §4.1 · p. 4, Figure 2 · p. 4
  • Explicitly placing environment Gaussians in 3D improves near-field reflections from nearby occluders and provides finer detail than low-frequency environment maps.The representation avoids the distant-lighting assumption used by environment maps.Source: §4.1 · p. 4, Figure 3 · p. 5

4.2. Differentiable Ray Tracing

  • The ray tracer renders 2,000,000 2DGS at 1292x839 and 30 FPS on an RTX 4090, supporting real-time environment-Gaussian rendering.This design is motivated by the impracticality of rasterizing a distinct reflection ray for every pixel.Source: §4.2 · p. 4
  • EnvGS converts 2D Gaussians into triangle primitives, organizes them in a BVH, and traces reflection rays through ordered intersections using CUDA and OptiX.Chunked ray traversal maintains a sorted k-buffer, with k = 16 reported as the best trade-off between traversal counts and sorting cost.Source: §4.2 · pp. 4–5
  • The fully differentiable renderer recomputes rays during backpropagation to avoid storing all intersections, enabling joint optimization of base and environment Gaussians.The paper reports that joint geometry-and-appearance optimization is essential for recovering geometrically accurate surfaces.Source: §4.2 · p. 5

4.3. Optimization

  • Optimization first bootstraps base Gaussians from an SfM point cloud, then initializes environment Gaussians in scene sub-grids and jointly trains both sets.The scene is divided into 32^3 sub-grids with 5 randomly sampled environment primitives per grid.Source: §4.3 · p. 5
  • Training combines photometric, normal-consistency, monocular-normal, and perceptual losses to supervise appearance and geometry.The loss weights are λ1 = 0.04, λ2 = 0.01, and λ3 = 0.01, while the photometric loss mixes L1 and D-SSIM at 0.8 and 0.2.Source: §4.3 · pp. 5–6

5. Experiments

Experiments evaluate EnvGS across real-world and synthetic settings, compare it with prior methods, and ablate its key components. EnvGS achieves strong real-time reflection reconstruction, while joint optimization, monocular normals, and Gaussian environment modeling materially affect quality.

5.1. Implementation Details

  • Implementation uses custom OptiX kernels, Adam optimization, 2DGS-matched Gaussian learning rates, and a single NVIDIA RTX 4090 GPU.Three-dimensional spatial gradients replace projected 2D-center gradients for densification, with depth scaling to avoid distant-region under-densification.Source: §5.1 · pp. 6–7

5.3. Baseline Comparisons

  • EnvGS delivers the highest rendering quality among real-time techniques and is competitive with NeRF-Casting while being 100 times faster.Table 1 covers Ref-Real and NeRF-Casting Scenes; all metrics are evaluated at 1/4 resolution.Source: Table 1 · p. 7, §5.3 · p. 7

5.2. Datasets and Evaluation Metrics

  • The evaluation uses Ref-Real, NeRF-Casting Shiny Scenes, two self-captured scenes, and Shiny Blender with PSNR, SSIM, and LPIPS under consistent splits and resolutions.Additional comparisons include Mip-NeRF 360 and object-level methods in the supplementary appendix.Source: §5.2 · p. 7, §5.3 · p. 7

5.3. Baseline Comparisons

  • On nine real-world scenes, EnvGS outperforms all explicit methods and captures near-field reflections and high-frequency details more effectively.The scenes come from Ref-Real, NeRF-Casting Shiny Scenes, and self-captured real-world data.Source: §5.3 · p. 7, §5.2 · p. 7

5.4. Ablation Studies

  • Removing joint optimization or the monocular normal constraint produces inaccurate geometry and reflection reconstruction, while replacing environment Gaussians with an environment map misses near-field and high-frequency reflections.The environment-map variant captures smooth distant reflections but produces bumpier geometry.Source: §5.4 · p. 8, Figure 5 · p. 8, Table 4 · p. 8
  • Removing color sabotage or normal propagation reduces rendering quality, whereas removing LPIPS causes only marginal improvement loss.The ablations are reported quantitatively in Table 4.Source: §5.4 · p. 8, Table 4 · p. 8

6. Conclusion and Discussion

The conclusion presents EnvGS as a reflective scene representation that combines environment and base Gaussians with differentiable GPU ray tracing. It reports strong complex-reflection performance across datasets while identifying transparent and refractive materials as a limitation.

6. Conclusion and Discussion

  • EnvGS combines environment Gaussian primitives for reflections with base Gaussian primitives for geometry, base color, and blending weight.The representation models the appearance of the whole scene through both Gaussian sets.Source: §6 · p. 8
  • A differentiable Gaussian ray tracer using the GPU’s RT core enables effective optimization and rendering of environment Gaussians.This supports the method’s real-time rendering objective.Source: §6 · p. 8
  • EnvGS demonstrates superior complex-reflection performance across various datasets.Source: §6 · p. 8
  • EnvGS has difficulty with transparent and refractive materials because it addresses only reflection direction.The paper identifies extending the method to these materials as future work.Source: §6 · p. 8

Supplementary Material

The supplementary material expands the empirical analysis and documents the Gaussian tracer’s gradient computation. It also provides continuous rendered videos for closer inspection of reflection quality and rendering.

Supplementary Material

A.1. Comparison on Reflective Regions

EnvGS improves reflection reconstruction across reflective, near-field, real-world, synthetic, diffuse, and object-level comparison settings, with particular gains in complex reflection details.

A.1. Comparison on Reflective Regions

  • Over 1.0 PSNR improvement on reflective regions and 2.0 PSNR improvement on near-field regions versus environment maps.The comparison uses masked regions on Ref-Real and NeRF-Casting datasets.Source: §A.1 · p. 11

A.2. Comparison on Real-World Shiny Scenes

  • EnvGS significantly improves reflection fidelity and overall rendering quality on indoor and outdoor real-world shiny scenes, especially for near-field and high-frequency details.Source: §A.2 · p. 11

A.3. Comparison on Shiny Blender [37]

  • On Shiny Blender, EnvGS matches or surpasses environment-map methods for distant reflections and considerably improves near-field reflections caused by self-occlusions.The method also reconstructs more accurate geometry in this synthetic setting.Source: §A.3 · p. 11

A.4. Comparison on Mip-NeRF 360 [3]

  • On Mip-NeRF 360, EnvGS achieves comparable or superior performance to state-of-the-art implicit and explicit methods on primarily diffuse, complex-geometry scenes.Source: §A.4 · p. 11, Table 5 · p. 12

A.5. Additional Baselines

  • Object-level methods perform well on synthetic data but struggle on real-world scenes and cannot maintain real-time rendering with backgrounds.Source: §A.5 · p. 12, Figure 7 · p. 12

B.1. Environment Representation Comparison

  • Replacing environment Gaussians with 128- or 256-resolution environment maps fails to capture near-field reflections and blurs high-frequency reflection details.The environment Gaussian representation preserves these complex reflections with higher fidelity.Source: §B.1 · p. 12

B.2. Speed Analysis

The speed analysis attributes real-time rendering to hardware-accelerated Gaussian ray tracing, while separate environment Gaussians support stable reflection optimization and compact representation.

B.2. Speed Analysis

  • PyTorch ray tracing and 1x1-tile rasterization each take over an hour per frame, whereas the Gaussian tracer achieves real-time rendering using hardware-accelerated ray tracing.All compared strategies use the same volume-rendering equation.Source: §B.2 · p. 12, Table 6 · p. 12
  • The environment Gaussian and base Gaussian are rendered in separate rounds, and their outputs are blended into the final color.Source: §B.2 · p. 12
  • Tracing only rays with high blending weights accelerates rendering because strong specular reflections occupy a small portion of the scene surface.The reported ablation evaluates the quality impact of this acceleration technique.Source: §B.2 · p. 13

B.3. Environment Gaussian Design

  • Using one Gaussian set for both base appearance and reflections consistently fails to converge because geometry and erroneous reflection intersections interfere during optimization.Source: §B.3 · p. 13

C. Details of Environment Gaussian

  • The environment Gaussian is jointly optimized with the base Gaussian and comprises about 15% of the base Gaussian, averaging 300k primitives and 70MB after training.Pruning retains at most 630k environment Gaussian primitives by rendering weight.Source: C. Details of Environment Gaussian · pp. 13–14
  • On synthetic scenes, EnvGS reconstructs both distant specular reflections and near-field reflections caused by self-occlusions.Source: Figure 10 · p. 14

D. Details of Gradient Computation

The differentiable Gaussian tracer propagates reflection-rendering gradients through ray intersections, surface position, and normal to jointly optimize environment and base Gaussians.

D. Details of Gradient Computation

Related papers

Loading 2412.15215v2…