PaperlayerView exact revision on arXiv

Source-linked AI summary

BRDFusion: Physics Meets Generation for Urban Scene Inverse Rendering

Yi-Ruei Liu, Jie-Ying Lee, Zheng-Hui Huang, Yu-Lun Liu, Chih-Hao Lin

arXiv:2606.17049v1cs.CV

TL;DR

Inverse rendering of sparse urban-scene videos must recover photorealistic, controllable scene properties despite ill-posed decomposition. BRDFusion combines physically based and generative models for inverse and forward rendering, achieving state-of-the-art performance while supporting relighting, night simulation, and dynamic object editing.

  • Problem

    Recovering photorealistic, controllable urban-scene edits from sparse videos requires decomposing geometry, materials, and lighting despite an ill-posed inverse-rendering problem.

  • Method

    BRDFusion jointly uses physically based models for explicit, consistent scene reconstruction and controllable rendering, with generative models regularizing decomposition and refining rendering artifacts.

  • Results

    BRDFusion achieves state-of-the-art performance on real and synthetic scenes across downstream tasks, including novel-view relighting, night simulation, and dynamic object editing.

  • Takeaways & Limitations

    The hybrid pipeline combines PBR controllability with generative photorealism for urban-scene relighting, nighttime simulation, and object insertion.

  • Takeaways & Limitations

    The method does not explicitly model emissive materials, making decomposition of nighttime sequences with active headlights or streetlights challenging.

Abstract

from arXiv · show

Inverse rendering of urban scenes from captured videos enables numerous applications, including content creation and autonomous driving simulation. Physically-based rendering methods follow and control lighting physics, but suffer from reconstruction and rendering artifacts. While generative models produce realistic videos, they offer limited consistency and controllability. We present BRDFusion, a unified framework that combines two complementary models for inverse and forward rendering. Specifically, BRDFusion recovers explicit, consistent scene properties with physical modeling and alleviates optimization ambiguity with generative priors. During forward rendering, the physical model provides controllable rendering from the scene configuration, and the generative model denoises and fixes artifacts. Therefore, our method produces high-quality videos while allowing precise control, outperforming baselines in real and synthetic scenes. Moreover, BRDFusion supports novel-view relighting, night simulation, and dynamic object insertion/editing. Project page: https://shigon255.github.io/brdfusion-page/

1 Introduction

BRDFusion addresses the ill-posed problem of reconstructing and editing dynamic urban scenes by combining physically based models for controllability and consistency with generative models for photorealism. It resolves ambiguities in scene decomposition and supports controllable novel-view relighting, night simulation with local lights, and dynamic object editing.

  • Motivation: Reconstructing large-scale urban environments from casual driving videos requires photorealism and controllability for applications including autonomous driving simulation, AR/VR, and content creation.The task requires decomposing captured scenes into underlying 3D geometry and material properties.
  • Limitations: Existing approaches struggle to achieve physical accuracy and visual realism simultaneously, while failing to produce consistent long sequences, precisely control local illumination, or simulate diverse scenarios such as night.The cited local-illumination example is vehicle-headlight intensity and direction.
  • BRDFusion: BRDFusion unifies physical and generative models in forward and inverse rendering to bridge physical accuracy and generative photorealism.It represents lighting with an HDR environment map, geometry and material with 3D Gaussians, and produces 2D G-buffers through efficient volume rendering.
  • Optimization: Its multi-stage optimization resolves ambiguities in geometry, material, and HDR lighting, while generative regularization supports decomposition and physical modeling maintains 3D consistency and plausibility.The framework combines physically based controllability with generative photorealism.
  • Results: BRDFusion achieves state-of-the-art performance on real and synthetic scenes and enables photorealistic, controllable novel-view relighting, night simulation with local lights, and dynamic object editing.These are identified as challenging downstream tasks evaluated across several real and synthetic scenes.

2 Related Works

Prior work addresses urban inverse rendering through differentiable physical models and increasingly through generative priors, while hybrid methods combine physical rendering with diffusion. BRDFusion differs by modeling dynamic urban objects with a scene graph and initializing denoising from physically based renders for controllable, consistent outputs.

  • Physically-Based Inverse Rendering in Urban Scenes: Differentiable NeRF- and mesh-based rendering enables joint optimization of geometry, materials, and lighting in urban scenes.Outdoor methods progress from single-image and learning-based decomposition to multi-view relighting, neural fields, and NeRF frameworks for complex illumination.
  • Physically-Based Inverse Rendering in Urban Scenes: Autonomous-driving methods address outdoor lighting with sky domes, SDF shadows, single-camera setups, LiDAR, and lighting-aware simulation, while 3DGS methods model static outdoor scenes.BRDFusion instead handles dynamic urban objects with a scene graph and complements physically based rendering with generative priors.
  • Generative and Diffusion Priors for Inverse Rendering: Diffusion models increasingly provide priors for geometry, materials, and lighting when single-illumination decomposition is underdetermined.Coupling these priors to optimization is critical, although SDS-based distillation is described as prone to over-s... in the supplied passage.
  • Hybrid PBR and Generative Rendering: Pure PBR methods suffer from Monte Carlo noise and artifacts, whereas purely generative methods lack the 3D grounding needed for consistent relighting.Hybrid approaches embed diffusion posteriors into path-tracing loops or relight input views before reconstruction.
  • Hybrid PBR and Generative Rendering: Initializing denoising trajectories with PBR renders makes generative outputs respect explicit lighting changes and remain consistent across viewpoints and time.The passage identifies this hybrid design as BRDFusion’s central contribution.

3 Method

BRDFusion unifies explicit 3D Gaussian scene modeling, physically based rendering, and generative denoising for controllable, relightable urban-scene rendering. Its forward pipeline rasterizes scene attributes, simulates light transport, and refines resulting artifacts while preserving PBR structure and lighting.

  • 3.1 Relightable Scene Representation: The scene is represented by 3D Gaussians encoding geometry, appearance, and materials, enabling ray-traced shadows, multi-view consistency, and controllable relighting.Movable entities are organized in a scene graph with rigid transformations, alongside a 512 × 1024 × 3 HDR environment map.
  • 3.2 Forward Rendering Pipeline: The forward renderer uses three passes: volume rendering for scene attributes, physically based rendering for light transport, and generative refinement for residual noise.The volume pass is fast but cannot model global-lighting effects such as shadows or reflections.
  • 3.2 Forward Rendering Pipeline: PBR estimates incident radiance with environment-map importance sampling, Gaussian ray-traced visibility, and a physically based Cook-Torrance BRDF.The method approximates the rendering-equation integral with Monte Carlo integration and blends emission and reflectance using opacity.
  • 3.2 Forward Rendering Pipeline: A pretrained video diffusion model denoises partially noised PBR videos, improving visual quality while preserving structure, lighting, and temporal coherence.Starting from a partially noised PBR result avoids hallucinated shading inconsistent with the physically based rendering.
  • 3.2 Forward Rendering Pipeline: The resulting generated rendering is both photorealistic and faithful to PBR lighting, supporting high-quality, accurate relighting.The generative step addresses Monte Carlo noise and reconstruction artifacts in the PBR output.

4 Experiments

BRDFusion is evaluated on real-world Waymo scenes and synthetic urban scenes for inverse rendering, novel-view synthesis, and relighting. Across qualitative comparisons, quantitative evaluations, and ablations, it produces physically plausible, consistent scene reconstructions and high-quality controllable renderings.

  • Datasets: The evaluation uses Waymo scenes with 50 consecutive single-camera frames per scene and six synthetic urban scenes rendered under diverse environment maps.Waymo frames reserve every 10th frame for testing; synthetic scenes provide separate training and testing trajectories and three relighting environments.
  • Baselines: BRDFusion is compared with physically based UrbanIR and InvRGB+L, and with the generative Gen3C+DR baseline.UrbanIR uses neural-field geometry and color with sun-sky lighting, while InvRGB+L uses 3D Gaussian attributes and LiDAR reflectance for material estimation.
  • Forward Rendering: On real-world Waymo scenes, BRDFusion produces realistic shading, high-quality rendering, and precise lighting control despite dynamic objects and local-light conditions.UrbanIR is limited to static scenes, while InvRGB+L suffers reconstruction artifacts and shadow baking; BRDFusion also supports sunset, starry-night, streetlight, and headlight conditions.
  • Inverse Rendering: BRDFusion estimates detailed, physically plausible, view- and temporally consistent geometry and materials, whereas competing methods exhibit reconstruction, material, or consistency failures.UrbanIR fails on dynamic scenes, InvRGB+L predicts incorrect materials and overly smooth scene buffers, and Gen3C+DR is detailed but physically inconsistent because of randomness.
  • Quantitative Evaluation: On synthetic data, BRDFusion achieves the highest accuracy for roughness, metallic, and normal estimation, with albedo comparable to Gen3C+DR and novel-view synthesis comparable to the baseline.Inverse-rendering metrics exclude sky regions because sky material definitions are ambiguous.
  • Ablation Study: Ablations show that PBR optimization is essential for disentangling materials, geometry, and lighting, while generative rendering reduces artifacts and improves final output quality.Removing PBR optimization causes inconsistent generative estimates, and bypassing generative rendering leaves Monte Carlo integration and reconstruction artifacts insufficiently corrected.

5 Conclusion

BRDFusion combines physically based inverse rendering with generative refinement to unite controllability and photorealism for dynamic urban scenes. It supports novel-view relighting, object insertion, and nighttime simulation, but remains limited by emissive-material modeling, floaters, and catastrophic generative failures.

  • BRDFusion integrates physically based inverse rendering with generative refinement for inverse rendering in dynamic urban scenes.
  • The hybrid pipeline bridges PBR controllability with generative-model photorealism.
  • The framework enables novel-view relighting, object insertion, and nighttime simulation.
  • Limitations: Emissive materials are not explicitly modeled, making nighttime decomposition with active headlights or streetlights challenging.The method supports inserting virtual local lights but does not explicitly represent emissive materials.
  • Limitations: Floaters may appear in unobserved regions, and catastrophic generative-model failures can still produce incorrect scene decompositions.Refinement mitigates minor inconsistencies but cannot prevent catastrophic failures.

Supplementary Material

The supplementary material expands the evaluation and methodology with synthetic-data results, component ablations, implementation details, baseline adaptations, and a Cook-Torrance BRDF derivation.

  • Supplementary Material: The supplement presents qualitative evaluations on the synthetic dataset, qualitative ablations of each proposed component, implementation details and baseline adaptations, and the Cook-Torrance BRDF derivation used in rendering.These materials are organized in Sections A–D, respectively.

A Additional Qualitative Results

BRDFusion qualitatively outperforms the baselines on a synthetic scene by avoiding severe reconstruction, novel-view, and novel-lighting artifacts.

  • Qualitative comparison: UrbanIR and InvRGB+L fail to reconstruct detailed geometry and materials, causing severe artifacts under different viewpoints or illumination.The comparison is shown in Fig. 8 on a synthetic-dataset scene.
  • Qualitative comparison: Gen3C + DR introduces artifacts in novel views and predicts inaccurate shading under novel lighting.These failures are visible in the bottom row of Fig. 8.

B Additional Ablation Studies

The ablations show that physically based optimization and generative priors are both necessary for artifact-free, consistent inverse rendering. Generative refinement further improves temporal consistency and reconstruction accuracy.

  • Removing physically based inverse rendering causes artifacts in novel-view synthesis and relighting, demonstrating the necessity of physically based optimization.
  • Removing the generative prior prevents proper decomposition of scene geometry, material, and lighting because inverse rendering is ill-posed.
  • DiffusionRenderer’s material prior is temporally inconsistent, with window metallic values changing across timesteps and producing noticeable artifacts.
  • Generative refinement makes the material prior temporally consistent and cleaner, leading to a more accurate reconstructed 3DGS.

C Implementation Details

The implementation uses staged optimization losses with empirically chosen weights, followed by joint refinement combining volume-rendering and physically based objectives. It also standardizes baseline lighting representations and supports direct rendering from localized point and spot lights.

  • Optimization: Volume rendering minimizes RGB, opacity, depth, normal, albedo, roughness, and mask losses with weighted terms.The objective is loss_vol = loss_rgb + λOloss_O + λDloss_D + λNloss_N + λAloss_A + λRloss_R + λMloss_M.
  • Optimization: The volume-rendering weights are λO = 0.05, λD = 0.01, λN = 0.3, λA = 0.5, and λR = λM = 0.3.These weights are empirically set for the Volume Rendering stage.
  • Optimization: Physically based inverse rendering minimizes color reconstruction error plus an environment-light loss, with λE initialized at 1.0 and decayed linearly to 0.1.The environment-light loss compares log environment radiance against ground truth over directions ω.
  • Optimization: Joint refinement combines the volume and physically based losses using λpbr = 0.1, while adjusting volume weights to λN = 0.3, λA = 0.4, and λR = λM = 0.1.The combined objective is loss_all = loss_vol + λpbrloss_pbr.
  • Light Representation Conversion for Baselines: For fair relighting comparisons, HDR environment maps are converted into UrbanIR’s sun-sky parameters and InvRGB+L’s spherical-harmonics and sun representation.The conversion detects the maximum-intensity pixel for sun direction and intensity, computes ambient intensity excluding the sun region, and directly samples HDR maps for sky rendering.
  • Local Light Rendering: Localized illumination adds direct point- and spot-light contributions using deterministic shadow-ray visibility, normalized light directions, distance-attenuated incident radiance, and BRDF evaluation.The rendering equation sums environment-light and local-light contributions at each surface point.

D Cook-Torrance BRDF derivation

The rendering pipeline uses a standard Cook-Torrance BRDF that combines Lambertian diffuse and microfacet specular responses. Its material response is parameterized by albedo, metallicity, surface normal, half-vector, and standard GGX, Smith-Schlick, and Fresnel-Schlick approximations.

  • BRDF formulation: The surface response combines a Lambertian diffuse term with a Cook-Torrance microfacet specular term.The BRDF is expressed as a linear combination of diffuse and specular components.
  • BRDF formulation: The formulation defines A as albedo, M ∈[0, 1] as metallic, N as the surface normal, and H as the normalized sum of incident and outgoing directions.The half-vector is H = (ωi+ωo) / ∥ωi+ωo∥.
  • Specular approximations: The normal distribution uses the GGX model, parameterized by α = R^2, where R denotes surface roughness.The GGX distribution depends on the squared roughness parameter and the normal-half-vector alignment.
  • Specular approximations: The geometry term uses Schlick’s approximation to the Smith model, factorized as G(ωi, ωo) = G1(ωo)G1(ωi) with k = (R + 1)^2/8.The single-direction term is defined using the normal-direction cosine and k.
  • Specular approximations: The Fresnel term follows Schlick’s approximation, F(ωo, H) = F0 + (1 - F0)(1 - ωo · H)^5, with F0 interpolated between 0.04 and albedo A.The baseline dielectric reflectance is 0.04.

E Computation Efficiency

BRDFusion’s efficiency is evaluated against physically based baselines using amortized chunk-level rendering speed and measured generative-processing times. Its runtime is comparable to the baselines, though InvRGB+L is evaluated on a different GPU, limiting direct comparison.

  • Evaluation setup: Efficiency is compared with UrbanIR and InvRGB+L using a single NVIDIA RTX A6000 at 960 × 640 resolution.Rendering speed is reported as FPS amortized over a 57-frame chunk.
  • Runtime comparison: BRDFusion achieves runtime comparable to the physically based baselines.The reported efficiency may differ from the InvRGB+L paper because the baselines were run on different machines.

F Failure Case

BRDFusion shares a limitation with reconstruction-based pipelines: limited training views make unobserved geometry difficult to resolve. Unconstrained regions can then create artifacts and inaccurate relighting shadows, including unexpected ground shadows from floaters in invisible regions.

  • Failure Case: Limited training views make unobserved geometry difficult to resolve, leaving some scene regions unconstrained.The paper identifies this as a limitation common to many reconstruction-based pipelines.
  • Failure Case: Unconstrained regions may produce artifacts and cast inaccurate shadows during relighting.These effects are illustrated in Fig. 11.
  • Failure Case: Floaters in invisible regions can cast unexpected shadows on the ground when relighting.This is the specific failure shown in Fig. 11.
Loading 2606.17049v1…