Source-linked AI summary
DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models
Ruofan Liang, Zan Gojcic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Zhi-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, Zian Wang
TL;DR
Real-world PBR is limited by the difficulty of obtaining precise geometry, materials, and lighting. DiffusionRenderer jointly learns inverse and forward rendering with video diffusion priors, outperforming state-of-the-art methods and supporting editing from a single video.
Problem
Real-world PBR requires precise geometry, material, and lighting representations that are challenging to acquire.
Method
DiffusionRenderer unifies neural inverse and forward rendering using two video diffusion models that reconstruct G-buffers and generate photorealistic videos from G-buffers and lighting.
Results
DiffusionRenderer consistently outperforms state-of-the-art methods while approximating inverse and forward rendering without explicit path tracing or precise 3D scene representations.
Takeaways & Limitations
From a single video input, the framework supports relighting, material editing, and realistic virtual object insertion across diverse scenes.
Takeaways & Limitations
Training requires paired ground-truth buffers for material, geometry, and lighting, which are typically available only in synthetic data.
Abstract
from arXiv · showhide
Understanding and modeling lighting effects are fundamental tasks in computer vision and graphics. Classic physically-based rendering (PBR) accurately simulates the light transport, but relies on precise scene representations--explicit 3D geometry, high-quality material properties, and lighting conditions--that are often impractical to obtain in real-world scenarios. Therefore, we introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering within a holistic framework. Leveraging powerful video diffusion model priors, the inverse rendering model accurately estimates G-buffers from real-world videos, providing an interface for image editing tasks, and training data for the rendering model. Conversely, our rendering model generates photorealistic images from G-buffers without explicit light transport simulation. Experiments demonstrate that DiffusionRenderer effectively approximates inverse and forwards rendering, consistently outperforming the state-of-the-art. Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion.
1. Introduction
The paper jointly addresses inverse and forward rendering with video diffusion priors, targeting practical photorealistic rendering despite incomplete scene representations. It trains inverse rendering on synthetic data and uses real-world pseudo-labels to support a forward renderer for editing applications.
- PBR quality depends heavily on accurate geometry, materials, and lighting, whose acquisition is challenging in real-world scenarios.
- The paper jointly considers physically based and inverse rendering rather than treating them as separate problems.The motivation draws on generative models that implicitly capture complex lighting effects.
- DIFFUSIONRENDERER uses video diffusion priors to synthesize shadows and reflections from geometry, material buffers, and environment-map lighting.It is designed to remain faithful to conditioning signals while handling imperfections in input data.
- The inverse renderer is trained on synthetic data, generalizes to real-world videos, and generates pseudo-labels used with synthetic data to train the forward renderer.
- DIFFUSIONRENDERER outperforms state-of-the-art methods while approximating inverse and forward rendering without explicit path tracing or 3D scene representation.
- From a single video input, the framework supports relighting, material editing, and virtual object insertion.
2. Related Work
Related work spans neural rendering, inverse rendering, and relighting, but existing approaches face data, representation, or editing constraints. DIFFUSIONRENDERER extends image-based diffusion approaches to video and uses editable G-buffers as an intermediate representation.
- Neural rendering replaces or extends traditional rendering pipelines with neural networks, including methods that approximate light transport or radiance caching.
- DIFFUSIONRENDERER extends RGB↔X-style image diffusion to video diffusion and targets neural relighting without requiring an irradiance estimate.
- Unlike volumetric approaches that typically bake radiance and have limited editing capabilities, the paper targets traditional, easy-to-edit G-buffers.
- Inverse rendering estimates intrinsic scene properties such as geometry, materials, and lighting from input images.
- Learning-based inverse-rendering algorithms are often data-hungry and task- or domain-specific, while sufficient diverse training data remains difficult to acquire.
- Recent image generative models improve reconstruction quality, but it remains insufficient for physically based rendering pipelines.
3. Preliminaries
The preliminaries define PBR as light-transport simulation and describe video diffusion models as iterative denoisers that can generate data-distribution samples under injected conditions.
- PBR simulates how incoming radiance contributes to outgoing radiance at a surface point.
- The rendering equation integrates the BRDF, incoming radiance, and a cosine factor over the hemisphere of incoming directions.The integral is evaluated using Monte Carlo methods.
- A diffusion model learns to approximate a data distribution through iterative denoising.
- Training constructs noisy latents as zτ = ατz0 + στϵ by adding Gaussian noise under a scheduled noise process.
- After training, repeatedly applying the denoising function to Gaussian noise produces samples from the learned data distribution.
- Video diffusion models inject conditions either by concatenating condition channels with image latents or through cross-attention layers.
4. Method
DiffusionRenderer unifies neural inverse and forward rendering with two video diffusion models. It estimates scene attributes from videos and renders photorealistic outputs from geometry, materials, and lighting conditions.
- The framework uses paired video diffusion models for inverse rendering and forward rendering.The inverse model reconstructs geometry and material buffers, while the forward model transforms G-buffers and lighting into photorealistic video.
- 4.1. Neural Forward Rendering: The forward renderer approximates light transport from geometry, material buffers, and environment-map lighting without explicit path tracing or classic 3D geometry.Environment-map information is encoded as lighting conditions for the diffusion model.
- 4.1. Neural Forward Rendering: Environment maps are encoded into multi-resolution features and injected through cross-attention layers at corresponding diffusion-UNet levels.This design preserves spatial details while conditioning generation on lighting information.
- 4.2. Neural Inverse Rendering: The inverse renderer estimates five scene attributes using dedicated passes selected by a domain embedding.The attributes are normals, depth, base color, roughness, and metallic properties.
- 4.4. Training pipeline: Training combines curated synthetic data with automatically labeled real-world videos because paired real geometry, materials, and lighting are impractical to capture.The inverse renderer generates G-buffer labels for real videos, which are then combined with synthetic data to train the forward renderer.
- 4.5. Editing Applications: The unified pipeline supports photorealistic editing through neural inverse rendering, G-buffer and lighting editing, and neural rendering.Supported applications include relighting, material editing, and virtual object insertion.
5. Experiments
Experiments evaluate forward rendering, inverse rendering, relighting, and editing applications across synthetic and real-world settings. DiffusionRenderer generally outperforms neural baselines, handles complex lighting effects, and supports realistic edits from G-buffers.
- Evaluation setup: Experiments cover forward rendering, inverse rendering, relighting, and applications including material editing and object insertion.Evaluations use synthetic and real-world datasets, with qualitative and quantitative comparisons against established baselines.
- Forward rendering: Our method exhibits a minor PSNR decrease of 2.3 dB from SyntheticObjects to SyntheticScenes, while other baselines show more substantial drops.Both classic PBR and neural methods perform well on single-object scenes but degrade with complex inter-reflections and occlusions.
- Forward rendering: Our method consistently outperforms all neural methods on both datasets and performs comparably to classic methods.In real-world editing applications, PBR techniques face limitations from missing 3D geometry and noisy G-buffers.
- Inverse rendering: The video model consistently enhances inverse rendering across all attributes, reducing RMSE by 41% for metallic and 20% for roughness versus the image model.The reported reductions are from 0.066 to 0.039 for metallic and from 0.098 to 0.078 for roughness.
- Relighting: Our method outperforms recent relighting baselines, particularly in scenes with complex shadows and inter-reflections.It produces higher-quality lighting effects with more accurate color and scale than DiLightNet and Neural Gaffer.
- Applications: Material edits and inserted objects blend naturally into scenes, generating realistic reflections and shadows.Examples adjust sphere roughness from 0.15 to 0.6, horse metallic from 0 to 1, and insert a bathtub and table.
6. Discussion
DiffusionRenderer is presented as a scalable, data-driven alternative to explicit path tracing and precise 3D scene representations. Its main practical limitations are offline inference, content variations during editing, and dependence on an off-the-shelf lighting estimator.
- Discussion: DiffusionRenderer achieves high-quality G-buffer estimation and photorealistic image generation without explicit path tracing or precise 3D scene representations.The approach is jointly trained on synthetic and auto-labeled real-world data and consistently outperforms state-of-the-art methods.
- Limitations and future work: Stable Video Diffusion operates offline, so inference speed would benefit from distillation techniques.This is identified as a limitation and future-work direction.
- Limitations and future work: Editing models preserve most original content but may introduce slight variations in color or texture.The paper suggests task-specific fine-tuning and neural intrinsic features to improve content consistency.
- Limitations and future work: Real-world auto-labeling currently uses an off-the-shelf lighting estimation model whose accuracy and robustness could be improved.Better lighting estimation is identified as a future-work opportunity.
Supplementary Material
The supplementary material provides additional implementation details and further results and analysis, with more qualitative comparisons available in the accompanying video.
- Supplementary material: The supplementary material contains additional implementation details and further results and analysis.These materials are organized in Sections A and B of the supplement.
- Supplementary material: The accompanying video provides additional qualitative results and comparisons.
A. Experimental Settings
The experiments train diffusion-based inverse and forward renderers with synthetic and real-world data, environment-light encodings, and G-buffer-based editing workflows. The setup evaluates relighting and object insertion while targeting photorealistic outputs and controlled lighting effects.
- Training setup: The inverse renderer maps RGB videos to intrinsic properties, while pseudo-labels from real-world videos support training the forward renderer.The inverse model is trained solely on synthetic data but generalizes robustly to real-world scenarios.
- Training setup: Both models are fine-tuned from Stable Video Diffusion and trained for 20,000 iterations at 512×512 resolution using mixed-precision training.Training takes around 2 days on 32 A100 GPUs.
- Rendering conditions: The forward renderer conditions on geometry, material buffers, and environment-map lighting to approximate path-traced shading without explicit light-transport simulation.Its environment-lighting condition is encoded through cross-attention, using LDR, log-space, and directional environment-map representations.
- Editing workflow: Object insertion blends background and object G-buffers using a foreground mask, then renders inserted and background images before applying a shading ratio to preserve the original background.Lighting is estimated with an off-the-shelf model, and the final edited image composites the masked foreground with the shading-adjusted background.
B. Additional Results
Additional experiments examine runtime, temporal consistency, perceptual preference, and comparisons with inverse-rendering baselines. The results support video-quality advantages and more photorealistic relighting without explicit 3D geometry constraints.
- Runtime: The inverse renderer takes 9.7 seconds and the forward renderer 20.3 seconds for 20 denoising steps on one A100 GPU.Both models use around 21 GB peak GPU memory for 24-frame 512×512 videos; the forward renderer is slower because of additional condition encoding.
- Temporal consistency: The method achieves the highest ColorVideoVDP score on both relighting test sets, consistent with visual inspections.The paper attributes improved temporal consistency relative to image-diffusion baselines to its use of video diffusion models.
- User study: Users selected the method over baselines in the reported relighting and neural-rendering comparisons on SyntheticScenes and SyntheticObjects.The user study compared each rendering with a reference path-traced image and asked which result more closely matched its lighting, shadows, and reflections.
- Baseline comparison: Compared with FEGR and UrbanIR, the method produces more photorealistic results without relying on explicit 3D geometry constraints.The compared reconstruction methods often retain shadow artifacts and struggle with highly detailed structures and dynamic scenes.