Source-linked AI summary
Object-Centric Neural Scene Rendering
Michelle Guo, Alireza Fathi, Jiajun Wu, Thomas Funkhouser
TL;DR
NeRFs model static scenes under fixed illumination, limiting their use when objects or lights move. The paper represents objects with neural scattering functions and combines them with volumetric path tracing. It reports photorealistic renderings of multi-object scenes under novel illumination and notes a single-point-light training assumption.
Problem
NeRFs assume static scenes and fixed illumination, so new object or lighting configurations require separate scene models.
Source: §1 · p. 1Method
The method learns lighting- and view-dependent per-object OSFs and combines them with volumetric path tracing for scene rendering.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Object-centric scattering functions can be reused across scenes as object placements, cameras, and lighting change.
Source: §6 · p. 8During training, the method assumes a single point light source with radiance (1, 1, 1).
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
We present a method for composing photorealistic scenes from captured images of objects. Our work builds upon neural radiance fields (NeRFs), which implicitly model the volumetric density and directionally-emitted radiance of a scene. While NeRFs synthesize realistic pictures, they only model static scenes and are closely tied to specific imaging conditions. This property makes NeRFs hard to generalize to new scenarios, including new lighting or new arrangements of objects. Instead of learning a scene radiance field as a NeRF does, we propose to learn object-centric neural scattering functions (OSFs), a representation that models per-object light transport implicitly using a lighting- and view-dependent neural network. This enables rendering scenes even when objects or lights move, without retraining. Combined with a volumetric path tracing procedure, our framework is capable of rendering both intra- and inter-object light transport effects including occlusions, specularities, shadows, and indirect illumination. We evaluate our approach on scene composition and show that it generalizes to novel illumination conditions, producing photorealistic, physically accurate renderings of multi-object scenes.
1. Introduction
The introduction frames dynamic-scene rendering as difficult for NeRFs and traditional graphics because they depend on fixed scenes or detailed scene models. It proposes reusable per-object scattering functions combined with path tracing to render changing arrangements and illumination.
1. Introduction
- NeRF assumes static scenes and fixed illumination, while traditional graphics needs detailed geometry, lighting, materials, and cameras for photorealistic real-world scenes.For moving lights and objects, a NeRF must be trained separately for each scene configuration.Source: §1 · p. 1
- OSFs represent each object with a lighting- and view-dependent neural function that predicts volumetric density and how incoming light scatters toward an outgoing direction.The function is learned from 2D images.Source: §1 · pp. 1–2
- Because an OSF is intrinsic to its object rather than the scene, it can be reused across object placements and lighting conditions without retraining.This allows independently learned object functions to be composed into new scenes.Source: §1 · pp. 1–2
- Volumetric path tracing integrates OSFs over incoming light directions to model inter-object transport, including shadows and indirect illumination.The learned component handles intra-object transport, while per-scene path tracing handles inter-object transport; the stated exception is occlusion of intra-object illumination by other objects.Source: §1 · p. 2
- The authors report that the method renders novel scene compositions and lighting conditions better than alternative learned approaches.They describe the resulting renderings as photorealistic and physically accurate for multi-object scenes.Source: §1 · p. 2
2. Related Work
Related work spans object-centric graphics, novel-view synthesis, and relighting. The paper contrasts its continuous object scattering representation and volumetric rendering with prior methods' discretization, static-scene assumptions, or input requirements.
Novel view synthesis
- On the depicted dynamic scene, NeRF produces blurred predictions under randomly moved objects and lighting, while the proposed method predicts crisp images.The figure contrasts NeRF in panel (a) with the proposed method in panel (b).Source: Figure 2 · p. 2
Classical object-centric representations
- Prior object-centric graphics methods use discrete object radiance representations, whereas this work encodes continuous scattering in a network and uses volumetric rendering for inter-object illumination.The authors state that discrete representations cannot reproduce accurate inter-object transport, especially when object bounding volumes intersect.Source: §2 · p. 2
Novel view synthesis
- Other novel-view approaches require 3D geometry, limit viewing ranges to input-view interpolation, or lack view consistency with 2D convolutional kernels.Voxel-grid volume rendering is more view-consistent, but resolution is limited by the cost of discretely sampled volumes.Source: §2 · p. 2
- NeRF and related implicit methods produce high-quality novel views but assume static scenes with fixed illumination; this method targets changing viewpoints, lighting, and object configurations.NeRF represents a continuous radiance field with a multilayer perceptron.Source: §2 · p. 2
Relighting
- Relighting methods either lack hard-shadow recovery or require 3D geometry; the paper says its method needs no known material model and renders multiple objects in dynamic scenes.Neural Reflectance Fields instead focus on single objects or scenes and require manual BRDF specification.Source: §2 · p. 3
3. Preliminaries
The preliminaries establish volume rendering along camera rays, its numerical approximation by ray marching, and neural representations used to model scene appearance. They also introduce the object-scattering function illustrated in the final subsection’s supplied passage.
3. Preliminaries
- The preliminaries frame volume rendering, ray marching, and NeRF as background for the paper’s methods.The section overview identifies those topics as prerequisites and describes NeRF as using volume rendering with a continuous scene representation.Source: §3 · p. 3
3.1. Volume Rendering
- Volume rendering computes pixel color from radiance along camera rays through a volume, with density and accumulated transmittance contributing to the ray integral.The rendering equation also obtains scattered radiance by integrating over incoming light directions; the paper notes that this matters when lights and objects move relative to one another.Source: §3.1 · p. 3
3.2. Ray Marching
- Ray marching approximates the continuous rendering integrals by sampling points along each ray and incoming light paths, then averaging their contributions.The supplied passage describes stratified ray samples and a sampled set of incoming light paths.Source: §3.2 · p. 3
3.3. Neural Radiance Fields
- The supplied Figure 3 passage depicts an object-scattering function whose network maps point position and incoming and outgoing directions to density and a scattered-light fraction.The network’s learned weights parameterize the multilayer perceptron.Source: Figure 3 · p. 4
4. Method
The method represents each object with a learned scattering function and combines those functions through ray-based rendering. This separates object-level light transport from scene-level interactions, including occlusion and indirect illumination.
4.1. Object-Centric Neural Scattering Function
- Each object’s 7D OSF maps position and incoming and outgoing light directions to volumetric density and the fraction of light scattered between directions.The inputs are expressed in the object coordinate frame, with inverse object transforms applied before network evaluation.Source: §4.1 · p. 4
4.2. Rendering Multiple OSFs
- Camera rays are intersected with object bounding boxes, and samples are evaluated only along rays that hit objects.The rendering procedure prunes misses before sampling within each intersecting region.Source: §4.2 · pp. 4–5, Figure 5 · p. 5
- Shadow rays evaluate occlusion by other objects, while sampled secondary directions account for indirect illumination between objects.The incoming radiance is attenuated by shadow-ray transmittance; the secondary-ray formulation supports an arbitrary number of bounces.Source: §4.2 · p. 5
- The final ray radiance combines samples across objects and light paths, with an upper bound on sample count that is often reduced because a ray intersects at most one object.The method describes rendering with paths containing direct and indirect illumination.Source: §4.2 · p. 5
4.3. Implementation Details
- The implementation uses an eight-layer, 256-channel MLP for density and a four-layer, 128-channel MLP for scattering.It uses positional-encoding frequency levels W = 10 for position and W = 4 for directions, with 64 coarse and 128 fine samples per ray.Source: §4.3 · pp. 5–6
5. Experiments
Across object-level novel lighting and multi-object scene composition, OSF produces more accurate appearances than the o-NeRF baselines, with shadows and indirect illumination represented in composed scenes. The experiments also test environment-map lighting and show that the full model is more realistic than variants omitting shadows or indirect illumination.
5. Experiments
- The evaluation uses Furniture-Single, Furniture-Random, and Furniture-Realistic datasets, rendered with Blender’s Cycles path tracer at 256 by 256 resolution.Quantitative metrics are PSNR, SSIM, and LPIPS.Source: §5 · p. 6
5.1. Novel Lighting
- Under novel lighting, OSF renders object appearance more accurately than o-NeRF, recovering couch and chair self-shadows and ottoman specular details.The evaluation trains on 400 images per object and tests on 20 images with novel viewpoints and lighting.Source: §5.1 · p. 6, Figure 6 · p. 6
5.2. Scene Composition
- OSF outperforms the baselines and ablations on Furniture-Random scene composition, both quantitatively and qualitatively, while reproducing self-shadows and inter-object light transport.It renders object-cast shadows and indirect illumination, including left-wall light brightening the couch and table.Source: §5.2 · pp. 7–8, Table 2 · p. 7
5.3. Complex Illumination
- With a point light and environment map, the Furniture-Realistic scene shows a green-blue tint from the environment lighting.The experiment samples 20 incoming directions per OSF sample point.Source: §5.3 · p. 8, Figure 9 · p. 8
5.4. Ablations
- The full OSF model is the most realistic ablation, compared with versions using direct illumination only, indirect illumination only, or direct illumination plus shadows.The full model includes both shadows and indirect illumination.Source: §5.4 · p. 8, Figure 10 · p. 8
6. Conclusion
The conclusion presents OSFs as a way to compose objects into photorealistic dynamic scenes, with object functions reusable across changing placements, cameras, and lighting. Integrating the functions with volumetric path tracing models inter-object shadows and indirect illumination.
6. Conclusion
- OSFs enable photorealistic renderings of dynamic scenes by composing objects whose learned functions can be reused as placements, cameras, and lighting change.The method integrates learned object functions with volumetric path tracing to model inter-object shadows and indirect illumination.Source: §6 · p. 8