Source-linked AI summary
PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting
Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, Noah Snavely
TL;DR
Inverse rendering must recover geometry, materials, and illumination from RGB images, including glossy objects under static natural lighting. PhySG uses a differentiable physics-based renderer with SDF geometry and spherical-Gaussian appearance representations to jointly estimate these factors. On synthetic and real data, the reconstructions support novel views, material editing, and relighting, while the method remains limited by assumptions about specular cues, object-level data, and BRDF form.
Problem
Inverse rendering seeks to recover shape, material, and lighting from images, including challenging multi-view RGB settings with glossy objects under static illumination.
Method
PhySG is an end-to-end physics-based differentiable rendering pipeline using SDFs for geometry, MLPs for SDF representation, and spherical Gaussians for lighting and specular BRDFs.
Results
PhySG jointly recovers environment maps, material BRDFs, and geometry from static-illumination multi-view inputs, producing novel views and appearance-editing results on synthetic and real data.
Takeaways & Limitations
The reconstructions support physics-based material editing and relighting in addition to novel-view synthesis.
Takeaways & Limitations
The method relies on specular highlights and assumes constant, monochrome specular BRDFs with spatially varying diffuse components; its SG renderer limits it to object-level data.
Abstract
from arXiv · showhide
We present PhySG, an end-to-end inverse rendering pipeline that includes a fully differentiable renderer and can reconstruct geometry, materials, and illumination from scratch from a set of RGB input images. Our framework represents specular BRDFs and environmental illumination using mixtures of spherical Gaussians, and represents geometry as a signed distance function parameterized as a Multi-Layer Perceptron. The use of spherical Gaussians allows us to efficiently solve for approximate light transport, and our method works on scenes with challenging non-Lambertian reflectance captured under natural, static illumination. We demonstrate, with both synthetic and real data, that our reconstructions not only enable rendering of novel viewpoints, but also physics-based appearance editing of materials and illumination.
1. Introduction
PhySG addresses multi-view inverse rendering from ordinary RGB images of glossy objects under static illumination, without scanned geometry. It jointly estimates geometry, materials, and lighting for novel-view synthesis, relighting, and material editing.
- Prior neural rendering methods support tasks such as shape reconstruction and novel-view synthesis, but texture- or radiance-field representations do not factorize appearance for physical editing.
- PhySG tackles multi-view inverse rendering from normal RGB images under shared static illumination without assuming scanned geometry.
- The pipeline jointly estimates lighting, material properties, geometry, and surface normals from posed multi-view images of specular objects.
- Spherical Gaussians approximately and efficiently evaluate the rendering equation, supporting physics-based appearance manipulation.
- PhySG supports novel viewpoints as well as physically intuitive material editing and relighting.
2. Background
The background contrasts neural surface-light-field approaches with PhySG’s physics-based appearance model. It introduces the rendering equation and motivates jointly estimating geometry, materials, and lighting without scanned geometry or a known environment map.
- Neural rendering: Neural rendering methods commonly represent appearance with an MLP that maps spatial and viewing information to color, whereas PhySG models appearance through the physical rendering equation.
- Material and environment estimation: Prior material-estimation methods generally require varying illumination, while some environment-estimation methods rely on RGBD geometry or entangle material with lighting.
- Joint shape and appearance refinement: PhySG estimates material, lighting, geometry, and surface normals end-to-end without scanned geometry or a known environment map.
- The rendering equation: The rendering equation integrates incident light, BRDF reflectance, and the cosine-weighted incident direction over the visible hemisphere to produce outgoing radiance.
3. Method
PhySG uses differentiable SDF-based geometry and spherical-Gaussian appearance models to jointly optimize geometry, materials, and illumination from multi-view images under static illumination. Its closed-form spherical-Gaussian rendering approximation supports end-to-end optimization while avoiding expensive numerical integration.
- Pipeline: The pipeline jointly models geometry, appearance, and forward rendering as differentiable components optimized end-to-end from images captured under static illumination.Its appearance includes an environment map, spatially varying diffuse albedo, and a shared monochrome isotropic specular component.
- Geometry modeling: SDFs represented by MLPs provide differentiable geometry, sphere-tracing intersections, and surface normals obtained as SDF gradients.The SDF MLP uses positional encoding and a skip connection to represent high-frequency geometric detail.
- Geometry modeling: Sphere tracing requires approximately 10 MLP evaluations per ray for SDFs, compared with over 100 for occupancy fields.This makes SDF-based ray casting substantially more efficient than tracing through occupancy fields.
- Appearance modeling: Environmental illumination is represented as a mixture of M = 128 spherical Gaussians, while diffuse albedo is predicted by an MLP from surface position.The environment uses distant illumination, dropping its spatial location dependence; the albedo MLP uses positional encoding for high-frequency texture.
- Appearance modeling: The shared specular component uses a simplified Disney BRDF with an isotropic single spherical-Gaussian distribution aligned to the surface normal.The isotropic assumption sets the lobe axis ξ to the normal n, and the monochrome assumption makes the three amplitude values identical.
- Forward rendering: PhySG approximates the rendering equation by representing its terms with spherical Gaussians, approximating the cosine factor with another spherical Gaussian, and integrating their product in closed form.This closed-form integration yields the observed color and makes the rendered output differentiable with respect to appearance and SDF parameters.
4. Experiments
Experiments on synthetic and real glossy-object captures evaluate PhySG for novel-view extrapolation, reconstruction, material editing, and relighting. The method produces plausible results under difficult viewpoint shifts and static illumination, while comparisons expose a trade-off between physical editability and real-world view quality.
- Experimental setup: PhySG is evaluated on synthetic and real-world data using held-out viewpoints and qualitative comparisons.The experiments include synthetic renders and captures from SLF, DeepVoxels, Bag of Chips, and DTU.
- Synthetic data: Synthetic test views closely match ground truth for novel views, diffuse albedo, surface normals, material editing, and relighting despite difficult view extrapolation.The evaluation includes quantitative results in Table 1 and qualitative results in Figure 7.
- Comparison with baselines: PhySG extrapolates specular highlights more reasonably than NeRF, IDR, and DVR through physics-based approximate light transport modeling.The comparison is qualitative on synthetic and real data.
- Real-world data: PhySG qualitatively generates photo-realistic novel views and plausible material editing and relighting results on real-world captures.Figure 5 also visualizes synthesized images, diffuse estimates, edited albedo, novel illumination, and surface normals.
- Comparison with baselines: PhySG’s estimated geometry is nearly as good as IDR’s and much better than other baselines, while also supporting relighting and material editing.The geometry comparison is reported in Table 2.
- Robustness and limitations: For purely Lambertian materials, PhySG faces a lighting-texture ambiguity and cannot recover lighting without additional priors.The method relies on specular highlights to estimate lighting and material properties.
- Robustness and limitations: Even weak specular highlights allow PhySG to reconstruct a reasonable-looking environment map on synthetic data.The roughness robustness study evaluates this behavior empirically.
5. Conclusion
PhySG jointly recovers environment maps, material BRDFs, and geometry from multi-view inputs under static illumination, enabling physics-based material editing and relighting. Its scope is limited by omitted indirect illumination and assumptions about specular BRDFs.
- PhySG uses physics-based differentiable rendering with SDFs for geometry and spherical Gaussians for appearance.
- Omitting indirect illumination limits the method to object-level data rather than scene-level data.
- The method assumes constant and monochrome specular BRDFs with spatially varying diffuse components because of illumination–reflectance scale ambiguity.