Source-linked AI summary
Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Images
Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, Ravi Ramamoorthi
TL;DR
Deep Reflectance Volumes addresses practical scene appearance acquisition from unstructured mobile-phone flash images, where conventional reconstruction and reflectance capture are challenging. It jointly learns volumetric geometry and reflectance and renders them with a physically based differentiable ray-marching framework. The method supports novel-view synthesis, relighting, and material editing, producing higher-quality results than mesh-based methods on challenging real scenes.
Problem
Conventional mesh reconstruction and reflectance acquisition struggle with textureless or non-Lambertian regions and typically require dense, calibrated capture with sophisticated devices.
Method
The method jointly learns opacity, normal, diffuse-color, and roughness volumes and optimizes them through differentiable physically based volume rendering.
Results
The method reconstructs challenging real-scene geometry and reflectance and renders novel viewpoints and lighting, with higher-quality images than mesh-based methods.
Takeaways & Limitations
The learned reflectance volumes support practical view synthesis, relighting, and editable material representations from mobile-phone flash captures.
Takeaways & Limitations
The 128^3 volume resolution may fail to fully reproduce scenes with highly complex surface-normal variations and texture details, and the reflectance model is most appropriate for opaque surfaces.
Abstract
from arXiv · showhide
We present a deep learning approach to reconstruct scene appearance from unstructured images captured under collocated point lighting. At the heart of Deep Reflectance Volumes is a novel volumetric scene representation consisting of opacity, surface normal and reflectance voxel grids. We present a novel physically-based differentiable volume ray marching framework to render these scene volumes under arbitrary viewpoint and lighting. This allows us to optimize the scene volumes to minimize the error between their rendered images and the captured images. Our method is able to reconstruct real scenes with challenging non-Lambertian reflectance and complex geometry with occlusions and shadowing. Moreover, it accurately generalizes to novel viewpoints and lighting, including non-collocated lighting, rendering photorealistic images that are significantly better than state-of-the-art mesh-based methods. We also show that our learned reflectance volumes are editable, allowing for modifying the materials of the captured scenes.
1 Introduction
Deep Reflectance Volumes targets practical scene acquisition from unstructured mobile-phone flash images, using learned geometry and reflectance volumes with differentiable rendering for novel views and lighting.
- 1 Introduction: Mobile-phone flash capture offers a practical setup, but specularities, shadows, and non-Lambertian regions make conventional mesh reconstruction unreliable.The method uses unstructured images captured around a scene by one phone in a dark room.
- 1 Introduction: The method jointly learns opacity, normal, diffuse-color, and roughness volumes that represent geometry, surface orientation, and spatially varying reflectance.A decoder-like network learns the scene representation during per-scene optimization.
- 1 Introduction: Its physically based differentiable renderer accumulates shaded colors along rays and supports collocated light-view settings during training and arbitrary settings at inference.Global opacity is accumulated from both viewing and lighting directions, enabling hard-shadow computation under novel lighting.
- 1 Introduction: The learned volume representation enables view synthesis, relighting, and material editing while modeling complex geometry, occlusions, and appearance from ordinary captures.The stated contributions include a practical neural rendering framework and a scene representation based on opacity, normal, and reflectance volumes.
2 Related Works
Prior work addresses geometry, reflectance acquisition, relighting, and view synthesis, but remains limited by difficult capture setups, restricted scene content, or mesh quality.
- Geometry reconstruction: Mesh reconstruction remains difficult for textureless regions and thin structures, while category-specific deep mesh methods do not produce photorealistic results.The paper instead uses a physically based opacity volume to express arbitrary scene geometry within a deep learning system.
- Reflectance acquisition: Dense reflectance measurement typically requires sophisticated devices, and practical mobile-phone methods mostly focus on flat planar objects or limited single-view content.Recovering complete shape with spatially varying BRDF from multiview inputs has commonly relied on an initial mesh and controlled capture.
- Relighting and view synthesis: Image-based relighting can reproduce realistic specularities and soft shadows from sparse inputs but cannot reproduce accurate hard shadows, while mesh-based methods inherit reconstruction errors.The paper extends volumetric view synthesis toward explicit geometry-and-reflectance modeling for relighting.
3 Rendering with Deep Reflectance Volumes
Deep Reflectance Volumes jointly represent geometry and reflectance in voxel grids and render them with differentiable, physically based ray marching. The renderer models both camera- and light-path opacity, enabling single-bounce rendering with complex reflectance, occlusions, and shadows.
- 3.1 Volume rendering overview: The framework simplifies radiative transfer to single-point-light, single-scattering, no-emission rendering using a reflectance function parameterized by surface normal and material properties.The generalized phase function becomes a reflectance function for the simplified setting.
- 3.2 A discretized, differentiable volume rendering module: The physically based renderer computes point-wise reflected radiance from local normals, reflectance, and incoming lighting, then composites contributions along camera rays.It approximates volume rendering through discretized ray marching and opacity compositing.
- 3.2 A discretized, differentiable volume rendering module: Light-path opacity is accumulated separately from camera-path opacity, allowing the representation to model both view occlusions and lighting shadows.The light transmittance term corresponds to opacity accumulated along the light-to-surface ray.
- 3.1 Volume rendering overview: Deep Reflectance Volumes represent a scene with opacity, normal, and BRDF volumes that jointly encode geometry and reflectance.The opacity volume depends on the ray-marching step size.
- 3.1 Volume rendering overview: Unlike prior alpha-compositing view-synthesis formulations, this differentiable framework models single-bounce light transport with complex reflectance, occlusions, and shadows.Because the ray-marching module is fully differentiable, it supports backpropagation while jointly learning geometry and reflectance.
4 Learning Deep Reflectance Volumes
The method learns scene-specific Deep Reflectance Volumes from collocated-light multiview images using a jointly optimized encoding and decoder network. Learned warping, volumetric decoding, differentiable rendering loss, and opacity regularization support reconstruction from practical captures.
- 4.1 Overview: A scene-specific encoding vector and decoder network jointly reconstruct opacity, normal, and reflectance volumes from multiview images with collocated lighting.Unlike an encoder-predicted code, the scene encoding and decoder parameters are optimized together.
- 4.1 Overview: A learned warping function maps world-space positions into transformed volume space, where geometry and reflectance volumes are inferred and sampled by bilinear interpolation.The warping combines global and spatially varying affine transformations with optimized rotation, translation, and scale parameters.
- 4.2 Network architecture: The geometry-and-reflectance decoder uses transposed 3D convolutions to upsample the scene code into an 8-channel 128 × 128 × 128 volume.The decoded channels contain opacity, normal, and reflectance information.
- 4.3 Loss function and training details: Training minimizes an L2 rendering loss on randomly sampled captured pixels while regularizing accumulated opacity toward foreground/background values and promoting spatial opacity sparsity.The method assumes opaque objects and applies the priors in addition to the image-data term.
- 4.3 Loss function and training details: Each training batch samples 128 × 128 pixels from 8 captured images, uses ray-marching step size 1/64, and increases regularization weights after 300000 iterations.The weight schedule helps remove background artifacts and recover sharp boundaries.
5 Results
Experiments show that Deep Reflectance Volumes reconstruct challenging real scenes, outperform mesh-based and DeepVoxels baselines, support practical capture and editing, and benefit from deep-prior optimization.
- Comparisons with mesh-based reconstruction: Our method produces significantly better geometry and reflectance than the state-of-the-art mesh-based method on challenging real scenes.The mesh baseline fails in textureless, specular, and thin-structured regions, while the volumetric representation preserves geometric details and realistic specularities.
- Evaluation of the number of inputs: Comparable House-scene performance is achieved with 200 training images, although performance drops noticeably below 100 images.The final experiments use 385 images, and performance is evaluated using PSNR and SSIM.
- Comparison with DeepVoxels: Higher PSNR/SSIM scores show more accurate novel-view renderings than DeepVoxels under collocated lighting.The comparison retrains the model at 512 × 512 resolution for fairness.
- Additional results: The method handles multiple objects, complex occlusions, realistic cast shadows, and handheld video captures under novel views and lighting.Handheld captures remain effective despite potential motion blur and non-uniform coverage.
- Comparison with direct optimization: Deep-prior optimization converges significantly faster than direct voxelwise optimization by correlating appearance information across voxels.The network regularizes optimization and helps disentangle ambiguous information in captured images.
- Material editing: Explicit reflectance volumes enable material editing while preserving geometry and colors and introducing new specularities.Decreasing learned roughness makes an object appear glossier, and component edits need not influence other scene components.
- Limitations: The 128^3 volume resolution is limited by GPU memory and may fail to fully reproduce scenes with highly complex normal variations and texture details.The paper suggests increasing volume resolution as a possible remedy.
6 Conclusion
The paper learns volumetric representations that jointly model geometry and reflectance from unstructured mobile-phone flash captures. Its physically based renderer supports novel viewpoints, relighting, and material editing for complex real scenes.
- The method models complex real scenes with per-voxel opacity, normals, and reflectance learned from unstructured mobile-phone flash captures.
- Physically based differentiable rendering enables images under arbitrary viewing and lighting directions.
- The resulting representation supports novel view synthesis, relighting, and material editing, with potential applications in 3D visualization and VR/AR.
A BRDF Model
The paper uses a microfacet BRDF to model view-dependent appearance from local normals, diffuse albedo, roughness, and view/light directions. The resulting reflected radiance combines BRDF shading with the light-facing normal component.
- The reflectance model is a simplified microfacet BRDF using normal distribution, Fresnel, and geometric terms.The paper notes that essentially any differentiable BRDF could be incorporated, but applies the Walter et al. model with Karis-style simplifications.
- The BRDF takes view direction, light direction, normal, diffuse albedo, and roughness as appearance inputs.The per-point quantities are evaluated at scene location x_s.
- The model fixes the Fresnel base reflectance at F0 = 0.05.
- Reflected radiance is computed by evaluating the BRDF and multiplying it by the normal’s dot product with the light direction.
B Network Architecture
The network decodes a learned latent encoding into warped volumetric scene components. Its template volume stores opacity, normals, diffuse albedo, and roughness at 128 × 128 × 128 resolution, while separate outputs model global and spatially varying warps.
- The architecture begins with a 512-channel encoding vector, then uses fully connected layers and decoders for warping parameters and a template volume.The decoders predict global warping, spatially varying warping, and template-volume outputs.
- Global warping uses scaling, translation, and quaternion rotation parameters, while spatial warping uses 16 corresponding bases and a learned weight volume.
- The template volume has 8 channels: opacity, three normal channels, three diffuse-albedo channels, and roughness.
- The spatially varying weight volume has 16 channels at 32 × 32 × 32 resolution, and the template volume has 128 × 128 × 128 resolution.
C Testing Specifications
Testing evaluates generalization from collocated-light training images to novel viewpoints and lighting. The supplementary results include environment-illumination relighting and comparisons with a prior novel-view method.
- The supplementary video tests viewpoints whose angular distances from the nearest training views are reported by minimum, maximum, and average values.The training images use collocated light and camera, while the tested renderings use novel viewpoints and lighting.
- Figure 10 compares relighting under environment illumination against ground truth and shows the environment map used for rendering.
- Figure 11 compares novel-view synthesis under collocated lights with Sitzmann et al. and reports that the proposed method produces fewer artifacts.
D Results on Synthetic Data
On synthetic data, the method supports environment-map relighting by combining relit images for individual environment-map lights and produces visually plausible results against physically based ground truth.
- D Results on Synthetic Data: The synthetic evaluation renders scenes from multiple viewpoints under collocated camera and light, comparing view synthesis and relighting with ground-truth renderings.
- D Results on Synthetic Data: Environment-map relighting produces visually plausible images compared with ground-truth renderings from a physically based renderer.The result is demonstrated in Fig. 10.
- D Results on Synthetic Data: The geometry reconstructed from Nam et al. [37] is included for comparison in Fig. 12.
E Comparison on View Synthesis
Compared with prior methods, the approach better handles challenging real-scene geometry and appearance for novel-view synthesis under collocated lighting.
- E Comparison on View Synthesis: Sitzmann et al.'s representation produces distorted novel views with undesired structures, whereas the proposed method synthesizes the challenging real scenes more correctly.The comparison is shown in Fig. 11.
- E Comparison on View Synthesis: Nam et al.'s MVS-based geometry refinement fails to recover faithful geometry in textureless and thin-structured regions, degrading reproduced appearance.The optimized geometry is shown in Fig. 12.