PaperlayerView exact revision on arXiv

Source-linked AI summary

NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images

Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, Wenping Wang

arXiv:2305.17398v1cs.CVcs.GR

TL;DR

Reflective-object reconstruction is difficult because view-dependent reflections break multiview consistency and environment lighting is unknown. NeRO uses a two-stage neural rendering approach to reconstruct geometry, environment lighting, and BRDF from posed RGB images without masks, achieving better reconstruction and BRDF estimation than state-of-the-art methods.

  • Problem

    Reflective-object reconstruction remains difficult because view-dependent reflections violate multiview consistency, while existing methods rely on masks and weak-specularity assumptions.

  • Method

    NeRO uses a novel direct-and-indirect light representation and two stages: approximate shading for geometry, then fixed-geometry Monte Carlo sampling for BRDF and environment-light recovery.

  • Results

    NeRO reconstructs reflective-object geometry and BRDF on benchmark datasets, outperforming state-of-the-art methods in surface reconstruction quality and BRDF estimation.

  • Takeaways & Limitations

    The method produces textured meshes with estimated BRDF parameters that can support downstream applications such as relighting.

  • Takeaways & Limitations

    NeRO may miss subtle details because neural SDFs produce smooth normals, and BRDF estimation suffers when the reconstructed geometry is incorrect.

Abstract

from arXiv · show

We present a neural rendering-based method called NeRO for reconstructing the geometry and the BRDF of reflective objects from multiview images captured in an unknown environment. Multiview reconstruction of reflective objects is extremely challenging because specular reflections are view-dependent and thus violate the multiview consistency, which is the cornerstone for most multiview reconstruction methods. Recent neural rendering techniques can model the interaction between environment lights and the object surfaces to fit the view-dependent reflections, thus making it possible to reconstruct reflective objects from multiview images. However, accurately modeling environment lights in the neural rendering is intractable, especially when the geometry is unknown. Most existing neural rendering methods, which can model environment lights, only consider direct lights and rely on object masks to reconstruct objects with weak specular reflections. Therefore, these methods fail to reconstruct reflective objects, especially when the object mask is not available and the object is illuminated by indirect lights. We propose a two-step approach to tackle this problem. First, by applying the split-sum approximation and the integrated directional encoding to approximate the shading effects of both direct and indirect lights, we are able to accurately reconstruct the geometry of reflective objects without any object masks. Then, with the object geometry fixed, we use more accurate sampling to recover the environment lights and the BRDF of the object. Extensive experiments demonstrate that our method is capable of accurately reconstructing the geometry and the BRDF of reflective objects from only posed RGB images without knowing the environment lights and the object masks. Codes and datasets are available at https://github.com/liuyuan-pal/NeRO.

1 INTRODUCTION

Reflective-object reconstruction remains difficult because view-dependent reflections violate multiview consistency and rendering-equation evaluation is intractable with unknown geometry and environment lighting. NeRO addresses this by reconstructing geometry and BRDF from posed RGB images using separate direct/indirect-light modeling, succeeding on synthetic and real datasets where baseline methods fail.

  • Motivation: Reflective objects remain challenging to reconstruct because glossy, view-dependent reflections violate the multiview consistency assumed by conventional methods.The introduction identifies reflective-object reconstruction as an outstanding problem despite major progress in multiview 3D reconstruction.
  • Motivation: Neural reconstruction represents geometry implicitly, but evaluating environment-light integrals remains intractable when surface locations and lighting are unknown.This difficulty arises when explicitly incorporating the rendering equation to model BRDF–environment-light interactions.
  • Method: NeRO reconstructs reflective-object geometry and BRDF from only posed RGB images by incorporating the rendering equation into neural reconstruction.Its light representation uses separate MLPs for direct and indirect lights and an occlusion probability to select between them.
  • Evaluation: NeRO is evaluated on synthetic and real datasets containing reflective objects illuminated by complex environment lights.The datasets are introduced specifically to evaluate reconstruction under complex lighting conditions.
  • Results: On both datasets, NeRO successfully reconstructs geometry and surface BRDFs, while baseline MVS and neural reconstruction methods fail.The method outputs a triangular mesh with estimated BRDF parameters suitable for downstream applications such as relighting.

2 RELATED WORKS

Related work spans correspondence-based multiview reconstruction, neural surface and image-based rendering, and inverse-rendering approaches for BRDF estimation. Strong specular reflections undermine multiview consistency, while prior reflective-object methods often require masks, reflection removal, known environments, or constrained settings.

  • Multiview reconstruction: Traditional correspondence-based multiview reconstruction performs poorly on reflective objects because strong specular reflections violate multiview consistency.These methods rely on consistent 3D points to establish correspondences and estimate depth across views.
  • Neural surface reconstruction: Neural rendering and neural representations have improved novel-view synthesis and introduced neural surface representations for multiview reconstruction.DVR first introduced neural rendering and neural surface representation into multiview reconstruction, while IDR improved reconstruction quality.
  • Reflective-object reconstruction: Prior reflective-object reconstruction methods use object masks, remove reflections, or assume known specular flows or environments in constrained settings.These approaches target reflective reconstruction under additional information or restrictions, including ideal mirror-like objects.
  • Reflective-object rendering: Image-based rendering methods model glossy or reflective objects, but ORCA relies on object masks and primarily reconstructs the scene radiance field rather than object geometry and BRDF.Related methods include NeRFRen, Neural Point Catacaustics, and Ref-NeRF for reflective-scene rendering and reconstruction.
  • BRDF estimation: BRDF estimation commonly uses inverse rendering, object or scene priors, and differentiable renderers to optimize material properties from image losses.Prior-based methods estimate BRDF and lighting directly, while differentiable renderers enable direct BRDF optimization.

3 METHOD

NeRO reconstructs reflective-object geometry and BRDF from posed RGB images without object masks or known environment lighting. Its two-stage pipeline first makes geometry reconstruction tractable with neural SDF rendering and approximated lighting, then fixes geometry to recover lighting and material properties more accurately.

  • Stage I: Geometry Reconstruction: NeRO represents the object surface as the zero-level set of an MLP-encoded neural SDF and uses NeuS volume rendering to optimize it from posed RGB images.Surface points are sampled along camera rays, and the SDF and color-related networks are learned by minimizing rendered-versus-input color differences.
  • Stage I: Geometry Reconstruction: The method predicts metalness, roughness, and albedo, then computes rendered colors with a micro-facet BRDF and rendering equation.The BRDF is parameterized by metalness m, roughness ρ, and albedo a, while normals come from SDF gradients.
  • Stage I: Geometry Reconstruction: Split-sum approximation separates diffuse and specular lighting effects, while integrated directional encoding makes the corresponding light integrals tractable during volume rendering.The approximation reduces the unknown lighting terms to Ldiffuse and Lspecular after material parameters are predicted.
  • Stage I: Geometry Reconstruction: NeRO models lighting inside a bounding sphere as direction-dependent direct light plus position-dependent indirect light, blended using an MLP-predicted occlusion probability.Direct light provides a shared global environment prior, whereas indirect light varies spatially with the surface position.
  • Stage II: BRDF Reconstruction: With geometry fixed in Stage II, NeRO uses the same material and light representations but computes occlusion by ray tracing the reconstructed geometry instead of predicting it with an occlusion MLP.Stage II uses more accurate light and BRDF evaluation to recover environment lighting and material properties.

4 EXPERIMENTS

Experiments show that NeRO reconstructs reflective-object geometry under direct and indirect illumination more accurately than the evaluated baselines and supports plausible BRDF and relighting results. Ablations clarify the roles of combined lighting, occlusion supervision, and regularization, while remaining limitations include missing subtle details and pose-recovery requirements.

  • BRDF: Relighting evaluation reports BRDF quality with PSNR, while baselines are limited by incorrect surfaces or overly rough materials with blurred reflections.NDR estimates BRDF inaccurately because of surface errors, whereas NDRMC tends to produce rough materials and blurred reflections on glossy objects.
  • Geometry: NeRO accurately reconstructs objects with large reflective surfaces under both direct and indirect lighting, whereas baseline methods produce distorted, noisy, incomplete, or cracked surfaces.COLMAP fails under strong reflections; Ref-NeRF struggles with noisy geometry and indirect lights; NDRMC produces holes or cracks; and NeuS distorts reflective surfaces.
  • Geometry: On the Glossy-Real dataset, NeRO similarly reconstructs reflective surfaces accurately while each baseline fails on several objects and Ref-NeRF produces very noisy surfaces.Quantitative geometry results use Chamfer distance (CD), with qualitative comparisons shown for the Glossy-Real dataset.
  • Ablation Study: Ablations show that combining direct and indirect lights, occlusion loss, and Eikonal regularization is important for stable reconstruction across reflective objects.Models omitting occlusion supervision converge unstably, while omitting Eikonal loss can create double-layer surfaces in indirectly illuminated regions.
  • Relighting: Importance sampling on the specular lobe is necessary to capture high-frequency specular colors and avoid overestimating surface roughness during relighting.Without it, estimated roughness becomes large; stage-I BRDF estimates otherwise tend to have very small roughness because smooth materials fit specular colors more easily.
  • Limitations: NeRO reconstructs overall shape successfully but misses subtle details because neural SDFs produce overly smooth surface normals, and pose recovery may require non-reflective objects or IMU assistance.Examples include cloth textures on “Angel” and beards or other textures on “Cat”; the supplied passage also notes recovering poses from co-visible non-reflective objects or devices like IMU.

5 CONCLUSION

NeRO reconstructs reflective-object geometry and BRDF without known environment lighting or object masks by incorporating the rendering equation, a novel light representation, and a two-stage approach. Experiments show better surface reconstruction and BRDF estimation than state-of-the-art methods.

  • Conclusion: NeRO reconstructs reflective-object geometry and BRDF without knowing environment light conditions or object masks.Its framework explicitly incorporates the rendering equation into neural reconstruction.
  • Conclusion: NeRO uses a novel light representation and a two-stage reconstruction approach.The second stage fixes geometry and uses Monte Carlo sampling to recover surface BRDF and environment light.
  • Conclusion: NeRO achieves better surface reconstruction quality and BRDF estimation for reflective objects than state-of-the-art methods.These results are supported by experiments on reflective-object reconstruction and appearance estimation.

A APPENDIX · A.1 BRDF model

The appendix specifies NeRO’s Cook–Torrance BRDF model, including metalness-dependent base reflectance, Schlick-GGX geometry, and a Stage II Trowbridge-Reitz GGX normal distribution. Its split-sum prefiltered BRDF terms are stored in an image and interpolated using roughness and the view-angle cosine.

  • A.1 BRDF model: NeRO adopts the Cook–Torrance BRDF for modeling reflective-object appearance.The model defines the basic reflection ratio as F0 = (m*a + (1 − m)*0.04), where a is albedo and m is metalness.
  • A.1 BRDF model: The basic reflection ratio F0 combines albedo-dependent and dielectric reflectance terms through metalness m.The appendix gives F0 = (m*a + (1 − m)*0.04), with a denoting albedo.
  • A.1 BRDF model: The Fresnel term is defined using the half-way vector h and viewing direction ω_o.The supplied description identifies h as the half-way vector and ω_o as the viewing direction.
  • A.1 BRDF model: The geometry function follows the Schlick-GGX formulation.Its sub-function is given as G_sub(n,v,k) = (n · v)^2 / ((n · v)(1 − k) + k).
  • A.1 BRDF model: Stage II uses the Trowbridge-Reitz GGX distribution for the normal distribution.The supplied passage introduces this distribution after defining the roughness-dependent geometry parameter.
  • A.1 BRDF model: Direct environment lights can be represented using directions alone while still providing a satisfactory approximation for surface reconstruction in an indoor environment.This approximation is illustrated with an indoor image and estimated direct lights.
  • A.1 BRDF model: The prefiltered BRDF terms F1 and F2 for the split-sum are stored in an image and retrieved by interpolation.The image axes represent roughness ρ and n · ω_o; given those values, F1 and F2 are interpolated from Fig. 21.

A.2 Discussion on the direct light representation

NeRO’s direction-only direct-light representation is sufficiently accurate for Stage I surface reconstruction, but not for Stage II BRDF estimation because finite-distance strong lights require spatial information. Stage II therefore adds sphere-intersection encoding, while using it in Stage I can encourage geometric overfitting.

  • Stage I direct-light representation: In Eq. 9, direction-only direct lighting assumes all light sources lie at infinity, yet this approximation remains accurate enough for surface reconstruction in challenging indoor environments.The method illustrates Stage I’s estimated environment lights in Fig. 22.
  • Stage II direct-light representation: For Stage II BRDF estimation, direction-only lighting is insufficient because BRDF recovery is highly sensitive to strong light sources at finite distances.The representation enlarges estimated strong light sources and can produce inaccurate, darker albedo regions.
  • Stage II direct-light representation: Stage II represents direct light as g_direct(q(p,ω_i),ω_i), where q(p,ω_i) is the ray’s intersection with the bounding sphere.This sphere-intersection representation supplies spatial information beyond the incident direction.
  • Sphere-intersection encoding: Adding sphere-intersection encoding in Stage I degenerates reconstruction performance because increased fitting ability can favor color fitting on distorted surfaces over faithful geometry.Stage II avoids this single-direction overfitting through Monte Carlo sampling of many rays per pixel.

A.3 Rationale of light integral approximations

The two light-integral approximations are justified by surface roughness and specular-lobe size. Smooth surfaces with small specular lobes are dominated by the reflective direction, while rough surfaces with large lobes produce slowly changing, predominantly white integrals that tolerate lower-frequency approximations.

  • Smooth surfaces: On smooth surfaces with small specular lobes, light integrals are mainly determined by light from the reflective direction.The occlusion probability of that direction therefore dominates the integral, making reflective-direction probability a good approximation for other directions.
  • Rough surfaces: On rough surfaces with large specular lobes, light integrals are influenced by more than the reflective direction, limiting the first approximation’s assumptions.However, these integrals are mainly white and change slowly with view direction, so exact direct or indirect prediction is unnecessary.
  • Rough surfaces: Integrated directional encoding produces low-frequency directional representations for slowly changing illumination on rough surfaces.The passage states that IDE with a large ρ produces a low-frequency directional encoding in this case.

A.4 Stabilization loss

The method uses a background NeRF, implemented as NeRF++, alongside the neural SDF, but imbalanced convergence between foreground shading and background color can collapse training. If foreground shading converges faster, the surface enlarges beyond the bounding sphere; otherwise, it shrinks and disappears.

  • Implementation: The background renderer follows NeuS and is implemented as NeRF++, providing the image background alongside the neural SDF.Neural SDF initialization follows Atzmon and Lipman (2020), while background rendering follows NeuS (2021b).
  • Failure mode: Imbalanced convergence between neural-SDF shading color and background-NeRF color causes the training process to collapse.The imbalance arises because the foreground shading and background colors converge at different speeds.
  • Failure mode: Faster foreground-shading convergence enlarges the surface beyond the bounding sphere, whereas slower convergence makes it shrink and eventually disappear.Both outcomes result from the convergence-speed imbalance between foreground shading and background color.

A.5 Network architectures and implementation details

The method uses a NeuS-style SDF network with positional encoding, eight 256-channel linear layers, and a fourth-layer skip connection, producing both an SDF value and a 256-dimensional feature vector.

  • Architectures: The g_sdf network applies positional encoding with frequency 6 and contains eight linear layers with 256 channels.Its architecture follows NeuS.
  • Architectures: A skip connection is placed on the fourth layer of g_sdf.The network outputs an SDF value and a 256-dimensional feature vector.

A.6 Dataset statistics

The datasets provide multiview training images for reflective-object reconstruction, with additional test views in Glossy-Blender for novel-view synthesis evaluation. Image resolutions differ between Glossy-Blender and Glossy-Real.

  • Glossy-Blender: Glossy-Blender provides 128 training images per object, uniformly distributed on the upper hemisphere.The training images have 800×800 resolution.
  • Glossy-Blender: Glossy-Blender additionally provides 8 evenly-distributed test images for PSNR, SSIM, and LPIPS evaluation.The test images also have 800×800 resolution.
  • Glossy-Real: Glossy-Real image counts per object are reported in Table 6, and all images have 1024×768 resolution.The passage states that these images are used for training.

A.7 Model size

NeRO is slightly larger than NeuS because its color function is decomposed into materials and lighting. The study also evaluates an approximately 8M-parameter NeuS-Large variant with increased color-network width and depth.

  • A.7 Model size: NeRO uses the same SDF and background networks as NeuS but is slightly larger because its color function is decomposed into materials and lighting.The comparison includes Ref-NeRF, NeuS, and NeRO in Table 8.
  • A.7 Model size: Increasing the width and depth of NeuS’s color network produces an approximately 8M model called NeuS-Large.

A.8 Relighting results in SSIM/LPIPS … A.12 Copyrights

The supplementary experiments report strong relighting and novel-view synthesis quality, including on less- or non-reflective objects. They also analyze a Ref-NeRF–NeuS combination and document dataset and image-source copyrights.

  • A.8 Relighting results in SSIM/LPIPS: On Glossy-Blender, NeRO outperforms relighting baselines by a significant margin in SSIM and LPIPS, with results similar to PSNR.The evaluated baselines are NDR, NDRMC, MII, and NeILF.
  • A.9 Novel-view synthesis quality: NVS quality is additionally evaluated by rendering 8 novel-view images on Glossy-Blender using PSNR, SSIM, and LPIPS.The comparison is reported in Table 11 for NeuS and Ref-NeRF.
  • A.10 Results on less- or non-reflective objects: NeRO also reconstructs less- or non-reflective objects, including three less-reflective objects and sets containing both reflective and non-reflective objects.The three less-reflective objects use approximately 100 images each at 1024×768 resolution, with camera poses recovered by COLMAP; mixed-object sets use approximately 200 images each.
  • A.10 Results on less- or non-reflective objects: On DTU, NeRO is evaluated after manually removing images with inconsistent light environments because the method assumes static lighting while DTU includes inconsistent lighting and shadows.Quantitative and qualitative results are presented for the filtered training set.
  • A.11 Direct combination of Ref-NeRF with NeuS: A direct combination of Ref-NeRF’s color function with NeuS’s neural SDF resembles Model 2 in the geometry ablation study, while differing in material integration and diffuse-color prediction.Both approaches predict specular lights from positional encoding and integrated directional encoding; Ref-NeRF omits the integral of material M_material in Eq. 7 and directly predicts diffuse color.
  • A.12 Copyrights: The Glossy-Blender dataset is built from models listed in Table 14, with some appearances and geometries modified for dataset creation.All HDR images used for relighting or rendering were downloaded from Poly Haven under the CC0 license.
  • A.12 Copyrights: Supplementary figures provide qualitative comparisons for Glossy-Blender relighting, Glossy-Real relighting, reconstructed surfaces, and other real objects.Comparisons include COLMAP, Ref-NeRF, NDRMC, NeuS, NDR, MII, and NeILF, with relevant notes on object masks and color normalization.
Loading 2305.17398v1…