Source-linked AI summary
NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild
Jason Y. Zhang, Gengshan Yang, Shubham Tulsiani, Deva Ramanan
TL;DR
Sparse-view reconstruction in the wild is difficult because volumetric representations are too flexible for limited images and rough cameras. NeRS uses watertight neural surfaces with factorized reflectance and evaluates them on marketplace imagery, where it outperforms volumetric and classic mesh-based approaches.
Problem
Sparse in-the-wild image sets have few views and rough camera estimates, while volumetric methods often require many precisely posed images to recover coherent 3D structure.
Method
NeRS represents shape as a watertight neural surface and factorizes appearance into illumination, diffuse color, and specularity, trained with image-based losses on marketplace images.
Results
NeRS significantly outperforms neural volumetric and classic mesh-based approaches across all reported metrics in fixed-camera and in-the-wild novel-view synthesis evaluations.
Takeaways & Limitations
Surface-based neural reconstructions can learn from sparse in-the-wild multi-view data while modeling diverse materials and view-dependent appearance.
Takeaways & Limitations
NeRS cannot model objects with non-genus-zero topologies, and illumination and material properties remain fundamentally ambiguous.
Abstract
from arXiv · showhide
Recent history has seen a tremendous growth of work exploring implicit representations of geometry and radiance, popularized through Neural Radiance Fields (NeRF). Such works are fundamentally based on a (implicit) volumetric representation of occupancy, allowing them to model diverse scene structure including translucent objects and atmospheric obscurants. But because the vast majority of real-world scenes are composed of well-defined surfaces, we introduce a surface analog of such implicit models called Neural Reflectance Surfaces (NeRS). NeRS learns a neural shape representation of a closed surface that is diffeomorphic to a sphere, guaranteeing water-tight reconstructions. Even more importantly, surface parameterizations allow NeRS to learn (neural) bidirectional surface reflectance functions (BRDFs) that factorize view-dependent appearance into environmental illumination, diffuse color (albedo), and specular "shininess." Finally, rather than illustrating our results on synthetic scenes or controlled in-the-lab capture, we assemble a novel dataset of multi-view images from online marketplaces for selling goods. Such "in-the-wild" multi-view image sets pose a number of challenges, including a small number of views with unknown/rough camera estimates. We demonstrate that surface-based neural reconstructions enable learning from such data, outperforming volumetric neural rendering-based reconstructions. We hope that NeRS serves as a first step toward building scalable, high-quality libraries of real-world shape, materials, and illumination. The project page with code and video visualizations can be found at https://jasonyzhang.com/ners.
1 Introduction
NeRS addresses sparse-view 3D reconstruction by replacing overly flexible volumetric representations with a surface-based model of geometry and appearance. Its watertight shape representation and factorized reflectance support reconstruction from diverse in-the-wild images with approximate cameras.
- Sparse-view reconstruction must recover 3D geometry and appearance from 2D observations while choosing an appropriate representation.
- Neural volumetric representations model diverse structures and view-dependent lighting but often require hundreds of images with precise camera poses.
- With sparse in-the-wild images and imprecise camera estimates, unconstrained volumetric representations fail to infer coherent 3D structure.
- NeRS models geometry as a 2D manifold embedded in 3D and appearance with neural bidirectional surface reflection functions.
- A neural displacement field over a canonical sphere constrains NeRS to a watertight surface, while surface normals support physically grounded view-dependent lighting.
- Using a coarse category-level template and approximate camera poses, NeRS reconstructs diverse object instances and outperforms volumetric and classic mesh-based approaches in the in-the-wild setting.
2 Related Work
Related work spans explicit and neural surface representations, volumetric reconstruction, radiance fields, and multi-view datasets. NeRS differs by reconstructing detailed single instances from sparse images with coarse cameras and by targeting varied real-world marketplace imagery.
- Surface-based 3D Representations: Morphable models parameterize shapes as deformations of canonical templates, while later methods use continuous neural parametrizations for shape and texture.
- Surface-based 3D Representations: Earlier surface methods primarily perform category-level single-view prediction, whereas NeRS infers a detailed representation for one instance from multiple images without prior training.
- Volumetric 3D and Radiance Fields: Volumetric and radiance-field methods model single instances and complex geometry or illumination, but typically rely on dense images with precise camera localization for coherent reconstruction.
- Volumetric 3D and Radiance Fields: NeRS instead targets efficient learning from sparse images with coarse cameras for single-instance reconstruction.
- Multi-view Datasets: Common multi-view datasets are often controlled, small-scale, or insufficiently diverse to represent the range of real-world objects and materials.
3 Method
NeRS represents shape and texture continuously over a deformed sphere, then renders appearance by combining environment illumination, diffuse color, and specular effects. It is designed to learn detailed object reconstructions from sparse in-the-wild images using approximate cameras and coarse initialization.
- Neural Surface Representation: The surface representation provides normals and constrains geometry to a watertight surface, supporting physically grounded view-dependent lighting.Unlike unconstrained volumetric radiance, surface appearance is factorized into diffuse color and specularity.
- Neural Surface Representation: NeRS uses a neural deformation of a unit sphere to represent a continuous surface shape and a neural texture field for surface color.The shape mapping sends points on the sphere to 3D coordinates, while texture is represented similarly over the surface.
- Modeling Illumination and Specular Rendering: A neural environment map models incoming radiance as direction-dependent illumination, simplifying the lighting problem by assuming distant light sources.Under this assumption, incoming radiance depends on direction ω rather than surface position.
- Modeling Illumination and Specular Rendering: Phong shading decomposes radiance into a view-independent diffuse component and a view-dependent specular component controlled by shininess α and coefficient k_s.The output is L_o(x, v) = T(x) · I_diffuse(x) + k_s · I_specular(x, v); α controls mirror-like sharpness, while k_s controls highlight intensity.
- Learning NeRS in the Wild: The method jointly optimizes cameras, neural shape, texture, environment illumination, and material parameters from sparse images, using coarse category templates and approximate poses.The pipeline typically uses N images, often 8, and can begin from a cuboid initialization.
- Learning NeRS in the Wild: Surface rendering enables a perceptual image loss that compares rendered and input images using deep features, unlike volumetric approaches that operate on batches of rays.An additional mean-texture rendering loss helps learn visually plausible lighting.
4 Evaluation
NeRS is evaluated on in-the-wild multi-view reconstruction and novel-view synthesis using the Multi-view Marketplace Cars dataset, approximate or pseudo-ground-truth cameras, and sparse image sets. Across fixed-camera and unconstrained in-the-wild settings, it outperforms volumetric and mesh-based baselines while recovering detailed appearance and plausible illumination.
- Dataset: The Multi-view Marketplace Cars dataset contains 600 listings with at least 8 exterior views, plus 20 evaluation instances under varied illumination conditions.The dataset averages 10 exterior images per listing for the curated subset and 9.1 images per listing for evaluation.
- Evaluation setup: Fixed-camera evaluation holds out one image and uses manually corrected pseudo-ground-truth cameras to assess novel-view synthesis.Each evaluation treats one image-camera pair as the target and the remaining pairs as training data, repeating this across 182 evaluation cases.
- Evaluation setup: In-the-wild evaluation optimizes approximate training and test cameras, measuring synthesis under the camera uncertainty present in marketplace imagery.The model remains fixed while the test camera is refined against the test image.
- Baselines: NeRS is compared against modified NeRF, meta-learned NeRF variants, and IDR, with baselines adapted or selected for sparse-view reconstruction.Vanilla NeRF struggles in the low-data regime, while IDR has difficulty producing view-consistent outputs from limited views.
- Quantitative results: NeRS significantly outperforms the baselines across MSE, PSNR, SSIM, LPIPS, and FID in both fixed-camera and in-the-wild evaluations.The quantitative comparison covers both pseudo-ground-truth-camera and approximate-camera settings.
- Qualitative results: Qualitative results show detailed texture recovery and plausible illumination from 8–16 views, and a coarse cuboid mesh suffices to initialize detailed reconstruction.The evaluations include marketplace cars and household objects, with reconstructions visualized from novel views.
5 Discussion
NeRS learns watertight neural surface models that capture geometry and surface reflectance from sparse in-the-wild multi-view data. The paper identifies camera recovery as a major unresolved bottleneck and notes limitations from reflectance ambiguity, topology, and privacy risks.
- NeRS learns neural surface models that capture geometry and surface reflectance while enforcing watertight, closed manifolds.The representation supports view-dependent specularities and normal-dependent diffuse appearance.
- Approximate off-the-shelf camera poses and a new evaluation protocol address in-the-wild reconstruction when accurate camera recovery remains unsolved.The paper describes accurate camera recovery in the wild as a significant bottleneck affecting all approaches, including NeRS.
- NeRS supports reconstruction from sparse in-the-wild multi-view data across diverse materials and indoor and outdoor illumination conditions.
- The paper proposes future high-quality libraries of real-world geometry, materials, and environments through improved neural approximations of shape, reflectance, and illuminants.
- NeRS reconstructions may reveal identifiable or proprietary information, including license plates, motivating efforts to blur license plates and other personally identifiable information.
6 Appendix
The appendix details ablations, evaluation protocols, implementation choices, and limitations of NeRS and its baselines. These analyses examine view dependence, image-count effects, camera refinement, lighting assumptions, and competing reconstruction methods.
- Ablations: The appendix evaluates how increasing training-image counts affects NeRS novel-view synthesis and separately ablates view-dependence prediction.
- View-Dependence Ablation: NeRF-style view-direction conditioning produces artifacts resembling the NeRF baseline, including large appearance changes under small viewing-direction changes.The authors interpret this observation as evidence that learning a BRDF can regularize and improve generalization of view-dependent effects.
- Limitations of illumination model: The illumination model assumes grayscale environment lighting, which can bake non-white lighting hues into predicted texture and leaves gray texture–illumination brightness ambiguous.
- Evaluation and baselines: The appendix compares NeRS with direct view-dependent radiance conditioning, volume carving, and modified NeRF, while evaluating fixed-camera and in-the-wild novel-view synthesis on MVMC.The in-the-wild protocol permits methods to refine approximate test cameras given the target image.
- Implementation Details: NeRS uses neural texture, environment, and shape networks with positional encoding, fully connected layers, instance normalization, and Leaky ReLU.