Source-linked AI summary
UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction
Michael Oechsle, Songyou Peng, Andreas Geiger
TL;DR
Existing surface-rendering methods need object masks, while NeRF’s density does not provide accurate surfaces. UNISURF unifies implicit surfaces and radiance fields with a coarse-to-fine sampling strategy, achieving accurate mask-free reconstruction competitive with IDR. Its scope is limited to solid objects and is sensitive to overexposed, textureless, and rarely visible regions.
Problem
Surface-rendering methods require accurate masks, whereas NeRF’s density-based geometry is often non-smooth and artifact-prone.
Method
UNISURF unifies implicit surfaces and radiance fields, using decreasing sampling regions to combine volumetric coarse-structure learning with precise surface representation.
Results
UNISURF reconstructs accurate geometry without masks on DTU, performs nearly on par with IDR, and is demonstrated on BlendedMVS and synthetic indoor scenes.
Takeaways & Limitations
The unified formulation provides a path toward optimizing implicit surfaces in more general settings than previous approaches.
Takeaways & Limitations
The model represents only solid, non-transparent surfaces and is less accurate in overexposed, textureless, and rarely visible regions.
Abstract
from arXiv · showhide
Neural implicit 3D representations have emerged as a powerful paradigm for reconstructing surfaces from multi-view images and synthesizing novel views. Unfortunately, existing methods such as DVR or IDR require accurate per-pixel object masks as supervision. At the same time, neural radiance fields have revolutionized novel view synthesis. However, NeRF's estimated volume density does not admit accurate surface reconstruction. Our key insight is that implicit surface models and radiance fields can be formulated in a unified way, enabling both surface and volume rendering using the same model. This unified perspective enables novel, more efficient sampling procedures and the ability to reconstruct accurate surfaces without input masks. We compare our method on the DTU, BlendedMVS, and a synthetic indoor dataset. Our experiments demonstrate that we outperform NeRF in terms of reconstruction quality while performing on par with IDR without requiring masks.
1. Introduction
Neural implicit representations support multi-view geometry and appearance reconstruction, but surface rendering requires masks while radiance-field density yields imperfect geometry. UNISURF unifies both paradigms to reconstruct accurate surfaces without input masks.
- Continuous neural implicit functions represent 3D geometry or appearance and support reconstruction and novel-view synthesis from multi-view images.
- Surface-rendering methods require per-pixel object masks and suitable initialization because they provide gradients only where rays intersect surfaces.This dependence limits them to object-level reconstruction and larger scenes.
- NeRF supports larger scenes without masks, but level sets of its density volume are typically non-smooth and artifact-prone.Radiance-field flexibility does not sufficiently constrain geometry under ambiguities.
- UNISURF unifies implicit surfaces and radiance fields, combining surface and volume rendering to reconstruct accurate geometry without mask supervision.Its sampling region decreases during optimization, beginning broadly for coarse geometry and later concentrating near the surface.
- UNISURF performs nearly on par with IDR on DTU while avoiding strong mask supervision and is also demonstrated on BlendedMVS and synthetic indoor scenes.
2. Related Work
Related work spans classic and learning-based multi-view stereo, neural implicit representations, surface rendering, and volumetric radiance models. UNISURF targets accurate geometry without masks by combining the latter two approaches.
- 3D Reconstruction from Multi-View Images: Classic multi-view stereo uses feature matching or voxel representations, while learning-based methods learn feature matching, depth fusion, or depth inference.
- Neural Implicit Representations: Neural implicit functions represent geometry and appearance continuously without discretization and with a small memory footprint, although most require 3D supervision.
- Surface Rendering: DVR and IDR optimize implicit surfaces through differentiable surface rendering but require pixel-accurate object masks for all views.IDR can capture detailed non-Lambertian surfaces by conditioning on viewing direction.
- Volumetric Radiance Fields: NeRF uses volume rendering without mask supervision and achieves strong novel-view synthesis, but its recovered geometry is unsatisfactory.Follow-up methods extract meshes from NeRF density without directly optimizing surfaces.
- Volumetric Radiance Fields: UNISURF differs by directly targeting accurate geometry with a volume-rendering formulation that approaches surface rendering in the limit.
3. Background
Implicit surface models define geometry through occupancy boundaries and render colors at surface intersections, whereas NeRF represents view-dependent radiance with volume densities integrated along rays. Their limitations motivate a unified treatment.
- Implicit Surface Models: Implicit surface models represent geometry with an occupancy field whose decision boundary defines the surface.The surface consists of points where occupancy probability equals 0.5.
- Implicit Surface Models: Surface rendering retrieves the first surface intersection along a ray and evaluates a learned color field there.
- Rendering Limitations: Surface rendering estimates geometry accurately but relies strongly on object-mask supervision because it reasons only about rays intersecting a surface.
- Volumetric Radiance Models: NeRF maps 3D locations and viewing directions to volume density and color, then integrates sampled radiance along rays using alpha blending.
- Volumetric Radiance Models: NeRF avoids object masks during training, but extracting geometry from density requires threshold tuning and can produce artifacts from density ambiguity.
4. Method
UNISURF unifies surface and volume rendering for solid-object reconstruction, using occupancy-based representations, adaptive sampling, and a regularized objective. The formulation supports mask-free optimization and efficient surface-based rendering.
- Unified representation: UNISURF combines volume rendering for coarse structure with surface rendering as an inductive bias for precise solid-object surfaces without mask supervision.The method focuses on solid objects represented by 3D surfaces and view-dependent surface colors.
- Unified representation: The method interprets occupancy rendering as selecting the first occupied, visible sample along each camera ray.The occupancy term identifies the first occupied sample, while the visibility product excludes samples occluded by earlier occupied points.
- Unified representation: UNISURF parameterizes a continuous occupancy field and conditions color on surface normals and geometry features to unify implicit surfaces with volumetric radiance.The model supports both volume and surface rendering within the same formulation.
- Optimization: The objective combines an ℓ1 reconstruction loss with ℓ2 surface regularization that encourages neighboring points to have similar normals.Surface normals are computed from the normalized occupancy-field gradient at surface points.
- Adaptive sampling: Training samples points around the estimated surface, beginning with a large interval and progressively shrinking it through an exponential decay schedule.As the interval approaches zero and the number of samples increases, volume rendering approaches surface rendering.
- Inference: Surface rendering produces similar image results to volume rendering while running twice as fast, and the representation supports mesh extraction with MISE.Inference can use either rendering mode; mesh extraction starts at resolution 64^3 and upsamples in three steps.
5. Experimental Evaluation
UNISURF is evaluated against classical and neural baselines on DTU, BlendedMVS, and SceneNet, with additional rendering and loss ablations. It performs comparably to IDR on DTU without masks and produces plausible, detailed geometry on more complex scenes.
- Experimental setup: The evaluation compares UNISURF with COLMAP, NeRF, and IDR on DTU, and with COLMAP and NeRF on unmasked BlendedMVS and SceneNet scenes.The study also includes an ablation of rendering procedures and the surface regularization term.
- Datasets: DTU contains 49–64 images per scan at 1200 × 1600 resolution, with camera parameters and ground-truth 3D shapes available for evaluation.The dataset includes objects with varied shapes and appearances, including challenging non-lambertian effects.
- Comparison on DTU: UNISURF performs nearly on par with IDR on DTU while avoiding strong mask supervision, whereas NeRF and watertight COLMAP reconstructions have worse Chamfer distance.COLMAP with trim parameter 7 achieves the best Chamfer distance but produces incomplete, non-watertight meshes.
- Comparison on DTU: On DTU qualitative results, UNISURF and IDR produce accurate surfaces with high-quality details, while COLMAP has incomplete geometry and NeRF shows holes and noise artifacts.UNISURF also captures overall scene arrangement and fine geometry such as skull teeth.
- Comparison on BlendedMVS and SceneNet: For complex scenes, UNISURF reconstructs plausible multi-object geometry and more accurate, less noisy surfaces than NeRF on BlendedMVS, while COLMAP shows artifacts in uniformly colored regions.IDR failed to converge on the evaluated unmasked complex scenes, producing degenerate outputs.
- Ablation study: The full unified rendering method performs best in the Buddha ablation, while isolated surface rendering fails to converge and volume-rendering baselines produce bloated shapes with missing details.Removing surface regularization makes surfaces less smooth, particularly in textureless flat regions such as the table.
6. Discussion and Conclusion
UNISURF unifies implicit surfaces and radiance fields to capture high-quality surface geometry from multi-view images without input masks. Its scope is limited by solid-surface assumptions, difficult visual regions, and sparse visibility, motivating stronger priors.
- UNISURF unifies implicit surfaces and radiance fields to capture high-quality geometry from multi-view images without input masks.The formulation is presented as a more general setting for optimizing implicit surfaces.
- The model is limited to representing solid, non-transparent surfaces.
- Overexposed and textureless regions can cause inaccuracies and non-smooth reconstructed surfaces.
- Reconstructions are less accurate in regions that are rarely visible in the images.
- Resolving ambiguities in rarely visible and textureless regions requires a prior beyond the explicit smoothness prior used during optimization.The paper suggests learning a probabilistic neural surface model to capture regularities and uncertainty across objects.