Source-linked AI summary
Neuralangelo: High-Fidelity Neural Surface Reconstruction
Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H. Taylor, Mathias Unberath, Ming-Yu Liu, Chen-Hsuan Lin
TL;DR
Existing methods struggle to recover detailed structures in real-world scenes from multi-view images. Neuralangelo combines multi-resolution hash-grid SDF representations with neural surface rendering, using numerical gradients and coarse-to-fine optimization, and recovers dense structures with high fidelity across object-centric and large-scale scenes.
Problem
Current methods struggle to recover detailed structures of real-world scenes from multi-view images.
Method
Neuralangelo combines multi-resolution hash encoding with neural SDF surface rendering, numerical gradients, and coarse-to-fine optimization.
Results
Neuralangelo recovers dense scene structures with extremely high fidelity across object-centric and large-scale indoor and outdoor scenes from RGB videos.
Takeaways & Limitations
The approach enables detailed large-scale scene reconstruction from RGB video captures without requiring auxiliary depth inputs.
Abstract
from arXiv · showhide
Neural surface reconstruction has been shown to be powerful for recovering dense 3D surfaces via image-based neural rendering. However, current methods struggle to recover detailed structures of real-world scenes. To address the issue, we present Neuralangelo, which combines the representation power of multi-resolution 3D hash grids with neural surface rendering. Two key ingredients enable our approach: (1) numerical gradients for computing higher-order derivatives as a smoothing operation and (2) coarse-to-fine optimization on the hash grids controlling different levels of details. Even without auxiliary inputs such as depth, Neuralangelo can effectively recover dense 3D surface structures from multi-view images with fidelity significantly surpassing previous methods, enabling detailed large-scale scene reconstruction from RGB video captures.
1. Introduction
Neuralangelo targets the limited detail recovery of existing neural surface reconstruction methods by combining multi-resolution hash encoding with neural SDF rendering. Numerical gradients and coarse-to-fine optimization unlock detailed, high-fidelity reconstruction without auxiliary inputs.
- Motivation: 3D surface reconstruction recovers dense geometric scene structures from multi-view images for applications including digital twins, environment mapping, and immersive 3D assets.Monocular RGB photogrammetry is especially attractive because ubiquitous mobile devices can support casual real-world capture.
- Limitations of Existing Methods: Classical multi-view stereo produces noisy or missing surfaces when observations contain homogeneous colors, repetitive textures, or strong color variations.These ambiguities limit reliable reconstruction from image correspondences.
- Neural Surface Reconstruction: Neural surface methods use continuous MLP-based implicit functions and neural volume rendering to interpolate surfaces between observed image evidence.Occupancy fields and signed distance functions are representative scene representations.
- Scalable Representations: Current neural methods do not scale reconstruction fidelity effectively with MLP capacity, motivating scalable multi-resolution hash encodings with lightweight MLPs.Instant NGP increases neural-field representation power while keeping memory growth loglinear in resolution.
- Neuralangelo: Neuralangelo incorporates multi-resolution hash encoding into neural SDF reconstruction and optimizes it from multi-view images through neural surface rendering.The framework is designed for high-fidelity surface reconstruction.
- Key Techniques: Numerical gradients for higher-order derivatives and coarse-to-fine optimization across detail levels are the two techniques that improve hash-encoded surface reconstruction.Experiments report significant improvements over previous methods in reconstruction accuracy and view synthesis quality.
2. Related work
Prior reconstruction methods face brittleness from photometric assumptions, noisy point clouds, limited cost-volume resolution, or heuristic geometry extraction. Neural surface methods provide more direct surfaces, while Neuralangelo uses hash encodings without auxiliary inputs.
- Classical Reconstruction: Classical volumetric photogrammetry relies on strict cross-view color constancy, which often fails under autoexposure and non-Lambertian materials.Relaxing this assumption is important for realistic 3D reconstruction.
- Point Clouds and Cost Volumes: Point-cloud-based pipelines inherit missing or noisy surfaces from multi-view stereo, while learned cost volumes remain limited by their resolution.These limitations prevent reliable recovery of fine geometric details.
- Neural Radiance Fields: NeRF supports photorealistic view synthesis but does not directly define geometry, so heuristic density thresholding can yield noisy or inaccurate surfaces.Direct surface representations are therefore preferred for photogrammetric reconstruction.
- Neural Surface Reconstruction: Occupancy grids and SDFs provide better-defined surfaces while retaining neural volume-rendering view synthesis capabilities.Neural implicit functions reparameterize these representations into volume density for rendering.
- Extensions and Auxiliary Inputs: Some follow-up methods trade surface fidelity for real-time operation, use auxiliary information, or remain bounded by sparse point-cloud quality.These approaches include patch warping, point-cloud supervision, depth, and segmentation.
- Neuralangelo: Neuralangelo builds on hash encodings to recover surfaces without the auxiliary inputs used by several prior methods.Its approach differs from concurrent coarse-to-fine work by using hierarchical hash grids rather than a displacement network.
3. Approach
Neuralangelo combines multi-resolution hash encoding with neural SDF rendering for dense surface reconstruction. Numerical gradients address hash-grid locality, while coarse-to-fine activation and smoothing support stable recovery of progressively finer details.
- Neural surface representation: Neuralangelo encodes sampled 3D positions with multi-resolution hash features, feeds them to SDF and color MLPs, and renders images using an SDF-based formulation.The hash features are formed by trilinear interpolation at multiple grid resolutions and concatenation.
- Neural surface representation: Density-based representations do not clearly define surfaces, whereas SDF geometry uses the zero-level set and can be optimized through volume rendering.The SDF-based formulation computes opacity from SDF values for rendering.
- Numerical gradient computation: Analytical hash-encoding gradients are discontinuous across grid cells, so eikonal-loss updates affect only locally sampled hash entries.This locality prevents neighboring cells spanning continuous surfaces from being jointly optimized for coherent normals.
- Numerical gradient computation: Numerical gradients sample neighboring SDF locations so multiple grid cells participate in surface-normal computation and receive updates simultaneously.With step sizes larger than a hash-grid cell, the numerical gradient acts as a smoothing operation on the analytical gradient.
- Progressive optimization: Neuralangelo progressively activates coarse-to-fine hash grids because activating all levels initially can cause fine grids to lose geometric details during coarse optimization.Coarse-to-fine optimization shapes the loss landscape and progressively reconstructs levels of detail.
4. Experiments
Experiments evaluate Neuralangelo on object-centric and large-scale scenes, showing strong reconstruction and image-synthesis quality without auxiliary inputs. Ablations further show that numerical gradients, progressive hash activation, curvature regularization, and topology warmup jointly improve surface quality and detail recovery.
- Datasets and evaluation: Experiments use 15 DTU object-centric scenes and 6 Tanks and Temples indoor/outdoor scenes, evaluated with Chamfer distance, F1 score, and PSNR.DTU images come from a monocular RGB camera with structured-light ground truth; Tanks and Temples uses monocular RGB images with LiDAR ground truth.
- DTU Benchmark: Neuralangelo achieves the lowest average Chamfer distance and highest average PSNR on DTU, despite not leading in every individual scene.The result is reported without using auxiliary inputs.
- Ablations: Numerical gradients improve coarse surface smoothness but can smooth details, whereas combining them with progressive hash activation produces smooth surfaces and fine details.Analytical-gradient variants often produce noisy or artifact-laden coarse surfaces.
- Tanks and Temples Benchmark: On Tanks and Temples, Neuralangelo achieves the highest average PSNR and best average F1 score, recovering intricate details better than NeuS, NeuralWarp, and COLMAP.COLMAP surfaces are sensitive to sparse-point-cloud outliers, while NeuralWarp can predict background surfaces that worsen F1 scores.
- Level of Details: Finer hash resolutions recover structures missed at coarse levels, while sufficiently fine resolutions are required to predict flat continuous surfaces.The observed behavior motivates numerical-gradient updates that back-propagate beyond local grid cells.
- Ablations: Curvature regularization removes surface noise, while topology warmup helps form concave regions that curvature regularization otherwise makes difficult to represent.Without curvature regularization, surfaces tend to develop undesirable sharp transitions.
5. Conclusion
Neuralangelo combines multi-resolution hash encoding with neural SDF rendering to reconstruct detailed surfaces from RGB videos. Its numerical-gradient and coarse-to-fine strategy supports high-fidelity reconstruction across object-centric and large-scale scenes, while long training remains a current efficiency limitation.
- Neuralangelo is a photogrammetric neural surface reconstruction approach that models scenes as SDFs using multi-resolution hash encoding.
- Numerical gradients for higher-order derivatives and coarse-to-fine optimization are the method's central techniques.
- The method recovers dense scene structures with extremely high fidelity in object-centric and large-scale indoor/outdoor scenes from RGB videos.
- Neuralangelo currently uses long training iterations because random pixel sampling does not track sampling statistics and errors.
A. Additional Hyper-parameters
Neuralangelo uses a fixed training and optimization configuration, including 500k iterations, AdamW, learning-rate scheduling, and regularization weights.
- A. Additional Hyper-parameters: Training runs for 500k iterations with AdamW optimization and weight decay of 10^-2.The learning rate is initialized at 1 × 10^-3, warmed up for 5k iterations, and decayed by a factor of 10 at 300k and 400k iterations.
- A. Additional Hyper-parameters: The region of interest is assumed to lie inside a unit sphere.
- A. Additional Hyper-parameters: Inactive hash-grid resolutions use zero feature vectors, while the eikonal and curvature regularization settings are explicitly weighted.The eikonal weight is 0.1, and the curvature regularization strength uses linear warmup.
B. Additional In-the-wild Results
Additional in-the-wild reconstructions use consumer-drone videos from NVIDIA HQ Park and Johns Hopkins University, with camera geometry recovered by COLMAP.
- The in-the-wild scenes are captured by a consumer drone, and COLMAP recovers the camera intrinsics and poses.
C. Additional Tanks and Temples Results
Additional Tanks and Temples results show that Neuralangelo reconstructs surfaces and synthesizes images competitively without relying on noisy COLMAP point clouds.
- Surface reconstruction: Neuralangelo achieves the best surface reconstruction quality and performs best on average for image synthesis on Tanks and Temples.
- Surface reconstruction: Geo-NeuS performs worse than NeuS and Neuralangelo in most Tanks and Temples scenes.The authors attribute this to noisy COLMAP point clouds degrading reconstruction.
- RGB image synthesis: Neuralangelo produces high-fidelity renderings with recovered details on buildings and objects compared with NeuS.PSNR is evaluated on temporally subsampled video frames and foreground objects.
D. Additional DTU Results
Additional DTU results show smoother surfaces, sharper details, and higher-fidelity renderings, while revealing weaker performance on simple or highly reflective objects.
- Surface reconstruction: Neuralangelo reconstructs smoother surfaces in several DTU scans and sharper details such as a pumpkin vine and statue face.
- Surface reconstruction: Neuralangelo misses button structures and eyes on highly reflective Scan 69, recovering fewer details than NeuS.This limitation is consistent with the reported weakness of multi-resolution hash encoding on reflective surfaces.
- RGB image synthesis: Neuralangelo generates high-fidelity RGB renderings with minute details compared with NeuS on DTU.
- RGB image synthesis: Neuralangelo recovers detailed textures while NeuS produces overly smoothed images in the RGB synthesis comparison.
- Evaluation: Foreground masks are used to remove background regions for proper DTU evaluation, but the IDR masks omit part of Scan 83's object.The authors re-annotate the masks to fix this issue.
E. Additional Ablations
Additional ablations examine appearance modeling, curvature regularization, numerical gradients, and computation, identifying benefits alongside explicit trade-offs.
- Color network: Per-image appearance embedding greatly reduces floating objects used to explain exposure variation across frames.
- Curvature regularization: Decaying curvature regularization as finer hash grids activate preserves details better.The regularization is reduced as step size ϵ decreases to avoid smoothing fine structures.
- Gradient comparison: Numerical and analytical gradients produce nearly identical surface normals at the end of optimization when ϵ is sufficiently small.
- Color network: The decomposed color network uses view-invariant albedo and view-dependent shading branches, with the rendered image formed from their sum.The sigmoid function normalizes predictions to the range 0 to 1.
- Color network: The decomposed color network's shading branch explains view-dependent effects, but flat surfaces may be carved away because reflection computation is unstable.
- Computation time: 1.2 times slowdown in training occurs with numerical gradients versus analytical gradients, while the method remains faster than NeuS.Inference for both Neuralangelo gradient variants is more than 2 times faster than NeuS at 128^3 surface extraction resolution.
F. Derivation of Frequency Encoding
The frequency-encoding analysis explains why Fourier features avoid hash-grid gradient locality and shows that coarse-to-fine optimization still improves reconstruction.
- Gradient locality: Analytical derivatives of multi-resolution hash encoding suffer from gradient locality, whereas Fourier frequency encoding does not.Hash-grid locality restricts eikonal-loss backpropagation to locally sampled entries.
- Frequency encoding: Fourier frequency encoding represents position with sine and cosine features at frequency 2^l.
- Color decomposition: The intrinsic color decomposition uses albedo and shading images, but the implemented design lacks smoothness in homogeneous-color regions.
- Derivatives: Its derivative remains continuous across space and retains position information, enabling second-order derivatives for curvature regularization.
- Coarse-to-fine optimization: 0.84 to 0.79 Chamfer distance is achieved when coarse-to-fine optimization is applied with Fourier encoding.The result supports annealing different frequencies through a varying step size for more robust optimization.