PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction

Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, Lingjie Liu

arXiv:2212.05231v3cs.CVcs.GR

TL;DR

NeuS2 addresses the high training cost of high-quality neural surface reconstruction, especially for dynamic sequences. It combines hash-encoded SDFs, efficient second-order derivatives, progressive training, and dynamic-frame transformation and incremental learning, achieving fast reconstruction while retaining high quality. The method is also subject to limitations in cross-frame surface correspondence and per-frame parameter storage.

  • Problem

    NeuS provides high-quality neural surface reconstruction but takes about 8 hours to train a static object, limiting its applicability to dynamic sequences.

  • Method

    NeuS2 combines multi-resolution hash-encoded SDFs, CUDA-efficient second-order derivatives, progressive training, and incremental dynamic reconstruction with global transformation prediction.

  • Results

    NeuS2 achieves fast, accurate multi-view reconstruction for both static and dynamic scenes, with reported runtime performance up to 20 seconds per dynamic frame.

  • Takeaways & Limitations

    NeuS2 provides a fast neural surface reconstruction approach for static objects and long dynamic sequences with large movements and deformations.

  • Takeaways & Limitations

    NeuS2 does not provide dense surface correspondences across dynamic frames and requires saving 25M network parameters for each frame.

Abstract

from arXiv · show

Recent methods for neural surface representation and rendering, for example NeuS, have demonstrated the remarkably high-quality reconstruction of static scenes. However, the training of NeuS takes an extremely long time (8 hours), which makes it almost impossible to apply them to dynamic scenes with thousands of frames. We propose a fast neural surface reconstruction approach, called NeuS2, which achieves two orders of magnitude improvement in terms of acceleration without compromising reconstruction quality. To accelerate the training process, we parameterize a neural surface representation by multi-resolution hash encodings and present a novel lightweight calculation of second-order derivatives tailored to our networks to leverage CUDA parallelism, achieving a factor two speed up. To further stabilize and expedite training, a progressive learning strategy is proposed to optimize multi-resolution hash encodings from coarse to fine. We extend our method for fast training of dynamic scenes, with a proposed incremental training strategy and a novel global transformation prediction component, which allow our method to handle challenging long sequences with large movements and deformations. Our experiments on various datasets demonstrate that NeuS2 significantly outperforms the state-of-the-arts in both surface reconstruction accuracy and training speed for both static and dynamic scenes. The code is available at our website: https://vcai.mpi-inf.mpg.de/projects/NeuS2/ .

1. Introduction

NeuS2 targets the long training time of high-quality neural surface reconstruction by combining hash-encoded SDFs with CUDA-efficient derivatives and progressive optimization. It extends these ideas to dynamic sequences through incremental learning and global transformation prediction.

  • NeuS delivers high-quality reconstruction but requires about 8 hours to train a static object, limiting its use for long dynamic sequences.
  • NeuS2 uses multi-resolution hash tables of learnable feature vectors to parameterize the neural SDF and accelerate reconstruction.
  • NeuS2 reconstructs static objects in minutes and moving-object sequences in up to 20 seconds per frame.
  • A lightweight second-order derivative formulation tailored to ReLU-based MLPs enables efficient GPU parallelization through CUDA.
  • Progressive training optimizes multi-resolution hash encodings from coarse to fine to improve convergence speed and stability.
  • Incremental learning and global transformation prediction support efficient reconstruction of long dynamic sequences with large movements and deformations.

2. Related Work

Related work spans classical stereo, model-based dynamic reconstruction, and neural implicit representations. NeuS2 is positioned as a model-free, RGB-only method that combines fast training with high-quality surface reconstruction for static and dynamic scenes.

  • Multi-view Stereo: Depth-based stereo depends on point correspondences, while voxel-based methods recover occupancy and color using photometric consistency.
  • Classical Multi-view 4D Reconstruction: Precomputed deformable models are widely used for multi-view 4D reconstruction, whereas NeuS2 does not rely on a precomputed model and handles topology changes.
  • Neural Implicit Representations: NeRF produces high-quality novel views but lacks surface constraints for extracting high-quality geometry, while NeuS represents surfaces as signed distance fields.
  • Neural Implicit Representations: NeuS requires about 8 hours to train, whereas NeuS2 is reported as 100 times faster and reaches 20 seconds per frame for dynamic reconstruction.
  • Neural Implicit Representations: Voxel-grid methods accelerate training but generally inherit NeRF’s volume-density geometry representation, unlike NeuS2’s surface-oriented representation.
  • Neural Implicit Representations: Several dynamic methods focus on novel view synthesis rather than high-quality surface geometry and appearance models.
  • Concurrent Work: NeuS2 is reported as over 3x faster than Voxurf and achieves better geometry quality than Voxurf’s reported results.
  • Concurrent Work: Neuralangelo uses multi-resolution hash grids for neural surface reconstruction but is reported as 100x slower than NeuS2 and is not designed for dynamic reconstruction.

3. Background

The background describes NeuS-style SDF rendering and Instant-NGP hash encoding, then identifies the difficulty of combining high-quality surface constraints with fast CUDA-based computation. For dynamic scenes, temporal reuse and large motion are the central challenges.

  • NeuS: NeuS represents geometry with an SDF and appearance with a radiance field, extracting the surface as the SDF’s zero-level set.
  • NeuS: NeuS renders rays by accumulating SDF-based densities and colors at sampled points, but training takes about 8 hours on a single GPU.
  • InstantNGP: Instant-NGP maps multi-resolution voxel grids to fixed-size hash tables of learnable feature vectors and concatenates interpolated level encodings.
  • InstantNGP: CUDA implementation and GPU parallelism substantially improve Instant-NGP runtime, although its geometry accuracy does not reach NeuS quality.
  • Challenges: Combining NeuS and Instant-NGP is nontrivial because high-quality surface learning requires the Eikonal constraint alongside CUDA-based computation.
  • Challenges: Dynamic reconstruction must exploit temporal information for acceleration while handling long sequences with large movements and deformations.

4. Static Neural Surface Reconstruction

NeuS2 reconstructs static neural surfaces with hash-encoded signed distance fields, shallow networks, volume rendering, and CUDA-efficient second-order derivatives. Progressive coarse-to-fine encoding training further improves convergence and reconstruction quality.

  • Hash-encoded SDF: NeuS2 represents each 3D position with multi-resolution hash encodings and uses a shallow MLP to predict its SDF and geometry feature.The network concatenates the position with its hash encoding, supporting more stable geometry learning.
  • Network architecture: The SDF network feeds the SDF value, geometry feature, position, normal, and viewing direction into a color network that predicts point color.Normals are computed as the SDF gradient with respect to position.
  • Training objective: NeuS2 combines color supervision from rendered pixels with an Eikonal regularizer for the learned signed distance field.The final loss is L = L_color + β L_eikonal, with the Eikonal term regularizing sampled-point normals toward unit norm.
  • Efficient derivatives: Because normals enter the color network and Eikonal loss, NeuS2 directly computes second-order derivatives for hash-table and SDF-network parameters using simplified formulas.The color network itself does not require second-order gradients of its parameters because it only takes the normal as input.
  • Efficient derivatives: CUDA implementation of the tailored ReLU-based derivative computation is more efficient than PyTorch's automatic computation-graph approach for second-order backpropagation.The formulation exploits simplifications in ReLU-based MLPs to reduce computational overhead.
  • Progressive training: Progressive training gradually increases hash-encoding bandwidth from coarse to fine, balancing faster training at low resolution with capacity for high-frequency details.NeuS2 initializes λ at 2 and increases it by 1 every 2.5% of training steps.

5. Dynamic Neural Surface Reconstruction

NeuS2 extends fast neural surface reconstruction to dynamic scenes by combining incremental training with global transformation prediction, enabling efficient reconstruction under large movements and deformations. Its evaluations report strong geometry and appearance results across synthetic and real scenes.

  • Incremental Training: NeuS2 learns the first frame from scratch and fine-tunes subsequent frames from the preceding frame’s hash-grid representation.This exploits shared geometry and appearance between consecutive frames to accelerate convergence.
  • Global Transformation Prediction: Global transformation prediction addresses local-minimum failures and surface holes when adjacent-frame movement is relatively large.The method roughly transforms the target SDF before further learning.
  • Global Transformation Prediction: Global transformation prediction permits a small hash grid to represent dynamic sequences with large movement, improving reconstruction accuracy while reducing memory cost.The grid models only a small portion of the entire scene after transformation.
  • Dynamic Representation: The combined design models large global movements separately from smaller adjacent-frame deformations, rather than learning every frame’s large motion to a common canonical space.The authors identify these two components as enabling handling of challenging movements and deformations.
  • Evaluation: On DTU, NeuS2 outperforms other baselines for geometry reconstruction by Chamfer Distance and matches Instant-NGP in novel-view-synthesis PSNR.The comparison reports both geometry and appearance outcomes.
  • Evaluation: On synthetic scenes, NeuS2 achieves much better appearance and geometry reconstruction than D-NeRF and TiNeuVox, while on real scenes it outperforms them across all metrics.Chamfer Distance is omitted for the Lion sequence because ground-truth geometry is unavailable.

6. Experiments

Experiments show that NeuS2 reconstructs static and dynamic scenes with high quality while substantially reducing training time. Its second-order derivative computation and dynamic-scene components improve efficiency and reconstruction quality.

  • Static Scene Reconstruction: NeuS2 outperforms baseline methods on static geometry reconstruction while matching Instant-NGP in novel view synthesis at the same 5-minute training time.Geometry is evaluated with Chamfer Distance and rendering with PSNR.
  • Static Scene Reconstruction: NeuS2 produces detailed, noise-free geometry and detailed renderings, while NeuS is smoother, Instant-NGP is noisy, and Instant-NSR exhibits artifacts.The comparison attributes Instant-NGP’s noise to missing surface constraints and Instant-NSR’s artifacts to finite-difference second derivatives.
  • Dynamic Scene Reconstruction: 20 seconds per frame enables NeuS2 to complete synthetic dynamic sequences in under 1 hour, compared with about 20 hours for D-NeRF.The first frame takes 40 seconds, or 80 seconds for the Lego sequence, followed by 20 seconds per subsequent frame.
  • Dynamic Scene Reconstruction: NeuS2 outperforms D-NeRF and TiNeuVox on synthetic dynamic novel view synthesis and geometry reconstruction.The comparison covers Lego, Lion, and human-character sequences.
  • Dynamic Scene Reconstruction: On real scenes, NeuS2 produces photo-realistic renderings and detailed geometry, whereas D-NeRF and TiNeuVox produce blurred renderings and inaccurate geometry.The real-scene evaluation uses Dynacap sequences with large and non-rigid movement.
  • Ablation Studies: NeuS2’s CUDA second-order derivative computation is faster than PyTorch across tested MLP layer numbers and batch sizes.Ablations also show that Global Transformation Prediction and Progressive Training improve full-model reconstruction quality.

7. Conclusion

NeuS2 concludes with a fast neural implicit reconstruction method for static and dynamic scenes, combining hash encodings, efficient derivatives, progressive training, and incremental dynamic learning. The authors also identify missing cross-frame surface correspondences and per-frame parameter storage as limitations.

  • Limitations: NeuS2 does not provide dense surface correspondences across dynamic frames.The authors suggest deforming a mesh template to fit the learned neural surfaces as a possible future direction.
  • Ablations: Global Transformation Prediction and Progressive Training both improve the overall reconstruction results in ablations.The ablation study evaluates these design choices quantitatively and qualitatively.
  • Limitations: Each dynamic frame currently requires storing 25M network parameters.The authors identify parameter compression as future work.
  • Conclusion: NeuS2 integrates multi-resolution hash encodings and tailored second-order derivatives to achieve fast, accurate reconstruction of static and dynamic scenes.Progressive training further improves convergence.
  • Conclusion: Incremental learning with Global Transformation Prediction reconstructs long sequences with large movements efficiently and stably.The contribution description gives 2000 frames as an example.

Supplementary Material

The supplementary material provides derivations, proofs, dataset information, additional quantitative and qualitative results, and implementation details.

  • Supplementary Material: Supplementary material includes derivations of Eqs. 5 and 6 and the proof of Theorem 1.It also documents datasets, additional results, and implementation details.
  • Supplementary Material: Additional quantitative, qualitative, and video results are provided in the supplementary material.The supplementary material also contains implementation details.

A. Derivation of Equation 5 and Equation 6

The derivation develops second-order derivatives for ReLU-based MLPs with respect to hash-table parameters and inputs, using chain-rule expressions and layerwise definitions.

  • Hash-table derivatives: The derivation targets second-order derivatives of hash-table parameters because the surface constraint and rendering formulation require them.The calculation is designed to be lightweight and memory-efficient for parallel implementation.
  • Network definitions: Definition 2 represents a ReLU-based MLP as layered matrix transformations with ReLU activations, input x, and output y.The layer matrices H_l and layer index l define the network structure.
  • First-order derivatives: Applying the chain rule expresses the input derivative as a product of layer matrices and activation-gradient factors.The intermediate quantities P_l^j and S_l^i capture forward and backward layer contributions.
  • Layer derivatives: The derivative with respect to an intermediate layer factors into the corresponding backward term and forward activation term.The resulting expression uses S_l^i and P_l^j to represent the two derivative components.
  • Second-order derivatives: For ReLU-based MLPs, the second derivative with respect to the network input is zero under the piecewise-linear formulation.The derivation propagates second-order terms through intermediate activations before reaching this result.

C. Dataset

The experiments use DTU scenes for static reconstruction and synthetic sequences with varied motions and deformations for dynamic reconstruction.

  • Static Scene Reconstruction: The static evaluation uses 15 DTU scenes spanning varied materials, appearances, and geometries, including non-Lambertian surfaces and fine structures.Each scene contains 49 or 64 images at 1600 × 1200 resolution, with training and testing splits following NeuS.
  • Dynamic Synthetic Scene Reconstruction: The dynamic synthetic evaluation includes Lego, Lion, and human sequences with different motions and deformations.The sequences contain 150, 177, and 100 frames respectively.
  • Dynamic Synthetic Scene Reconstruction: The synthetic sequences use multi-view training and testing cameras, with image resolutions of 400 × 400 for Lego and 512 × 512 for the human sequence.Lego uses 40 training and 40 test views, while the human sequence uses 48 training and 12 test views.

D. Additional Result

Additional experiments examine qualitative and quantitative reconstruction results, baseline comparisons, architecture details, and robustness to transformation-prediction errors.

  • Additional Comparisons: The supplementary evaluation compares NeuS2 with Voxurf and Instant-NGP on DTU and synthetic scenes.The Lion Chamfer Distance is omitted because its ground-truth geometry is unavailable.
  • Transformation Robustness: Reconstruction remains robust to minor global transformation errors, with performance dropping only when predicted rotation and translation are very inaccurate.The experiment adds noise to SMPL rotation R and translation T predictions.
  • Static Scene Reconstruction: The DTU results show that NeuS2 outperforms baselines on geometry reconstruction and matches Instant-NGP on novel-view PSNR.The reported geometry metric is Chamfer Distance, while novel-view synthesis uses PSNR.
  • Static Scene Reconstruction: Qualitative DTU comparisons report high rendering quality, complex-texture results comparable to Instant-NGP, and detailed geometry without noise.The comparison includes NeuS and Instant-NSR for rendering and all listed baselines for geometry.
  • Network Architecture: The NeuS2 architecture uses a 14-level multi-resolution hash grid, a 1-layer 64-unit SDF MLP, and a 2-layer 64-unit RGB MLP.The hash-grid resolutions range from 16 to 2048.

E.3. Training Details

Training details combine NeuS-style unbiased volume rendering with ray-marching acceleration and short static or incremental dynamic optimization schedules.

  • Unbiased Volume Rendering: NeuS2 converts the signed distance field into a logistic density field and uses differentiable volume rendering to learn geometry and radiance from images.The logistic density is derived from the Sigmoid function through a learnable parameter s.
  • Unbiased Volume Rendering: The rendered ray color accumulates sample-point densities and colors using transmittance and density weights along each camera ray.Samples are placed as p(t_i) = o + t_i v, where o is the camera center and v is the viewing direction.
  • Ray Marching Strategy: An occupancy grid accelerates ray marching by preventing sampling in empty voxels and is periodically updated from predicted SDF values.This strategy follows Instant-NGP’s occupancy-grid acceleration approach.
  • Training Schedule: Static models train for 15k iterations in around 5 minutes, while dynamic training initializes the first frame from scratch and fine-tunes subsequent frames.Dynamic training uses 2k iterations for the first frame, or 4k for Lego, followed by 1.1k iterations per subsequent frame.
Loading 2212.05231v3…