PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Sparse Voxel Fields

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, Christian Theobalt

arXiv:2007.11571v2cs.CVcs.GRcs.LG

TL;DR

Photo-realistic free-viewpoint rendering is difficult because detailed geometry and appearance are hard to acquire, while existing neural methods can be blurry and computationally expensive. NSVF combines voxel-bounded implicit fields with a sparse voxel octree learned from posed 2D images, skipping empty space during rendering. It is typically over 10 times faster than NeRF while achieving higher quality, and supports scene editing, composition, and several challenging rendering tasks.

  • Problem

    Photo-realistic free-viewpoint rendering is difficult because detailed scene geometry and material models are often infeasible to acquire, while existing neural methods can be blurry and require time-consuming ray marching.

  • Method

    NSVF combines voxel-bounded implicit fields in a sparse voxel octree with progressive differentiable ray-marching learning from posed 2D images.

  • Results

    NSVF is typically over 10 times faster than NeRF at inference time while achieving higher quality results across challenging rendering tasks.

  • Takeaways & Limitations

    NSVF’s explicit sparse voxel representation supports efficient rendering and can be applied to scene editing and composition.

  • Takeaways & Limitations

    Applying the method to images containing identifiable individuals requires appropriate privacy-preserving steps.

Abstract

from arXiv · show

Photo-realistic free-viewpoint rendering of real-world scenes using classical computer graphics techniques is challenging, because it requires the difficult step of capturing detailed appearance and geometry models. Recent studies have demonstrated promising results by learning scene representations that implicitly encode both geometry and appearance without 3D supervision. However, existing approaches in practice often show blurry renderings caused by the limited network capacity or the difficulty in finding accurate intersections of camera rays with the scene geometry. Synthesizing high-resolution imagery from these representations often requires time-consuming optical ray marching. In this work, we introduce Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering. NSVF defines a set of voxel-bounded implicit fields organized in a sparse voxel octree to model local properties in each cell. We progressively learn the underlying voxel structures with a differentiable ray-marching operation from only a set of posed RGB images. With the sparse voxel octree structure, rendering novel views can be accelerated by skipping the voxels containing no relevant scene content. Our method is typically over 10 times faster than the state-of-the-art (namely, NeRF(Mildenhall et al., 2020)) at inference time while achieving higher quality results. Furthermore, by utilizing an explicit sparse voxel representation, our method can easily be applied to scene editing and scene composition. We also demonstrate several challenging tasks, including multi-scene learning, free-viewpoint rendering of a moving human, and large-scale scene rendering. Code and data are available at our website: https://github.com/facebookresearch/NSVF.

1 Introduction

Neural scene representations address difficult real-world free-viewpoint rendering by learning geometry and appearance from 2D observations. NSVF combines voxel-bounded implicit fields with sparse voxel structures for efficient, high-quality rendering and progressive learning.

  • Real-world free-viewpoint rendering is challenging because acquiring detailed scene geometry and material models is often infeasible.
  • Existing neural representations can produce blurry renderings, require time-consuming ray marching, and complicate scene editing or recomposition.
  • NSVF organizes voxel-bounded implicit fields in a sparse voxel octree instead of modeling the entire space with one implicit function.
  • The method is typically over 10 times faster than NeRF at inference time while achieving higher quality results.
  • NSVF learns voxel embeddings that encode local properties for high-quality rendering.
  • NSVF uses sparse voxels to achieve efficient rendering and progressively learns the structure through differentiable ray marching from posed 2D images.

2 Background

Neural rendering represents scene geometry and appearance implicitly and trains through differentiable rendering from 2D images. Existing surface and volume methods face accuracy or efficiency challenges, especially when sampling along camera rays.

  • Neural rendering learns a function mapping spatial locations to feature representations that describe local scene geometry and appearance.
  • A neural scene function predicts color and density from spatial location and viewing direction, which are used to render camera rays.
  • To encourage multiview consistency, density depends only on spatial position while color depends on both position and viewing direction.
  • Surface Rendering: Surface-rendering methods model density as a Dirac function at the camera ray’s intersection with scene geometry.
  • Volume Rendering: Volume-rendering methods densely sample camera rays and accumulate sampled colors and densities into a 2D image.
  • Around 30 seconds are required for NeRF to render an 800 × 800 image, illustrating the inefficiency of evaluating many ray samples.

3 Neural Sparse Voxel Fields

NSVF combines voxel-bounded implicit fields with a sparse voxel octree, then renders by intersecting rays with occupied voxels and marching within them. Progressive pruning and refinement learn this structure from posed images while reducing computation in empty space.

  • 3 Neural Sparse Voxel Fields: NSVF represents scenes as voxel-bounded implicit fields organized in a sparse voxel octree rather than one field over the entire space.Each voxel has vertex embeddings that are aggregated to represent query points before prediction by an implicit network.
  • 3.1 Voxel-bounded Implicit Fields: Voxel embeddings aggregate region-specific information such as geometry, materials, and colors, easing subsequent field learning and supporting high-quality rendering.
  • 3.2 Volume Rendering: NSVF renders rays in two stages: testing ray-voxel intersections and marching through intersected voxels while sampling and accumulating color and density.Axis-aligned bounding-box tests efficiently identify intersected voxels, especially within the hierarchical octree.
  • 3.2 Volume Rendering: Sparse-voxel sampling avoids repeatedly evaluating empty space, unlike uniform sampling across the whole scene and the additional network used for NeRF importance sampling.
  • 3.3 Learning: The differentiable rendering process enables end-to-end optimization from target posed images without 3D supervision.Training minimizes a rendering loss over sampled camera rays with a beta-distribution regularizer.
  • 3.3 Learning: Self-pruning removes non-essential voxels using predicted density, progressively adapting voxelization and computational resources to important scene regions.Pruning is performed periodically after coarse geometry emerges; voxel and ray-marching sizes are then progressively halved.

4 Experiments

NSVF is evaluated across diverse synthetic, real, indoor, dynamic, multi-scene, editing, composition, and ablation settings. It consistently delivers high-quality rendering while improving efficiency and supporting flexible scene manipulation.

  • NSVF is evaluated on multi-scene learning, dynamic and large-scale indoor rendering, scene editing and composition, and feature and progressive-training ablations.
  • The experiments use Synthetic-NeRF, Synthetic-NSVF, BlendedMVS, Tanks & Temples, ScanNet, and Maria Sequence datasets with three neural-rendering baselines.The baselines are SRN, NV, and NeRF.
  • Quality Comparison: NSVF produces photo-realistic renderings on scenes with complex geometry, thin structures, and lighting effects.Qualitative comparisons report sharper results than SRN, NV, and NeRF.
  • Quality Comparison: NSVF significantly outperforms the three baselines across all four datasets and all reported rendering-quality metrics.Table 1 evaluates PSNR, SSIM, and LPIPS on test images averaged across scenes.
  • Speed Comparison: NSVF renders approximately 10–20 times faster than NeRF and approaches the rendering speed of SRN and NV.Its rendering time varies with foreground-to-background ratio and intersected voxels, while other methods use nearly constant fixed-step sampling.
  • Storage Comparison: NSVF uses 3.2–16MB for network weights, compared with approximately 5MB for NeRF.NSVF storage varies with the number of used voxels, reported as 10–100K.
  • Rendering of Indoor Scenes & Dynamic Scenes: NSVF supports rendering dynamic Maria Sequence and ScanNet indoor scenes, outperforming all baselines in the reported quantitative comparisons.The dynamic-scene adaptation uses a hypernetwork, while ScanNet initialization uses point-cloud-derived voxels.
  • Scene Editing and Scene Composition: The learned multi-object model supports scene composition by duplicating and moving voxels and editing by adjusting sparse-voxel presence.These operations can be rendered without additional overhead.

5 Related Work

Related work spans neural rendering and novel-view synthesis with and without 3D inputs. Neural implicit representations offer continuous, high-resolution modeling but differ in supervision, geometric explicitness, and rendering quality.

  • Neural rendering replaces or augments traditional graphics rendering with neural networks.
  • Novel View Synthesis with 3D inputs: Methods using 3D inputs render explicit or neural textures over geometric proxies before translating coarse outputs into high-quality images.
  • Novel View Synthesis without 3D inputs: Methods without 3D inputs learn scene representations from 2D images, but early approaches produced coarse renderings or modeled geometry less explicitly than NSVF.
  • Neural Implicit Representations: Neural implicit representations are continuous and can provide high spatial resolution, although most prior methods require 3D supervision for geometry learning.

6 Conclusion

NSVF is presented as a hybrid neural scene representation for fast, high-quality free-viewpoint rendering. Experiments report faster and better-quality rendering than NeRF, alongside editing, composition, and challenging-scene demonstrations.

  • NSVF is a hybrid neural scene representation for fast and high-quality free-viewpoint rendering.
  • NSVF is typically over 10 times faster than NeRF while achieving better quality.
  • NSVF supports scene editing and composition and is demonstrated on multi-scene learning, moving humans, and large-scale scene rendering.

7 Broader Impact

NSVF allocates representational capacity to relevant scene regions, supporting detailed large-scale rendering, faster rendering, and more convenient editing and compositing. Its ability to reconstruct identifiable humans from 2D images creates a privacy consideration.

  • NSVF allocates network capacity to relevant scene regions, enabling higher-detail representations of large-scale scenes and higher visual quality.
  • The representation enables faster rendering and more convenient scene editing and compositing.
  • Reconstructing and re-rendering humans from 2D images may require privacy-preserving steps for identifiable individuals.

A Additional Details of the Method

NSVF renders rays by intersecting sparse voxels, sampling points inside them, predicting color and density, and accumulating their contributions. The method also reports transparency and expected depth, while progressive training refines voxelization and sampling.

  • Rendering: The algorithm initializes transparency, color, and expected depth before processing ray-voxel intersections and sampled points.
  • Rendering: Ray marching uses a step size and threshold, processing samples only while transparency exceeds the threshold and samples lie inside intersected voxels.
  • Rendering: Samples contribute color and depth through density-derived alpha, while transparency is updated multiplicatively along the ray.
  • Rendering: After ray processing, background color and maximum depth are weighted by the remaining transparency before returning color, depth, and transparency.
  • Rendering: For each camera ray, NSVF finds intersected sparse voxels, samples points inside them, predicts color and density, and accumulates the results.

B.1 Datasets

The experiments span synthetic, blended, real, indoor, and dynamic-scene datasets to test single-scene, multi-scene, and moving-human reconstruction.

  • Datasets: Experiments cover five datasets: Synthetic-NeRF, Synthetic-NSVF, BlendedMVS, Tanks & Temples, and ScanNet.The first four support scene rendering evaluations, while ScanNet targets large-scale indoor scenes.
  • Datasets: Synthetic-NeRF contains eight path-traced objects with 100 training views and 200 test views at 800 × 800 pixels.
  • Datasets: Synthetic-NSVF adds eight 800 × 800 objects with more complex geometry and lighting effects.
  • Datasets: BlendedMVS contributes four realistically lit synthetic objects, while Tanks & Temples contributes five real scenes.BlendedMVS images are 768 × 576; Tanks & Temples images are 1920 × 1080.
  • Datasets: ScanNet uses two real indoor scenes, with RGB-D images split into 20% training and the remainder testing at 640 × 480 resolution.
  • Multi-scene and dynamic data: Multi-scene experiments train across Synthetic-NeRF and Synthetic-NSVF, while Maria Sequence contains 200 frames of a moving female rendered from 50 training viewpoints.The sequence also includes 50 additional circular-trajectory test views at 1024 × 1024 pixels.

B.2 Implementation Details

NSVF combines voxel embeddings, positional encoding, and a shared MLP within a sparse-octree architecture, then uses progressive training and ray-aware evaluation procedures.

  • Architecture: Each voxel vertex receives a learnable embedding, whose trilinearly interpolated eight-vertex aggregate is positionally encoded before MLP prediction.The default embedding dimension is 32 and the maximum positional-encoding frequency is L = 6.
  • Architecture: The architecture uses a shared MLP to predict density and color from voxel-interpolated features, with approximately 0.5M parameters excluding voxel embeddings.
  • Training and inference: Training samples 2048 rays per image with biased ray selection, Adam optimization, an initial learning rate of 0.001, and step size τ = l/8.
  • Training and inference: Voxels are pruned every 2500 steps, while voxel and step sizes are progressively halved at 5k, 25k, and 75k iterations.Models train for 100k–150k iterations and use ϵ = 0.01 for inference-time early termination.
  • Multi-scene learning: For multiple objects, NSVF either uses separate embedding matrices with a shared MLP or shared voxel representations conditioned through hypernetworks.
  • Evaluation: NSVF achieves higher quality and is typically over 10 times faster than NeRF at inference across challenging scene tasks.The evaluations include multi-object, dynamic, and indoor scenes.

C.3 Additional Analysis

Additional analyses examine voxel resolution, geometry, termination, progressive training, indoor and dynamic scenes, and the editability of learned sparse representations.

  • Voxel representation: Smaller voxel sizes improve rendering quality, indicating that progressively increasing model capacity helps represent finer details.
  • Geometry and scene types: NSVF avoids some geometry failures observed in surface rendering and achieves the best performance on ScanNet and Maria Sequence comparisons.
  • Rendering behavior: The model naturally supports zooming in and out, with Figure 11 reporting no visible artifacts.
  • Rendering efficiency: An early-termination threshold of ϵ = 0.01 provides the best trade-off between quality and rendering speed on Wineholder.
  • Training strategy: Progressive training is faster, easier to train, more space-efficient, and higher quality than one-round final-resolution training on Wineholder.
  • Indoor scenes: For ScanNet, point clouds initialize voxels and an additional depth loss uses ground-truth depth; NSVF achieves the best reported performance.
  • Editing and composition: Learned sparse voxels can be translated, cloned, or removed before rendering, and multiple objects can be used for scene composition.

D Limitations and Future Work

The authors identify limitations involving background complexity, insufficient or biased training views, and camera-pose requirements.

  • Limitations: NSVF assumes a simple constant background and cannot handle scenes whose backgrounds vary across viewpoints.
  • Limitations: Because color and density are learned as a black-box function of location and viewing direction, insufficient or biased training data can produce severe artifacts.The paper cites incorrect refraction on a glass bottle as an example.
  • Limitations: The learning paradigm requires known camera poses and lacks a mechanism for unavoidable real-world camera-calibration errors.Accurately registering poses is more difficult for single-view images of multiple objects.
Loading 2007.11571v2…