PaperlayerView exact revision on arXiv

Source-linked AI summary

Point-NeRF: Point-based Neural Radiance Fields

Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, Ulrich Neumann

arXiv:2201.08845v7cs.CV

TL;DR

NeRF provides high-quality novel-view synthesis but requires slow per-scene fitting, while multi-view stereo enables faster geometry reconstruction. Point-NeRF combines neural point-cloud radiance fields with feed-forward initialization and point optimization, achieving comparable or superior quality with substantially shorter reconstruction times.

  • Problem

    NeRF-based radiance fields require slow per-scene fitting and sample vast empty space, motivating more efficient reconstruction methods.

  • Method

    Point-NeRF represents a continuous volumetric radiance field with neural 3D points, initializes it through feed-forward multi-view stereo inference, and refines it using point growing and pruning.

  • Results

    Point-NeRF achieves rendering quality comparable to or better than NeRF while requiring only 20–40 minutes of per-scene optimization instead of NeRF’s 20+ hours.

  • Takeaways & Limitations

    Point-NeRF provides efficient, high-quality neural scene reconstruction and can improve radiance fields initialized from external point-cloud reconstruction methods.

  • Takeaways & Limitations

    The implementation does not focus on rendering speed and leaves point querying and feature aggregation unoptimized.

Abstract

from arXiv · show

Volumetric neural rendering methods like NeRF generate high-quality view synthesis results but are optimized per-scene leading to prohibitive reconstruction time. On the other hand, deep multi-view stereo methods can quickly reconstruct scene geometry via direct network inference. Point-NeRF combines the advantages of these two approaches by using neural 3D point clouds, with associated neural features, to model a radiance field. Point-NeRF can be rendered efficiently by aggregating neural point features near scene surfaces, in a ray marching-based rendering pipeline. Moreover, Point-NeRF can be initialized via direct inference of a pre-trained deep network to produce a neural point cloud; this point cloud can be finetuned to surpass the visual quality of NeRF with 30X faster training time. Point-NeRF can be combined with other 3D reconstruction methods and handles the errors and outliers in such methods via a novel pruning and growing mechanism. The experiments on the DTU, the NeRF Synthetics , the ScanNet and the Tanks and Temples datasets demonstrate Point-NeRF can surpass the existing methods and achieve the state-of-the-art results.

1. Introduction

Point-NeRF addresses NeRF’s slow per-scene reconstruction with a neural point-based radiance field initialized by feed-forward inference. It supports efficient optimization, handles imperfect point clouds, and achieves strong novel-view synthesis results.

  • NeRF’s global per-scene MLP fitting and empty-space sampling cause long reconstruction times.
  • Point-NeRF uses 3D neural points to represent a continuous volumetric radiance field.
  • A cost-volume MVS network predicts depth and a CNN extracts image features to generate an initial neural point cloud.
  • Point-NeRF supports external point clouds and improves holes and outliers through point growing and pruning during optimization.
  • Experiments across DTU, NeRF Synthetic, Tanks & Temples, and ScanNet report state-of-the-art novel-view synthesis against prior methods.

2. Related Work

Point-NeRF combines localized neural point representations with volumetric rendering to improve efficiency and scene-surface adaptation. Its cross-scene initialization and 3D rendering distinguish it from global MLP, voxel, rasterization, and image-feature approaches.

  • Point-NeRF localizes neural features on fine-grained points, improving scene-surface adaptation over global MLP representations.
  • Unlike voxel-based methods, Point-NeRF adapts to actual surfaces and directly predicts initial neural point features across scenes.
  • Point clouds from MVS or depth sensors are commonly converted to meshes, but meshing can introduce errors and require image-based rendering.
  • Previous neural point renderers splat features and use 2D CNNs, whereas Point-NeRF uses 3D volume rendering.
  • Point-NeRF avoids per-scene MLP training through initialization by a pretrained network that generalizes across scenes.
  • Compared with PixelNeRF and IBRNet, Point-NeRF uses 3D neural points near surfaces rather than sampling features at every ray point.

3. Point-NeRF Representation

Point-NeRF represents a continuous radiance field with neural points whose local features are aggregated around each shading location. Differentiable ray marching then estimates density and view-dependent radiance while avoiding empty-space computation.

  • Point-NeRF is designed for efficient reconstruction and rendering using a point-based radiance field.
  • A radiance field assigns volume density and view-dependent radiance at each 3D location, which Point-NeRF computes from a neural point cloud.
  • Each neural point has a location, local scene feature, and confidence indicating its likelihood of lying near an actual scene surface.
  • At a shading location, Point-NeRF queries K nearby points and regresses density and radiance from their positions, features, and confidences.
  • Relative positions let the local MLP produce features at the shading location while remaining invariant to point translation.
  • Inverse-distance weighting gives closer points greater influence when aggregating features for radiance regression.
  • Density is formed by first regressing per-point densities and then applying inverse-distance-based weighting.
  • Differentiable ray marching shades only near the neural point cloud, accumulates radiance using density, and enables end-to-end rendering optimization.

4. Point-NeRF Reconstruction

Point-NeRF reconstructs radiance fields by generating neural point clouds from multiple views, then optimizing them with differentiable rendering and point growing and pruning. This workflow initializes scenes efficiently while addressing holes and outliers in input geometry.

  • Pipeline overview: A cross-scene network directly generates an initial point-based radiance field, which is then optimized per scene with point growing and pruning.The workflow separates feed-forward initialization from scene-specific refinement.
  • Initial field generation: Depth prediction from a cost-volume MVS network is unprojected into 3D points, while a 2D CNN supplies aligned per-point neural features.Point clouds from multiple viewpoints are combined into the neural point cloud.
  • Initial field generation: End-to-end training with a rendering loss produces reasonable initial radiance fields and initializes representation MLPs with weights that reduce per-scene fitting time.The same framework can also provide initial neural features for point clouds reconstructed by methods such as COLMAP.
  • Point pruning: Pruning removes outlier points and reduces rendering artifacts.The confidence-based criterion links low confidence to low local volume density and empty regions.
  • Point growing: Point growing adds neural points near point-cloud boundaries when ray-marched locations have high opacity and are sufficiently distant from existing points.The method expands sparse initial clouds to cover missing scene regions, including clouds with only 1000 initial points.

5. Implementation details

The implementation uses multi-scale image features and trains the reconstruction pipeline with rendering supervision. Per-scene refinement combines rendering and sparsity losses with periodic point growing and pruning.

  • Network details: The per-point neural feature is a 59-channel vector formed from multi-scale image features and viewing directions.Features from three resolutions provide 56 channels, with viewing-direction information added for view-dependent effects.
  • Training: The full pipeline is trained on DTU using an MVSNet-based depth network pretrained with ground-truth depth and subsequent end-to-end training.The training and testing split matches PixelNeRF and MVSNeRF.
  • Training: The feed-forward network generates a point cloud from three input views in 0.2s.The full pipeline uses an Adam optimizer with an initial learning rate of 5e−4 and an L2 rendering loss.
  • Optimization: Per-scene optimization combines rendering and sparsity losses as Lopt = Lrender + aLsparse.The sparsity-loss weight is a = 2e−3.
  • Optimization: Point growing and pruning are performed every 10K iterations during final reconstruction.These operations are part of the per-scene optimization procedure.

6. Experiments

Experiments show Point-NeRF provides efficient, high-quality novel-view synthesis across DTU, NeRF Synthetic, Tanks & Temples, and ScanNet, while pruning and growing improve robustness to imperfect point clouds.

  • DTU testing set: 10k fine-tuning iterations achieve the best DTU SSIM and LPIPS, while IBRNet has slightly better PSNR but takes 5× longer to fine-tune.Point-NeRF also produces high-quality results after only 1k iterations and comparable visual quality after 2 minutes.
  • NeRF Synthetic dataset: 20k iterations outperform IBRNet's converged results on PSNR, SSIM, and LIP-IPS, with better geometry and texture details on NeRF Synthetic.This comparison uses a dataset with a 360° camera distribution, substantially wider than DTU's.
  • NeRF Synthetic dataset: 20k iterations achieve quantitatively close results to NeRF trained for 200k iterations, while requiring 40 minutes instead of NeRF's 20+ hours.At convergence, Point-NeRF can surpass NeRF, NSVF, and other comparison methods, with point growing recovering thin rope details in the Ship scene.
  • Point-based rendering: Point-NeRF's volumetric rendering produces photo-realistic results, whereas the compared rasterization and 2D CNN point-based method produces blurry renderings.The comparison uses the same point cloud generated by the Point-NeRF MVSNet-based network.
  • Additional experiments: Point-NeRF converts COLMAP point clouds into high-quality radiance fields despite holes and noise, while improving their geometry through point growing and pruning.The method is presented as compatible with point clouds reconstructed by other techniques.
  • Additional experiments: Starting from only 1000 sparse points, point growing progressively fills the scene surface and produces high-quality novel views under RGB supervision.The ablation also shows significant improvements with pruning and growing for both generated and COLMAP point clouds.

7. Conclusion

The conclusion presents Point-NeRF as a neural point-cloud radiance-field approach that combines efficient feed-forward initialization with high-quality rendering and robust point-cloud optimization.

  • Conclusion: Point-NeRF models volumetric radiance fields with neural point clouds and directly reconstructs their initialization from input images via network inference.The initialized field can be efficiently fine-tuned for a specific scene.
  • Conclusion: 20–40 min per-scene optimization yields rendering quality comparable to or better than NeRF, which requires 20+ hours of training.Growing and pruning improve results and robustness across different point-cloud quality levels.

A. Ablation Studies on Point Features Initialization

The ablation study evaluates whether extracted image features provide a better initialization for neural point features than random initialization.

  • Feature initialization: Image-feature initialization produces better converged performance than random feature initialization after 200K iterations.The experiment is conducted on the NeRF Synthetic dataset using Kaiming initialization as the random baseline.
  • Feature initialization: Image features also make the neural point representation converge faster during optimization.The passage reports faster convergence in addition to better performance after convergence.

B. Per-scene Breakdown Results of the DTU Dataset

On DTU, Point-NeRF achieves strong per-scene reconstruction quality, with the best SSIM and LPIPS scores on most scenes while IBRNet attains better PSNR.

  • Point-NeRF has the best SSIM scores in most DTU scenes, reflecting faithful scene geometry reconstruction.
  • Point-NeRF also achieves the best LPIPS for most DTU scenes, producing more visually authentic results.
  • IBRNet achieves better PSNR by combining colors from source views during shading.
  • Point-NeRF is more temporally consistent because local radiance and geometry are stored consistently at each neural point.

C. Per-scene Breakdown Results of the NeRF Synthetic Dataset

On NeRF Synthetic, Point-NeRF leads most per-scene metrics and converges substantially faster than NeRF, while COLMAP initialization remains competitive with NeRF.

  • Point-NeRF achieves the best PSNRs, SSIMs, and LPIPSs on most NeRF Synthetic scenes.
  • Point-NeRF outperforms state-of-the-art methods with a big margin on the NeRF Synthetic scenes.
  • Point-NeRF initialized with COLMAP points is on par with NeRF.
  • 20K iterations of Point-NeRF match NeRF results at 500K iterations, demonstrating fast convergence.
  • Point pruning and growing improve geometry reconstruction and radiance-field quality even from unideal initial points.

D. Evaluation on Large-scale 3D Scenes (ScanNet).

Point-NeRF generalizes from DTU training to large-scale ScanNet and Tanks and Temples scenes, where it is reported to outperform prior methods across metrics.

  • The model generalizes from object-focused DTU training to large-scale ScanNet scenes.Two ScanNet scenes are evaluated, using RGB and depth images with one of every five frames for training.
  • Point-NeRF outperforms previous studies in all metrics on the evaluated ScanNet scenes.
  • The ScanNet qualitative evaluation contains five generated test frames each for scenes 101 and 241.
  • Point-NeRF also outperforms state-of-the-art models in all metrics on five Tanks and Temples scenes.

G. Networks Architectures

The architecture combines multi-view stereo and image features with grid-based neural-point queries, while restricting shading to occupied regions for efficiency; implementation and data-handling details define its scope.

  • Network architecture: A cost-volume CNN predicts depth, which is unprojected into 3D to initialize neural points and their radiance-field representation.
  • Network architecture: Radiance computation aggregates features from K neighboring neural points and outputs radiance color r and density σ.
  • Network architecture: A 2D CNN extracts multi-scale image features that are unprojected onto 3D points to provide per-point features.
  • Efficient querying: Grid-point indices identify occupied regions and nearby grids, enabling efficient retrieval of neural-point neighbors for ray marching.
  • Efficient querying: Shading uses eight nearest neural-point neighbors and skips empty space, accelerating point queries and radiance computation.
  • Limitations: Rendering is naturally 3X faster than NeRF, although point querying and feature aggregation were not optimized for fast rendering.
  • External reconstruction: Point-NeRF can initialize from external point clouds such as COLMAP, whose holes and outliers are addressed through pruning and growing.
  • External reconstruction: For Tanks and Temples, MVSNet points are filtered in pure background regions because the dataset lacks an alpha channel.
Loading 2201.08845v7…