PaperlayerView exact revision on arXiv

Source-linked AI summary

PlenOctrees for Real-time Rendering of Neural Radiance Fields

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, Angjoo Kanazawa

arXiv:2103.14024v2cs.CVcs.GR

TL;DR

NeRFs render detailed scenes from arbitrary viewpoints but are too slow for real-time applications. The paper distills them into PlenOctrees with spherical-harmonic view dependence and direct octree optimization, achieving 167.68 FPS at 800×800 resolution while retaining comparable or better quality. The representation also supports browser-based rendering, but has substantially higher memory usage and remains less suitable for unbounded or forward-facing scenes.

  • Problem

    NeRF rendering is impractical for interactive use because dense sampling requires costly neural-network queries, including more than 100 million inputs for an 800×800 image.

  • Method

    The method converts NeRFs into PlenOctrees that store density and spherical-harmonic radiance coefficients, then directly fine-tunes the octree.

  • Results

    167.68 FPS is achieved for 800×800 images on an NVIDIA V100 GPU, with quality equal to or better than the original NeRF formulation.

  • Takeaways & Limitations

    PlenOctrees enable real-time, view-dependent NeRF rendering without a deep neural network during test time and support interactive browser rendering on consumer laptops.

  • Takeaways & Limitations

    The octree uses substantially more memory than the original NeRF, averaging 1.93 GB on synthetic scenes and 3.53 GB on Tanks and Temples before compression.

Abstract

from arXiv · show

We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects. Our method can render 800x800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs. We do so without sacrificing quality while preserving the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects. Real-time performance is achieved by pre-tabulating the NeRF into a PlenOctree. In order to preserve view-dependent effects such as specularities, we factorize the appearance via closed-form spherical basis functions. Specifically, we show that it is possible to train NeRFs to predict a spherical harmonic representation of radiance, removing the viewing direction as an input to the neural network. Furthermore, we show that PlenOctrees can be directly optimized to further minimize the reconstruction loss, which leads to equal or better quality compared to competing methods. Moreover, this octree optimization step can be used to reduce the training time, as we no longer need to wait for the NeRF training to converge fully. Our real-time neural rendering approach may potentially enable new applications such as 6-DOF industrial and product visualizations, as well as next generation AR/VR systems. PlenOctrees are amenable to in-browser rendering as well; please visit the project page for the interactive online demo, as well as video and code: https://alexyu.net/plenoctrees

1. Introduction

NeRFs provide high-quality, view-dependent novel-view synthesis but are too slow for interactive rendering. PlenOctrees address this by tabulating NeRF radiance in an octree while preserving quality and accelerating both rendering and training.

  • Motivation: 30 seconds are required to render an 800x800 NeRF image on a high-performance GPU, making interactive applications impractical.Dense sampling and neural-network queries cause the bottleneck.
  • Results: 167.68 FPS is achieved for 800x800 images on an NVIDIA V100 GPU without a deep neural network during test-time rendering.The method preserves arbitrary geometry and view-dependent effects.
  • Method: PlenOctrees pre-sample NeRFs into sparse hierarchical volumes that store density and view-dependent appearance for efficient rendering.Spherical harmonics represent view-dependent effects such as specularities.
  • Results: The approach reports five orders of magnitude faster NeRF rendering without image-quality loss and state-of-the-art-level image quality and speed on standard benchmarks.The experiments use scenes and objects captured from 360° views.
  • Method: NeRF-SH predicts spherical-harmonic coefficients instead of RGB values, enabling direct storage of appearance in PlenOctree leaves.This removes the viewing-direction input from the network.
  • Training: Direct PlenOctree fine-tuning improves quality and permits early NeRF termination, making training substantially faster because later optimization uses no neural networks.The pipeline can stop NeRF training early before converting and optimizing the octree.

2. Related Work

Prior novel-view methods trade among topology flexibility, editing, memory, and rendering speed. PlenOctrees combine octree efficiency with spherical-basis representations for real-time view-dependent rendering, while related neural methods accelerate NeRFs through different factorizations or learned structures.

  • Novel View Synthesis: Mesh representations render efficiently but are difficult to optimize for complex arbitrary topology, whereas image-based methods can be fast and photorealistic but restrict viewing angles or editing.These approaches represent different trade-offs for novel-view synthesis.
  • Volume Representations: Voxel grids and multi-plane images support straightforward optimization and real-time rendering, but naive volumetric representations are memory-bound.Octrees reduce memory and compute for volumetric representations.
  • PlenOctrees: PlenOctrees use octrees with spherical-harmonic coefficients to encode view-dependent radiance for efficient rendering of complex scenes.This combines hierarchical volumetric storage with view-dependent appearance.
  • Coordinate-Based Networks: Coordinate-based neural networks represent spatial properties with multilayer perceptrons queried by coordinates, including occupancy, signed distance, and radiance.NeRF is the radiance-field example in this line of work.
  • NeRF Accelerations: NeRF acceleration methods include learned priors, sparse voxel features, decomposed networks, and related approaches that reduce training or inference cost.Neural Sparse Voxel Fields improve render time by skipping empty regions.

3. Preliminaries

NeRF encodes a continuous radiance field in an MLP and renders pixels by accumulating sampled colors and densities along rays. Its quality comes with substantial inference cost because every sample requires a network query, including many samples in empty space.

  • NeRF Representation: NeRF queries an MLP with position and viewing direction to recover density and RGB color for each spatial sample.The field supports arbitrary novel viewpoints and view-dependent appearance.
  • Volume Rendering: A pixel color is computed by casting a ray and accumulating density-weighted colors from N samples along that ray.Volume rendering integrates contributions from samples through the scene.
  • Training: NeRF uses positional encoding and separate coarse and fine sampling stages to represent high-frequency scene details.These stages improve the representation of fine details.
  • Computational Cost: 192 samples are taken per ray in the NeRF examples, and an 800×800 image requires over 100 million network inputs.Most samples lie in free space and do not contribute to integrated color.
  • Computational Cost: 30 seconds are needed to render one 800×800 frame on an NVIDIA V100 GPU, preventing real-time use.Sparse voxel octrees and precomputed voxel values avoid excess empty-space computation and inference-time network queries.

4. Method

The method converts NeRF-SH into a sparse PlenOctree that stores density and spherical-harmonic coefficients for view-dependent radiance, then directly fine-tunes the fixed tree. This representation supports fast rendering while preserving image quality, and octree optimization also enables earlier termination of NeRF training.

  • PlenOctree representation: PlenOctrees store density and spherical-harmonic coefficients at octree leaves, encoding view-dependent radiance for real-time rendering.The representation supports non-Lambertian effects while avoiding a neural network during rendering.
  • NeRF-SH: NeRF-SH predicts spherical-harmonic coefficients instead of RGB values, eliminating the view-direction input and simplifying conversion into the octree.Conversion evaluates the network on a grid and thresholds voxels before storing coefficients in octree leaves.
  • Sparsity prior: A sparsity prior encourages empty space when both empty and solid-color solutions fit, reducing unobserved geometry that would otherwise occupy voxel space.The final loss combines the sparsity term with the RGB reconstruction loss, weighted by a hyperparameter.
  • Evaluation: The fine-tuned PlenOctree is significantly faster than existing methods while remaining on par in image metrics on Tanks and Temples; conversion without fine-tuning harms quality.Qualitative NeRF-synthetic comparisons report no significant image-quality difference while rendering more than 3500x faster.
  • Conversion: The conversion evaluates density on a grid, filters voxels by visibility, samples 256 points per remaining voxel, and averages values into leaf coefficients.The resulting tree stores density and RGB-channel spherical-harmonic coefficients; the full extraction process takes about 15 minutes.
  • PlenOctree optimization: Direct octree fine-tuning improves reconstruction quality while optimizing about 3 million rays per second, compared with about 9000 for NeRF training.The tree structure remains fixed during optimization, which uses the NeRF loss on the original training images.

5. Results

Across synthetic and real datasets, PlenOctrees achieve competitive or better image quality while delivering real-time rendering and substantially faster training transitions. The experiments also examine quality–speed trade-offs, qualitative detail recovery, and practical interactive operations.

  • 5.1. Experimental Setup: Four PlenOctree variants sweep the accuracy–speed trade-off by changing conversion parameters and tree construction choices.Figure 5 uses a logarithmic time axis; Table 3 reports average metrics across NeRF-synthetic scenes.
  • 5.2. Quality Evaluation: On Tanks and Temples, the method better recovers fine scene details than NeRF while producing otherwise similar results.The reported render time is over 3000× faster.
  • 5.2. Quality Evaluation: PlenOctree quality is competitive in all evaluated cases and better on some metrics, while none of the baselines achieve real-time performance.The comparison covers synthetic and real datasets against NeRF, NSVF, AutoInt, SRN, and Neural Volumes.
  • 5.2. Quality Evaluation: PlenOctree conversion does not perceptually worsen NeRF renderings, and octree optimization can enhance fine details such as text.The NeRF-SH modification itself does not significantly change performance.
  • 5.2. Quality Evaluation: Over 3000× faster than NeRF and at least 30× faster than other compared methods, PlenOctree inference remains best or second best across image-quality metrics.Measurements use a Tesla V100 across the synthetic and Tanks and Temples datasets.
  • 5.4. Indirect Acceleration of NeRF Training: Given a time constraint, stopping NeRF training and transitioning to PlenOctree optimization is almost always preferable.Fine-tuning the octree can yield larger image-quality improvements than continuing NeRF-SH training for an equivalent time.
  • 5.5. Interactive Demos: The interactive viewer supports mesh insertion with occlusion, octree slicing, depth rendering, radiance probing, and inspection of spherical-harmonic components.These operations are demonstrated on the PlenOctree representation.
  • 5.3. Additional Real Scenes: FastNeRF performs reasonably well on additional real scenes despite not being designed for unbounded or forward-facing scenes.For forward-facing scenes, the octree is constructed in NDC coordinates.

6. Discussion

PlenOctrees provide a real-time representation for arbitrary objects and scenes, including browser-based 6-DOF viewing. Their main practical constraints are substantially larger memory footprints and limited optimality for unbounded or forward-facing scenes.

  • Discussion: More than 3000× rendering acceleration and equal or better quality than NeRF make PlenOctrees suitable for real-time rendering of arbitrary objects and scenes.The authors also demonstrate a WebGL in-browser viewer with real-time and 6-DOF rendering on consumer laptops.
  • Discussion: The approach may support virtual online stores in VR by enabling real-time 6-DOF visualization of products with arbitrary complexity and materials.This is presented as a future application enabled by the method.
  • Limitations and Future Work: The full uncompressed octree averages 1.93 GB on NeRF-synthetic and 3.53 GB on Tanks and Temples, exceeding the original NeRF’s compact representation.Compressed lower-resolution online models are approximately 30–120 MB.
  • Limitations and Future Work: Optimal application to unbounded and forward-facing scenes requires further work because their data distribution differs; forward-facing scenes also do not support 6-DOF viewing.The authors suggest MPIs may be more appropriate for forward-facing scenes.
  • Limitations and Future Work: Future extensions target real-time 6-DOF immersive viewing of large-scale and dynamic scenes.These directions are identified as future work rather than demonstrated capabilities in the current system.

A. Additional Results

Additional experiments provide further qualitative comparisons with several baselines and report per-scene quantitative breakdowns across multiple tables.

  • A. Additional Results: Additional results include qualitative comparisons with SRN, Neural Volumes, and NSVF, plus further examples of the proposed method.Per-scene quantitative metrics against all approaches are reported in Tables 5–8.

A.2. Spherical Basis Function Ablation

The ablation compares spherical harmonics with spherical Gaussians and evaluates basis-function choices before and after PlenOctree conversion. SH-16 and SH-25 perform similarly, while SG's slight PSNR advantage disappears after conversion and fine-tuning.

  • Experimental setup: The experiments report average metrics on NeRF-synthetic before and after converting spherical-basis NeRFs to fine-tuned PlenOctrees.The ablation covers basis-function count and spherical Gaussians.
  • Number of SH basis functions: Switching between ℓmax = 3 (SH-16) and ℓmax = 4 (SH-25) makes very little difference in metrics or visual quality.Both the modified NeRF and corresponding PlenOctree are evaluated.
  • Spherical Gaussians: Spherical Gaussians use learnable Gaussian kernels as an alternative spherical basis to spherical harmonics.The SG-25 model uses 25 learnable components with lobe axes and bandwidths.
  • Spherical Gaussians: SG-25 has marginally better PSNR before conversion, but that advantage disappears after PlenOctree conversion and fine-tuning.The comparison is specific to the reported NeRF-synthetic ablation.
  • Spherical Harmonics: Spherical harmonics form a complete basis for functions on the sphere, and real spherical functions can be expressed in that basis.The appendix provides the technical SH formulation for completeness.

B.3.2 Derivation of the Derivatives

This section derives analytic derivatives for the piecewise-constant NeRF volume-rendering model and explains how PlenOctree rendering computes them efficiently. The derivatives cover color and density, including spherical-harmonic colors.

  • Rendering model: The rendered ray color is expressed using segment weights determined by accumulated transmittance and density-dependent absorption.The background color is included as a final term in the rendering model.
  • Color derivative: Color derivatives are straightforward because rendered color is a convex combination of segment colors.For spherical-harmonic colors, the derivative follows by applying the chain rule because basis values are constant across the ray.
  • Density derivative: Density derivatives account for how changing one segment density affects later transmittance and therefore future segment contributions.The derivation uses an indicator for terms whose segment index is greater than the differentiated index.
  • Implementation: The PlenOctree renderer computes the gradient in two rendering passes because density derivatives depend on future weights and colors.The first pass stores accumulated color-related quantities, followed by a prefix subtraction; auxiliary memory remains constant.
  • Implementation: For nonnegative density parameterization, gradients are set to zero when the underlying network output is nonpositive.Multiple color derivatives are combined by summing the density derivatives across colors.

B.4. NeRF-SH Training Details

The NeRF-SH training setup uses a Jax NeRF reimplementation with fixed ray and sample counts, Adam optimization, exponential learning-rate decay, and a 2M-iteration schedule.

  • Training configuration: The model uses batches of 1024 rays with 64 coarse samples and 128 additional fine samples per ray.These settings are used throughout the reported training protocol.
  • Training configuration: Adam optimization starts at a learning rate of 5 × 10^-4 and decays exponentially to 5 × 10^-6.The learning-rate schedule is part of the common training protocol.
  • Training duration: All models are trained for 2M iterations, and training takes around 50 hours to converge.The reported duration refers to the stated convergence setup.

B.5. PlenOctree Optimization Details

After NeRF-SH conversion, PlenOctrees are directly optimized with the NeRF loss using SGD, with dataset-specific learning rates, epoch limits, and early stopping.

  • Optimization procedure: After conversion, PlenOctrees are optimized on the training set with SGD using the NeRF loss and no sparsity loss.The octree is already sparse at this stage.
  • Dataset-specific settings: NeRF-synthetic optimization uses a constant 1 × 10^7 learning rate for at most 80 epochs.Early stopping monitors PSNR during optimization.
  • Dataset-specific settings: Tanks&Temples optimization uses a 1.5 × 10^6 learning rate for at most 40 epochs.Early stopping is also applied for this dataset.
  • Validation: For Tanks&Temples, 10% of the training set is held out as validation data only for PlenOctree optimization.The holdout is specific to the optimization process.
  • Evaluation: The appendix reports per-scene quantitative results for NeRF-synthetic, Tanks&Temples, PlenOctree conversion, and spherical-basis ablations.These results are organized in Tables 5–8.
Loading 2103.14024v2…