PaperlayerView exact revision on arXiv

Source-linked AI summary

Baking Neural Radiance Fields for Real-Time View Synthesis

Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, Paul Debevec

arXiv:2103.14645v1cs.CVcs.GR

TL;DR

NeRF enables detailed, view-dependent novel-view synthesis, but its hundreds of per-ray MLP queries make real-time rendering impractical. The paper bakes a trained NeRF into a Sparse Neural Radiance Grid that stores sparse scene content while deferring view-dependent computation. SNeRG renders above 30 frames per second on a laptop GPU, averages under 90 MB per scene, and retains fine details and convincing view-dependent effects.

  • Problem

    NeRF’s hundreds of MLP queries per ray make rendering too slow for real-time applications.

  • Method

    The method reformulates NeRF as a deferred architecture and bakes it into a sparse voxel grid containing opacity, diffuse color, and learned feature vectors.

  • Results

    SNeRG renders above 30 frames per second on a laptop GPU, averages less than 90 MB per scene, and retains fine geometric details and convincing view-dependent effects.

  • Takeaways & Limitations

    SNeRG provides a practical commodity-hardware representation for serving and real-time rendering of scenes reconstructed by NeRF.

  • Takeaways & Limitations

    Quantization and compression lower baked-SNeRG image quality relative to the corresponding deferred NeRF, although fine-tuning keeps final quality competitive.

Abstract

from arXiv · show

Neural volumetric representations such as Neural Radiance Fields (NeRF) have emerged as a compelling technique for learning to represent 3D scenes from images with the goal of rendering photorealistic images of the scene from unobserved viewpoints. However, NeRF's computational requirements are prohibitive for real-time applications: rendering views from a trained NeRF requires querying a multilayer perceptron (MLP) hundreds of times per ray. We present a method to train a NeRF, then precompute and store (i.e. "bake") it as a novel representation called a Sparse Neural Radiance Grid (SNeRG) that enables real-time rendering on commodity hardware. To achieve this, we introduce 1) a reformulation of NeRF's architecture, and 2) a sparse voxel grid representation with learned feature vectors. The resulting scene representation retains NeRF's ability to render fine geometric details and view-dependent appearance, is compact (averaging less than 90 MB per scene), and can be rendered in real-time (higher than 30 frames per second on a laptop GPU). Actual screen captures are shown in our video.

1. Introduction

NeRF produces detailed, view-dependent novel views but is too slow for real-time rendering. SNeRG bakes NeRF into a sparse voxel representation while retaining quality and enabling compact, real-time rendering.

  • NeRF represents scenes as continuous volumetric functions but requires hundreds of MLP queries per ray, making rendering slow.
  • Three orders of magnitude faster rendering yields 12 milliseconds per frame on a single GPU.
  • SNeRG stores opacity, diffuse color, and learned feature vectors in a sparse voxel grid, then predicts view-dependent residuals with a lightweight MLP.
  • A deferred NeRF runs the view-dependent MLP once per pixel, while opacity regularization encourages sparsity for lower storage and rendering costs.
  • Less than 90 MB per scene is required on average while retaining fine geometric details and convincing view-dependent effects.

2. Related work

Prior view-synthesis methods trade off rendering speed, optimization, scalability, or appearance quality. SNeRG combines NeRF’s reconstruction quality and view dependence with efficient sparse-volume rendering on commodity hardware.

  • Scene Representations for View Synthesis: Mesh methods support real-time rasterization but struggle to reconstruct fine structures and detailed scene geometry through gradient-based optimization.
  • Scene Representations for View Synthesis: Discretized voxel and multiplane representations support gradient-based optimization but face cubic scaling or limited viewpoint ranges.
  • Scene Representations for View Synthesis: NeRF replaces discretized volumes with a continuous MLP-based volumetric function that represents detailed geometry and realistic appearance.
  • Efficient Representation and Rendering: Efficiency-focused methods reduce MLP evaluations, skip free space, or factorize radiance representations to accelerate neural volumetric rendering.
  • Efficient Representation and Rendering: SNeRG extends deferred neural rendering to volumetric scenes, combining NeRF-like reconstruction quality and view dependence with commodity-hardware speed.

3. Method Overview

The method targets practical NeRF serving under strict latency, storage, and GPU-memory requirements. It exchanges computation for sparse storage while deferring view-dependent effects until rendering.

  • The target is rendering an 800×800 frame in under 30 milliseconds on commodity hardware.
  • The representation should compress to 100 MB or less and fit uncompressed within approximately 4 GB of GPU memory without streaming.
  • A standard NeRF requires about 100 teraflops per 800×800 frame and can take 10 seconds on an NVIDIA RTX 2080.
  • The proposed hybrid approach stores selected content in a sparse 3D structure while deferring view-dependent computation to rendering time.

4. Modifying NeRF for Real-time Rendering

The method reformulates NeRF to reduce view-dependent computation and encourages sparse opacity so the trained representation can be efficiently stored and rendered. It shifts expensive computation from every ray sample toward precomputed voxel-grid data and a single per-pixel network evaluation.

  • Reformulating NeRF: The reformulation limits view-dependent effects to one network evaluation per ray, adds an 8-bit storage bottleneck, and introduces sparsity loss during training.The sparsity loss concentrates the opacity field around scene surfaces.
  • Review of NeRF: NeRF predicts density from 3D position and radiance from both 3D position and viewing direction, rendering pixels by numerically integrating sampled densities and colors along rays.Training minimizes squared error between rendered pixel colors and pixels from posed input images.
  • Motivation: NeRF’s standard rendering requires hundreds of MLP queries for a single ray, creating a space-time tradeoff that favors compact storage but makes rendering extremely slow.The MLP replaces memory lookup with computation at each sampled location; this is useful during training but costly after optimization.
  • Deferred NeRF Architecture: Deferred rendering accumulates diffuse colors and feature vectors along each ray, then feeds them with the ray direction to a small MLP that produces a view-dependent residual.The residual is added to the accumulated diffuse color; the MLP has 2 layers with 16 channels each.
  • Deferred NeRF Architecture: Precomputed diffuse colors and 4-dimensional feature vectors can be stored in a sparse voxel grid, while the view-dependent MLP is evaluated once per pixel instead of once per 3D sample.This preserves the deferred-rendering structure while reducing repeated evaluations along each ray.
  • Opacity Regularization: A Cauchy-loss regularizer penalizes predicted density during training to encourage opacity sparsity, using coarse uniformly distributed samples to avoid uneven application from hierarchical sampling.The experiments use λs = 10^-4 and c = 1/2.

5. Sparse Neural Radiance Grids

SNeRG bakes a trained NeRF into a sparse voxel representation that replaces most MLP evaluations with texture lookups while retaining view-dependent rendering through a lightweight MLP. Sparsity, visibility culling, compression, and fine-tuning jointly target compact storage, fast rendering, and reduced quality loss.

  • SNeRG Data Structure: SNeRG replaces per-sample NeRF MLP evaluations with lookups of precomputed diffuse colors and features in a sparse voxel grid.A lightweight MLP is evaluated once per pixel to produce the view-dependent residual.
  • SNeRG Data Structure: The block-sparse representation stores occupied macroblocks in a 3D texture atlas and uses an indirection grid to skip empty space.Spatially close macroblock content remains nearby in the atlas, supporting efficient rendering.
  • Rendering: Ray marching skips empty macroblocks, fetches atlas values by trilinear interpolation, accumulates diffuse colors and features, and stops when opacity saturates.Features are fetched only where volume density is non-zero, reducing memory bandwidth.
  • Baking: Sparsity loss and visibility culling allocate content primarily around visible scene surfaces, making the representation more compact.Removing either sparsity loss or both sparsity loss and visibility culling produces a much less compact representation.
  • Compression and Fine-tuning: 8-bit quantization and compression reduce storage but lower baked-rendering quality relative to the deferred NeRF; fine-tuning the shading MLP recovers almost all lost accuracy.The texture atlas can be compressed with PNG, JPEG, or H264, with different quality-versus-storage tradeoffs.

6. Implementation Details

The implementation uses JAXNeRF-based training and a WebGL renderer with quantized 3D textures and an uncompressed view-dependence MLP. Visual ablations compare successive view-dependence simplifications, while real-scene examples include reflective objects.

  • Training: The deferred NeRF implementation is based on JAXNeRF and uses positional encoding for positions and viewing directions.Networks are trained for 250k iterations with learning-rate scheduling and gradient clipping.
  • Ablations: Visual ablations show minimal quality differences among JAXNeRF+, Tinyview, Deferred NeRF, and SNeRG, while Diffuse introduces floating alpha artifacts.The approximations successively simplify view-dependent appearance to accelerate rendering.
  • Rendering: The renderer uses Javascript and WebGL, loading the indirection grid and texture atlas into 8-bit 3D textures.The view-dependence MLP is stored uncompressed and implemented in a WebGL shader.
  • Real Scenes: The real 360° garden-spheres scene demonstrates that the real-time method models mirror-like reflective surfaces.The evaluation hardware is a 2019 MacBook Pro with an AMD Radeon Pro 5500M GPU.

7. Experiments

Experiments evaluate SNeRG through ablations and baseline comparisons spanning rendering speed, GPU memory, storage, and image quality. The method achieves competitive quality with substantially faster rendering and compact representations, while sparsity and deferred rendering are key contributors.

  • Evaluation: Experiments measure render-time performance, GPU memory, storage cost, and rendering quality across ablations and competing methods.Quality uses PSNR, SSIM, and LPIPS; performance is evaluated with frames per second and GPU memory.
  • Ablations: 11.9±4.5 ms/frame and 1.73 ± 1.48 GB GPU memory are reported for the full method in the performance ablation.The comparison uses combinations of the view-dependence MLP, sparsity loss, and deferred rendering components.
  • Ablations: 343.6±247.5 ms/frame results when deferred rendering is replaced by per-sample MLP queries, making rendering prohibitively slow.This ablation restores NeRF’s approach of querying an MLP at each sample along a ray.
  • Ablations: After fine tuning, SNeRG remains competitive with its derived neural model despite quality reductions from deferred rendering, a smaller view-dependent network, and compression.Quantization and compression initially cause a significant quality drop, which fine tuning mitigates.
  • Storage: More than 200× storage compactness is achieved with JPEG or H264 compression while sacrificing less than 1dB of PSNR.Removing the sparsity loss significantly increases model size because it no longer concentrates opaque voxels around surfaces.
  • Ablations: Sparsity loss more than doubles rendering speed and halves compressed disk and uncompressed GPU-memory requirements with minimal quality impact.The effect is observed across the reported ablations.
  • Baseline Comparisons: SNeRG quality is comparable to competing methods, while rendering is an order of magnitude faster than Neural Volumes and more than a thousand times faster than NeRF.SNeRG is measured on an 85W laptop GPU, whereas competing methods use substantially more powerful hardware.

8. Conclusion

The paper presents SNeRG as a baked sparse voxel representation for real-time NeRF rendering. It combines precomputed scene geometry with a neural representation for view-dependent appearance and reaches over 30 frames per second on a laptop GPU.

  • Conclusion: SNeRG bakes a trained NeRF into a sparse voxel grid while retaining a neural representation for view-dependent appearance.The grid stores precomputed scene geometry, while neural features preserve view-dependent shading.
  • Conclusion: Over 30 frames per second is achieved on a laptop GPU for typical NeRF scenes by evaluating the view-dependent shading network only on visible scene parts.The stated goal is real-time rendering on commodity graphics hardware.

A. WebGL Implementation Details

The WebGL implementation separates grid data into compact textures and skips empty space during ray marching. Performance is measured on a laptop GPU under controlled browser conditions, with lower grid resolution offering a speed and storage trade-off.

  • WebGL Implementation Details: The renderer loads alpha, RGB, and feature data as separate 8-bit 3D textures, with a low-resolution 8-bit indirection grid.This organization is intended to conserve memory bandwidth.
  • WebGL Implementation Details: Empty macroblocks and zero-alpha voxels are skipped before fetching high-resolution colors and features.For occupied voxels, trilinear interpolation retrieves alpha, colors, and features; empty-space traversal uses only a one-byte alpha query.
  • WebGL Implementation Details: The view-dependence MLP runs only for pixels with non-zero accumulated alpha.Network weights use 32-bit floating-point textures and biases are hard-coded in the GLSL shader.
  • Evaluation Setup: Performance is measured in Chrome on a 2019 MacBook Pro with an 85 watt AMD Radeon Pro 5500M GPU and 8GB of GPU RAM.The protocol closes other applications, connects the laptop to power, and disables browser frame-rate limiting.
  • Resolution Trade-off: Lower-resolution baked grids provide higher rendering speeds and lower storage costs at the expense of a slight quality decrease.The resolution trade-off is evaluated in the voxel-grid ablation.

C.1. Experiments with Changing 3D Resolution

Real-scene experiments examine resolution, compression, quality, and performance for 360° and forward-facing scenes. SNeRG maintains quality close to trained NeRF models while reaching about 30 frames per second and roughly 50 MB storage on the evaluated real 360° scenes.

  • Changing 3D Resolution: Lower-resolution 3D grids increase rendering speed and reduce storage cost, with a slight decrease in rendering quality.This establishes a direct speed, storage, and quality trade-off for SNeRG baking resolution.
  • Real 360° Scenes: About 30 frames per second and about 50 MB storage are achieved on the evaluated real 360° scenes while maintaining quality close to trained NeRF models.The scenes include Flowers, Pine Cone, Toy Car, and Spheres; Spheres contains glossy objects that are difficult to model with diffuse geometry alone.
  • Real Forward-Facing Scenes: Real forward-facing scenes are compared with layered representations because their cameras and viewpoints cover only a limited forward-facing range.The experiments include quality and performance comparisons for these scenes.
  • Implementation Adjustment: Baking can reduce total alpha mass and introduce small semitransparent holes, which are addressed by partially un-premultiplying alpha after ray marching.The adjustment saturates alpha values above 0.66 while preserving soft edges and smooth fall-off.
  • Baselines: JAXNeRF+ serves as the starting architecture for modifications, using more samples per ray to recover a sparser representation concentrated near object surfaces.Baseline runtimes are measured on NVIDIA V100 GPUs, unlike the laptop measurements for SNeRG.

C.5. Experiments with Changing 3D Resolution

Baking the 3D grids at lower resolution increases rendering speed and reduces storage costs, with a slight decrease in rendering quality.

  • Lower-resolution 3D grids achieve higher rendering speeds and lower storage costs.These gains come with a slight decrease in rendering quality.
  • The storage and speed improvements are obtained by reducing the resolution used to bake the grids.
  • The quality trade-off from lower-resolution baking is characterized as slight.

C.6. Per-Scene Quality and Performance Metrics

The experiments report per-scene quality, frame-time, and GPU-memory breakdowns across synthetic and real scene categories, including performance ablations.

  • Per-Scene Quality Metrics: Tables 13–18 provide per-scene PSNR, SSIM, and LPIPS quality metrics for Synthetic 360° and Real Forward-Facing scenes.The passages also identify Real 360° as an experimental scene category.
  • Per-Scene Performance Metrics: Table 19 reports per-scene frame time, while Table 20 reports per-scene GPU memory consumption.
  • Performance Ablations: The performance ablations remove view dependence, remove the sparsity loss, or replace deferred rendering with per-sample MLP queries.
  • Storage Ablation: Table 12 is identified as the storage ablation table.
  • Quality Table Organization: The listed quality tables separately cover PSNR, SSIM, and LPIPS for Synthetic 360° and Real Forward-Facing scenes.
Loading 2103.14645v1…