PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction

Cheng Sun, Min Sun, Hwann-Tzong Chen

arXiv:2111.11215v2cs.CV

TL;DR

NeRF-quality radiance-field reconstruction is limited by lengthy per-scene training. The paper directly optimizes voxel grids with post-activation interpolation and geometry priors, achieving NeRF-comparable quality while reducing training to 15 minutes from scratch on one GPU.

  • Problem

    NeRF achieves strong novel-view synthesis quality but requires lengthy training, typically several hours per scene or a day of pretraining.

  • Method

    The method directly optimizes a density voxel grid, uses a feature grid with a shallow MLP for view-dependent color, and adds post-activation interpolation plus geometry priors.

  • Results

    Two orders of magnitude faster training reduces 10−20 hours to 15 minutes on one NVIDIA RTX 2080 Ti GPU while achieving NeRF-comparable quality.

  • Takeaways & Limitations

    Direct voxel-grid optimization enables fast per-scene reconstruction from scratch without cross-scene pretraining, while retaining NeRF-comparable quality.

  • Takeaways & Limitations

    The method does not address unbounded or forward-facing scenes.

Abstract

from arXiv · show

We present a super-fast convergence approach to reconstructing the per-scene radiance field from a set of images that capture the scene with known poses. This task, which is often applied to novel view synthesis, is recently revolutionized by Neural Radiance Field (NeRF) for its state-of-the-art quality and flexibility. However, NeRF and its variants require a lengthy training time ranging from hours to days for a single scene. In contrast, our approach achieves NeRF-comparable quality and converges rapidly from scratch in less than 15 minutes with a single GPU. We adopt a representation consisting of a density voxel grid for scene geometry and a feature voxel grid with a shallow network for complex view-dependent appearance. Modeling with explicit and discretized volume representations is not new, but we propose two simple yet non-trivial techniques that contribute to fast convergence speed and high-quality output. First, we introduce the post-activation interpolation on voxel density, which is capable of producing sharp surfaces in lower grid resolution. Second, direct voxel density optimization is prone to suboptimal geometry solutions, so we robustify the optimization process by imposing several priors. Finally, evaluation on five inward-facing benchmarks shows that our method matches, if not surpasses, NeRF's quality, yet it only takes about 15 minutes to train from scratch for a new scene.

1. Introduction

The paper targets NeRF’s lengthy per-scene training and inefficient rendering by directly optimizing voxel-based scene geometry. It combines post-activation interpolation and geometry priors to achieve fast convergence with NeRF-comparable quality.

  • Motivation: NeRF provides state-of-the-art novel-view synthesis but typically requires several hours per scene or a day of pretraining.Its rendering inefficiency also limits some applications.
  • Method: The method directly models volume density with a dense voxel grid and uses a feature grid with a shallow MLP for view-dependent colors.This representation is trained from scratch for each scene without converting a trained implicit model.
  • Method: Two priors reduce suboptimal geometry: near-zero initial opacity avoids near-plane bias, while lower learning rates for rarely viewed voxels limit redundant allocations.These priors address cloud-like free-space solutions during direct density optimization.
  • Method: Post-activation interpolates density before applying activation functions, enabling sharper surfaces at lower grid resolution than conventional interpolation.The method uses fewer voxels while retaining NeRF-comparable quality.
  • Results: Two orders of magnitude faster training reduces NeRF’s 10−20 hours to 15 minutes on a single NVIDIA RTX 2080 Ti GPU.The reported convergence speed is accompanied by NeRF-comparable visual quality and approximately 45× faster rendering.
  • Results: The method achieves NeRF-comparable quality with about 160^3 voxels, whereas prior voxel methods use 512^3 to 1300^3 resolutions.It also does not require cross-scene pretraining.

2. Related work

Related work spans image-, mesh-, CNN-, implicit-, and hybrid volumetric representations for novel-view synthesis. The paper positions direct voxel-grid optimization as a route to fast rendering and convergence without conversion, pretraining, or external depth information.

  • Representations for novel view synthesis: Traditional novel-view representations include light fields, layered depth images, meshes, and CNN-predicted multiplane or voxel grids, each with stated capture, depth, optimization, or scene-setting constraints.The paper uses gradient descent to optimize voxel grids directly rather than predicting grid values with neural networks.
  • Neural radiance fields: NeRF uses coordinate-based MLPs as implicit continuous volumetric representations and offers strong quality, flexibility, and extensibility.This contrasts with explicit discretized representations such as voxel grids and multiplane images.
  • Hybrid volumetric representations: Hybrid volumetric methods condition coordinate-based MLPs on local grid features to combine implicit representations with fast-rendering grid structures.The paper uses a hybrid representation for view-dependent color.
  • Fast NeRF rendering: Fast-rendering NeRF variants use octrees, tiny MLPs, or explicit volumes, but prior methods often retain long training or require conversion from trained implicit models.The paper attributes these limitations to topological structures or missing optimization priors.
  • Fast NeRF convergence: Fast-convergence methods commonly rely on generalizable pretraining or external MVS depth, while the proposed method does not.The paper reports NeRF-quality reconstruction in 15 minutes per scene under its evaluated setup.

3. Preliminaries

NeRF represents scenes implicitly with MLPs that predict density and view-dependent color, then renders pixels by accumulating sampled ray values. Training minimizes photometric mean squared error between rendered and observed colors.

  • NeRF maps each queried 3D position and viewing direction to density and view-dependent color using MLPs.
  • NeRF samples ordered points along each camera ray and accumulates their queried densities and colors into a pixel color through volume rendering.
  • The rendered color is optimized against observed pixel colors using photometric mean squared error over sampled rays.
  • A toy grid-cell comparison motivates post-activation because it recovers a sharp linear surface where pre-activation and in-activation produce smooth results.

4. Post-activated density voxel grid

The method uses explicit voxel grids and places nonlinear density activations after trilinear interpolation. This post-activation ordering enables sharp surfaces and sharper low-resolution image boundaries than alternative orderings.

  • Voxel grids explicitly store density, color, or feature modalities and support efficient queries through interpolation.
  • Density grids store raw scalar densities that are mapped to nonnegative values by a shifted softplus activation.
  • Post-activation applies nonlinear density activations after trilinear interpolation, unlike pre-activation and in-activation orderings.
  • In binary-image fitting, post-activation produces sharp boundaries at low grid resolution and outperforms the other orderings across grid resolutions.
  • A single grid cell with post-activation can recover a sharp linear boundary, whereas pre-activation and in-activation produce only smooth results.

5. Fast and direct voxel grid optimization

The approach searches coarse geometry directly in a density voxel grid, then reconstructs fine detail and view-dependent effects in a restricted region. Initialization, view-count learning rates, staged grids, and free-space skipping support optimization and efficiency.

  • Overview: The pipeline first searches coarse geometry and then reconstructs fine detail, including view-dependent effects.
  • 5.1. Coarse geometry searching: The coarse representation uses post-activated density and view-invariant color voxel grids queried by interpolation.
  • 5.1. Coarse geometry searching: A tightly enclosing camera-frustum bounding box allocates the coarse voxel grid, with voxel dimensions derived from the expected voxel count.
  • 5.1. Coarse geometry searching: Low-density initialization keeps sampled points visible early in training and avoids geometry biased toward camera near planes.
  • 5.1. Coarse geometry searching: View-count-based learning rates favor voxels visible to more training views, while the coarse objective combines color reconstruction with background entropy regularization.
  • 5.2. Fine detail reconstruction: The fine stage freezes optimized coarse geometry, allocates a higher-resolution density grid, and uses a feature grid with a shallow MLP for view-dependent color.
  • 5.2. Fine detail reconstruction: Progressive scaling doubles fine-grid resolution at checkpoints, while free-space skipping removes known free-space and low-alpha samples during training and testing.
  • 5.2. Fine detail reconstruction: Fine-stage sampling restricts rays to box intersections and uses the same training losses as the coarse stage with smaller regularization weights.

6. Experiments

Experiments evaluate the method on five inward-facing datasets, comparing reconstruction quality, training and rendering speed, qualitative results, and the proposed techniques. The method reaches NeRF-comparable or better quality while substantially reducing optimization time.

  • Evaluation setup: Five inward-facing datasets are used for novel-view synthesis evaluation, including synthetic and real-world scenes with held-out test views.The datasets include Synthetic-NeRF, Synthetic-NSVF, BlendedMVS, and Tanks&Temples among the reported benchmarks.
  • Quantitative comparisons: 15 minutes per scene replaces many hours or days of optimization while maintaining good PSNR.Training-time comparisons use reported training times and GPU specifications from previous works.
  • Quantitative comparisons: The method exceeds original NeRF and JaxNeRF on the four datasets under all reported metrics and remains comparable to most recent methods.A larger 256^3 fine grid further improves all metrics and produces more comparable results to JaxNeRF+ and Mip-NeRF.
  • Quantitative comparisons: 45× faster rendering than NeRF is achieved at comparable visual quality.The reported comparison is 0.64 seconds versus 29 seconds per 800 × 800 image.
  • Qualitative comparison: Qualitative comparisons show fewer blocking or background artifacts, better pot recovery, and better color-tone recovery, although ship details are blurrier.The comparisons use challenging portions of ficus, ship, and real-world Ignatius scenes.
  • Ablation studies: Post-activation improves reconstruction with a 160^3 fine grid, while low-density initialization and view-count-based learning rates improve geometry and final PSNR.Without low-density initialization, quality drops severely across scenes; the effective initialization range remains scene-dependent.

7. Conclusion

The method directly optimizes voxel grids to obtain NeRF-comparable quality with 15-minute per-scene convergence. Its demonstrated scope is limited to inward-facing scenes, excluding unbounded and forward-facing settings.

  • Conclusion: Direct voxel-grid optimization achieves NeRF-comparable quality while reducing per-scene training from many hours to 15 minutes.The conclusion frames fast convergence as the method’s central result.
  • Scope: The method is not evaluated on unbounded or forward-facing scenes.The authors describe extending the approach to those scenarios as future work.

H. Additional implementation details

Implementation uses generally shared coarse- and fine-stage voxel-grid settings, half-voxel ray steps, Adam optimization, and progressive learning-rate decay.

  • Grid setup: The coarse stage uses 100^3 voxels and initializes activated alpha to 10^-6.These settings are generally shared across scenes.
  • Grid setup: The fine stage uses 160^3 voxels, alpha initialization 0.01, a 12-channel feature grid, and a two-hidden-layer 128-channel MLP.The higher fine-stage alpha reflects skipping query points in known free space.
  • Sampling: Ray-sampling steps are set to half the corresponding voxel sizes: δ(c) = 0.5s(c) and δ(f) = 0.5s(f).The same relative step-size rule is applied in the coarse and fine stages.
  • Optimization: Adam optimizes the representations with batches of 8,192 rays for 10k coarse and 20k fine iterations.Base learning rates are 0.1 for voxel grids and 10^-3 for the MLP.

I. Additional ablation experiments

Additional ablations characterize quality–efficiency trade-offs and validate progressive scaling, free-space skipping, hybrid color modeling, and auxiliary losses.

  • Voxel count: More voxels improve quality at the cost of computation and training time, while coarse-stage voxel count affects final results through fine-stage geometry.The coarse representation provides geometry on which the fine stage relies.
  • Sampling step size: Finer sampling improves rendering quality but increases computation and training time.The step size is defined relative to voxel size in both stages.
  • Progressive scaling: Progressively scaling fine-stage grid resolution improves training efficiency with slightly better rendering quality.The comparison is reported for fine-stage training.
  • Free-space skipping: Free-space skipping avoids expensive fine-density and view-dependent-color queries for points classified as known free space.Without skipping at both stages, the fine stage runs out of memory.
  • View-dependent colors: The hybrid feature-grid and shallow-MLP representation balances view-dependent color quality against computation, with larger models improving quality at higher cost.Using only the shallow MLP worsens outputs, while using only the voxel grid degrades rendering quality.
  • Losses: Adding per-point RGB and background-entropy losses improves results compared with the main photometric loss alone.The per-point loss directly supervises query-point color emissions, while background entropy concentrates rendered background probability.

J. Main ablation studies details

The ablations evaluate post-activation and the geometry priors across subsampled scenes and show that post-activation is the strongest quality contributor, while low-density initialization is essential and view-count learning rates offer smaller gains.

  • The detailed ablations cover Materials, Mic, Robot, Lifestyle, Character, Statues, Ignatius, and Truck across four datasets.
  • Post-activation significantly improves reconstruction quality across the ablation experiments.
  • Low-density initialization is essential to the method and functions as an important hyperparameter.
  • −1.22 PSNR is the worst-case degradation observed when the view-count prior is removed.
  • The view-count-based learning rate produces only a slight improvement compared with the other ablation settings.

K. Additional training time details

The supplementary results report per-scene optimization times and qualitative and quantitative comparisons across five datasets. The method generally achieves comparable quality while requiring about 15 minutes per scene, with Tanks&Temples taking a few additional minutes.

  • Less than 15 minutes is required for per-scene training except on Tanks&Temples, which takes a few additional minutes.
  • The method achieves results comparable to most recent methods across Synthetic-NeRF, Synthetic-NSVF, BlendedMVS, Tanks&Temples, and DeepVoxels.
  • The supplementary material includes per-scene ablation results, auxiliary-loss results, and detailed training-time tables.
  • About 15 minutes of training on one RTX2080 Ti outperforms early-stopped Mip-NeRF trained for 6 hours by a large margin.
  • Qualitative comparisons report no consistently finer detail across methods; the method is better on fine detail for Ficus and Lego, while stronger baselines excel on other scenes.

N.1. Derivation for a 1D grid cell

The derivation shows that post-activated interpolation can approximate a shifted unit-step boundary within prescribed error and tolerance by selecting voxel-grid values, extending from 1D cells to higher-dimensional cases.

  • S(x; a, b) models a 1D grid cell using endpoint values a and b with post-activated interpolation.
  • Given 0 < ϵ < 1 and 0 < ∆ < min(c, 1 − c), suitable a and b can make S arbitrarily close to the target step function T(x; c).
  • The derivation converts approximation requirements into inequalities and an upper bound that determine feasible grid values.
  • With ϵ = 10−4, ∆ = 10−2, and δ = 0.5, derived grid values faithfully reproduce example target functions.
  • The 2D and 3D extensions recover linear boundaries by determining endpoint grid values on successive slices, with extrapolation handled by minor modifications.
  • The proof directly establishes approximation of linear surfaces, while tuned edge tolerances can also produce sharp non-linear surfaces.
Loading 2111.11215v2…