Source-linked AI summary
Plenoxels: Radiance Fields without Neural Networks
Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa
TL;DR
Photorealistic radiance-field reconstruction is computationally expensive, especially for neural methods. Plenoxels address this with a directly optimized sparse spherical-harmonic voxel representation and achieve comparable quality with dramatically faster optimization.
Problem
Neural radiance fields provide photorealistic novel-view synthesis but require extensive computation time for training and rendering.
Method
Plenoxels represent scenes with a sparse voxel grid storing opacity and spherical harmonic coefficients, optimized end-to-end from calibrated images using differentiable rendering and regularization without neural networks.
Results
Two orders of magnitude faster optimization than NeRF is achieved while maintaining comparable rendering quality, including 11-minute bounded-scene optimization versus roughly 1 day for NeRF.
Takeaways & Limitations
The results suggest that differentiable volumetric rendering, rather than a neural network specifically, is the key element enabling high-quality radiance-field reconstruction.
Takeaways & Limitations
The method remains susceptible to artifacts because volumetric reconstruction is an underdetermined inverse problem, and its artifacts differ from those of neural methods.
Abstract
from arXiv · showhide
We introduce Plenoxels (plenoptic voxels), a system for photorealistic view synthesis. Plenoxels represent a scene as a sparse 3D grid with spherical harmonics. This representation can be optimized from calibrated images via gradient methods and regularization without any neural components. On standard, benchmark tasks, Plenoxels are optimized two orders of magnitude faster than Neural Radiance Fields with no loss in visual quality.
1. Introduction
Plenoxels replace neural radiance-field representations with an explicit sparse voxel model that preserves photorealistic quality while greatly reducing optimization time. The approach uses spherical harmonics, differentiable rendering, regularization, and efficient scene handling without neural networks.
- NeRF achieves photorealistic novel-view rendering but requires more than a day to train and about 30 seconds per frame to render on one GPU.
- Plenoxels train radiance fields from scratch without neural networks while maintaining NeRF quality and reducing optimization time by two orders of magnitude.
- 11 minutes on a single Titan RTX GPU versus roughly 1 day for NeRF yields more than a 100× speedup on bounded scenes.
- The explicit model stores opacity and spherical harmonic coefficients in a sparse voxel grid, with empty voxels pruned and optimization proceeding from coarse to fine.
- Plenoxels support bounded and unbounded scenes using voxel grids with normalized device coordinates or multisphere background images.
- The method frames photorealistic volumetric reconstruction as an inverse problem whose simple representation, renderer, regularizer, and optimizer can achieve state-of-the-art results.
2. Related Work
Prior work spans classical voxel-based reconstruction and neural implicit radiance fields, with efficiency improvements focused largely on rendering or post-processing. Plenoxels instead optimize a sparse spherical-harmonic voxel representation directly, without neural networks or warping functions.
- Classical Volume Reconstruction: Classical volume-reconstruction methods commonly use voxel grids or multiplane images, while high-resolution voxel grids can be memory limited.
- Neural Volume Reconstruction: Neural implicit methods extend occupancy, signed-distance, and point-cloud representations to calibrated 2D image supervision through differentiable volume rendering.
- Neural Volume Reconstruction: NeRF trains a coordinate-based MLP to predict color and opacity from 3D position and viewing direction, but full training takes several days and rendering takes about half a minute per image.
- Neural Volume Reconstruction: Neural Volumes use an interpolated voxel grid optimized through a convolutional network and learned warping, whereas Plenoxels optimize the voxel grid directly without either component.
- Accelerating NeRF: Efficiency research accelerates NeRF through spatial subdivision, computational changes, and preprocessing or post-processing methods that remove bottlenecks.
- Accelerating NeRF: PlenOctrees extract a pretrained NeRF into a sparse spherical-harmonic voxel grid, achieving roughly 3× training and greater-than-3000× rendering speedups.
- Accelerating NeRF: Plenoxels extend PlenOctrees with end-to-end optimization, arbitrary-resolution sparse grids, and trilinear interpolation while delivering two orders of magnitude faster training than NeRF.
3. Method
Plenoxels directly optimize a sparse voxel representation with opacity and spherical-harmonic coefficients through differentiable volume rendering, interpolation, regularization, and gradient-based optimization. The method extends across scene settings while using sparsity and pruning to make optimization efficient.
- Representation: Each occupied voxel corner stores opacity σ and spherical-harmonic coefficients, which are interpolated to produce view-dependent color and opacity.Degree-2 spherical harmonics use 27 coefficients per voxel, with nine coefficients per color channel.
- Volume rendering: The differentiable volume-rendering model integrates sampled colors and opacities along each ray, enabling updates from training-ray errors.Transmittance, opacity, sample distance, and color determine each sample’s contribution to the rendered ray color.
- Interpolation: Trilinear interpolation defines a continuous plenoptic function and improves both effective resolution and optimization compared with nearest-neighbor interpolation.Averaged over the 8 NeRF synthetic scenes, trilinear interpolation at resolution 128^3 outperforms nearest-neighbor interpolation at 256^3.
- Resolution and sparsity: A coarse-to-fine strategy repeatedly optimizes, prunes unnecessary voxels, subdivides the remaining grid, and initializes new values by trilinear interpolation.The synthetic example begins at 256^3 and upsamples to 512^3.
- Optimization objective: The base objective combines rendered-color mean squared error with total-variation regularization, evaluated using stochastic samples of rays and voxels.Different fixed weights are used for opacity and spherical-harmonic coefficients across bounded, forward-facing, and 360° scene types.
- Scene settings: With minor modifications, Plenoxels support synthetic, bounded, forward-facing, and unbounded 360° scenes, including normalized device coordinates and multisphere-image backgrounds.The method is reported to work beyond the synthetic scenes of the original NeRF paper.
- Regularization: TV regularization prevents visual artifacts in forward-facing scenes, while sparsity regularization encourages empty voxels to save memory and reduce quality loss during upsampling.Opacity regularization has a quantitatively larger effect than spherical-harmonic regularization, although both help avoid artifacts.
- Optimization: Fewer than 10% of voxels have nonzero gradients within the first 1–2 minutes, enabling the implementation’s optimization speed.RMSProp is used to ease ill-conditioning without the computational complexity of a second-order method.
4. Results
Plenoxels achieve comparable quality across synthetic and real scene settings while optimizing rapidly, including strong results after only one epoch. Ablations identify continuous interpolation, rendering formulation, and TV regularization as important design factors.
- Benchmark results: The experiments span synthetic bounded, real forward-facing, and real 360◦ scenes.The 360◦ evaluation uses four Tanks and Temples scenes, while the forward-facing evaluation uses eight scenes from NeRF.
- Optimization speed: Quality results appear after the first epoch of optimization, requiring less than 1.5 minutes.Figure 5 reports an average of 1 minute and 20 seconds on the synthetic scenes.
- Benchmark results: Plenoxels achieve comparable quality to the best baseline while training in an average of 11 minutes per synthetic scene on one GPU.The evaluation covers eight synthetic scenes and includes interactive rendering.
- Ablations: Continuous trilinear interpolation substantially improves fidelity compared with nearest-neighbor interpolation.Interpolation also represents sub-voxel variations and provides a continuous function approximation that supports optimization.
- Ablations: Increasing TV regularization allows Plenoxels to exceed NeRF fidelity when training uses 25 rather than 100 views.The stronger regularizer helps optimization extend smoothly into sparsely supervised regions.
- Ablations: The rendering formula from Max yields a substantial improvement over the Neural Volumes formula.The comparison uses the relative-transmittance formulation from Max versus Neural Volumes’ absolute-transmittance formulation.
5. Discussion
The discussion positions Plenoxels as a simple, differentiable volumetric approach with state-of-the-art fidelity and much faster training. It also notes artifacts, scene-dependent regularization, and possible extensions for broader reconstruction pipelines.
- Discussion: Plenoxels attain comparable fidelity to the state of the art while requiring orders of magnitude less training time.The method uses a differentiable forward model, continuous representation, and regularization.
- Limitations: Artifacts remain because scene reconstruction is an underdetermined inverse problem, and their appearance differs between Plenoxel and neural models.Both methods achieve similar quality under standard metrics despite exhibiting different artifacts.
- Limitations: There is no optimal a priori TV weight because scene scale, smoothness, and training-view count vary between scenes.Scene-by-scene tuning may improve results and is practical because training is fast.
- Future work: Voxel cone tracing, tone mapping, and hierarchical data structures are proposed as future extensions.An octree may provide additional speedup if differentiable interpolation is preserved.
- Implications: The two-orders-of-magnitude speedup may enable downstream applications such as multi-bounce lighting and large-scale 3D generative models.Combining Plenoxels with camera optimization and voxel hashing could support end-to-end photorealistic 3D reconstruction.
Supplementary Material
The supplementary material provides additional experimental details, ablations, and complete per-scene comparisons with prior work.
- Supplementary contents: The supplement contains further ablation studies, experimental details, and full per-scene quantitative and visual comparisons.It also includes additional visualizations and encourages viewing the accompanying video for broader scene results.
B.1. Implementation Details
The implementation maintains sparse voxel data while accelerating rendering and optimization with CUDA parallelism, stochastic regularization, and explicit learning-rate schedules.
- Data structure: A dense grid plus a data table stores density and spherical-harmonic coefficients while preserving sparsity through NULL cells and interpolation.The structure supports reasonably efficient trilinear interpolation in both forward and backward passes.
- CUDA implementation: CUDA kernels parallelize across rays, colors, and spherical-harmonic coefficients, using coalesced memory access and warp-level color operations.Each warp handles one ray, with threads processing individual spherical-harmonic coefficients.
- Color interpolation: Negative interpolated colors are clipped to zero with ReLU instead of sigmoid to preserve linearity during trilinear color interpolation.This differs from NeRF’s bounded sigmoid color parameterization.
- Voxel pruning: Thresholding differs by scene type: weight-based pruning is used for synthetic and 360◦ scenes, while forward-facing scenes use opacity-based pruning.Weight-based pruning can remove sparsely supervised edge content in forward-facing scenes.
- Optimization: Optimization uses 5000 rays per batch, RMSProp, separate schedules for opacity and spherical-harmonic coefficients, and stochastic TV losses.Each optimization step evaluates TV losses on 1% of grid voxels.
- Regularization: The total-variation quantity measures adjacent-voxel differences for density or spherical-harmonic coefficients along each grid dimension.The same loss is applied in NDC and to the background model, with wraparound at equirectangular background edges.
B.2. Synthetic experiments
Synthetic scenes use staged optimization: the grid starts at 256^3, is pruned and upsampled to 512^3, and TV regularization is needed only initially without materially changing quality.
- Synthetic scenes perform nearly identically when TV regularization is used only during the first optimization stage.Turning regularization off after pruning and increasing resolution modestly reduces training time.
- The grid starts at 256^3, is pruned and upsampled to 512^3 after 38400 steps, and runs for 128000 total steps.TV regularization is applied only during the initial 38400 steps before pruning and upsampling.
B.3. Forward-facing experiments
Forward-facing scenes use progressively finer grids with scene-specific TV regularization and sparsity settings. The method notes that TV weights can reduce artifacts but may also blur scenes, making hyperparameter tuning useful.
- Forward-facing grids progress from 256 × 256 × 128 to 512 × 512 × 128 and finally 1408 × 1156 × 128 during optimization.Pruning and upsampling occur at steps 38400 and 76800, with 128000 total optimization steps.
- TV parameters that work for forward-facing NeRF scenes may not generalize, because stronger weights reduce artifacts while blurring the scene more.The paper suggests sweeping TV weights because scenes differ in content, camera noise, and scale.
- A hyperparameter sweep of TV weights can improve metrics for NeRF scenes when different values are used across scenes.
B.4. 360◦experiments
For 360° scenes, Plenoxels use a progressively upsampled foreground grid together with a separately encoded background. More general scenes may require additional ray bounds, delayed foreground optimization, or sparsity losses.
- The 360° foreground grid progresses from 128^3 through 256^3, 512^3, and 640^3, while the background uses 64 layers at 2048 × 1024 resolution.The model uses separate TV weights and regularization terms for the foreground and background.
- For more general scenes, a near-bound on camera rays can prevent floaters close to the camera.The paper also mentions beginning foreground optimization later as another useful setting.
- A hyperparameter sweep of TV weights can be helpful beyond the scenes with fixed TV settings.
C. Ablation Studies
Ablations show that interpolation, regularization, optimizer choice, and the rendering formula materially affect Plenoxel behavior. Trilinear interpolation and the Max transmittance formula are particularly important, while optimization remains broadly robust.
- Regularization: TV regularization is essential for forward-facing scenes, particularly for opacity, and also helps spherical harmonics avoid visual artifacts.Without TV regularization on either component, three of eight scenes run out of GPU memory.
- Interpolation: Tricubic interpolation offers negligible improvement over trilinear interpolation despite substantially increasing computation.
- Optimization: The method is robust to learning-rate schedules and optimizers, although RMSProp noticeably improves optimization over SGD, especially for spherical harmonic coefficients.
- Regularization: With 25 training images, higher TV regularization resolves artifacts that remain under low TV regularization.
- Rendering formula: The Max rendering formula substantially outperforms the Neural Volumes formula because its transmittance accounts for preceding samples along the ray.The paper characterizes this as more physically accurate modeling of transmittance.
D. Per-Scene Results
This section reports per-scene results for synthetic, forward-facing, and 360° scenes, with visual comparisons against several baselines. It also documents evaluation differences and distinct behavior in unsupervised regions.
- Per-scene evaluation: Per-scene results cover 8 synthetic scenes, 8 forward-facing scenes, and four 360° scenes from Tanks and Temples.The synthetic, forward-facing, and 360° results are presented in Tables 10, 11, and 9, respectively.
- Evaluation procedure: JAXNeRF timings come from reruns with centered pixels and are multiplied by four to account for parallel execution across four GPUs.This timing adjustment is stated for both the synthetic and forward-facing evaluations.
- Evaluation procedure: NeRF++ metrics were independently re-evaluated using VGG LPIPS and standard SSIM from rendered images shared by the original authors.The re-evaluated values differ slightly from those reported in the NeRF++ paper.
- Visual comparisons: Visual comparisons use ground truth and competing methods across synthetic, forward-facing, and 360° scene categories.Synthetic scenes compare Neural Volumes, JAXNeRF, and Plenoxels; forward-facing scenes compare JAXNeRF and Plenoxels; 360° scenes compare NeRF++ and Plenoxels.
- Unsupervised regions: In unsupervised regions, JAXNeRF fills in plausible textures whereas Plenoxels default to gray.The figure notes this difference using the bottom-right region of the orchids view as an example.