PaperlayerView exact revision on arXiv

Source-linked AI summary

DeRF: Decomposed Radiance Fields

Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, Andrea Tagliasacchi

arXiv:2011.12490v1cs.CVcs.GR

TL;DR

Neural volume rendering is computationally intensive, especially during NeRF inference. DeRF spatially decomposes scenes among smaller networks using differentiable Voronoi cells and Painter’s Algorithm compositing, achieving faster rendering or higher quality at comparable cost.

  • Problem

    NeRF inference is slow because rendering requires many network evaluations per pixel, while reducing network capacity naively lowers rendering quality.

  • Method

    DeRF spatially decomposes a scene into Voronoi cells, assigns smaller neural networks to those regions, and composites their outputs with the Painter’s Algorithm.

  • Results

    Up to 3x faster inference is reported without rendering-quality loss, or up to 1.0dB higher PSNR at the same rendering cost.

  • Takeaways & Limitations

    Voronoi decomposition makes the independently rendered parts compatible with GPU-friendly Painter’s Algorithm compositing while improving rendering efficiency or quality.

  • Takeaways & Limitations

    The method observes diminishing returns as the number of decomposition heads increases, and the paper leaves heterogeneous-head designs for future work.

Abstract

from arXiv · show

With the advent of Neural Radiance Fields (NeRF), neural networks can now render novel views of a 3D scene with quality that fools the human eye. Yet, generating these images is very computationally intensive, limiting their applicability in practical scenarios. In this paper, we propose a technique based on spatial decomposition capable of mitigating this issue. Our key observation is that there are diminishing returns in employing larger (deeper and/or wider) networks. Hence, we propose to spatially decompose a scene and dedicate smaller networks for each decomposed part. When working together, these networks can render the whole scene. This allows us near-constant inference time regardless of the number of decomposed parts. Moreover, we show that a Voronoi spatial decomposition is preferable for this purpose, as it is provably compatible with the Painter's Algorithm for efficient and GPU-friendly rendering. Our experiments show that for real-world scenes, our method provides up to 3x more efficient inference than NeRF (with the same rendering quality), or an improvement of up to 1.0~dB in PSNR (for the same inference cost).

1. Introduction

NeRF enables high-quality neural rendering but is slow because it requires many MLP evaluations per pixel. DeRF addresses this with spatial decomposition, using smaller networks and Voronoi-based compositing to improve efficiency or quality.

  • Motivation: Neural volume rendering remains limited by inference performance, with NeRF requiring hundreds of MLP invocations per pixel.The paper focuses specifically on inference efficiency.
  • Motivation: Larger NeRF networks provide diminishing rendering-quality returns as depth or width increases.Figure 2 varies network depth and width to illustrate this capacity-quality trade-off.
  • Approach: DeRF spatially decomposes scenes and assigns smaller neural networks to separate areas, allowing the parts to render the whole scene together.Each localized network renders a convex image portion, which is composited into the final output.
  • Approach: Voronoi decomposition addresses memory-access constraints because convex cells can be rendered independently and composited with the Painter’s Algorithm.The decomposition is designed to avoid the incoherent accesses caused by random partitioning.
  • Results: Up to 3x faster inference is achieved without loss in rendering quality, or up to 1.0dB higher PSNR at the same rendering cost.These are reported comparisons against the NeRF baseline.

2. Related Work

Related neural-rendering methods include image-space CNN approaches and 3D volumetric representations. DeRF’s framework uses independent NeRF networks over spatial regions and GPU-conscious compositing.

  • Image-space neural rendering: Image-space neural rendering commonly uses CNNs for image-to-image transformations, sometimes aided by 3D reasoning.Regular grids and point clouds are examples of intermediate 3D representations used with CNN post-processing.
  • DeRF framework: DeRF represents a scene with independent NeRF networks, each responsible for a region within a Voronoi cell.The framework combines these region-specific radiance fields rather than using one network over the full space.
  • DeRF framework: Each ray’s final color is formed by volume rendering cell segments and alpha-compositing the resulting colors.The rendering heads provide radiance c and density σ for their assigned spatial regions.

3. Method

DeRF decomposes a scene into spatially assigned neural heads, then renders and composites those parts efficiently. A differentiable Voronoi partition supports balanced, accelerator-friendly evaluation and Painter’s Algorithm compositing.

  • 3.2. Decomposed radiance fields (DeRFs): NeRF represents an entire scene with one network, while DeRF models density and radiance as weighted sums of functions from multiple neural heads.The decomposition function assigns coordinatewise-positive weights summing to one across heads.
  • 3.2. Decomposed radiance fields (DeRFs): Spatial partitioning makes zero-weight heads unnecessary, so only one head is evaluated at each position to accelerate inference.The decomposition is trained so exactly one weight is non-zero at any spatial position.
  • 3.2. Decomposed radiance fields (DeRFs): The decomposition is balanced by encouraging the heads to contribute uniformly, so regions represent similar amounts of visual complexity.The balancing objective minimizes a loss whose optimum uses all heads equally.
  • 3.3. Voronoi learnable decompositions: Voronoi sites provide a differentiable spatial partition that becomes hard as β → ∞ and remains compatible with Painter’s Algorithm.The soft Voronoi formulation uses β as a temperature controlling approximation softness.
  • 3.3. Voronoi learnable decompositions: Painter’s Algorithm renders independent parts from back to front, while the Voronoi ordering provides the partial order needed for correct compositing.The ordering compares distances from each Voronoi site to the camera location.
  • 3.3. Voronoi learnable decompositions: Voronoi cells can be rendered separately without memory-cache incoherence, improving GPU throughput before their images are composited into the final image.The method renders one cell at a time and composites cell images back-to-front.
  • 3.4. Training details: Successful training first learns the decomposition function, then trains the network heads after coarse scene-wide density and radiance pre-training.This ordering avoids interference from shifting decomposition boundaries; coarse pre-training lasts approximately 100k iterations.

4. Results

DeRF achieves favorable quality–efficiency trade-offs on real forward-facing scenes by decomposing rendering across specialized networks. Voronoi decomposition preserves practical efficiency, while naive alternatives suffer from memory-access costs or require more regions.

  • Efficiency: Up to 3× faster inference is achieved with no loss in rendering quality through Voronoi decomposition.The method is evaluated on real forward-facing scenes and compares rendering quality against computational cost.
  • Efficiency: 49.8% average and 62.8% maximum FLOP increases occur from no decomposition to eight decompositions, versus quadratic growth when using more units.These figures concern the “fern” scene’s theoretical operation counts.
  • Efficiency: 29.3% average and 47.5% maximum runtime increases occur from no decomposition to eight decompositions, confirming the theoretical efficiency trend in practice.Voronoi decomposition keeps actual runtime growth substantially below the cost of increasing neuron counts.
  • Quality: A 16-head DeRF model with 128-unit networks outperforms a 256-unit NeRF baseline while each DeRF network has one quarter as many parameters.The qualitative comparison reports better performance and quality for the DeRF configuration.
  • Quality: At equal rendering cost, finer decompositions improve PSNR, SSIM, and LPIPS on a representative scene.The reported quality trend holds across all three evaluation metrics as the number of decompositions increases.
  • Alternative decompositions: MLP-based decomposition becomes less efficient as heads increase because random memory access prevents effective acceleration.Although its theoretical operation count should be similar, its runtime grows faster than DeRF.
  • Alternative decompositions: Regular-grid decomposition requires more regions for non-homogeneous scenes and produces incoherent weight-memory access patterns that reduce computational performance.The limitation is attributed to the curse of dimensionality and scene structure.

5. Conclusions

DeRF improves neural-rendering inference by spatially decomposing scenes into Voronoi cells rendered with smaller networks and composited efficiently. The method addresses diminishing returns in network capacity while delivering faster or higher-quality rendering, though several scaling and training questions remain.

  • 5. Conclusions: DeRF increases neural-rendering inference efficiency through spatial decomposition into multiple cells.Each decomposed part uses a separate network, addressing diminishing returns from increasing network capacity.
  • 5. Conclusions: Voronoi decompositions are compatible with Painter’s Algorithm, making the inference pipeline GPU-friendly.
  • 5. Conclusions: DeRF can render faster and deliver higher-quality images than the baseline approach.
  • 5. Conclusions: Increasing the number of decomposition heads also exhibits diminishing returns, and heterogeneous head capacities remain future work.The paper also identifies efficient scatter/gather operations as a possible way to accelerate training, which is currently slower than training models without decomposition.

A. Voronoi vs na¨ıve decomposition – Table 3

The learned decomposition improves accuracy compared with a naïve fixed-grid decomposition under the tested conditions.

  • A. Voronoi vs na¨ıve decomposition – Table 3: The learned decomposition improves all metrics over a naïve decomposition with cells fixed into a grid.

B. Additional experiments on real data

Experiments on real capture scenes show that DeRF generally offers the strongest quality–computation trade-off, especially with finer decompositions and perceptual or structural metrics. Results also vary across initialization, while the supplied figures document scene-specific quantitative evaluations.

  • B. Additional experiments on real data: DeRF models provide the best quality–computation trade-off for all LPIPS cases and almost all PSNR and SSIM cases.
  • B. Additional experiments on real data: Highly decomposed DeRF models perform best for the strong majority of scenes, especially on perceptual and structural metrics.
  • B. Additional experiments on real data: DeRF’s advantage is most consistent on scenes with higher absolute error and when less inference computation is used.
  • B. Additional experiments on real data: Initialization produces significant variation in reconstruction quality, particularly for PSNR, motivating multiple runs for more robust statistics.

C. Synthetic data – Figure 18

DeRF can learn decomposed models from synthetic data, in addition to the photorealistic scenes of primary interest. The demonstrated synthetic models use 192 units and 8 heads.

  • DeRF can learn decomposed models from synthetic data, demonstrated on two synthetic scenes.The models include renders and visualizations of their spatial decompositions.
  • Each synthetic-scene model uses 192 units and 8 heads.
Loading 2011.12490v1…