PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoInt: Automatic Integration for Fast Neural Volume Rendering

David B. Lindell, Julien N. P. Martel, Gordon Wetzstein

arXiv:2012.01714v2cs.CVcs.GRcs.LG

TL;DR

Neural volume rendering is costly because Monte Carlo integration requires hundreds of network evaluations along each of millions of rays. AutoInt learns a closed-form antiderivative with coordinate-based networks, reducing definite-integral evaluation to two network queries. Applied to neural rendering, it achieves greater than 10× faster rendering with slightly reduced image quality, while training remains slow and approximation accuracy depends on design choices.

  • Problem

    Neural volume rendering is extremely slow and memory inefficient because Monte Carlo integration requires hundreds of forward passes for each of millions of rays.

  • Method

    AutoInt trains a derivative computational graph on the signal to integrate, then reassembles shared parameters into a network representing its antiderivative.

  • Results

    Greater than 10× faster rendering improves the speed–image-quality tradeoff, with a slight reduction in image quality.

  • Takeaways & Limitations

    AutoInt enables efficient neural volume rendering by evaluating learned closed-form integral solutions instead of relying on traditional sampling.

  • Takeaways & Limitations

    Approximation accuracy degrades as the piecewise-rendering sample count N decreases, and optimizing a network for posed 2D images remains slow.

Abstract

from arXiv · show

Numerical integration is a foundational technique in scientific computing and is at the core of many computer vision applications. Among these applications, neural volume rendering has recently been proposed as a new paradigm for view synthesis, achieving photorealistic image quality. However, a fundamental obstacle to making these methods practical is the extreme computational and memory requirements caused by the required volume integrations along the rendered rays during training and inference. Millions of rays, each requiring hundreds of forward passes through a neural network are needed to approximate those integrations with Monte Carlo sampling. Here, we propose automatic integration, a new framework for learning efficient, closed-form solutions to integrals using coordinate-based neural networks. For training, we instantiate the computational graph corresponding to the derivative of the network. The graph is fitted to the signal to integrate. After optimization, we reassemble the graph to obtain a network that represents the antiderivative. By the fundamental theorem of calculus, this enables the calculation of any definite integral in two evaluations of the network. Applying this approach to neural rendering, we improve a tradeoff between rendering speed and image quality: improving render times by greater than 10 times with a tradeoff of slightly reduced image quality.

1. Introduction

Neural volume rendering achieves photorealistic view synthesis but is computationally and memory intensive because rendering requires costly integrations along millions of rays. AutoInt learns closed-form antiderivatives to evaluate these integrals efficiently, yielding over 10× faster rendering with slightly reduced image quality.

  • Neural rendering offers photorealistic image quality for view synthesis but remains extremely slow and memory inefficient.
  • Hundreds of forward passes through the volume network are typically required for each of millions of rendered rays.
  • Greater than 10× faster rendering improves the speed–image-quality tradeoff, though with a slight reduction in image quality.
  • AutoInt trains a grad network to represent the target signal, then reassembles its parameters into an antiderivative network.
  • Definite integrals can be computed in two evaluations of the antiderivative MLP using the fundamental theorem of calculus.
  • Efficient integral evaluation accelerates rendering, but optimizing a network for posed 2D images remains slow.
  • AutoInt provides a computation- and memory-efficient neural volume-rendering model through automatic integration.

2. Related Work

Prior neural rendering methods use coordinate-based scene representations but rely on expensive Monte Carlo integration along rays. AutoInt instead reduces integral evaluation to two network queries by constructing a parameter-sharing integral and grad network pair.

  • Coordinate-based networks provide continuous, memory-efficient neural scene representations whose detail depends on network capacity rather than voxel-grid resolution.
  • NeRF combines a coordinate-based volume network with a neural volume renderer for high-quality novel-view synthesis.
  • NeRF’s Monte Carlo rendering requires hundreds of MLP forward passes per ray, with training taking days and frame rendering tens of seconds to minutes.
  • AutoInt reduces integral evaluation from hundreds of network queries to two for neural rendering.
  • AutoInt builds a grad network from an integral-network architecture, optimizes it on a function, and evaluates the shared-parameter integral network for definite integrals.
  • The framework develops neural-network-based analytical integration beyond prior closed-form results limited to simple shallow architectures.

3. AutoInt for Neural Integration

AutoInt learns an antiderivative by training a derivative-derived coordinate network and reassembling its shared parameters. Definite integrals then require two evaluations, with tomography experiments showing that Swish generalizes better than sine activations.

  • Principles: AutoInt trains a grad network to match a target signal, then reassembles its shared parameters into an integral network representing the antiderivative.The grad network is obtained from the integral network's derivative computational graph.
  • Evaluating Antiderivatives & Definite Integrals: The grad network is instantiated from an MLP architecture and optimized with a loss penalizing discrepancies between its output and the target signal.AutoInt supports choices of layers, features, and nonlinearities, and extends to high-dimensional signals.
  • Evaluating Antiderivatives & Definite Integrals: Two evaluations of the integral network compute any definite integral through the fundamental theorem of calculus.The framework avoids traditional sampling or finite-difference integration at evaluation time.
  • Example in Computed Tomography: In computed tomography, AutoInt trains on sparse ray measurements and evaluates the integral network to produce unseen measurements.The setup parallels novel-view synthesis: Monte Carlo evaluations occur during training, while inference uses two integral-network evaluations.
  • Example in Computed Tomography: Swish with normalized positional encoding generalizes well to unseen tomography views, whereas SIREN fits observed measurements better but fails to generalize.The figure reports that Swish performs best in PSNR and interpolates missing sinogram data more effectively than sine activations.
  • Example in Computed Tomography: ReLU integral networks may produce step functions in their grad networks, whose zero-almost-everywhere derivatives can make training difficult.The paper identifies the choice of nonlinearities in grad networks as an avenue for future work.

4. Neural Volume Rendering

Neural volume rendering traces rays through an absorptive, emissive volume and requires integrating radiance while accounting for transmittance. AutoInt makes this tractable by approximating the nested volume-rendering integral with piecewise sections evaluated using closed-form antiderivatives.

  • Volume Rendering: Volume rendering assigns pixel colors by integrating emissive radiance along camera rays while accounting for absorption through transmittance.The volume is parameterized by position and viewing direction, with density σ controlling absorption and radiance c controlling emitted color.
  • Volume Rendering: Conventional volume rendering computes the VRE numerically using Riemann sums, quadratures, or Monte Carlo methods whose accuracy depends on ray samples.
  • Approximate Volume Rendering: AutoInt cannot directly evaluate the VRE because it contains nested integrations of radiance weighted by cumulative transmittance.The framework therefore approximates the VRE using piecewise sections, allowing each absorption and radiance integral to be evaluated efficiently.
  • Approximate Volume Rendering: The piecewise VRE expresses rendering through interval-wise density and color integrals combined with transmittance factors along the ray.Each interval has length δi, and the expression can also be viewed as repeated alpha compositing with alpha values of ¯σiδi.
  • Approximate Volume Rendering: The piecewise formulation is an approximation whose lower interval count improves computational efficiency but degrades accuracy.The paper evaluates this efficiency–accuracy tradeoff in volume rendering and learned novel view synthesis.

5. Optimization Framework

The optimization framework trains gradient networks and a sampling network for piecewise volume rendering, then uses the resulting representation for efficient inference. It addresses training and sampling choices while comparing runtime–quality tradeoffs across interval counts and rendering configurations.

  • Network Parameterization: Two MLPs compute integrals over volume density σ and color c within the optimization framework.The networks receive ray-defining inputs and calculate positions along each ray as x = o + t d.
  • Grad Network Positional Encoding: A normalized positional encoding is used for the integral network because direct encoding in the grad network introduces exponentially increasing derivative amplitudes and produces poor results.The encoding uses sine and cosine frequencies controlled by L.
  • Predictive Sampling: A sampling MLP predicts interval lengths δ, and stratified samples are drawn within subdivided intervals for training-time quadrature.Intervals are adjusted to place more samples where σ and c vary substantially, improving sample efficiency.
  • Predictive Sampling: The sampling-network ablation varies N = {2, 4, 8, 16, 32, 64} intervals with and without learned sampling, evaluating PSNR on 200 Lego test views.
  • Evaluation: Table 1 aggregates AutoInt, NeRF, and Neural Volumes results over 8 Blender scenes to compare image quality, runtime, and memory tradeoffs.Table 2 additionally compares PSNR/SSIM and LPIPS across piecewise and full VRE configurations, grad networks, standard MLP sampling, and varying NeRF samples.

6. Results

AutoInt improves the rendering speed–image-quality tradeoff across quantitative and qualitative evaluations, with rendering more than 10× faster than NeRF but slightly reduced image quality. More piecewise sections improve quality at additional computational cost.

  • More than 10× faster rendering than NeRF is achieved, though with a tradeoff in image quality.
  • AutoInt quantitatively outperforms Neural Volumes while improving render time relative to NeRF.
  • Increasing the number of piecewise sections in the approximate VRE improves rendered quality at the cost of computation.
  • The sampling network improves performance and sample efficiency by allocating more sections where volume density varies substantially.
  • NeRF quality degrades significantly when using fewer ray samples, whereas AutoInt retains higher quality with the same number of samples.
  • AutoInt exhibits fewer Materials-scene artifacts than Neural Volumes and better models view-dependent effects than both baselines in the Drums scene.

7. Discussion

The discussion presents AutoInt as a learned closed-form integration framework that improves computational efficiency for neural volume rendering, while retaining an image-quality tradeoff and leaving architecture and training improvements open.

  • AutoInt learns closed-form integral solutions with coordinate-based neural networks to improve computational efficiency in neural volume rendering.
  • AutoInt currently retains a tradeoff between computational speedups and image quality.
  • The approach uses approximations that trade image quality for computationally efficient rendering, paralleling conventional fast VRE methods.
  • Improved training strategies and more expressive grad-network architectures remain important future directions.
  • The framework may apply beyond computer vision to inverse rendering, sparse-view tomography, and compressive sensing.

Supplemental Material AutoInt: Automatic Integration for Fast Neural Volumetric Rendering

The supplemental material identifies the paper and its authors, with the work released as arXiv:2012.01714v2 in May 2021.

  • The paper is authored by David B. Lindell, Julien N. P. Martel, and Gordon Wetzstein.
  • The listed contact address is lindell, jnmartel, and gordon.wetzstein at Stanford University.
  • The supplemental material lists arXiv:2012.01714v2, dated 23 May 2021.

1. Multivariable Integration with AutoInt

AutoInt extends the grad-network and integral-network relationship to multivariable integration. It constructs partial-derivative networks, trains them on the target signal, then reassembles and evaluates the integral network at domain bounds.

  • Setup: An implicit neural representation maps low-dimensional coordinates to outputs and admits a sub-gradient with respect to its input coordinates.
  • Network relationship: The grad network is the input derivative of the coordinate-based network, while the integral network is the original network that represents the antiderivative.
  • Single integration: For one-dimensional integration, the fundamental theorem of calculus relates the grad and integral networks, allowing a definite integral in two integral-network evaluations.
  • Multivariable extension: For multiple integration, AutoInt introduces partial derivatives with respect to each integration variable and extends the construction to double and n-dimensional integrals.
  • Procedure: The multivariable procedure constructs partial-derivative grad networks, trains them, reassembles the integral network, and evaluates it at the domain bounds.

2. Deriving the VRE Approximation and Quadrature

The VRE approximation uses piecewise-constant sections along each ray, with section lengths, densities, and radiances defining the resulting quadrature expression.

  • The volume rendering equation is approximated as a Riemann integral with N piecewise-constant sections.
  • Each section length is defined by δ_i = t_i − t_{i−1}.
  • The section density σ̄_i and radiance c̄_i are defined before substitution into the rendering equation.
  • Substituting the section definitions into the VRE and simplifying yields the piecewise volume rendering expression.
  • The NeRF quadrature rule is obtained using a Taylor expansion in the limit σ̄_iδ_i → 0.

3. AutoInt Implementation

AutoInt implements integral and derivative networks as directed acyclic computational graphs, constructs grad networks through automatic differentiation, and evaluates them with graph-specific ordering and reuse. Its custom implementation improves training efficiency over a direct PyTorch approach.

  • AutoInt specifies the integral-network architecture and represents its computational dependencies as a directed acyclic graph.The architecture includes layers, features, nonlinearities, and input parameterization.
  • Automatic differentiation of the integral-network DAG recursively constructs the grad-network computation graph in topological order.
  • The grad network is evaluated in reverse topological order, separately from the integral network and without relying on the latter’s forward-pass intermediates.
  • Figure 1 contrasts the integral network with its tree-like grad network, which computes a partial derivative with respect to an input.
  • The implementation uses lexicographic-topological forward evaluation with cumulative computation reuse across network legs.
  • The custom Python and PyTorch implementation uses NetworkX and computational nodes wrapping PyTorch modules.
  • Over 15% GPU-memory savings and more than 1.8x faster training iterations were measured versus direct PyTorch for volume rendering.

4. Supplemental Results

Supplemental experiments examine AutoInt across sparse-view tomography, direct antiderivative supervision, captured scenes, and image-quality metrics, highlighting the effects of nonlinearities and section counts.

  • 4.1. Sparse-View Computed Tomography: Figure 2 compares sinograms and center scanlines across activation functions and angular subsampling levels.
  • 4.1. Sparse-View Computed Tomography: ReLU is particularly unsuited to this representation because its derivative is zero-valued almost everywhere, producing noisy inpainted results.
  • 4.2. Direct Integral-Network Supervision: Direct supervision with sampled definite integrals trains a Swish network to fit the sigmoid antiderivative up to a scalar constant.The learned fit is shown after removing the offset for visualization.
  • 4.3. Additional Neural-Rendering Results: Both AutoInt variants using 8 and 32 sections achieve similarly high image quality on held-out DeepVoxels scenes.
  • 4.3. Additional Neural-Rendering Results: Additional LLFF evaluations report PSNR, SSIM, and LPIPS, alongside qualitative comparisons between NeRF and AutoInt.
Loading 2012.01714v2…