PaperlayerView exact revision on arXiv

Source-linked AI summary

Instant Neural Graphics Primitives with a Multiresolution Hash Encoding

Thomas Müller, Alex Evans, Christoph Schied, Alexander Keller

arXiv:2201.05989v2cs.CVcs.GRcs.LG

TL;DR

Neural graphics primitives can be costly to train and evaluate, while existing task-specific encodings complicate training or limit applicability. This paper introduces a multiresolution hash encoding that enables compact neural networks, achieving several-orders-of-magnitude speedups and single-GPU training in seconds.

  • Problem

    Existing task-specific encodings can complicate training, restrict methods to particular tasks, or perform poorly on GPUs due to control flow and pointer chasing.

  • Method

    The paper introduces an adaptive, task-independent multiresolution hash encoding that augments a small neural network with trainable feature tables and requires no structural updates during training.

  • Results

    The encoding achieves several-orders-of-magnitude speedups for NeRF and brings single-GPU training times for many graphics applications into the seconds range.

  • Takeaways & Limitations

    The approach makes neural graphics applications practical in time-constrained settings such as online training and inference.

  • Takeaways & Limitations

    The hash encoding adds complexity in generative settings because its features are not arranged bijectively with a regular grid, leaving the best solution for future work.

Abstract

from arXiv · show

Neural graphics primitives, parameterized by fully connected neural networks, can be costly to train and evaluate. We reduce this cost with a versatile new input encoding that permits the use of a smaller network without sacrificing quality, thus significantly reducing the number of floating point and memory access operations: a small neural network is augmented by a multiresolution hash table of trainable feature vectors whose values are optimized through stochastic gradient descent. The multiresolution structure allows the network to disambiguate hash collisions, making for a simple architecture that is trivial to parallelize on modern GPUs. We leverage this parallelism by implementing the whole system using fully-fused CUDA kernels with a focus on minimizing wasted bandwidth and compute operations. We achieve a combined speedup of several orders of magnitude, enabling training of high-quality neural graphics primitives in a matter of seconds, and rendering in tens of milliseconds at a resolution of ${1920\!\times\!1080}$.

1 INTRODUCTION

The paper introduces a task-independent, adaptive multiresolution hash encoding that enables compact neural graphics models to achieve high approximation quality. Configured by only the parameter count T and finest resolution N_max, it achieves state-of-the-art quality across several tasks after seconds of training.

  • Motivation: Prior successful encodings map inputs to higher-dimensional spaces and absorb much of the learning task, enabling smaller, more efficient MLPs.The paper notes that existing trainable, task-specific data structures rely on heuristics.
  • Contribution: The multiresolution hash encoding is adaptive, efficient, and task-independent, with configuration determined by T and N_max.The two configuration values are the number of parameters T and the desired finest resolution N_max.
  • Validation: The encoding reaches state-of-the-art quality on a variety of tasks after a few seconds of training.The paper validates it on gigapixel images, neural signed distance functions, neural radiance caching, and neural radiance and density fields.
  • Encoding mechanism: A hierarchy of hash tables maps grids to fixed-size feature-vector arrays, using direct entries at coarse resolutions and hashed indexing with collisions at fine resolutions.Colliding training gradients average, allowing the largest loss-relevant gradients to dominate.

2 BACKGROUND AND RELATED WORK

This section traces input encodings from classical feature mappings through frequency and parametric encodings, motivating a multiresolution hash representation that reduces storage waste without requiring known geometry or progressive pruning.

  • Input encodings: Input encodings map model inputs into higher-dimensional spaces, helping represent complex data arrangements and identify positions processed by neural networks.Examples include one-hot encoding, the kernel trick, recurrent attention, and transformers’ multiresolution sine-and-cosine functions.
  • Frequency encodings: Frequency encodings apply multiresolution sine-and-cosine mappings to NeRF’s five-dimensional spatiodirectional light field and volume density.Later extensions include randomly oriented parallel wavefronts and level-of-detail filtering.
  • Parametric encodings: Parametric encodings add trainable parameters in auxiliary grids or trees, often improving accuracy but increasing memory or computational costs.ACORN generates dense feature grids in tree leaves, providing greater adaptivity at greater computational cost.
  • Sparse parametric encodings: The dense grid allocates features inefficiently: parameters grow as O(N^3), whereas visible surface area grows as O(N^2).For a 128^3 grid, only 53 807 cells, or 2.57%, touch the visible surface.
  • Sparse parametric encodings: Multiresolution grids can preserve reconstruction quality with fewer parameters: eight grids from 16^3 to 173^3 achieved similar quality with less than half the parameters of a comparable encoding.Each grid stores interpolated 2-dimensional feature vectors, concatenated into a 16-dimensional network input.
  • Sparse parametric encodings: Our method uses compact spatial hash tables at multiple resolutions, with tunable size T, avoiding progressive pruning and prior scene geometry.The neural network learns to disambiguate hash collisions, avoiding probing, bucketing, and chaining while reducing control-flow divergence and implementation complexity.

3 MULTIRESOLUTION HASH ENCODING

The multiresolution hash encoding represents inputs with trainable feature tables across resolutions, combines interpolated features with auxiliary inputs, and feeds them to a compact MLP. Its design handles collisions through multiple scales, adapts to input distributions, and balances quality, memory, and performance through tunable hyperparameters.

  • Encoding structure: Trainable feature vectors are organized into L levels, each with up to T entries of dimensionality F, and jointly optimized with the MLP weights.The encoding parameters θ and network parameters Φ are both trainable.
  • Encoding structure: At each level, corner features are d-linearly interpolated, then concatenated across levels with auxiliary inputs to form y ∈ R^(LF+E) for the MLP.Auxiliary inputs may include encoded view directions and textures.
  • Performance vs. quality: (F = 2, L = 16) is recommended as the default Pareto-optimal configuration, while T trades memory, quality, and performance.Memory footprint is linear in T, whereas quality and performance tend to scale sub-linearly.
  • Implicit hash collision resolution: Hash collisions are resolved implicitly because coarse levels remain collision-free while collisions at finer levels are pseudo-randomly scattered and unlikely to coincide across every level.When collisions occur, gradients average; samples with greater reconstruction importance can produce larger table updates.
  • Implicit hash collision resolution: O(log(Nmax/Nmin)) levels cover scales from a collision-free coarse resolution Nmin to the task’s finest required resolution Nmax.Geometric scaling enables a conservatively large Nmax while including all meaningful learning scales.
  • Online adaptivity: Changing input distributions during training can reduce collisions at finer levels, allowing the encoding to learn a more accurate function without task-specific data-structure maintenance.This online adaptivity inherits benefits associated with tree-based encodings.

4 IMPLEMENTATION

The implementation combines a CUDA multiresolution hash encoding with fully fused MLPs, using GPU-aware memory and computation strategies for efficient training and inference. It also provides released source code and PyTorch bindings for integration into existing projects.

  • Implementation: The hash encoding is implemented in CUDA and integrated with tiny-cuda-nn’s fast fully fused MLPs.Source code for the encoding and neural graphics primitives is released publicly.
  • Performance considerations: Hash-table entries use 2-byte half precision, while full-precision master parameters support stable mixed-precision updates.This storage strategy is used to optimize inference and backpropagation performance.
  • Performance considerations: Level-by-level evaluation keeps only a small number of consecutive hash tables in GPU caches when processing input batches.The schedule evaluates each encoding level for all inputs before proceeding to the next level.
  • Implementation: 10× speed-ups are observed over a naïve Python implementation, motivating PyTorch bindings for existing projects.The bindings cover both the hash encoding and fully fused MLPs with little overhead.
  • Training: Adam jointly optimizes network weights and hash entries, while skipping zero-gradient hash-entry updates saves ∼10% performance without degrading convergence.The heuristic is especially useful when gradients are sparse, a common condition with T≫BatchSize.

5 EXPERIMENTS

Experiments across image fitting, signed distance functions, neural radiance caching, and NeRF demonstrate that multiresolution hash encoding combines high quality with substantial speed, while introducing hash-collision artifacts and limitations from smaller MLPs. The encoding also supports online rendering and scene-wide SDF evaluation.

  • Image fitting: 2.5 minutes matched ACORN’s 38.59 dB PSNR, while 4 minutes reached 41.9 dB on the Tokyo panorama with similar parameters.ACORN required 36.9 h to achieve 38.59 dB; the comparison is confounded by fully fused CUDA kernels and smaller MLPs.
  • Signed distance functions: The hash encoding approached NGLOD’s reconstruction fidelity at roughly equal parameter count, while supporting SDF evaluation throughout the training volume.NGLOD achieved the highest visual quality using a shape-tailored octree, whereas the hash method permits off-surface rendering techniques such as approximate soft shadows.
  • Neural radiance caching: 133 versus 147 frames per second at 1920 × 1080px delivered sharper neural radiance-cache reconstruction with only a 0.7 ms overhead.Because training occurs online during rendering, the overhead includes both encoding training and runtime costs.
  • Signed distance functions: Hash collisions produced persistent fine-grid microstructure that remained visible with longer training and was stronger than analogous artifacts in other primitives.The artifact was attributed to collisions because NGLOD provides a collision-free analogue and does not exhibit the same microstructure.
  • NeRF: 15 s of training produced competitive NeRF results within 1 min to 5 min, with best PSNR on high-detail scenes but weaker performance on complex view-dependent reflections.The speedup comes from using a much smaller MLP; mip-NeRF and NSVF outperformed the method on Materials.

6 DISCUSSION AND FUTURE WORK

The discussion favors concatenating multiresolution features by default, while identifying hash-collision microstructure and generative use as limitations. It also outlines future directions involving optimized hashing and applications to volumetric fields and other high-frequency tasks.

  • Concatenation vs. reduction: Concatenation is preferred by default because resolutions can be processed independently in parallel and reduction may leave the encoding too small.In the authors’ applications, concatenation with F = 2 consistently produced by far the best results.
  • Concatenation vs. reduction: Reduction may nevertheless be favorable when the neural network is significantly more expensive than the encoding, making increased F computationally insignificant.The authors present concatenation as a default rather than a hard-and-fast rule.
  • Choice of hash function: Alternative hash functions either failed to improve reconstruction quality or provided only marginal speedups while reducing quality.The evaluated alternatives included PCG32, higher-bit hashing after space-filling ordering, and dense-grid tiling.
  • Microstructure due to hash collisions: Hash collisions produce grainy microstructure in learned signed distance functions because the MLP cannot fully compensate for the collisions.Filtering hash-table lookups or imposing an additional constraint are proposed ways to overcome this artifact.
  • Generative setting: The hash encoding complicates generative use because its features are not arranged in a regular grid or bijective with regular grid points.This adds complexity relative to dense-grid parametric encodings populated by a separate generator network.
  • Other applications: Future applications include accurate high-frequency fits, attention-based tasks, and heterogeneous volumetric density fields with empty space, solid cores, and sparse surface detail.A preliminary implementation fits radiance and density fields from noisy volumetric path-tracer output, with promising initial results.

7 CONCLUSION

The multiresolution hash encoding offers a task-independent, low-overhead alternative to task-specific data structures and can serve as a drop-in neural-network input encoding. Its efficiency enables seconds-scale single-GPU training for graphics applications and accelerates NeRF by several orders of magnitude.

  • Task-independent encoding: The multiresolution hash encoding automatically focuses on relevant detail independently of the task, providing a practical alternative to task-specific data structures.It is designed to exploit neither task-specific sparsity nor smoothness assumptions.
  • Efficient deployment: Its low overhead supports time-constrained online training and inference, while serving as a drop-in replacement for neural-network input encodings.In NeRF, it speeds up computation by several orders of magnitude and matches concurrent non-neural 3D reconstruction techniques.
  • Practical impact: Single-GPU training times measured in seconds are within reach for many graphics applications, reducing iteration times and broadening where neural approaches can be applied.The conclusion connects shorter training workflows with applications ranging from lightmap baking to neural-network training.

A SMOOTH INTERPOLATION

The paper recommends applying smoothstep to d-linear interpolation weights as a low-cost way to make the encoding C1-smooth, while offsetting levels prevents aligned zero derivatives. Higher-order smoothstep can provide greater smoothness, but may reduce reconstruction quality.

  • Smooth interpolation: d-Quadratic and d-cubic interpolation are more expensive because they require lookup of 3^d and 4^d vertices instead of 2^d.Smoothstep is recommended as the low-cost alternative for obtaining smoother interpolation.
  • Smooth interpolation: Applying smoothstep to d-linear interpolation weights removes derivative discontinuities by making the encoding C1-smooth.The smoothstep derivative vanishes at 0 and 1, so the chain rule eliminates the encoding’s derivative discontinuity.
  • Smooth interpolation: Offsetting each level by half its voxel size, 1/(2N_l), prevents zero derivatives from aligning across all levels.This enables the encoding to learn smooth, non-zero derivatives at every spatial location x.
  • Smooth interpolation: Higher-order smoothstep functions S_n provide higher-order smoothness at small additional cost, but reconstruction quality tends to decrease with higher-order interpolation.The first-order smoothstep S_1 is essentially free because its computational cost is hidden by memory bottlenecks, so it is not used by default.

B IMPLEMENTATION DETAILS OF NGLOD

The NGLOD implementation mirrors the hash-encoding implementation but replaces hash tables with collision-free octree feature vectors. Its configuration uses fewer levels and begins look-ups at level 4 to match the coarsest hash resolution and avoid coarse-entry gradient bottlenecks without reducing quality.

  • NGLOD stores collision-free feature vectors at octree vertices around the ground-truth triangle mesh instead of using hash tables, and concatenates looked-up vectors rather than summing them.
  • The selected configuration uses F=8 feature dimensions per entry, L=10 levels, and look-ups starting at level l=4, at roughly equal trainable parameters.The octree’s fixed growth factor is b=2, resulting in fewer levels than the hash encoding.
  • Starting look-ups at level l=4 matches the hash tables’ coarsest resolution N_min=16 and avoids a GPU gradient-accumulation bottleneck without reducing quality.Looking up the entire hierarchy would cause all GPU threads to atomically accumulate gradients in few coarse entries.

C REAL-TIME SDF TRAINING DATA GENERATION

SDF training requires rapid generation of large volumes of ground-truth signed distances from high-resolution meshes to avoid becoming a bottleneck.

  • C REAL-TIME SDF TRAINING DATA GENERATION: ∼millions per second of ground-truth signed distances must be generated from high-resolution meshes to prevent an SDF training bottleneck.The data-generation process must support a large number of samples at this rate.

C.1 Efficient Sampling of 3D Training Positions · C.2 Efficient Signed Distances to the Triangle Mesh

The method samples training positions using a mixture of cube, mesh-surface, and perturbed-surface distributions, then computes signed mesh distances with BVH queries and ray-based sign determination. GPU-specific implementations accelerate both sampling and ray-shape intersections.

  • C.1 Efficient Sampling of 3D Training Positions: Training positions allocate 1/8 uniformly in the unit cube, 4/8 uniformly on the mesh surface, and 3/8 as perturbed surface samples.This distribution follows prior work by Takikawa et al. (2021).
  • C.1 Efficient Sampling of 3D Training Positions: Uniform cube samples are generated with a GPU implementation of the PCG32 pseudorandom number generator.The unit-cube sampling routine is described as trivial to generate with any pseudorandom number generator.
  • C.1 Efficient Sampling of 3D Training Positions: Surface samples select triangles proportional to precomputed area using a cumulative distribution function and binary-search inversion, then apply standard sample warping.Triangle areas are normalized into a probability distribution before constructing the CDF array.
  • C.1 Efficient Sampling of 3D Training Positions: Perturbed surface samples add a random 3D vector whose independently drawn dimensions follow a logistic distribution with standard deviation r/1024.Here, r is the mesh’s bounding radius; the logistic distribution is chosen for its Gaussian-like shape and lower computation cost.
  • C.1 Efficient Sampling of 3D Training Positions: For NGLOD training, cube sampling is replaced by rejection sampling of octree leaf nodes followed by uniform sampling within each selected voxel.This reduces the frequency of training positions generated outside octree leaf nodes.
  • C.2 Efficient Signed Distances to the Triangle Mesh: A triangle BVH provides efficient unsigned distance queries for each sampled position, with average complexity O(log N_triangles).The signed-distance computation begins by querying the triangle mesh through this hierarchy.
  • C.2 Efficient Signed Distances to the Triangle Mesh: The method signs distances by tracing 32 uniformly sphere-distributed Fibonacci-lattice stab rays with an independent pseudorandom offset per position.A position is outside with positive distance if any ray reaches infinity; otherwise its distance is negative.
  • C.2 Efficient Signed Distances to the Triangle Mesh: OptiX 7 uses NVIDIA ray-tracing hardware for ray-shape intersections and is over an order of magnitude faster than the triangle BVH on an RTX 3090 GPU.This hardware path is used for maximum efficiency during sign determination.

D BASELINE MLPS WITH FREQUENCY ENCODING · E ACCELERATED NERF RAY MARCHING · E.1 Ray Marching Step Size and Stopping

The appendix specifies frequency-encoded MLP baselines and an accelerated NeRF ray-marching scheme. It uses nonlinear, scene-dependent stepping with empty-space skipping, sample compaction, and transmittance-based termination to reduce computation while preserving quality.

  • D BASELINE MLPS WITH FREQUENCY ENCODING: The SDF, NRC, and NeRF baselines replace hash encoding with sine/cosine or triangle-wave frequency encodings and use larger MLPs.For NeRF, the listed architecture numbers correspond first to the density MLP and second to the color MLP.
  • D BASELINE MLPS WITH FREQUENCY ENCODING: The SDF baselines use relative L2 loss and perturb samples with standard deviation r/128, producing a smoother loss landscape and better reconstruction.These changes replace the main-text MAPE loss and the Appendix C.1 perturbation value r/1024.
  • D BASELINE MLPS WITH FREQUENCY ENCODING: Frequency-encoded MLPs offer favorable performance-versus-quality trade-offs, while equal-parameter or equal-throughput comparisons make pure MLPs impractical or understate reconstruction quality.Pure MLPs scale as O(n^2), whereas trainable encodings scale sub-linearly.
  • D BASELINE MLPS WITH FREQUENCY ENCODING: Fourier features did not improve results compared with the previously described axis-aligned frequency encodings.
  • E ACCELERATED NERF RAY MARCHING: The accelerated NeRF marcher combines exponential stepping, empty-space and occlusion skipping, and dense-buffer sample compaction with imperceivable error.These techniques target the marching scheme’s impact on NeRF performance and efficient execution.
  • E.1 Ray Marching Step Size and Stopping: For unit-cube synthetic NeRF scenes, the method uses a fixed step size equal to Δt.The scenes are bounded to the unit cube [0, 1]^3.
  • E.1 Ray Marching Step Size and Stopping: For other scenes, step size is proportional to ray distance, clamped by scene scale; its exponential growth makes cost logarithmic in scene diameter without perceivable quality loss.The proportional rule is Δt := t/256, with clamping involving the largest axis size s of the scene bounding box.
  • E.1 Ray Marching Step Size and Stopping: Ray marching stops when transmittance falls below ε = 10^-4, after which the remaining contribution is set to zero.

E.2 Occupancy Grids · E.3 Number of Rays Versus Batch Size

The method uses multiscale occupancy grids to skip empty-space ray-marching samples while updating occupancy estimates during training. It also uses large fixed-size batches of as many rays as occupancy permits, improving convergence speed and quality.

  • E.2 Occupancy Grids: K=1 grid serves synthetic NeRF scenes, while K∈[1, 5] grids cover larger real-world scenes; each grid has resolution 128^3.The grids span geometrically growing domains centered around (0.5, 0.5, 0.5).
  • E.2 Occupancy Grids: Single-bit occupancy cells are stored in Morton order, enabling memory-coherent DDA traversal and skipping samples whose cell bit is low.This culls samples during ray marching when their cells are marked unoccupied.
  • E.2 Occupancy Grids: The queried grid is the finest grid covering sample position x whose cell side length exceeds the step size Δt.Grid selection depends jointly on sample position and ray-marching step size.
  • E.2 Occupancy Grids: Every 16 training iterations, floating-point density grids are decayed by 0.95, updated from M sampled candidate cells, and thresholded into occupancy bits.Candidate cells receive the maximum of their current value and the NeRF density at a random within-cell location.
  • E.2 Occupancy Grids: During the first 256 training steps, M=K·128^3 cells are sampled uniformly without repetition; afterward, M=K·128^3/2 samples combine uniform and rejection sampling.The strategy changes because occupancy grids are unreliable early in training.
  • E.3 Number of Rays Versus Batch Size: Larger ray batches incorporate more viewpoint variation and reach lower error in fewer steps, so the implementation fills fixed-size batches with as many rays as variable occupancy permits.The number of samples per ray varies because occupancy changes the ray-marching workload.
  • E.3 Number of Rays Versus Batch Size: A batch size of 256 Ki produced the fastest wall-clock convergence and was 4× smaller than mip-NeRF’s choice.The authors attribute this difference likely to mip-NeRF requiring more samples per ray, while noting that other implementation differences prevent a definitive conclusion.
  • E.3 Number of Rays Versus Batch Size: The frequency-encoding baseline produces fewer samples than hash encoding because finer hash-encoded detail leaves surrounding empty space below occupancy-grid resolution and uncullable.Those regions must therefore be traversed by extra ray-marching steps.
Loading 2201.05989v2…