PaperlayerView exact revision on arXiv

Source-linked AI summary

Variable Bitrate Neural Fields

Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, Sanja Fidler

arXiv:2206.07707v1cs.CVcs.GRcs.LGcs.MM

TL;DR

Feature grids improve neural-field representations but impose substantial memory costs, motivating compression that preserves quality while supporting variable bitrate and streaming. The paper introduces a vector-quantized auto-decoder that learns discrete feature-grid representations end to end, reporting up to two orders of magnitude less storage with relatively little visual quality loss. Training requires an additional memory and compute footprint for softened indices.

  • Problem

    High-resolution feature grids can dominate neural-field memory, despite enabling high-quality representations with smaller neural networks.

  • Method

    VQAD jointly trains feature-grid indices, a learned codebook, and a decoder MLP to learn compressed feature grids without direct supervision.

  • Results

    Two orders of magnitude reduction in feature-grid storage is achieved with relatively little visual quality loss without entropy encoding.

  • Takeaways & Limitations

    The learned representation supports progressive, variable-bitrate streaming and can scale quality with available bandwidth or desired level of detail.

  • Takeaways & Limitations

    Training requires allocating an m × 2^b matrix of softmax coefficients before converting them into inference and storage indices.

Abstract

from arXiv · show

Neural approximations of scalar and vector fields, such as signed distance functions and radiance fields, have emerged as accurate, high-quality representations. State-of-the-art results are obtained by conditioning a neural approximation with a lookup from trainable feature grids that take on part of the learning task and allow for smaller, more efficient neural networks. Unfortunately, these feature grids usually come at the cost of significantly increased memory consumption compared to stand-alone neural network models. We present a dictionary method for compressing such feature grids, reducing their memory consumption by up to 100x and permitting a multiresolution representation which can be useful for out-of-core streaming. We formulate the dictionary optimization as a vector-quantized auto-decoder problem which lets us learn end-to-end discrete neural representations in a space where no direct supervision is available and with dynamic topology and structure. Our source code will be available at https://github.com/nv-tlabs/vqad.

1 INTRODUCTION

Feature-grid neural fields improve reconstruction quality and efficiency but can consume substantially more memory than standalone neural networks. VQAD compresses these grids into a streamable, variable-bitrate representation with relatively little visual quality loss.

  • Motivation: Feature-grid methods achieve state-of-the-art reconstruction quality and interactive training or rendering by moving representation complexity from the MLP into a spatial feature structure.They embed coordinates through learned feature-grid lookups, such as sparse grids or hash tables.
  • Contribution: 60x compression reduces an uncompressed 15 207 kB feature grid, while a coarse LOD can appear after receiving 10 kB of data.The figure reports PSNR relative to training images and excludes additional entropy encoding.
  • Contribution: VQAD directly learns compressed feature grids without direct supervision and supports progressive, variable-bitrate streaming according to bandwidth or desired level of detail.The method jointly optimizes compression-aware representations for neural fields.
  • Contribution: Two orders of magnitude reduction in storage is achieved with relatively little visual quality loss without entropy encoding.The reported result concerns feature grids representing neural radiance fields.

2 RELATED WORKS

Prior work explores compression through dynamic architectures, differentiable rate-distortion optimization, quantized neural parameters, multiresolution structures, wavelets, and hash tables. These approaches provide variable bitrate or compactness through different representations, but feature-grid compression remains an active comparison point.

  • Graphics context: Streaming and compression are established requirements in graphics systems, including out-of-core meshes, massive-volume rendering, and progressive mesh representations.The cited systems use levels of detail, streaming, or compression to manage large assets and datasets.
  • Scope: Compression is widely cited as a neural-field benefit, yet relatively few works evaluate it for both standalone networks and feature-grid methods.The related-work review covers both global and feature-grid compression approaches.
  • Neural compression: Architecture-search methods use hyperparameter sweeps to select rate-distortion tradeoffs, while variable bitrate requires storing multiple models.Other work directly minimizes rate-distortion with differentiable entropy approximations or tunes different tradeoffs.
  • Neural compression: Vector quantization has been applied to MLP parameters alongside quantization at different bitwidths to provide variable bitrate.The passage places this among global neural-network compression methods.
  • Feature-grid compression: Multiresolution trees support adaptive bitrate by streaming breadth-first prefixes, while adaptive Haar wavelets learn transform coefficients for feature grids.These methods adapt representation detail through hierarchical or transformed coefficients.
  • Feature-grid compression: Hash tables provide compact fixed-size feature grids, but large tables are required for good quality.The paper presents these approaches as complementary to its contribution.

3 BACKGROUND

Neural fields represent continuous signals with coordinate-based functions, while feature-grid methods augment the MLP with learned local embeddings. Because these grids can dominate memory, the background motivates transform coding, block processing, and vector quantization as compression foundations.

  • Neural fields: A neural field is a parametric function that approximates a continuous signal and is fitted through stochastic optimization.Global methods store the MLP weights and biases, whereas feature-grid methods add learned grid parameters.
  • Feature grids: Feature-grid methods interpolate local feature vectors from a grid and provide them to the MLP alongside the queried coordinate.The nonlinear combination can reconstruct frequencies above the usual Nyquist limit, motivating coarser grids for compression.
  • Compression motivation: Feature vectors are often the most memory-hungry component because a grid of m points and k-dimensional vectors can exceed the MLP size substantially.One example uses 12.6 million feature-grid parameters and 10,000 MLP weights.
  • Compression foundations: Transform coding maps a discrete signal to transform coefficients that can be quantized or truncated after decorrelation.Linear transforms can be fixed or data-driven, including the Karhunen-Loève transform.
  • Compression foundations: Nonlinear transform coding uses a parametric function and inverse to encode and decode signals, while compressed auto-decoders adapt the auto-decoder framework for compression.The paper uses this framework as conceptual background for learning compressed feature representations.
  • Vector-quantized auto-decoder: A compact codebook representation replaces bulky feature vectors with indices, which are trained jointly with the codebook and decoder MLP.The paper describes this as the vector-quantized auto-decoder framework.
  • Compression foundations: Block-based coding reshapes long signals into smaller chunks, applies a block transform, and can cluster block rows through vector quantization.The feature-grid compression approach is inspired by this block-based procedure.

4 METHOD

VQAD compresses neural-field feature grids by jointly learning codebook indices, decoder parameters, and a reconstruction objective under indirect supervision. The method supports differentiable training, compact storage, and multiresolution streaming through vector-quantized representations.

  • Compressed auto-decoder: VQAD replaces bulky feature vectors with indices into a learned codebook, jointly training the indices, codebook, and decoder MLP.The approach avoids an encoder and can learn compressed representations with respect to arbitrary domains, including indirectly supervised neural radiance fields.
  • Compressed auto-decoder: The auto-decoder reconstructs feature-grid coefficients using supervision in a different domain from the target signal, such as 2D rendered images for radiance fields.A differentiable forward map lifts the signal into the supervision domain; for radiance fields, this map is a differentiable renderer.
  • Feature-grid compression: Feature-grid compression substitutes a block-based inverse transform into the neural-field pipeline, where interpolated local embeddings are passed to an MLP and then reconstructed by a forward map.Each feature-grid row acts as a local block, and the learned inverse transform produces the compressed feature grid.
  • Vector quantization: Vector quantization represents the compressed grid with integer indices V∈Z^m that select entries from a codebook D∈R^(2^b×k).The decoder uses codebook indexing D(V)=D[V], with b controlling the index bitwidth.
  • Vector quantization: A softened matrix C makes codebook selection differentiable during training, while a straight-through estimator preserves awareness of hard indexing.At storage and inference, C is discarded and only the integer vector V remains; the resulting compression ratio is 16mk/(mb+k2^b).
  • Multiresolution representation: Arranging the compressed indices in a multiresolution sparse octree enables streaming level of detail across tree levels.Feature vectors from levels can be summed or concatenated before entering the MLP, with a separate codebook trained for each level.

5 EXPERIMENTS

Experiments evaluate VQ-AD against feature-grid, post-processing, hash-based, and neural-field baselines across radiance-field, geometry, compression, and streaming settings. The results show that learning indices and multiresolution representations improves quality or enables bitrate adaptation, while introducing artifacts or training-resource costs in some settings.

  • 5.1 Baseline and Implementation Details: NGLOD-NeRF provides a strong baseline with reconstruction quality comparable to NeRF, mip-NeRF, and Plenoxels.The evaluation uses RTMV scenes and reports LPIPS, SSIM, and PSNR at 400 × 400 resolution.
  • 5.1 Baseline and Implementation Details: The evaluation initializes NGLOD’s octree from calibrated depth maps and does not test refining the octree during training without such depth information.The authors note that compared reference methods did not use depth during training, so the comparisons serve mainly as quality reference points.
  • 5.2 Feature Grid Compression: Learning vector quantization end-to-end improves quality over low-rank approximation and post-processing k-means quantization at comparable storage.The post-processing baselines use half-precision values without entropy coding, and k-means produces noticeable discoloration.
  • 5.2 Feature Grid Compression: VQ-AD compresses signed distance functions with significant bitrate reduction, although the resulting TSDF representations show visible high-frequency or normal artifacts.A quantized Draco mesh provides a comparison at a similar entropy-coded bitrate, including a 2 MB decompressed binary .ply mesh.
  • 5.3 Random vs. Learned Indices: Learned indices reconstruct with less noise than random hash-style indices at roughly equal storage cost, despite using smaller codebook bitwidths.The qualitative comparison contrasts 4-bit learned indices with 12-bit hash-based codebooks.
  • 5.4 Streaming Level of Detail: VQ-AD jointly filters and compresses multiresolution representations, allowing bitrate reduction at lower resolutions for progressive streaming and level of detail.Its rate-distortion representation supports variable bitrate, while Mip-NeRF’s filtering changes quality without changing bitrate.

6 CONCLUSION

The paper concludes that VQ-AD provides a streamable, compressive neural representation with minimal visual quality loss, while its main drawback is substantial training-time memory and compute.

  • VQ-AD learns a streamable, compressive representation with minimal visual quality loss, supporting filtering and compression in graphics systems.
  • The method’s major drawback is its training-time memory and compute footprint from storing an m × 2^b softmax-coefficient matrix.

7 IMPLEMENTATION DETAILS

The implementation adapts NGLOD for neural radiance fields and uses a sparse multiresolution feature grid, custom volumetric integration, and sampled level-of-detail training.

  • 7.1 Minor Modifications to NGLOD: The radiance-field implementation modifies NGLOD feature lookup so coarsest-level locations remain sampleable while missing finer levels return zero vectors.The zero vectors are a no-op when summed in the sparse Laplacian pyramid.
  • 7.2 Architectural Hyperparameters: The model uses 16-dimensional feature vectors, four grid resolutions from 25 through 28, and a 2-layer network with hidden size 128 and four outputs.The feature vector is combined with a 27-dimensional positional encoding of view direction.
  • 7.3 Initialization: The sparse grid is initialized from camera rays and pixelwise depths, then normalized into a cube with range [−1, 1].
  • 7.4 Optimization: Training and evaluation use sRGB images, area-weighted bilinear downscaling, premultiplied ground-truth alphas, and Adam optimization with a 100× feature-grid learning-rate scale.
  • 7.5 Level-of-Detail Training: For multiple levels of detail, the method samples one level per batch, weighting coarser levels twice as likely as each subsequent level.Training only on the finest level still produces lower-level effects, but with compromised lower-LOD quality.

8 OTHER EXPERIMENTAL DETAILS

The compressed model has similar inference speed and memory to the uncompressed baseline, but higher training cost; entropy coding provides limited gains unless quality and streaming are sacrificed.

  • 8.1.1 Inference: Inference runs at around 15 FPS at 720p with 8 GB memory for both uncompressed NGLOD-NeRF and compressed 4- and 6-bit models.The measurements depend heavily on viewpoints, batching, and implementation choices.
  • 8.1.2 Training: 600 training epochs take around 20 minutes for the uncompressed model and around 40 minutes for the compressed model.Both models achieve PSNR above 30 within 50 epochs.
  • 8.1.2 Training: Peak training memory is 8 GB for the uncompressed model, 8 GB for 4-bit compression, and 18 GB for 6-bit compression.Inference memory is not affected by the high training memory usage.
  • 8.2 Entropy Coding: Entropy coding reduces uncompressed weights by 7% and compressed weights by 4% in the reported experiments.The trained indices are somewhat uniformly distributed, limiting entropy-coding gains.
  • 8.2 Entropy Coding: Entropy minimization can yield up to a 56% size reduction through entropy coding, but causes a large quality drop and prevents streaming.
Loading 2206.07707v1…