PaperlayerView exact revision on arXiv

Source-linked AI summary

BACON: Band-limited Coordinate Networks for Multiscale Scene Representation

David B. Lindell, Dave Van Veen, Jeong Joon Park, Gordon Wetzstein

arXiv:2112.04645v2cs.CVcs.GRcs.LG

TL;DR

Coordinate networks are difficult to analyze, constrain at unsupervised points, and adapt across signal scales. BACON introduces analytically band-limited coordinate networks with Fourier-characterized behavior and demonstrates multiscale representations across images, radiance fields, and 3D scenes. The method outperforms conventional single-scale coordinate networks in these multiscale tasks.

  • Problem

    Existing coordinate networks have difficult-to-analyze spectra, unpredictable behavior at unsupervised points, and artifacts when naively queried at different resolutions.

  • Method

    BACON uses analytically controlled band-limited outputs and a principled frequency initialization scheme to characterize network behavior through Fourier spectra.

  • Results

    BACON outperforms conventional single-scale coordinate networks for multiscale image fitting, neural rendering, and 3D scene representation.

  • Takeaways & Limitations

    BACON provides interpretable, scale-aware coordinate representations with constrained behavior at unsupervised locations and multiscale outputs without explicit per-scale supervision.

  • Takeaways & Limitations

    Fitting higher-dimensional signals may require more parameters for dense spectral coverage, and the current work is limited to single-scene overfitting.

Abstract

from arXiv · show

Coordinate-based networks have emerged as a powerful tool for 3D representation and scene reconstruction. These networks are trained to map continuous input coordinates to the value of a signal at each point. Still, current architectures are black boxes: their spectral characteristics cannot be easily analyzed, and their behavior at unsupervised points is difficult to predict. Moreover, these networks are typically trained to represent a signal at a single scale, so naive downsampling or upsampling results in artifacts. We introduce band-limited coordinate networks (BACON), a network architecture with an analytical Fourier spectrum. BACON has constrained behavior at unsupervised points, can be designed based on the spectral characteristics of the represented signal, and can represent signals at multiple scales without per-scale supervision. We demonstrate BACON for multiscale neural representation of images, radiance fields, and 3D scenes using signed distance functions and show that it outperforms conventional single-scale coordinate networks in terms of interpretability and quality.

1. Introduction

Coordinate networks provide continuous signal representations but are difficult to analyze, constrain, and use across scales. BACON addresses these limitations with analytically controlled spectral bandwidths and demonstrates multiscale performance across several signal types.

  • Coordinate networks map input coordinates to signal values using continuous functions embedded in neural-network weights.
  • Black-box spectral behavior makes unsupervised-coordinate predictions difficult and prevents reliable filtering or anti-aliasing for multiscale representation.
  • Multiresolution signal representation supports applications including image pyramids, optical flow, compression, filtering, mipmapping, and efficient rendering.
  • BACON analytically controls each layer’s maximum frequency and characterizes trained-network behavior through its Fourier spectrum.
  • The paper introduces band-limited coordinate networks, spectral-analysis methods, and a principled band-limited initialization scheme.
  • BACON outperforms conventional single-scale coordinate networks for multiscale image fitting, neural rendering, and 3D scene representation.

2. Related Work

Prior scene-representation methods include feature-based, coordinate-based, hybrid, and multiscale architectures. BACON extends multiplicative filter networks with analytically constrained bandwidths, differing from approaches that rely on multiscale training or scale-dependent encodings.

  • Neural scene representations encode parts, objects, or scenes continuously and can be optimized as signed distance functions, occupancy networks, or neural-rendered models.
  • Feature-based scene representations evaluate quickly but typically require substantial memory, whereas coordinate-based networks use MLPs to map coordinates to signal values.
  • Hybrid architectures combine feature-based and coordinate representations, while BACON instead builds on multiplicative filter networks rather than conventional MLPs.
  • Existing multiscale methods use octrees, feature hierarchies, specialized architectures, progressive training, or scale-dependent positional encoding.
  • BACON initializes sine-layer frequencies within limited bands and accumulates input bandwidths to specify each output layer’s bandwidth.
  • Unlike Mip-NeRF, BACON constrains output bandwidth by design and learns a band-limited multiscale decomposition without explicit training at multiple scales.

3. Method

BACON builds on multiplicative filter networks to create coordinate networks with analytically controllable bandwidth, characterized Fourier spectra, multiscale outputs, and principled initialization. Its architecture and analysis control represented frequencies, derive their distribution, support simultaneous resolution supervision, and maintain well-scaled activations during training.

  • BACON architecture: BACON extends multiplicative filter networks with multiscale, band-limited outputs, frequency formulas, and a principled initialization scheme.The architecture uses an MFN backbone but adds changes for multiscale outputs and analytical spectral control.
  • Forward pass: Each layer applies sine-filtered linear features, intermediate outputs, and linear readouts; Hadamard products combine the activations.The network maps coordinates through sine functions, Hadamard products, and output layers.
  • Fourier representation: MFN outputs reduce to sums of sines with varying amplitudes, frequencies, and phases, enabling analytical characterization of the network spectrum.The reduction follows from repeatedly applying a trigonometric identity through the network.
  • Frequency control: Freezing filter frequencies within [−B_i, B_i] bounds the output bandwidth by the sum of per-layer bandwidths, while discrete frequencies permit FFT-based spectrum characterization.For finite domains, periodicity determines the spacing of represented discrete frequencies.
  • Frequency distribution: The network’s frequency distribution is approximately zero-mean Gaussian, yielding greater parameterization of low frequencies.This distribution provides an inductive bias toward low frequencies, which often have greater amplitude in natural signals.
  • Multiscale outputs: Intermediate linear layers extract band-limited outputs for simultaneous multiscale supervision, including semi-supervised training with mismatched supervisory bandwidths.The outputs’ band limitation permits the supervisory signal’s bandwidth to differ from the desired output bandwidth.
  • Initialization scheme: The proposed initialization accounts for Hadamard products and sine-layer inputs, avoiding vanishingly small deep activations and producing standard-normal activations after linear layers.For d_h = 1024, the activation distributions closely match the analytical derivations shown in Figure 3.

4. Experiments

BACON is evaluated for image fitting, neural radiance fields, and 3D shape representation across multiple resolutions and levels of detail. Across these tasks, its band-limited outputs support multiscale reconstruction, frequency-controlled behavior, and efficient rendering.

  • BACON is demonstrated on image fitting, neural radiance fields, and 3D shape fitting using signed distance functions.
  • Neural Radiance Fields: BACON learns multiscale radiance-field representations and captures higher-frequency details than NeRF while using fewer parameters for lower-resolution rendering.The comparison covers 1/2, 1/4, and 1/8 resolution outputs on a multiscale synthetic dataset.
  • Neural Radiance Fields: BACON produces band-limited outputs at 1/8, 1/4, 1/2, and full resolution for neural rendering.Its output bandwidths are designed to correspond to the dataset’s multiple image scales.
  • 3D Shape Representation: For 3D shape fitting, BACON achieves comparable highest-detail performance while learning a smooth multiscale decomposition with band-limited signed-distance outputs.The evaluation averages Chamfer distance and intersection over union across five scenes.
  • 3D Shape Representation: Adaptive-frequency evaluation makes signed-distance-function evaluation roughly 80× faster than the naive approach.The timings are reported for multiscale grids on an NVIDIA RTX A6000 GPU.

5. Conclusion

BACON makes coordinate networks interpretable and scale aware by analytically controlling spectral bandwidth and constraining behavior at unsupervised locations. It supports multiscale representation without explicit per-scale supervision and outperforms single-scale coordinate networks across several tasks.

  • BACON enables analysis and control of spectral bandwidth at intermediate layers, supporting multiscale signal representation without explicit supervision.
  • Fourier analysis provably constrains BACON’s behavior even at unsupervised locations.
  • BACON’s intermediate outputs can improve inference times through adaptive frequency evaluation.
  • BACON outperforms other single-scale coordinate networks for multiscale image fitting, neural rendering, and 3D scene representation.

Supplemental Material

The supplemental material contains derivations, experiments, and supporting documentation for the paper. The listed material includes image-related content and the paper’s publication metadata.

  • The document identifies the authors as David B. Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein.
  • The listed version is arXiv:2112.04645v2, dated 28 Mar 2022.
  • The supplemental material includes an image-related section listed as “Images.”

1. Supplemental Derivations

The supplemental derivations analyze how multiplicative filter networks generate and distribute sine frequencies. They derive frequency-count recursions and show that BACON frequencies become approximately Gaussian distributed as depth increases.

  • The product of two sines produces two sines whose frequencies are the sum and difference of the original frequencies.
  • A multiplicative filter network’s sine count grows through layerwise multiplication, Hadamard-product doubling, and additional bias-term contributions.
  • Removing bias layers removes the additional dh term from the sine-count recursion.
  • Network frequencies form a compound distribution because represented frequencies sum between one and NL random variables.
  • As hidden-layer count increases, the frequency distribution is approximated by a Gaussian under the Central Limit Theorem.
  • Empirically, a 5-layer network with dh = 1024 and Bi = 10 produced frequencies well approximated by a Gaussian with the derived variance, especially in deeper layers.

2.1. Images

The image experiments evaluate BACON against Fourier Features, SIREN, and integrated positional encoding across multiple output scales. BACON produces anti-aliased lower-resolution outputs and interpretable band-limited upsampling while being supervised only at 1× resolution.

  • Image fitting setup: BACON learns a multiscale image decomposition from supervision at a single 1× scale, unlike baselines trained at one or multiple specified scales.
  • Image fitting setup: The evaluation uses 16 DIV2K images and reports PSNR and SSIM on training and offset validation grids.
  • Multiscale results: BACON and integrated positional encoding show the best performance on low-resolution outputs in the multiscale quantitative evaluation.
  • Multiscale results: Fourier Features and SIREN exhibit aliasing when subsampled, whereas BACON closely represents low-pass filtered images despite 1×-only supervision.
  • Upsampling behavior: BACON avoids spurious high frequencies during 4× upsampling because its output is band limited; unconstrained integrated positional encoding retains aliasing artifacts after low-pass filtering.
  • Deep BACON: 38.8 dB PSNR was reached by 8-layer BACON versus 29.8 dB for 8-layer MFN, while 16-layer MFN failed to optimize because of numerical instabilities.
  • Scale interpolation: Linear interpolation between BACON’s discrete output scales provides a kind of continuous output scale, analogous to trilinear filtering for mipmaps.

2.2. Neural Radiance Fields

BACON is evaluated for multiscale neural radiance fields, including regularized training, computational efficiency, and comparisons with NeRF and Mip-NeRF. It learns multiscale outputs and achieves favorable quality and parameter-efficiency tradeoffs at selected resolutions.

  • Regularization: BACON benefits significantly from opacity regularization, whereas NeRF and Mip-NeRF show few opacity artifacts and gain little from it.The regularizer penalizes non-zero off-surface opacity that can produce hazy rendered regions.
  • Efficiency: Roughly 30% speedup is possible without a noticeable performance drop by reusing outputs across layers with equal bandwidths.This reduces the number of input layers that need computation by roughly half.
  • Efficiency: BACON has a computational advantage for low-resolution outputs because only the first few layers require evaluation.The implementation is generally slower than NeRF and Mip-NeRF overall, partly because of framework differences and greater architectural complexity.
  • Evaluation: BACON, NeRF, and Mip-NeRF are evaluated on multiscale Blender radiance-field outputs using PSNR and SSIM across scenes.The section reports averaged results in Tables 5 and 6 and qualitative outputs in Figure 8.
  • Quality and efficiency: BACON outperforms NeRF at the lowest and highest resolution outputs while using a fraction of the parameters for low-resolution rendering.Mip-NeRF achieves the best overall performance in the reported per-scene averages.
  • Multiscale supervision: When only full-resolution images supervise training, BACON automatically learns a multiscale decomposition of the neural radiance field.Additional qualitative results are shown for this semi-supervised setting in Figure 9.

2.3. 3D Shape Representation

BACON is compared with Fourier Features, SIREN, and NGLOD for signed-distance-function shape representation. It provides smooth multiscale shape decompositions and comparable high-detail quality while scaling differently in parameter count.

  • Evaluation: BACON, Fourier Features, SIREN, and NGLOD are trained to fit signed distance functions and evaluated using Chamfer distance and intersection over union.Meshes are extracted at 512^3 resolution, while IOU is computed on a 128^3 occupancy grid.
  • Low-detail results: At lower detail levels, BACON produces smooth multiscale shape decompositions, whereas NGLOD exhibits angular artifacts associated with high-frequency spectra.The qualitative comparisons include rendered objects, normal maps, and Fourier spectra in Figures 10–14.
  • High-detail results: At the highest detail level, all methods perform similarly, with BACON matching comparable performance while representing multiple levels of detail simultaneously.The comparison covers Armadillo, Dragon, Lucy, Thai Statue, and Sphere scenes.
  • Parameter scaling: NGLOD generally improves at higher detail levels while its model parameter count increases significantly.NGLOD levels 1–4 come from a model trained to maximum level 4, while separate models are also trained for levels 5–6.
  • Parameter scaling: At lower detail levels, NGLOD has fewer parameters than BACON, while BACON generally achieves better performance.For full-resolution models, NGLOD parameter counts scale superlinearly, compared with linear scaling for BACON.

2.4. Accelerated Marching Cubes

BACON’s band-limited multiscale structure enables two acceleration strategies for Marching Cubes: adaptive-frequency evaluation and hierarchical SDF grids. Combining them further accelerates mesh extraction while exploiting SDF-based empty-space guarantees.

  • Adaptive-frequency evaluation: Adaptive-frequency evaluation computes a low-frequency SDF first and skips higher-frequency layers for cells far from the zero-level set.Cells whose low-frequency SDF magnitude exceeds threshold τ are not evaluated through the higher BACON layers.
  • Hierarchical evaluation: Hierarchical multi-scale SDF evaluation starts at coarse grids and subdivides cells only when their SDF values do not certify empty space.Under accurate SDFs, a cell is empty when the center magnitude exceeds the circumsphere radius.
  • Combined strategy: Combining adaptive-frequency evaluation with multi-scale evaluation applies early stopping at every level, including the computationally dominant finest level.The combined procedure is evaluated as a proposed acceleration strategy in Table 10.
  • Scope and comparison: The adaptive-frequency strategy relies on BACON’s band-limited outputs, while occupancy networks cannot use it and lack the same SDF empty-space guarantees.Occupancy-based octree evaluation can therefore lead to errors, and all query points require full-network evaluation for the adaptive-frequency approach.
  • Results: The proposed procedures produce extracted 3D shapes while reducing the computation required for mesh extraction.The resulting shapes are shown in Figure 17, with asymptotic comparisons in Figure 18.

2.5. Comparison to Explicit Fourier Basis

BACON is compared with explicit Fourier-transform methods for evaluating a discrete frequency grid. Its compressive spectral representation can be more efficient for few samples than computing a full inverse transform.

  • Evaluation comparison: For few samples, evaluating BACON on a 512^3 discrete-frequency grid is more efficient than IFFT or naive IDFT evaluation on an explicit coefficient grid.The comparison uses the grid from the shape-fitting experiments.
  • Complexity: IFFT and IDFT have asymptotic complexities O(N log(N)) and O(N), while BACON’s complexity scales as O(d^2h) from matrix multiplication.The comparison is plotted for n output samples in Figure 18.
  • Complexity: IFFT always computes 512^3 outputs, so its cost remains constant as the number of requested output samples changes.BACON’s compressive representation is evaluated through the network rather than reconstructing the entire explicit grid.
Loading 2112.04645v2…