PaperlayerView exact revision on arXiv

Source-linked AI summary

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, Ren Ng

arXiv:2006.10739v1cs.CVcs.LG

TL;DR

Standard coordinate-based MLPs struggle to learn high-frequency functions in low-dimensional domains. The paper uses Fourier feature mappings to tune the neural tangent kernel’s frequency spectrum, substantially improving MLP performance across computer vision and graphics regression tasks.

  • Problem

    Standard coordinate-based MLPs are poorly suited to low-dimensional tasks because spectral bias makes high-frequency components extremely slow or effectively impossible to learn.

  • Method

    Fourier feature mappings transform the MLP’s neural tangent kernel into a stationary kernel with tunable frequency bandwidth, using randomly sampled features whose scale controls the learned frequencies.

  • Results

    Fourier feature mappings dramatically improve coordinate-based MLP performance across a range of low-dimensional computer vision and graphics tasks.

  • Takeaways & Limitations

    Tuning Fourier feature parameters provides practitioners a simple strategy to improve coordinate-based MLP results in graphics and imaging applications.

  • Takeaways & Limitations

    In multiple dimensions, densely sampling Fourier basis functions is infeasible because the number of basis functions grows exponentially with dimension, motivating random feature sampling.

Abstract

from arXiv · show

We show that passing input points through a simple Fourier feature mapping enables a multilayer perceptron (MLP) to learn high-frequency functions in low-dimensional problem domains. These results shed light on recent advances in computer vision and graphics that achieve state-of-the-art results by using MLPs to represent complex 3D objects and scenes. Using tools from the neural tangent kernel (NTK) literature, we show that a standard MLP fails to learn high frequencies both in theory and in practice. To overcome this spectral bias, we use a Fourier feature mapping to transform the effective NTK into a stationary kernel with a tunable bandwidth. We suggest an approach for selecting problem-specific Fourier features that greatly improves the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities.

1 Introduction

The paper argues that standard coordinate-based MLPs are poorly suited to low-dimensional vision and graphics tasks because spectral bias limits their ability to learn high-frequency functions. It introduces Fourier feature mappings, which transform the NTK into a tunable stationary kernel and improve MLP performance when their scale is appropriately chosen.

  • Motivation: Coordinate-based MLPs replace discrete object, geometry, and appearance representations with continuous functions parameterized by deep fully-connected networks.They typically take low-dimensional coordinates, such as points in R3, as inputs and output shape, density, or color.
  • Motivation: Standard coordinate-based MLPs struggle with high-frequency functions because their kernels exhibit rapid frequency falloff, a phenomenon known as spectral bias.This frequency falloff prevents effective representation of high-frequency content in natural images and scenes.
  • Fourier features: A Fourier feature mapping transforms the NTK into a stationary kernel whose spectrum can be tuned by modifying the frequency vectors b_j.This controls the range of frequencies that the corresponding MLP can learn.
  • Results: Random Fourier features with an appropriately chosen scale dramatically improve coordinate-based MLP performance across many low-dimensional computer vision and graphics tasks.The simple strategy a_j = 1 with isotropically sampled b_j achieves good performance, while the sampling distribution’s standard deviation matters.
  • Results: Fourier feature mappings overcome coordinate-based MLPs’ low-frequency spectral bias by enabling them to learn much higher frequencies.The paper supports this claim using NTK theory and simple experiments.

2 Related Work

This section situates the work among coordinate-based MLPs and positional encodings used for visual signals, language, and time series. It extends prior Fourier-feature approaches by explaining their effect through the NTK and analyzing multidimensional coordinates and convergence.

  • Related applications: Coordinate-based MLPs are widely used to represent images and 3D scenes, with sinusoidal positional encodings improving novel-view-synthesis performance.The referenced encodings use logarithmically spaced, axis-aligned frequencies.
  • Prior positional encodings: Prior NLP and time-series studies used positional encodings for time or 1D position, while Xu et al. used RFFs to approximate stationary kernels and tune mapping parameters.This work builds on those sinusoidal input mappings.
  • Contribution: The work extends these approaches by interpreting the mappings as modifications of the network’s NTK and addressing multidimensional coordinate embeddings for vision and graphics.The multidimensional embedding is presented as necessary for vision and graphics tasks.
  • Theoretical framework: Its analysis uses infinite-width, infinitesimal-learning-rate theory that models neural networks as kernel regression through the NTK.The section relies on analyses showing that gradient-descent outputs remain close to those of a linear dynamical system.
  • Theoretical framework: Fourier feature mappings significantly alter the NTK eigenvalue spectrum and the corresponding network’s convergence properties in practice.This conclusion follows from analyzing the mapping added before the MLP.

3 Background and Notation

This section establishes kernel regression and its connection to infinite-width fully connected networks through the neural tangent kernel (NTK). It explains that NTK eigenvalues determine learning rates, causing conventional MLPs to learn high-frequency components extremely slowly.

  • Kernel regression: Kernel regression estimates a function as a similarity-weighted sum of training labels using a symmetric positive semidefinite kernel.The kernel Gram matrix contains pairwise similarities between training inputs.
  • Approximating deep networks with kernel regression: As network width tends to infinity and the SGD learning rate tends to zero, a fully connected network converges during training to kernel regression using its NTK.For hyperspherical inputs, the MLP NTK can be represented as a dot product kernel.
  • Spectral bias when training neural networks: In the NTK eigenbasis, the ith absolute-error component decays approximately exponentially at rate ηλ_i, so larger-eigenvalue components are learned faster.The NTK matrix is positive semidefinite and admits an eigendecomposition K = QΛQ^T with λ_i ≥ 0.
  • Spectral bias when training neural networks: Conventional MLP NTK eigenvalues decay rapidly, producing extremely slow convergence for high-frequency target components and making them effectively unlearnable.This spectral bias motivates analyzing Fourier feature mappings in later sections.

4 Fourier Features for a Tunable Stationary Neural Tangent Kernel

For low-dimensional regression with densely distributed Euclidean coordinates, Fourier feature mappings make the composed neural tangent kernel stationary and tunable. Training the coordinate-based MLP then corresponds to kernel regression whose frequency response governs reconstructed signals.

  • Low-dimensional regression setting: Low-dimensional tasks use densely distributed coordinates in a subset of R^d, unlike conventional analyses of sparse, high-dimensional inputs.This setting motivates treating the network through kernel regression and seeking shift invariance over the input domain.
  • Fourier features and stationarity: A Fourier feature mapping γ featurizes input coordinates before the coordinate-based MLP and makes the composed NTK stationary, or shift-invariant.Stationarity means the resulting kernel depends only on the difference between input points.
  • Fourier features and stationarity: The mapping represents a stationary kernel through Fourier basis frequencies b_j and corresponding Fourier series coefficients a_j^2.The induced kernel follows from the trigonometric identity cos(α − β) = cos α cos β + sin α sin β.
  • Kernel-regression interpretation: Training the MLP on embedded inputs corresponds to kernel regression with the stationary composed NTK h_NTK ◦ h_γ.The network output is approximated by the composed NTK convolved with weighted Dirac deltas at the training inputs, where w = K^-1y.
  • Kernel-regression interpretation: The composed NTK acts like a reconstruction filter, and its frequency decay determines the behavior of the reconstructed signal.This connects the method to signal processing and motivates analyzing the kernel’s frequency response.

5 Manipulating the Fourier Feature Mapping

Manipulating Fourier feature parameters makes the composed NTK stationary and tunable, changing convergence and generalization in 1D regression. Lower mapping p values increase high-frequency NTK power, while practical random-feature performance depends more on frequency scale than distribution family.

  • Overview: Fourier feature parameters aj and bj make the composed NTK stationary and tunable, altering network convergence and generalization in 1D regression.The section investigates these effects for coordinate-based MLPs on 1D functions.
  • Experimental setup: On linearly sampled periodic domains, the stationary composed NTK matrix is circulant, representing a convolution and allowing Fourier-based eigenvalue computation.The training set uses every cth point from cn linearly spaced samples, with the remaining points used for testing.
  • Visualizing the composed NTK: Lower p values produce a slower NTK spectral falloff and more power in higher frequencies, enabling networks to learn higher-frequency functions.The basic mapping corresponds to p = ∞ and wraps [0, 1) around the unit circle.
  • Effects on convergence: NTK linear dynamics accurately predict how changing Fourier feature mappings affects convergence, with narrower spectra converging faster on low-frequency error components.The experiment uses 4-layer, 1024-channel ReLU MLPs fitting a bandlimited 1/f^1 noise signal with c = 8 and n = 32.
  • Tuning Fourier features in practice: In higher dimensions, dense Fourier bases are infeasible because their size scales with training points and grows exponentially with dimension, so random features are sampled parametrically.The exact distribution family matters less than its scale, and Figure 4 finds sparse sampling can perform as well as dense features while distribution width matters more than shape.

6 Experiments

Experiments across direct and indirect low-dimensional regression tasks validate Fourier feature mappings for coordinate-based MLPs. All mappings improve performance over no input mapping, with Gaussian random Fourier features performing best.

  • Overall results: Fourier feature mappings improve coordinate-based MLP performance across regression tasks, and Gaussian RFF performs best.The experiments cover tasks relevant to computer vision and graphics.
  • Experimental design: The evaluation includes direct regression and indirect regression through forward models, with Fourier-feature scales tuned on held-out signals.Direct supervision matches network outputs to labels, whereas indirect supervision compares forward-model observations with supervision labels.
  • Experimental design: Table 1 compares no mapping, Basic, positional encoding, and Gaussian Fourier maps using PSNR, except 3D shape regression, which uses IoU.Higher is better for both metrics.
  • Tasks: The tasks span 2D image regression, 3D shape regression, 2D computed tomography, 3D MRI, and 3D inverse rendering for view synthesis.These tasks include both direct image or occupancy prediction and indirect reconstruction from projections, Fourier coefficients, or rendered views.
  • Tasks: For 2D image regression, MLPs predict RGB values from pixel coordinates and train on one quarter of image pixels before testing on the remainder.The study uses natural-image and text-image datasets.

7 Conclusion · A Further experiments

The paper concludes that Fourier feature mappings overcome the spectral bias of coordinate-based MLPs in low-dimensional function modeling. Tuning Fourier feature parameters controls NTK frequency falloff and improves performance across graphics and imaging tasks.

  • 7 Conclusion: Fourier feature mappings make coordinate-based MLPs better suited to modeling low-dimensional functions.The conclusion attributes this improvement to leveraging neural tangent kernel (NTK) theory.
  • 7 Conclusion: Fourier feature mappings overcome the spectral bias inherent in coordinate-based MLPs.
  • 7 Conclusion: NTK theory provides the framework for explaining why Fourier feature mappings improve coordinate-based MLPs.
  • 7 Conclusion: Tuning Fourier feature parameters controls the frequency falloff of the combined NTK.
  • 7 Conclusion: The experiments show that Fourier feature parameter tuning significantly improves performance across graphics tasks.
  • 7 Conclusion: The experiments also show significant performance improvements across imaging tasks.

A.1 Optimizing validation error through the NTK linear dynamics … B.2 Main text Figure 4 (different random feature distributions in 1D)

The paper uses NTK-based analysis and experiments to optimize Fourier feature mappings, explain depth-dependent feature sparsity, and characterize underfitting, overfitting, and distributional effects across regression tasks. These results also show limits of jointly learning features and axis-aligned positional encodings.

  • A.1 Optimizing validation error through the NTK linear dynamics: NTK linear dynamics predict validation loss as a function of Fourier feature parameters, enabling feature mappings to be optimized for signal classes.Gradient descent on the theoretical loss produces feature values matching the performance of the best power-law choices for three 1/f^α noise families.
  • A.2 Feature sparsity and network depth: Deeper MLPs achieve similar performance with fewer Fourier features because their NTKs spread each discrete frequency over a broader local spectral neighborhood.The frequency-spreading effect helps sparse Fourier features reconstruct complex signals that sparse Fourier regression or a coordinate-based MLP reconstruct poorly.
  • A.3 Gradient descent does not optimize Fourier features: Jointly optimizing Fourier feature parameters and network weights with Adam does not improve performance over keeping the feature mapping fixed.The sampled frequencies remain close to their initial values across multiple Gaussian initialization scales.
  • A.4 Visualizing underfitting and overfitting in 2D: Increasing Fourier-feature scale produces an underfitting-to-overfitting transition in both 1D and 2D regression, with low scales oversmoothing and high scales adding noise.For the Natural image dataset, σ = 10 performs best in the reported 2D experiment.
  • A.5 Failures of positional encoding (axis-aligned bias): Isotropic Gaussian random Fourier features fit sinusoidal images consistently across angles, whereas axis-aligned positional encoding performs worse away from coordinate axes.Positional encoding is suited to axis-dominant signals but is less effective for natural signals.
  • B.1 Main text Figure 3 (effect of feature mapping on convergence speed): For mappings a_j = 1/j^p and b_j = j, lower p values accelerate convergence at mid and high frequencies, while p = 1 achieves the lowest test error.The denser sampling in Figure 12 makes the effect of Fourier-feature falloff easier to observe.
  • B.2 Main text Figure 4 (different random feature distributions in 1D): Figure 4 compares frequency-sampling distributions by scale, and its alternate view separates low-frequency underfitting from high-frequency overfitting using training and test errors.Low frequencies yield similar train and test errors, whereas high frequencies drive test error upward while training error approaches machine precision.
  • B.2 Main text Figure 4 (different random feature distributions in 1D): The randomized equivalent of positional encoding samples frequencies from a distribution proportional to a 1/f power-law falloff.This corresponds to the Uniform log distribution used among the Figure 4 sampling choices.

C Stationary kernels

Fourier feature mappings make the composed NTK stationary, replacing the plain MLP’s rotation-invariant but non-translation-invariant behavior with translation-invariant similarity. This suits low-dimensional graphics and imaging tasks, where objects or scenes should be modeled equally well regardless of location.

  • Motivation: A plain MLP’s NTK depends on input dot products and norms, making it rotation-invariant but not translation-invariant.Translation-invariance is desirable when modeling objects or scenes regardless of their location.
  • Motivation: Stationary kernels measure similarity through input differences, matching low-dimensional settings with dense, nearly uniform sampling.Regression with a stationary kernel corresponds to reconstruction with a convolution filter.
  • Kernel transformation: The sinusoidal mapping transforms a dot product kernel into a stationary kernel: k(γ(u), γ(v)) = h(γ(u)^Tγ(v)) = h̃(u − v).This makes the resulting kernel better suited to the low-dimensional regime.
  • Empirical illustration: Basic Fourier embedding enables an MLP to fit shifted 1D Gaussian density functions equally well, whereas a plain MLP struggles with shifted functions.The plain MLP successfully fits a zero-centered function but not translated versions; the Fourier-embedded MLP removes this shift sensitivity.

D Indirect supervision through a linear map · E Task details · E.1 2D image

The paper extends its NTK analysis to indirect supervision through linear forward models, showing that convergence depends on the composed kernel K A^T A K and on measurement coverage. It also specifies the implementation and 2D image-regression settings, including datasets, sampling, model architecture, and training procedures.

  • D Indirect supervision through a linear map: Indirect measurements require computing the loss after a physical forward model processes the network output, rather than comparing outputs directly with observed quantities.The paper gives image and 3D shape regression as direct-supervision examples and contrasts them with graphics and imaging applications using indirect measurements.
  • D Indirect supervision through a linear map: The indirect-supervision derivation linearizes the network around its initial parameters and analyzes continuous-time gradient-flow dynamics through the neural tangent kernel.The parameter change is denoted ω_t ≜ θ_t − θ_0, while the time-dependent NTK matrix is defined from network parameter gradients.
  • D Indirect supervision through a linear map: Training under a linear map is governed by the spectrum of K A^T A K rather than K alone; unitary A preserves direct-measurement behavior, whereas rank deficiency limits affected components.When A is not full rank, only components associated with nonzero eigenvalues of K A^T A K are trained.
  • D Indirect supervision through a linear map: For CT and MRI, A^T A is diagonalizable by the DFT matrix, with diagonal entries equal to how often each frequency is measured by the sampling patterns.Both measurement processes can be formulated as Fourier-space sampling.
  • E Task details: The task experiments use JAX and train on a single K80 or RTX2080Ti GPU, with individual MLP training lasting from 10 seconds for 2D images to 30 minutes for inverse rendering.These timings describe the reported implementation range across tasks.
  • E.1 2D image: The 2D image tasks use 512 × 512 images, train on a subsampled 256 × 256 grid, and test on an offset 256 × 256 grid.The training and testing grids have the same resolution but are spatially offset.
  • E.1 2D image: The Natural and Text datasets each contain 32 images, and feature-scale sweeps identify σ_g = 10 and σ_p = 6 as the best scales.The sweep is performed on 16 images per dataset; Natural images use Div2K center crops, while Text images contain randomly styled strings on white backgrounds.
  • E.1 2D image: Each 2D model has 4 layers, 256 channels, ReLU activation, and sigmoid output, and is trained for 2000 Adam iterations with default β_1 = 0.9, β_2 = 0.999, and ϵ = 10^-8.Learning rates are manually tuned for each dataset and method; for Natural images, Gaussian RFF and positional encoding use 10^-3, while basic and no mapping use 10^-2.

E.2 3D shape … E.5 3D inverse rendering for view synthesis

Across 3D shape regression, tomographic reconstruction, MRI, and view synthesis, Fourier feature mappings are evaluated as representations for coordinate-based MLPs. The experiments vary task-specific data, validation-based mapping scales, and reconstruction metrics across these domains.

  • E.2 3D shape: Fourier feature mappings are tested on four complex meshes, with separate coordinate-based MLPs trained to represent individual objects rather than generalize across meshes.The meshes are Dragon, Armadillo, Buddha, and Lucy, each containing hundreds of thousands of vertices.
  • E.2 3D shape: Evaluation uses both uniformly sampled points and points near mesh boundaries to assess reconstruction of fine surface details.The near-surface test points perturb randomly chosen mesh vertices with Gaussian noise of standard deviation 0.01; both test sets contain 643 points.
  • E.3 2D CT: The 2D CT task reconstructs 512 × 512 images from 20 synthetic integral projections, with test PSNR measured on the original images.Experiments use randomized Shepp-Logan phantoms and the ATLAS brain dataset, selecting mapping scales through sweeps on 8 examples.
  • E.4 3D MRI: The 3D MRI task reconstructs 96 × 96 × 96 volumes from approximately 13% of randomly sampled Fourier coefficients.The experiments use the ATLAS brain dataset and select σg = 5 and σp = 4 through a sweep on 6 examples.
  • E.5 3D inverse rendering for view synthesis: For view synthesis, a simplified tiny NeRF predicts color and volume density at 3D points and uses volumetric rendering to generate novel viewpoints.The method removes hierarchical sampling and view dependence from NeRF.
  • E.5 3D inverse rendering for view synthesis: The NeRF experiment uses 120 Lego images at 400 × 400 resolution, split into 100 training, 7 validation, and 13 test images.Validation reconstruction quality determines the mapping scale, with σg = 6.05 and σp = 1.27 performing best for this scene.
  • E.5 3D inverse rendering for view synthesis: The comparison includes the original NeRF positional encoding, which uses 8 frequencies versus 256 frequencies in the authors’ mappings and is biased toward on-axis frequency content.The Lego scene is rotated in the experiments, and held-out-image PSNR is reported because the ground-truth radiance field is unavailable.

F Additional results figures

Additional figures present results across 2D image regression, 3D shape occupancy, 2D CT, 3D MRI, and inverse rendering tasks.

  • 2D image regression: Additional 2D image regression results cover three Natural-dataset images and two Text-dataset images.These results are shown in Figure 15.
  • 3D shape occupancy: Additional results are provided for the 3D shape occupancy task.The task is associated with reference [24].
  • 2D CT: Results are provided for the 2D CT task.These results are shown in Figure 17.
  • 3D MRI: Additional results are provided for the 3D MRI task.These results are shown in Figure 18.
  • Inverse rendering: Additional results are provided for the inverse rendering task.The task is associated with reference [27].
Loading 2006.10739v1…