Source-linked AI summary
A Structured Dictionary Perspective on Implicit Neural Representations
Gizem Yüce, Guillermo Ortiz-Jiménez, Beril Besbinar, Pascal Frossard
TL;DR
Although INRs are practically successful, how they represent signals remains poorly understood. The paper analyzes their expressive power and inductive bias through harmonic and NTK dictionary perspectives, finding that harmonic structure supports broad frequency representations and meta-learning increases NTK encoding efficiency.
Problem
The mechanisms behind INR signal representation remain poorly understood despite their practical success, limiting clear principles for their analysis and design.
Source: §1 · p. 1Method
Results
INRs express integer-harmonic combinations with broad frequency support, while meta-learning modifies the NTK and increases its signal-encoding efficiency.
Source: §3 · pp. 3–4, §7 · p. 8Takeaways & Limitations
The unified analysis is intended to inform INR architecture design and tuning and may also interest the wider deep-learning theory community.
Source: §1 · p. 2Most practical results use one- or two-dimensional signals, and the effect of out-of-distribution data on post-meta-training NTK alignment remains for future study.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Implicit neural representations (INRs) have recently emerged as a promising alternative to classical discretized representations of signals. Nevertheless, despite their practical success, we still do not understand how INRs represent signals. We propose a novel unified perspective to theoretically analyse INRs. Leveraging results from harmonic analysis and deep learning theory, we show that most INR families are analogous to structured signal dictionaries whose atoms are integer harmonics of the set of initial mapping frequencies. This structure allows INRs to express signals with an exponentially increasing frequency support using a number of parameters that only grows linearly with depth. We also explore the inductive bias of INRs exploiting recent results about the empirical neural tangent kernel (NTK). Specifically, we show that the eigenfunctions of the NTK can be seen as dictionary atoms whose inner product with the target signal determines the final performance of their reconstruction. In this regard, we reveal that meta-learning has a reshaping effect on the NTK analogous to dictionary learning, building dictionary atoms as a combination of the examples seen during meta-training. Our results permit to design and tune novel INR architectures, but can also be of interest for the wider deep learning theory community.
1. Introduction
INRs offer compact, continuous signal representations, but their mechanisms and design principles remain poorly understood. The paper introduces a unified perspective for analyzing their representation and behavior.
1. Introduction
- INRs parameterize continuous coordinate-to-signal mappings with neural networks, enabling compact representations that integrate into differentiable learning pipelines.They are presented as an alternative to classical discretized signal representations.Source: §1 · p. 1
- Periodic representations can mitigate standard networks’ bias toward low frequencies, helping INRs represent signals with high-frequency content.The introduction links their use to applications including surface representation, volume rendering, and generative modelling.Source: §1 · p. 1
- Despite practical success, INR design lacks clear selection criteria; parameters are often heuristic and limitations remain poorly understood.The paper identifies these gaps as slowing further research development.Source: §1 · p. 1
1. What is the expressive power of INRs?
The paper analyzes INR expressive power and inductive bias through a unified theoretical perspective. It connects harmonic structure, NTK eigenfunctions, and meta-learning to signal-dictionary representations.
1. What is the expressive power of INRs?
- Most INRs have expressive power equivalent to a structured signal dictionary whose atoms are integer harmonics of their input-mapping frequencies.This perspective is presented as a way to unify INR architectures and understand shared problems.Source: §1 · p. 2
- The empirical NTK’s eigenfunctions act like dictionary atoms, and signal-learning difficulty depends on how efficiently those atoms encode the target.The paper builds on prior work [56] and recent deep-learning theory.Source: §1 · p. 2
- Meta-learning reshapes the NTK into a signal dictionary whose atoms combine examples from meta-training, increasing target-signal encoding efficiency and improving performance and training speed.The paper relates this effect to dictionary learning.Source: §1 · p. 2
- The framework is intended to support principled INR design and tuning, with the meta-learning analysis also relevant to deep-learning theory.The paper frames these as potential impacts of its findings.Source: §1 · p. 2
2. Implicit Neural Representations
INRs represent signals by mapping coordinates to values with neural networks, often using a coordinate mapping followed by an MLP. Their diversity and limited theoretical understanding motivate a unified analysis.
2. Implicit Neural Representations
- An INR learns a coordinate-to-signal mapping by minimizing a distortion measure such as mean-squared error, typically with gradient descent.Coordinates may represent pixels and signal values may represent RGB colors.Source: §2 · p. 2
- Continuous parameterization keeps memory cost constant across spatial resolutions, but reconstructing high-frequency details remains a challenge.The section notes that standard networks’ low-frequency spectral bias has motivated architectures designed to improve convergence and fidelity.Source: §2 · p. 2
- Most INR architectures can be formulated as an input mapping followed by an MLP with layer-specific weights, biases, and activation functions.The formulation applies the activation elementwise at each hidden layer.Source: §2 · p. 2
- Fourier feature networks use a Fourier input mapping with ReLU activations, while SIRENs use sinusoidal activations.Fourier features may use random or deterministic frequency initialization; larger random-feature scales bias FFNs toward higher frequencies.Source: §2 · pp. 2–3
- Despite effectiveness on natural and visual signals, INR architectures have limited theoretical explanation, and their diverse design principles complicate analysis.The paper proposes a unified perspective to analyze expressive power and inductive bias and to inform design and tuning.Source: §2 · p. 3
3. Expressive Power of INRs
The paper characterizes INR expressive power through harmonics of the input mapping. This structure permits broad frequency support with compact parameterization, while activation and parameter choices shape the represented spectrum.
3. Expressive Power of INRs
- INRs with sinusoidal mappings and polynomial activations represent linear combinations of sinusoids at sums and differences of integer harmonics of mapping frequencies.The theorem’s formulation assumes polynomial activations; the paper notes that analytic activations and ReLUs can be approximated by polynomials.Source: §3 · p. 3
- The multilayer structure imposes low-rank structure on harmonic coefficients, reducing storage needs compared with directly learning every coefficient.The passage contrasts O(TKL) coefficients with O(T 2L) parameters.Source: §3 · p. 3
- Layered sinusoidal activations can produce signals supported by infinitely many frequency harmonics using only a few coefficients.This follows from expanding the nested sinusoids into a harmonic series.Source: §3 · p. 4
- Bessel coefficients make higher-order harmonics weaker, concentrating output energy near input frequencies; increasing inner-layer coefficient scale strengthens higher harmonics and broadens frequency coverage.The section relates this modulation to frequency-modulation signals.Source: §3 · p. 4
- The shared harmonic structure across INR families may explain both their practical expressive power and common failure modes.The paper identifies imperfect recovery and aliasing as failure modes for further analysis.Source: §3 · p. 4
4. Failure modes of INRs
INRs’ harmonic expansion can produce imperfect recovery when the initial mapping frequencies do not cover the target spectrum, and aliasing when the implicit bias selects a fit that fails away from training samples. The section illustrates these failure modes and explains how frequency choices affect them.
4.1. Imperfect recovery
- With f0 = 1, an FFN represents only even multiples of 2π/N, producing displaced image copies and a reconstruction spectrum that misses odd coefficients.The restricted frequency support follows from the initial mapping and does not cover all spectral coefficients.Source: §4.1 · p. 5
- Setting f0 = 0.5 removes this artifact and permits energy at all spectral coefficients, but the image remains blurry because high-frequency harmonic weights decay rapidly.Increasing the initial mapping’s frequency coverage can alleviate the blur; larger T and σ can reduce the chance of limited frequency coverage, while σ that is too large can introduce other problems.Source: §4.1 · p. 5
4.2. Aliasing
- At ω0 = 300 rad/s, a SIREN fits the 128 Hz training samples exactly yet adds a 105 Hz spectral peak at 256 Hz; at ω0 = 30 rad/s, that aliased peak disappears.The figure compares the target and reconstruction spectra at the two sampling rates and initialization settings.Source: Figure 3 · p. 5, §4.2 · p. 5
- High fundamental frequencies include higher components without relying on high-order harmonics, but give those components larger weights and increase aliasing risk.The section attributes this to the rapid decay of coefficients for common nonlinearities such as ReLU or sine.Source: §4.2 · p. 5
- Because low-rate samples cannot distinguish aliased high-frequency components from lower frequencies, the underspecified reconstruction problem can impair generalization beyond training support.The authors identify super-resolution and scene reconstruction as applications that require such generalization.Source: §4.2 · pp. 5–6
5. Inductive bias of INRs
The section interprets INR learnability through the empirical NTK: its eigenfunctions act as dictionary atoms, and target alignment with those atoms helps explain which signals are easier to learn. It uses this perspective to analyze SIREN frequency choices and meta-learning’s effects on the NTK.
5.1. Overview of NTK theory
- The kernel norm links target projections onto NTK eigenfunctions to learning complexity, with larger-eigenvalue components associated with easier learning.The kernel representation is exact only for infinitely wide networks of a specific structure; for practical finite networks it is a rough approximation.Source: §5.1 · p. 6
- Signals concentrated in the span of NTK eigenfunctions with larger eigenvalues are easier to learn, motivating an eigenfunction-dictionary view of INR inductive bias.For finite networks, the NTK approximation is rough, but kernel norm remains a proxy for learnability.Source: §5.1 · p. 6, §5.2 · p. 6
5.2. NTK eigenfunctions as dictionary atoms
- The section uses the NTK eigenfunction dictionary to characterize how initialization choices, including SIREN’s ω0, affect inductive bias.This perspective also supports comparisons across networks and initializations.Source: §5.2 · p. 6
- More energy on larger-eigenvalue eigenfunctions corresponds to better test PSNR, while excessively high-frequency SIREN atoms span natural-image signals inefficiently.The energy profile shifts toward larger eigenvalues as ω0 increases, then recedes for ω0 > 40.Source: Figure 4 · p. 6, §5.2 · p. 7, Figure 5 · p. 7
5.3. Meta-learning as dictionary learning
- Meta-learned weights concentrate target-image energy on the largest-eigenvalue NTK eigenfunctions, and the authors report faster convergence and superior generalization.Their experiment pre-trains on 1,000 CelebA training images using MAML.Source: §5.3 · pp. 7–8
- The leading eigenfunctions of the meta-learned NTK look like faces, indicating that they correlate strongly with the target images.The learned initialization is a SIREN meta-learned on CelebA.Source: Figure 5 · p. 7, §5.3 · p. 8
- The authors report that meta-learning can reshape a larger NTK dictionary by combining tasks, increasing its capacity to encode a full metadistribution of signals.They propose dictionary learning as an abstraction for understanding this behavior and as an avenue for future research.Source: §5.3 · p. 8
6. Related work
Related work spans INR applications and architectural advances, while theoretical analyses have addressed narrower architectures or specific properties. This paper builds on NTK research to study finite networks with arbitrary weights and activations, including meta-learned SIRENs.
6. Related work
- Recent INR architecture work has focused on inference and training costs, aliasing, and generalization, across applications including rendering, reconstruction, synthesis, modeling, and compression.Source: §6 · p. 8
- Prior INR theory studied expressive power for a specific multiplicative filter network or compared activation functions’ rank and distance-preserving properties.The cited work also includes a static kernel account of random Fourier features with infinitely wide ReLU networks.Source: §6 · p. 8
- The paper extends NTK analysis to finite networks with arbitrary weights and activations, enabling investigation of dynamical effects such as pre-training.The authors contrast this with the static intuition from random Fourier features and infinitely wide ReLU networks.Source: §6 · p. 8
- Prior NTK visualizations studied training-time alignment with target signals, while this work uses a similar approach to report MAML’s dictionary-learning behavior.The authors describe their observation as the first report of this behavior.Source: §6 · p. 8
- Other related analyses connect harmonic expansions to deeper-layer loss-landscape roughness and to skip-connections avoiding shattered gradients.Source: §6 · p. 8
- The paper grounds its analogies in classical signal processing, signal representation, and dictionary design.Source: §6 · p. 8
7. Conclusion
The paper unifies the expressive-power and inductive-bias analysis of INRs: sinusoidal mappings yield structured harmonic dictionaries, while empirical NTKs characterize signal-encoding efficiency and meta-learning can modify that bias. The conclusion notes limits in current theory and experiments, and the preliminary lemmas establish how products and powers of sinusoids generate the frequency combinations used in the proof.
7. Conclusion
- INRs with sinusoidal mappings represent linear combinations of integer harmonics, enabling exponentially large frequency support but leaving imperfect recovery and aliasing as failure modes.The paper also characterizes inductive bias through empirical-NTK signal-encoding efficiency and reports that meta-learning can increase that efficiency.Source: §7 · p. 8
- The theorem could be extended beyond sinusoidal input mappings or to include normalization layers; alignment with the NTK after meta-training on out-of-distribution data remains a topic for study.Source: §7 · p. 8
- Most practical results use one- or two-dimensional signals, although the authors expect their insights to extend to higher dimensions; visualizing higher-dimensional NTK eigenfunctions could inform practitioners.Source: §7 · pp. 8–9
A.1.1 Preliminary lemmas
- Products of sinusoidal sums produce frequencies that are sums or differences of their input frequencies, and induction bounds the resulting integer-coefficient combinations by total absolute coefficient size.The preliminary lemmas apply this closure under products to powers of sinusoidal sums.Source: §A.1.1 · pp. 13–14
A.1.2 Main proof
The proof shows that an INR with a sinusoidal input mapping and polynomial activations represents a sum of sinusoids at bounded integer combinations of the mapping frequencies. It derives this layer by layer, illustrates the expansion for a two-layer SIREN, and specifies the FFN experiment setup.
A.1.2 Main proof
- For sinusoidal mappings and polynomial activations, the INR output is a sinusoid sum whose frequencies are integer combinations of mapping frequencies with total absolute coefficient at most K^(L-1).The theorem gives the representable function form as a linear combination of sinusoids at those frequencies.Source: §A.1.2 · p. 14
- Applying polynomial activations raises the harmonic order, while induction carries the bounded integer-combination frequency set through successive network layers.Source: §A.1.2 · p. 15
A.2. Two-layer SIREN example
- The two-layer SIREN expansion uses frequencies Σ s_t ω_t, with coefficients formed from first-kind Bessel factors and output-layer weights.Source: §A.2 · p. 17
B.1. Experimental details
- The Section 4.1 FFN uses three 256-wide hidden layers and a C=3 output layer, trained for 2,000 Adam iterations on all pixels of a 512 × 512 image.Source: §B.1 · p. 19
B.2. Additional experiments
These experiments show how SIREN reconstruction changes with its input mapping, including when the initial layer is trainable. The same experiments with FFNs show findings consistent with those for SIRENs.
B.2. Additional experiments
- With fixed initial-layer parameters, the tested SIREN mapping yields H(Ω) ⊆ {2πk | k ∈ Z}, producing imperfect, aliased reconstruction.The mapping uses frequencies f0 = 1 and f0 = 0.5 followed by a rich mapping; the initial-layer parameters are not updated during training.Source: §B.2 · p. 19
- The Figure 6 comparison presents SIREN image reconstructions alongside the magnitude of each reconstruction’s DFT and a 32 × 32 center crop.The accompanying experiment compares configurations while keeping the initial-layer parameters fixed for a fair comparison with FFNs.Source: Figure 6 · p. 19, §B.2 · p. 19
- Trainable initial-layer parameters mostly eliminate SIREN aliasing by making the input mapping dynamic, but low ω0 still produces blurry reconstructions.The dynamic mapping spreads energy across spectral coefficients rather than concentrating it only on even ones.Source: §B.2 · p. 19
- FFN experiments with learnable input mappings show findings identical to those for the SIREN experiments.The tested FFN variants are SFM*, with trainable frequency f initialized at f0, and FFN* with trainable Ωi,j.Source: §B.2 · p. 19
C.1. Experimental details
- The Section 4.2 SIREN experiment trains for 2000 iterations on 128 samples of a single-frequency signal, then tests at fs = 256 Hz.The three-layer network has two 128-dimensional hidden layers, uses Adam at 1 × 10^-4, and trains on samples at fs = 128 Hz.Source: §C.1 · p. 21
C.2. Additional experiments
FFNs show the same aliasing phenomenon observed with SIRENs. The section also describes how the NTK eigenfunctions and their overlaps with target images are estimated.
C.2. Additional experiments
- FFNs initialized with different σ exhibit the same aliasing phenomenon observed for SIRENs.Figure 9 compares spectra for σ = 100 and σ = 3, with signals sampled at fs = 128 Hz and fs = 256 Hz.Source: §C.2 · p. 21, Figure 9 · p. 21
- Figure 9 shows the magnitude of the spectrum of g(r) = sin(2π · 23r) and its FFN [56] reconstruction, trained at fs = 128 Hz.The top and bottom rows use σ = 100 and σ = 3, respectively. The left and right columns show signals sampled at fs = 128 Hz and fs = 256 Hz.Source: Figure 9 · p. 21
D.1. Estimation of eigenfunctions of the NTK
- NTK eigenvectors from the pixel-coordinate Gram matrix approximate eigenfunction values, and their inner products with target images are approximated from pixel samples.The Gram matrix uses all pixels on a 64 × 64 grid; this computation scales quadratically with sample and output counts, so experiments use grayscale images with a single output.Source: §D.1 · p. 22
D.2. Training details
Across SIREN and FFN experiments, stronger concentration of image energy on leading NTK eigenspaces accompanies faster training and better reconstruction performance. The appendix tests this relationship under additional networks and meta-learning settings.
D.2. Training details
- Networks with energy more concentrated on the largest NTK eigenvalues train faster and reach the best test performance in the SIREN experiments.Figure 10 tracks reconstruction performance for 100 CelebA validation images during the first 10 training iterations; the SIREN instances vary in ω0, and Meta marks the learned initialization.Source: §D.2 · p. 22, Figure 10 · p. 22
D.3. Experiments on additional networks
- FFNs with energy profiles more concentrated on the largest NTK eigenvalues perform much better, matching the SIREN pattern.Figure 11 reports average energy concentration for 100 CelebA validation images and average test PSNR after reconstructing from 50% randomly selected pixels for 3 iterations.Source: §D.3 · p. 22, Figure 11 · p. 23
E.1. Experimental details
- The MAML experiment meta-trains a SIREN initialization on 5,000 CelebA images for 5,000 meta-iterations before fine-tuning.The SIREN has four 256-dimensional hidden layers and ω0 = 30; NTK eigenfunctions at the meta-learned weights are estimated using the same setting.Source: §E.1 · pp. 22–23
E.2. Experiments with an additional meta-learning algorithm
- Replacing MAML with Reptile yields results that agree with MAML, suggesting the NTK reshaping effect may arise with multiple meta-learning algorithms.The comparison includes energy concentration plots and eigenfunctions at Reptile meta-learned weights.Source: §E.2 · p. 23, Figure 12 · p. 23
E.3. Experiments on additional networks
- For FFNs, meta-learning improves validation-image energy concentration on the NTK principal eigenspace and significantly improves performance.The tested FFN uses a sinusoidal mapping of size 256 and three 256-dimensional hidden layers; its mapping is initialized with σ = 1 before meta-learning.Source: §E.3 · p. 23, Figure 11 · p. 23
E.4. Comparison with standard training
Single-image pretraining can worsen fine-tuning on other images, while meta-learning builds a dictionary from broader training data. Separately, larger-eigenvalue NTK eigenfunctions are easier to learn.
E.4. Comparison with standard training
- Fine-tuning from single-image-pretrained SIRENs performs worse than starting from scratch, indicating negative transfer across tasks.More pretraining shifts validation-image energy toward smaller eigenvalues, and the more-pretrained networks take longer to converge and generalize worse.Source: §E.4 · p. 24, Figure 14 · p. 24, Figure 13 · p. 24
- Standardly trained NTK eigenfunctions resemble their specific pretraining targets, whereas meta-learning uses more data to build atoms composable into face images.The authors describe the standard-training dictionaries as overfitted to their particular pretraining tasks.Source: §E.4 · p. 24, Figure 16 · p. 26
F. Performance on NTK eigenfunctions
- NTK eigenfunctions associated with larger eigenvalues are easier for a network to learn, a pattern the authors say also applies beyond meta-learned weights.Figure 15 reports PSNR after 2000 training iterations on NTK eigenfunctions at initialization.Source: F. Performance on NTK eigenfunctions · p. 25, Figure 15 · p. 25