Learnable Spectral Activations

1 Introduction

• The structured-dictionary view treats an INR as a weighted combination of harmonic atoms induced by its architecture.Input encoding determines base frequencies, while nonlinear activations generate harmonics.Source: §1 · p. 1
• Existing INR methods mainly change which frequencies are available, leaving each neuron's spectral profile tied to the weights that also define its input feature.The introduction uses reconstructing an image edge as an example requiring coordinated energy across many harmonics with specific relative amplitudes.Source: §1 · p. 2
• LSA replaces fixed activations with residual truncated Fourier series whose learned harmonic amplitudes give hidden units multi-harmonic responses over the same pre-activation.The residual form starts near the identity and adapts a layer-specific harmonic profile during training.Source: §1 · p. 2
• LSA changes representation factorization rather than asymptotic function class: linear weights select features while activation coefficients shape their spectra through separate gradients.Given fixed pre-activations, the activation output is affine in its coefficients, making spectral tuning a more direct optimization subproblem.Source: §1 · p. 2
• LSA improves reconstruction quality across audio, image, neural radiance field, and neural acoustic field tasks, while reducing structured spatial and spectral residuals.Audio experiments also report more stable off-grid behavior.Source: §1 · p. 2

2 Related Work

• The structured-dictionary perspective represents signals as weighted combinations of architecture-induced harmonic atoms, with input encodings, activations, and initialization determining the dictionary.For base frequencies Ω, nonlinear activations generate harmonics.Source: §2 · pp. 2–3
• Fourier features, SIREN, FINER, and meta-learning approaches control the induced dictionary chiefly by changing available or emphasized frequencies, while leaving neuron-level nonlinearities prescribed.Other approaches address localized bases, initialization, input-frequency schedules, conditioning, or network composition.Source: §2 · p. 3
• LSA fixes an integer-harmonic basis for each pre-activation and learns relative amplitudes, unlike STAF, which learns layer-shared sinusoidal mixtures with trainable amplitudes, frequencies, and phases.LSA focuses on composing coordinated harmonics rather than learning the sinusoidal basis itself.Source: §2 · p. 3
• Leading empirical NTK eigenfunctions act as high-eigenvalue atoms learned more rapidly by gradient descent, motivating alignment of the induced basis with the target rather than only expanded frequency support.LSA adapts neuron-level spectral shaping while retaining a compact layer-wise parameterization.Source: §2 · p. 3

3 Method

• The activation is a residual truncated Fourier series applied elementwise to layer pre-activations, with harmonic coefficients shared across neurons within each layer.The coefficients are learned jointly with the network weights and initialized with small variance and zero mean so the model starts close to identity.Source: §3 · pp. 3–4
• Input encodings set input spectral support, while linear weights form hidden features and LSA coefficients shape those features' spectra without changing their projections.LSA's pointwise learned operator adapts hidden units' harmonic responses over the same pre-activations.Source: §3 · p. 4
• Unlike prescribed activations, LSA applies multiple harmonic transformations to one pre-activation and learns their relative amplitudes, changing spectral factorization rather than asymptotic expressivity.A fixed-basis expansion remains affine in its coefficients.Source: §3 · p. 4
• At identity initialization, LSA retains harmonic tangent directions, while its local optimization benefit depends on residual alignment.The activation equals u at a = 0 and has harmonic coefficient derivatives; the rate-weighted tangent-kernel component contributes to initial loss decay as described in the passage.Source: §3 · p. 4
• The NTK analysis measures zero-mean target energy in the span of leading eigenfunctions on fixed sampled coordinates, whose high-eigenvalue directions gradient descent learns most efficiently.LSA captures substantially more target energy in the same number of leading modes than SIREN or FINER.Source: §3 · pp. 4–5
• LSA captures more target energy in the same number of leading empirical NTK modes than SIREN or FINER, and its harmonic factorization reaches the target structure with fewer atoms in the controlled 1D phantom.Figure 1a contrasts prescribed single-unit responses with LSA's multi-harmonic waveforms; Figure 1b compares target-energy concentration in leading modes.Source: §3 · pp. 4–5, Figure 1 · p. 5

4.1.1 Audio Signal Fitting

• At moderate and high Fourier-feature bandwidths, LSA achieves the strongest on-grid audio performance on both speech and musical-note signals.With raw coordinates (σ = 0), FINER is slightly higher on both datasets; LSA overtakes it once Fourier-feature support is available. Under the tested 5,000-step budget, LSA reaches 86.72/90.78 dB on NSynth/LibriSpeech versus 68.79/68.40 dB for the strongest tested STAF configuration.Source: Figure 2 · p. 6, §4.1.1 · p. 6

4.1.2 Continuous Approximation

• As Fourier-feature bandwidth rises, SIREN's on-grid PSNR improves but its off-grid reconstruction degrades, while LSA remains consistent across seen and unseen grids.The comparison evaluates a fixed grid of 16k samples and an unseen midpoint grid.Source: §4.1.2 · pp. 6–7, Figure 3 · p. 7

4.2 Image Data

• 38.110 dB mean PSNR for LSA versus 37.317 dB for FINER, 35.369 dB for matched-budget global STAF, and 34.990 dB for SIREN across 24 Kodak images.LSA wins on all 24 images with per-image tuning, with a paired mean gain of 2.883 dB over STAF; the 95% bootstrap interval is [2.439, 3.295] dB.Source: Figure 4 · p. 7, §4.2 · p. 8
• With gradient-only supervision, LSA yields lower gradient-field residuals and a more coherent image reconstruction than FINER or SIREN.FINER has structured gradient residuals and large low-frequency distortions; SIREN exhibits global drift and inconsistent reconstruction.Source: Figure 5 · p. 8, §4.2 · p. 8
• On the 18-image Kodak holdout, LSA reaches 38.045 dB with 199,043 parameters, versus 37.857 dB for full SL2A with 330,243 parameters.At nearly matched parameter counts, LSA's mean advantage is 2.135 dB, outperforming SL2A on 18/18 images.Source: §4.2 · p. 8

4.3 Neural Fields

• Across eight Blender scenes, LSA achieves the best PSNR on five scenes and best SSIM on seven, with moderate gains across complementary metrics.The controlled comparison uses an InstantNGP-style neural-field backbone for novel-view synthesis.Source: Table 1 · p. 9, §4.3 · p. 9

4.2 Image Data

• On a six-image Kodak ablation, PSNR rises from 30.45 dB at K = 4 to 39.50 dB at K = 32, with no further improvement at K = 64.Freezing the near-identity coefficients reduces held-out Kodak PSNR from 38.05 to 19.91 dB.Source: §4.2 · p. 9

4.3 Neural Fields

• Replacing the NAF baseline activation with LSA yields consistent acoustic-field gains, especially in log-spectral and multi-resolution STFT errors across spatial queries.The evaluation keeps the NAF architecture and training protocol fixed and uses K = 8 harmonics.Source: Figure 7 · p. 10, §4.3 · p. 10

5 Conclusion

• Across audio, images, and neural fields, LSA improves reconstruction quality and reduces structured spatial and spectral residuals; audio experiments also show more stable off-grid behavior.The authors frame learned harmonic composition as a structured inductive bias for INR fitting and spatial-query prediction.Source: §5 · p. 10
• LSA adds 64 parameters but incurs roughly 7× training-time and 9× peak-memory overhead relative to FINER in a representative full-grid audio setting.Inference overheads are similar; more efficient fused or checkpointed implementations may reduce the cost.Source: §5 · p. 10

A Implementation Details

• Unless otherwise stated, experiments use a single NVIDIA RTX A6000 GPU.Source: A Implementation Details · p. 13

A.1 Audio Data

• Audio fitting uses fixed LibriSpeech and NSynth clip manifests, shared Fourier-feature samples, and method-specific selected σ settings.Signals are resampled to 48 kHz; main runs use full-grid training for 5,000 steps, while continuous approximation tests seen-grid and midpoint queries on a 16,000-sample grid.Source: §A.1 · pp. 13–15

A.2 Images and Derivatives

• The image experiments compare coordinate-to-RGB fitting without input lifting and separately train on RGB spatial derivatives rather than direct color values.Derivative targets use finite differences; because derivative supervision leaves an additive constant ambiguity, predictions are mean-aligned channel-wise before PSNR evaluation.Source: §A.2 · pp. 15–16

A.3 Novel View Synthesis

• Novel-view synthesis uses eight Blender scenes with every fourth training view retained, held-out test views, and a shared NeRF backbone whose activation family varies.Models are trained from scratch for 150,000 steps, and reported values use the final checkpoint.Source: §A.3 · p. 17

A.4 Neural acoustic fields

• The neural acoustic field experiment replaces only the hidden activation in the NAF decoder, retaining its spatial feature grid, conditioning, data protocol, and other modules.NAF and LSA variants use the same train/test splits and optimization budget, with final checkpoints reported at epoch 50.Source: §A.4 · pp. 17–18

A.5 NTK Energy

• The NTK analysis projects a zero-mean target onto empirical-kernel eigenvectors and measures normalized cumulative target energy in the top modes.The same coordinates and target preprocessing are used across methods, and the Jacobian includes all trainable parameters, including LSA activation coefficients.Source: §A.5 · p. 19

A.6 Local Optimization

• The local-optimization analysis characterizes harmonic-coefficient contributions to squared-loss descent at a fixed network state and examines amplitude parameterization near identity initialization.Its conclusions concern the residual on the coordinates used to form the Jacobian.Source: §A.6 · p. 19

A.6.1 Tangent Features and Descent

• For fixed inputs, the tangent-feature analysis defines coefficient and ordinary-weight Jacobians, with shared coefficients producing output tangent features across channels.The stacked coefficient tangent features form Ψ = ∂f(X)/∂a.Source: §A.6.1 · pp. 19–20
• Coefficient updates follow residual correlations with propagated harmonic features, adding instantaneous descent η_a∥Ψ^T r∥_2^2 when the residual projects nontrivially onto those features.The finite-width identity holds even as tangent features change during training.Source: §A.6.1 · p. 20
• At initialization, coefficient learning adds a positive-semidefinite kernel update of rank at most (L − 1)K, and the loss-decay-rate difference is η_a∥Ψ_0^T r_0∥_2^2.The fixed-kernel comparison evaluates both Jacobians at shared initialization.Source: §A.6.1 · p. 20
• The guarantee concerns instantaneous descent and its fixed-kernel local approximation; subsequent nonlinear trajectories develop their own Jacobians and residuals.Generalization to unobserved coordinates is evaluated empirically in the audio and neural-field experiments.Source: §A.6.1 · p. 20
• The resulting inductive bias is target-dependent because the exact propagated-feature quantity reflects feature scale and correlation.The harmonic choice determines the alignment of the added directions.Source: §A.6.1 · p. 20

A.6.2 Amplitude Conditioning

• As amplitude A approaches zero, frequency and phase output-Jacobian blocks are O(A) with Gram matrices O(A^2), while the amplitude block is O(1) under the stated boundedness assumptions.The amplitude block has a nonzero limit when propagated sine features are nondegenerate; the result also holds for a finite bank with amplitudes A_k = εc_k and bounded c_k.Source: §A.6.2 · p. 20
• LSA’s identity residual parameterization preserves the activation and its input derivative while coefficient learning accesses harmonic output directions; STAF also learns amplitudes and is evaluated separately.For LSA, ω_k = 2πk and φ_k = 0.Source: §A.6.2 · p. 20

A.6.3 Controlled Validation

• The one-step validation compares a joint learned-and-weight update with an ordinary-weight-only update, and the normalized loss difference tends to 1 as the step size approaches zero.The smallest tested step has a positive difference in all 18 image-seed pairs; exact-zero initialization reaches mean full-image PSNR within 0.044 dB of small-random initialization.Source: §A.6.3 · pp. 20–21, Table 3 · p. 21
• For a zero-mean, unit-RMS edge target, the harmonic adapter attains D_a = 0.0611, versus 0.00020–0.00481 for random dictionaries and 0.0616 for the mean jittered result.The same ordering holds for the tested spectral-stack target; the comparison associates the gain with coordinated harmonic structure under a controlled kernel scale.Source: §A.6.3 · p. 21

B.1.1 Audio Signal Fitting

• Learned spectral activations closely match transient and harmonic waveform structure on LibriSpeech and NSynth, leaving smaller residuals than FINER and SIREN.The passage connects this behavior to learning the harmonic mixture of the hidden nonlinear response, rather than merely shifting oscillatory components.Source: §B.1.1 · pp. 21–22, Figure 8 · p. 22

B.1.2 Continuous Approximation

• With 16,000 training samples on a fixed grid, the continuous-approximation experiment evaluates predictions both on-grid and at unseen midpoints between adjacent samples.This experiment is separate from the main audio fitting experiments, which use one-second clips represented at 48 kHz.Source: §B.1.2 · p. 22

B.2 Image Fitting

• LSA leads FINER on 21 of 24 Kodak images at 5,000 updates and SIREN on all 24, with its advantage over FINER emerging progressively.LSA leads FINER on 5, 16, 21, 22, 21, and 21 images at steps 500 through 5,000; it leads SIREN on 23/24 at step 500 and all 24 at later listed steps.Source: §B.2 · p. 22

B.3 Neural Fields

• On Chair and Ficus, LSA reduces structured residuals around thin geometry, object boundaries, and textured regions while preserving comparable global appearance.The qualitative comparison contrasts LSA with SIREN and FINER.Source: §B.3 · p. 22

B.4 Neural-Acoustic Fields

• On Kodak images, LSA separates from SIREN early, while its advantage over FINER emerges progressively across a matched 5,000-update budget.Early-stage behavior varies across images.Source: Figure 10 · p. 24
• LSA improves every reported neural acoustic field metric under both aggregation schemes, with TDOA gains of 67.55% and 72.41%.Log spectral L1 improves by 44.90% and 50.84%, while MRSTFT log-L1 improves by 45.97% and 51.54%, under macro-scene and micro-position aggregation, respectively.Source: §B.4 · pp. 23–24
• Waveform-domain metrics also improve consistently, with MSE reduced by approximately 9–10% and correlation increased by roughly 40–43%.Metrics are reported with macro-scene and micro-position aggregation.Source: §B.4 · pp. 23–24
• Additional spatial improvement maps report results similar to Figure 7, indicating that the conclusions from Section 4.3 are stably reproduced across different scenes.Source: §B.4 · p. 24

B.5 Additional Use-cases

• Additional maps report spatial improvement over the NAF baseline for apartment, room, and office scenes.The captions identify apartment scenes 1, 2, and 4, a room scene, and an office scene.Source: Figure 12 · p. 25, Figure 13 · p. 26, Figure 14 · p. 26
• No positional encoding yields oversmoothed reconstructions, learned table embeddings perform poorly, and fixed sinusoidal encoding leaves structured artifacts near edges and fine details.The task is highly sensitive to the positional encoding choice.Source: §B.5 · p. 25
• The PE-only variant performs best, with sharper reconstructions and lower errors particularly in high-frequency regions.In this variant, spatial information is provided exclusively through the learned spectral encoding.Source: §B.5 · p. 25
• Learned spectral positional encoding adapts its basis to the task, whereas fixed positional encodings impose a predefined spectral basis.The comparison interprets the improvement as learning spatial relationships rather than merely adding features.Source: §B.5 · p. 25

C Additional Comparisons

• The broader comparisons include released implementations of STAF and SL2A, wavelet and multiplicative-filter comparisons, and a single-degree sequential fJNB construction.The passage distinguishes the fJNB construction from the Fourier KAN discussed elsewhere.Source: C Additional Comparisons · p. 28

C.1 Image Fitting

• The mean paired PSNR difference is 2.883 dB in LSA’s favor, with a 95% bootstrap interval of [2.439, 3.295] dB against STAF.LSA also has higher SSIM on every image; these results use the specified per-image search budgets.Source: §C.1 · p. 28, Table 5 · p. 28
• The matched-budget global STAF configuration has an all-24 mean of 35.369 dB, while the separate extended-budget result has an all-24 mean of 35.146 dB.Both full-set means include calibration images and are descriptive.Source: §C.1 · p. 28
• The primary SL2A comparison uses an 18-image holdout after calibration, while the 24-image means also include the six calibration images.Paired PSNR intervals resample image-level differences 20,000 times and report the 2.5th and 97.5th percentiles of bootstrap means.Source: §C.1 · p. 28, Table 6 · p. 29
• LSA’s mean difference against full SL2A is 0.188 dB, with ten wins and eight losses; full SL2A has a slightly lower descriptive mean.Full SL2A uses 65.9% more parameters, while LSA gains 2.135 dB and wins all 18 images in the nearly parameter-matched comparison.Source: §C.1 · pp. 28–29, Table 6 · p. 29
• On the holdout, LSA and fJNB obtain SSIM 0.9545 ± 0.0115 and 0.5449 ± 0.1274, respectively.The comparison uses a single-degree sequential fJNB construction.Source: §C.1 · p. 29

C.2 Audio Fitting

• Audio comparisons use fixed NSynth and LibriSpeech clip manifests, one-second mono signals resampled to 48 kHz, and PSNR selected from evaluated fitting-grid checkpoints.Coordinate MLP comparisons use width 256, four hidden layers, Adam, and 5,000 updates; settings are selected separately for each method and evaluated on the same clips.Source: §C.2 · p. 29
• 86.72 ± 21.18 dB on NSynth and 90.78 ± 18.57 dB on LibriSpeech, LSA at FF16, σ = 100, K = 16 wins 8/10 and 37/40 clips over the strongest STAF setting.The raw-coordinate STAF setting is strongest among the STAF configurations tested, reaching 68.79 dB on NSynth and 68.40 dB on LibriSpeech.Source: §C.2 · p. 31
• 68.03 dB on NSynth and 47.42 dB on LibriSpeech, SL2A’s degree-512, rank-128 setting is its strongest tested configuration, while LSA wins on 7/10 and 40/40 clips.The SL2A sweep tests degrees 256 and 512 and ranks 64 and 128 at learning rate 10−3.Source: §C.2 · p. 31
• The audio fJNB sweep includes multiple degrees, learning rates, Fourier scales, and raw-coordinate runs, with the strongest completed configurations reported.Table 12 includes both input regimes for fJNB and the corresponding LSA results.Source: §C.2 · p. 31, Table 12 · p. 31

C.3 Novel View Synthesis

• For novel view synthesis, fJNB and LSA use the same reduced-view Blender split, backbone, ray budget, sample count, optimizer family, and final-step evaluation rule.The section also reproduces WIRE PSNR from FINER [24], whose source protocol uses 25 training views; the WIRE values are not from the shared-backbone evaluation.Source: §C.3 · p. 31, Table 13 · p. 32

C.4 Ablations

• On six Kodak images, reconstruction quality increases through K = 32 and saturates at K = 64 under the tested protocol.The table reports standard deviations and medians across the six images.Source: §C.4 · p. 32, Table 14 · p. 32
• K controls both reconstruction quality and activation cost in the reported image and audio settings.The quality ablation and resource measurements concern their respective settings.Source: §C.4 · p. 32
• The ablation evaluates six Kodak images across harmonic counts, with image-level standard deviations and medians reported.Source: Table 14 · p. 32, §C.4 · p. 32

K Mean PSNR Std. PSNR Median PSNR Mean SSIM

• 18.13 dB on the holdout and 18.35 dB across all 24 images are the paired mean PSNR gains from learning coefficients rather than freezing them near identity.Both variants share the same initial state, data order, random seed, optimizer settings, and 5,000-step budget; the frozen coefficients remain bitwise unchanged.Source: K Mean PSNR Std. PSNR Median PSNR Mean… · p. 32, Table 15 · p. 32

D.1 Hidden-Representation Diagnostics

• LSA produces less purely periodic and more spatially localized hidden features in early layers than SIREN and FINER, whose later layers become more image-aligned.The radial spectra of selected high-variance channels are diagnostic rather than a complete spectrum of the full representation; they show similar behavior across LSA layers and stronger layer-wise changes for SIREN and FINER.Source: §D.1 · p. 33, Figure 18 · p. 33

D.2 Learned Activation Coefficient Spectra

• Learned coefficient spectra have layer-dependent structure and image-dependent variation: early layers selectively enhance or suppress harmonics, while later layers retain broader support.The coefficient plots show the learned operator in parameter space, complementing hidden-channel visualizations of its spatial manifestation.Source: §D.2 · pp. 33–34

D.3 Learned Activation Profiles

• Learned activations stay close to the residual identity mapping but add localized deviations and small oscillatory corrections, especially around frequently used pre-activation regions.Their profiles vary across fitted images and layers, showing targeted harmonic corrections rather than a fixed activation profile.Source: §D.3 · pp. 34–35

• For B coordinates, width W, Lact activation layers, and K harmonics, pointwise evaluation computes BWLactK harmonic responses, making intermediate tensors and autograd storage the practical cost drivers.Source: §D.4 · p. 35
• At depth 4 and K = 16, LSA adds only 64 parameters over FF16 baselines but requires additional runtime and peak memory to evaluate and store harmonic responses.The measured comparison uses matched FF16 inputs for full-grid 48k audio fitting, with training time including forward, backward, and optimizer update.Source: §D.4 · pp. 35–36, Table 16 · p. 36
• Runtime and memory both increase with K, making harmonic count a direct trade-off between spectral flexibility and computational cost.The scaling measurements use the batched PyTorch harmonic evaluation under the matched FF16 full-grid 48k setting.Source: §D.4 · p. 36, Table 17 · p. 37
• The batched implementation is faster but uses more memory; sequential accumulation reduces memory, especially at inference, but is slower.The accumulate implementation avoids materializing the full harmonic tensor.Source: §D.4 · p. 37, Table 18 · p. 37
• The current overhead is partly implementation-dependent, and the reported implementation is a straightforward reference rather than a fully optimized activation kernel.Fused kernels, checkpointed harmonic evaluation, or adaptive sparsification are proposed as ways to reduce cost without changing LSA’s mathematical form.Source: §D.4 · p. 37