PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fields

Amir Mallak, Alaa Maalouf, Lior Wolf, Daniela Rus, Dan Rosenbaum

arXiv:2609.03117v1cs.LGcs.CV

TL;DR

Fast, high-quality neural-field reconstruction from sparse observations is difficult because classical NTK regression is linear and lacks reusable task priors. The paper develops NTK-KIP, MetaQuill, and their fusion to add distilled nonlinear supports and meta-learned adaptation. The resulting methods support high-PSNR reconstruction and semantically plausible sparse inpainting with lightweight per-instance adaptation.

  • Problem

    Classical NTK neural-field solvers are fast and analytic but remain fixed-kernel, coefficient-space methods that struggle with semantic structure, long-range context, and reusable task-adaptive features.

    Source: §I · p. 2, A. Exploring NTK-based NFs in… · p. 5

  • Method

    NTK-KIP distills task-specific supports, MetaQuill learns a shared initialization with small residual updates, and MetaQuill-KIP combines both mechanisms.

    Source: §I · p. 2, C. MetaQuill-KIP Hybrid Adaptation · p. 9, §VI · p. 17

  • Results

    MetaQuill-KIP produces high-PSNR reconstructions and coherent sparse-mask inpainting with adaptation taking only a few seconds for new tasks.

    Source: 1) NTK-KIP · p. 3

  • Takeaways & Limitations

    NTK-driven neural fields can combine analytic, fast adaptation with nonlinear representation and meta-learned features for single-image reconstruction and inpainting.

    Source: §I · p. 2, §VI · p. 17

Abstract

from arXiv · show

Neural fields (NFs) map continuous coordinates to signals such as color or density, but fast high-quality reconstruction from sparse observations remains difficult. Classical Neural Tangent Kernel (NTK) regression gives closed-form fits, yet it is fundamentally linear and cannot accumulate reusable task priors. We develop three algorithms that address these gaps. NTK-KIP learns a distilled support set of coordinates (and optional labels) so that a finite NTK can inpaint large missing regions from little observed data, yielding a compact non-linear representation instead of a raw kernel solve. MetaQuill meta-learns a shared initialization for an INR so that new scenes can be adapted by updating only a small task-specific weight offset, which provides true feature learning and a reusable prior. Finally, MetaQuill-KIP fuses both ideas: it seeds the task with a KIP-style non-linear warm start, then refines only that small offset around the meta-learned initialization. MetaQuill-KIP achieves high-PSNR reconstructions and semantically plausible inpainting under very sparse observations, while requiring only lightweight per-instance adaptation, whereas diffusion-style baselines typically depend on large pretrained generative priors and costly per-image tuning. This shows that NTK-driven neural fields can be made both non-linear and meta-learnable, narrowing the gap between analytic kernels and practical few-shot reconstruction.

I. INTRODUCTION

Neural fields represent continuous signals but require repeated nonlinear training, motivating NTK-based approaches that offer fast closed-form fitting. The paper identifies NTK's linearity, sensitivity to sparse or masked data, and computational costs, then introduces methods to add adaptive representations and meta-learning.

  • Neural fields map positional coordinates to continuous outputs such as color, density, or geometry for image, 3D, and audio representation.Source: §I · p. 1
  • Representation learning with neural fields is nonlinear, while encoding samples as network weights restricts network size and requires training a separate field for each sample.Source: §I · p. 1
  • Infinite-width NTKs can remain constant during training, reducing neural-network fitting under MSE to closed-form kernel-ridge regression.Source: §I · p. 1
  • NTK regression can bypass model training and accelerate fitting, but it incurs the cost of inverting the kernel matrix.Source: §I · p. 2
  • Classical NTK pipelines are limited by linear function-space predictions, sensitivity to noise and masking, and high memory or solve costs for dense finite-width neural fields.Source: §I · p. 2, A. Our Contributions · p. 2
  • The paper develops NTK-KIP, MetaQuill, and MetaQuill-KIP to add distilled nonlinear supports, reusable meta-learned initialization, and lightweight task-specific adaptation.Source: §I · p. 2, A. Our Contributions · p. 2

1) NTK-KIP:

NTK-KIP distills NTK fitting into a learned support set, adding task-adaptive nonlinearity while reducing the solve to a small subset. The section motivates this design through NTK’s strong fully observed reconstruction but weaker semantic and inpainting behavior.

NTK-KIP

  • NTK-KIP learns a compact set of spatial support coordinates and optional target values through the NTK.The method replaces solving on every pixel with solving on learned inducing points.Source: 1) NTK-KIP · p. 2
  • The learned support set makes the NTK representation nonlinear and task-adaptive rather than a strictly linear regressor over the raw grid.Source: 1) NTK-KIP · pp. 2–3

Results

  • ≈20% observed pixels reconstruct RGB Flowers at PSNR ≈31 dB with Fourier positional encodings, while ≈5% observed reaches PSNR ≈20 dB.Source: 1) NTK-KIP · p. 3
  • With raw positional encodings, NTK-KIP achieves ≈80% compression at PSNR > 36 dB and ≈50% compression at PSNR > 23 dB.Source: 1) NTK-KIP · p. 3

Inpainting

Representation Rigidity

Limitations

  • Fixed kernel correlations transfer poorly from fully observed reconstruction to partial observation, where semantic structure, edge continuity, texture, and long-range context matter.Source: A. Exploring NTK-based NFs in… · p. 5

B. Exploring NTK’s Robustness to Noise

The paper finds that NTK representations are highly sensitive to noise and inefficient for practical representation tasks. Noise propagates through kernel-based reconstructions, while NTK representations retain limited structure and incur substantial computational cost.

III. METHOD

The method uses NTK regression to learn inducing points for neural-field images rather than directly mapping complete pixel data to a coefficient vector. Optimizing coordinates and positional features yields a distilled, potentially nonlinear representation.

B. MetaQuill Algorithm

MetaQuill combines a shared meta-learned neural-field initialization with lightweight task-specific residual adaptation, while MetaQuill-KIP adds a distilled NTK warm start and nonlinear refinement. The hybrid targets both missing cross-task transfer and limited inpainting capacity.

MetaQuill-KIP Hybrid Adaptation

  • The hybrid addresses complementary weaknesses: NTK-KIP lacks shared task structure, whereas MetaQuill remains locally linear and may struggle with large missing regions.Source: C. MetaQuill-KIP Hybrid Adaptation · p. 9

MetaQuill Algorithm

MetaQuill-KIP Hybrid Adaptation

IV. RESULTS

The results evaluate NTK-KIP across positional encodings, Fourier bandwidths, and distillation levels. Performance improves with richer Fourier features before saturating, while the method achieves strong compression but retains NTK’s lack of feature learning.

IV. RESULTS

  • PSNR increases with Fourier Kbands and then exhibits diminishing returns once the positional features are sufficiently expressive.Source: A. NTK-KIP Results · pp. 10–11
  • The main experiments use Kbands = 20 as a practical operating point beyond the steep gain region while avoiding higher computational cost.Source: A. NTK-KIP Results · p. 11, Figure 12 · p. 11
  • At Kbands = 30, the best reported PSNR is approximately 48.40 at 50% distillation and approximately 54.24 at 20% distillation.Source: Figure 13 · p. 11
  • Despite its distilled nonlinear representation, NTK-KIP remains limited by NTK’s inherent lack of learning and feature extraction.Source: A. NTK-KIP Results · p. 11

B. MetaQuill Algorithm Results

MetaQuill-KIP combines a shared MetaQuill initialization with KIP-based nonlinear refinement, achieving strong reconstruction quality from lightweight per-task updates. Across MNIST and Flowers, it improves over kernel-only and tangent-only baselines, while sparse-mask results show semantic structure can emerge even when global PSNR is not maximal.

MetaQuill setup

MNIST reconstruction

Flowers reconstruction

Long-horizon refinement

  • Nonlinear ∆θ refinement improves Flowers reconstruction from 9.37 dB at KIP initialization to 26.29 dB after 500 steps.Intermediate checkpoints reach 16.73, 21.77, 24.31, and 25.39 dB at 100, 200, 300, and 400 steps, respectively.Source: 2) Flowers: inpainting, hole filling,… · p. 13

Sparse inpainting

  • At observed fractions ϕ≤0.20, MetaQuill-KIP may have lower global PSNR than Single-INR while producing more coherent petal, stem, and leaf structure.At ϕ=0.10, the scores are 11.97 versus 14.74 dB; at ϕ=0.20, they are 13.59 versus 14.70 dB.Source: 2) Flowers: inpainting, hole filling,… · pp. 13–14

Hole filling

3) Comparison to diffusion-based inpainting baselines:

MetaQuill-KIP is compared with diffusion-based inpainting systems under different observation levels and adaptation budgets. The paper emphasizes a self-contained, local alternative rather than a head-to-head replacement for large pretrained generative priors.

Experimental regime

  • Diffusion baselines use large pretrained generative priors, whereas MetaQuill-KIP adapts only a small ∆θ around task-agnostic θS on the test image.MetaQuill-KIP uses no external dataset or text guidance, while SD-IA uses about 150 steps and 30–35 seconds of per-image finetuning.Source: 3) Comparison to diffusion-based… · pp. 14–15, Table V · p. 15

Extreme sparsity

  • At ϕ=0.10, MetaQuill-KIP reports 11.97 dB versus 14.74 dB for Single-INR(masked), despite producing more semantically coherent missing-region structure.The comparison highlights a divergence between global PSNR and qualitative plausibility under extreme sparsity.Source: Figure 20 · p. 14

Interpretation

  • The authors frame the experiments as evidence that NTK-based neural fields can become nonlinear, rapidly adaptable, and self-contained, not as a universal diffusion replacement.The claimed scope is fast reconstruction with learned structure and local adaptation.Source: 3) Comparison to diffusion-based… · pp. 15–16

MNIST comparison

Flowers comparison

  • MetaQuill-KIP reaches competitive PSNR in moderate- and high-observation Flowers masks while adapting locally in a few seconds.The paper cites observed fractions ϕ∈{0.5,0.8} as representative settings.Source: 3) Comparison to diffusion-based… · p. 15

APPENDIX A LIST OF ACRONYMS AND ABBREVIATIONS

The appendix defines implementation terms and examines NTK-based representations through Fourier encoding, kernel-derived W vectors, and Functa comparisons. It reports representational and computational limitations alongside selected frequency-band results.

Fourier positional encoding

  • FPE transforms each d-dimensional positional encoding into d + 2·d·k Fourier features, so increasing k expands the input feature space.With d=2 and k=20, the encoding uses the selected Fourier-band configuration for experiments.Source: Appendix B · p. 19

Frequency-band selection

  • The experiments choose kband=20, although the reported elbow value is 15 and the maximum PSNR occurs at kband=50.The chosen value yields slightly higher PSNR than the elbow and somewhat more feature representation.Source: Appendix B · p. 19

Functa comparison

  • The Functa comparison treats each data sample as a neural function and reports distillation to approximately 0.5% of the original dataset size while preserving reconstruction features.The experiment tests whether NTK representations support comparable function-level compression and downstream use.Source: Appendix D · p. 21

Functa via NTK

  • NTK-derived W vectors replace learned neural fields as dataset representations for downstream modeling in the Functa comparison.The construction uses kernel matrices and labels, with W = Kt(x,x′;θ)^−1Y described for the representation.Source: Appendix D · p. 21

NTK limitations

  • The appendix identifies NTK weaknesses in representational structure, representational effectiveness, and runtime complexity.The NTK W representation matches ground-truth image size but lacks efficient structure and effective performance, with runtime at least an order of magnitude less efficient.Source: Appendix D · p. 23

A. Representation Learning via NTK Novel Algorithms

The paper proposes NTK-KIP and related finite-width, feature-learning approaches to address NTK’s representational linearity, inefficiency, and weak feature extraction. NTK-KIP distills coordinates and labels into compact inducing supports, producing nonlinear representations while retaining reconstruction capability.

NTK-KIP limitations

  • Fourier positional encodings expand each 2D coordinate to 82 features, so a 5% distilled dataset still produces 410% of the original dataset’s feature count.This feature expansion remains a limitation of the Fourier-based algorithm.Source: A. Representation Learning via NTK… · p. 24

NTK-KIP

  • NTK-KIP addresses NTK’s representational linearity by updating positional coordinates through an NTK-derived reconstruction objective.The learned coordinates make the kernel and resulting representation dependent on target labels.Source: A. Representation Learning via NTK… · pp. 23–24
  • NTK-KIP distills positional coordinates and optionally labels into a compact support set, replacing pixel-wise kernel solving with a smaller inducing-point solve.The method optimizes the support representation through the NTK to improve feature representation and data distillation.Source: A. Representation Learning via NTK… · pp. 23–24

NTK-KIP results

  • Raw positional encodings achieve 20% distillation above 36 PSNR and 50% distillation above 23 PSNR, while Fourier encodings achieve 80% distillation above 31 PSNR and 95% above 20 PSNR.The Fourier experiments also report improved feature learning and distilled inpainting reconstruction relative to raw positional encodings.Source: A. Representation Learning via NTK… · p. 24

Unified direction

  • The paper positions the combined approach as integrating finite-width NTK, model feature learning, and MAML-like meta-learning without an inner loop.The claimed goal is to address linearity, efficiency, and kernel representation weaknesses together.Source: A. Representation Learning via NTK… · pp. 23, 25

Finite-width NTK

  • Finite-width NTK is used because finite networks can exhibit feature learning, whereas infinite-width NTK models have fixed underlying feature maps.The finite-width formulation approximates network training locally around a working point using a first-order Taylor expansion.Source: A. Representation Learning via NTK… · p. 25

SAMPLE A NEW NF

This section develops an efficient meta-learning approach for adapting neural fields through shared parameters and task-specific offsets. Experiments report efficient feature learning and adaptation from a shared model, while finite-NTK conditioning remains a reconstruction challenge.

Shared NF training

  • A shared neural field learns common parameters θS together with task-specific offsets ∆θi across a subset of images.The example trains on 100 images, converting the image dataset into multiple neural-field tasks.Source: SAMPLE A NEW NF · p. 26

Task adaptation

  • New neural fields are initialized from the shared model and adapted by obtaining only the corresponding task offset ∆θi.The procedure starts from the shared NF base parameters rather than training each new field independently.Source: SAMPLE A NEW NF · p. 26

Results

  • Using only 100 images, the algorithm achieves high accuracy with efficient runtime and resources and behaves similarly to MAML without its inner loop.The reported procedure makes the MAML inner-loop process redundant in the authors’ experiment.Source: SAMPLE A NEW NF · pp. 26–27

Limitation

  • Finite-NTK feature learning can lose representation power because the shared-model kernel matrix is ill-conditioned.The high condition number makes the matrix nearly singular and contributes to blurry reconstructions.Source: SAMPLE A NEW NF · pp. 26–27

Combined direction

  • The authors propose combining the meta-learning approach with NF KIP to jointly address feature learning, nonlinear representation, and efficiency.The unified method is presented as a response to both representational linearity and ill-conditioning-related reconstruction problems.Source: SAMPLE A NEW NF · p. 27

3) MetaQuill-KIP Algorithm:

MetaQuill-KIP combines a KIP-based warm start with nonlinear offset refinement around a shared neural-field initialization. The robustness protocol corrupts only observed pixels and compares reconstruction methods under Gaussian and impulse noise.

MetaQuill-KIP

  • MetaQuill-KIP represents each task as a shared initialization θS plus a compact offset ∆θi, updating only ∆θi at test time.The shared parameters remain fixed during per-instance adaptation.Source: 3) MetaQuill-KIP Algorithm · p. 27

Tangent prediction

  • The tangent prediction uses the INR Jacobian evaluated at θS, providing the linearized output around the shared initialization.The input is the coordinate representation used by the INR, including Fourier positional encodings in the experiments.Source: 3) MetaQuill-KIP Algorithm · p. 27

KIP warm start

  • KIP warm-starts ∆θi by optimizing compact support coordinates and optional pseudo-labels so kernel ridge regression predicts the target from observed pixels.For inpainting, the KIP objective is restricted to observed pixels, producing a structure-aligned but still linearized initialization.Source: 3) MetaQuill-KIP Algorithm · p. 27

Nonlinear refinement

  • Nonlinear refinement updates only ∆θi using the full model output and a masked reconstruction loss on observed pixels, with mild offset regularization.This stage injects nonlinearity while preserving lightweight adaptation.Source: 3) MetaQuill-KIP Algorithm · p. 27

Robustness protocol

  • The robustness evaluation uses Flowers random masks with observed fractions 0.10, 0.20, and 0.40, corrupting only observed pixels with Gaussian or salt-and-pepper noise.Evaluation compares against clean ground truth, and qualitative panels include KRR, KIP, and combined KIP-to-nonlinear methods.Source: 3) MetaQuill-KIP Algorithm · p. 27, Table VIII · p. 28, Figure 26 · p. 29

Evaluation caveat

  • Hole PSNR is treated as a complementary diagnostic because PSNR can be misleading under extreme sparsity, so qualitative panels assess structural plausibility.The reported PSNR convention normalizes each compared region by its own intensity range.Source: 3) MetaQuill-KIP Algorithm · pp. 27–28

Robustness results

  • Across most corruption settings, MetaQuill-KIP degrades gracefully with increasing noise and benefits from a higher observed fraction.Qualitative results indicate more globally coherent structure in several low-coverage cases despite small quantitative differences.Source: 3) MetaQuill-KIP Algorithm · p. 28

Related papers

Loading 2609.03117v1…