PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fields

Amir Mallak, Alaa Maalouf, Lior Wolf, Daniela Rus, Dan Rosenbaum

arXiv:2609.03117v1cs.LGcs.CV

TL;DR

Fast, high-quality neural-field reconstruction from sparse observations is difficult because classical NTK regression is linear and lacks reusable task priors. The paper develops NTK-KIP, MetaQuill, and their fusion to add distilled nonlinear supports and meta-learned adaptation. The resulting methods support high-PSNR reconstruction and semantically plausible sparse inpainting with lightweight per-instance adaptation.

  • Problem

    Classical NTK neural-field solvers are fast and analytic but remain fixed-kernel, coefficient-space methods that struggle with semantic structure, long-range context, and reusable task-adaptive features.

  • Method

    NTK-KIP distills task-specific supports, MetaQuill learns a shared initialization with small residual updates, and MetaQuill-KIP combines both mechanisms.

  • Results

    MetaQuill-KIP produces high-PSNR reconstructions and coherent sparse-mask inpainting with adaptation taking only a few seconds for new tasks.

  • Takeaways & Limitations

    NTK-driven neural fields can combine analytic, fast adaptation with nonlinear representation and meta-learned features for single-image reconstruction and inpainting.

Abstract

from arXiv · show

Neural fields (NFs) map continuous coordinates to signals such as color or density, but fast high-quality reconstruction from sparse observations remains difficult. Classical Neural Tangent Kernel (NTK) regression gives closed-form fits, yet it is fundamentally linear and cannot accumulate reusable task priors. We develop three algorithms that address these gaps. NTK-KIP learns a distilled support set of coordinates (and optional labels) so that a finite NTK can inpaint large missing regions from little observed data, yielding a compact non-linear representation instead of a raw kernel solve. MetaQuill meta-learns a shared initialization for an INR so that new scenes can be adapted by updating only a small task-specific weight offset, which provides true feature learning and a reusable prior. Finally, MetaQuill-KIP fuses both ideas: it seeds the task with a KIP-style non-linear warm start, then refines only that small offset around the meta-learned initialization. MetaQuill-KIP achieves high-PSNR reconstructions and semantically plausible inpainting under very sparse observations, while requiring only lightweight per-instance adaptation, whereas diffusion-style baselines typically depend on large pretrained generative priors and costly per-image tuning. This shows that NTK-driven neural fields can be made both non-linear and meta-learnable, narrowing the gap between analytic kernels and practical few-shot reconstruction.

I. INTRODUCTION

Neural fields represent continuous signals but require repeated nonlinear training, motivating NTK-based approaches that offer fast closed-form fitting. The paper identifies NTK's linearity, sensitivity to sparse or masked data, and computational costs, then introduces methods to add adaptive representations and meta-learning.

  • Neural fields map positional coordinates to continuous outputs such as color, density, or geometry for image, 3D, and audio representation.
  • Representation learning with neural fields is nonlinear, while encoding samples as network weights restricts network size and requires training a separate field for each sample.
  • Infinite-width NTKs can remain constant during training, reducing neural-network fitting under MSE to closed-form kernel-ridge regression.
  • NTK regression can bypass model training and accelerate fitting, but it incurs the cost of inverting the kernel matrix.
  • Classical NTK pipelines are limited by linear function-space predictions, sensitivity to noise and masking, and high memory or solve costs for dense finite-width neural fields.
  • The paper develops NTK-KIP, MetaQuill, and MetaQuill-KIP to add distilled nonlinear supports, reusable meta-learned initialization, and lightweight task-specific adaptation.

1) NTK-KIP:

NTK-KIP distills NTK fitting into a learned support set, adding task-adaptive nonlinearity while reducing the solve to a small subset. The section motivates this design through NTK’s strong fully observed reconstruction but weaker semantic and inpainting behavior.

  • NTK-KIP: NTK-KIP learns a compact set of spatial support coordinates and optional target values through the NTK.The method replaces solving on every pixel with solving on learned inducing points.
  • NTK-KIP: The learned support set makes the NTK representation nonlinear and task-adaptive rather than a strictly linear regressor over the raw grid.
  • Results: ≈20% observed pixels reconstruct RGB Flowers at PSNR ≈31 dB with Fourier positional encodings, while ≈5% observed reaches PSNR ≈20 dB.
  • Results: With raw positional encodings, NTK-KIP achieves ≈80% compression at PSNR > 36 dB and ≈50% compression at PSNR > 23 dB.
  • Inpainting: In inpainting, NTK reconstruction reaches PSNR ≈20.21 dB and masked PSNR ≈12.07 dB, but lacks the semantic coherence and object-shape connections of NF predictions.
  • Representation Rigidity: With all labels known, infinite NTK reconstruction reaches PSNR 101.79 dB, compared with 32.12 for NF and 18.93 for finite NTK.
  • Limitations: Fixed kernel correlations transfer poorly from fully observed reconstruction to partial observation, where semantic structure, edge continuity, texture, and long-range context matter.

B. Exploring NTK’s Robustness to Noise

The paper finds that NTK representations are highly sensitive to noise and inefficient for practical representation tasks. Noise propagates through kernel-based reconstructions, while NTK representations retain limited structure and incur substantial computational cost.

  • A single noisy pixel in W affects multiple reconstructed rows and columns, while noise across W severely perturbs image structure.
  • NTK representations are extremely noise-sensitive, producing blurry reconstructions containing kernel-induced frequencies.
  • The experiments compare classic Functa with an NTK-based variant that replaces each neural field by its coefficient representation W.
  • NTK’s W representation matches the ground-truth image size but lacks efficient structure and effective task representation.
  • NTK representation and computation are less efficient than neural-field alternatives, with runtime at least one order of magnitude worse.

III. METHOD

The method uses NTK regression to learn inducing points for neural-field images rather than directly mapping complete pixel data to a coefficient vector. Optimizing coordinates and positional features yields a distilled, potentially nonlinear representation.

  • NTK-KIP learns a small distilled set of induced pixel coordinates that serves as a compressed training set for a neural field.
  • The method optimizes either pixel positions or Fourier positional features to minimize reconstruction error on the full image.
  • The shared neural-field pipeline can be trained from a distilled support set rather than all image pixels.
  • Unlike direct NTK mapping from image labels to W, inducing-point optimization produces distilled data and a representation that is no longer a simple linear signal mapping.

B. MetaQuill Algorithm

MetaQuill combines a shared meta-learned neural-field initialization with lightweight task-specific residual adaptation, while MetaQuill-KIP adds a distilled NTK warm start and nonlinear refinement. The hybrid targets both missing cross-task transfer and limited inpainting capacity.

  • MetaQuill-KIP Hybrid Adaptation: The hybrid addresses complementary weaknesses: NTK-KIP lacks shared task structure, whereas MetaQuill remains locally linear and may struggle with large missing regions.
  • MetaQuill Algorithm: MetaQuill learns a shared initialization θS across tasks and adapts each new image by optimizing only a small residual Δθ.
  • MetaQuill Algorithm: Finite-width NTK linearization accelerates inner-loop adaptation without retraining the full network, making θS a meta-learned feature extractor.
  • MetaQuill-KIP Hybrid Adaptation: NTK-KIP creates a compact support set whose finite-NTK solution supplies an implicit task-specific direction in parameter space.
  • MetaQuill-KIP Hybrid Adaptation: MetaQuill-KIP initializes the residual with the KIP-derived update instead of zero, then refines only Δθ while freezing θS.
  • MetaQuill-KIP Hybrid Adaptation: MetaQuill-KIP combines reusable backbone structure, task-specific inducing points, and fast nonlinear residual refinement instead of full-network retraining.

IV. RESULTS

The results evaluate NTK-KIP across positional encodings, Fourier bandwidths, and distillation levels. Performance improves with richer Fourier features before saturating, while the method achieves strong compression but retains NTK’s lack of feature learning.

  • IV. RESULTS: PSNR increases with Fourier Kbands and then exhibits diminishing returns once the positional features are sufficiently expressive.
  • IV. RESULTS: The main experiments use Kbands = 20 as a practical operating point beyond the steep gain region while avoiding higher computational cost.
  • IV. RESULTS: At Kbands = 30, the best reported PSNR is approximately 48.40 at 50% distillation and approximately 54.24 at 20% distillation.
  • IV. RESULTS: Despite its distilled nonlinear representation, NTK-KIP remains limited by NTK’s inherent lack of learning and feature extraction.

B. MetaQuill Algorithm Results

MetaQuill-KIP combines a shared MetaQuill initialization with KIP-based nonlinear refinement, achieving strong reconstruction quality from lightweight per-task updates. Across MNIST and Flowers, it improves over kernel-only and tangent-only baselines, while sparse-mask results show semantic structure can emerge even when global PSNR is not maximal.

  • MetaQuill setup: MetaQuill learns a shared initialization θS and adapts each new neural field using only a task-specific offset ∆θ.The shared model is trained across MNIST images and supports efficient adaptation to unseen samples.
  • MNIST reconstruction: 32.49 dB PSNR is achieved on MNIST in about 0.46 seconds after 50 nonlinear refinement steps on ∆θ.Combined(KIP→nonlin) starts from a distilled KIP support and keeps θS fixed during refinement.
  • Flowers reconstruction: 26.29 dB PSNR is achieved on Flowers in roughly 3.12 seconds, outperforming KIP(init) and Single-INR reconstruction.The result uses small-step updates to ∆θ around the frozen shared initialization.
  • Long-horizon refinement: Nonlinear ∆θ refinement improves Flowers reconstruction from 9.37 dB at KIP initialization to 26.29 dB after 500 steps.Intermediate checkpoints reach 16.73, 21.77, 24.31, and 25.39 dB at 100, 200, 300, and 400 steps, respectively.
  • Sparse inpainting: At observed fractions ϕ≤0.20, MetaQuill-KIP may have lower global PSNR than Single-INR while producing more coherent petal, stem, and leaf structure.At ϕ=0.10, the scores are 11.97 versus 14.74 dB; at ϕ=0.20, they are 13.59 versus 14.70 dB.
  • Hole filling: In a circular-hole case, MetaQuill-KIP reproduces realistic autumn-leaf texture and correct petal colors without obvious seams.Its global PSNR is similar to Single-INR in that setting, at roughly 13–14 dB.

3) Comparison to diffusion-based inpainting baselines:

MetaQuill-KIP is compared with diffusion-based inpainting systems under different observation levels and adaptation budgets. The paper emphasizes a self-contained, local alternative rather than a head-to-head replacement for large pretrained generative priors.

  • Experimental regime: Diffusion baselines use large pretrained generative priors, whereas MetaQuill-KIP adapts only a small ∆θ around task-agnostic θS on the test image.MetaQuill-KIP uses no external dataset or text guidance, while SD-IA uses about 150 steps and 30–35 seconds of per-image finetuning.
  • Extreme sparsity: At ϕ=0.10, MetaQuill-KIP reports 11.97 dB versus 14.74 dB for Single-INR(masked), despite producing more semantically coherent missing-region structure.The comparison highlights a divergence between global PSNR and qualitative plausibility under extreme sparsity.
  • Interpretation: The authors frame the experiments as evidence that NTK-based neural fields can become nonlinear, rapidly adaptable, and self-contained, not as a universal diffusion replacement.The claimed scope is fast reconstruction with learned structure and local adaptation.
  • MNIST comparison: On MNIST, MetaQuill-KIP exceeds 30 dB PSNR, while DDPM remains under 17 dB and GSDM under 7 dB at comparable lightweight budgets.The table describes adaptation in tens of steps and well under one second of wall-clock time.
  • Flowers comparison: MetaQuill-KIP reaches competitive PSNR in moderate- and high-observation Flowers masks while adapting locally in a few seconds.The paper cites observed fractions ϕ∈{0.5,0.8} as representative settings.

APPENDIX A LIST OF ACRONYMS AND ABBREVIATIONS

The appendix defines implementation terms and examines NTK-based representations through Fourier encoding, kernel-derived W vectors, and Functa comparisons. It reports representational and computational limitations alongside selected frequency-band results.

  • Fourier positional encoding: FPE transforms each d-dimensional positional encoding into d + 2·d·k Fourier features, so increasing k expands the input feature space.With d=2 and k=20, the encoding uses the selected Fourier-band configuration for experiments.
  • Frequency-band selection: The experiments choose kband=20, although the reported elbow value is 15 and the maximum PSNR occurs at kband=50.The chosen value yields slightly higher PSNR than the elbow and somewhat more feature representation.
  • Functa comparison: The Functa comparison treats each data sample as a neural function and reports distillation to approximately 0.5% of the original dataset size while preserving reconstruction features.The experiment tests whether NTK representations support comparable function-level compression and downstream use.
  • Functa via NTK: NTK-derived W vectors replace learned neural fields as dataset representations for downstream modeling in the Functa comparison.The construction uses kernel matrices and labels, with W = Kt(x,x′;θ)^−1Y described for the representation.
  • NTK limitations: The appendix identifies NTK weaknesses in representational structure, representational effectiveness, and runtime complexity.The NTK W representation matches ground-truth image size but lacks efficient structure and effective performance, with runtime at least an order of magnitude less efficient.

A. Representation Learning via NTK Novel Algorithms

The paper proposes NTK-KIP and related finite-width, feature-learning approaches to address NTK’s representational linearity, inefficiency, and weak feature extraction. NTK-KIP distills coordinates and labels into compact inducing supports, producing nonlinear representations while retaining reconstruction capability.

  • NTK-KIP limitations: Fourier positional encodings expand each 2D coordinate to 82 features, so a 5% distilled dataset still produces 410% of the original dataset’s feature count.This feature expansion remains a limitation of the Fourier-based algorithm.
  • NTK-KIP: NTK-KIP addresses NTK’s representational linearity by updating positional coordinates through an NTK-derived reconstruction objective.The learned coordinates make the kernel and resulting representation dependent on target labels.
  • NTK-KIP: NTK-KIP distills positional coordinates and optionally labels into a compact support set, replacing pixel-wise kernel solving with a smaller inducing-point solve.The method optimizes the support representation through the NTK to improve feature representation and data distillation.
  • NTK-KIP results: Raw positional encodings achieve 20% distillation above 36 PSNR and 50% distillation above 23 PSNR, while Fourier encodings achieve 80% distillation above 31 PSNR and 95% above 20 PSNR.The Fourier experiments also report improved feature learning and distilled inpainting reconstruction relative to raw positional encodings.
  • Unified direction: The paper positions the combined approach as integrating finite-width NTK, model feature learning, and MAML-like meta-learning without an inner loop.The claimed goal is to address linearity, efficiency, and kernel representation weaknesses together.
  • Finite-width NTK: Finite-width NTK is used because finite networks can exhibit feature learning, whereas infinite-width NTK models have fixed underlying feature maps.The finite-width formulation approximates network training locally around a working point using a first-order Taylor expansion.

SAMPLE A NEW NF

This section develops an efficient meta-learning approach for adapting neural fields through shared parameters and task-specific offsets. Experiments report efficient feature learning and adaptation from a shared model, while finite-NTK conditioning remains a reconstruction challenge.

  • Shared NF training: A shared neural field learns common parameters θS together with task-specific offsets ∆θi across a subset of images.The example trains on 100 images, converting the image dataset into multiple neural-field tasks.
  • Task adaptation: New neural fields are initialized from the shared model and adapted by obtaining only the corresponding task offset ∆θi.The procedure starts from the shared NF base parameters rather than training each new field independently.
  • Results: Using only 100 images, the algorithm achieves high accuracy with efficient runtime and resources and behaves similarly to MAML without its inner loop.The reported procedure makes the MAML inner-loop process redundant in the authors’ experiment.
  • Limitation: Finite-NTK feature learning can lose representation power because the shared-model kernel matrix is ill-conditioned.The high condition number makes the matrix nearly singular and contributes to blurry reconstructions.
  • Combined direction: The authors propose combining the meta-learning approach with NF KIP to jointly address feature learning, nonlinear representation, and efficiency.The unified method is presented as a response to both representational linearity and ill-conditioning-related reconstruction problems.

3) MetaQuill-KIP Algorithm:

MetaQuill-KIP combines a KIP-based warm start with nonlinear offset refinement around a shared neural-field initialization. The robustness protocol corrupts only observed pixels and compares reconstruction methods under Gaussian and impulse noise.

  • MetaQuill-KIP: MetaQuill-KIP represents each task as a shared initialization θS plus a compact offset ∆θi, updating only ∆θi at test time.The shared parameters remain fixed during per-instance adaptation.
  • Tangent prediction: The tangent prediction uses the INR Jacobian evaluated at θS, providing the linearized output around the shared initialization.The input is the coordinate representation used by the INR, including Fourier positional encodings in the experiments.
  • KIP warm start: KIP warm-starts ∆θi by optimizing compact support coordinates and optional pseudo-labels so kernel ridge regression predicts the target from observed pixels.For inpainting, the KIP objective is restricted to observed pixels, producing a structure-aligned but still linearized initialization.
  • Nonlinear refinement: Nonlinear refinement updates only ∆θi using the full model output and a masked reconstruction loss on observed pixels, with mild offset regularization.This stage injects nonlinearity while preserving lightweight adaptation.
  • Robustness protocol: The robustness evaluation uses Flowers random masks with observed fractions 0.10, 0.20, and 0.40, corrupting only observed pixels with Gaussian or salt-and-pepper noise.Evaluation compares against clean ground truth, and qualitative panels include KRR, KIP, and combined KIP-to-nonlinear methods.
  • Evaluation caveat: Hole PSNR is treated as a complementary diagnostic because PSNR can be misleading under extreme sparsity, so qualitative panels assess structural plausibility.The reported PSNR convention normalizes each compared region by its own intensity range.
  • Robustness results: Across most corruption settings, MetaQuill-KIP degrades gracefully with increasing noise and benefits from a higher observed fraction.Qualitative results indicate more globally coherent structure in several low-coverage cases despite small quantitative differences.
Loading 2609.03117v1…