On the Inductive Bias of Neural Tangent Kernels

Alberto Bietti; Julien Mairal

Source-linked AI summary

On the Inductive Bias of Neural Tangent Kernels

Alberto Bietti, Julien Mairal

arXiv:1905.12173v2May 29, 2019stat.MLcs.LG

TL;DR

Over-parameterized neural networks can exhibit learning dynamics governed by an initialization-defined neural tangent kernel, motivating analysis of its inductive bias. The paper characterizes NTK-associated RKHS properties, deriving hierarchical CNN feature maps and studying smoothness, approximation, and deformation stability. ReLU NTKs offer improved approximation properties but weaker smoothness, while CNN mappings still admit deformation-stability guarantees.

Problem
The paper asks what inductive bias is imposed by neural tangent kernels and their associated RKHSs in the over-parameterized learning regime.
Method
The paper analyzes NTK RKHS smoothness, approximation, and stability, and derives hierarchical feature maps for convolutional architectures with generic patch and pooling operators.
Results
ReLU NTKs have better approximation properties but weaker smoothness; CNN mappings remain stable to deformations under a weaker guarantee.
Takeaways & Limitations
The NTK regime provides an inductive bias relevant to natural signals by combining improved approximation with a weaker form of deformation stability.
Takeaways & Limitations
Finite-neuron networks do not belong to the same RKHS as their limiting NTK, and deep convolutional approximation is left for future work.

Abstract

from arXiv · show

State-of-the-art neural networks are heavily over-parameterized, making the optimization algorithm a crucial ingredient for learning predictive models with good generalization properties. A recent line of work has shown that in a certain over-parameterized regime, the learning dynamics of gradient descent are governed by a certain kernel obtained at initialization, called the neural tangent kernel. We study the inductive bias of learning in such a regime by analyzing this kernel and the corresponding function space (RKHS). In particular, we study smoothness, approximation, and stability properties of functions with finite norm, including stability to image deformations in the case of convolutional networks, and compare to other known kernels for similar architectures.

1 Introduction

The paper studies the inductive bias of neural tangent kernels in an over-parameterized regime, focusing on RKHS smoothness, approximation, and deformation stability. It finds that ReLU NTKs trade weaker smoothness for improved approximation, while CNN mappings remain deformation-stable under a weaker guarantee.

Over-parameterized networks can fit many models, making gradient-descent optimization important for selecting models that generalize well.
The paper analyzes the RKHS associated with NTKs to characterize functions learned efficiently in the lazy-training regime.
For ReLU networks, NTK mappings are not Lipschitz but satisfy a weaker Hölder smoothness property.
For CNNs, the weaker smoothness property still yields stability of kernel mappings to input deformations such as translations or small rotations.
The full two-layer ReLU NTK has better approximation properties than function classes associated with last-layer training or Gaussian-process limits.

2 Neural Tangent Kernels

The paper introduces NTKs as limiting kernels governing linearized training dynamics when networks are sufficiently over-parameterized. It derives hierarchical feature-map representations for fully connected and convolutional architectures, including generic patch extraction, pooling, and downsampling operators.

2.1 Lazy training and neural tangent kernels: In sufficiently over-parameterized networks, weights remain close to initialization and training is approximated by linearization around the initial parameters.
2.1 Lazy training and neural tangent kernels: As width tends to infinity, linearized features converge to the neural tangent kernel, which remains fixed during training under appropriate assumptions.
2.1 Lazy training and neural tangent kernels: For ℓ2 loss, gradient descent in this limit leads to the minimum-norm kernel least-squares fit of the training set.
2.1 Lazy training and neural tangent kernels: Fully connected NTKs admit recursive feature-map representations built from arc-cosine kernel mappings and tensor products.
2.2 Neural tangent kernel for convolutional networks: The CNN construction applies to signals in ℓ2(Z^d, R^m0), with infinite support avoiding boundary conditions for deformations and pooling.
2.2 Neural tangent kernel for convolutional networks: For CNNs, the NTK has a hierarchical feature map supporting arbitrary patches, linear pooling, and downsampling through generic linear operators.

3 Two-Layer Networks

For two-layer ReLU networks, the NTK RKHS trades weaker smoothness for stronger approximation properties, while its behavior depends on activation, input domain, and training regime.

3.1 Smoothness of two-layer ReLU networks: The two-layer ReLU NTK mapping is not Lipschitz, unlike the kernel obtained by fixing first-layer weights and training only the second layer.This non-Lipschitzness holds even on the unit sphere and implies that the RKHS contains unit-norm functions with arbitrarily large Lipschitz constants.
3.1 Smoothness of two-layer ReLU networks: The ReLU NTK nevertheless satisfies a weaker Hölder-type smoothness property that supports deformation-stability guarantees.The weaker behavior is attributed to the first-layer gradient contribution in the kernel mapping.
3.2 Approximation properties for the two-layer ReLU NTK: The NTK has better approximation properties than the fixed-first-layer kernel, creating a tradeoff between smoothness and approximation.Its spectral analysis uses a Mercer decomposition in spherical harmonics and studies eigenvalue decay.
3.2 Approximation properties for the two-layer ReLU NTK: For the ReLU NTK, nonzero eigenvalues decay asymptotically as µk ∼ C(p)k^-p, while selected odd-degree components have zero eigenvalues.Zero eigenvalues exclude functions with nonzero coefficients on the corresponding basis elements; adding a bias may prevent them.
3.2 Approximation properties for the two-layer ReLU NTK: Even smooth functions with sufficiently many bounded derivatives belong to the RKHS, and even Lipschitz functions admit approximations with improved rates over the fixed-first-layer kernel.The stated sufficient derivative condition is s ≥ p/2, compared with s ≥ p/2 + 1 for the fixed-first-layer kernel.
3.2 Approximation properties for the two-layer ReLU NTK: For inputs in R^p, the homogeneous NTK RKHS consists of functions f(x)=∥x∥g(x/∥x∥), with g in the spherical RKHS and equal corresponding norms.Non-homogeneous functions can be represented using an augmented variable while preserving regularity properties on bounded balls.

4 Deep Convolutional Networks

The deep-CNN NTK mapping is analyzed for deformation stability under explicit pooling and patch-scale assumptions. ReLU-induced weaker smoothness still yields a stability guarantee, but with weaker depth and deformation dependence than a fixed-layer alternative.

Stability guarantee: For C1 diffeomorphisms, the stability bound separates translation invariance from deformation stability through ||τ||∞ and ||∇τ||∞ terms.The Jacobian norm measures deformation size, while the translation term decreases with the scale of the final pooling layer.
Stability guarantee: The ReLU NTK mapping has faster-than-linear growth for small deformation size because of weaker smoothness from the arc-cosine 0 kernel.This contrasts with settings where the smoothness function is linear in the deformation magnitude.
Architectural assumptions: The NTK representation Φn(x) is formed after an initial anti-aliasing pooling layer A0, with patch sizes controlled by pooling scales.The architecture assumes patch scale is tied to the preceding pooling scale, which typically grows exponentially with depth.
Comparison: Compared with the bound in, stability is weaker because deformation dependence grows faster with ||∇τ||∞ and depth dependence is n2 instead of n.The authors attribute part of the depth gap to norm and smoothness constants that grow with depth.
Comparison: Fixing all layers except the last yields feature maps with the improved stability guarantee of, while training all layers may support less stable but more discriminative functions.The deep-convolutional approximation question is left for future work.

5 Discussion

The discussion characterizes the NTK lazy-training regime through approximation, smoothness, stability, and statistical trade-offs. It also cautions that finite networks and feature-learning behavior are not fully represented by the limiting RKHS.

Main trade-off: The ReLU NTK has better approximation properties than other neural-network kernels, but weaker smoothness that still guarantees deformation stability for CNNs.These properties may help with abundant data but can worsen estimation error when data is scarce.
Scope limitations: Finite-neuron networks do not belong to the limiting RKHS, so they may be less stable outside training data because of linearization and kernel-approximation errors.Approximating some non-smooth functions may also require very many neurons.
Scope limitations: The lazy regime does not capture early-layer feature movement, such as learning Gabor filters, which may improve statistical efficiency.The authors suggest studying an intermediate regime where neurons can move in early layers.

A.1 Proof of Lemma 1

The proof of Lemma 1 derives the stated feature-map representation by induction from the preceding kernel relations and feature-map definitions.

Inductive proof: The proof proceeds by induction using equations (6) and (7) together with the definitions of ϕ1 and ϕ0.A Hilbert-space relation for three vector pairs is then invoked to complete the result.

A.2 Proof of Proposition 2 (NTK for CNNs)

The proof establishes the convolutional NTK through Gaussian-process limits and recursive gradient calculations. Infinite-width limits and the law of large numbers identify the covariance and gradient terms needed for Proposition 2.

Gaussian-process limit: The pre-activations converge in law to independent centered Gaussian processes with covariance Σk as layer widths grow.The base case follows from Gaussian weights, while induction and the law of large numbers extend the result across layers.
Limit argument: The proof repeatedly converts width limits into expectations using the law of large numbers, completing the covariance and gradient identities.The final proposition follows after combining the separately established terms.
Gradient recursion: The convolutional-layer gradient recursion is proved inductively for pre-activation gradients as widths m1,…,mn−1 tend to infinity.The proof uses linearity, the chain rule, and the inductive hypothesis for preceding layers.
NTK construction: The NTK is obtained by decomposing the prediction-gradient inner product into the final-layer and preceding-layer parameter contributions.The two terms are evaluated in the infinite-width limit and then combined.

B.1 Proof of Proposition 3

The proof shows that the relevant supremum is unbounded and uses this to construct unit-norm functions with arbitrarily large Lipschitz constants.

The supremum over x and y is unbounded.
For any L > 0, there exist x and y whose kernel-mapping distance exceeds L times their input distance.
A unit-RKHS-norm function can therefore have a Lipschitz constant larger than any prescribed L.

B.2 Proof of Proposition 4 (smoothness of 2-layer ReLU NTK)

The proof establishes smoothness bounds for the two-layer ReLU NTK by reducing general inputs to normalized vectors on the sphere and applying kernel and deformation inequalities.

For spherical inputs, the relevant distance quantity decreases from 1 to 1/2π and is therefore bounded above by 1.
When α ≥ 1, scaling one spherical vector cannot reduce its distance from another vector.
The proof transfers the spherical bound to normalized general inputs using the 0-homogeneity of ϕ0.
For inputs with unequal norms, the argument combines bounds for ϕ0 and ϕ1 and concludes by symmetry.
The NTK on the sphere is represented as a dot-product kernel κσ(u) = uκ1(u) + κ0(u), with smoothness inherited through its component kernels.
The convolutional deformation proof uses prior stability results, pointwise commutation with deformation operators, intermediate feature maps, and an unrolled recurrence.

C.2 Dot-product kernels and spherical harmonics

This section uses spherical harmonics to decompose dot-product kernels and characterize their RKHS through eigenvalue supports, decay, and differentiability conditions.

The Mercer decomposition expresses a positive-definite dot-product kernel through spherical-harmonic components and their eigenvalues.
A function belongs to the RKHS only if its coefficients vanish at degrees whose kernel eigenvalues are zero.
If f is s-times differentiable with derivatives bounded by η and maxk≥1,µk≠0 1/(k2sµk) < C, then its RKHS norm is bounded by C′η2.
For arc-cosine kernels, eigenvalues vanish beyond degree α in the parity-restricted set and otherwise decay as k−p/2−α.
The two-layer ReLU NTK has positive zeroth- and first-degree eigenvalues, and its approximation analysis uses the square-root integral operator to scale Fourier coefficients by √µk.

D Details on Numerical Experiments

The numerical study evaluates RKHS distances between transformed MNIST images and reference digits using convolutional kernels built from a two-layer ReLU architecture.

The experiments use Infinite MNIST images with random translations and small smooth deformations approximated through image tangent vector fields.
The convolutional architecture has two 3x3 patch-based layers, ReLU activations, Gaussian pooling, and subsampling factors 2 and 5.
Figure 2 compares deformations, translated deformations, same-label training digits, and arbitrary-label training digits against a reference “5”.
Figure 1 reports average relative RKHS distances for sets of 20 images across deformation sizes α ∈ {0.01, 0.03, 0.1, 0.3, 1, 3}.
The relative distance is normalized by the RKHS norm of the reference image to reduce sensitivity to kernel choice.