PaperlayerView exact revision on arXiv

Source-linked AI summary

The proton structure one mode at a time: a Neural Tangent Kernel analysis of a DIS fit

Amedeo Chiefa, Luigi Del Debbio, Richard Kenway

arXiv:2610.36034v1hep-phcs.LG

TL;DR

PDF extraction is an ill-posed inverse problem, so methodology helps determine what finite data leave unconstrained. This paper extends NTK gradient-flow analysis to a multi-flavour DIS fit and examines training and initialisation. It finds that a small number of leading modes reconstruct the underlying law, while sub-leading modes carry noise.

  • Problem

    Finite experimental data cannot determine the continuous PDF functions, leaving methodology to fix unconstrained directions in an ill-posed extraction problem.

    Source: §1 · p. 3

  • Method

    The study extends analytical gradient-flow and NTK analysis to a multi-flavour PDF fit and compares it with a trained network.

    Source: §6 · p. 45

  • Results

    About 30 leading modes reconstruct the underlying law, versus 763 network parameters, while sub-leading modes are poorly constrained and carry noise into the solution.

    Source: §6 · p. 45

  • Takeaways & Limitations

    The parameter count can overestimate the effective degrees of freedom needed to reconstruct the underlying law.

    Source: §6 · p. 45

    The analytical treatment does not include parameter-dependent normalization factors that make the model non-local in x, so the sum rules are not imposed.

    Source: §2.1 · p. 6

Abstract

from arXiv · show

We study the training dynamics of a feed-forward neural network (NN) through the analysis of the neural tangent kernel (NTK) in a phenomenologically relevant fit of parton distribution functions (PDFs) from deep inelastic scattering (DIS) data. The exact integration of the gradient flow derived in a previous single-flavour analysis is generalised to the multi-flavour case, yielding an analytical perspective on the relationship between data and methodology during training. The Shapley value recently used in the context of PDFs is derived within our statistical framework for observables that depend linearly on the PDFs, and we outline a geometric interpretation of training. We expose the role of gradient-based optimisation algorithms and of the parametrisation in the regularisation of the ill-posed inverse problem that affects the determination of PDFs from experimental data, demonstrating that the regularisation amounts to the incremental inclusion of NTK modes, each learned at the timescale set by the corresponding eigenvalue. We find that only a few leading modes of the NTK are necessary to reconstruct the target PDFs, while the noise of the data is carried by the sub-leading ones learned at later stages of training. Finally, we consider the impact of the initialisation on the training dynamics, and recover a transient behaviour observed in the previous analysis and in the machine learning (ML) literature. Beyond sharpening our understanding of the training dynamics, our framework can be applied to other parametrisations (such as fixed functional forms) and ML applications in physics, as well as to stress tests in the presence of inconsistent data.

1 Introduction

PDF extraction is an ill-posed inverse problem in which methodology helps determine what finite data cannot constrain, yet the effects of standard PDF-fitting methods remain insufficiently understood. This work extends NTK analysis to a multi-flavour DIS fit, using it to examine training, implicit regularisation, and the influence of initialisation.

1 Introduction

  • PDF extraction is ill-posed because finite data cannot determine a continuous function, leaving methodology to fix unconstrained directions through implicit and explicit assumptions.PDFs encode the proton’s inner structure and are inputs to theoretical predictions and searches for new physics.Source: §1 · p. 3

Related works

  • Although tools for comparing PDF methods, making assumptions explicit, and explaining data’s role exist, systematic study of standard PDF-determination methodologies remains lacking.The authors extend a previous single-flavour NTK analysis, whose applicability to realistic fits was still an open question.Source: §1 · p. 3

Our contribution

  • The paper studies NNPDF4.0 with plain gradient descent in a synthetic-data DIS fit of eight flavours, without sum rules or positivity constraints.Synthetic data generated from known PDFs allow validation against the underlying law.Source: Our contribution · p. 4
  • Only a few leading NTK modes reconstruct the underlying PDF law, while sub-leading modes carry noise and make stopping a signal–noise trade-off.The study generalises the analytical gradient-flow solution to multiple flavours and validates it against the empirically trained network.Source: Our contribution · p. 4
  • The central PDF values and most of their error budget are data-driven, while initialisation contributes uncertainty that varies by flavour and x-region.In extrapolation regions, the initial-condition contribution remains comparable to that of the data.Source: Our contribution · p. 4

2 Methodological and theoretical framework

This section presents the DIS-fit methodology and extends the gradient-flow analysis to multiple PDF flavours, describing how the NTK and data-weighted operator govern training. It explains how training progressively adds modes and how early stopping balances movement toward the noiseless-data target against the effects of data noise and poorly constrained modes.

2.1 Dataset and methodology

  • The fit uses 2,968 linear-in-PDF DIS measurements and an eight-output neural network to parametrise independent flavours; sum-rule normalisation and positivity and integrability penalties are omitted.The dataset uses NNPDF4.0 DIS inputs, and the analysis uses closure-test replicas with synthetic data. The omitted constraints are excluded because they are not included in the analytical treatment.Source: §2.1 · pp. 5–7

2.2 The analytical solution of the gradient flow

  • The multi-flavour gradient-flow treatment describes neural-network parameter updates as continuous evolution along the negative loss gradient.For sufficiently small learning rates, this flow approximates discrete gradient descent; preprocessing factors remain fixed after initialisation.Source: §2.2 · pp. 7–8

The continuous gradient flow

  • PDF evolution is driven by the interplay of the NTK, which encodes network architecture, and M, which combines physical flavour correlations with experimental uncertainties.Directions in the kernel of M are unconstrained by the data and remain unchanged under the flow.Source: The continuous gradient flow · p. 8

The frozen NTK and the training Hamiltonian

  • In the lazy regime, projecting M onto the NTK image regularises the inverse problem when their kernels intersect only trivially, as for the neural networks considered.The learned modes are not generally single-flavour directions, and each mode’s learning timescale is set by the inverse training-Hamiltonian eigenvalue; that eigenvalue reflects both parametrisation coverage and data resolution.Source: The frozen NTK and the training… · pp. 8, 10, 11

The analytical solution

  • In the lazy regime, the trained PDFs are a linear transformation of the initial PDFs plus a data contribution, with training time acting as a regularisation parameter.The filter factor 1 − e^(-h(i)t) resolves modes with h(i)t ≫ 1 while suppressing modes with h(i)t ≪ 1.Source: The analytical solution · p. 11

2.3 Early stopping as a bias-variance trade-off

  • The early-stopping analysis recasts the solution for L2 pseudodata into teacher and noise components, interpreting them with deterministic initial PDFs and M.This interpretation assumes NTK fluctuations are neglected.Source: §2.3 · p. 12
  • The teacher term incrementally includes modes resolved by the filter and, at infinite time, yields the noiseless-data least-squares solution within f0 + Im(Θ).This term corrects the initial condition using regularised inversion and carries no data noise.Source: §2.3 · p. 12
  • Poorly constrained data directions have the largest impact on the central value and variance, and correspond to slower modes with small h(i).For t ≫ 1/h(i), the mode weights governing the central-value shift and variance reduce to the same factor.Source: §2.3 · p. 12
  • Early stopping sets the trade-off between reducing teacher-term bias and limiting the noise shift and variance introduced as new modes are included.Adding modes improves the teacher term but can worsen overall performance; the noise shift affects the central value, while replica fluctuations contribute to variance.Source: §2.3 · p. 12
  • A small h(i) can also reflect a direction poorly covered by the parametrisation rather than poorly constrained by data, though the authors expect neural networks to cover data-constrained directions.Such a mode would be slow without injecting a large amount of noise.Source: §2.3 · p. 12

3 Information content and geometry of a PDF fit

The section interprets M statistically as information about PDFs carried by the data and geometrically through the interaction of the data, parametrization, and training flow. It also derives the DIS Shapley analysis from M and presents a visual, explicitly intuitive account of training geometry.

3.1 Resolving power of the data

  • M is the Fisher information matrix (FIM) and a positive-semidefinite metric on PDF space induced by the statistical structure of the data.Source: §3.1 · pp. 13–14
  • Large M directions are tightly constrained by data, while its flat directions, especially Ker(M), remain undetermined by data and are fixed only by parametrization.Source: §3.1 · p. 14

3.2 The Shapley value in the DIS case

  • In the DIS Shapley analysis, M-based quadratic forms encode individual-flavour sensitivity through M_ii and interference between perturbed flavours through off-diagonal M_ij terms.Source: §3.2 · pp. 14–15

3.3 A geometric interpretation of training

  • The geometric account is an intuitive visualization rather than a rigorous treatment of the geometric structure of parametric PDF fits.Source: §3.3 · p. 15

The role of the model

  • The PDF manifold and tangent space are invariant under reparametrization, but the NTK depends on parametrization because it encodes how tangent vectors cover that space.Source: The role of the model · p. 16

Training as a flow on the manifold

  • With the Euclidean parameter-space metric, tangent vectors contribute independently with equal weighting; choosing the parametric-model FIM instead gives natural gradient flow.Source: Training as a flow on the manifold · p. 17

The frozen NTK dynamics as a tangent plane

The physical directions

  • The training Hamiltonian combines the NTK spectrum, which describes parametrization coverage, with M, which weights directions by how well the data resolve them.Source: The physical directions · p. 18

4 The NTK analysis of a DIS-only fit

This section analyzes the DIS-only PDF fit through its NTK training dynamics, tracing how hierarchical modes enter training and how initialization and data shape PDF uncertainties. It also documents how initialized-PDF one-sigma widths vary with x, flavour, and initialization exponent.

4.1 The model at initialisation

  • At initialisation, PDFs are flavour-similar without preprocessing; preprocessing deforms flavour shapes and usually narrows their spread, especially at small x.The exceptions to the reduced variance are g and Σ; including log x also prevents the network from collapsing to zero at small x. [44]Source: §4.1 · p. 19

4.2 Diagnostic of the NTK during training

  • Unlike the plateau-then-drop pattern in [44], training and validation losses decay smoothly, falling rapidly in early epochs before slowing toward a plateau.The analysis does not impose momentum or valence sum rules, positivity, or integrability constraints in these fits.Source: §4.2 · pp. 20, 22

Spectrum of the NTK

  • The NTK spectrum is hierarchical: new modes enter its image mainly early in training, after which leading eigenvalues stabilise and lower ones rise slowly.Preprocessing preserves this qualitative behaviour but changes eigenvalue magnitudes and spread; it suppresses modes in regions where its factors are small, leaving fewer active modes than the bare network. [Figure 8]Source: Spectrum of the NTK · pp. 22–23, Figure 8 · p. 23

Modes of the training Hamiltonian

  • Hamiltonian modes have non-trivial shapes at initialisation, but ranks show no systematic flavour pattern and reorganise across training; their forms depend on parametrisation and initialisation.The first mode remains tied to Σ over the examined epochs, while other ranks' flavour contributions change without visible correlation. [Figure 9]Source: Modes of the training Hamiltonian · pp. 23, 25, Figure 9 · p. 24

4.3 Analytical training

  • Each analytical mode is incorporated on a timescale set by its eigenvalue and learning rate, so larger-eigenvalue modes are learned first and smaller ones later.The timescales span about 14 orders of magnitude, and evolution to the 20,000-epoch horizon retains less than a third of available modes.Source: §4.3 · p. 25

Convergence of the analytical solution

The initial condition and the data

  • The data-driven term dominates PDF central values, while the initial-condition term mainly contributes uncertainty; initialisation adds no significant bias in the studied setting.The initial-condition uncertainty varies by flavour and training time, remaining up to 80% for T3 until x ≲ 10^-1; beyond the training horizon, data-driven uncertainties dominate small x. [Figures 13–14]Source: The initial condition and the data · pp. 28, 30, Figure 13 · p. 29, Figure 14 · p. 31
  • Figure 15 plots the initial one-sigma PDF widths versus x at Q0 for eight fitted flavours and six initialization exponents, using 120 full-model replicas with preprocessing factors included.Both axes are logarithmic.Source: Figure 15 · p. 32

5 Dependence on the initialisation

This section examines how initialization choices—including the initialization exponent, network width, and initialization variance—shape the NTK and training dynamics. The analysis uses vanilla gradient descent, so whether the findings extend to more efficient optimizers remains unclear.

5.1 A linear model at initialisation

  • In the linear regime, increasing γ rescales the initial PDF widths without changing their x-shape; for γ ≳3, consecutive widths approach a ratio of 65.21.At smaller γ, larger weights saturate the hyperbolic tangents, especially at small x, compressing the spacing between curves and breaking the scaling.Source: §5.1 · p. 33, Figure 16 · p. 33

5.2 Training dynamics at different initialisations

  • For γ ≲2, loss decreases monotonically, whereas γ ≳3 produces a plateau and sudden drop; median escape epochs are ≈2500, 5400, and ≳10^4 for γ = 3, 4, and 5.For γ = 5, 22 of 120 replicas do not escape within 2 × 10^4 epochs. During the plateau the NTK spectrum remains nearly constant; its bulk grows by several orders of magnitude around escape, when hidden-layer saturation takes the network out of its linear regime.Source: §5.2 · pp. 34, 36, Figure 17 · p. 34, Figure 18 · p. 35
  • For γ ≲1, parameter displacement stays within a fraction of its initial norm and the slowly varying NTK signals lazy training; for γ ≥2, parameters travel several initial norms.At the end of training, the relative displacement is about 0.2 for γ = 0.001 and 0.6 for γ = 1; for γ = 5 it exceeds two orders of magnitude, so the initial NTK does not describe the full trajectory.Source: §5.2 · pp. 34, 36, Figure 19 · p. 35
  • The plateau and drop for small initialisations differ from deep-linear saddle-to-saddle transitions: here a single plateau ends when hidden-layer saturation takes the network out of the linear regime.The NTK and loss together diagnose this departure from linear behaviour.Source: §5.2 · pp. 34, 36

Freezing at initialisation

  • Freezing the NTK at initialisation reproduces training for γ = 0.001 and 1, but for γ ≥2 it tracks the plateau, not the post-escape drop or underlying law.In the linear regime, tangent directions span at most 24 network dimensions, or 15 with preprocessing; only after escape do new NTK directions enter and allow the trained network to reach the underlying law.Source: Freezing at initialisation · pp. 36–37, Figure 20 · p. 37, Figure 21 · p. 38

Freezing at the end of training

  • Freezing the NTK at the end of training reproduces the final PDFs for γ = 1, 3, and 5, and reaches the underlying law by 500 epochs for γ = 3 and 5.At that epoch, the trained networks remain on their plateaus, about a factor of two below the underlying law; the late-time NTK can therefore learn the law much faster than the actual trajectory.Source: Freezing at the end of training · p. 38, Figure 23 · p. 39

5.4 Is there an optimal initialisation?

  • The lowest validation loss and fastest training occur near γmin ∼0.6, while γ = 0.5 and 0.75 also provide the closest analytical control over training.The agreement between analytical and trained solutions worsens as γ increases, except for T3, which shows discrepancies even at baseline.Source: §5.4 · pp. 40, 42, Figure 24 · p. 40, Figure 25 · p. 41
  • With γ = 1, ηhmax starts at 0.95 and approaches unity; for γ ≥2 it remains below the stability bound after 2 × 10^4 epochs, increasingly so at larger γ.For γ <1, ηhmax starts above unity and collapses toward one; rescaling the learning rate to satisfy the bound initially keeps its final value closer to its initial value.Source: §5.4 · pp. 40, 42, Figure 24 · p. 40
  • Whether the observed behavior persists with more efficient optimizers such as Adam or Nadam is unclear because the analysis adopts vanilla gradient descent.The authors defer this question to future work.Source: §5.4 · p. 42

5.5 A wide NN is not a substitute for a small initialisation

  • The 500-node-per-layer network’s NTK eigenvalues match the baseline’s scale and exceed those of larger-γ baseline networks, despite a variance matching γ ≈ 2.The wide network’s spectrum also has a tier structure resembling that for γ = 2, whose origin the authors leave unexplained.Source: §5.5 · p. 42
  • Width keeps the NTK O(1) with depth, whereas γ > 1 shrinks it geometrically, explaining their different eigenvalue scales.The authors attribute the difference to the distinct scaling properties of width and γ in the layer-by-layer NTK recursion.Source: §5.5 · pp. 43–44
  • At γ = 1, the wide network had not converged within 2 × 10^4 epochs and showed a different monotonic loss decrease, without the large-γ plateau and sudden drop.Its learning rate was reduced to 10^-7 to account for the increased leading NTK eigenvalues; the comparison in gradient-flow time accounts for the differing learning rates.Source: §5.5 · p. 44, Figure 27 · p. 44

6 Conclusions and outlook

The conclusions characterize PDF training through NTK modes, gradient-flow dynamics, and the effects of initialization. They find that a small leading set reconstructs the underlying law, while later-learned modes carry data noise.

6 Conclusions and outlook

  • The study generalizes the gradient-flow solution to multiple flavours and validates it numerically, while identifying M as the PDFs' Fisher information matrix and relating it to the Shapley value.It also outlines a geometric interpretation of training.Source: §6 · p. 45
  • The NTK spectrum is hierarchical: a brief early rich phase incrementally adds sub-leading modes, then the kernel becomes lazy and the gradient flow is analytically integrable.Modes of the training Hamiltonian capture much of the underlying law's shape at initialization.Source: §6 · p. 45
  • The analytical solution reproduces the trained network well across flavours except for T3 uncertainty at small x; data shape the central PDFs, while initial-condition uncertainty matters more in extrapolation regions.Within the data-covered region, initial-condition uncertainty is suppressed except for T3; its relative weight also depends on flavour and training time.Source: §6 · p. 45
  • About 30 leading NTK modes reconstruct the underlying law, compared with 763 network parameters; sub-leading modes carry noise, making early stopping a signal–noise trade-off.The sub-leading modes are poorly constrained by the data.Source: §6 · p. 45
  • Initialization width substantially affects training: overly narrow parameters cause a plateau until hidden layers saturate, while a slightly wider-than-standard initialization gives the best and fastest fits.The sharp loss and eigenvalue transition is qualitatively distinct from saddle-to-saddle dynamics; small initialization is not equivalent to large width.Source: §6 · p. 45
  • The framework could be extended to optimizers and physical constraints omitted here, and applied to inconsistent-data tests, fixed functional forms, and other particle-physics neural networks.The authors identify these as possible extensions and applications.Source: §6 · p. 45

Code and data availability

The paper identifies the software used for its fits and NTK analysis and states that fit outputs are available from the authors. The appendix introduces a deep linear network approximation and outlines its derivations.

Code and data availability

  • Fits used the open-source NNPDF code's ntk colibri branch, and NTK analysis used ntkpdf, an interface built on Colibri.The passage specifies the branches and commit identifiers for these tools.Source: Code and data availability · p. 47
  • Fit outputs are available from the authors upon request.The passage does not state that the outputs are openly hosted.Source: Code and data availability · p. 47

A The deep linear network approximation

  • The appendix derives initialization scaling and saturation limits for the deep linear network approximation, then derives its tangent subspace and leading NTK tier.It also gives a compact NTK form reducing the eigenvalue problem to 3Nout dimensions.Source: A The deep linear network approximation · p. 48

A.1 Deep linear networks

In the deep linear regime, the network output is linear in x and log x, while its initialization scaling follows a power of the parameter-scale ratio. The analysis relates that scaling to the observed curves and the regime where the approximation holds.

A.1 Deep linear networks

  • With biases zero at initialization, the network output is A x + B log x, linear in its inputs but not in its parameters.The network's output also has bias-related terms; only the final-layer bias term is unaffected by γ scaling.Source: §A.1 · p. 48
  • The figure compares saturation across x for six γ values and shows it is negligible for γ ≳3, but significant at smaller γ, especially at small x.The plotted curves average over 120 replicas.Source: Figure 28 · p. 49
  • At initialization, the γ dependence factors into a power of r, and consecutive integer γ values differ in scale by 1/r = 65.21 for this architecture.The random matrix product is shared across γ when random seeds are held fixed.Source: §A.1 · pp. 48–49
  • The γ = 4 and γ = 5 curves reproduce the predicted asymptotic ratio, validating the linear approximation and explaining why curves are parallel on a logarithmic scale for γ ≳3.The width scaling makes the logarithmic separation independent of x.Source: §A.1 · p. 49

A.2 Saturation of the hidden layers

The appendix quantifies hidden-layer saturation to assess the deep linear network approximation. Saturation is negligible for γ ≳3 and significant at smaller γ, particularly at small x.

A.2 Saturation of the hidden layers

  • Saturation is defined as one minus the layer-average derivative of the hyperbolic tangent; it is zero in the linear regime and approaches one as units saturate.The quantity is evaluated by layer and input x.Source: §A.2 · p. 49
  • For γ ≳3, saturation is negligible across x; at smaller γ it becomes significant, especially at small x, where the log x input is largest.The authors identify γ ≲2 as nonlinear and γ ≳3 as linear, while leaving intermediate regimes for future work.Source: §A.2 · p. 50

A.3 The span of the tangent plane

The network’s tangent vectors span a three-dimensional functional subspace, which constrains the dimension and functional form of the NTK modes. The full model applies a flavour rotation and preprocessing to the network outputs.

A.3 The span of the tangent plane

  • Each output’s tangent-vector form combines constant, x, and log x terms with coefficients that depend on network parameters.The same form applies to the full model after rotating outputs to the FK flavour basis and applying large- and small-x preprocessing factors.Source: §A.3 · p. 50
  • The tangent vectors arise from three parameter groups: last-layer biases, hidden-layer biases, and weights, with weights contributing constant and non-constant terms.The last-layer biases contribute only constant terms; hidden-layer biases also contribute only constant terms, while weights contribute both types.Source: §A.3 · pp. 50–51
  • Although the network has 763 tangent vectors, they lie in the three-dimensional subspace Vlin, constraining the NTK modes’ dimension and functional form.The modes are linear combinations of tangent vectors and are therefore also contained in Vlin.Source: §A.3 · p. 51

A.4 The NTK of a deep linear network

For the deep linear network, the NTK factorises into grid- and flavour-dependent structure and a parameter-dependent matrix. This factorisation reduces its non-zero eigenvalue problem and makes the role of the input-function basis explicit.

A.4 The NTK of a deep linear network

  • The NTK factorises into a quadratic form: Ψ depends on the grid, input functions, and flavour map, while G carries parameter dependence.For a DLN, the tangent-plane orientation is fixed by Ψ; parameter-space directions that leave G unchanged leave the NTK unchanged, including at least 463 directions in the stated 763-parameter model.Source: §A.4 · p. 53
  • The non-zero NTK spectrum is determined by a 24 × 24 eigenvalue problem, rather than the full Nvec-dimensional problem.The reduced matrix has real, non-negative eigenvalues, and its eigenvectors map to orthogonal NTK eigenvectors.Source: §A.4 · p. 53
  • With the chosen simple input functions, the linear regime approximates only curves proportional to x and log x; richer input functions can broaden that expressiveness.The DLN with arbitrary basis functions is equivalent to a fixed functional form formed by their linear combinations.Source: §A.4 · p. 53
  • Different weight-and-bias combinations can realise the same fixed functional form, yet yield different tangent-vector sets and learning dynamics.The span Vlin can be the same even when the models explore neighbouring regions differently.Source: §A.4 · p. 53

The reduced matrix G

The reduced matrix G separates constant and non-constant sectors at zero-bias initialisation. As the initialisation width grows, the constant block approaches the identity while the non-constant block is exponentially suppressed.

The reduced matrix G

  • At zero-bias initialisation, the off-diagonal G12 and G13 blocks vanish because bias derivatives contribute only constant terms and the relevant weight contribution disappears.This yields G12 = G21 = 0; the same reasoning applies to G13.Source: The reduced matrix G · p. 54
  • The constant block receives an identity contribution from last-layer biases, independent of γ, while hidden-layer contributions depend on their weight chains.The correction to the constant block is exponentially small in γ and is dominated by the shortest chain.Source: The reduced matrix G · p. 54
  • As the initialisation width grows, Gcc tends to the identity while Gnn is driven to zero exponentially, separating the constant and non-constant sectors.The reduced eigenvalue problem is governed by the product of these blocks with the corresponding grid-moment blocks.Source: The reduced matrix G · p. 55
  • The reduced block structure makes the leading eigenvalues those of the constant-sector block when the lower-left block is negligible.The remaining 2Nout eigenvalues require eliminating ζc because their lower-left and lower-right blocks are both proportional to Gnn.Source: The reduced matrix G · p. 55

The constant sector

In the vanishing-width limit, the constant-sector eigenvalues are set by the flavour multiplicities. The corresponding leading-tier modes are constant in x.

The constant sector

  • The constant-sector eigenvalues reproduce the first tier of the observed spectrum, whose multiplicities come from the FK flavour map rather than the network.In the vanishing-width limit, the eigenvalues are obtained from GccScc, with Scc determined by Ngrid and the fitted-flavour multiplicities.Source: The constant sector · p. 55
  • The leading-tier modes are constant in x and correspond to individual fitted outputs in the vanishing-width limit.Their leading-order coefficient vectors have one nonzero constant-sector component and no non-constant-sector component.Source: The constant sector · p. 55

The non-constant sector

The non-constant sector of the NTK has eigenvalues suppressed by the network factor and approximately separates into two tiers. Its modes combine flavour structure with x- and log x-dependence.

The non-constant sector

  • The 2Nout non-constant eigenvalues scale as O(Gnn), suppressed relative to the constant-sector eigenvalues.The block equations leave a 2Nout-dimensional problem after the constant components are eliminated.Source: The non-constant sector · p. 56
  • Approximate factorisation holds because layer-to-layer fluctuations enter the remainder quadratically; corrections are a few percent at the widths used.The leading corrections are absorbed into G2 and F.Source: The non-constant sector · pp. 56–57
  • The non-constant sector splits into two tiers of Nout eigenvalues, separated by µ+/µ−, with exact eigenvalues reproduced to a few percent at large γ.This accuracy applies where the separation of scales holds.Source: The non-constant sector · p. 57
  • The modes factor into a flavour term and a centred combination of x and log x, with the second and third tiers separated by µ+/µ− rather than by input variable.Because log x has much larger variance on the grid, modes beyond r = Nout are dominated by log x and appear straight on the logarithmic scale.Source: The non-constant sector · p. 57

The tier structure of the NTK spectrum

The initial NTK spectrum develops tiers of Nout modes at large initialisation exponents, while smaller exponents show a smooth decay. The accompanying analysis examines how the kernel changes across initialisation widths during training.

The tier structure of the NTK spectrum

  • The bare-network image dimension approaches the bound of 24 monotonically as γ increases, reaching 24 = 3 × 8 at γ = 5; preprocessing reduces the full-model image dimension.The image-dimension and saturation diagnostics agree on where the linear regime sets in.Source: The tier structure of the NTK spectrum · p. 58
  • At γ = 5, the 24 non-zero eigenvalues form three clear tiers of eight; γ = 4 shows similar grouping, γ = 3 retains it through r = 16, and γ ≤ 2 has no visible tiers.For γ = 5, six eigenvalues in the leading tier are exactly degenerate; the first two tiers are more closely spaced at γ = 4.Source: The tier structure of the NTK spectrum · p. 58
  • The leading tier follows the predicted 1, 0.75, 0.25 × 6 pattern and represents constant functions; the next two tiers mix x and log x sectors.The leading pattern is reported for the γ = 5 curve.Source: The tier structure of the NTK spectrum · p. 58

B NTK dynamics at different initialisation widths

  • The appendix tracks image growth, hidden-layer saturation, mode shapes, eigenvector orientation, and frozen-NTK solutions across six initialisation widths.It focuses on escape from the linear regime and comparisons with the frozen-kernel analytical solution.Source: B NTK dynamics at different… · p. 59

B.1 Escape from the linear regime

As training escapes the linear regime, the NTK image grows, its modes leave the initial linear span, and its eigenvectors rotate. Hidden-layer saturation accompanies the growth, while the timing and pattern of these changes depend on the initialisation width.

B.1 Escape from the linear regime

  • At γ = 5, the median NTK image dimension grows from 24 at initialisation to 71 at escape and 109 at the final epoch, alongside hidden-layer saturation.For γ ≥ 3, new modes enter the image around the escape epoch; saturation makes higher-order activation terms relevant and allows a richer functional space.Source: §B.1 · p. 59
  • At initialisation, γ = 4 and 5 modes largely lie in Vlin: the leading tier is constant in x, while modes beyond it are approximately straight in log x.At γ ≲ 2, the modes are far from Vlin and lack the tier structure; after escape, modes become more complex even at large γ.Source: §B.1 · p. 60
  • At escape, residuals from Vlin are large for every γ; initially they are small and tiered at large γ but grow and merge as γ decreases.For γ ≲ 2, the initial modes are already far from Vlin and no tier structure remains.Source: §B.1 · p. 60
  • At γ = 5, the leading mode and tier barely drift through the plateau, then rotate at escape; at γ = 1, drift rises smoothly from the first epochs.At about 8 × 10^3 epochs, d1 = 0.002 and d8 = 0.000 to three decimals; at escape they reach 0.26 and 0.11, respectively.Source: §B.1 · pp. 60, 62
  • By the final epoch, d24 is 0.14–0.26 across five initialisation widths, while the γ = 0.001 fit has d1 = 0.90; the NTK is stationary within the linear regime.At γ = 5, stationarity comprises fixed eigenvalues, eigenvectors, and image dimension, with the image dimension exactly 3Nout.Source: §B.1 · p. 62

B.2 Timescales of the frozen NTK at initialisation

The frozen initial NTK resolves only the constant-sector modes within the training horizon for γ = 4 and 5. Directions with x or log x dependence remain unresolved, supporting the conclusion that escape requires nonlinear network evolution.

B.2 Timescales of the frozen NTK at initialisation

  • Within the training horizon, the frozen NTK for γ = 4 and 5 resolves exactly eight constant-sector modes, generated by last-layer biases and independent of γ.These are the modes of the constant sector described by eq. (128).Source: §B.2 · p. 62
  • Every frozen-kernel direction carrying x or log x dependence has a resolution time of at least 3 × 10^6 epochs and remains unresolved.The comparison is against the training horizon.Source: §B.2 · p. 63
  • The escape is therefore attributed to nonlinear evolution driven by weight growth and hidden-layer saturation, rather than slow learning under the initial kernel.The saturation takes the network out of Vlin.Source: §B.2 · p. 63

B.3 The frozen NTK along the training

The frozen-NTK solution is compared with the trained network at several freezing epochs, using a discrepancy weighted by experimental uncertainties. Its accuracy depends on initialization: for lazy fits it remains close throughout training, while for larger γ it improves around the escape epoch.

B.3 The frozen NTK along the training

  • The comparison evolves the analytical solution from the trained network’s initial condition and measures its end-of-training discrepancy using a dataset-wide metric weighted by experimental uncertainties.The metric projects PDFs onto data space and uses the whole dataset rather than each replica’s training split.Source: §B.3 · p. 63
  • Figure 34 plots initial frozen-NTK mode resolution times against mode rank for six γ values, marking the training horizon, constant-sector modes, and escape-limited resolution.Modes above the horizon have not had time to be learned; each star marks the slowest mode resolved by that fit’s escape epoch.Source: Figure 34 · p. 64
  • For γ ≲1, discrepancy stays below 0.1: it falls from D ≃0.08 to D ≃0.01 for γ = 1, while remaining between 0.02 and 0.08 for γ = 0.001.For both values, the frozen-NTK description holds approximately throughout training, consistent with the lazy regime.Source: §B.3 · pp. 63–64
  • For γ ≥ 2, discrepancy falls from D ≃0.4–0.5 at initialization to D ≃0.03–0.07 after escape, with curves sharing a shape when time is measured in escape epochs.Between τNTK = 0.5 τesc and 1.5 τesc, D drops by a factor of four to ten; initialization mainly controls when the frozen-NTK description becomes accurate, not its eventual accuracy.Source: Figure 35 · p. 64, §B.3 · p. 65

Related papers

Loading 2610.36034v1…