PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Networks as Kernel Learners: The Silent Alignment Effect

Alexander Atanasov, Blake Bordelon, Cengiz Pehlevan

arXiv:2111.00034v2stat.MLcs.LG

TL;DR

The paper asks whether rich feature-learning neural networks can behave as kernel machines with data-dependent kernels. It analyzes silent alignment, in which the NTK aligns its eigenstructure before substantial loss reduction and later grows mainly in scale. The resulting network function is equivalent to kernel regression with the final NTK under small initialization and approximately whitened data, while anisotropy weakens the effect.

  • Problem

    The paper addresses whether neural networks in the rich feature-learning regime can learn a kernel machine with a data-dependent kernel.

    Source: §1 · p. 1

  • Method

    The paper analyzes two-phase NTK evolution and provides an analytical treatment for small-initialization linear networks, supplemented by nonlinear-network experiments.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Under small initialization and approximately whitened data, the final neural network function satisfies a representer theorem for the final NTK, with analogous results observed for nonlinear and multiple-output networks.

    Source: §6 · pp. 9–10

  • Takeaways & Limitations

    Representation learning is not necessarily incompatible with a neural network function being a kernel regression solution, and the learned kernel can be shaped by data.

    Source: §1 · p. 2, §6 · p. 10

    Silent alignment is sensitive to anisotropy: when data are not whitened, the effect can weaken, and the network output can deviate from final-NTK kernel regression.

    Source: §3.2 · p. 7, §6 · p. 10

Abstract

from arXiv · show

Neural networks in the lazy training regime converge to kernel machines. Can neural networks in the rich feature learning regime learn a kernel machine with a data-dependent kernel? We demonstrate that this can indeed happen due to a phenomenon we term silent alignment, which requires that the tangent kernel of a network evolves in eigenstructure while small and before the loss appreciably decreases, and grows only in overall scale afterwards. We show that such an effect takes place in homogenous neural networks with small initialization and whitened data. We provide an analytical treatment of this effect in the linear network case. In general, we find that the kernel develops a low-rank contribution in the early phase of training, and then evolves in overall scale, yielding a function equivalent to a kernel regression solution with the final network's tangent kernel. The early spectral learning of the kernel depends on the depth. We also demonstrate that non-whitened data can weaken the silent alignment effect.

1 INTRODUCTION

The paper asks whether rich feature-learning networks can still implement kernel regression with a data-dependent kernel. It introduces silent alignment and analyzes when evolving neural tangent kernels yield this behavior.

1 INTRODUCTION

  • Silent alignment has two phases: the kernel first aligns its eigenvectors with task-relevant directions while small, then grows mainly in overall scale as the loss decreases.Feature learning therefore occurs before the loss appreciably decreases.Source: §1 · p. 2
  • The paper theoretically shows that when these dynamics hold, the final neural network is a kernel machine using its final, data-dependent NTK.This establishes a concrete link between representation learning and kernel regression beyond the static infinite-width regime.Source: §1 · pp. 1–2
  • For two-layer linear networks with small initialization and whitened data, the NTK aligns early with the optimal linear function while the loss remains nearly unchanged, then changes only in scale.This provides an analytic realization of the silent-alignment conditions.Source: §1 · p. 2
  • In deep MLPs, alignment and appreciable loss decrease have related initialization-dependent timescales that can nevertheless separate, and the final kernel aligns with task-relevant directions.The paper also gives an explicit final-kernel formula for linear networks of arbitrary depth and width.Source: §1 · p. 2
  • Experiments extend the phenomenon to ReLU and Tanh networks and multiple outputs on isotropic data, whereas anisotropy can force eigenvector changes during loss reduction and weaken the effect.In anisotropic settings, the final network output can deviate from a kernel machine using the final NTK.Source: §1 · p. 2

1.1 RELATED WORKS

  • Unlike several prior analyses, the results do not rely on perturbative expansions in width or particular initial conditions, relying instead on small initialization and whitened data.The paper situates this as a distinction from prior work on kernel evolution and linear-network dynamics.Source: §1.1 · pp. 2–3

2 THE SILENT ALIGNMENT EFFECT AND APPROXIMATE KERNEL SOLUTION

The section explains how evolving NTK dynamics can produce a final kernel-regression solution, then models and demonstrates the required two-phase evolution. The key condition is early eigenstructure alignment followed by scale-only growth before substantial loss reduction.

2 THE SILENT ALIGNMENT EFFECT AND APPROXIMATE KERNEL SOLUTION

  • In the infinite-width lazy regime, a static NTK gives ordinary kernel regression, but evolving kernels in the rich regime contribute throughout training and can depend on the data.The paper studies a regime where the initial kernel is small, aligns early, and then evolves monotonically in scale.Source: §2 · p. 3
  • When the kernel evolves early and subsequently increases only in scale, the final network approaches kernel regression with the final kernel rather than the initial one.The approximation holds in the small-initialization limit, with error O(σ^L).Source: §2 · p. 3
  • Figure 1 shows alignment near t ≈50 and kernel-scale and loss changes near t = 300 for the whitened MNIST experiment, while initial-NTK regression poorly predicts the network.The final NTK instead closely describes the trained network’s predictions.Source: Figure 1 · p. 4
  • For whitened MNIST and CIFAR-10, alignment rises and stabilizes before the loss decreases, while Wide-ResNet predictions strongly correlate with the final-NTK predictor.The experiments use ReLU networks and also show the effect with Adam on whitened CIFAR-10.Source: §2 · pp. 3–4, Figure 1 · p. 4

3 KERNEL EVOLUTION IN 2 LAYER LINEAR NETWORKS

The two-layer linear network exhibits silent alignment: its kernel first learns task-relevant structure while small, then grows mainly in scale during data fitting. With whitened data this yields kernel regression with the final NTK, whereas unwhitened data causes spectral realignment that violates silent-alignment conditions.

3 KERNEL EVOLUTION IN 2 LAYER LINEAR NETWORKS

  • The evolving anisotropic term x^T W^T W x′ learns a representation, unlike the isotropic linear-kernel term.The network uses a two-layer linear form f(x) = a^T W x with small Gaussian initialization and gradient-flow training.Source: §3 · p. 5

3.1 PHASES OF TRAINING IN TWO LAYER LINEAR NETWORK

  • For whitened inputs, the network has an early alignment phase followed by data fitting, with the final function solving kernel interpolation using the final NTK.The phases separate when initialization scale σ is small.Source: §3.1 · p. 5, §3.1.2 · p. 6

3.1.1 PHASE I: EARLY ALIGNMENT FOR SMALL INITIALIZATION

  • In Phase I, the kernel gains a rank-one component in the task direction while the loss and network function change little.The correction grows exponentially on timescale 1/s and can overwhelm the initial kernel.Source: §3.1.1 · p. 5, §3.1 · p. 5

3.1.2 PHASE II: SPECTRAL LEARNING

  • In Phase II, the kernel changes only in overall scale as the network escapes the initial saddle point and the loss decreases toward zero.The transition occurs around t ≈ s^-1 log(s/σ^2).Source: §3.1 · p. 5, §3.1.2 · p. 6
  • The early eigenvector-learning timescale t1 ∼ 1/s is independent of σ^2, while active data fitting begins near t2 ≈ t1 log(s/σ^2).Small initialization therefore creates the temporal separation required for silent alignment.Source: §3.1.1 · p. 5, §3.1.2 · p. 6

3.2 UNWHITENED DATA

  • For unwhitened data, the kernel initially aligns toward Σβ but later realigns during spectral learning, so it is not constant up to overall scale.The learned function remains a final-NTK kernel-regression solution in this linear case, but not through silent alignment.Source: §3.2 · pp. 6–7

4 EXTENSION TO DEEP LINEAR NETWORKS

Deep linear networks with small initialization develop task-aligned NTKs, with whitening enabling early alignment followed by scale-only evolution. The analysis extends this picture to multiple depths and output channels, while anisotropy can disrupt it.

4 EXTENSION TO DEEP LINEAR NETWORKS

  • In deep linear networks, balancing adjacent layers constrains weight evolution and provides the basis for analyzing the NTK.The balancing condition holds during gradient descent and is not specific to mean squared error loss.Source: §4 · p. 7

4.1 NTK UNDER SMALL INITIALIZATION

  • Under small initialization, the NTK is an isotropic linear kernel plus a rank-one task-aligned correction whose alignment increases with depth.At convergence, the spike direction approaches the optimal target direction β.Source: §4.1 · p. 7

4.1.1 WHITENED DATA VS ANISOTROPIC DATA

  • For whitened data, the first-layer direction rapidly aligns with β before substantial loss reduction, after which the kernel evolves primarily in scale.The spectral-learning timescale scales as t1/2 ≈ L s(L−2)σ−L+2.Source: §4.1.1 · pp. 7–8, Figure 3 · p. 8
  • For unwhitened data, the first-layer direction initially aligns with Σβ but must realign later, violating the silent-alignment conditions.Anisotropy introduces multiple timescales and prevents the same clean separation between alignment and loss reduction.Source: §4.1.1 · p. 8, Figure 3 · p. 8

4.2 MULTIPLE OUTPUT CHANNELS

  • With C output channels, balancing implies rank-C weight structure and a rank-C kernel, with separate singular modes governing learning dynamics.Silent alignment requires all Phase I alignment timescales to precede all Phase II spectral-learning timescales.Source: §4.2 · p. 8

5 SILENT ALIGNMENT ON REAL DATA AND RELU NETS

Small initialization makes NTK and feature evolution faster than loss evolution in homogeneous networks, enabling silent alignment in whitened nonlinear networks. Data anisotropy introduces later kernel evolution and weakens the final-NTK kernel-machine approximation.

5 SILENT ALIGNMENT ON REAL DATA AND RELU NETS

  • For small initialization, relative feature change can scale as O(σ−L), making NTK evolution much faster than loss evolution but not guaranteeing silent alignment.The gradients must finish changing apart from overall scale before the loss appreciably decreases.Source: §5 · pp. 8–9
  • For whitened ReLU networks, empirical evidence shows the kernel stabilizes before loss reduction, while anisotropic data causes continued alignment changes during learning.The anisotropic setting has multiple loss timescales rather than a single sigmoidal learning curve.Source: §5 · p. 9, Figure 4 · p. 9
  • As data becomes more anisotropic, the kernel evolves nontrivially while loss decreases, so the neural-network function increasingly deviates from regression with the final NTK.Partial whitening of CIFAR-10 provides an empirical test of this dependence on anisotropy.Source: §5 · p. 9, Figure 4 · p. 9

6 CONCLUSION

The paper concludes that rich feature learning can still produce a kernel-regression solution when NTK eigenstructure aligns early and later changes are mainly scalar. It also develops time-varying-kernel dynamics for training and test predictions.

6 CONCLUSION

  • Silent alignment provides a mechanism by which richly trained networks become kernel machines with a data-dependent final NTK.The mechanism is analytically established for approximately whitened linear networks and observed in nonlinear and multi-output networks.Source: §6 · pp. 9–10
  • The paper argues that representation learning is compatible with kernel-regression solutions and may clarify generalization and transfer-learning behavior.It presents silent alignment as one possible temporal mechanism rather than the only route to data-dependent kernel behavior.Source: §6 · pp. 9–10

A.1 TRAINING POINT PREDICTIONS WITH TIME VARYING KERNEL

  • Training errors under a time-varying NTK can be represented through a transition matrix satisfying a linear matrix evolution equation.When kernels at different times commute, the formal Peano-Baker solution simplifies to a matrix exponential.Source: §A.1 · p. 13

A.2 TEST POINT PREDICTIONS WITH TIME VARYING KERNEL

  • Test-point predictions are obtained by integrating the time-varying kernel’s coupling to training errors throughout the full training interval.The constant-kernel infinite-width case is a special case in which the kernel remains fixed during training.Source: §A.2 · p. 13, §A.1 · p. 13

B KERNEL EVOLUTION IN SCALE ONLY

If an initially small kernel first aligns its eigenstructure and then changes only in scale, the learned function approaches kernel regression with the final NTK. Two-layer linear dynamics illustrate the early task-alignment phase underlying this result.

B KERNEL EVOLUTION IN SCALE ONLY

  • The model assumes a small early kernel followed after time τ by evolution through a fixed final kernel multiplied by a scalar g(t).For deep linear networks, the early kernel scale and alignment time satisfy ϵ ∼σ2L−2 and τ ∼σ−L+2.Source: B KERNEL EVOLUTION IN SCALE ONLY · p. 14
  • After alignment, the static eigenbasis of the final kernel permits integrating-factor solutions for each training-error component.The resulting vector dynamics are Δ(t)=exp(−h(t)K∞)Δ(τ).Source: B KERNEL EVOLUTION IN SCALE ONLY · pp. 14–15
  • Under these dynamics, the final predictor converges to kernel regression with the final NTK when limϵ→0 ϵτ(ϵ)=0.For linear depth-L networks, ϵτ ∼σ^L, while the initial output scales as σ^L and becomes negligible.Source: B KERNEL EVOLUTION IN SCALE ONLY · p. 15

C.1 PHASE I: TWO LAYER NETWORK AND KERNEL ALIGNMENT

  • The two-layer alignment variables form a closed early-time system whose alignment grows exponentially, with random initialization typically giving stronger initial kernel than network-teacher alignment.In the large-input limit, q0 ≫ r0 and q(t)=q0 cosh(2ηst).Source: §C.1 · p. 16
  • During the early phase of a two-layer linear network, the NTK develops a growing rank-one component in the teacher direction β while the loss remains relatively high.This component eventually overwhelms the randomly initialized O(σ2) kernel.Source: §C.1 · p. 16

C.2 PHASE I: ERROR IN THE LEADING ORDER APPROXIMATION

The appendix controls approximation error across the early alignment and later growth phases, analyzes anisotropic inputs, and establishes balanced deep-linear dynamics. It also combines phase solutions into accurate full-training formulas.

C.2 PHASE I: ERROR IN THE LEADING ORDER APPROXIMATION

  • The leading-order truncation produces an O(σ2) kernel error while s ≫ σ4, and alignment remains guaranteed when σ2 ≪ s.The correction is bounded by σ4/s, and the alignment time t = 1/s lies within the valid approximation range.Source: §C.2 · p. 17

C.3 PHASE I: TWO LAYER ANALYSIS WITH UNWHITENED DATA

  • For unwhitened data, the early kernel aligns along Σβ, creating a spike that later realigns toward the minimum-norm solution.The late realignment removes the conditions required for silent alignment.Source: §C.3 · p. 18, Figure C.1 · p. 18

C.4 PHASE II: WHITENED DATA

  • Under balance, phase-II scalar dynamics reduce to the Saxe et al. (2014) two-layer solution, enabling matched formulas for the full training trajectory.The matched solutions for r(t) and q(t) are exact up to O(σ2) corrections.Source: §C.4 · p. 18, §C.5 · p. 18

C.5 SOLUTIONS TO THE FULL TRAINING DYNAMICS OF LINEAR NETWORKS AT SMALL INITIALIZATION

  • The exact full-training equations for q(t) and r(t) agree closely with empirical trajectories across network sizes, initialization scales, and whitened datasets.The comparison is illustrated for synthetic and two-class MNIST settings.Source: §C.5 · pp. 18–19, Figure C.3 · p. 19

D BALANCING OF WEIGHTS IN DEEP LINEAR NETWORKS

  • Deep linear networks preserve balance for losses summed over examples, and each single-output layer has a rank-one factorization with a layer-specific singular direction.The rank-one structure supports the subsequent analysis of tangent-kernel evolution.Source: D BALANCING OF WEIGHTS IN DEEP LINEAR… · pp. 19–20

E NTK FORMULA FOR DEEP LINEAR NETWORKS

The deep-linear NTK can be expanded from balanced weights, with alignment and learning times governed by depth and initialization. Its final task-aligned structure also shapes intermediate representations and transfer performance.

E NTK FORMULA FOR DEEP LINEAR NETWORKS

  • For balanced weights, the NTK admits a leading-order expansion in the small initialization scale σ.The expansion uses Wℓ = u(t)rℓ+1(t)rℓ(t)⊤ + O(σ).Source: E NTK FORMULA FOR DEEP LINEAR NETWORKS · p. 20
  • Final kernel alignment increases monotonically with depth L and approaches 1, while loss decay becomes sharper with depth.On whitened synthetic data, the half-loss time scales as t ∼ σ^(−L+2) for L > 2.Source: Figure E.1 · p. 20

E.1 DEEP LINEAR NETWORK DYNAMICS UNDER BALANCE

  • Under balance, the scalar amplitude obeys deep-network dynamics with a learning timescale proportional to σ^(−L+2).The amplitude equation follows from ˙u(t) = u(t)^(L−1)(s − u(t)^L).Source: §E.1 · p. 21
  • Alignment in deep networks occurs on the same initialization scaling as learning but can precede a significant loss decrease, enabling silent alignment.The separation is empirical rather than a strict asymptotic separation in σ.Source: §E.1 · p. 21

E.2 FINAL NTK FOR DEEP LINEAR NETWORKS

  • The final NTK aligns with the optimal task direction independently of input structure, and its class-direction alignment strengthens with depth.The depth trend is supported by the final-NTK experiment on partially whitened CIFAR-10.Source: §E.2 · p. 21, Figure E.3 · p. 22

E.3 FORMULA FOR THE INTERMEDIATE NNGP KERNELS

  • The final intermediate NNGP kernels can be computed for arbitrary input correlation structure once the network reaches a balanced optimum.The derivation expands the weights and collects leading-order terms in σ.Source: §E.3 · p. 22

F GENERALIZATION ERROR IN TRANSFER LEARNING TASK

  • Increasing feature-learning alignment strictly lowers transfer generalization error for tasks aligned with the learned direction.For w = β, increasing A strictly decreases generalization error; larger A also helps correlated transfer tasks at small sample sizes.Source: F GENERALIZATION ERROR IN TRANSFER… · p. 22

F.1 DERIVATION OF LEARNING CURVES

The transfer analysis derives generalization curves for a learned linear kernel by reducing them to its eigenstructure. Alignment with the learned direction determines which new tasks are easiest to learn.

F.1 DERIVATION OF LEARNING CURVES

  • For tasks w = β, increasing alignment A strictly decreases generalization error, as illustrated by the transfer-learning curve.The associated figure evaluates offline generalization error for the learned linear kernel.Source: Figure F.1 · p. 23
  • At fixed alignment strength A, tasks with larger correlation α to β are easier to learn, and the lowest error occurs when both A and α are large.The result follows from the fraction of task variance along the large-eigenvalue direction.Source: §F.1 · p. 23
  • The transfer kernel has one eigenvalue A + 1 along β and eigenvalue 1 in the D − 1 orthogonal directions.This follows from identity-covariance inputs and the linear-kernel eigenfunction decomposition.Source: §F.1 · p. 23

G LINEAR NTKS DURING GD LEARN THE SAME FUNCTION

Linear networks trained by gradient descent can produce the same interpolating function throughout NTK evolution, namely the minimum-norm pseudo-inverse solution. In multi-class settings, final kernels and intermediate NNGP kernels reflect class structure and depth-dependent spectral learning.

G LINEAR NTKS DURING GD LEARN THE SAME FUNCTION

  • For whitened data, the kernels encountered during gradient descent yield the same minimum-norm interpolator despite changing in scale and alignment.The final pseudo-inverse solution minimizes the RKHS norm for every kernel along the trajectory.Source: G LINEAR NTKS DURING GD LEARN THE SAME… · p. 24

G.1 DEEP LINEAR NETWORKS FROM SMALL INITIALIZATION LEARN PSEUDO-INVERSE

  • Deep linear networks from small initialization learn β* = X(X⊤X)−1y, the minimum ℓ2-norm interpolating solution.The learned direction remains in the span of the training examples, forcing the pseudo-inverse form.Source: §G.1 · pp. 24–25
  • Anisotropy does not change the final linear-network function, but it can prevent nonlinear networks from satisfying the silent-alignment conditions.For unwhitened nonlinear data, late kernel realignment can make the final output deviate from final-NTK regression.Source: §G.1 · p. 25

H FINAL KERNEL IN MULTI-CLASS NETWORKS

  • In multi-class networks, final-kernel contributions depend on how output channels align with class directions.Balancing and alignment produce an orthogonal, balanced configuration whose terms are indexed by output channels.Source: H FINAL KERNEL IN MULTI-CLASS NETWORKS · p. 25
  • For depth L, alignment and singular-value learning times scale with sα^−1σ^(2−L), while differing singular values can be learned at separated times.The time differences depend on the singular-value structure of the data.Source: H FINAL KERNEL IN MULTI-CLASS NETWORKS · p. 25

H.1 FINAL NNGP IN MULTI-OUTPUT CASE

  • Final NNGP kernels interpolate between the input Gram matrix X⊤X at layer 0 and the label Gram matrix Y⊤Y at layer L.This describes how learned representations evolve across depth.Source: §H.1 · p. 25
  • The multi-class phase separation becomes less clear in deeper networks, where singular values are learned at different times and produce separated loss drops.Similar dynamics occur for tanh networks on whitened MNIST.Source: Figure H.1 · p. 26

H.2 MORE REFINED BALANCING ANALYSIS

The refined balancing analysis shows how initialization-dependent corrections vary with depth and widths, while the laziness analysis finds that gradient features can evolve much faster than the loss from small initialization.

H.2 MORE REFINED BALANCING ANALYSIS

  • The refined derivation recursively constructs singular value decompositions after incorporating initial conditions into the conservation laws.The analysis also notes that the network need not be in the lazy regime.Source: §H.2 · p. 26
  • For large-width networks, initial weight-matrix products concentrate around their random-initialization expectations, enabling refined balancing corrections.The concentration improves as adjacent widths grow, with an analogous last-layer condition when the number of classes is sufficiently large.Source: §H.2 · p. 26
  • When all network widths are equal, the correction term g vanishes, so the first moment of the random weights contributes nothing.The correction otherwise depends on initialization scale, depth, and network widths.Source: §H.2 · pp. 26–27

I LAZINESS IN HOMOGENOUS NETWORKS

  • Small initialization makes gradient NTK features and the kernel evolve more rapidly than the loss, with the separation amplified by greater depth and smaller initialization scale.The scaling argument uses homogeneous-network estimates for the gradient and Hessian.Source: I LAZINESS IN HOMOGENOUS NETWORKS · p. 27

J RESNET EXPERIMENTAL DETAILS

The ResNet experiments vary depth, width, and optimizer-related kernel dynamics to assess silent alignment and prediction by the final tangent kernel. Across the reported settings, deeper or wider models train faster, and the final kernel better predicts network functions than the initial kernel.

J RESNET EXPERIMENTAL DETAILS

  • For the k = 4, b = 1 experiment, loss decreases significantly before final alignment, so the final NTK is not perfectly correlated with the final neural-network function.The setup uses Adam with learning rate η = 10^-5 and specified initial weight scales; a small first-layer variance produces less stable learning and worse alignment.Source: J RESNET EXPERIMENTAL DETAILS · p. 27, Figure J.1 · p. 28
  • The depth study varies the number of blocks per group b, producing WideResNets with 6b + 1 trainable convolutional layers.This setup supplies the depth range used for the reported alignment experiments.Source: J RESNET EXPERIMENTAL DETAILS · p. 27

J.1 ADAPTIVE OPTIMIZERS AND THE RELEVANT KERNEL

  • Adaptive optimizers require a time-dependent relevant kernel determined by parameter-specific learning-rate functions, but this kernel was not calculated because the available Neural Tangents API did not support it.The passage identifies this calculation as a possible future investigation.Source: §J.1 · pp. 27–28
  • The final NTK closely predicts WideResNet functions across depths, while the initial NTK does not; alignment is nearly reached when the kernel norm is only 10% of its final value.Deeper models train faster with Adam, and the final NTK norm increases with depth.Source: Figure J.2 · p. 28
  • Wider WideResNets train faster, their kernel norm grows more rapidly but by a smaller amount, and the final kernel predicts the network function better than the initial kernel.Alignment approaches its asymptote by the time the kernel norm reaches 10% of its final value.Source: Figure J.3 · p. 29
Loading 2111.00034v2…