PaperlayerView exact revision on arXiv

Source-linked AI summary

On the linearity of large non-linear models: when and why the tangent kernel is constant

Chaoyue Liu, Libin Zhu, Mikhail Belkin

arXiv:2010.01092v3cs.LGstat.ML

TL;DR

The paper asks when and why wide nonlinear networks become linear in their parameters and their tangent kernels become constant. It develops a Hessian-scaling framework, finding that vanishing Hessian norms explain this behavior for certain architectures, while nonlinear output layers can prevent it and successful optimization can occur without near-linearity.

  • Problem

    The paper addresses which structural and scaling conditions make tangent kernels constant in wide nonlinear networks, and when this transition fails.

  • Method

    The paper analyzes model Hessian spectral-norm scaling relative to gradient and tangent-kernel scaling across standard neural-network architectures.

  • Results

    Certain wide networks with linear output layers have vanishing Hessian norms and near-constant tangent kernels, whereas nonlinear output layers generally prevent constancy; efficient optimization can still occur with non-constant kernels.

  • Takeaways & Limitations

    Tangent-kernel constancy is a structural consequence of Hessian scaling, not a universal property of wide networks or a necessary condition for gradient-descent convergence.

  • Takeaways & Limitations

    The analysis assumes specific architectural settings, including standard wide networks and, for the constant-kernel result, linear output layers; nonlinear output layers fall outside that regime.

Abstract

from arXiv · show

The goal of this work is to shed light on the remarkable phenomenon of transition to linearity of certain neural networks as their width approaches infinity. We show that the transition to linearity of the model and, equivalently, constancy of the (neural) tangent kernel (NTK) result from the scaling properties of the norm of the Hessian matrix of the network as a function of the network width. We present a general framework for understanding the constancy of the tangent kernel via Hessian scaling applicable to the standard classes of neural networks. Our analysis provides a new perspective on the phenomenon of constant tangent kernel, which is different from the widely accepted "lazy training". Furthermore, we show that the transition to linearity is not a general property of wide neural networks and does not hold when the last layer of the network is non-linear. It is also not necessary for successful optimization by gradient descent.

1 Introduction

The paper explains when wide nonlinear networks become linear in their parameters and their tangent kernels become constant, attributing this to Hessian scaling rather than lazy training. It identifies structural conditions that support or prevent this transition and shows that near-linearity is not required for optimization.

  • Core connection: The tangent kernel is defined as K(x,z)(w) := ∇wf(w; x)T ∇wf(w; z), and its constancy is equivalent to linearity of the model in w.The paper treats the tangent kernel as a property of the model, independent of the optimization algorithm and loss function.
  • Hessian mechanism: A small Hessian spectral norm relative to the gradient makes the model close to linear and gives it a near-constant tangent kernel within a fixed-radius ball.The analysis concerns the Hessian of the model f, not the Hessian of a related loss function.
  • Conditions for constancy: For wide networks with linear output layers, the gradient norm is Θ(1), while the Hessian spectral norm scales as 1/√m, yielding a vanishing Hessian in the infinite-width limit.The resulting near-constant kernel holds in a ball of fixed radius, rather than only near initialization.
  • Scaling explanation: The Hessian and gradient scale differently because increasing width separates the ∞-norm and 2-norm of hidden-layer derivatives while layer-wise (2,1,1)-norms remain comparable.Hessian scaling depends on ∞-norms and layer-wise derivative norms, whereas gradient and tangent-kernel scaling depends on 2-norms.
  • Failure regimes: A nonlinear output layer or a narrow bottleneck generally prevents tangent-kernel constancy, because the Hessian norm does not vanish or becomes large relative to the gradient.These failures can occur even when individual parameters change only slightly.
  • Optimization: Non-constant tangent kernels do not preclude efficient gradient-descent optimization: convergence can follow from uniform kernel conditioning even when the Hessian norm is Ω(1).The paper distinguishes optimization convergence from linearity, which is controlled by the Hessian norm.
  • Against lazy-training explanations: Small individual parameter changes do not explain constant tangent kernels, because wide-network training can have ∥w∗−w0∥∞ = O(1/√m) while ∥w∗−w0∥2 = Θ(1).The second-order Taylor term depends on Euclidean distance and Hessian spectral norm, not only coordinate-wise parameter changes.
  • Interpretation: The paper attributes near-constant tangent kernels to vanishing Hessian norms in suitable wide networks, while rescaled models can satisfy the relevant criterion through a small scaling factor.For neural networks with linear output layers, the paper identifies B ≪ 1 with A = Θ(1) as the relevant mechanism.

2 Notation and Basic Results on Tangent Kernel and Hessian

The section formalizes tangent kernels for scalar, twice-differentiable models and establishes their equivalence to linearity in the parameters. It also shows that a uniformly small Hessian bounds tangent-kernel variation within a fixed-radius ball.

  • Notation: The analysis uses Euclidean and infinity norms for vectors, spectral and Frobenius norms for matrices, and a defined (2,2,1)-norm for order-3 tensors.These norms provide the notation used in the subsequent Hessian and derivative analysis.
  • Setup: The paper assumes a twice-differentiable scalar model f(w; x), with w as trainable parameters and x as input.For input points {x_i}, the tangent kernel becomes an n × n matrix with entries K_ij(w).
  • Basic equivalence: The tangent kernel of a differentiable function is constant if and only if the function is linear in its parameters w.This proposition identifies constant tangent kernels with the representation of the model as linear in w.
  • Hessian control: If the Hessian spectral norm is uniformly less than ϵ within a radius-R ball, tangent-kernel changes are bounded by O(ϵR).The bound applies for every w in the ball and for all input pairs x and z.
  • Implication: Previously established wide-network results with near-constant tangent kernels correspond to models whose Hessian spectral norms are small, vanishing in the infinite-width limit.The statement connects observed kernel constancy to Hessian scaling rather than merely to training dynamics.

3 Transition to linearity: non-linear neural networks with linear output layer

For wide networks with a linear output layer, the Hessian spectral norm decreases with width while gradient and tangent-kernel norms remain order one, causing models to become linear with constant tangent kernels.

  • Scope: The analysis covers fully connected networks and extends to CNNs, ResNets, and mixed architectures with linear output layers.The general framework uses the minimum hidden-layer width and applies within a finite-radius parameter ball.
  • Mechanism: The Hessian spectral norm is controlled by infinity norms of layer sensitivities, whereas the tangent kernel and gradient are controlled by their Euclidean norms.This norm disparity is identified as the underlying reason for the transition to linearity as width increases.
  • General theorem: The framework derives its bounds through a theorem controlling Hessian norms for general networks and applies them to fully connected, convolutional, residual, and mixed architectures.The theorem is stated for W within B(W0,R), with high probability over initialization.
  • Infinite-width limit: As m →∞, the Hessian norm converges to zero in the parameter ball, which implies tangent-kernel constancy and model linearity there.The same conclusion is stated for the multilayer setting after applying the Hessian bounds.

4 Constant tangent kernel is not a general property of wide networks

Wide networks need not develop constant tangent kernels: non-linear output layers and narrow bottlenecks preserve nonlinearity as width grows. Hessian spectral-norm scaling explains these failures of transition to linearity.

  • Non-linear output layers generally do not have constant tangent kernels, and a linear output layer is necessary for transition to linearity.The only structural difference is applying a non-linear activation to the output of a sufficiently wide network.
  • The non-linear-output model’s tangent kernel changes by Θ(1) during training even as width tends to infinity.Its Hessian retains an Ω(1) contribution from the output nonlinearity, whereas the underlying model’s Hessian is O(1/√m).
  • A narrow bottleneck breaks the O(1/√m) Hessian scaling that supports transition to linearity in fully wide networks.For a bottleneck width mb = 1, the relevant quantity grows as O(m), so the Hessian spectral norm is not arbitrarily small.
  • For the bottleneck network, the Hessian is lower-bounded by a positive constant around initialization, implying nonlinearity and a non-constant tangent kernel along optimization.The theorem holds within any fixed-radius ball with high probability, and the parameter displacement is Ω(1).
  • The analysis compares linear-output, non-linear-output, and bottleneck models through their Hessian spectral-norm scaling and whether transition to linearity occurs.In this framework, transition to linearity is equivalent to a constant tangent kernel in the infinite-width limit.

5 Optimization of wide neural networks

Near-linearity and successful gradient-descent optimization are distinct properties. A non-linear-output network can lack a constant tangent kernel while still converging rapidly under suitable conditioning assumptions.

  • Near-constant tangent kernels are not necessary for successful optimization; a well-conditioned tangent kernel along the optimization path is sufficient and weaker.The paper separates model near-linearity from convergence of gradient descent.
  • A sufficiently wide non-linear-output network can lack a constant tangent kernel while gradient descent still converges rapidly.This example directly contrasts non-constant kernel behavior with optimization success.
  • If |φ′(z)| ≥ ρ > 0 for all z and width is sufficiently large, gradient descent with a small enough step size converges to a global square-loss minimizer with high probability.
  • The convergence argument uses preservation of tangent-kernel conditioning under the non-linear transformation when φ′ remains separated from zero.The original model’s non-degenerate, well-conditioned tangent kernel transfers to the transformed model.

6 Numerical Verification

Experiments verify that linear output layers approach constant tangent kernels with width, whereas non-linear outputs and narrow bottlenecks retain measurable kernel changes. Alternative parameterizations preserve the paper’s conclusions.

  • Wide neural networks with non-linear output layers: As width increases, linear-output networks’ tangent kernels become nearly constant, while softmax- and swish-output networks retain significant changes.The comparison uses widths from 30 through 10^6 and measures relative kernel change from initialization to convergence.
  • Wide neural networks with non-linear output layers: At width m = 10^4, the non-linear-output kernel changes significantly over epochs, whereas the linear-output kernel is nearly unchanged.
  • Wide neural networks with a bottleneck: Increasing bottleneck width reduces tangent-kernel change, with the observed scaling close to Θ(1/√m).A narrow bottleneck prevents constancy even when the other hidden layers have width 10^4.
  • Different parameterizations: Different parameterizations rescale the tangent kernel and Hessian spectral norm by the same factor, preserving their ratio and the near-constant-kernel conclusion.The paper explicitly states this for alternatives such as LeCun initialization.
  • Experimental setup: The experiments train the networks by gradient descent to convergence and estimate maximum relative kernel change across ten independent runs.The metric is ΔK := sup_t>0 ||K(w_t)−K(w_0)||_F / ||K(w_0)||_F.

C Proof for Proposition 2.2

The proof establishes that a model has a constant tangent kernel exactly when it is linear in its parameters. It derives linearity from constant gradient norm on input-wise level sets.

  • The argument proves the equivalence input by input, using the tangent-kernel definition K_ij(w) = ∇f(w; x_i)^T∇f(w; x_j).
  • Linearity in the parameters implies a constant tangent kernel because the parameter gradient is constant for each input.
  • For a constant tangent kernel, each diagonal entry fixes the parameter-gradient norm for every input.The identity K_ii(w) = ||∇_w f(w, x_i)||^2 makes the gradient norm constant in w.
  • The proof analyzes level sets of f and shows their tangent spaces are orthogonal to the constant-norm gradient direction.This geometric characterization leads to parallel level sets.
  • A one-dimensional restriction g(t) has constant derivative, which establishes that f is linear in the parameter vector.

D Proof of Proposition 2.3

Taylor expansion bounds how gradient changes within a convex parameter ball, which in turn bounds variation of the tangent kernel. The kernel difference is O(ϵR) under smoothness and bounded-gradient conditions.

  • Gradient variation: Taylor’s theorem expresses the gradient change through the Hessian along the segment between w0 and w.The segment remains inside B(w0, R) because the ball is convex.
  • Gradient variation: The Euclidean norm of the gradient change is bounded within the parameter ball.
  • Kernel constancy: For any inputs x and z, the tangent-kernel difference is bounded by products of gradient changes and gradient norms.
  • Kernel constancy: O(ϵR) bounds |K(x,z)(w) − K(x,z)(w0)| when the network is smooth and its gradients are bounded.

E Proof of Theorem 3.1

The proof decomposes the network Hessian into layer-wise blocks and bounds its spectral norm by the norms of those blocks. The final-layer Hessian block is zero, and the stated result follows from the block bounds and Lemma E.1.

  • Hessian decomposition: The Hessian is partitioned into blocks H(l1,l2), each containing second derivatives with respect to two layer parameter groups.The final-layer parameters v are treated as w(L+1).
  • Block analysis: Lemma E.1 bounds the Hessian spectral norm by the sum of the spectral norms of its blocks.
  • Block analysis: By symmetry, the proof analyzes only blocks satisfying 1 ≤ l1 ≤ l2 ≤ L + 1.
  • Block analysis: For hidden-layer blocks, the proof applies chain-rule expressions and layer-wise bounds to control their spectral norms.
  • Final-layer block: The final-layer block H(L+1,L+1) is zero, so its spectral norm is zero; applying Lemma E.1 yields the desired result.

F Proof for Lemma 3.1

The fully connected-network proof establishes uniform layer-wise bounds inside a finite ball around Gaussian initialization. These bounds control activations, derivatives, and tensor norms needed for Hessian scaling.

  • Proof strategy: The analysis reduces the target quantities to layer-wise properties that must hold throughout B(W0, R) with high probability.
  • Network and initialization: The network uses width m in hidden layers, Gaussian initialization, and parameters grouped as W(1), …, W(L), and v.
  • Assumptions: Initial weight spectral norms are controlled with high probability, while bounded inputs and a finite parameter ball provide the main assumptions.
  • Derivative bounds: O(1) bounds hold for layer derivatives ∂α(l)/∂w(l) and ∂α(l)/∂α(l−1), establishing layer-wise Lipschitz control.
  • Layer-wise estimates: The proof separately handles the first layer and later layers, using bounded activations, Lipschitz continuity, and tensor-norm estimates.
  • Uniform control: At initialization, relevant backpropagated components and hidden-layer quantities have the stated high-probability order bounds, which extend across the parameter ball.

G Generalization to other architectures

The Hessian-scaling analysis extends beyond fully connected networks to CNNs and ResNets. Both architectures have Hessian spectral norm of order ˜O(1/√m) when the final layer is linear.

  • Scope: The analysis applies to CNNs and ResNets and establishes small Hessian spectral norms for sufficiently large width with a linear last layer.
  • CNNs: For CNNs, convolutional layers are represented using channel, pixel, and filter indices and can be reformulated into fully-connected-like matrix operations.
  • CNNs: CNN bounds control weight and hidden-layer norms, layer derivatives, and third-order tensor (2,2,1)-norms using analogous fully connected arguments.
  • CNNs: CNN backpropagated components satisfy ˜O(1/√m) bounds with high probability when the number of filter positions K is finite.
  • ResNets: For ResNets, skip connections add the previous activation to each layer, so the proof closely parallels the fully connected case.
  • ResNets: Bounded layer derivatives make the ResNet layer function Lipschitz continuous, supporting the same Hessian-scaling analysis.
  • Common scaling: Fully connected networks, CNNs, and ResNets all have Hessian spectral norm of order ˜O(1/√m).

H Proof of Theorem 4.1

The proof establishes bounds on Hessian blocks and related layerwise quantities for parameters near initialization. It combines Gaussian concentration, induction, and norm estimates to obtain probabilistic bounds used in Theorem 4.1.

  • The proof considers an arbitrary parameter setting W within the radius-R ball around initialization W0.
  • The Hessian spectral norm is lower bounded by the norm of a Hessian block, reducing the analysis to a specific block.
  • Gaussian concentration and norm estimates are applied to gradients, Hessian entries, weights, and layerwise quantities throughout the proof.
  • The resulting bounds include high-probability estimates for layerwise terms and Hessian-related expressions involving width m and radius R.
  • Inductive arguments propagate bounds across layers using Lipschitz continuity of the activation and concentration inequalities.

I.9 Proof of Lemma H.1

Lemma H.1 bounds a random-variable expression for vectors close to Gaussian reference vectors. The proof combines anti-concentration for the leading term with concentration and norm bounds for the remaining terms.

  • The lemma assumes independent standard Gaussian coordinates and vectors x̄ and ȳ within radius R of x and y.
  • The final term is bounded directly from the assumptions ∥s∥≤R and ∥t∥≤R.
  • The proof lower bounds the first term using an anti-concentration inequality with failure probability controlled by δ1.
  • The second and third terms are controlled using Gaussian concentration and a union bound with probability parameter δ2.
  • Combining the bounds yields probability estimates at least 1−δ1−δ2 and, when δ1=δ2, at least 1−2δ1.
Loading 2010.01092v3…