Source-linked AI summary
Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
Jiaoyang Huang, Horng-Tzer Yau
TL;DR
The paper addresses the gap between limiting-NTK kernel regression and finite-width deep-network behavior by studying NTK dynamics during training. It derives the data-dependent neural tangent hierarchy and shows that a finite truncation can approximate those dynamics under the paper’s assumptions. This provides a framework for directly studying NTK change in deep fully-connected networks.
Problem
A performance gap exists between kernel regression using the limiting NTK and deep neural networks, motivating analysis of finite-width NTK changes during training.
Method
The paper derives an infinite data-dependent hierarchy of ordinary differential equations, the neural tangent hierarchy, and constructs a finite truncated hierarchy.
Results
The truncated hierarchy approximates NTK dynamics up to arbitrary precision under suitable conditions on network width and dataset dimension.
Takeaways & Limitations
The hierarchy provides a direct description of how the NTK changes during training in deep fully-connected neural networks.
Takeaways & Limitations
The main analysis focuses on deep fully-connected networks and continuous-time gradient descent.
Abstract
from arXiv · showhide
The evolution of a deep neural network trained by the gradient descent can be described by its neural tangent kernel (NTK) as introduced in [20], where it was proven that in the infinite width limit the NTK converges to an explicit limiting kernel and it stays constant during training. The NTK was also implicit in some other recent papers [6,13,14]. In the overparametrization regime, a fully-trained deep neural network is indeed equivalent to the kernel regression predictor using the limiting NTK. And the gradient descent achieves zero training loss for a deep overparameterized neural network. However, it was observed in [5] that there is a performance gap between the kernel regression using the limiting NTK and the deep neural networks. This performance gap is likely to originate from the change of the NTK along training due to the finite width effect. The change of the NTK along the training is central to describe the generalization features of deep neural networks. In the current paper, we study the dynamic of the NTK for finite width deep fully-connected neural networks. We derive an infinite hierarchy of ordinary differential equations, the neural tangent hierarchy (NTH) which captures the gradient descent dynamic of the deep neural network. Moreover, under certain conditions on the neural network width and the data set dimension, we prove that the truncated hierarchy of NTH approximates the dynamic of the NTK up to arbitrary precision. This description makes it possible to directly study the change of the NTK for deep neural networks, and sheds light on the observation that deep neural networks outperform kernel regressions using the corresponding limiting NTK.
1 Introduction
Deep-network training is difficult to describe because of large parameter counts and non-convex optimization, while limiting-NTK theory simplifies dynamics in the infinite-width regime. This paper addresses the finite-width gap by introducing the data-dependent neural tangent hierarchy and a truncated approximation.
- Motivation: Deep-network training involves many parameters and highly non-convex optimization, motivating a succinct description of its dynamics.The paper frames this as a central challenge for gradient-based training.
- Limiting NTK: In the infinite-width limit, the NTK remains constant and the network dynamics become equivalent to kernel regression using the limiting NTK.For sufficiently wide networks, this linearized description can approximate training dynamics and yield zero training loss under stated assumptions.
- Motivation: A 5% - 6% performance gap separates convolutional neural networks from corresponding kernel regressions using the limiting NTK.The passage attributes this gap likely to finite-width changes in the NTK during training.
- Contribution: The neural tangent hierarchy (NTH) captures gradient-descent dynamics through an infinite hierarchy of ordinary differential equations and learns data-dependent features.Unlike the limiting NTK, the NTH depends on the dataset as well as the network dynamics.
- Contribution: A priori estimates show that the NTK varies at rate O(1/m), and networks with m ≳n3 converge to zero training loss at a linear rate.The stated width requirement improves an earlier m ≳n4 result.
- Contribution: A truncated NTH approximates NTK dynamics up to arbitrary precision for a suitable truncation, enabling direct study of finite-width changes during training.Without truncation, the infinite hierarchy cannot determine the NTK dynamics directly.
2 Main results
The paper develops the neural tangent hierarchy (NTH) to describe finite-width training dynamics and proves that a truncated version approximates NTK evolution under stated assumptions. It also establishes an O(1/m) NTK variation rate and improved width conditions for linear convergence to zero training loss.
- Assumptions: The analysis assumes smooth activations with bounded derivatives and normalized, sufficiently nondegenerate training inputs.The input assumptions require bounded norms and lower-bounded singular values for small subsets of distinct samples.
- Neural tangent hierarchy: The NTH represents continuous-time gradient descent as an infinite hierarchy of ordinary differential equations for higher-order kernels.The hierarchy is data dependent, unlike the limiting NTK, and can capture data-dependent features.
- NTK dynamics: O(1/m) is the NTK variation rate under the paper’s assumptions, confirming the rate suggested by numerical experiments.Earlier theory provided an O(1/√m) upper bound; the paper’s corollary sharpens this behavior to order O(1/m).
- Training convergence: m ≳ n^3 suffices for linear-rate convergence of gradient descent to zero training loss under the stated assumptions.This improves an earlier quartic width dependence on n to a cubic dependence.
- Truncated hierarchy: The truncated NTH replaces the infinite hierarchy with p levels and approximates the original NTK dynamics to arbitrary precision when p is sufficiently large.The approximation holds subject to the theorem’s time constraint, with higher-order-kernel estimates controlling the truncation error.
- Prediction: The truncated dynamics can estimate predictions on new data points by sequentially solving the approximated outputs and higher-order kernels.The method uses the training-set dynamics to construct an estimate for a new test point.
3 Technique overview
The paper constructs higher-order kernels recursively and differentiates them using the gradient-flow equations. Tensor-program arguments then control their initialization and evolution, supporting a hierarchy-based description of NTK dynamics.
- Dynamic equations: The dynamics of K(r) are derived by applying the chain rule to the parameter and weight evolution equations.Replacing the relevant parameter or weight factors generates the terms appearing in the derivative hierarchy.
- Kernel construction: Higher-order kernels K(r) are constructed recursively from inner products of vectors transformed by network matrices, activations, and diagonalization operations.The vector-expression classes D0, D1, …, Dr track the number of diagonalization operations.
- Initialization: At initialization, tensor programs express the relevant vectors through projections of independent Gaussian vectors, whose normalized inner products concentrate with high probability.This concentration yields the initial estimates needed for the hierarchy analysis.
- A priori control: A self-consistent differential equation for ξ(t), together with its initial estimate, controls the relevant vector norms over the analyzed time range.The resulting bound is stated as ∥v_j(t)∥∞ ≲ (ln m)^C with high probability.
- Consequences: A priori estimates imply that the higher-order kernels K(r)_t vary slowly along continuous-time gradient descent.The paper uses a Grönwall-type argument to derive its corollaries and theorem from these estimates.
4 Discussion and future directions
The paper concludes that the NTK dynamics can be represented by an infinite neural tangent hierarchy and approximated by a finite truncation. It identifies extensions to other architectures and discrete-time optimization as future directions.
- Conclusions: The training dynamics of deep fully-connected networks are represented by a data-dependent infinite hierarchy of ordinary differential equations, the NTH.The hierarchy describes the continuous-time gradient-flow setting studied in the paper.
- Conclusions: The NTK dynamics can be approximated to arbitrary precision by a finite truncated hierarchy under the paper’s stated conditions.The truncation provides a direct way to study changes in the NTK during training.
- Future directions: The analysis mainly covers deep fully-connected neural networks, while analogous statements for convolutional and residual networks are left as future work.The authors state that they believe such extensions can be proven.
- Future directions: The approach is developed for continuous-time gradient descent, but the authors outline its extension to discrete-time gradient descent.The discrete-time analysis uses a Taylor expansion of the NTK across training steps and higher-order kernels.
A Initial Estimates
The initial-estimate analysis characterizes the kernel expressions at initialization through recursively structured vector classes and Gaussian representations. Concentration and conditioning arguments establish deterministic limiting behavior and high-probability bounds.
- Expression structure: The kernel K(r)_t is decomposed into sums of products of inner products involving recursively generated vectors from the classes D0 through Dr.Each vector class records a prescribed number of diagonalization operations.
- Structural induction: The recursive replacement rules preserve the stated structure of the vector classes, while the time parameter in Proposition A.1 does not itself introduce dynamics.This structural proposition supports the later initialization estimates.
- Concentration: Normalized inner products of the initial Gaussian-based vectors concentrate around deterministic limits with high probability as the width m tends to infinity.The same concentration argument applies to products of such normalized inner products.
- Gaussian representation: Conditioning Gaussian weight matrices expresses network evaluations as combinations of independent standard Gaussian vectors with coefficients determined by Gram-Schmidt procedures.For the input layer, the coefficients depend on input inner products; later-layer coefficients may be random.
- Nondegeneracy: The limiting Gram matrix of the first hidden-layer representations is full rank, ensuring diagonal Gram-Schmidt coefficients remain of order one with high probability.The argument is repeated across layers to obtain new independent Gaussian components.
B A Priori Estimates
The a priori-estimate analysis bounds the network expressions and their derivatives over time under the stated activation and data assumptions. These bounds support control of the hierarchy terms with high probability.
- L2 bounds: Under the stated assumptions, the analysis establishes a priori L2 bounds for the recursively defined expressions.The proof bounds products of operator norms and diagonalized-vector norms by induction on the number of diagonalization operations.
- High-probability estimates: The estimates yield bounds on the relevant expressions with high probability over the random initialization.The argument uses the smoothness and bounded-derivative assumptions on the activation function.
- Inductive control: The proof controls higher-complexity expressions by induction, separating cases according to how many diagonalization operations each factor contains.At most one factor can exceed the threshold r+1 under the stated total-count constraint.
- Proof caveat: The norm inequalities used in the proof are not optimal and incur a factor involving √m.This factor is explicitly identified in the discussion of the norm bound.
- Time evolution: A self-consistent differential equation for ξ(t) controls higher derivatives and extends the initial estimates across the analyzed time interval.The resulting control includes logarithmic dependence on the width through bounds such as ξ(t) ≲ (ln m)^C.
C Proof of Corollary 2.4 and 2.5, and Theorem 2.6
The appendix derives bounds on NTK evolution and uses them to establish training-error decay and control the difference between the original and truncated dynamics over a finite time interval.
- Corollary 2.4: The proof bounds the NTK entrywise change and then derives corresponding L∞ and norm bounds.These estimates are used to control the NTK's smallest-eigenvalue change.
- Corollary 2.4: The NTK norm remains sufficiently close to its initial value on a width- and dimension-dependent time interval.The proof combines the norm-change estimate with the initial spectral bound.
- Corollary 2.5: The resulting lower bound implies exponential decay of the training error during the controlled interval.The proof further gives a logarithmic time scale for reaching error ε.
- Corollary 2.5: Under the stated conditions, reaching training error ε takes time proportional to (n/λ) ln(n/ε), while the argument requires a sufficiently wide network.The passage distinguishes this time scale from the bound involving 2n/λ and states a width requirement for the estimate.
- Theorem 2.6: The discrepancy remains below the stopping threshold throughout the stated interval, yielding the theorem's approximation guarantee.The stopping-time argument establishes a lower bound on T and closes the proof.