The Principles of Deep Learning Theory

Daniel A. Roberts; Sho Yaida; Boris Hanin

Source-linked AI summary

The Principles of Deep Learning Theory

Daniel A. Roberts, Sho Yaida, Boris Hanin

arXiv:2106.10165v2Jun 18, 2021cs.LGcs.AIhep-thstat.ML

TL;DR

The formal infinite-width limit mismatches practical observations of deep networks and does not capture representation learning. This book derives recursive equations for NTK statistics and reports that zero-mean deterministic weight initialization leads to exponential decay toward a trivial fixed point.

Problem
The formal infinite-width limit poorly models deep networks, mismatching practical observations and leaving hidden representations unchanged from initialization.
Source: A Principle of Sparsity · p. 16
Method
The book derives a layer-to-layer forward equation for NTK statistics from the MLP iteration equation for preactivations.
Source: RG Flow of the Neural Tangent Kernel · p. 208
Results
Zero-mean deterministic weight initializations cannot satisfy both criticality conditions and produce exponential decay toward a trivial fixed point.
Source: Summary · p. 130
Takeaways & Limitations
The framework is presented as a way to analyze both deep representation-group flow and finite-width nonlinear representation learning.
Source: Deep Learning: A Non-Minimal Model of… · p. 341

Abstract

from arXiv · show

This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks, we explain how to determine an accurate description of the output of trained networks by solving layer-to-layer iteration equations and nonlinear learning dynamics. A main result is that the predictions of networks are described by nearly-Gaussian distributions, with the depth-to-width aspect ratio of the network controlling the deviations from the infinite-width Gaussian description. We explain how these effectively-deep networks learn nontrivial representations from training and more broadly analyze the mechanism of representation learning for nonlinear models. From a nearly-kernel-methods perspective, we find that the dependence of such models' predictions on the underlying learning algorithm can be expressed in a simple and universal way. To obtain these results, we develop the notion of representation group flow (RG flow) to characterize the propagation of signals through the network. By tuning networks to criticality, we give a practical solution to the exploding and vanishing gradient problem. We further explain how RG flow leads to near-universal behavior and lets us categorize networks built from different activation functions into universality classes. Altogether, we show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks. By using information-theoretic techniques, we estimate the optimal aspect ratio at which we expect the network to be practically most useful and show how residual connections can be used to push this scale to arbitrary depths. With these tools, we can learn in detail about the inductive bias of architectures, hyperparameters, and optimizers.

7 Gradient-Based Learning

Section 7 is titled “Gradient-Based Learning” and lists subsections on supervised learning and gradient descent and function approximation.

7 Gradient-Based Learning

The section is titled “Gradient-Based Learning.”Source: §7 · p. 4
The listed topics are supervised learning and gradient descent and function approximation.Source: §7 · p. 4

8 RG Flow of the Neural Tangent Kernel

Section 8 develops the NTK forward equation and follows its layer-wise behavior from a deterministic first-layer NTK to second-layer fluctuations and their accumulation in deeper layers.

8 RG Flow of the Neural Tangent Kernel

The section traces NTK behavior from a forward equation through a deterministic first layer, a fluctuating second layer, and fluctuation accumulation in deeper layers.These topics appear in subsections 8.0–8.3.Source: §8 · p. 4

8.3 Deeper Layers: Accumulation of NTK Fluctuations

For deeper layers, the section examines interlayer correlations, the NTK mean, NTK–preactivation cross-correlations, and NTK variance.These topics are listed in subsections 8.3.0–8.3.3.Source: §8 · p. 4

8.1 First Layer: Deterministic NTK

The first-layer subsection treats the NTK as deterministic, while the second-layer subsection treats it as fluctuating.Source: §8 · p. 4

9 Effective Theory of the NTK at Initialization

This section presents an effective theory of the NTK at initialization, organized around criticality analysis, two universality classes, and criticality’s relation to exploding and vanishing problems.

9.1 Criticality Analysis of the NTK

The section includes a criticality analysis of the NTK.Source: §9 · p. 4

9.2 Scale-Invariant Universality Class

The section examines the scale-invariant universality class.Source: §9 · p. 4

9.3 K⋆= 0 Universality Class

The section examines the K⋆= 0 universality class.Source: §9 · p. 4

9.4 Criticality, Exploding and Vanishing Problems, and None of That

The section addresses criticality alongside exploding and vanishing problems.Source: §9 · p. 4

11 Representation Learning

Section 11 covers the differential neural tangent kernel (dNTK), its RG flow and effective theory at initialization, and connections between nonlinear models and nearly-kernel methods.

11.2 RG Flow of the dNTK

The section examines the differential of the neural tangent kernel and its RG flow, from a zero first-layer dNTK to a growing dNTK in deeper layers.Its listed topics include the forward equation and the transition from a nonzero second-layer dNTK to growth in deeper layers.Source: §11 · p. 5

11.3 Effective Theory of the dNTK at Initialization

The effective theory of the dNTK at initialization is organized around scale-invariant and K⋆=0 universality classes.Source: §11 · p. 5

11.4 Nonlinear Models and Nearly-Kernel Methods

The section treats nonlinear models, nearly-kernel methods, and finite-width networks as nonlinear models.Source: §11 · p. 5

11.1 Differential of the Neural Tangent Kernel

The section begins by examining the differential of the neural tangent kernel.Source: §11 · p. 5

335 ∞.1 Two More Differentials

The supplied passages list later topics in the book: finite-width training and RG-flow expressions, information-theoretic analyses of model complexity, and residual-network analyses.

335 ∞.1 Two More Differentials

The contents list sections on finite-width training and prediction, followed by full expressions for RG flow of the ddNTKs.The listed subsections cover a small step after a giant leap, many gradient-descent steps, and finite-width prediction.Source: 335 ∞.1 Two More Differentials · p. 5

Epilogue: Model Complexity from the Macroscopic Perspective

The epilogue contents list entropy and mutual information, information at infinite width and criticality, and finite-width optimal aspect ratio.Source: A Information in Deep Learning · p. 6

A Information in Deep Learning

The listed residual-network topics include residual multilayer perceptrons, criticality at infinite width, optimal aspect ratio at finite width, and residual building blocks.Source: B Residual Learning · p. 6

Initialization

The book aims to close the gap between deep learning practice and theory by developing principles for analyzing relevant deep networks, using their known microscopic computation to explain macroscopic behavior.

Initialization

Deep learning theory has lagged practice, especially in addressing depth, motivating principles that analyze deep neural networks of practical relevance.The book’s introductory chapter previews why such an analysis is attainable and how to carry it out.Source: Initialization · pp. 9–10

0.1 An Effective Theory Approach

Because networks contain many elementary components with known computational rules, the theory can seek macroscopic regularities emerging from their collective statistics.The book draws on effective-theory reasoning, where many-component systems can admit simpler macroscopic descriptions.Source: §0.1 · pp. 10–11

0.2 The Theoretical Minimum

Networks are randomly initialized and then trained by adjusting parameters to fit observed training pairs to a partially observed target function.The particular procedure for tuning parameters is the learning algorithm.Source: §0.2 · p. 13

Problem 1

A Taylor expansion around initialization has infinitely many terms, and larger changes from initial to trained parameters require more terms for accurate approximation.Using the expansion may therefore require computing an unbounded series.Source: Problem 1 · p. 14

Problem 2

Random initialization induces a joint distribution over the network function and its derivatives, whose statistical dependence is intricate and whose function marginal may be analytically intractable.The distribution must map initial parameters to the function, gradient, Hessian, and higher derivatives.Source: Problem 2 · p. 14

Problem 3

The trained parameters depend nonlinearly on initialization, the learning algorithm, and training data, making their analytical determination the third central problem.Training iteratively accumulates changes, and the resulting parameters can depend on the initial function and all its derivatives.Source: Problem 3 · p. 14
The book seeks an analytical method for computing distributions over trained network functions conditioned simply on the learning algorithm and training data.The method should make it easy to evaluate the distribution for different algorithms or training data without re-solving the training-dependence problem each time.Source: Problem 3 · p. 15

A Principle of Sparsity … Second-layer conditional distribution

The section develops a large-width effective theory for finite-width neural networks, using Gaussian moments, cumulants, connected correlators, and perturbative descriptions to characterize deviations from Gaussianity. It applies this framework to network architectures and activation functions, tracing layer-by-layer distributions and how initialization, depth, and width shape their statistics.

A Principle of Sparsity

At infinite width, higher derivatives of f with respect to θ vanish, random functions become independent, training dynamics become linear and algorithm-independent, and the trained distribution is Gaussian with nonzero mean.These simplifications make the three problems tractable, but the resulting description does not capture important properties of practical multilayer networks.Source: A Principle of Sparsity · p. 16
Strict infinite width mismatches practical multilayer networks because learning-algorithm effects and nontrivial representation learning are absent.Hidden-layer representations remain at their random initial values, limiting the functions these networks can learn.Source: A Principle of Sparsity · pp. 16–17
Finite-width corrections at order 1/n yield a nearly-Gaussian trained distribution that depends on the learning algorithm and includes nontrivial representation learning.The 1/n expansion restores interactions between neurons while retaining an analytically tractable description.Source: A Principle of Sparsity · pp. 17–18
The depth-to-width ratio r=L/n governs the correction scale: small positive r supports an accurate finite-width theory, while r≫1 brings strong coupling and chaotic behavior.At r→0, interactions turn off and the infinite-width description applies; residual connections can shift the optimal ratio r⋆ upward.Source: A Principle of Sparsity · p. 18

Pretraining

The pretraining chapter introduces Gaussian integration, perturbation theory, and statistics concepts needed to analyze nearly-Gaussian network distributions.Connected correlators characterize deviations from Gaussianity, and perturbation theory relates small non-Gaussian couplings to observables.Source: Pretraining · pp. 21–22

1.1 Gaussian Integrals

Wick’s theorem provides an operational formula for computing any moment of a multivariable Gaussian distribution.The section introduces Gaussian integrals and distributions to derive the theorem for use throughout the book.Source: §1.1 · p. 22

Single-variable Gaussian integrals

For a single-variable Gaussian, odd moments vanish and even moments follow a simple formula identified as Wick’s theorem.A source-term partition function provides another derivation that extends naturally to multivariable Gaussians.Source: Single-variable Gaussian integrals · pp. 24–25
The second derivation completes Wick’s theorem for the single-variable Gaussian distribution.The text notes that this derivation is longer than the first and extends naturally to multivariable Gaussians.Source: Single-variable Gaussian integrals · p. 25

Multivariable Gaussian integrals

For multivariable Gaussians, odd moments vanish and each even 2m-point moment is a sum over (2m −1)!! distinct pairings of covariance factors.Each covariance factor is a Wick contraction; the resulting pairing formula is Wick’s theorem.Source: Multivariable Gaussian integrals · pp. 30–31

1.2 Probability, Correlation and Statistics, and All That

All correlators can characterize a distribution, but measuring high-order correlators quickly becomes impractical; a zero-mean Gaussian is specified by its covariance, which determines its higher correlators.The covariance has N(N +1)/2 independent components, and Gaussian higher-point correlators are fixed by Wick’s theorem.Source: §1.2 · pp. 32–33
For nearly-Gaussian distributions, cumulants (connected correlators) offer useful observables; under sign-flip symmetry odd moments vanish, while a nonzero connected four-point correlator signals non-Gaussianity.The connected four-point correlator vanishes for a Gaussian distribution and is described as a simple measure of non-Gaussianity.Source: §1.2 · pp. 33–34
Connected correlators of order M > 2 vanish for zero-mean Gaussians, so they measure departures from Gaussianity; in wide networks, higher-order correlators become progressively smaller.A few leading connected correlators can therefore concisely describe a nearly-Gaussian distribution.Source: §1.2 · p. 36

1.3 Nearly-Gaussian Distributions

A nearly-Gaussian distribution can be represented by an action that perturbs the quadratic Gaussian action, with its probability density proportional to the exponential of minus the action.Given a distribution, its action is −log p(z) up to an additive constant.Source: §1.3 · pp. 36–37

Quadratic action and the Gaussian distribution

A quadratic action describes the Gaussian distribution and serves as the starting point for other distributions; Gaussian expectations provide a primitive for evaluating nearly-Gaussian expectations.Wick’s theorem can be written using the Gaussian-expectation notation.Source: Quadratic action and the Gaussian… · p. 37

Quartic action and perturbation theory

The quartic coupling directly controls the connected four-point correlator and can also shift the covariance from its Gaussian value.For networks with a large but finite layer width n, the nearly-Gaussian description can be expanded in 1/n.Source: Quartic action and perturbation theory · p. 41

Aside: statistical independence and interactions

Nonzero non-Gaussian couplings generically prevent an action from separating into independent terms, so the joint distribution cannot factorize into independent variables.A Gaussian with diagonal covariance has independent components, and a correlated Gaussian can be diagonalized by rotating basis.Source: Aside: statistical independence and… · p. 42

Nearly-Gaussian actions

With parity symmetry, non-Gaussian actions contain even-degree terms whose small couplings control the degree and pattern of connected correlations.The action can be truncated to a reasonably small degree when that provides a good representation of the statistical process.Source: Nearly-Gaussian actions · pp. 42–43
Hierarchically small connected correlators become less important at higher orders as ϵ shrinks, making a quartic action a tractable model of salient finite-width non-Gaussian effects.The quartic action captures nontrivial component interactions; higher-order correlation differences are suppressed as O(ϵ^2).Source: Nearly-Gaussian actions · p. 44

Neural Networks

The book uses MLPs as an archetype for deep learning theory, shifting the analysis from random weights and biases to induced distributions over neural activities and outputs.MLPs are presented as rich enough to capture essential aspects of deep learning while keeping the analysis pedagogically focused.Source: Neural Networks · p. 45

2.1 Function Approximation

Function approximation tunes parameters in flexible functions, while an MLP builds such functions by iterating layers of weighted, biased preactivations and componentwise activations.Deep learning emphasizes stacking many structurally similar layers; hidden-layer activations can encode information useful to the approximation.Source: §2.1 · pp. 46–47
Architectural inductive biases can encode data structure: convolutions exploit spatial locality and translational invariance, while transformers encourage correlations across sequence elements.When these biases are well founded, convolution curtails the number of weights to train and store; the theory formalism can also be specialized by changing the layer iteration equation.Source: §2.1 · pp. 50–51
The effective-theory approach is presented as applicable across architectures built from iterated layers, with extensions outlined for residual networks, residual convolutional networks, and transformers.Residual connections are described as enabling training of deeper networks.Source: §2.1 · p. 51

2.2 Activation Functions

The activation-function overview is non-exhaustive; a later section uses the effective theory to evaluate how well these functions pass input signals through deep networks.Source: §2.2 · p. 51

Perceptron

The perceptron’s on/off simplicity is unsuitable for deep networks, which benefit from propagating more than one bit of preactivation information.The text says the perceptron is never used in deep neural networks.Source: Perceptron · pp. 51, 53

Sigmoid and tanh

The sigmoid preserves preactivation magnitude mainly near zero but saturates and fails to pass through the origin, while tanh is shifted so σ(0) = 0.The sigmoid’s differentiability was essential to developing backpropagation for training networks with hidden layers.Source: §2 · p. 53, Tanh · p. 53

Scale-invariant: linear, ReLU, and leaky ReLU

Scale-invariant activations are piecewise linear with separate slopes on either side of zero; ReLU and leaky ReLU introduce nonlinearities through a kink at the origin.The class includes linear, ReLU, and leaky ReLU; stacking nonlinear activations lets deep networks express highly nonlinear functions.Source: Scale-invariant: linear, ReLU, and… · p. 54

ReLU-like: softplus, SWISH, and GELU

Smoothing ReLU with softplus, SWISH, or GELU introduces an intrinsic scale and violates scale invariance; unlike softplus, SWISH and GELU cross the origin.Softplus has σ(0) = log(2), whereas SWISH has σ(0) = 0; the text also says GELU crosses the origin.Source: ReLU-like: softplus, SWISH, and GELU · p. 55

2.3 Ensembles

Zero initialization fails to break permutation symmetry among hidden-layer neurons, making the network behave as if each layer had only one neuron; independent random draws can break that symmetry.The initialization distribution should be chosen so the resulting network ensemble is well behaved for the function-approximation task.Source: §2.3 · p. 56

Initialization distribution of biases and weights

The authors initialize each bias and weight independently from zero-mean Gaussian distributions, using weight-variance normalization by 1/nℓ−1 and potentially layer-specific bias variances.The corresponding variances are initialization hyperparameters, for which the effective-theory approach aims to provide prescriptions.Source: Initialization distribution of biases… · pp. 56–57

Induced distributions

Randomly drawing parameters induces a distribution over network outputs, although each output is deterministic conditional on the inputs and parameter values.Computing the induced output distribution requires integrating over model parameters.Source: Induced distributions · pp. 57–58

Induced distributions, redux

Dirac delta functions encode deterministic mappings and express both network-output distributions and layer-to-layer preactivation distributions after integrating out parameters.The general construction also applies to parameterized models with outputs zout_i; the delta function has an integral representation depending only on z − s.Source: Induced distributions, redux · pp. 59–60, Deterministic distributions and the… · p. 59

3.2 Criticality

Initialization hyperparameters govern the two-point correlator, with sensitivity increasing exponentially with depth and requiring fine tuning for well-behaved networks.The two-point recursion is solved exactly in the deep linear network.Source: Effective Theory of Deep Linear… · p. 61

3.3 Fluctuations

Network-function fluctuations across parameter draws are measured by the connected four-point correlator and scale with the depth-to-width ratio, so keeping that ratio sufficiently small controls them.The section simplifies the analysis by considering correlators evaluated on a single input.Source: Effective Theory of Deep Linear… · p. 62, §3.3 · p. 67

Effective Theory of Deep Linear Networks at Initialization

At any width and depth, arbitrary M-point correlator recursions are exactly solvable; the large-width and large-depth limits do not commute, and L/n controls interactions.An interpolating solution at fixed L/n identifies this ratio as the parameter controlling the validity of perturbative analysis.Source: Effective Theory of Deep Linear… · p. 62

3.1 Deep Linear Networks

Deep linear networks generically represent fewer functions than one-layer linear networks and have non-Gaussian preactivation statistics that depend on depth and layer widths.The chapter aims to determine these statistics by computing the network’s nontrivial distribution.Source: §3.1 · p. 63

Math: recursion for the two-point correlator

Each layer multiplies the two-input covariance by C_W, yielding exponential evolution from the input inner product.The resulting covariance describes the inputs after passing through the network.Source: Math: recursion for the two-point… · p. 65

Physics: criticality

When C_W > 1 covariance grows exponentially, and when C_W < 1 it decays, respectively causing divergent outputs or curtailing data dependence; C_W = 1 preserves input covariance exactly.The tuned value is critical initialization for deep linear networks, separating the growing and decaying regimes.Source: Physics: criticality · pp. 66–67

Math: recursion for the four-point correlator

The first-layer full four-point correlator matches the Gaussian expectation, but this Gaussianity does not hold in deeper layers.The deeper-layer correlator is analyzed through a layer-to-layer recursion.Source: Math: recursion for the four-point… · p. 68
The four-point recursion reduces to a recursion for G(ℓ), which has a simple solution.The solution uses the initial condition and the covariance solution.Source: Math: recursion for the four-point… · p. 69

Physics: large-n expansion, non-Gaussianities, interactions, and fluctuations

At criticality, leading finite-width effects scale with depth/width, increasing non-Gaussianity and neuron interactions while causing preactivation-observable fluctuations.When depth approaches width, fluctuations can dominate even at CW=1; the authors suggest a small but nonzero aspect ratio may balance representation learning with controlled fluctuations.Source: Physics: large-n expansion,… · pp. 70–72

3.4 Chaos

Wick contractions extend the two- and four-point calculations to all higher-point correlators, completing the goal of determining the output distribution.The correlators are evaluated on a single input to simplify the algebra.Source: §3.4 · p. 73

Math: recursions for six-point and higher-point correlators

The first-layer preactivation distribution is Gaussian, while deeper-layer 2m-point correlators follow a recursion with a simple solution for their layer dependence.The general correlators are decomposed into a neural-index tensor and a layer-dependent factor G(ℓ).Source: Math: recursions for six-point and… · pp. 73, 75

Physics: breakdown of perturbation theory and the emergence of chaos

At fixed width, sending depth to infinity makes higher-point correlators grow exponentially and yields extremely non-Gaussian, unusable outputs, even at CW=1.This limit shows how large-width perturbation theory breaks down as depth becomes too large.Source: Physics: breakdown of perturbation… · p. 76
At fixed aspect ratio r, the joint large-depth, large-width limit interpolates from Gaussian behavior as r→0 to chaotic behavior as r→∞; connected six-point effects scale as r^2.For sufficiently small r, the connected six-point correlator is more suppressed than the connected four-point correlator.Source: Physics: breakdown of perturbation… · p. 77
Higher-point connected correlators are suppressed by successively higher powers of r, enabling small-r truncation, though the exponential approximation can fail for sufficiently large m at finite n and L.The interpolating solution assumes fixed m, so it need not hold across the full set of correlators.Source: Physics: breakdown of perturbation… · pp. 77–78

RG Flow of Preactivations

The large-n expansion organizes finite-width preactivation distributions in powers of 1/n, while representation-group flow formalizes how layerwise marginalization changes neuron interactions with depth.This makes representation coarse-graining concrete and provides a way to track how preactivation distributions change as layers are added.Source: RG Flow of Preactivations · pp. 80–81
Keeping leading 1/n corrections yields depth recursions for two- and four-point correlators, with each layer combining inherited and newly generated near-Gaussianity.Solving these recursions is presented as a route to extending criticality and depth-to-width analysis to general activation functions.Source: RG Flow of Preactivations · pp. 80–81

Wick this way: combinatorial derivation via correlators

Wick contractions show that all odd-point and connected higher-point first-layer correlators vanish, so full correlators follow from Gaussian pairings.The two-point variance is specified by the first-layer metric.Source: Wick this way: combinatorial… · pp. 82–83

4.1 First Layer: Good-Old Gaussian

At initialization, first-layer preactivations have a zero-mean Gaussian distribution whose variance is determined by the first-layer metric.The chapter gives both Wick-contraction and Hubbard-Stratonovich derivations of this distribution.Source: Wick this way: combinatorial… · p. 83, Hubbard-Stratonovich this way:… · p. 85

Hubbard-Stratonovich this way: algebraic derivation via action

The Hubbard-Stratonovich derivation integrates out Gaussian weights and biases to recover independent first-layer neuron distributions, with cross-sample correlations encoded by the metric.It trades parameter integrals and delta-function constraints for auxiliary variables.Source: Hubbard-Stratonovich this way:… · p. 85

Gaussian action in action

First-layer activation expectations reduce to Gaussian integrals over sample indices, and factorization across neurons shows there are no first-layer neuron interactions.The discussion contrasts this simple factorization with the more complicated nearly-Gaussian distributions in deeper layers.Source: Gaussian action in action · pp. 86–87

4.2 Second Layer: Genesis of Non-Gaussianity

The second-layer analysis begins by factoring the joint first- and second-layer preactivation distribution and using the Gaussian first-layer marginal.The supplied passage introduces the conditional distribution but does not include its resulting form.Source: §4.2 · pp. 87–88
The marginal distribution of second-layer preactivations is obtained by integrating the conditional distribution over first-layer preactivations governed by a Gaussian distribution.The passage introduces evaluating this marginal by integrating out the first-layer preactivations.Source: §4.2 · p. 88

Second-layer conditional distribution

The second-layer conditional distribution is Gaussian, but its variance is a random metric depending on the first-layer activations.The formalism can be adapted beyond multilayer perceptrons, whose architecture-specific Dirac delta expression is used here.Source: Second-layer conditional distribution · pp. 88–89
The second-layer metric fluctuation has mean zero, with its typical size determined by a two-point correlator involving a symmetric four-point vertex.The vertex depends on four input data points and is symmetric under exchanges within either pair and between the pairs.Source: Second-layer conditional distribution · p. 89
As the first-layer width n1 grows, metric fluctuations are suppressed; at infinite n1, they vanish through self-averaging.The suppression follows because the four-point vertex is of order one, and the fluctuations become more Gaussian by the central limit theorem.Source: Second-layer conditional distribution · pp. 89–90

Wick Wick Wick: combinatorial derivation … Running couplings with partial marginalizations

These sections develop layer-to-layer recursions for preactivation statistics, correlators, and effective couplings, using partial marginalization to make the calculations tractable. They frame depth as successive coarse-graining through representation group flow and derive finite-width corrections to the infinite-width description.

Wick Wick Wick: combinatorial derivation

The second-layer distribution is non-Gaussian at finite width, but its connected four-point correlator is controlled by a vertex suppressed for wide networks, yielding a Gaussian limit as n1 → ∞.A matched quartic action reproduces the correlators to leading order in 1/n1; its coupling is O(1/n1), and the effective interaction depends on the network inputs.Source: Wick Wick Wick: combinatorial derivation · pp. 90, 91, 93

Schwinger-Dyson this way: algebraic derivation

Expanding the marginal over metric fluctuations derives the second-layer quartic action, whose quartic coupling gives the leading connected four-point contribution while the quadratic finite-width correction is a small quantitative effect.The quadratic correction is omitted at this order; the quartic term captures the leading nontrivial neuron-neuron interaction.Source: Schwinger-Dyson this way: algebraic… · pp. 95–96

Nearly-Gaussian action in action

The quartic coupling changes same-neuron activation expectations and enables covariance between distinct neurons, indicating a finite-width route to feature learning.For distinct neurons, the covariance arises through the quartic interaction in the action.Source: Nearly-Gaussian action in action · pp. 96–97

4.3 Deeper Layers: Accumulation of Non-Gaussianity

Deeper-layer preactivation distributions are obtained recursively, using an action and a 1/n expansion to handle the non-Gaussian distribution inherited from the previous layer.The method extends the marginalization procedure used for the second layer.Source: §4.3 · p. 98

Recursion

A conditional distribution bridges adjacent layers: activation correlators determine the next layer’s preactivation correlators, with the mean metric governing two-point terms and its fluctuation governing connected four-point terms.The resulting correlators can be used to reconstruct the next layer’s marginal distribution.Source: Recursion · pp. 98–99

Action

The action ansatz expands around a Gaussian using even powers of preactivations, with data-dependent couplings governing interactions and related to preactivation correlators.It encompasses the first- and second-layer actions described earlier.Source: Action · p. 100
Neglecting higher-order action terms is self-consistent only when quartic and higher couplings are perturbatively small, as they are in sufficiently wide networks.The large-width induction gives v(ℓ) = O(1/n) and g(ℓ) = G(ℓ) + O(1/n).Source: Action · p. 100, Large-width expansion · p. 101

Large-width expansion

At infinite width, preactivation marginals are Gaussian; finite-width quartic interactions give the first nontrivial corrections, with higher-order terms expected to change quantitative rather than qualitative behavior.The large-width expansion yields recursions for the two- and connected four-point correlators, with the mean metric and four-point vertex remaining O(1).Source: Large-width expansion · p. 103

4.4 Marginalization Rules

The marginalization rule allows expectations to be computed from the distribution restricted to the samples and neurons on which the observable depends.This uses consistency of probability distributions under marginalization.Source: §4.4 · p. 104

Marginalization over samples

Sample marginalization reduces integrals and dataset sums to the samples involved in the correlator; inverse metrics for the four-point vertex must use the restricted metric submatrix.Two- and four-activation Gaussian integrals then involve at most two and four variables, respectively.Source: Marginalization over samples · pp. 104–105

Marginalization over neurons

Integrating out spectator neurons resolves the apparent scale-counting problem: correlators involve only mℓ ≪ nℓ neurons, so action sums can be restricted to those relevant neurons.The remaining previous-layer widths are parametrically large.Source: Marginalization over neurons · p. 105

Running couplings with partial marginalizations

Partial marginalization makes the quadratic coupling depend on the retained neuron count, while observables without neural indices should remain unchanged across action representations.The section derives a running equation for the quadratic coupling and verifies it through a single-input metric calculation; it recommends expressing observables via G(ℓ) and V(ℓ) when in doubt.Source: Running couplings with partial… · pp. 107–108

4.5 Subleading Corrections

Finite-width metric and four-point-vertex correlators admit 1/n expansions, and the section begins deriving a recursion for the next-to-leading-order metric correction.The leading mean metric describes the infinite-width limit and is named the kernel; its next-order term is the NLO metric.Source: §4.5 · p. 108
The subleading-correction calculation recovers the kernel recursion and derives a recursion for the next-to-leading-order metric.It also shows cancellation of spectator-neuron contributions, as required by the marginalization rule.Source: §4.5 · pp. 110–111
Higher-order corrections can be computed systematically: the subleading four-point vertex requires the leading sextic coupling, which enters the action at order 1/n^2.That coupling contributes to a connected six-point function.Source: §4.5 · p. 111

4.6 RG Flow and RG Flow

Representation group flow tracks how layerwise transformations of representations arise by marginalizing fine-grained features into coarser output representations.The text defines each neuron's feature as a scalar function of the input and the layer's representation as a vector-valued function.Source: §4.6 · p. 113
Sequentially marginalizing layer preactivations yields recursions that determine an effective description of network outputs at the depth of interest.The recursions also describe how preactivation correlators and statistics evolve from layer to layer.Source: §4.6 · pp. 111, 112, 115
Relevant finite-width couplings make networks deviate increasingly from infinite-width models as depth grows, with consequences for fluctuations and representation learning.The section identifies the four-point vertex as the interaction whose relevance determines this behavior.Source: §4.6 · p. 116

Effective Theory of Preactivations at Initialization … Summary

This section develops a criticality and fluctuation framework for preactivation propagation in nonlinear MLPs, using kernel recursions and susceptibilities to characterize initialization across activation functions. It examines how activation universality classes, including the K⋆ = 0 class, shape RG flow and large-depth kernel behavior.

Effective Theory of Preactivations at Initialization

Initialization behavior grows more sensitive to hyperparameters with depth; the chapter gives activation- and architecture-specific critical tuning, identifies activations without criticality, and notes similarities among critically tuned activations.The chapter also resolves a subtlety in criticality analysis for nonsmooth activations such as ReLU.Source: Effective Theory of Preactivations at… · p. 118
At criticality, the depth-to-width ratio L/n controls the effective theory’s validity and finite-width corrections, from near-infinite-width behavior in short, wide networks to non-Gaussian fluctuations in tall, narrow ones.The analysis takes the infinite-width limit before the large-depth limit and derives asymptotic two- and four-point preactivation correlators.Source: Effective Theory of Preactivations at… · pp. 117–118

5.1 Criticality Analysis of the Kernel

Nonlinear kernel recursions mix off-diagonal components, unlike linear networks, and this mixing can support generalization while complicating analysis; at most two inputs need to be analyzed.For nonlinear activations, the activation expectation is a nonlinear function of the kernel, and the recursion mixes components for distinct inputs.Source: §5.1 · p. 119

A single input

For one input, criticality requires tuning initialization to prevent exponential kernel growth or collapse; the kernel may remain fixed, decay toward zero, or decay toward a nonzero fixed point.The chapter associates zero-fixed-point power-law decay with activations including tanh and sin, and nonzero-fixed-point decay with SWISH and GELU.Source: A single input · p. 121

Two inputs

For two inputs, analyzing the evolution of their difference toward the coincident-input fixed point yields a second criticality condition that, together with the single-input condition, determines critical initialization.The difference perturbation can remain nonzero even when the inputs have equal magnitudes.Source: Two inputs · pp. 121–122
Parity-based δ expansions reduce two-input kernel dynamics to sequentially solved decoupled recursions, rather than three coupled nonlinear recursions.The expansions assume sufficiently smooth activations; nonsmooth activations such as ReLU require more complicated analysis.Source: Two inputs · pp. 124–125

Deriving bootstrapped recursions

To derive the bootstrapped recursions, the analysis evaluates a general two-variable Gaussian expectation in kernel eigen-coordinates, then expands it to obtain component-wise recursions.The eigenbasis factorizes the Gaussian distribution, and the calculation retains terms through second order in δ.Source: Deriving bootstrapped recursions · pp. 125, 127, 128

Summary

The parallel susceptibility prevents exponential growth or decay of norm perturbations, while the perpendicular susceptibility governs whether input distances avoid exponential behavior.The perpendicular condition keeps the distance between propagated inputs well behaved.Source: Summary · p. 130
Criticality requires χ∥(K⋆00)=χ⊥(K⋆00)=1, preserving the entire kernel matrix to leading order and enabling critical initialization hyperparameters for nonlinear activations.These conditions generalize criticality for deep linear networks.Source: Summary · pp. 130–131
Zero-mean deterministic weight distributions, including zero initialization, cannot meet both criticality conditions and decay exponentially toward a trivial fixed point.Both susceptibilities are proportional to CW, so they cannot equal one when CW = 0.Source: Summary · p. 130

5.2 Criticality for Scale-Invariant Activations

For scale-invariant activations, the criticality conditions can be solved exactly; this class includes linear and ReLU activations.The class has piecewise-linear form with separate slopes on positive and negative inputs.Source: §5.2 · p. 131
For scale-invariant activations, χ > 1 causes exponential explosion, χ < 1 gives a stable finite fixed point, and χ = 1 yields critical or semi-critical behavior depending on Cb.At CW = 1/A2 and Cb = 0, there is a line of nontrivial fixed points; for Cb > 0, the midpoint kernel grows linearly rather than exponentially. For ReLU, the critical initialization is (Cb, CW) = (0, 2) [48].Source: §5.2 · p. 132
At criticality, nonlinear scale-invariant activations can show a benign power-law decay rather than preserving the whole kernel matrix or exhibiting exponential behavior.Finite-angle analysis qualifies the infinitesimal-perturbation result: K(ℓ) crosses over from near constancy to decay ∼1/ℓ2 at large ℓ.Source: §5.2 · p. 132

5.3 Universality beyond Scale-Invariant Activations

Scale-invariant activations form a universality class because their equal, kernel-independent susceptibilities simplify criticality analysis and yield similar representation group flow behavior.More generally, activation functions in a universality class share the same effective preactivation description after many layers, independent of fine activation details.Source: §5.3 · p. 133

5.3.1 General Strategy

The general procedure finds critical initialization by solving a hyperparameter-independent susceptibility-ratio condition for the fixed-point kernel, then evaluating the critical hyperparameters.For tanh, the condition is met at K⋆_00 = 0, giving (CW, Cb) = (1, 0); the procedure also recovers the scale-invariant critical initialization.Source: §5.3.1 · pp. 134–135

5.3.2 No Criticality: sigmoid, softplus, nonlinear monomials, etc.

Sigmoid, softplus, and nonlinear monomial activations cannot be tuned to criticality under the analyzed conditions, so the section discourages their use in deep networks.The sigmoid requires an unphysical negative bias variance, softplus fails the criticality condition, and nonlinear monomials fail it when p ≠ 1.Source: §5.3.2 · pp. 136–137

5.3.3 K⋆= 0 Universality Class: tanh, sin, etc.

For a smooth activation function, the conditions σ(0)=0 and σ′(0)≠0 are both necessary and sufficient for having a nontrivial fixed point at K⋆_00=0.A finite variance also requires σ1 ≠ 0.Source: §5.3.3 · pp. 137–138
Activations satisfying conditions (5.89), when initialized with (5.90), have a nontrivial fixed point at K⋆.Source: §5.3.3 · p. 138
Activations that vanish at the origin and have a nonzero first derivative form the K⋆ = 0 universality class, with tanh as a canonical member and sin as another example.Source: §5.3.3 · p. 138
When tuned to criticality, activations satisfying (5.89) behave similarly under RG flow, and large-depth kernel behavior depends only on the first few Taylor coefficients of σ(z).Source: §5.3.3 · p. 138

Deep asymptotic analysis for the midpoint kernel · Deep asymptotic analysis for parallel perturbations

At criticality, midpoint-kernel perturbations decay with a universal 1/ℓ law, while parallel perturbations decay as 1/ℓ^2. Their leading exponents are universal within the K⋆=0 class, whereas data and activation details appear in subleading terms and coefficients.

Deep asymptotic analysis for the midpoint kernel

The midpoint-kernel perturbation has a universal critical exponent p0 = 1, giving mild power-law decay independent of the activation function within the K⋆=0 class.Source: Deep asymptotic analysis for the… · p. 139
A consistent positive midpoint-kernel power-law solution requires (−a1) > 0; otherwise the fixed point is unstable and repels the kernel exponentially.SWISH and GELU exhibit this instability near K⋆=0.Source: Deep asymptotic analysis for the… · p. 139
Subtracting log(2) from softplus does not restore criticality: the kernel is repelled from the candidate fixed point and diverges exponentially.Source: Deep asymptotic analysis for the… · p. 139
The subleading midpoint-kernel expansion requires a log(ℓ)/ℓ^2 term to cancel 1/ℓ^3 contributions, and its crossover constant ℓ0 depends on the input norm.Higher-order corrections can refine the expansion to arbitrary degree.Source: Deep asymptotic analysis for the… · pp. 139–140
At large depth, midpoint-kernel terms depend on progressively fewer activation Taylor coefficients, making activations with the same first three coefficients indistinguishable in this asymptotic limit.The leading term depends only on a1.Source: Deep asymptotic analysis for the… · pp. 140–141
Fine-tuning activation Taylor coefficients can produce critical exponents other than p0 = 1; the generic 1/ℓ behavior does not require such tuning.Source: Deep asymptotic analysis for the… · p. 141

Deep asymptotic analysis for parallel perturbations

The parallel perturbation has critical exponent p∥ = 2, corresponding to a 1/ℓ^2 falloff.The leading difference reflects distinct input-dependent constants in the diagonal-kernel expansions.Source: Deep asymptotic analysis for parallel… · p. 141
The 1/ℓ^2 parallel-perturbation coefficient δ∥ carries data dependence through the logarithm of the ratio of input-dependent constants ℓ+ and ℓ−.Source: Deep asymptotic analysis for parallel… · p. 141

Deep asymptotic analysis for perpendicular perturbations … RG flow of the polar angle

The section analyzes deep RG flow of kernels and polar angles, including perpendicular perturbations, finite-width and NLO corrections, and scale-invariant activations beyond small-perturbation treatments. It characterizes their scaling and evolution, including nonlinear polar-angle recursion and nonzero kernel fixed points for SWISH and GELU.

Deep asymptotic analysis for perpendicular perturbations

For odd activations, p⊥ = 1: perpendicular perturbations decay as 1/ℓ, preserving angles between nearby inputs under deep RG flow.More generally, the presumptive exponent p⊥ = b1/a1 depends mildly on the activation function.Source: Deep asymptotic analysis for… · p. 142

5.3.4 Half-Stable Universality Classes: SWISH, etc. and GELU, etc.

SWISH and GELU form half-stable universality classes with nonzero kernel fixed points; these smooth ReLU-like activations weakly break scale invariance and are judged inferior to ReLU.For SWISH, the nontrivial fixed point is approximately 14.3; for GELU, it is 17/2.Source: §5.3.4 · pp. 143–144

5.4 Fluctuations

Leading finite-width fluctuations scale with L/n, so deeper networks deviate further from the infinite-width limit; with an O(1/n) correction to CW, the NLO metric remains subdominant to the kernel.These fluctuations matter because a single network instantiation can deviate from the ensemble mean.Source: §5.4 · pp. 145–146

A single input, reloaded

Single-input analysis suffices for criticality because only the single-input NLO metric adds tuning; multi-input vertex and NLO metric solutions add none, while Jacobian variance is deferred to NTK variance.Source: A single input, reloaded · p. 146

5.4.1 Fluctuations for the Scale-Invariant Universality Class

For scale-invariant activations, the single-input metric has no O(1/n) correction, while at criticality the four-point vertex grows linearly under RG flow (pV = −1).Source: §5.4.1 · p. 148
The reported fluctuation strengths are 2 for linear activations and 5 for ReLU, showing that fluctuation strength is not universal.Even under semi-criticality, the dimensionless four-point vertex grows linearly with depth, preserving the universal ℓ/n scaling of finite-width corrections.Source: §5.4.1 · p. 148

5.4.2 Fluctuations for the K⋆ = 0 Universality Class

For equal-norm inputs, the K⋆ = 0 universality class comprises smooth activations with σ(0) = 0 and σ′(0) ≠ 0, with tanh as an exemplar.Source: §5.4.2 · p. 148

Four-Point Vertex

The normalized four-point vertex follows the same scaling across universality classes, so its leading finite-width corrections remain relevant under representation group flow.For K⋆ = 0, pV = 1 and V(ℓ) ∼ 1/ℓ, unlike pV = −1 and V(ℓ) ∼ ℓ for the scale-invariant class.Source: Four-Point Vertex · p. 150

NLO metric, bare

At criticality, the bare NLO metric has a constant-in-depth leading contribution, and its relative correction scales with the depth-to-width ratio ℓ/n.The ℓ/n scaling is an artifact of not tuning initialization hyperparameters at finite width.Source: NLO metric, bare · p. 151

NLO metric, renormalized

Under the finite-width tunings, p1 − p0 = 0 and the dimensionless ratio (5.137) is O(1) in depth at leading order, making the NLO metric correction marginal under RG flow.These tunings affect the NLO metric but not leading-order observables depending on the four-point vertex, whose leading contributions are O(1/n).Source: NLO metric, renormalized · pp. 152–153
With finite-width critical tunings, the NLO-metric correction is marginal and O(1) in depth, while four-point-vertex corrections remain relevant.The required initialization corrections become increasingly fine as depth approaches width, eventually exceeding floating-point precision for reasonable network widths and making critical tuning practically impossible.Source: NLO metric, renormalized · pp. 152–153

5.5 Finite-Angle Analysis for the Scale-Invariant Universality Class

For nonlinear scale-invariant activations, the apparently layer-constant perpendicular kernel perturbation is an artifact of the perturbative δ expansion, motivating a nonperturbative finite-angle analysis.The analysis considers two equal-norm inputs without assuming their difference is small.Source: §5.5 · p. 154

RG flow of the polar angle

At criticality, setting Cb = 0 and A2CW = 1 keeps the diagonal kernel constant; for ReLU, the analysis evaluates the Gaussian expectation to derive polar-angle flow.The exact coordinate transformations simplify the ReLU integral, whereas general activations would require a perturbative expansion in the polar angle.Source: RG flow of the polar angle · pp. 155–156
The arbitrary scale-invariant activation expectation separates into first- and third-quadrant terms proportional to a_+^2 and a_-^2, and second- and fourth-quadrant terms proportional to a_+a_−.The first and third quadrants have z+z−>0, while the second and fourth have z+z−<0.Source: RG flow of the polar angle · p. 157
The full nonperturbative recursion for the off-diagonal kernel is evaluated, but it remains highly nonlinear in ψ(ℓ+1).The recursion is for the scale-invariant activation class.Source: RG flow of the polar angle · p. 157
The Gaussian expectation ⟨σ′(z+) σ′(z−)⟩K(ℓ) is also evaluated, using that σ′(z+) σ′(z−) is constant within each quadrant.This is an additional nonperturbative expectation for the scale-invariant universality class.Source: RG flow of the polar angle · p. 157

Criticality analysis of the polar angle … An equivalence principle for learning rates

This section develops an account of Bayesian inference and learning in neural networks, from model fitting and evidence to finite-width interactions that update representations. It then analyzes gradient-based learning through NTK dynamics and statistics, relating criticality and finite-width scaling to learning-rate choices across layers, depth, and width.

Criticality analysis of the polar angle

Analysis of the polar-angle recursion focuses on its zero-angle fixed point and linearizes in the small-angle regime to determine its asymptotic decay.Source: Criticality analysis of the polar angle · p. 158
The polar-angle exponent is pψ = 1 universally except for the degenerate linear case ρ = 0, where the angle is preserved and pψ = 0.The angle is exactly preserved only when ρ = 0.Source: Criticality analysis of the polar angle · pp. 158–159
Perpendicular perturbations cross from nearly constant behavior at small depth to ∼1/ℓ² decay at large depth, giving p⊥ = 2.Smaller initial angles between inputs delay the onset of the power-law regime.Source: Criticality analysis of the polar angle · p. 159
For nonlinear scale-invariant activations, singular Gaussian expectations require caution: a non-integer 3/2 correction cannot be generated by Taylor expansion.The passage contrasts this limitation with the smooth-activation K⋆ = 0 universality class, where the Taylor-expansion analysis is reliable.Source: Criticality analysis of the polar angle · p. 160

Bayesian Learning

Bayesian model comparison uses evidence to select among architectures and hyperparameters, quantify inductive biases, and incorporate Occam’s razor.The chapter also previews that model comparison prefers critical initialization, while finite-width correlated fluctuations induce a propensity for Hebbian learning; exact Bayesian learning yields output correlations and representation learning at finite width but not infinite width.Source: Bayesian Learning · p. 162

6.1 Bayesian Probability

Bayesian probability treats probabilities as strengths of belief rather than frequencies, and Bayes’ rule is the logically consistent way to update beliefs after observations.The framework derives from logical constraints expressed through the product and sum rules.Source: §6.1 · pp. 163–164

6.2 Bayesian Inference and Neural Networks

The chapter applies Bayesian inference to deep learning through model fitting and model comparison.Source: §6.2 · p. 164

6.2.1 Bayesian Model Fitting

For Bayesian model fitting, the initialization distribution can serve as the prior over network parameters, with architecture and initialization hyperparameters included in the hypothesis.The prior over preactivations represents beliefs about hidden-layer features and the output function.Source: §6.2.1 · p. 165
Bayesian fitting updates parameter beliefs by balancing observation likelihood against the prior, with the likelihood choice determining the loss, including MSE for uncertain outputs and cross-entropy for categorical outputs.The posterior favors functions that fit observations while adhering to prior beliefs; the likelihood’s form is part of the hypothesis.Source: §6.2.1 · p. 168

Approximation methods for model marginalization: MAP and MLE

MAP minimizes the data loss plus a prior-based regularizer, whereas MLE minimizes the loss alone; with sufficiently many observations, their estimates are expected to become similar.The loss grows with dataset size while the regularization term remains constant.Source: Approximation methods for model… · pp. 169–170
MLE optima form a continuum, and their distribution depends on initialization, learning algorithm, and training hyperparameters.Consequently, a point estimate does not remove the need to consider how optima are obtained.Source: Approximation methods for model… · p. 170

Exact method for model marginalization: effective theory

The effective-theory method incorporates observations by conditioning the joint output prior, yielding posterior beliefs about unobserved outputs without separately integrating model parameters.Parameter marginalization has already occurred in forming the output prior; predictions represent the full ensemble of potential networks weighted by posterior plausibility, rather than a typical realization.Source: Exact method for model… · pp. 171–172

6.2.2 Bayesian Model Comparison

Bayesian model comparison uses evidence to compare hypotheses such as hyperparameter settings; the Bayes factor updates their relative prior beliefs using the observations.A Bayes factor above one favors H1, while a ratio below one favors H2.Source: §6.2.2 · pp. 173–174

Occam’s razor

Bayes’ factors automatically implement Occam’s razor, favoring simple hypotheses that fit observations; model complexity is governed by effective couplings rather than the number of tunable weights and biases.For finite-width networks, the 1/n expansion yields effective theories with increasing complexity.Source: Occam’s razor · pp. 174–175

Inductive Bias

Inductive biases are prior assumptions about the desired function, and effective theory provides a blueprint for computing evidence to compare architectures on a dataset.Source: Inductive Bias · pp. 175–176

6.3 Bayesian Inference at Infinite Width

At infinite width, Bayesian analysis finds criticality favored for sufficiently deep networks, independent output components, and identical penultimate-layer prior and posterior distributions, implying no representation learning.Source: §6.3 · p. 176

6.3.1 The Evidence for Criticality

For a single input, criticality (χ∥(K⋆)=1) gives a finite, nonzero kernel and comes closest to maximal evidence, unlike the zero- and infinite-kernel limits.For polynomially vanishing kernels, evidence can remain finite at practical depths; at absurdly large depths, the Bayesian-optimal CW may lie slightly above its critical value.Source: §6.3.1 · pp. 178–179
Bayesian model comparison favors criticality when both parallel and perpendicular susceptibilities equal one; the distinction requires nonzero Y[1].For tasks where all true outputs have the same norm, the distinction may not matter.Source: §6.3.1 · p. 180

6.3.2 Let’s Not Wire Together

At infinite width, conditioning on observed outputs yields a Gaussian posterior with a shifted mean and covariance encoding prediction confidence, but output components remain independent.The text notes that computing the posterior mean requires inverting an observation-sized kernel matrix, making naive Bayesian learning impractical for large datasets.Source: §6.3.2 · pp. 183–185

6.3.3 Absence of Representation Learning

At infinite width, the posterior over penultimate-layer representations equals the prior, so observations do not update representations in any hidden layer before the output.The stated cause is a lack of interlayer correlation in the joint distribution.Source: §6.3.3 · p. 186

6.4 Bayesian Inference at Finite Width

Finite-width Bayesian learning is presented as producing neural association, correlated output components, and representation learning through non-Gaussian neuron interactions.The passage attributes output correlations to intralayer interactions and a posterior shift in penultimate-layer preactivations to interlayer interactions.Source: §6.4 · pp. 186–187

6.4.1 Hebbian Learning, Inc.

At finite width, interactions in the prior give neural networks an inductive bias for neural association, aligning with Hebbian learning.The section proposes determining how one preactivation influences another in the effective preactivation distribution at initialization.Source: §6.4.1 · p. 187
At finite width, atypical firing in one neuron makes other neurons more likely to fire atypically too, with associations growing with depth at criticality.The association is a propensity, not a guarantee, and scales with the normalized four-point vertex, which is proportional to ℓ/n at criticality.Source: §6.4.1 · p. 189

6.4.2 Let’s Wire Together

Finite-width posterior predictions couple output components, allowing observations of one component to influence predictions for another.This dependence descends from the finite-width prior’s fire-together bias and reflects posterior learning to wire together.Source: §6.4.2 · pp. 190, 192, 193
Finite-width networks incorporate Hebbian learning automatically, whether learning is Bayesian or gradient-based.The prior and initialization distribution share the relevant finite-width form.Source: §6.4.2 · p. 190
Finite-width Bayesian prediction is computationally costly because quartic couplings require large four-point tensors and inverse-kernel contractions.Both computation and memory requirements grow rapidly with the size of the observed dataset.Source: §6.4.2 · p. 193

6.4.3 Presence of Representation Learning

Finite-width interlayer interactions update penultimate-layer preactivations after observations, producing representation learning that is absent at infinite width.The update is defined as the difference between posterior and prior expectations of a penultimate-layer observable.Source: §6.4.3 · pp. 193–194
At large but finite width, observed outputs update hidden representations through 1/n-suppressed covariance and non-Gaussian interactions.The update depends on the difference between observed outputs and prior expectations; propagation to the (L −2)-th layer is further suppressed to O(1/n^2). The analysis provides a blueprint for studying finite-width representation learning under gradient-based training.Source: §6.4.3 · pp. 196–197

Gradient-Based Learning

Gradient-based learning updates one sampled network’s parameters to fit outputs to labels, enabling data-efficient training but making naive Bayesian inference computationally intractable at scale.The NTK is introduced as the Hamiltonian controlling observables’ training dynamics.Source: Gradient-Based Learning · pp. 199–200

7.1 Supervised Learning

Supervised learning fits p(y|x) by minimizing finite-sample training loss as a proxy for expected loss, then uses held-out test data to assess generalization.The parameters’ initialization distribution contains no information about the data distribution and must be adjusted for good predictions.Source: §7.1 · pp. 201–202

7.2 Gradient Descent and Function Approximation

Gradient descent iteratively moves parameters along the negative loss gradient; sufficiently small learning rates decrease training loss, and iterations typically reach at least a local minimum.Its computational cost scales linearly with the training-set size.Source: §7.2 · p. 203

Tensorial Gradient Descent

Tensorial gradient descent generalizes the global learning rate to a parameter-space tensor, and sufficiently small learning rates decrease loss when that tensor is positive semidefinite.Ordinary gradient descent is the special case where the tensor is the Kronecker delta.Source: Tensorial Gradient Descent · p. 204

Neural Tangent Kernel

The neural tangent kernel (NTK) is independent of the loss and governs training dynamics for observables that depend on model outputs.Function-approximation updates combine an error factor with the NTK.Source: Neural Tangent Kernel · p. 205
Off-diagonal NTK components mediate generalization across examples and let one output feature affect another, while test loss may still increase during training.Unlike training loss, a general observable is not guaranteed to decrease after an update; this includes test loss for models that overfit.Source: Neural Tangent Kernel · p. 206

RG Flow of the Neural Tangent Kernel

Analyzing typical trained networks requires tracking the mean behavior and fluctuations across initializations, with initial observable changes governed by initialization NTK statistics.The chapter applies this statistical perspective to recursively compute joint NTK–preactivation statistics for deep MLPs.Source: RG Flow of the Neural Tangent Kernel · pp. 207–208

8.0 Forward Equation for the NTK

The NTK obeys a layer-to-layer forward iteration that computes it for each realization of the network parameters, analogous to the forward equation for preactivations.A statistical treatment then uses this recursion to compute joint NTK–preactivation distributions across layers.Source: §8.0 · p. 211

Scaling in the effective theory

A 1/n scaling of weight learning rates keeps the NTK order one at large width; ordinary gradient descent can instead overwhelm bias contributions and under-train biases.The framework also argues that making scales explicit can improve hyperparameter comparisons across architectures and robustness when scaling models.Source: Scaling in the effective theory · p. 212

Getting things backwards

Although backpropagation iterates derivatives backward through layers, the effective-theory analysis computes NTK statistics using the forward equation.The backward equation is the efficient algorithm used by deep-learning packages to compute parameter derivatives.Source: Getting things backwards · p. 213

8.1 First Layer: Deterministic NTK

First-layer preactivations are zero-mean Gaussian, and the absence of interactions between neurons lets expectations of observables factor into separate Gaussian integrals.The first-layer deterministic metric is a function of the inputs.Source: §8.1 · p. 214
The first-layer NTK is deterministic and diagonal in neural indices, while its joint distribution with preactivations is independent and representative of deeper layers at infinite width.In single-layer networks, a feature at one neuron therefore cannot affect another neuron's gradient-descent update.Source: §8.1 · pp. 214–215

8.2 Second Layer: Fluctuating NTK

The mean second-layer NTK is diagonal in neural indices, with its sample dependence separated from its stochastic fluctuations.Source: §8.2 · p. 216
At leading nontrivial order, second-layer statistics are nearly Gaussian with quartic preactivation interactions, a fluctuating NTK, and preactivation–NTK cross-correlation, with variance and cross-correlation suppressed by 1/n1.These effects vanish in the strict infinite-width limit.Source: §8.2 · pp. 215, 217, 218, 219

8.3 Deeper Layers: Accumulation of NTK Fluctuations

Correlated dependence on earlier-layer preactivations accumulates non-Gaussianity with depth and likewise amplifies NTK variance and NTK–preactivation cross-correlation.Recursions for the NTK mean, cross-correlation, and variance determine the deeper-layer joint distribution at leading nontrivial order in 1/n.Source: §8.3 · p. 219

8.3.0 Interlude: Interlayer Correlations

A generating-function marginalization trick reduces interlayer correlations to expectations over earlier-layer variables, yielding neighboring-layer preactivation correlations by averaging a Gaussian expectation against the stochastic metric.Source: §8.3.0 · pp. 220, 222, 223
Interlayer correlation formulas, together with intralayer correlations, enable recursive analysis of the joint NTK–preactivation statistics in deeper layers.Source: §8.3.0 · p. 223

8.3.1 NTK Mean

The NTK mean is diagonal in neural indices at every depth, and its leading-order recursion depends on the previous layer’s mean and fluctuation.Source: §8.3.1 · pp. 223–224

8.3.2 NTK-Preactivation Cross Correlations

General NTK–preactivation cross-correlations are governed by the same sample-index tensors D and F as the elementary cross-correlation.Source: §8.3.2 · p. 227

D-recursion

At leading order, the D recursion mixes prior-layer D correlations with the four-point vertex V, but not with F correlations or NTK variance.Source: D-recursion · p. 228

F-recursion

The F recursion does not mix with other finite-width tensors; both D and F remain order one, so all cross-correlations are suppressed by 1/n and vanish at infinite width.Source: F-recursion · p. 229

8.3.3 NTK Variance

Deeper-layer NTK variance analysis accounts for prior-layer intralayer correlations and the fluctuating NTK, decomposing the variance into A- and B-type tensors.Source: §8.3.3 · pp. 229–230

B-recursion

The B-type variance recursion remains order one and, at leading order, does not mix with other finite-width tensors.Source: B-recursion · p. 232

A-recursion

The A-type variance recursion remains order one and mixes with V and D, but not B or F; together, leading cross-correlations and NTK variance scale as 1/n and vanish at infinite width.Source: A-recursion · p. 234

Effective Theory of the NTK at Initialization

NTK fluctuations and cross-correlations accumulate with depth, finite-width effects are controlled by L/n, and criticality mitigates exploding and vanishing gradients.The NTK is deterministic in the first layer and becomes stochastic and cross-correlated in deeper layers.Source: Effective Theory of the NTK at… · p. 235

9.1 Criticality Analysis of the NTK

The analysis assumes uniform hidden widths and layer-independent initialization variances, retains layer-dependent bias learning rates, and focuses on leading-order single-input statistics.Source: §9.1 · p. 236

Leading-order NTK recursions for a single input

The leading O(1) NTK mean, called the frozen NTK, controls training dynamics in the infinite-width limit.Source: Leading-order NTK recursions for a… · p. 236

The relevance of scaling laws

For both scale-invariant and K⋆ = 0 classes, the leading finite-width effects of the NTK-preactivation joint distribution are relevant and controlled by the same ℓ/n cutoff.The scaling relations include pA − 2pΘ = −1, pB − 2pΘ = −1, pD − pΘ − p0 = −1, and pF − pΘ − p0 = −1.Source: The relevance of scaling laws · p. 240

Formalities: perpendicular perturbations and the frozen NTK

Avoiding exponential behavior in the frozen NTK recursion requires χ⊥ = 1, the same criticality condition that preserves differences between nearby inputs as they propagate.Source: Formalities: perpendicular… · p. 241

9.2 Scale-Invariant Universality Class

For the scale-invariant class, whose canonical activations include ReLU and linear functions, critical initialization sets both susceptibilities to unity.Source: §9.2 · pp. 241–242

NTK mean (frozen NTK)

With uniform learning rates, the critical scale-invariant frozen NTK grows linearly with depth, giving uniform layer contributions and critical exponent pΘ = −1.Source: NTK mean (frozen NTK) · pp. 242–243

NTK variance and NTK-preactivation cross correlation (agitated NTK)

The scale-invariant class has critical exponents pD = −2, pF = −2, pB = −3, and pA = −3 for its leading NTK variance and cross-correlation observables.Source: NTK variance and NTK-preactivation… · p. 244

9.3 K⋆= 0 Universality Class

A nontrivial K⋆=0 fixed point requires an activation that vanishes at the origin with nonzero slope; the single-input kernel approaches it slowly, and normalized quantities scale as ℓ/n.tanh and sin are examples in this class.Source: §9.3 · pp. 244–245

NTK mean (frozen NTK)

With layer-independent rates, weight contributions to the frozen NTK decline relative to bias contributions while deeper layers dominate; layer-dependent rates balance them, giving pΘ = 0 for tanh and sin.The balanced rates scale with depth to equalize the per-layer contributions.Source: NTK mean (frozen NTK) · pp. 246–247

NTK variance and NTK-preactivation cross correlation (agitated NTK)

The leading finite-width NTK variance and NTK–preactivation cross-correlation scale as ℓ/n regardless of activation function.For the K⋆=0 class, pF = pD = p⊥−1 and pB = pA = 2p⊥−3; these exponents, together with the kernel and frozen-NTK exponents, obey the scaling law across universality classes.Source: NTK variance and NTK-preactivation… · p. 248

9.4 Criticality, Exploding and Vanishing Problems, and None of That

The discussion returns to criticality’s original motivation—exploding and vanishing problems—and reframes it through the behavior of gradients during training.The authors state that criticality is sufficient to mitigate both problems.Source: §9.4 · p. 249, Traditional view on the exploding and… · p. 250

Traditional view on the exploding and vanishing gradient problem

Without fine-tuning, products in the chain-rule factor can grow or decay exponentially, leaving shallower-layer parameters with poorly behaved or negligible training signals.Exploding gradients can indicate instability in which the loss may increase or blow up.Source: Traditional view on the exploding and… · pp. 249–250

Critical view on the exploding and vanishing gradient problem

Criticality mitigates exploding and vanishing gradient problems by setting χ∥(K⋆) = 1 and χ⊥(K⋆) = 1, controlling kernel-related factors and chain-rule factors in the NTK.When χ⊥(K⋆) exceeds one, deeper-layer NTKs can explode; when it is below one, shallower-layer contributions are exponentially diminished.Source: Critical view on the exploding and… · p. 251

An equivalence principle for learning rates

The equivalence principle requires parametrically equal NTK contributions from every layer for each parameter group; scale-invariant activations meet it with layer-independent learning rates.Learning rates require appropriate depth and width scalings, including rescaling weight rates by the previous layer’s width and by overall network depth.Source: An equivalence principle for learning… · p. 252
Critical initialization can mitigate vanishing gradients for any activation function admitting critical initialization hyperparameters, including saturating tanh.The passage contrasts this with practitioners’ adoption of ReLU over saturating activations such as tanh.Source: An equivalence principle for learning… · p. 252
For odd smooth activations such as tanh and sin, the critical exponent for perpendicular perturbations is p_⊥ = 1, yielding a simpler learning-rate prescription.The prescription uses separate depth scalings for bias and weight learning rates to ensure uniform contributions across parameter groups and layers to the asymptotic frozen NTK solution.Source: An equivalence principle for learning… · p. 253
The passage suggests that ReLU’s layer-independent learning rates, unlike tanh’s required layer rescalings, might partly explain ReLU’s empirical preference.This is posed as a possible explanation rather than an established result.Source: An equivalence principle for learning… · p. 253
The equivalence principle’s layer-, depth-, and width-dependent learning-rate scalings are expected to reduce variation across architectures with different widths and depths.The order-one training hyperparameters still depend on the task.Source: An equivalence principle for learning… · p. 253
Relative layer scaling lets deeper-layer weights receive polynomially non-vanishing gradients and contribute polynomially equally to other parameters’ training dynamics.The passage characterizes this as a polynomial version of the vanishing-kernel problem.Source: An equivalence principle for learning… · p. 253

Kernel Learning … Linear ∗-Polation by Deep Linear Networks

This section analyzes gradient-based learning and generalization in infinite-width networks, characterizing trained predictions through frozen-NTK kernel rules and initialization-driven variability, and examining how robustness, kernel perturbations, critical initialization, and learning-rate scaling shape generalization. It also contrasts tuned nonlinear scale-invariant networks with deep linear networks, which linearly *-polate and are restricted to linear target functions.

10.1 A Small Step

In the infinite-width limit, training is governed by the frozen NTK, and the output update is exactly linear in the global learning rate.The section uses this small-step analysis to establish that output components move independently and hidden-layer representations do not change.Source: Kernel Learning · p. 255, §10.1 · p. 257

10.1.1 No Wiring

Each output component updates independently, so gradient training at infinite width does not generate correlations between output components.For the MSE loss, the update depends only on the corresponding component of the error factor.Source: §10.1.1 · p. 258

10.1.2 No Representation Learning

At infinite width, the penultimate-layer representation does not change under a gradient update because the update’s mean and covariance vanish with 1/n.The section finds the same absence of representation learning as for exact Bayesian inference.Source: §10.1.2 · pp. 259–260

10.2 A Giant Leap

The section previews a fully trained infinite-width solution obtainable in one theoretical step, equivalent to gradient descent or stochastic gradient descent with decreasing learning rates.The solution memorizes the training set and is independent of the loss form; test predictions depend on the initial output, frozen NTK, and training set.Source: Kernel Learning · p. 256, §10.2 · pp. 260–261

10.2.1 Newton’s Method

A scalar learning rate cannot undo sample mixing by the frozen NTK, motivating a Newton tensor that enables anisotropic updates in sample space.The tensor extends the update so the fully trained condition can be satisfied.Source: §10.2.1 · pp. 261–262
Newton’s method’s giant leap reaches a fully trained network in one step, exactly matching training targets and minimizing any loss whose minimum requires that fit.Source: §10.2.1 · pp. 263–265

10.2.2 Algorithm Independence

At infinite width, the fully trained solution is fixed by the frozen NTK, initial outputs, and training targets, not by the loss, optimizer, or number of steps.Source: §10.2.2 · pp. 266–267

10.2.3 Aside: Cross-Entropy Loss

Cross-entropy is suited to classification, and matching its softmax outputs yields the same giant-leap solution; hard labels require infinite logits, making early stopping a regularizer.Source: §10.2.3 · pp. 267–268

10.2.4 Kernel Prediction

Infinite-width fully trained MLP test predictions are governed by a frozen-NTK kernel rule, regardless of one-step or many-step training or the learning algorithm’s loss.Source: §10.2.4 · pp. 268–269

initialization:

The stochasticity in a fully trained network’s prediction arises from its initialization outputs, which depend on the particular realization of initialized parameters.Source: initialization · p. 269
The trained infinite-width ensemble has a closed-form generalized posterior whose mean and covariance describe average predictions and variation across network initializations.Its covariance reflects instantiation-to-instantiation variation, unlike Bayesian posterior covariance, which represents uncertainty about unseen data.Source: initialization · pp. 270–271

That One Place Where Gradient Descent = Exact Bayesian Inference

In the infinite-width limit, training only final-layer biases and weights makes gradient descent reproduce exact Bayesian inference; ensemble implementation avoids kernel inversion but may require many wide models.The equivalence holds for the specified training-hyperparameter choices, and becomes exact for an infinite ensemble.Source: That One Place Where Gradient Descent… · pp. 271–272

10.3 Generalization

The ensemble’s generalization error decomposes into bias from the mean prediction and variance from prediction fluctuations across models; for general losses, variance depends on covariance and loss curvature.The bias-variance terms proxy generalization error when individual model predictions are close to the ensemble mean.Source: §10.3 · pp. 273–274

10.3.1 Bias-Variance Tradeoff and Criticality

For nearby test and training inputs, prediction change measures robustness: hyperparameters should support networks that are robust but not so inflexible that they cannot adapt.When the prediction is correct, the two terms defining the bias cancel; the analysis uses a local expansion of the frozen NTK.Source: §10.3.1 · pp. 275–276
Setting χ⊥(K⋆)=1 can balance bias and variance, while χ∥(K⋆)=1 is also a natural choice because it makes contributions from all layers equal in the midpoint frozen NTK.The robustness measure captures nearby-input prediction robustness, and the leading variance term depends on the perpendicular kernel perturbation.Source: §10.3.1 · pp. 276–277

K⋆= 0 Universality Class

For the scale-invariant universality class, Θ[1] has the same depth dependence as the single-input frozen NTK; with χ∥=χ⊥=χ=1, exponential explosions and vanishments are mitigated.Source: K⋆= 0 Universality Class · p. 278
For K⋆=0 activations, order-one robustness at great depth selects p⊥=1; non-odd activations with σ2≠0 have p⊥<1 and a bias term that grows with depth.Any odd K⋆=0 activation function, including tanh and sin, satisfies the p⊥=1 condition.Source: K⋆= 0 Universality Class · pp. 280–281
Critical initialization and the learning-rate equivalence principle keep the one-training-one-test generalization error’s bias–variance tradeoff in check, with tunings remaining useful across widths and depths.The stated validity extends beyond infinite width to nonzero aspect ratios L/n ≪1.Source: K⋆= 0 Universality Class · p. 281
Robustness slowly but surely decays with overall network depth L, making this training-hyperparameter choice polynomially suboptimal compared with the choice based on the equivalence principle.35The text says considering more general activation functions with p⊥ ≠ 1 does not change this conclusion.Source: K⋆= 0 Universality Class · p. 281

Scale-Invariant Universality Class

For nonlinear scale-invariant activations, finite-angle analysis links kernel angle evolution to frozen-NTK angle evolution; tuning χ=1 yields an order-one bias-term robustness measure.The finite-angle analysis is used because the Gaussian expectation involving σ′′ is singular for these activations.Source: Scale-Invariant Universality Class · pp. 282–283
With criticality and equivalence tuning, nonlinear scale-invariant networks achieve constant bias and variance; their asymptotic generalization errors are hyperparameter-independent, although the depth at which they apply depends on training and data conditions.The asymptotic errors also do not depend on δ2, which encodes test-training point separation.Source: Scale-Invariant Universality Class · p. 284
The deep-linear frozen-NTK ratio polynomially approaches a constant that depends in detail on first-layer observables, unlike the nonlinear scale-invariant case.This input-dependent behavior connects generalization to the input and highlights deep linear networks’ limitations in interpolation and extrapolation.Source: Scale-Invariant Universality Class · p. 284
In exact Bayesian inference with learning rates only in the last layer, robustness decays with depth, making this restricted case inferior to uniformly trained gradient-descent ensembles.The comparison is with fully trained networks using uniform learning rates across layers.Source: Scale-Invariant Universality Class · p. 284

10.3.2 Interpolation and Extrapolation

The analysis defines *-polation along the line through two training inputs: it is interpolation for s in [0, 1] and extrapolation outside that interval.The subsection asks what a fully trained network predicts along this one-parameter family when outputs at both training inputs are known.Source: §10.3.2 · p. 285

Linear ∗-Polation by Deep Linear Networks

Fully trained deep linear networks linearly *-polate: predictions on a line through two training inputs are the corresponding linear combinations of their outputs, limiting them to linear target functions.The relation holds at initialization and after training; nonlinear targets require nonlinear activation functions.Source: Linear ∗-Polation by Deep Linear… · p. 286

Nonlinear ∗-Polation by Smooth Nonlinear Deep Networks … P-recursion

The section develops nonlinear ∗-polation for smooth deep networks and connects kernel prediction to linear models in parameter and sample spaces. It analyzes finite-width representation learning through evolving NTK features and stochastic dNTK updates, including their cross-correlations and the recursion for the P tensor.

Nonlinear ∗-Polation by Smooth Nonlinear Deep Networks

For smooth nonlinear activations, fully trained networks can nonlinearly ∗-polate, with predictions also depending on initialization outputs unlike deep linear networks.For scale-invariant activations with kinks, extending the analysis to unequal input norms is left as a challenge.Source: Nonlinear ∗-Polation by Smooth… · pp. 286, 288
The curvature ratio δδΘ[0]/Θ00 encodes activation- and architecture-dependent inductive bias, while generalization bias compares that curvature with true-output nonlinearity.Deep linear networks promote only linear functions; as the training inputs coincide, the curvature vanishes and ∗-polation approaches linear ∗-polation.Source: Nonlinear ∗-Polation by Smooth… · p. 289
As a test input approaches either training input, its ∗-polation approaches that training output with certainty, and the nearest training point contributes most.This locality suggests weighting nearby training predictions and using the network’s inductive biases to guide sampling over the data manifold.Source: Nonlinear ∗-Polation by Smooth… · p. 289

10.4 Linear Models and Kernel Methods

Kernel predictions can be viewed either as kernel-based ∗-polation of training data or as outputs of a trained model linear in its parameters.The section introduces this dual perspective to connect kernel prediction formulas with linear models.Source: §10.4 · p. 290

10.4.1 Linear Models

Linear models are linear in parameters, often use engineered input features, and admit a closed-form MSE solution whose learned weights can replace the training set.The model is linear in its parameters, not necessarily in its inputs.Source: §10.4.1 · pp. 290–292

10.4.2 Kernel Methods

Kernel methods exchange explicit feature vectors and feature-space inversion for training-sample similarities and an inverse, predicting by similarity-weighting training outputs.This memory-based form retains the training set; its prediction makes explicit that each output component is determined by the corresponding training-output components.Source: §10.4.2 · pp. 293–294
Kernel methods generalize a linear kernel by substituting a more complex kernel, while local methods such as k-nearest neighbors use nearby training points to avoid some prediction impracticalities.The text suggests extending local methods to finite-width exact Bayesian inference or finite-width gradient-based learning predictions.Source: §10.4.2 · p. 294

10.4.3 Infinite-Width Networks as Linear Models

At infinite width, Bayesian and frozen-NTK predictions match kernel-method predictions, and the fully trained mean network is a linear model on fixed random features.The resulting predictions are linear in the training outputs; finite-width networks can instead learn useful features.Source: §10.4.3 · pp. 295–298

Representation Learning

Finite-width NTK evolution is driven by its differential, whose statistics are O(1/n) and whose effect scales as L/n across the two universality classes.The differential of the NTK governs leading-order-in-learning-rate NTK dynamics; the NTK governs leading-order preactivation dynamics.Source: Representation Learning · pp. 299–300
A quadratic model provides an analyzable minimal model of representation learning, and its nearly-kernel description predicts with a trained kernel that learns from data.Deep networks add an architecture-induced inductive bias through layer-to-layer RG flow, providing initial features for learning.Source: Representation Learning · pp. 300–301

11.1 Differential of the Neural Tangent Kernel

To move beyond infinite width, Section 11.1 expands preactivation updates to second order in parameter changes and rewrites them using the chain rule for contributing layers.This analysis contributes to identifying the differential of the NTK as the source of representation learning.Source: Representation Learning · p. 300, §11.1 · p. 301
The dNTK, contracted with a layer error factor and multiplied by the learning rate, gives that layer’s NTK update after a gradient-descent step.The dNTK is a finite-width effect and vanishes in the strict infinite-width limit.Source: §11.1 · p. 304

11.2 RG Flow of the dNTK

The dNTK’s order-1/n statistics are needed for leading finite-width corrections and can establish representation learning when the dNTK is nonvanishing.The analysis develops a joint distribution of preactivations, the NTK, and the dNTK.Source: §11.2 · p. 304

11.2.0 Forward Equation for the dNTK

Combining three derivative cases yields a stochastic forward iteration equation for the dNTK, which provides the basis for recursions of its statistics.The cases distinguish whether neither, one, or both summed layers are the newest layer.Source: §11.2.0 · pp. 305, 307

11.2.1 First Layer: Zero dNTK

The dNTK vanishes identically in the first layer, providing the initial condition for the recursions and matching the infinite-width limit across layers.The first-layer NTK is parameter-independent and cannot change during training.Source: §11.2.1 · p. 307

11.2.2 Second Layer: Nonzero dNTK

In the second layer, the dNTK mean vanishes, but its cross-correlation with preactivations is order 1/n and captures leading finite-width correlations.The cross-correlation tensors and their deeper-layer counterparts encapsulate the dNTK’s entire effect at order 1/n.Source: §11.2.2 · pp. 308–309

11.2.3 Deeper Layers: Growing dNTK

The deeper-layer analysis proceeds by evaluating an interlayer formula with three weight insertions and using it to derive recursions for dNTK–preactivation cross-correlations.Source: §11.2.3 · p. 309

Interlude 2: Interlayer Correlations Reloaded

The interlayer-correlation interlude includes a new three-weight-insertion formula, derived by differentiating the generating function three times.The one-insertion formula is recalled, and the generating function can include the previous layer’s dNTK among its arguments.Source: Interlude 2: Interlayer Correlations… · pp. 309–310

dNTK-Preactivation Cross Correlations

At order 1/n, all dNTK contributions to the joint-distribution statistics are captured by the dNTK’s cross-correlation with a single preactivation.Source: dNTK-Preactivation Cross Correlations · p. 311
At leading non-vanishing order in 1/n, tensors P(ℓ) and Q(ℓ) from the single-preactivation dNTK correlator fix general dNTK–preactivation cross-correlations.Thus, incorporating leading dNTK effects requires evaluating recursions for P(ℓ) and Q(ℓ).Source: dNTK-Preactivation Cross Correlations · pp. 314–315

P-recursion

In deriving the recursion, mean–fluctuation cross terms are suppressed, while leading off-diagonal terms use Gaussian factorization into products of single-neuron expectations.The leading terms have k≠m, and the derivative acts on two of the four activations.Source: P-recursion · pp. 315–316
P(ℓ) mixes with the NTK mean H(ℓ) and variance tensor B(ℓ); since both are O(1) and P(1)=0, P(ℓ) stays O(1) at every layer.Source: P-recursion · p. 316

Q-recursion · Nearly-Linear Quadratic Regression

Together, these parts characterize representation learning in finite-width, nearly-linear settings, describing the Q recursion’s coupling of Q, H, and F and the quadratic model’s parameter-dependent effective features. They also outline perturbative optimization around linear regression and the need for a more complex cubic description for finite-width networks.

Q-recursion

The Q(ℓ) recursion mixes dNTK tensor Q with NTK mean H and cross-correlation F; because H and F are order one and Q(1)=0, Q remains order one.The leading dNTK–preactivation cross-correlation is 1/n-suppressed, so dNTK effects are visible at finite width.Source: Q-recursion · pp. 317–318

11.3.1 Scale-Invariant Universality Class

For scale-invariant activations, P(ℓ) vanishes identically, while Q has critical exponent pQ = −3 and verifies ℓ/n scaling for the leading dNTK–preactivation cross-correlation.This result uses scale-invariant critical initialization, with χ = 1 and K(ℓ) = K⋆.Source: §11.3.1 · pp. 321–322

11.3.2 K⋆= 0 Universality Class

For K⋆= 0, the critical exponents satisfy pP = pQ = 2p⊥−3; the dimensionless P correlation scales as ℓ/n, independently of training hyperparameters, and is negative.Here p⊥ ≤ 1.Source: §11.3.2 · p. 324
Across both universality classes, nonzero dimensionless dNTK–preactivation cross-correlations scale as ℓ/n, making deep finite-width networks representation learners.The leading finite-width effects share the same ℓ/n perturbative cutoff.Source: §11.3 · p. 320, §11.3.2 · p. 324

11.4 Nonlinear Models and Nearly-Kernel Methods

The dNTK captures finite-width effects involving the network function’s second derivative and lets the NTK evolve during training, enabling nontrivial representation learning; the section proposes a minimal model to study this more broadly.The framework is presented as extending the standard machine-learning toolbox.Source: §11.4 · p. 325

11.4.1 Nonlinear Models

The nonlinear-model discussion begins by recalling the linear model as the starting point for extending linear models to nonlinear ones.The book then previews a dual description called nearly-kernel methods.Source: §11.4 · p. 325, §11.4.1 · p. 325
The quadratic model’s parameter-dependent effective features evolve as a linear model, making it a minimal model of representation learning.The meta features play the role of features for the evolving effective features.Source: §11.4.1 · pp. 326–327

Nearly-Linear Quadratic Regression

Because the quadratic loss is quartic and generally lacks an analytical solution, small ϵ permits a perturbative solution around linear regression.The nonlinear terms are proportional to ϵ, yielding weakly interacting optimization dynamics.Source: §11.4.1 · p. 327, Nearly-Linear Quadratic Regression · p. 328
At leading order in the 1/n expansion, finite-width networks cannot be self-consistently described by a truncated quadratic model and instead require cubic models with evolving meta features.This is why quadratic models are treated before the more complicated analysis of finite-width networks with dynamical dNTKs.Source: Nearly-Linear Quadratic Regression · p. 327
In the overparameterized regime, the free linear solution and interacting correction together solve the nonlinear optimization problem to order ϵ.The interacting weights are O(ϵ); the linear solution fits the training examples, and the correction is obtained using a regularized inverse.Source: Nearly-Linear Quadratic Regression · pp. 328–329
The optimized prediction can be written as a linear model over fixed features and learned effective features that depend on the training data.This quadratic model captures the mechanism of feature evolution for a nonzero but fixed dNTK.Source: Nearly-Linear Quadratic Regression · p. 330

Aside: Model Comparison of Linear Regression and Quadratic Regression … Chapter ∞

The section examines inductive bias and representation learning through comparisons of linear and quadratic regression, a sample-space nearly-kernel account of quadratic models, and connections to trained kernels and finite-width networks. It also considers stochastic cubic models of finite-width nonlinear learning dynamics and treats representation learning and deep architecture as distinct, essential aspects of deep learning.

Aside: Model Comparison of Linear Regression and Quadratic Regression

Quadratic regression’s meta-feature constraints add an explicit representation-learning bias: if useful, their small O(ϵ) tunings can reduce overfitting and improve generalization.The models have the same parameter count and can memorize the same training set, so parameter-counting Occam’s razor alone does not distinguish them.Source: Aside: Model Comparison of Linear… · pp. 330–331

11.4.2 Nearly-Kernel Methods

The meta kernel is a parameter-independent, symmetric three-sample tensor built from fixed feature and meta-feature functions, and it enables quadratic predictions in sample space.The resulting nearly-kernel prediction uses sample-space quantities rather than explicit features or model parameters; it is computed by comparison with training examples and includes terms linear and quadratic in their true outputs.Source: §11.4.2 · pp. 333–335
Nearly-kernel predictions depend on the learning algorithm: gradient descent gives an algorithm-specific prediction, while this section continues with direct optimization.Algorithm dependence is expected for nonlinear overparameterized models and is also characteristic of finite-width networks.Source: §11.4.2 · p. 334

Trained-Kernel Prediction

The trained kernel incorporates nontrivial feature evolution into a kernel-form prediction, expressing representation learning in nearly-kernel methods.The effective kernel shifts from the original kernel through a meta-kernel contribution weighted by training outputs, making it output-component dependent.Source: Trained-Kernel Prediction · pp. 335–336

11.4.3 Finite-Width Networks as Nonlinear Models

Unlike the fixed features and non-evolving NTK of the infinite-width linear description, finite-width networks have effective features and kernels that evolve during training.At quadratic order, finite-width networks have nonzero random meta-features sampled at initialization and fixed during training.Source: §11.4.3 · pp. 337–338
The random meta kernel formed from finite-width random features and random meta features is identified with the L-th-layer stochastic dNTK.These feature functions are stochastic, determined by network architecture and randomly sampled initialization parameters.Source: §11.4.3 · p. 339
Unlike the minimal quadratic model, finite-width network predictions wire together different training-output components; feature constraints allow this at finite width but not infinite width.The quadratic model omits wiring by design to simplify the minimal model of representation learning.Source: §11.4.3 · p. 339
For finite-width networks with L > 1, a cubic random meta-feature model gives a consistent description, with meta-features and features evolving effectively as linear and quadratic models.The relevant combinations are all of the same order in ϵ = L/n, so a cubic model is needed for a self-consistent description.Source: §11.4.3 · p. 340, Deep Learning: A Non-Minimal Model of… · p. 340

Deep Learning: A Non-Minimal Model of Representation Learning

The quadratic model demonstrates that nontrivial representation learning can be analyzed separately from depth, with the meta kernel enabling features to be learned from training data.The model permits general feature and meta-feature functions, or equivalently a kernel and meta kernel.Source: Deep Learning: A Non-Minimal Model of… · p. 340
Representation-group-flow recursions encode architecture and hyperparameter effects in kernel statistics, making RG flow the architecture’s inductive bias; the framework also extends to architectures admitting an infinite-width expansion.Different architectures yield different features, kernels, and meta kernels.Source: Deep Learning: A Non-Minimal Model of… · p. 340
The book attributes deep learning’s power to deep representation-group flow working together with finite-width representation learning from nonlinear dynamics.These are presented as two irreducible elements that the theory analyzes in full generality.Source: Deep Learning: A Non-Minimal Model of… · p. 341

The End of Training … B.1 Residual Multilayer Perceptrons

These parts develop a finite-width account of trained neural networks, covering optimization dynamics, algorithm-dependent predictions, and the evolution of kernels and representations. They also examine how depth-to-width ratio, information-theoretic measures, and residual-network design relate to network complexity and effectively deep learning.

Scale-Invariant Universality Class

Taylor expansion in the global learning rate breaks down for nonsmooth activations, including ReLU and other scale-invariant activations.The text instead points to critically tunable smooth activations such as GELU and SWISH.Source: The End of Training · p. 344, Scale-Invariant Universality Class · p. 352

The End of Training

Fully trained finite-width networks depend on the training algorithm, unlike the algorithm-independent infinite-width solution.Their function approximation separates into a universal part and algorithm projectors encoding the optimizer’s influence; the analysis also separates training-dynamics inductive bias from architectural inductive bias and examines depth’s tradeoff between representation learning and fluctuations.Source: The End of Training · p. 344

∞.1 Two More Differentials

Consistent O(1/n) updates require expanding preactivations to third order, the NTK to second order, and the dNTK to first order.The ddNTKs scale as ℓ/n and give the dNTK its own dynamics, so the joint statistics of preactivations, NTK, dNTK, and ddNTKs are needed.Source: The End of Training · p. 344, ∞.1 Two More Differentials · pp. 346–347

ddNTK Statistics

Both ddNTKs have nontrivial order-1/n statistics, so they must be included in finite-width training-dynamics analysis.Their leading-order statistics come from their means; cross-correlations and variances are subleading.Source: ddNTK Statistics · p. 348

ddNTK Scalings

Criticality tuning and dimensionless ratios are used to determine ddNTK depth-scaling exponents; rescaling outputs or training targets can fix the overall output scaling.The text notes that this rescaling is appropriate for regression and discusses how it can affect fully trained test predictions.Source: ddNTK Scalings · pp. 349–350

K⋆= 0 Universality Class

For K⋆ = 0 activations, pR = pS = pT = −2 and pU = 0; R, S, and T contribute at the leading finite-width order.The first ddNTK contributes through R, while the second contributes through S and T.Source: K⋆= 0 Universality Class · pp. 351–352

Scale-Invariant Universality Class

For ReLU, activation threshold crossings can produce η-independent updates; measured NTK updates scale as ∼1/√n with width and linearly with depth ℓ, not the predicted ℓ/n.Because the Taylor expansion in η misses the threshold discontinuity, these methods cannot quantitatively describe finite-width dynamics for nonsmooth scale-invariant activations; smoothed ReLU-like activations remain covered.Source: Scale-Invariant Universality Class · pp. 353–354
For critical deep linear networks, the second ddNTK contributes to dynamics at leading order through S(ℓ) and T(ℓ), whose critical exponents are pS = pT = −4.The first ddNTK does not contribute at O(1/n) in this case.Source: Scale-Invariant Universality Class · pp. 354–355

∞.2 Training at Finite Width

Finite-width fully trained predictions depend on the optimization algorithm, but their solutions take a universal form with algorithm-specific details captured by six projective tensors.The resulting framework describes trained-network distributions for different algorithms, including Newton-like updates and vanilla gradient descent.Source: ∞.2 Training at Finite Width · p. 356

An Infinite-Width Giant Leap at Finite Width

The infinite-width giant leap leaves an order-1/n training error, from NTK fluctuations and nonlinear output changes that reflect representation learning; accounting for both could reduce error to order 1/n^2.The second-order corrections arise from the dNTK and first ddNTK effects, as the network’s effective features evolve.Source: An Infinite-Width Giant Leap at Finite… · p. 359

∞.2.1 A Small Step Following a Giant Leap

The proposed two-update training procedure first takes a giant leap accounting for NTK fluctuations, then a small step accounting for the 1/n representation change from the dNTK and ddNTK.The updates loosely correspond to distinct phases of practical gradient-based neural-network training.Source: ∞.2.1 A Small Step Following a Giant… · p. 360

First Update: One Final Generalization of Gradient Decent

Finite-width NTK fluctuations, dNTK, and ddNTK mix output components, so the generalized second-order update allows off-diagonal Newton-tensor terms to tune that mixing.The mixing means the initial error in one output component can affect a different component after the update.Source: First Update: One Final Generalization… · pp. 360–361
The first generalized Newton update accounts for network-specific stochastic NTK fluctuations, reducing the initial order-one training error while leaving finite-width representation-learning corrections.The remaining error is attributed to the dNTK and first ddNTK.Source: First Update: One Final Generalization… · pp. 362–363

Second Update: Representation Learning Strikes Back

A second, smaller update reduces residual training error by another factor of 1/n and expresses the fully trained output as a functional of initialization variables.The second update accounts for representation evolution; the resulting network is fully trained on all training inputs.Source: Second Update: Representation Learning… · pp. 364–365

∞.2.2 Many Many Steps of Gradient Descent

Many-step dynamics couple nonlinear output and NTK difference equations; the proposed solution first solves the dNTK/ddNTK-free case, then adds finite-width effects by perturbation theory.The single-update alternative combines stochastic NTK inversion with dNTK- and ddNTK-induced representation learning, but requires a finely tuned generalized Newton tensor.Source: ∞.2.2 Many Many Steps of Gradient… · pp. 366–367

Free Theory: Step-Independent NTK

With dNTK and ddNTK set to zero, the NTK remains at initialization, and for a positive-definite NTK with sufficiently small learning rate, training error vanishes exponentially and general-input predictions converge.The NTK sets the decay scale of individual error components.Source: Free Theory: Step-Independent NTK · pp. 367–369
With the dNTK and ddNTKs turned off, the free-theory solution matches the first Newton-tensor update, so finite-width algorithm dependence comes solely from NTK differentials and nonlinear dynamics.For T ≳ (η bH)^-1, prediction error is exponentially close to its T → ∞ value.Source: Free Theory: Step-Independent NTK · p. 370

Interacting Theory: Dynamical NTK and dNTK

At leading order in 1/n, interaction-output products with NTK differentials are negligible, enabling a systematic perturbative solution of the nonlinear dynamics.The interacting output and NTK corrections are organized in terms of the nonzero NTK differentials.Source: Interacting Theory: Dynamical NTK and… · p. 370
Larger training-error reduction drives greater dNTK evolution and meta-feature learning; larger NTK change likewise means greater feature evolution and representation learning.The text also states that more initial error entails more representation learning over training.Source: Interacting Theory: Dynamical NTK and… · pp. 371, 374
Gradient descent converges exponentially quickly to a zero-error minimum for finite-width, nonzero-depth realistic networks, up to errors at most quadratic in L/n.The interacting solution converges slightly more slowly than the free solution; the result is understood up to O(1/n^2) corrections under the fully-trained condition.Source: Interacting Theory: Dynamical NTK and… · pp. 376–377
In the ODE limit, gradient flow and gradient descent have different algorithm projectors, capturing their difference for fully trained finite-width networks.The ODE limit is obtained by taking the learning rate η to zero.Source: Interacting Theory: Dynamical NTK and… · pp. 381–382
At leading order, the fully trained finite-width solution is conjectured to have a universal form whose six algorithm projectors encode optimization-algorithm dependence.The projectors act on combinations of the dNTK and first and second ddNTKs; the solution recovers the fully trained condition on training-set inputs.Source: Interacting Theory: Dynamical NTK and… · pp. 381–382
Finite learning-rate gradient descent shares the same finite-width prediction form, with its algorithm dependence encoded by projectors; the effective description is about 1% off for L = 10 and n = 100.The stated 1% is the error in the effective description of a particular fully trained network, not the training error of that network.Source: Interacting Theory: Dynamical NTK and… · p. 382, ∞.2.3 Prediction at Finite Width · p. 384

∞.2.3 Prediction at Finite Width

The mean prediction depends on both NTK differentials and NTK variance, while finite-width output wiring lets one observed component influence another component’s mean prediction.Wiring also appears in the covariance: initialization fluctuations break output-neuron permutation symmetry and can correlate test predictions across components.Source: ∞.2.3 Prediction at Finite Width · pp. 386–387
Finite-width predictions have non-Gaussian statistics, with higher-point connected correlators arising from a nonlinear functional of initialization outputs and NTK, dNTK, and ddNTK quantities.The joint distribution of those initialized quantities is nearly Gaussian, but the fully trained prediction need not be.Source: ∞.2.3 Prediction at Finite Width · p. 387

Generalization at Finite Width

Leading finite-width corrections to generalization scale as O(L/n), with the same order expected for both bias and variance contributions.The analysis confirms this scaling for the bias correction and expects the covariance-based variance correction to have the same order.Source: Generalization at Finite Width · pp. 389–390
The optimal depth-to-width ratio is expected to be small but nonzero: depth enhances representation learning, while large L/n can undermine typicality and produce exponential generalization behavior.The authors characterize O(L/n) theory as simple enough to describe practical networks, unlike the infinite-width limit or the small-width, overly-deep regime.Source: Generalization at Finite Width · p. 390
The end-of-training prediction formula cannot optimize L/n at the stated order because generalization depends linearly on the ratio; fluctuations can affect ensemble bias and individual-network criticality.At order O(L/n), the formula can at most indicate whether nonzero L/n helps via the sign of its overall coefficient.Source: Generalization at Finite Width · pp. 390–391

Inductive Bias of the Training Algorithm

Algorithm projectors isolate training-algorithm effects in finite-width predictions: they adjust NTK-differential contributions, while NTK-fluctuation contributions remain irreducible.The differential contribution enables nontrivial representation learning, whereas fluctuations can disrupt criticality; projectors offer a way to enhance the former while holding the latter cost fixed.Source: Inductive Bias of the Training Algorithm · pp. 391–392

There’s No Place Where Gradient Descent = Exact Bayesian Inference

At finite width, gradient-based learning does not match exact Bayesian inference: the matching hyperparameters disable hidden-layer learning, eliminating dNTK and ddNTK effects and representation learning.Those settings reduce the model to fixed random features, whereas exact Bayesian inference at finite width exhibits nontrivial representation learning.Source: There’s No Place Where Gradient… · p. 393

∞.3 RG Flow of the ddNTKs: The Full Expressions; \ ddIH Recursion; \ ddIIH Recursions

The ddNTK appendix gives full expressions for checking finite-width Bayesian mismatch, depth-to-width scaling, and ddNTK contributions to mean predictions or higher statistics.Its recursions specify R for the first ddNTK and S, T, and U for the second ddNTK.Source: ∞.3 RG Flow of the ddNTKs: The Full… · pp. 393–394, \ ddIH Recursion · pp. 395–396, \ ddIIH Recursions · p. 396

Model Complexity from the Macroscopic Perspective

The depth-to-width ratio controls model complexity through the number of data-dependent couplings in the truncated nearly-Gaussian distribution, rather than the number of parameters.The ratio also acts as the effective-theory expansion parameter or cutoff, allowing sparse intuition and overparameterized models’ practical success to coexist.Source: Model Complexity from the Macroscopic… · p. 400

Sparsity at Infinite Width

At infinite width, higher derivatives vanish, leaving only the output and first derivative; the trained distribution is Gaussian and its solution is independent of the training algorithm.Its sparse description depends on the training labels, output-layer kernel, and frozen NTK.Source: Sparsity at Infinite Width · pp. 401–402

Near-Sparsity at Finite Width

At large but finite width, derivatives of order four and higher are O(1/n^2), so near-sparse descriptions retain statistical variables through the third derivative.The retained variables include the output, NTK, dNTK, and ddNTKs.Source: Near-Sparsity at Finite Width · p. 402
At finite width, trained-network predictions remain nearly Gaussian and nearly sparse: O(1/n) data-dependent couplings involve exactly four input samples, while algorithm dependence enters through a handful of projectors.The conditional description includes finite-width data-dependent couplings, whose functional forms determine the distribution’s overall data dependence.Source: Near-Sparsity at Finite Width · p. 403

Model Complexity of Fully-Trained Neural Networks

In overparameterized networks, model complexity is better identified with data-dependent couplings than parameter count, and the macroscopic description is simplest at small depth-to-width ratios.The authors characterize this as a microscopic-macroscopic duality: parameter-space complexity becomes sample-space structure, with density exchanged for sparsity.Source: Model Complexity of Fully-Trained… · pp. 404, 407
Accuracy of order O(L^2/n^2) requires six-sample-index 1/n^2 tensors, illustrating how higher-order effective theories increase sample-space complexity.The text expects a general O(L^k/n^k) description to be dominated by couplings with 2k sample indices.Source: Model Complexity of Fully-Trained… · pp. 405–406
The depth-to-width ratio distinguishes limits: r→0 gives an infinite-width Gaussian description without representation learning, while 0<r≪1 supports wide, deep networks that learn representations.The latter regime is described by a nearly-Gaussian finite-width theory truncated at order 1/n.Source: Model Complexity of Fully-Trained… · p. 406

Information in Deep Learning

A maximum-entropy variational principle computes entropies and information through order 1/n^3 using preactivation distributions truncated at order 1/n; at order 1/n^2, neuron-group mutual information grows quadratically with depth.This gives an information-theoretic interpretation and generalization of the book’s Hebbian learning result.Source: Information in Deep Learning · p. 410
At order 1/n^3, maximizing mutual information yields an activation-function-dependent optimal ratio r⋆ that separates effectively-deep networks from overly-deep networks that are no longer trainable.The optimal ratio arises from an unsupervised learning objective.Source: Information in Deep Learning · p. 410
A three-group mutual-information generalization indicates that information is stored redundantly across neurons, clarifying how RG-flow coarse-graining represents and shares input information in deeper layers.Source: Information in Deep Learning · p. 410

A.1 Entropy and Mutual Information

Entropy is a property of an entire probability distribution: it is minimal for certainty and maximal for a uniform distribution, where it equals the logarithm of the number of possible outcomes.The section introduces entropy and mutual information as shared concepts in statistical mechanics and information theory.Source: §A.1 · p. 411
The maximum-entropy principle assigns equal a priori probabilities when no other information about a system is available.When information about the state is available, the entropy need not be maximized by a uniform distribution.Source: §A.1 · p. 411
Entropy is additive for independent variables and subadditive for dependent variables, reflecting how statistical dependence constrains their joint information.The text connects this subadditivity to nonnegative mutual information.Source: §A.1 · pp. 412–414
Mutual information measures how much observing one variable informs us about another, is nonnegative, and is zero when the variables are statistically independent.It is the KL divergence between the joint distribution and the product of the marginal distributions.Source: §A.1 · pp. 414–415
Continuous mutual information is invariant under coordinate transformations and independent of the precision cutoff, unlike continuous entropy, which depends on the measurement units.With a consistent, fixed choice of units, entropy can be used as an intermediate step when computing mutual information.Source: §A.1 · pp. 416–417
Tripartite information is symmetric in its three variables and can be positive or negative, indicating redundant or synergistic distribution of information.Positive values indicate redundancy; negative values indicate synergy.Source: §A.1 · p. 418

A.2 Information at Infinite Width: Criticality

At infinite width, neural-network preactivations are described as zero-mean Gaussian variables.The passage introduces this distribution as the starting point for applying information-theoretic definitions to the neural-network prior.Source: §A.2 · p. 418
At infinite width, entropy is extensive in neuron count and mutual information between distinct neuron sets is zero, indicating independent neurons.Learning the activities of some neurons therefore conveys no information about the others in the same layer.Source: §A.2 · p. 419
For a fixed neuron, input mutual information is governed by K[2]/K[0]; setting χ∥=1 and χ⊥=1 avoids extremes that hinder learning same-class or different-class inputs.The authors frame this balance as an information-theoretic perspective on criticality.Source: §A.2 · p. 420

A.3 Information at Finite Width: Optimal Aspect Ratio

At finite width, nonzero neuron mutual information motivates an optimal critical depth-to-width ratio r⋆, separating effectively deep from overly deep networks.The latter are not described by the effective theory and are not trainable due to large fluctuations.Source: §A.3 · pp. 420–421
A maximum-entropy variational action fixes the exact two- and four-point correlators, cancels the linear deformation term, and yields a negative leading correction away from Gaussianity.The Gaussian term is the entropy for the same covariance, and the quadratic correction is negative because Gaussianity maximizes entropy at fixed first and second moments.Source: §A.3 · pp. 422, 424

Leading-Order Correction: Nonzero Mutual Information

The leading entropy correction is negative and scales as ∼V^2/n^2; its quadratic dependence on neuron count m breaks additivity and signals interactions.The correction must be negative because the Gaussian maximizes entropy at fixed covariance.Source: Leading-Order Correction: Nonzero… · p. 426
At finite width, observing one group of neurons conveys information about another non-overlapping group, yielding nonzero mutual information.The authors describe this as an information-theoretic reformulation and generalization of the Hebbian learning principle.Source: Leading-Order Correction: Nonzero… · p. 426
Leading-order mutual information scales quadratically with hidden-layer depth and is relevant under RG flow, reducing the independently available degrees of freedom in deeper layers.Source: Leading-Order Correction: Nonzero… · pp. 426–427

NLO Correction: Optimal Aspect Ratio and Tripartite Information

For small depth-to-width ratios, mutual information increases with depth, but its next-to-leading correction yields a local maximum at a nonzero, not-too-large ratio.The optimal aspect ratio marks a scale separating the effectively-deep regime from an overly-deep regime where the theory is strongly coupled and networks are no longer trainable.Source: NLO Correction: Optimal Aspect Ratio… · pp. 430–432
For equal mutual information in a layer, networks with tanh activations should be deeper than networks with ReLU activations.Source: NLO Correction: Optimal Aspect Ratio… · p. 431
Tripartite information scales cubically with depth and is exclusively positive, indicating redundant representations among any three neuron groups under RG flow.A nonzero result first appears at O(1/n^3).Source: NLO Correction: Optimal Aspect Ratio… · p. 432

Residual Learning

Degradation makes deeper networks empirically prone to higher training and test errors, because optimization struggles to tune their hidden layers to near-identity maps.Source: Residual Learning · p. 433
He et al. use residual connections so layers learn a residual function, which is easier to set near zero than to make a generic nonlinear layer approximate identity; this can support deeper models and improve test performance.The preactivation may both coarse-grain the input through RG flow and propagate an undegraded copy of it.Source: Residual Learning · p. 434
The analysis proceeds from infinite-width criticality to finite-width behavior, then offers a hybrid theoretical-empirical recipe for general residual networks with complex blocks.The appendix also demonstrates how effective theory can apply beyond vanilla MLPs.Source: Residual Learning · p. 436
Finite-width analysis predicts residual connections can shift optimal aspect ratio r⋆ upward, but are harmful below a threshold under the auxiliary unsupervised criterion, balancing nonlinear-block against identity propagation.This offers a macroscopic perspective on how residual connections address the degradation problem.Source: Residual Learning · p. 436

B.1 Residual Multilayer Perceptrons

The residual MLP uses a standard MLP layer as its residual block, with scalar hyperparameters ξ(ℓ) and γ(ℓ) controlling residual-block and identity-term magnitudes; connections are included only in hidden layers.Source: §B.1 · pp. 436–437
ξ(ℓ) is redundant because its effect can be reproduced by rescaling initialization hyperparameters, leaving γ(ℓ); γ(ℓ) → 0 recovers a vanilla MLP, while γ(ℓ) = 1 breaks criticality.Source: §B.1 · p. 437

B.2 Residual Infinite Width: Criticality Analysis … Index

This section analyzes critical initialization in residual networks, including how critical solutions relate to optimal depth-to-width ratios and how theory can be combined with four-point connected-correlator measurements to estimate them. It concludes with an index of topics and page references, including residual networks, RG flow and criticality, non-Gaussian distributions, and model complexity.

B.2 Residual Infinite Width: Criticality Analysis

For either universality class, criticality requires a weight variance C_W(γ) for each 0 < γ² < 1; the customary γ = 1 choice cannot satisfy the criticality conditions.Residual connections shift the critical initialization conditions.Source: §B.2 · p. 439

B.3 Residual Finite Width: Optimal Aspect Ratio

Finite-width optimal aspect ratios are estimated using initialization mutual information, since infinite-width networks have r = L/n → 0 and cannot capture depth-related degradation.The analysis motivates this criterion by proposing that residual connections tuned to criticality should shift optimal aspect ratios higher.Source: §B.3 · pp. 439–440

Scale-Invariant Universality Class

In the scale-invariant class, ν(γ) is strictly positive for 0 < γ² < 1 and decreases monotonically to ν(1) = 0 as γ increases.It reduces to the vanilla-MLP result as γ → 0.Source: Scale-Invariant Universality Class · p. 442

K⋆= 0 Universality Class

In the K⋆ = 0 class, ν(γ) is strictly positive for 0 < γ² < 1 and decreases monotonically to ν(1) = 0, qualitatively matching the scale-invariant class.The result reduces to the vanilla K⋆ = 0 MLP result as γ → 0.Source: K⋆= 0 Universality Class · p. 442

Physics of the Optimal Aspect Ratio

Because ν(γ) decreases with γ in both classes, the predicted optimal aspect ratio r⋆(γ) increases with γ, supporting residual connections as a way to address degradation.For r > r⋆(γ = 0), the optimal residual hyperparameter approaches 1 as depth-to-width ratio grows; maintaining criticality requires adjusting C_W accordingly.Source: Physics of the Optimal Aspect Ratio · pp. 443–444

B.4 Residual Building Blocks

For general residual blocks, measure input and output two-point correlators, tune initialization to keep their magnitudes comparable, then solve the correlator recursion for residual hyperparameters.For internally deep blocks, avoid exponential growth or decay; the framework applies schematically to ResNets and transformers, and tunes ξ after setting block initialization for internal criticality.Source: §B.4 · pp. 444–446
Each critical solution (γ, ξ) yields a different optimal aspect ratio r⋆(γ, ξ), estimable by combining §B.3 theory with four-point connected-correlator measurements.The measurements use an appropriate combination of four-point connected correlators.Source: §B.4 · p. 446

Index

The index lists residual connections (pages 425–427 and 430), RG flow (pages 101 and 123), criticality, and the gradient problem (pages 239–241).Source: Index · pp. 465, 468, 469, 471
The index also catalogs nearly-kernel methods, effective kernels, trained kernels, and their relation to dynamical NTK.These entries point to discussions on pages 325–328, including the effective kernel and its relation to dynamical NTK.Source: Index · p. 466
The index lists a non-Gaussian distribution at page 421 and an optimal aspect ratio entry that points readers to effectively deep; it also indexes model complexity at pages 164, 320, 390, and 391.Source: Index · pp. 465–467