Dynamically Stable Infinite-Width Limits of Neural Classifiers

1 Introduction

• The framework derives dynamically stable scalings for existing MF and NTK limits, the intermediate limit, and other infinite-width models.Source: §1 · p. 1
• Only 13 distinct stable model evolution equations arise from hyperparameter scalings, despite the continuous space of possible scalings.Source: §1 · p. 1
• MF, NTK, and sym-default limits satisfy most finite-width properties, but none generally satisfies all of them simultaneously.Source: §1 · p. 1, Figure 1 · p. 2
• IC-MF modifies the model so its infinite-width limit satisfies all listed finite-width properties.Source: §1 · p. 2, Figure 1 · p. 2
• IC-MF is reported as the best limit-model proxy for finite-width training dynamics among the possible limits.Source: §1 · p. 2
• The analysis is restricted to single-hidden-layer networks, with deep-network generalization discussed only as a future plan.Source: §1 · p. 2

2 Training a one hidden layer net with SGD

• Power-law scalings of initialization variances and learning rates induce limit models, and 13 subclasses correspond to distinct stable evolution equations.Source: §2 · pp. 2, 4
• Dynamic stability requires single-step logit changes to remain comparable to logits, allowing learning to persist in the infinite-width limit.Source: §2 · p. 3
• qσ + ˜q ∈ [-1/2, 0] is necessary for dynamic stability when the learning-rate exponents are equal.Source: §2 · p. 4
• The tangent-kernel definition incorporates learning-rate growth factors to avoid 0 × ∞ ambiguity under mean-field scaling.Source: §2 · p. 4
• NTK kernels remain stationary at infinite width, whereas mean-field kernels evolve because the weights continue evolving in the limit.Source: §2 · p. 4
• Four conditions distinguish limits by finite initialization logits, finite initialization kernels, matching kernel-logit orders, and evolving tangent kernels.Source: §2 · pp. 4–5
• The four conditions partition the stability band into 13 regions, each producing one distinct limit evolution; intermediate scalings from [8] form one such region.Source: §2 · p. 5

3 Capturing the behavior of finite-width nets

• Finite-width networks typically have finite initial logits and tangent kernels, while their tangent kernels evolve substantially during training.These properties motivate evaluating limit models by how many finite-width conditions they satisfy.Source: §3 · p. 5

3.1 "Non-dominated" limit models: MF, NTK and "sym-default"

• The paper discusses three non-dominated limit models, represented by exactly three distinct zero-dimensional regions (points) in Figure 1; one corresponding point is the NTK case.For NTK, qσ = −1/2 and ˜q = 0, with σ ∝d−1/2 and ˆη = const. For sym-default, qσ = −1/2 and ˜q = 1/2.Source: §3.1 · pp. 5–6
• The mean-field limit has zero initialization logits because its initial parameter-space measure has zero mean, violating finite-width properties concerning initial logits and kernel behavior.Its tangent-kernel dynamics are represented through evolution of a weight-space measure rather than a closed kernel equation.Source: §3.1 · pp. 5–6

3.2 Initialization-corrected mean-field (IC-MF) limit

• IC-MF adds a finite, training-independent initialization bias to mean-field dynamics, preserving finite initial logits while retaining evolving tangent kernels.The measure evolves with training, so the tangent kernels evolve as in the mean-field case.Source: §3.2 · p. 7

3.3 Experiments

• The NTK limit tracks the reference well only during the first 20 training steps, whereas the mean-field limit becomes less divergent after the 80th step.The mean-field limit begins with high divergence because its initial limit model is zero; default scaling instead produces diverging logits and KL divergence.Source: §3.3 · p. 7
• IC-MF retains small KL-divergence from the finite-width reference throughout training, outperforming the other tested limits as a proxy for finite-width networks.The authors contrast this simple modification with more complicated finite-width corrections for NTK.Source: §3.3 · p. 7

4 Related work

• NTK limits formalize lazy training, but the laziness assumption has limitations for explaining deep-learning success.Rich training instead allows the tangent kernel to evolve significantly.Source: §4 · p. 7
• Mean-field limits describe deterministic weight-space-measure evolution, whereas NTK limits use a constant kernel under an appropriate parameterization.The corresponding parameter scalings differ between the two literatures.Source: §4 · p. 7
• In a one-hidden-layer CIFAR2 experiment, IC-MF is reported to capture a finite-width reference network better than other infinite-width limits.The comparison uses KL-divergence between logits of each limit and the reference network.Source: Figure 2 · p. 8
• The proposed framework differs from prior scaling analyses by working directly with logits and kernels and by using a weaker stability criterion suited to classification.It also includes default, sym-default, and IC-MF limits not previously discussed in the same framework.Source: §4 · p. 8

5 Conclusions

• A weaker dynamical-stability requirement produces a richer class of limit models, including a default finite-learning-rate limit that does not satisfy the earlier well-definiteness condition [8].The paper studies how width-dependent hyperparameter scaling controls convergence to dynamically stable training dynamics.Source: §5 · p. 8
• The proposed modification yields a limit model satisfying all stated finite-width conditions and tracking a reference finite-width network better than other limits.These conditions include bounded initialization quantities and evolving tangent kernels.Source: §5 · p. 8, A Formal conditions for Section 2 · p. 11
• Hyperparameter-scaling space is partitioned into regions of training-dynamics conditions, with each region corresponding to one limit model.Thus, infinitely many or uncountably many scalings can induce only finitely many distinct limit models.Source: §5 · p. 8

A Formal conditions for Section 2

• Formal stability constraints depend on scaling exponents and, in general, on the data distribution and proportionality factors.A data- and hyperparameter-independent condition is obtained by requiring the stability condition to hold for arbitrary terminal-kernel and proportionality-factor values.Source: A Formal conditions for Section 2 · p. 11
• The formal conditions require finite initialization limits and tangent kernels, comparable initial orders, and nonstationary tangent kernels.These conditions separate the dynamically stable band considered in the paper.Source: A Formal conditions for Section 2 · p. 11

B Proofs of propositions

• The propositions assume a real-analytic asymptotically linear nonlinearity, binary cross-entropy loss, data sampled from D, and power-law hyperparameter scaling.The training dynamics are expressed through the specified loss and width-dependent parameterization.Source: B Proofs of propositions · pp. 11–12

B.1 Proof of Proposition 1

• For equal parameter exponents, model orders remain zero when qσ + ˜q ≤ 0, but grow as k(qσ + ˜q) with positive probability when qσ + ˜q > 0.This dichotomy is established under a continuous data distribution and kterm = +∞.Source: §B.1 · p. 14
• Condition 4 requires qσ + ˜q ∈ [−1/2, 0] when ˜qa = ˜qw = ˜q and the data distribution is continuous.Positive qσ + ˜q is excluded, while the lower bound follows from the model-order constraints.Source: §B.1 · p. 14

B.2 Proof of Proposition 2

• Finite initialization logits, finite tangent kernels, matching orders, and evolving kernels correspond respectively to linear constraints on qσ and ˜q.The stated equivalences include qσ + 1/2 = 0, 2qσ + ˜q + 1 = 0, qσ + ˜q + 1/2 = 0, and qσ + ˜q = 0.Source: §B.2 · pp. 14–15

C The number of distinct limit models is finite

• With fixed proportionality factors, choosing a limit evolution is equivalent to selecting one of 13 regions determined by the signs of the relevant exponents.Eight arise for constant normalized kernels, while five arise when normalized kernels are non-stationary.Source: C The number of distinct limit models… · p. 15, §C.1 · p. 17

C.1 Constant normalized kernels case

• In the constant-normalized-kernel case, the limit dynamics is determined by proportionality factors and the signs of three exponents, including logit and kernel orders.The NTK model is the special scaling qσ = −1/2 and ˜q = 0.Source: §C.1 · p. 17, §C.1.1 · p. 17

C.2 Non-stationary normalized kernels case

• Non-stationary normalized kernels can evolve at infinite width, and the resulting dynamics is closed after taking the width limit.The limiting equations are derived for this regime under fixed proportionality factors.Source: §C.2 · pp. 18–19
• Following the constant-kernel derivation directly can produce a non-closed limit dynamics because the kernels are non-stationary.The workaround expresses the evolution through a weight-space measure.Source: §C.2 · p. 18
• Five non-stationary-kernel limit dynamics combine with eight constant-kernel dynamics to produce 13 distinct limit dynamics.The classification depends on the signs of 1/2 + qσ and 1 + qσ in this case.Source: §C.2 · p. 19

C.2.1 MF limit model

• The mean-field scaling is qσ = −1 and ˜q = 1, while the sym-default scaling is qσ = −1/2 and ˜q = 1/2.Source: §C.2.1 · p. 19, §C.2.2 · p. 19

• The default scaling is represented by qσ = −1/2, ˜qa = 1, and ˜qw = 0.Source: D Default scaling · p. 20
• The default limit dynamics differs from sym-default only through its initial measure.Source: D Default scaling · p. 21
• At initialization, the output remains finite, but the tangent kernel violates boundedness and stationarity conditions.Source: D Default scaling · p. 21
• Kernel increments also scale as Θd→∞(1), violating the final condition for dynamically stable finite-width behavior.Source: D Default scaling · p. 21

• The initialization-corrected construction uses the same training dynamics as mean-field scaling with a modified model definition.Source: E Initialization-corrected mean-field… · p. 21
• As in the mean-field case, the dynamics is rewritten using a weight-space measure before taking the infinite-width limit.Source: E Initialization-corrected mean-field… · pp. 21–22
• Its finite-width logits remain finite for every k ≥ 0 as d →∞, allowing the limit to be taken directly.Source: E Initialization-corrected mean-field… · p. 22

• The experiments use a bias-free single-hidden-layer network trained as a binary classifier on 1,024 CIFAR2 examples, evaluated on 2,000 test examples.Source: F Experimental details · p. 22
• Networks are trained for 2,000 steps with full-batch gradient descent and binary cross-entropy, using 10 random seeds for reported means and deviations.Source: F Experimental details · p. 22
• Additional setups vary minibatch size, training-set size, and task dimensionality, while experiments use PyTorch on one NVIDIA GeForce GTX 1080 Ti GPU.Source: F Experimental details · p. 22
• The Gaussian initialization assumption can be replaced by σξ for any symmetric width-independent random variable ξ without changing the analysis.Source: F Experimental details · p. 22
• The reference network has width d∗ = 128, Kaiming He uniform initialization, and output-layer standard deviation proportional to (d∗)^−1/2.Source: F Experimental details · p. 22
• Reference learning rates and scaled initial parameters are adjusted according to each candidate power-law scaling.Source: F Experimental details · p. 22

• Changing batch size and training-set size produces marginal, non-qualitative differences in the experimental results.Source: G Experiments for other setups · p. 23
• Figure 3 compares test accuracy for different limit models and the reference model across CIFAR2 subset sizes and varying SGD batch sizes.Source: Figure 3 · p. 23

H Generalization to deep nets proposal

• The proposed deep-network analysis seeks regions of dynamically stable scalings whose distinct limit models satisfy finite initialization, comparable model and kernel scales, and kernel evolution.The authors expect separate conditions for hidden kernels and input-plus-output kernels.Source: H Generalization to deep nets proposal · pp. 23–24
• Deep-network scalings use 2H + 1 hyperparameters, so dimensionality reduction is needed to make the scaling space tractable.The hyperparameters are H + 1 learning rates and H + 1 initialization variances, with input variance fixed to 1.Source: H Generalization to deep nets proposal · p. 23
• Using a homogeneous leaky-ReLU activation reduces the relevant initialization dependence to the product σH × . . . × σ0, but sacrifices smoothness and mathematical rigor.This reduces the number of hyperparameters from 2H + 1 to H + 1.Source: H Generalization to deep nets proposal · p. 23
• Equal learning-rate exponents yield a lower-dimensional scaling family containing the NTK case, while a separate hidden-layer exponent preserves a broader three-dimensional family.The alternatives include a common exponent ˜q or distinct input/output and hidden-layer exponents (˜q, ˜qhid).Source: H Generalization to deep nets proposal · p. 24
• For deep networks, the mean-field limit reportedly vanishes when H > 2, motivating separate analyses for H = 2 and H > 2.The paper also identifies finding a non-dominated limit model beyond both NTK and mean-field formalisms as a valuable research direction.Source: H Generalization to deep nets proposal · pp. 24–25