Tensor Programs II: Neural Tangent Kernel for Any Architecture

• Earlier work showed that large-width gradient-descent training evolves like a linear model under kernel gradient descent with a constant limiting NTK [39].The NTK is defined as the inner product of parameter gradients and converges to a deterministic kernel under suitable random initialization [39].Source: §1 · p. 1
• By establishing a universal Neural Network–Tangent Kernel correspondence, the paper opens a route to studying inductive biases for networks of varied architectures trained under SGD.The motivation is to extend NTK analysis beyond modern architectures such as Faster R-CNNs, transformers, and generative adversarial networks [56] [59] [25].Source: §1 · pp. 1–2
• For any randomly initialized neural network of standard architecture, the NTK converges almost surely to a deterministic limit as widths tend to infinity, which the paper computes exactly.The scope includes compositions of MLPs, RNNs, skip connections, convolutions, graph convolutions, pooling, normalization, and attention.Source: §1 · pp. 1–2
• The Simple GIA Check rigorously justifies the Gradient Independence Assumption, while violating it can make GIA-based calculations incorrect.GIA treats W^⊤ in backpropagation as independent from W in forward propagation.Source: §1 · p. 2
• The paper gives concrete infinite-width NTK algorithms for batchnorm-ReLU MLPs, transformers, and RNNs, and verifies agreement with simulations across widths 2^6 through 2^13.The reported deviations use 95% confidence intervals for means over 100 seeds.Source: §1 · p. 2
• The results extend tensor programs with NETSOR^⊤, adding matrix transposes so forward propagation and backpropagation can be expressed and infinite-width gradients analyzed.This enables reasoning about NTKs, which NETSOR alone cannot provide.Source: §1 · p. 2

Infinite-width NTK

• As widths tend to infinity, the NTK converges to a deterministic limit, and neural-network features become deterministic despite random parameters [39].The convergence is taken over the randomness from Gaussian-style parameter initialization.Source: §2 · pp. 2–3

NTK training

• For fixed training time, infinite-width gradient descent follows kernel gradient descent with the fixed limiting NTK [39].Under square loss, the limiting dynamics become a linear differential equation whose solution decays exponentially according to the NTK spectrum.Source: §2 · p. 3
• Because the limiting NTK is generally nonsingular, sufficiently wide networks can fit arbitrary training data [39].This explains how a small parameter change can still produce substantial function change in high-dimensional parameter spaces.Source: §2 · pp. 2–3

Motivation and scope

• The NTK acts as an implicit prior determined by architecture and initialization, with its spectrum indicating which functions gradient descent learns quickly and generalizes well.The paper extends this perspective by presenting a general rigorous procedure for computing infinite-width NTKs beyond MLPs.Source: §2 · p. 3

• NN-GP associates a Gaussian-process kernel with wide networks, while NTK describes full-network gradient-descent training and the infinite-width GP describes last-layer training [16] [30] [44] [50] [52].Source: §3 · p. 3
• The original work established architectural universality for NTK and NN-GP [63], while Yang [62] simplified and generalized the GP presentation; this paper similarly simplifies the NTK results.Source: §3 · p. 3
• Subsequent studies examined specialized NTKs for convolutions, graphs, hypernetworks, RNNs, attention, and orthogonal initialization, alongside higher-order terms, ensembles, and finite-width corrections [6].Source: §3 · p. 3
• Signal-propagation research studies pathological behavior in deep random networks, with forward propagation corresponding to infinite-depth GP limits and backward propagation to infinite-depth NTK limits [13] [26] [28] [29] [53] [54] [55] [58] [68].Source: §3 · p. 3
• NTK theory addresses why highly nonconvex neural-network training works well [4] [5] [18] [39] [69], supports deep-reinforcement-learning convergence studies, and shows better low-data performance for corresponding kernel regression.NTK spectra provide finer-grained analyses [10] [24], while low-data benefits may matter for settings such as medical decision-making.Source: §3 · p. 3

4 Warmup: Neural Tangent Kernel for a Multi-Layer Perceptron

• The warmup uses an MLP with recursively defined hidden states and the NTK parametrization, where each weight matrix is scaled by width.The network is defined by h_l = W_l x_{l−1} + b_l and x_l = φ(h_l), with scalar output dimension.Source: §4 · pp. 3–4

4.1 Decomposing NTK

• The MLP NTK is decomposed into weight and bias contributions, and convergence follows by showing each summand converges after factoring gradient terms.The framework uses an explicit sum of products whose limits can be evaluated, rather than relying only on depth-inductive reasoning.Source: §4.1 · p. 4, §4 · p. 4

4.2 Limits of Forward Quantities xl⊤¯xl/nl

• Forward inner products x_l(ξ)^⊤x_l(¯ξ)/n_l converge to deterministic scalars C_l(ξ, ¯ξ) through central-limit reasoning and Gaussian activation recursions.Each preactivation coordinate is modeled as a jointly Gaussian pair, after which applying φ yields the forward covariance recursion, including the bias contribution.Source: §4.2 · p. 4

4.3 Limits of Backward Quantities dhl⊤d¯hl/nl

• Under GIA, backward quantities converge through Gaussian, approximately independent coordinates to a deterministic recurrence for D_l(ξ, ¯ξ).GIA treats W^⊤ in backpropagation as independent from W in the forward pass, and the resulting calculation agrees with simulation when widths are large.Source: §4.3 · p. 5
• Combining the forward and backward recurrences recovers the NTK limit formula in Jacot et al. [39].The backward limit is represented by the deterministic scalar D_l(ξ, ¯ξ), satisfying a recurrence involving E φ′(ζ)φ′(¯ζ).Source: §4.3 · p. 5

• The proposal is motivated by the greater generalizability of the preceding MLP NTK method compared with Jacot et al. [39], while acknowledging unresolved technical issues.These issues concern whether the NTK decomposition remains meaningful under tied weights, whether correlated forward and backward weights permit GIA, and whether diverse modern architectures can be handled systematically.Source: §5 · p. 5
• The NTK can be generalized into sums of inner-product products, with each term converging to a deterministic finite constant and hence the NTK converging.This extends the decomposition beyond the MLP setting, including cases with additional correlations such as weight tying; see Eqs. (10) and (12).Source: §5 · pp. 5–6
• GIA holds conditionally under Simple GIA Check: the output layer must be independently, zero-mean sampled from all other parameters and unused elsewhere in the network.The condition works because interactions between a weight matrix and its transpose must pass through the independently sampled, zero-mean output weights; violating it can make GIA fail.Source: §5 · pp. 5–6
• NETSOR⊤ provides a unified language for matrix multiplications and nonlinearities, allowing standard architectures to receive provably convergent NTKs under mild regularity conditions.Infinite-width NTKs can be computed mechanically by recursively applying the Master Theorem, provided Condition 1 holds and forward and backward computations are expressible in NETSOR⊤.Source: §5 · pp. 5–6

6.1 The Canonical Decomposition

• The NTK is computed by summing contributions from factorized weights and biases, whose intermediate kernel terms converge almost surely to deterministic infinite-width limits.This strategy extends the earlier NTK calculation framework to general architectures.Source: §6.1 · pp. 6–7

6.2 Intuitive Rules for Computing Intermediate Kernels C and D

• For wide networks, roughly iid coordinate variables and recursive Nonlin and MatMul rules determine the forward and backward kernels C and D.The MatMul rule remains applicable even when the matrix is correlated with vectors it transforms, while using a transposed matrix invokes the gradient-independence assumption.Source: §6.2 · pp. 7–8

6.2.1 Example: RNN

• For an RNN, the rules produce Gaussian forward covariances recursively from input and recurrent-state covariances, and combined forward-backward recursions yield its infinite-width NTK.The construction handles shared recurrent weights W and input weights U across time steps.Source: §6.2.1 · pp. 8–9

6.3 GIA Makes or Breaks the Intuitive Rules of Box 1

• When the final output averages embeddings rather than using an independent output-weight readout, Condition 1 can fail and treating W^⊤ as independent from W gives an incorrect backward distribution.In the example, the independence assumption predicts a zero-mean N(0, 4) distribution for dx1, but direct calculation shows its mean is not zero.Source: §6.3 · p. 9
• Under Condition 1, independent last-layer weights eliminate potential correlation pathways between W and W^⊤, explaining why the intuitive rules can correctly use GIA.The section closes by motivating a rigorous justification of these rules.Source: §6.3 · p. 9

7 NETSOR⊤

• NETSOR⊤ expresses both forward and backward neural-network propagation, and its Master Theorem rigorously justifies the resulting infinite-width calculations.The framework covers standard architectures when their propagations are expressible in NETSOR⊤ and satisfy the stated regularity conditions.Source: §7 · pp. 9–10
• Under Gaussian initialization, BP-like structure, and polynomially bounded nonlinearities, program observables converge almost surely to expectations over recursively defined limiting variables.The theorem applies to collections of program vectors and polynomially bounded test functions ψ.Source: §7 · p. 10
• Consequently, standard feedforward or recurrent networks with Condition 1 and polynomially bounded weak derivatives have NTKs that converge almost surely to deterministic kernels over finite input sets.The result includes typical activations such as ReLU and recovers prior NTK formulas rigorously for MLPs.Source: §7 · p. 11
• The BP-like condition is essential: valid NETSOR⊤ programs that are not BP-like can invalidate the Master Theorem, including the earlier GIA counterexample.The paper leaves extension to non-BP-like cases for future work because it requires substantially more machinery.Source: §7 · p. 11

Guide to the Appendix

• The framework generalizes to unequal widths and multidimensional outputs, while layer normalization and attention require the more powerful NETSOR⊤+ language.The appendices also develop examples for RNNs, CNNs, transformers, and batch normalization, alongside the formal proofs and theoretical tools.Source: Guide to the Appendix · p. 11

8 Conclusion

• The conclusion presents NETSOR⊤ as a general tool showing that randomly initialized standard feedforward and recurrent networks have almost-sure deterministic NTK limits.The authors frame this result as a foundation for understanding modern overparameterized neural networks.Source: §8 · p. 11

B.1 The Simplified NETSOR⊤+

• NETSOR⊤+ extends the simplified language with parameterized coordinatewise nonlinearities, moment operations, matrix multiplications, and scalar variables while retaining equivalent expressive power.Its notation collapses chains of parameterized nonlinearities into representations based on G-vars and associated parameters.Source: §B.1 · pp. 21–22
• The Simplified BP-like NETSOR⊤+ Master Theorem assumes Gaussian matrix initialization, iid Gaussian initial-vector coordinates, convergent initial scalars, parameter control, BP-like structure, and rank stability.It recursively defines limiting random variables for vectors and deterministic limits for scalars, supporting the asymptotic analysis of parameterized programs.Source: §B.1 · p. 22

Dimension constraints and CDCs

• Common Dimension Classes group H-vars whose dimensions are forced equal, while different classes may have different dimensions.The dimensionality rules preserve dimensions through nonlinear and moment operations and tie input/output dimensions across matrix multiplication.Source: C Programs with Variable Dimensions · p. 23

BP-like NETSOR⊤ theorem

• The variable-dimension BP-like NETSOR⊤ Master Theorem assumes polynomially bounded nonlinearities and sends all dimensions to infinity while every pairwise width ratio approaches a finite nonzero limit.Under these conditions, the theorem uses the variable-dimension sampling scheme for weights and input G-vars, with dimensions constant within each CDC.Source: C Programs with Variable Dimensions · p. 23
• For every CDC, empirical averages of polynomially bounded functions of its G-vars converge almost surely to expectations under a Gaussian distribution with recursively defined mean and covariance.The convergence applies as all program dimensions diverge, not only the dimension of the CDC being evaluated.Source: C Programs with Variable Dimensions · p. 24

Limitation across CDCs

• Equation (23) does not directly extend to G-vars from different CDCs because unequal dimensions prevent a naive empirical average, although their randomness can be viewed as roughly independent.The intuition is that different CDCs derive their main randomness from different A-vars.Source: C Programs with Variable Dimensions · p. 24

BP-like NETSOR⊤+ theorem

• The analogous BP-like NETSOR⊤+ theorem extends this convergence to parameter-controlled functions when rank stability holds and parameters converge almost surely to deterministic limits.It applies to polynomially bounded functions within a CDC and more generally to functions parameter-controlled at the limiting parameter vector.Source: C Programs with Variable Dimensions · pp. 24–25

D Writing Backpropagation of Standard Architectures in NETSOR⊤

• Forward computations expressible in NETSOR can be translated into NETSOR⊤ backpropagation, enabling explicit treatments of standard architectures.This framework is stated to apply whenever forward propagation can be written in NETSOR, including all standard architectures noted in Yang [62].Source: D Writing Backpropagation of Standard… · p. 25
• Convolutions become sums of weight-shared dense matrix multiplications, and their forward and backward equations are valid NETSOR⊤ snippets.The representation supports common zero padding and generalizes to different strides, paddings, dilations, and graph convolutions.Source: D Writing Backpropagation of Standard… · pp. 25–26
• Pooling operations, including global average pooling and coordinatewise local maxpooling, can be expressed through NETSOR⊤ nonlinear operations.Local maxpooling is defined over kernel positions and applies the maximum coordinatewise.Source: D Writing Backpropagation of Standard… · p. 26
• Batch normalization followed by a coordinatewise nonlinearity has NETSOR⊤ forward and backward forms when batch size B remains fixed as width n tends to infinity.The construction requires ϵ > 0; at ϵ = 0, the batch-normalization Jacobian is singular and the BP-like Master Theorem does not apply.Source: D Writing Backpropagation of Standard… · p. 26
• GRU and LSTM recurrences are naturally expressible in NETSOR⊤ because they consist of recurrent dense matrix multiplications and coordinatewise nonlinearities.The section works out GRU forward and backward propagation concretely, including gradients through its update and reset gates.Source: D Writing Backpropagation of Standard… · p. 27
• Layer normalization and attention require the NETSOR⊤+ extension to represent scalar moments such as means and variances during forward and backward propagation.The attention construction represents tokenwise key-query interactions, weighted values, and their gradients using parametrized nonlinearities.Source: D Writing Backpropagation of Standard… · pp. 27–28

E Example NTK Computations

• The NTK computations begin with the V-transform, which maps a multivariate nonlinearity and a positive semidefinite covariance matrix to the covariance of its Gaussian outputs.For scalar nonlinearities, the transform applies the function coordinatewise, and common V-transforms are collected for later kernel calculations.Source: E Example NTK Computations · p. 29

E.1 MLP

• The MLP appendix applies formal NETSOR⊤ and Theorem A.6 calculations to derive its infinite-width NTK, including transpose-dependent gradient limits beyond the Master Theorem.The calculation checks BP-like structure and polynomially bounded nonlinearities, with ReLU and its derivative satisfying the required conditions.Source: §E.1 · pp. 29–32

E.2 Simple Recurrent Neural Network and Average Pooling

• The NTK-limit derivation extends from projecting the last RNN state to projecting the average of all states, using different initial conditions for D in the two cases.For last-state output, the initial condition implies that most D terms are zero by induction; for average pooling, the majority of D terms cannot be zeroed out.Source: §E.2 · pp. 32–33

E.3 Convolution Neural Network

• The convolutional-network treatment derives deterministic forward covariance and backward-gradient recurrences for spatial feature maps, with distinct output and NTK contributions for global average pooling versus vectorization.The setup indexes image positions P and kernel positions K, and the NTK sums parameter-gradient and readout contributions over valid spatial locations.Source: §E.3 · pp. 33–35

E.3.1 Vectorized NTK Formula

• The vectorized NTK formula packages convolutional covariance recurrences into tensors and a convolution-implementable linear operator, yielding the infinite-width NTK while requiring only diagonal spatial entries in the vectorization case.When K ≠ −K, the backward recurrence uses the adjoint operator T† rather than T.Source: §E.3.1 · pp. 35–36

E.4 Batchnorm

• The batch-normalization analysis gives forward single-batch and cross-batch moment transformations and backward gradient transformations for batchnorm followed by coordinatewise nonlinearities, using Gaussian second-moment formulas [62] [68].The backward transformation is expressed through the adjoint of the forward covariance operator.Source: §E.4 · p. 36

E.4.1 NTK

• Batchnorm requires batch-to-batch NTK construction, using within-batch covariance recursions and reverse derivative recursions for same-batch inputs.For cross-batch inputs, the procedure additionally propagates cross-covariances and derivative quantities between the two batches.Source: §E.4.1 · pp. 37–38

E.5 Transformer

• The transformer forward pass recursively computes jointly Gaussian random variables and their covariances, while backpropagation supplies corresponding derivative-variable equations.Attention and Layernorm require the simplified NETSOR⊤+ Master Theorem.Source: §E.5 · pp. 38–39
• The transformer’s NETSOR⊤+ formulation satisfies the Master Theorem, giving an almost-sure NTK limit computed by vectorized forward and backward recursions.The construction handles Layernorm and Attention through the simplified NETSOR⊤+ Master Theorem.Source: §E.5 · pp. 39–40

E.5.1 NTK

• Transformer NTK implementation uses four-tensors indexed by sequences and tokens, with diagonalization, tensor multiplication, correlation, and SoftMax operations.These operations provide the vectorized representation for computing the NTK across multiple sequences.Source: §E.5.1 · p. 40

F Theoretical Tools

• The theoretical-tools section introduces elementary moment inequalities and probability facts used throughout the infinite-width analysis.It includes conditional distribution notation, convergence, Gaussian conditioning, Stein’s lemma, and Gaussian derivative identities.Source: F Theoretical Tools · p. 41, §F.1 · pp. 41–42

F.1 Probability Facts

• The probability appendix establishes notation for conditional distributions and convergence, and records Gaussian conditioning, Stein, and gradient-Hessian identities.These results are presented as standard probability facts, with prior proofs cited to Yang [62].Source: §F.1 · pp. 41–42

F.2 Gaussian Conditioning Trick

• The Gaussian conditioning trick gives the distribution of a Gaussian matrix conditioned on simultaneous constraints Y = AQ and X = A⊤P, expressed using Moore-Penrose projections.The projection matrices ΠP = PP+ and ΠQ = QQ+ encode the column spaces of P and Q.Source: §F.2 · p. 42

F.3 Law of Large Numbers for Images of Weakly Correlated Gaussians

• For Gaussian vectors with uniformly bounded correlations, Theorem F.8 provides a law of large numbers for coordinatewise functions under finite centered-moment conditions.The assumptions include p ≥ 6, with stronger 2pL moments yielding an additional convergence statement; high-rank orthogonal projections satisfy the correlation condition.Source: §F.3 · p. 42

G Proof of Main Theorem

• Theorem A.6 proves that polynomially bounded functions of all G-vars converge almost surely to their corresponding Gaussian expectation as n →∞.The Gaussian has mean µ and covariance Σ specified by Eq. (18), and the theorem applies to any BP-like NETSOR⊤ program satisfying Setup A.5 with polynomially bounded nonlinearities.Source: G Proof of Main Theorem · p. 43
• The proof’s inductive step is complicated by correlations between a matrix and its transpose, requiring rank stability, zero stability, and weakly correlated-variable laws of large numbers.The proof organization introduces notation first, then develops rank and zero stability before completing the inductive steps.Source: G Proof of Main Theorem · p. 43, A Bit of Notation and Terminology · p. 44

A Bit of Notation and Terminology

• Almost-sure statements are interpreted over a product probability space of input-variable instantiations, meaning the claim holds for all sufficiently large n outside a probability-zero set of sequences.The order of the qualifiers matters: one fixes an input sequence, then requires the claim for all large n.Source: A Bit of Notation and Terminology · p. 43
• The proof uses simultaneous induction on Moments and CoreSet rather than induction only on the number of program variables.The induction establishes Moments(M), which yields Theorem A.6, while CoreSet supplies additional structural control required by the general argument.Source: A Bit of Notation and Terminology · pp. 43–44
• CoreSet’s Basis condition reduces Moments to core G-vars because every other G-var is asymptotically a linear combination of them.Basis also implies that the core G-vars are linearly independent.Source: A Bit of Notation and Terminology · pp. 43–44
• CoreSet’s NullAvoid condition prevents core G-vars from asymptotically concentrating on Lebesgue-null sets where polynomial feature functions become linearly dependent.This preserves covariance-submatrix rank and supports convergence of the pseudoinverse needed for Gaussian conditioning.Source: A Bit of Notation and Terminology · pp. 43–44

G.1 Base Cases: Moments and CoreSet for Input Variables

• For input variables, Moments follows directly from the law of large numbers, while any subset whose restricted covariance has full rank provides a valid CoreSet.Specifically, choose M ⊆ [k] such that rank Σ|M = rank Σ; Basis and NullAvoid then hold.Source: §G.1 · p. 44

G.2 Inductive Case: Setup

• The new variable is written as g_m = Ah, with h formed from earlier variables, while previous uses of A and A^⊤ are tracked as ˆg_i = Aˆh_i and ˇg_j = A^⊤ˇh_j.This setup supports the induction from Moments(m −1) and CoreSet(m −1) to the corresponding properties at m.Source: §G.2 · pp. 44–45
• Conditioning on the σ-algebra of earlier G-vars constrains A through ˆG = AˆH and ˇG = A^⊤ˇH, introducing projections onto the column spaces of ˆH and ˇH.The projection involving ˇH prevents treating g as having independent Gaussian coordinates and creates much of the proof’s added complexity.Source: §G.2 · p. 45
• Under the induction hypothesis, the finite-dimensional matrices and vectors ˆΛ, ˇΛ, Γ, ˆη, and ˇη converge almost surely to their corresponding limits.These quantities contain moments of the preceding variables Z_1, ..., Z_{m−1}.Source: §G.2 · p. 45
• Stable limiting ranks imply almost-sure convergence of the pseudoinverses ˆΛ+ and ˇΛ+ to ˚ˆΛ+ and ˚ˇΛ+.The argument uses eventual rank equality and continuity of the pseudoinverse on fixed-rank matrices.Source: §G.2 · pp. 45–46
• The conditional variance converges almost surely to ˚σ2 = Σ(g, g) − Σ(g, ˆG)Σ(ˆG, ˆG)+Σ(ˆG, g), while the conditional mean is expressed using the limiting pseudoinverse and vanishing errors.Specifically, ω = ˆG(˚v + ˆε) + ˇHˇε, with ˆε and ˇε tending almost surely to zero.Source: §G.2 · p. 46

G.3 Rank Stability and Zero Stability

• Almost surely for sufficiently large n, K and ˚K have identical kernels, images, and ranks.This is the main Rank Stability conclusion.Source: §G.3 · pp. 46–47
• Zero Stability shows that a nonnegative function with zero limiting Gaussian expectation vanishes almost everywhere and eventually vanishes on the finite-width samples.The proof uses the density of the Gaussian law on the core coordinates and NullAvoid to transfer almost-everywhere vanishing to sampled values.Source: §G.3 · pp. 47–48
• Rank Stability follows by applying Zero Stability to null-space vectors and then using rank semicontinuity to obtain both rank inequalities.Equality of null spaces and image spaces then follows from equality of ranks and orthogonal-complement relations.Source: §G.3 · p. 47
• Proposition G.8 shows that the core-indexed Gaussian variables span and are linearly independent, so their covariance submatrix has the required full rank.The spanning and independence conclusion supplies the limiting linear structure used in Zero Stability.Source: §G.3 · pp. 47–48

G.4 Inductive Step: CoreSet(m)

• The CoreSet(m) induction asks whether the new index m must be added to preserve the Basis and NullAvoid properties.The section shows that these properties can be maintained without enlarging the existing core set.Source: §G.4 · p. 48, §G.4.1 · p. 49

G.4.1 If ˚σ = 0.

• When ˚σ = 0, φ is, after modification on a Gaussian-null set, a linear combination of the auxiliary functions {ˆφi}.NullAvoid removes the exceptional set from the finite-width samples, allowing the corresponding relation to preserve Basis with the same core set M.Source: §G.4.1 · p. 49
• Lemma G.9 records that ˚σ = 0 yields coefficients expressing the relevant function through the auxiliary functions, which is the key observation for later Moments(m) arguments.Because M remains unchanged, NullAvoid follows from the induction hypothesis.Source: §G.4.1 · p. 49

G.4.2 If ˚σ > 0.

• When ˚σ > 0, adding g to the core set preserves Basis and, using conditional density, establishes NullAvoid for the enlarged set M′.The proof reduces measure-zero slices to the existing NullAvoid property and concludes CoreSet(m), conditional on Lemma G.10.Source: §G.4.2 · p. 50
• Lemma G.10 proves that Π⊥Ĥ has no zero diagonal entries by reducing to full rank and showing the relevant leave-one-out matrices are nonsingular.Sherman–Morrison and convergence of the minimum eigenvalue of ˇΛ bound the projection diagonals away from degeneracy.Source: §G.4.2 · p. 50

G.5 Inductive Step: Moments(m)

• The G.5 induction proves Moments(m) by decomposing the target into A, B, and C, with each term converging almost surely to 0.The argument assumes ˚σ > 0 because the ˚σ = 0 case makes g a fixed linear combination of earlier variables.Source: §G.5 · pp. 51–52

G.5.1 C Converges Almost Surely to 0

• For C, conditioning on earlier Gaussian coordinates represents g_m as ˚v⊤Ẑ + ˚σz, so Gaussian averaging produces a polynomially bounded function to which induction applies.The conditional variance is ˚σ, and the resulting expectation is rewritten as an expectation over z ∼ N(0, 1).Source: §G.5.1 · p. 52

G.5.2 A Converges Almost Surely to 0

• For A, almost all sequences of earlier variables are amenable, and a conditional moment bound combined with Lemma F.2 yields almost-sure convergence to 0.Amenability is established through almost-sure uniform boundedness of the required moment expressions.Source: §G.5.2 · pp. 53–54
• The uniform-boundedness proof for A relies on induction-hypothesis limits, eventual control of σ by ˚σ + 1, and projection-diagonal bounds in [0, 1].The proof emphasizes that the order of almost-sure qualifiers matters when asserting bounds for sufficiently large n.Source: §G.5.2 · pp. 53–54
• Lemma G.13 supplies the reusable bound that polynomially bounded functions of the relevant Gaussian variables are almost surely uniformly bounded in n.Its proof reduces to powers |x|^d and controls the resulting summands using induction and decaying perturbations.Source: §G.5.2 · pp. 54–55

G.5.3 B Converges Almost Surely to 0

• Polynomially bounded nonlinearities are required here to ensure Gaussian integrability and support the smoothness estimates used for Eq. (50).This is the only place the proof needs the polynomial-boundedness assumption.Source: §G.5.3 · pp. 55, 57
• B converges almost surely to 0 by proving that Eqs. (49) and (50) each converge almost surely to 0.The argument establishes Eq. (49)'s convergence directly and reduces Eq. (50) to two claims proved in Lemmas G.15 and G.16.Source: §G.5.3 · pp. 56–57
• The Eq. (49) terms vanish independently almost surely, using that U has uniformly bounded size because rank ˇH is bounded.A sufficiently large moment parameter q controls the relevant uniformly bounded quantities and yields almost-sure convergence to zero.Source: §G.5.3 · pp. 56, 58
• Eq. (50) converges to 0 almost surely, using a smoothness bound for Ψα and perturbation vectors that go to 0 almost surely with n; ψ is polynomially bounded.The factor 1 + min(σ²/2, ˚σ²)−1 is almost surely uniformly bounded in n.Source: §G.5.3 · p. 57