Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation

Core framework

• Tensor programs express most neural-network computations and, under width scaling with Glorot initialization, characterize limits for inference, backpropagation, and general programs.The main theorems use Gaussian conditioning to analyze these scaling regimes.Source: §1 · p. 1

Corollaries

• The framework yields rigorous corollaries for scaling-limit results spanning neural-network Gaussian processes, signal propagation, small-learning-rate SGD dynamics, and Approximate Message Passing.These applications are presented as consequences that justify and strengthen prior semirigorous derivations.Source: §1 · p. 1

Scope

• The results apply to standard architectures formed from MLPs, RNNs, skip connections, attention, convolution, batch normalization, and readout layers.The stated architecture class includes examples such as LSTMs and GRUs.Source: §1 · pp. 1–2

2.1. Gaussian Behavior of Wide Neural Networks

• The DNN-GP correspondence extends to fixed standard architectures and general nonlinearities, with Gaussian-process convergence for independently read out networks under the stated growth condition.The result covers finite input sets of appropriate signatures, including batches for batch normalization and sequences for recurrent networks.Source: §2.1 · p. 2

2.2. Signal Propagation in Neural Networks

• The signal-propagation framework connects GP kernels and gradient covariances, while its fixed-iteration scope does not rigorously justify large-depth limits.Prior work used these dynamics to prevent gradient explosion or vanishing, including training a 10,000-layer CNN without batch normalization or skip connections.Source: §2.2 · pp. 2–3
• Gradient independence is conditionally correct for MLPs with polynomially bounded weak derivatives, but can fail with global mean pooling; corrected equations and Jacobian singular-value limits cover standard architectures without batch normalization.The condition also requires an independently sampled, mean-zero readout layer; the framework agrees for tied and untied simple RNNs but differs for general gated RNNs.Source: §2.2 · p. 3

2.3. Neural Tangent Kernel

• For fixed standard architectures without batch normalization, the Neural Tangent Kernel converges almost surely as all widths grow together, generalizing prior sequential-width results.The paper also gives a CNN NTK computation and rigorously justifies related empirical Fisher spectral calculations through the NTK–FIM correspondence.Source: §2.3 · p. 4

2.4. Other Works

• The tensor-program technique rederives the semicircle law for Gaussian Orthogonal Ensembles and the Marchenko–Pastur law for Wishart matrices.These results demonstrate that the framework reaches beyond neural-network limits to classical random-matrix theory.Source: §2.4 · p. 5
• For Approximate Message Passing, the paper proves State Evolution for arbitrary polynomially bounded nonlinearities and test functions, removing a prior smoothness assumption under a stronger measurement-moment condition.The proof uses the same Gaussian conditioning technique underlying earlier State Evolution results.Source: §2.4 · p. 5

3. Tensor Programs

• Tensor programs are typed straightline programs with vector and matrix inputs, transposes, matrix multiplication, linear combinations, and nonlinear functions, supporting weight sharing and type checking.Variables are classified as G-, A-, or H-vars, and partial programs are represented as skeletons with dimension constraints.Source: §3 · p. 5

3.1. Examples

• Tensor programs express most neural-network computations, including MLPs, residual networks, RNNs, batch normalization, and CNNs, as well as combinations of these architectures.Layer normalization is not directly captured, although Section 8 addresses its scaling limit within the framework.Source: §3.1 · p. 6

3.2. Setup

• Common dimension classes group G-vars whose dimensions are equated by explicit constraints or program operations, capturing varying widths across network blocks.The framework studies randomized realizations of each skeleton as the dimensions of these classes grow, with dimensions, initialization scales, means, and covariances satisfying stated asymptotic conditions.Source: §3.2 · p. 6
• Under suitable conditions, G-vars converge to Gaussians with specific means and covariances, but this convergence does not hold in general.The setup treats tensor programs as the large-dimensional body of a neural network, with fixed-dimensional inputs and outputs represented indirectly or through selected coordinates.Source: §3.2 · pp. 6–7

Rank convergence

• Almost-sure rank convergence is automatic when there are no linear-combination lines and no polynomial nonlinearities, while LinComb lines introduce only width-independent dependencies.In the latter case, rank convergence still holds because those dependencies remain constant as width increases.Source: §4 · p. 7

Theorem 4.3

• Theorem 4.3 establishes transpose-free scaling limits when all functions are α-controlled with α < 2 and almost-sure rank convergence holds.The theorem applies to fixed dimension constraints and skeletons without transpose lines, with inputs sampled as specified in Section 3.2.Source: §4 · p. 7

Gaussian behavior

• Variables generated from the same matrix can remain correlated, while otherwise they become asymptotically independent modulo linear-combination dependencies.For large width, the relevant coordinates are described by iid Gaussian samples with means and covariances determined by the recursion.Source: §4 · p. 7

Deep linear networks

• The theorem’s apparent contradiction for tied versus untied deep linear networks arises because it applies only to fixed skeletons, hence fixed depth, as width grows.For fixed depth, the largest eigenvalue of W^L scales like 1 + O(n^-1/2), while input components aligned with eigenvalues at least 1 vanish in probability; exponential effects therefore emerge only at depths increasing with n.Source: §4 · p. 8

Backpropagation representation

• The construction requires appended matrix multiplications to use transposed matrices and later variables to be odd functions of the appended vector inputs.These restrictions also prevent LinComb lines from involving earlier g-variables.Source: §5 · p. 8
• The extended program ˜π represents backpropagation and gradients with respect to hidden states by appending transposed matrix variables and new vector inputs.Multiple appended vectors support multiple neural-network outputs, and gradients such as ∂x are odd functions of the output variables.Source: §5 · p. 8

Zero-mean gradient conditions

• Gradient variables are sampled with asymptotically zero mean and independently from the original input variables, while the extended program assumes polynomially bounded functions and almost-sure rank convergence.The covariance between original-program variables and newly introduced variables is zero.Source: §5 · pp. 8–9

Theorem scope

• The theorem rigorously justifies the gradient independence assumption for the extended tensor-program setting.Its scope is restricted by the stated dimension constraints and regularity assumptions.Source: §5 · p. 9

Limitations

• The result does not apply to batch normalization because its Jacobian is singular on a one-dimensional affine subspace, including the origin.This limitation prevents the theorem from covering batch-normalization gradients.Source: §5 · p. 9

• General tensor-program theory provides the correct gradient computation when variables do not have zero mean, where the earlier theorem can fail.The motivating quadratic-activation MLP example exposes the failure, and the section develops a replacement theory for general tensor programs.Source: §6 · p. 9
• Extended-syntax programs allow coordinatewise nonlinear functions to consume H-vars, while remaining semantically equivalent to original-syntax programs through expansion.The compilation framework also converts original programs to extended programs without transposes while preserving the scaling limit in a suitable sense.Source: §6 · pp. 9–10
• Detransposition systematically replaces transposed matrix operations with new inputs and compensating construction steps, propagating sampling statistics through an injective variable mapping.The construction handles vector, matrix, transpose, linear-combination, nonlinear, and matrix-multiplication lines inductively.Source: §6 · p. 10
• Theorem 6.3 establishes almost-sure convergence for polynomially bounded programs under rank convergence, with a differentiable-case formula for the compensating coefficients.The theorem applies to arbitrary dimension constraints and polynomially bounded test functions, using the Gaussian parameters of the detransposed program.Source: §6 · p. 11
• Unlike restricted programs, general-program outputs need not become Gaussian; detransposition instead tracks a Gaussian component plus a linear combination of prior activation outputs.The linear-combination coefficients correspond to Stein-lemma corrections, and the framework rederives semicircle and Marchenko-Pastur laws in example applications.Source: §6 · p. 11

Gaussian conditioning

• Gaussian conditioning represents a conditioned Gaussian matrix as a mean plus projected iid noise, enabling induction over tensor-program lines.The conditioning applies jointly to MatMul lines involving A and A^⊤ through previous G- and H-vars.Source: §7 · p. 11

Moment and smoothness assumptions

• Finite moments of input G-vars simplify the induction and remove the smoothness assumption required by Bayati & Montanari (2011).Gaussian averaging smoothness and near-independence of projected coordinates provide the key technical advantages.Source: §7 · p. 11

Projection estimates

• Projection off-diagonal correlation bounds and Hermite expansions establish law-of-large-numbers control for functions of projected Gaussian vectors.Linear programming duality bounds projection correlations, while Hermite expansions bound the resulting moments.Source: §7 · p. 11

Nonsmooth and singular cases

• Tempered-distribution derivatives preserve the theorem for nondifferentiable functions when covariance is nonsingular, and maximal-rank subsets handle singular covariance.This interpretation supports functions such as the step-function gradient arising from ReLU.Source: §7 · p. 11

8. Discussion

• Tensor programs express almost all modern neural-network computations and characterize their scaling limits, generalizing DNN-GP results, signal propagation, and NTK convergence.These are presented as corollaries among many results developed in the paper.Source: §8 · p. 12
• The authors expect the Gaussian-sampling results to extend to other “nice” distributions with a few finite moments, and rank-convergence assumptions may be removable through tensor-program stability.The expected distributional universality is motivated by central-limit intuition and precedents in random-matrix and statistical-physics literature.Source: §8 · p. 12
• The framework does not yet directly cover binary dropout, changing-signature layer normalization, or batch-normalization gradients with singular derivatives, though extensions are expected for the first two.Binary dropout can use Bernoulli variables, layer normalization is asymptotically linear in the forward pass with similarly expressible gradients, and batch-normalization gradients remain future work.Source: §8 · p. 12
• Scaling limits currently apply only to fixed tensor-program skeletons, so datasets comparable to or larger than network width require a joint limit over skeleton size, data distribution, and dimensions.The paper leaves this extension for future work and points to analogous two- or three-layer settings.Source: §8 · p. 12
• The framework naturally supports automation: software for PyTorch tapes or TensorFlow computation graphs could compute the corresponding µc and Kc automatically.The discussion proposes such a module as a practical research tool for random neural-network analysis.Source: §8 · p. 12

A. Common Dimension Classes

• Common dimension classes are computed by propagating equivalence relations and merging dimensions across vector, matrix, transpose, linear-combination, nonlinear, and composition operations.The procedure inducts over skeleton lines and tracks left- and right-dimension classes for matrix variables.Source: A. Common Dimension Classes · p. 18

B. Example Programs

• The framework covers common architectures through tensor-program constructions, including MLPs, detransposition, batched inputs, residual networks, and weight-tied RNNs.The examples encode matrix multiplication, shared weights, residual merges, and recurrent inputs within straightline programs.Source: §B.1 · p. 18, §B.1.1 · p. 19, §B.2 · p. 19

B.5. Batchnorm, fully-connected / B.6. Convolution

• Batch-normalized fully connected networks and convolutional networks are likewise represented by tensor programs using batchwise normalization and spatially indexed linear combinations.The examples also note that transformers without layer normalization, including softmax and self-attention, can be expressed similarly.Source: §B.5 · p. 21, §B.6 · pp. 21–22

D.1. Warmup: MLP

• D.1 shows that backpropagated gradient variables have limiting covariance Π_l(x,x′), enabling formal NTK computation beyond an unexamined gradient-independence assumption.The gradient covariance follows from the tensor-program treatment of forward and backward variables.Source: §D.1 · pp. 23, 25
• D.1 proves that randomized wide MLPs converge to Gaussian processes and that their NTKs converge almost surely under fixed-input width-ratio limits.The theorem assumes a nonlinearity with a polynomially bounded weak derivative and formally justifies prior GP and NTK recurrences.Source: §D.1 · pp. 24–25
• The MLP tensor-program proof models forward activations and backward derivatives together, with parameter randomization producing the limiting Gaussian expectations.This construction also handles global mean-pooling readout through a detransposed program.Source: §D.1 · pp. 24–26
• For an MLP with global mean-pooling readout and odd final nonlinearity, gradient independence correctly computes gradient covariance and the NTK.The global mean-pooling NTK has an almost-sure limiting formula given in Corollary D.3.Source: §D.1 · pp. 26–27

D.2. Warmup: Semicircle Law

• D.2 rederives the Gaussian Orthogonal Ensemble semicircle law by encoding spectral moments as a tensor program and solving their Catalan-number recurrence.The limiting moments are zero for odd powers and C_k/2 for even powers.Source: §D.2 · pp. 27–28
• Across both spectral warmups, detransposition and Theorem 6.3 reduce matrix-power traces to Gaussian tensor-program limits, with vanishing cross-covariances simplifying the resulting recurrences.The semicircle and Marchenko–Pastur constructions both use independently sampled transposed variables in the detransposed program.Source: §D.2 · pp. 27–28, §D.3 · pp. 28–29

D.3. Warmup: Marchenko-Pastur Law

• D.3 applies the same moment-method construction to Gaussian sample-covariance matrices with m_n/n → α, recovering the Marchenko–Pastur moment equations.The derivation relates Marchenko–Pastur moments to semicircle moments through E[x^r] = E[(√αy + 1 + α)^(r−1)].Source: §D.3 · pp. 28–29

• Gaussian embeddings have jointly Gaussian inputs whose covariance depends on pairwise products of the embedded inputs.This applies when the embedding is linear and its matrices are sampled from zero-mean Gaussian distributions.Source: §D.4 · p. 30
• Under polynomial growth and almost-sure rank convergence, normalized representation inner products converge almost surely to entries of a computable positive-semidefinite kernel C.The convergence holds for representations with matching widths as all tensor-program dimensions tend to infinity.Source: §D.4 · p. 30
• With suitably scaled Gaussian readout weights, parallel processing of a batch through the same network converges in distribution to a Gaussian process at infinite width.The readout weights are sampled from N(0, 1/n(v)), with shared or independent weights according to the output structure.Source: §D.4 · p. 30
• The same correspondence gives multivariate Gaussian-process convergence for parallel sequence-to-sequence RNNs and similar convergence results for standard architectures.The RNN case processes a batch of sequences of length S in parallel.Source: §D.4 · p. 30

Gradient Independence Assumption

• Gradient independence computes correct gradient dynamics for MLPs and, more generally, tensor-program networks with zero-mean readout weights and polynomially bounded weak derivatives.The method computes Kc and µc while treating forward G-vars as independent of backward-pass weights.Source: §D.5 · p. 30
• The result covers standard architectures without batch normalization and without transposed forward-pass weight sharing, while batch normalization remains outside the theorems because its gradient can be singular.Simulations suggest the theorem may still hold with batch normalization, but this is not established by the stated results.Source: §D.5 · p. 30

Singular value distribution

• For MLP Jacobians, tensor-program moment calculations show that forward derivative matrices and weight matrices can be treated as asymptotically free, justifying the required S-transform assumption.The argument establishes the relevant independence in the moments of tr((JJ⊤)^k) and then extends it to mixed moments.Source: §D.5 · pp. 30–31
• Corollary D.10 implies that the singular value distributions of the analyzed MLP Jacobians converge almost surely to their limits as hidden widths grow with finite positive width ratios.This justifies the Stieljes-transform calculations of Pennington et al. (2017).Source: §D.5 · p. 31
• The same tensor-program moment machinery can compute neural-network Jacobian singular-value distributions for arbitrary architectures, including weight tying.The framework expresses the required moment computations as tensor programs and applies Theorem 6.3.Source: §D.5 · p. 31

Simple RNN

• Weight tying in a simple RNN yields the same covariance kernel as untied layers, mathematically confirming experiments by Chen et al. (2018).The kernel between hidden states at different layers is zero, and the within-layer kernel matches the untied model.Source: §D.6 · p. 31

Autoencoder

• For a weight-tied autoencoder, all layer widths tend to infinity with consecutive width ratios converging to positive finite constants.The encoder and decoder use separately specified activations, biases, and shared weight structure under this proportional-width limit.Source: §D.6 · p. 32
• In this limit, each encoder representation converges to a centered Gaussian vector with layer-specific variance, and the autoencoder output satisfies the stated limiting characterization.The result follows by expressing the computation as a tensor program and applying Theorem 6.3.Source: §D.6 · p. 32
• The autoencoder theorem requires only polynomially bounded nonlinearities and coordinatewise polynomially bounded test functions, while guaranteeing almost-sure convergence without smoothness assumptions.Compared with Li & Nguyen (2018), this weakens global-Lipschitz requirements and strengthens convergence from in probability to almost sure convergence, while restricting the test functions to be coordinatewise.Source: §D.6 · pp. 32–33

D.6.1. JUSTIFYING SEMIRIGOROUS EQUATIONS

• The framework rigorously justifies previously heuristic signal-propagation equations across several neural-network architectures.These examples include MLPs, residual networks, convolutional networks, and batch normalization.Source: §D.6.1 · pp. 33–35
• Residual-network equations from Yang & Schoenholz (2017) are recovered rigorously using Theorem 4.3.Source: §D.6.1 · pp. 33–34
• Convolutional-network equations calculated semirigorously by Xiao et al. (2018) are recovered rigorously using Theorem 4.3.Source: §D.6.1 · p. 34
• For batch-normalized networks, the framework establishes finite-batch results for α-controlled activations, but cannot rigorously justify gradient dynamics because batch normalization has a singular Jacobian.The batch size and input dimension are fixed while hidden widths tend to infinity.Source: §D.6.1 · pp. 34–35

D.6.2. A TASTE OF WEIGHT-TYING

• For α-controlled test functions, the weight-tied residual dynamics have asymptotically i.i.d. zero-mean Gaussian coordinates.Source: §D.6.2 · p. 35
• The limiting covariance and cross-covariance quantities satisfy recursive equations with symmetry and boundary identities.Source: §D.6.2 · p. 35
• Weight tying in residual networks produces faster hidden-state norm growth than residual networks with untied weights.Source: §D.6.2 · p. 36
• The same limiting framework describes simultaneous forward evolution of two vectors and backward gradient dynamics in weight-tied residual networks.The backward equations assume a zero-mean Gaussian vector with normalized norm 1 and a residual network running S steps.Source: §D.6.2 · pp. 36–37

D.7. Neural Tangent Kernel

• For tensor-program networks, each parameter’s NTK contribution can be computed by expanding the network and its gradients into a joint tensor program.The resulting expression scales like (n_l)^2, and input G-vars have a similar simpler contribution.Source: §D.7 · pp. 37–38
• This NTK computation requires almost-sure rank convergence of relevant programs and polynomially bounded weak derivatives for the nonlinearities.Source: §D.7 · p. 38
• The framework computes a CNN NTK by combining contributions from convolutional weights, biases, and final-layer weights.Source: §D.7 · p. 38
• The CNN NTK converges almost surely to the stated limiting expression when the activation function has a polynomially bounded weak derivative.Source: §D.7 · p. 39

D.8. Approximate Message Passing

• AMP reconstructs sparse signals from noisy linear observations, and its high-dimensional behavior is characterized by state evolution as n,N→∞ with n/N→δ.The framework covers coordinatewise and more general vector nonlinearities, with applications including robust regression, Bayesian estimation, low-rank recovery, phase retrieval, and community detection.Source: §D.8 · p. 39
• A corollary of Theorem 6.3 establishes AMP state-evolution conclusions for Gaussian x0 and noise when f_t and g_t are polynomially bounded, without smoothness assumptions.Compared with Bayati & Montanari (2011), the result assumes Gaussian empirical limits for x0 and w, while removing smoothness requirements on the nonlinearities and test function ψ.Source: §D.8 · pp. 39–40

E.1. The Conditioning Trick

• Gaussian conditioning is expressed through Moore–Penrose pseudoinverses and orthogonal projections, including the conditional law of Gaussian matrices subject to linear constraints.The lemmas generalize conditioning results from Bayati & Montanari (2011) to singular matrices and characterize the minimum-norm solution enforcing AQ=Y and A^⊤P=X.Source: §E.1 · p. 41

E.2. Probability Facts

• The probability toolkit includes almost-sure convergence criteria, multivariate Gaussian conditioning identities, Gaussian integration formulas, and Stein’s lemma.These results support moment and derivative manipulations for Gaussian variables, including measurable test functions when the relevant expectations exist.Source: §E.2 · p. 42

E.3. α-controlled functions

• The α-controlled-function lemmas bound function growth and Gaussian expectations in terms of the mean and covariance norms, with monotonicity in those norms.For α-controlled φ, the relevant upper bound does not decrease when ∥μ∥2 or ∥Σ∥2 increases.Source: §E.3 · p. 43

E.4. Hermite Polynomials

• Hermite polynomials form an orthogonal basis under the standard Gaussian measure, enabling moment expansions and bounds for weakly correlated Gaussian inputs.The bounds apply when most pairwise correlations are at most λ/(k−1), with variants allowing one exceptional correlation and centered functions on an index set.Source: §E.4 · pp. 43–45

E.5. Moment Bounds of Lightly Correlated Gaussians

• For an orthogonal projection matrix, diagonal entries lie in [0,1], and its normalized correlation matrix has controlled off-diagonal entries when the projection rank satisfies k≥n/2.The resulting estimate bounds the sum of squared off-diagonal correlations by 2.5(n−k)^2.Source: §E.5 · p. 46
• Lemma E.20 considers functions of a rank-k Gaussian projection with finite coordinate variances and bounds their joint centered moments.The proof normalizes the covariance to a correlation matrix and expands centered functions in normalized Hermite polynomials.Source: §E.5 · p. 49
• Theorem E.21 assumes a rank n−O(1) projection and finite centered moments through order 2p for p≥6, with stronger assumptions allowing moments through 2pL for L>1.Its proof exploits that all but O(√n) ordered correlations satisfy |C_(ij)|≤n^-1/4, then separates injective maps, single collisions, and multiple collisions.Source: §E.5 · pp. 50–52
• The collision analysis bounds exceptional contributions by O(n^(2p−1.5)B_2p) for one close-correlation case and O(n^(2p−2)B_2p) when multiple input pairs collide.The argument combines correlation ordering, moment inequalities, and separate treatment of large and small off-diagonal correlations.Source: §E.5 · pp. 51–52
• Lemma E.22 further treats polynomially bounded functions, with a universal constant R depending only on the growth degree p and not on µ or σ.The proof derives bounds on Gaussian expectations and their µ-derivatives whose constants depend only on p.Source: §E.5 · pp. 52–53

Theorem 4.3 proof

• Theorem 4.3 is proved by induction over program length, with Gaussian convergence established for the base case and preserved by linear combinations and nonlinear lines.The argument uses moment bounds for α-controlled functions, dominated convergence, and almost-sure convergence of empirical averages.Source: F. Proof of Main Theorems · p. 53
• Conditioning on preceding G-vars makes MatMul coordinates conditionally independent Gaussian variables, with limits determined by kernel covariances and projections onto previously generated features.The conditional mean converges to Kc(G,G)+Kc(G,g), while the conditional variance converges to Kc(g,g) − Kc(g,G)Kc(G,G)+Kc(G,g).Source: F. Proof of Main Theorems · pp. 53, 54, 57
• The projection and pseudoinverse quantities converge almost surely under rank convergence, supplying the covariance and regression limits needed to complete the nonlinear induction step.The proof derives limits for empirical products, (Ht⊤Ht)+, and (Ht)+ht before applying Gaussian smoothing and uniform integrability arguments.Source: F. Proof of Main Theorems · pp. 53–55

Transposed-matrix extension

• The transposed extension assumes all program functions are polynomially bounded, because compositions of merely α-controlled functions need not be Gaussian-integrable.The proof also handles varying conditional variances by separating a low-rank exceptional set and using rank H′t = O(1) to control it.Source: F. Proof of Main Theorems · pp. 56, 59, 60
• The transposed-matrix extension conditions on all prior G-vars and tracks covariance, cross-covariance, and pseudoinverse limits for both forward and transposed matrix products.Independence and oddness arguments force selected cross terms to vanish, yielding the limiting conditional mean and variance formulas.Source: F. Proof of Main Theorems · pp. 56–57

Gaussian and Hilbert-space tools

• Gaussian conditioning and Stein’s lemma identify the limiting coefficients through inner products in L2(Z), while orthogonal projections control dependencies among previously computed matrix-multiplication lines.The construction uses maximal linearly independent Gaussian variables and projection operators onto spans of corresponding square-integrable functions.Source: F. Proof of Main Theorems · pp. 62–63

Main theorem

• Theorem 6.3 establishes almost-sure convergence for polynomially bounded tensor programs under rank convergence of the detransposed program, for arbitrary dimension constraints and polynomially bounded test functions.The theorem applies to sampled input variables and identifies the limit through the Gaussian vector Z ∼ N(µˇc, Kˇc).Source: F. Proof of Main Theorems · p. 64

Differentiable extension

• If all program functions are differentiable, the theorem remains valid with coefficients expressed using Gaussian expectations of derivatives and the activation’s limiting variance.This derivative-based formula is obtained through Stein’s lemma and the covariance structure of the detransposed program.Source: F. Proof of Main Theorems · pp. 64, 67

Inductive proof

• The proof proceeds by induction over program lines, with matrix multiplication as the only nontrivial case and covariance-related quantities converging almost surely by the induction hypothesis.The matrix-multiplication step conditions on preceding vector variables, introduces empirical covariance and cross-covariance matrices, and integrates out an auxiliary Gaussian matrix.Source: F. Proof of Main Theorems · pp. 57, 64, 65

Degenerate covariance

• When Gaussian covariance is nonsingular, polynomially bounded functions can be treated via distributional derivatives; when it is singular, restricting to a maximal-rank independent subset preserves the argument.The singular-covariance treatment expands dependent Gaussian coordinates as linear combinations of an independent subset having a density.Source: F. Proof of Main Theorems · pp. 64, 67