Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes

Motivation

• The neural network–Gaussian process correspondence was established for infinitely wide, shallow networks and later rigorously for deep networks, while prior work focused on multilayer perceptrons and simple convolutional neural networks.This correspondence transforms implicit neural-network priors into explicit, analytically understandable priors [46] [49] [65] and has produced state-of-the-art kernels from deep neural networks [37] [43].Source: §1 · p. 1

Problem

• The central question is whether all infinitely wide, randomly initialized neural networks correspond to Gaussian processes, including recurrent networks.The paper also notes that architecture-specific proofs cannot keep pace with practice and that the question is less straightforward for recurrent networks.Source: §1 · p. 1

Contributions

• The authors define Gaussian processes with variable-dimensional outputs and prove convergence for standard feedforward and recurrent architectures as widths or channel counts tend to infinity.Standard architectures include compositions of MLPs, RNNs such as LSTMs and GRUs [10], skip connections [24] [27], convolutions, graph convolutions [8, 25, 15, 38, 14, 31], pooling, batch normalization [28], layer normalization [1], and attention [2].Source: §1 · pp. 1–2
• NETSOR expresses neural-network computations broadly, and its ability to implement standard architectures yields Gaussian-process convergence for all such expressible networks.The results for RNNs, batch normalization, layer normalization, attention, and their combinations with other layers are new.Source: §1 · p. 2

Relation to [60]

• The paper introduces tensor programs from [60], redesigns their notation, generalizes its Gaussian-process results, and provides kernel calculations and implementations for several architectures.Reference implementations cover vanilla RNNs, GRUs, transformers, and batch-normalized feedforward networks; this paper excludes forward passes using both a weight W and its transpose W^⊤.Source: §1 · p. 2

• For variable-dimensional f(x) ∈ R^l(x), Gaussian-process status means outputs over any finite inputs are jointly Gaussian in dimension Σ_i l(x_i).This extends the fixed-output definition, where k evaluations form a km-dimensional Gaussian.Source: §2 · p. 2
• A simple RNN processing 7- and 9-word sentences emits 16 scalars whose joint distribution should be Gaussian over random initialization.The resulting kernel is 16 × 16 because the RNN emits one scalar after each token.Source: §2 · p. 2
• At width 1000, the RNN’s final outputs exhibit a Gaussian distribution that closely matches the theoretically predicted distribution in Fig. 2(E).The figure visualizes the joint distribution of the last scalar from each sentence and the predicted Gaussian as dashed ovals.Source: §2 · p. 3

• Prior work [37] [40] [43] establishes GP convergence for wide MLPs with randomly initialized weights and biases, while this argument does not extend directly to RNNs.Source: §3 · p. 3
• An MLP computes h1(x) = W 1x + b1 and hl(x) = W lφ(hl−1(x)) + bl for l ≥2.Source: §3 · p. 3
• For two inputs x and x′, conditioned on h^(l−1)(x) and h^(l−1)(x′), the corresponding output pairs (h^l(x)_α, h^l(x′)_α) are iid across α.Source: §3 · p. 3
• If previous-layer coordinates are iid Gaussian, a law-of-large-numbers argument makes the covariance converge deterministically as width grows, and iterating layers yields a deep-MLP GP.Source: §3 · p. 3
• Tied weights break conditional iid Gaussianity, so the layerwise proof fails for RNNs and motivates the general NETSOR framework for standard architectures.Source: §3 · p. 3

Language and types

• NETSOR uses G-vars, H-vars, and A-vars for asymptotically Gaussian vectors, coordinatewise nonlinear transformations, and iid-Gaussian matrices, respectively.Variables carry dimensionality annotations such as G(n), H(n), and A(n_1,n_2), with G a subtype of H.Source: §4 · pp. 3–4

Core computation rules

• NETSOR programs take G- or A-vars as inputs, build variables through MatMul, LinComb, and coordinatewise Nonlin rules, and produce normalized scalar inner products.MatMul forms Gaussian vectors from matrix-vector products, LinComb preserves Gaussianity, and Nonlin maps Gaussian vectors into H-vars.Source: §4 · p. 4

Architecture examples

• NETSOR represents MLPs and simple RNNs, including deterministic inputs through random embeddings and recurrent weight sharing through reuse of A-vars.The examples show an MLP computation and an RNN operating on two input sequences.Source: §4 · pp. 4–5

Parameterized nonlinearities

• Nonlin+ allows nonlinearities to depend on parameters, enabling broader computations such as layer normalization and attention.In this paper, the parameters are continuous functions of empirical moments of G-vars.Source: §4 · pp. 4–5
• Empirical moments used by Nonlin+ converge almost surely to deterministic limits, so under suitable smoothness conditions the learned nonlinearity becomes approximately fixed at large width.This makes Nonlin+ intuitive as a self-determined Nonlin rule in the scaling limit.Source: §4 · p. 5

Assumptions and Scope

• The theorem applies under equal program dimensions and Gaussian random weights with Gaussian input coordinates, while the equal-dimension restriction can be removed and controlled nonlinearities need not be smooth.The GP result concerns finite-dimensional marginals; stronger convergence over function spaces would additionally require tightness.Source: §5 · pp. 5, 6, 8

NETSOR Master Theorem

• The NETSOR Master Theorem shows that empirical averages of controlled functions of program variables converge almost surely to expectations under a recursively defined Gaussian vector.The recursive mean and covariance are determined from the program structure, and each coordinate slice becomes asymptotically Gaussian and effectively iid for controlled empirical averages.Source: §5 · pp. 6–7

Computing the GP Kernel

• The GP-kernel corollary states that outputs of the form (v^T x_1/√n, ..., v^T x_k/√n) converge to N(0, K), with K_ij = σ_v^2 E_Z[φ_i(Z)φ_j(Z)].The covariance is deterministic because the conditional finite-width covariance converges to K; multiple independent output vectors are also supported.Source: §5 · pp. 7–8

GP Convergence

• Any standard feedforward or recurrent neural network with controlled nonlinearities converges in finite-dimensional distribution to a Gaussian process as widths increase under randomized weights and biases.The result also covers parameter-controlled nonlinearities, including attention and layer normalization, when the additional stated assumption holds.Source: §5 · p. 8

Empirical Demonstration

• With 1000-neuron simple RNNs, empirical covariance concentrates around the theoretical infinite-width covariance, with entrywise standard deviations typically at least an order of magnitude smaller than the covariance values.The predicted two-dimensional output distribution matches simulations from 100,000 randomly initialized RNNs, while the correlation structure captures context-sensitive similarities absent from the input embedding correlations.Source: §5 · pp. 8–9, Figure 2 · p. 8

Examples and Extensions

• Appendix constructions extend the kernel computation to NETSOR+, layer normalization, transformers, GRUs, batchnorm, and unequal-dimensional programs, with finite-width simulations approaching the theoretical kernels.Reference implementations and worked examples cover simple RNNs, GRUs, transformers, and batchnorm+ReLU networks.Source: §5 · p. 9, Figure 3 · p. 9

NN-GP Correspondence

• The NN-GP correspondence has been observed for architectural subsets [22] [13] [37] [40] [43], used to build models [12] [7] [32] [4] [6] [18] [43], and applied to training and generalization [42] [53].Lee et al. [37] constructed the SOTA permutation-invariant GP on MNIST, while Novak et al. [43] was until recently SOTA on CIFAR10 among GPs with untrainable kernels.Source: §6 · p. 9
• This paper generalizes NN-GP correspondence to standard architectures and controlled functions, whereas prior results impose linear-growth, smoothness, boundedness, ReLU, or derivative-growth conditions [40] [13] [43].The paper notes that the generality of Novak et al.’s three conditions remains unclear relative to its own conditions.Source: §6 · pp. 9–10

Signal Propagation in Neural Networks

• Signal-propagation studies analyze initialization and covariance dynamics in deep networks, using related covariance calculations but typically restricting architectures and emphasizing depth effects [23] [46] [21] [9] [45].One motivating question is how to choose initialization to avoid gradient vanishing.Source: §6 · p. 10

Reservoir Computing

• In reservoir computing, only the readout is trained while the randomly initialized recurrent network remains fixed, yielding exact GP inference at infinite width with the RNN kernel from Appendix B.2.Inputs are processed sequentially, and the final readout maps the random RNN state to the output.Source: §6 · p. 10

7 Conclusion

• Wide feedforward and recurrent neural networks of standard architectures converge in distribution to Gaussian processes with variable-dimensional outputs, substantially generalizing prior NN-GP results.Source: §7 · p. 10
• NETSOR provides a language for expressing standard deep-learning computations, while Theorem 5.4 characterizes randomized tensor-program behavior as dimensions tend to infinity.Source: §7 · p. 10
• The theory was empirically verified for simple RNNs, GRUs, transformers, and batch normalization, with corresponding infinite-width kernels open-sourced at github.com/thegregyang/GP4A.Source: §7 · p. 10
• The next paper will extend tensor programs to matrix transposes and use them to compute Neural Tangent Kernels [29] for any architecture.Source: §7 · p. 10

A Writing Standard Architectures in NETSOR

• NETSOR expresses batchnorm, skip connections, convolution, pooling, GRU/LSTM, and scaled attention, and compositions of these components remain expressible.The relevant nonlinearities are controlled or parameter-controlled, enabling the stated GP convergence theorems; GP convergence for batchnorm, GRU/LSTM, and attention is identified as new.Source: A Writing Standard Architectures in… · p. 16

B Example GP Kernel Computation with NETSOR

• GP kernels for architectures are computed recursively from Theorem 5.4 using V-transforms, which map Gaussian covariance matrices through nonlinearities.For a multivariate nonlinearity Φ, VΦ(K) is defined as E_z∼N(0,K) Φ(z)Φ(z)^⊤.Source: B Example GP Kernel Computation with… · p. 19

B.1 MLP

• The MLP subsection works out the usual kernel computation through tensor programs, illustrating how NETSOR recovers established multilayer-perceptron kernels.The construction emphasizes recursive or compositional structure in Σ for practical kernel computation.Source: §B.1 · p. 20, §B.1.1 · p. 20

B.1.1 MLP with Single Input (Program 1)

• With ReLU activations and equal widths n1 = n2 = n, the single-input MLP output is Gaussian with variance σ2 Ez φ2(z), computed through recursive µ and Σ evaluations.The derivation explicitly computes covariance entries for each G-var in program order, including Σ(h1,h1) = 2 and independence from b2 and v.Source: §B.1.1 · pp. 20–21
• The single-input MLP example illustrates NETSOR’s covariance bookkeeping, showing that successive preactivations can be independent while bias addition contributes directly to variance.In the example, Σ(h2, h1) = 0, whereas Σ(h2, h2) = 2 after combining the affine components.Source: §B.1.1 · p. 22

B.1.2 MLP with Multiple Inputs

• For multiple-input MLPs, the infinite-width network converges to a Gaussian process whose kernel is computed recursively from the activation’s V-transform.The recursion is based on the covariance of final-layer preactivations across the B inputs.Source: §B.1.2 · p. 24

B.2 Simple RNN (Program 2)

• Simple RNN outputs converge jointly to a Gaussian, with covariance obtained by recursively propagating token and hidden-state covariances across both input sequences.The algorithm initializes input covariance, iterates over sequence positions, forms the hidden-state covariance, and then computes the output kernel.Source: §B.2 · p. 26

B.3.1 Batchnorm with Single Batch

• With a single batch, batchnorm followed by a coordinatewise nonlinearity can be treated as a multivariate nonlinearity, and ReLU reduces the required Gaussian expectation to a 1D integral.Without this reduction, the expectation would require a B-dimensional integral that becomes intractable for large B.Source: §B.3 · p. 26, §B.3.1 · p. 27

B.3.2 Batchnorm with Multiple Batches

• For multiple batches, cross-batch batchnorm covariances can likewise be reduced from an apparently (Ba + Bb)-dimensional expectation to a 2D integral.The resulting algorithm separately evaluates within-batch and cross-batch covariance terms before returning the kernel.Source: §B.3.2 · pp. 27–28
• For k batches, with batch b containing B_b inputs, batchnorm network outputs converge jointly in distribution to Gaussian N(0, K), where K uses 1D and 2D integrals.Biases are omitted because batchnorm is shift invariant; the 1D integral applies within batches, while the 2D integral applies to distinct batches.Source: §B.3.2 · p. 28
• The repository implements single- and multi-batch batch normalization using vectorized quadrature and eigendecomposition-based integrand simplification.The implementation uses the quadpy package for vectorized quadrature integration.Source: §B.3.2 · p. 28

B.4 Convolution and Pooling (Program 4)

• The section revisits convolution and pooling in the neural-network–Gaussian-process correspondence from the tensor-program perspective [43] [18].Source: §B.4 · p. 28

B.4.1 CNN with Single Input

• For the single-input CNN, inputs are channel vectors indexed by pixel locations, with equal layer widths n1 = n2 = n and sampled parameters defining the setup.The example represents each image as {x_i ∈ R^m}_{i∈pos0}, where pos0 denotes pixel locations and m the number of channels.Source: §B.4.1 · p. 28
• The CNN’s input covariance is nonzero only for valid shared convolutional index pairs, while the input mean is identically zero and therefore the propagated mean remains zero.The covariance condition requires i + j and i′ + j′ to lie in the input-position set pos0.Source: §B.4.1 · p. 28
• The covariance recursion follows by applying Eq. (2), yielding a lemma that sums over valid kernel-index pairs for positions in the preceding convolutional layer.The summation is over j, j′ ∈ ker2 such that i + j and i′ + j′ lie in pos1.Source: §B.4.1 · p. 29
• The CNN output converges in distribution to a zero-mean Gaussian N(0, K), with K given by the derived scalar Gaussian-process kernel.The kernel expression is evaluated using Z ∼ N(µ, Σ) and independently uniformly sampled output positions.Source: §B.4.1 · p. 29

B.4.2 CNN with Multiple Inputs

• The CNN covariance recursion tracks input pairs and spatial locations across layers, while zero input means imply zero means for all intermediate variables.The resulting equations determine the GP kernel from local convolutional covariance updates and final spatial averaging.Source: §B.4.2 · pp. 29–30
• The CNN kernel computation can be vectorized over all input pairs and implemented with CUDA convolutions, including the final spatial averaging operation.The vectorized formulation treats layerwise covariances as matrices indexed by input and spatial position.Source: §B.4.2 · p. 31
• For multiple-input CNNs, the infinite-width output converges to N(0, K), with K computed recursively from the input covariance and spatially averaged final-layer covariance.The result applies to L-layer, bias-free CNNs with coordinatewise nonlinearity and multiple image inputs.Source: §B.4.2 · pp. 30–31

B.5 GRU (Program 5)

• For GRUs, the GP derivation requires Gaussian expectations whose dimension can grow with the number of unrolled timesteps because gate products couple values across time.The single-sequence analysis obtains zero means and a covariance matrix K for the GRU outputs in the infinite-width limit.Source: §B.5 · pp. 31–32
• For general activations and gates, the GRU covariance equations involve potentially T-dimensional Gaussian integrals that are generally intractable.When φ = erf and σ = (1 + erf)/2, the expectation reduces to a Gaussian orthant probability that can be evaluated efficiently.Source: §B.5 · p. 32
• The GRU recurrence extends to multiple input sequences, yielding output covariances across sequence indices and time steps through analogous covariance relations.The multi-sequence construction uses the same erf and scaled-erf gates and reduces its required expectation to an orthant-probability computation.Source: §B.5 · pp. 34–35
• A reference implementation computes the GRU GP kernel by evaluating the orthant probabilities with the R package mvtnorm.The implementation allows slightly more general initialization hyperparameters than the displayed derivation.Source: §B.5 · p. 35

Definitions

• Parameter control bounds a parameterized function’s deviation from its limiting function by a continuous factor that vanishes at the limiting parameter.Pseudo-Lipschitz functions in the inputs and parameters provide a common example, while some discontinuous functions can also satisfy the condition.Source: C NETSOR+ Master Theorem · p. 37

Assumptions

• Rank stability requires the rank of finite-width G-var collections to eventually equal the rank of their almost-sure limiting covariance matrix.The condition commonly holds when all limiting covariance matrices are full rank.Source: C NETSOR+ Master Theorem · p. 37

Master Theorem

• Theorem C.4 establishes almost-sure convergence for NETSOR+ programs under Assumptions 5.1 and C.3 when parameterized nonlinearities are parameter-controlled and their parameters converge to deterministic limits.The limiting Gaussian quantities are computed by replacing each parameterized nonlinearity with its parameterless version evaluated at the limiting parameters.Source: C NETSOR+ Master Theorem · p. 38

Practical use

• The theorem supports recursively instantiating nonlinear parameters with empirical moments, but its simultaneous use for parameter convergence and moment convergence creates a circularity addressed by Theorem C.11.Appendix C.1 supplies the additional setup, and Theorem C.11 becomes the primary practical tool for NETSOR+ programs.Source: C NETSOR+ Master Theorem · p. 38

Assumptions and limitations

• The remarks show that parameter control and rank stability are necessary: without them, the theorem can fail, whereas parameterless NETSOR programs obtain rank stability automatically.The automatic rank-stability result follows from the interleaved inductive proof of CoreSet and empirical-moment convergence.Source: C NETSOR+ Master Theorem · pp. 38–39

C.1 Self-Parametrized NETSOR+ Programs and Their Master Theorem

• Theorem C.11 proves the self-parametrized NETSOR+ master theorem by inductively unwinding the potential circularity between empirical parameters and program variables.The theorem assumes parameter-controlled nonlinearities at limiting parameters and uses simultaneous induction on moments and C-var limits.Source: §C.1 · pp. 39–41
• Self-parametrized NETSOR+ introduces C-vars that converge almost surely to deterministic constants and can serve as parameters of coordinatewise nonlinearities.The construction makes recursive parameters explicit and requires input C-vars to converge almost surely to deterministic limits.Source: §C.1 · p. 39

C.2 Gaussian Process Behavior of NETSOR+ Programs

• Parametrized and self-parametrized NETSOR+ outputs converge in distribution to N(0, K), with kernels computed by replacing parameters with their deterministic limits inside Gaussian expectations.For self-parametrized programs, the kernel uses ϕxi(Z; ˚Θxi)ϕxj(Z; ˚Θxj) under Z ∼ N(µ, Σ).Source: §C.2 · p. 42

D.1 Layernorm: Concrete Example (Program 9)

• The layer-normalization example represents mean, variance, and nonlinear transformations as a self-parametrized NETSOR+ program with Gaussian weight and output-vector inputs.The example uses inputs x and x′, matrices W 1 and W 2, vector v, and Moment operations for intermediate statistics.Source: §D.1 · pp. 42–43

D.1.1 Computing µ, Σ, and Limit Parameters ˚θ

• For the layer-normalization computation, all G-var means are zero, and the first variance limit is obtained as an expectation over a Gaussian variable.The supplied calculation gives ˚var1 = E z∼N(0,σ2w∥x∥2/m) z2 = σ2.Source: §D.1.1 · pp. 43–44
• The limit parameters for the worked NETSOR+ variables are computed by Gaussian moments, with the associated nonlinearities shown to be parameter-controlled at their limiting parameters.This includes the mean, variance, and activation variables for the two branches and their primed counterparts.Source: §D.1.1 · pp. 44–46

D.2 Layernorm: General Case

• For general self-parametrized nonlinearities with almost-surely convergent deterministic parameters, the master theorems provide Gaussian means and covariances under generic conditions and parameter-control.Layernorm is expressed through a self-parametrized Nonlin+ rule, which reduces to Nonlin when the limiting variance parameter is nonzero and rank stability is preserved.Source: §D.2 · p. 46
• The layer-normalization kernel is obtained from the nonlinear variables’ covariance matrix and mean vector using D = Diag(Ω−˚ν˚ν⊤).The limiting means are expectations of the defining nonlinearities under Z ∼ N(µ, Σ).Source: §D.2 · p. 47

D.3 Transformer (Program 10)

• The transformer program expresses masked attention, skip connections, feedforward transformations, and layer normalization using NETSOR+ operations across layers and tokens.Its presentation simplifies the standard transformer by omitting positional embeddings, using shared key/value/query projections, using one attention head, and modifying the skip connection.Source: §D.3 · pp. 47–48

D.3.1 Expressing the Composition of Attention, Skip Connection, and Layernorm via

• Attention logits, post-attention means and variances, and post-layernorm activations are represented by Moment and Nonlin+ rules whose nonlinearities are parameter-controlled under the stated conditions.The variance condition holds when all input tokens are nonzero and weight variances satisfy σw, σu > 0.Source: §D.3.1 · p. 49

D.3.2 Computing the GP Kernel

• The transformer output vector converges in distribution to a multivariate Gaussian N(0, K), with K determined from the final-layer covariance restriction.The kernel can be computed from the covariance of the final-layer normalized variables.Source: §D.3.2 · p. 50

D.3.3 Computing Σ

• The covariance recursion reduces transformer-kernel computation to equations linking attention-logit, post-attention, and post-layernorm covariance matrices.Equations (49)–(52), together with Eq. (49), form the complete equation set for computing the transformer Gaussian-process kernel.Source: §D.3.3 · pp. 50–51

D.3.4 Vectorized Implementation: Single Sequence

• The vectorized implementation already expresses Eqs. (49), (51), and (52) in vectorized form and also vectorizes Eq. (50) for a single sequence.This provides a vectorized route for the transformer-kernel equations.Source: §D.3.4 · p. 51
• The implementation defines attention weights as ∆ = SoftMax(Mask(Σkl)) row-wise, with the upper triangular entries of Σkl masked to −∞.The mask enforces causal attention by preventing a token’s embedding from depending on future tokens.Source: §D.3.4 · p. 51
• The covariance uses (I + ∆)Σkl(I + ∆)⊤, with I from the skip connection and D−1/2 from layer normalization, where D contains its diagonal variances.D is the diagonal matrix whose diagonal equals the diagonal of (I + ∆)Σkl(I + ∆)⊤.Source: §D.3.4 · p. 51

Computing Transformer Kernel

• The vectorized implementation represents each layer’s covariance as a tensor containing the corresponding covariance blocks for every pair of input sequences.The same representation is used for the associated Σyl tensors.Source: §D.3.5 · p. 51
• For p input sequences of t tokens processed by the transformer with nonzero parameter variances, the token outputs converge in distribution to N(0, K), where K ∈ R^{pt×pt}.The joint kernel covers all token embeddings across the input sequences.Source: §D.3.5 · p. 52
• At each layer, causal attention is computed per sequence by replacing the upper-triangular entries with −∞ before applying row-wise SoftMax.This produces each sequence’s attention matrix from its diagonal covariance block.Source: §D.3.5 · p. 52
• The per-sequence attention matrices are assembled into a block-diagonal matrix, while subsequent tensors are treated as pt × pt matrices.The implementation is available in the GP4A repository.Source: §D.3.5 · p. 52

• NETSOR− programs omit LinComb but remain semantically equivalent to NETSOR programs, since linear combinations can be absorbed into downstream nonlinearities.NETSOR is retained as canonical because LinComb expresses gadgets such as skip connections and convolutions more easily.Source: E Different Versions of Tensor Programs · p. 52
• The NETSOR− Master Theorem gives almost-sure Gaussian convergence for controlled observables of all G-vars, with mean and covariance specified as in Eq. (2).It is equivalent to Theorem 5.4, and LinComb cases in Eq. (2) are vacuous for NETSOR− programs.Source: E Different Versions of Tensor Programs · p. 52
• NETSOR◦ programs allow nonlinearities to take H-vars and are semantically reducible to NETSOR programs by recursively unwinding H-vars into G-vars.The same reduction applies to NETSOR◦+ programs, which additionally support parametrized nonlinearities.Source: E Different Versions of Tensor Programs · p. 53
• The NETSOR◦ Master Theorem preserves almost-sure Gaussian convergence when all unwinded nonlinearities are controlled, using means and covariances computed from the unwinded forms in Eq. (53).This formulation can be more concise because H-vars are reusable, but shifts unwinding into the scaling-limit computation.Source: E Different Versions of Tensor Programs · p. 53
• For NETSOR◦+ programs, parametrized unwinded nonlinearities converge to deterministic parameter values, after which the Gaussian limit uses parameterless nonlinearities evaluated at those limits.The theorem assumes almost-sure parameter convergence and parameter control at the limiting vector.Source: E Different Versions of Tensor Programs · pp. 53–54

F Programs with Variable Dimensions

• Variable-dimensional NETSOR programs partition H-vars into Common Dimension Classes whose dimensions may differ across classes.The partition is induced by input equivalence and dimensionality constraints from linear combinations and matrix multiplication.Source: F Programs with Variable Dimensions · p. 55
• The variable-dimension NETSOR Master Theorem gives almost-sure convergence of controlled functions of G-vars to expectations under a CDC-specific Gaussian distribution as all program dimensions diverge.The theorem requires dimensions within each CDC to match, but does not require different CDC dimensions to have convergent finite nonzero ratios.Source: F Programs with Variable Dimensions · p. 55

G Theoretical Tools

• The theoretical-tools section supplies probability lemmas for conditional distributions, Gaussian conditioning, Stein’s lemma, and convergence of output vectors from convergent second moments.These results support converting covariance limits into Gaussian output limits and controlling almost-sure convergence.Source: G Theoretical Tools · p. 56, §G.1 · p. 56

G.2 Review of Moore-Penrose Pseudoinverse

• The Moore–Penrose pseudoinverse review records existence, uniqueness, projection, singular-value, and Hilbert-space span properties used by later conditioning arguments.It also recovers the ordinary inverse when a matrix is invertible.Source: §G.2 · p. 57

G.3 Gaussian Conditioning Trick

• The Gaussian conditioning trick represents conditioned Gaussian vectors using a pseudoinverse, an orthogonal null-space projection, and an independent Gaussian residual.The generalized matrix lemma applies this construction to Gaussian matrices constrained by AQ = Y and A^⊤P = X.Source: §G.3 · pp. 57–58
• The matrix-conditioning construction is verified by showing its explicit solution satisfies both linear constraints and has minimum Frobenius norm.This establishes the pseudoinverse-based conditional representation for possibly singular constraint systems.Source: §G.3 · p. 58

G.4 α-Controlled Functions

• The α-controlled-function section defines growth control for functions on R^k and proves equivalent norm-based formulations plus Gaussian-expectation bounds that are monotone in the mean and covariance norms.The stated auxiliary results are formulated for α ≥ 1 and positive semidefinite covariance matrices.Source: §G.4 · pp. 58–59

A Bit of Notation and Terminology

• The proof simultaneously inducts on Moments and CoreSet, obtaining Moments(M) and hence Theorem 5.4.Basis reduces other G-vars to core-set variables, while NullAvoid prevents asymptotic concentration on problematic measure-zero sets.Source: A Bit of Notation and Terminology · p. 60

H.1 Base Cases: Moments and CoreSet for Input Variables

• For input variables, Moments follows from the law of large numbers, and any rank-preserving subset of inputs satisfies Basis and NullAvoid.Source: §H.1 · p. 60

H.2 Inductive Case: Setup

• The inductive setup conditions on prior G-vars, uses Gaussian projection identities, and proves convergence of the relevant pseudoinverse and conditional variance.Rank stability yields eventual equality of the finite-n and limiting ranks, making pseudoinverse continuity applicable; the limiting variance is the Schur-complement expression given in Lemma H.2.Source: §H.2 · p. 61

H.3 Rank Stability and Zero Stability

• Rank Stability shows that, almost surely for all sufficiently large n, K and its limit have identical kernels, images, and ranks.The proof combines Zero Stability with lower semicontinuity of rank and identifies image spaces as orthogonal complements of null spaces.Source: §H.3 · pp. 62–63
• Zero Stability transfers vanishing expectations of nonnegative functions under the limiting Gaussian law into eventual almost-sure vanishing across the triangular array.The argument uses the core variables’ density and absolute continuity with respect to Lebesgue measure, then applies NullAvoid to the resulting zero set.Source: §H.3 · pp. 62, 64

H.4 Inductive Step: CoreSet(m)

• The CoreSet induction preserves Basis and NullAvoid by deciding whether the new variable g belongs in the existing core set.The decision is governed by the limiting conditional variance: zero variance permits retaining the old core set, whereas positive variance requires adding g.Source: §H.4 · p. 64, §H.4.1 · p. 64, §H.4.2 · p. 65

H.4.1 If ˚σ = 0

• When ˚σ = 0, g is asymptotically a linear combination of the existing variables, so the old core set preserves Basis and inherits NullAvoid.The linear relation holds after excluding a measure-zero set, which NullAvoid eliminates asymptotically.Source: §H.4.1 · p. 65

H.4.2 If ˚σ > 0

• When ˚σ > 0, g cannot lie asymptotically in the span of the existing core variables, so the proof adds g to the core set and establishes Basis immediately.The remaining task is to prove NullAvoid for the enlarged core set M′.Source: §H.4.2 · p. 65
• When ˚σ > 0, slicing measure-zero triangular-array events and applying NullAvoid establishes NullAvoid for M′ and completes CoreSet(m).Source: §H.4.2 · p. 65

H.5 Inductive Step: Moments(m)

• The Moments(m) induction reduces the proof to showing that A, B, and C each converge almost surely to 0.If ˚σ = 0, gm is eventually a fixed linear combination of g1, …, gm−1, making Moments(m) trivial; otherwise Gaussian averaging supplies crucial smoothness.Source: §H.5 · pp. 66–67

H.5.1 C Converges Almost Surely to 0

• Conditioning Zm on earlier coordinates gives a Gaussian representation, and Gaussian averaging defines a controlled function to which the induction hypothesis applies.Source: §H.5.1 · p. 67

H.5.2 A Converges Almost Surely to 0

• A converges almost surely to 0 by bounding centered moments through uncentered Gaussian moments, uniformly controlling the resulting quantities, and applying Lemma G.1.Source: §H.5.2 · p. 68

H.5.3 B Converges Almost Surely to 0

• Assuming ˚σ > 0, Gaussian integration smooths the relevant function; Lemma H.9 yields deterministic limits, and the triangle inequality proves B converges almost surely to 0.Source: §H.5.3 · pp. 68–69

I Proof of NETSOR+ Master Theorem

• Theorem C.4 extends the Theorem E.3 proof by accounting for NETSOR+ LinComb rules, rank stability, and parameter fluctuations.These are the three structural differences identified in the proof augmentation.Source: I Proof of NETSOR+ Master Theorem · p. 70

I.1 LinComb

• LinComb applications can be absorbed into downstream nonlinearities or expressed as Nonlin+ rules, so the proof may assume programs contain no LinComb.This is a without-loss-of-generality reduction.Source: §I.1 · p. 70

I.2 Rank Stability

• Rank stability is necessary for the NETSOR+ Master Theorem and permits induction on Moments while replacing CoreSet-based uses of Lemma H.4.It is invoked when proving pseudo-inverse continuity under almost-sure limits and when establishing that σ is almost surely zero or positive.Source: §I.2 · p. 70

I.3 Fluctuation of the Parameters

• With parameters in nonlinearities, Eq. (62) gains an additional term D, while A, B, and C use ψ(−; ˚Θ).The modified terms retain the earlier structure after evaluating the nonlinearity at the reference parameter ˚Θ.Source: §I.3 · p. 70
• The terms A, B, and C still converge almost surely to zero under the same arguments, using rank stability instead of CoreSet where needed.The convergence remains valid under the parameterized-nonlinearity modification.Source: §I.3 · p. 70
• Parameter control supplies a controlled function ¯ψ and a continuous nonnegative error bound f with f(˚Θ) = 0.Both ¯ψ and f may depend on ˚Θ.Source: §I.3 · p. 70

I.4 Summary

• The proof of Theorem C.4 inducts on Moments, uses Assumption C.3 for Proposition H.1, adds D to Eq. (62), and assumes ˚σ > 0 and σ > 0 almost surely.The same arguments for A, B, and C from Appendix H.5 still hold, except in the proof of B.Source: §I.4 · p. 71

Formal specifications

• The appendix gives formal specifications for NETSOR−, NETSOR, and self-parametrized NETSOR+, corresponding to Figures 4–12.These specifications replace the main text’s informal treatment with formal definitions.Source: J Formal Specification of Tensor… · p. 71, Figure 6 · p. 71, Figure 7 · p. 72

Notation

• Matrix multiplication and self-parametrized moments are represented explicitly as MatMul and Moment, while type annotations use :: instead of :.These choices simplify the presentation of the formal specifications.Source: J Formal Specification of Tensor… · p. 71

NETSOR

• NETSOR extends NETSOR− with LinComb in its expression grammar and specifies its inference rules and semantics.The semantics include coordinatewise application of scalar functions and vector addition.Source: J Formal Specification of Tensor… · pp. 71–72, Figure 7 · p. 72, Figure 8 · p. 72

Self-parametrized NETSOR+

• Self-parametrized NETSOR+ extends NETSOR with a new type C and the Moment expression.Its formal treatment includes dedicated grammar, inference-rule, and semantics figures.Source: Figure 10 · p. 73, Figure 11 · p. 73, Figure 12 · p. 73