A Closed Form Depth-Width Law for Neural Collapse in Deep Linear Networks

1 Introduction

• The depth-to-width ratio has been studied for finite-width kernel effects, and this work shows it also controls a representation-level observable.The motivating gap is that architectural effects on learned representations are usually measured after training rather than derived in advance.Source: §1 · p. 1
• Neural collapse tracks within-class variability, class-mean geometry, and classifier–mean alignment, raising questions about their evolution with depth, data, and architecture.Prior theory has examined idealized feature optima, perturbations, intermediate-layer compression, and data-dependent kernel descriptions.Source: §1 · p. 1
• The paper extends the feature-kernel formulation of NC1 to angular NC2 and NC3, then evaluates these observables on self-consistent feature-learning kernels.For exchangeable deep linear networks, the equations reduce to scalar within- and between-class channels.Source: §1 · pp. 1–2
• The resulting theory gives kernel forms for all three training-set observables and an exact geometric compression law whose leading architectural dependence is L/N.The contraction rate is determined self-consistently by the data, prior, and architecture.Source: §1 · p. 2

2 Neural collapse observables from feature kernels

• For balanced classes and nonzero normalizations, NC1, angular NC2, and kernel-ridge NC3 are functions of the feature kernel and class labels alone.The class-mean Gram matrix is formed from the feature kernel and centered class-averaging operators; the ridge readout uses κ > 0.Source: §2 · p. 2
• The NC1 identity is due to Kothapalli and Tirer [2025], while the NC2 and NC3 kernel forms are new.These quantify within-class compression, angular deviation from a simplex ETF, and readout class-mean misalignment, respectively.Source: §2 · p. 2
• The identities hold for arbitrary representations, but NC4 additionally requires a test-train cross-kernel and layerwise prediction requires a theory of learned kernels.The kernel identities alone do not predict how the observables evolve across layers.Source: §2 · p. 2

3 Exact depth law for exchangeable deep linear networks

• The MAP layerwise kernels are obtained from coupled forward–backward self-consistency equations in a Bayesian setting with Gaussian weight priors and observation noise.The underlying large-deviation theory is exact in the proportional limit P = αN →∞.Source: §3 · pp. 2–3
• Exchangeable class geometry restricts kernels to a closed algebra, decoupling the matrix equations into three scalar channels; centered NC observables discard the E channel.Under this symmetry, NC2 and NC3 vanish, leaving NC1 as the nontrivial layerwise observable.Source: §3 · p. 3
• Feature learning applies fixed within-class and between-class factors, so NC1 decreases geometrically whenever tB > tW.The input geometry sets the initial NC1, while the self-consistent feature-learning solution determines its depthwise contraction.Source: §3 · p. 3
• At large width, the leading contraction rate is r/N, making L/N the controlling architectural variable, with r determined by the data, prior, and architecture.The small-κ form applies only while κ is negligible relative to terminal channel eigenvalues; otherwise the exact finite-κ fixed point applies.Source: §3 · p. 3
• Figure 1 compares the geometric law with numerical fixed points, examines rate versus SNR and terminal NC1 versus L/N, and compares predictions with a trained network; Figure 2 compares NC1–NC3 from theory and trained XOR networks.Figure 1(d) uses a sampled four-class Gaussian task; Figure 2 reports variation across seeds.Source: Figure 1 · p. 4, Figure 2 · p. 4
• The terminal collapse separates into an input-geometry contribution and a network-induced contraction controlled by depth relative to width.This separates the starting representation geometry from compression accumulated through the network.Source: §3 · p. 4

• The combined kernel observables and feature-learning theory make representation geometry predictable from data, prior, and architecture without training each candidate network.Source: §4 · p. 4
• At large width, accumulated NC1 compression is controlled at leading order by L/N, with its rate determined self-consistently by data, prior, and architecture.Source: §4 · p. 5
• The resulting geometric law links architectural choices directly to learned geometry in exchangeable deep linear networks.Source: §4 · p. 5

• NC1, angular NC2, and ridge-readout NC3 require no explicit representation coordinates once the feature kernel and labels are known.The proof uses projected kernel traces for NC1, the centered class-mean Gram matrix for NC2, and kernel inner products for ridge-readout alignment in NC3.Source: A Derivation of the kernel… · p. 7
• The NC1 kernel identity was previously derived by Kothapalli and Tirer [2025], while the angular NC2 and ridge-readout NC3 kernel forms are new.Source: A Derivation of the kernel… · p. 7

A.1 Proof of the NC1 identity

• The proof represents within- and between-class variation through the corresponding covariance matrices and a projector onto the centered class-indicator subspace.Source: §A.1 · p. 7
• The class-indicator projector extracts centered class means repeated across each class, while the within-class projector extracts within-class residuals.Source: §A.1 · p. 8
• These projected components establish the NC1 identity.Source: §A.1 · p. 8

A.2 Proof of the NC2 identity

• The centered class-mean Gram matrix gives the cosine Gram matrix used to assess angular deviation from a unit-norm simplex ETF.Source: §A.2 · p. 9

A.3 Proof of the NC3 identity

• The NC3 construction removes the common class component before comparing class-specific readout directions.Source: §A.3 · p. 10
• Averaging cosine misalignment across classes yields NC3 quantities determined by K, Y, and κ.Source: §A.3 · p. 10

B Multi-output terminal condition

• The scalar-output theory requires a multiclass terminal boundary condition because the terminal conjugate kernel supplies the backward boundary for the feature-learning equations.For one-hot C-class targets, the analysis uses Nout = C.Source: B Multi-output terminal condition · pp. 10–11
• All C output coordinates share the same scalar feature kernel, and the output dimension enters only through the data-dependent part of the action.Conditioned on the penultimate representations, the output channels are independent Gaussian processes with common covariance K(L) + κIP.Source: B Multi-output terminal condition · pp. 10–11
• Only the subtraction term in the scalar-output terminal condition gains a factor C; the two outer resolvents remain unchanged.The condition reduces to the scalar-output case at Nout = 1, and is the only modification needed for the multiclass analysis.Source: B Multi-output terminal condition · p. 11

C Proof of Theorem 2

• The proof derives the NC1 depth law by reducing forward–backward kernel equations to scalar channels and using a conserved product to make transfer factors layer-independent.The relevant centered channels are the between-class and within-class channels.Source: C Proof of Theorem 2 · p. 11, §C.3 · pp. 12–13

C.1 Exact deep-linear feature-learning equations

• For a linear network, the MAP equations close exactly, with the input data setting the forward boundary and the multiclass terminal condition setting the backward boundary.The equations use forward feature kernels and conjugate kernels at each layer.Source: §C.1 · p. 11

C.2 Exchangeable projector decomposition

• Exchangeability gives kernels three entry types—diagonal, within-class off-diagonal, and between-class—and reduces the matrix equations to scalar eigenspace equations.The global-mean channel drops out of centered NC observables, leaving the between- and within-class channels.Source: §C.2 · p. 12

C.3 Channel conservation law

• The scalar forward and backward recursions conserve a product across depth, so each channel’s transfer factor is independent of layer.The proof uses this conservation and geometric propagation to establish Theorem 2.Source: §C.3 · pp. 12–13

C.4 Geometric depth law for NC1

• NC1 evolves geometrically with the ratio of within-class to between-class transfer factors.The common product of weight-variance factors cancels in the ratio.Source: §C.4 · p. 13

D.1 Exact scalar terminal closure

• The terminal closure determines the between- and within-class transfer factors, which are simplified when κ is small relative to the channel eigenvalues.The exact finite-κ fixed point applies when that small-noise condition does not hold.Source: D Proof of Corollary 1 · p. 13, §D.1 · p. 14

D Proof of Corollary 1

• The exact scalar fixed-point closure yields the depth law before any small-κ or large-width approximation is made.The proof then simplifies transfer factors for κ ≪ ΛW, ΛB before expanding their ratio at large width.Source: D Proof of Corollary 1 · p. 13, §D.1 · p. 14

D.3 Large-width depth–width scaling

• At large width, the accumulated compression is controlled at leading order by L/N.This scaling follows by substituting the large-width transfer-factor ratio into the exact geometric law.Source: D Proof of Corollary 1 · p. 13, §D.3 · p. 15

D.4 Self-consistent determination of the rate

• The contraction rate is not externally fitted; it is determined self-consistently by the input channel scale, prior variance schedule, depth, and width.The closure’s architectural dependence enters through L and N via their ratio in the large-width limit.Source: §D.4 · p. 15

Network training

• Finite-width training uses depth-L, width-N MLPs trained with full-batch SGLD and summed squared-error loss, with results averaged over 10 data and initialization seeds.Configurations run for 105 epochs; reported empirical values are across-seed means and standard deviations.Source: E Experimental and numerical details · p. 16

Kernel NC observables

• At each layer, NC1–NC3 are computed from the post-activation feature kernel using the kernel expressions, with ridge strength matched to the terminal readout.For raw-feature NC3, κ is rescaled to make ridge regularization comparable across kernel conventions.Source: E Experimental and numerical details · p. 16

Exchangeable linear experiments

• The exchangeable population experiments vary depth and width to isolate τ = L/N, using an idealized kernel without finite-dataset sampling or seed averaging.They use C = 5, P = 50, D = 400, and κ = 10^-4; the prediction uses exact finite-κ scalar self-consistency equations.Source: E Experimental and numerical details · pp. 16–17
• The closed-form prediction and full P × P forward–backward matrix solutions agree to numerical precision for N ∈ {64, 128, 256, 512}.The matrix equations are solved with width-annealed Anderson acceleration at tolerance 10^-6.Source: E Experimental and numerical details · p. 17

Trained linear-network comparison

• A separate panel compares SGLD-trained layerwise NC1 from a sampled Gaussian-multi dataset with the exchangeable prediction using measured input channel eigenvalues.Unlike the exchangeable population panels, this is a direct comparison with representations learned by a trained finite-width network.Source: E Experimental and numerical details · p. 17