A Qualitative Test-Risk Mechanism for Scaling Behavior in Normalized Residual Networks

Problem and framework

• Depth expansion is analyzed through representational gain, optimization gain, and generalization transfer rather than hypothesis-class size alone.The setting inserts a residual block after an intermediate layer of a trained reference model and asks when the expansion improves test risk.Source: §1 · pp. 1–2

Representational gain

• A nontrivial first-order descent direction near zero initialization guarantees an expanded-class jumpboard model with strictly lower population risk.This establishes the representational gain that optimization must realize.Source: §1 · p. 2

Test-risk guarantees

• The population-risk route gives a tighter test-risk inequality when a positive deterministic margin is available, while the direct train/test route avoids Hoeffding transfer and remains robust when that margin is too small.Both routes quantify how optimization and generalization determine whether the representational improvement survives finite-sample error.Source: §1 · p. 2

Scaling mechanism

• Depth improves test performance when representational and optimization gains outweigh generalization costs; under stronger conditions, the improvement is strict.The framework therefore distinguishes a non-worsening guarantee from a strict-improvement guarantee.Source: §1 · pp. 1–2

Roles of depth, width, and data

• The scaling picture is joint: depth creates improving directions, width improves finite-sample observability of weak signals, and data controls the decay of the generalization penalty.The framework extends naturally to post-normalized ResNet-like and Transformer-like blocks under corresponding regularity and first-order non-degeneracy conditions.Source: §1 · p. 2

• Empirical studies report power-law gains with increased scale, while recent theory examines global scaling trends [Hestness et al., 2017, Kaplan et al., 2020, Hoffmann et al., 2022, Rosenfeld et al., 2019, Hernandez et al., 2021, Bahri et al., 2024].Source: §2 · p. 2
• Prior analyses show that skip connections stabilize training and depth increases expressivity, alongside work on trainability, optimization, dynamical stability, and initialization [He et al., 2016a,b, Veit et al., 2016, Balduzzi et al., 2017, Telgarsky, 2016, Eldan and Shamir, 2016].Source: §2 · p. 2
• Classical and ResNet-specific Rademacher analyses provide complexity upper bounds, whereas this paper proves when residual blocks directly improve test risk [Bartlett et al., 2017, Golowich et al., 2018, Neyshabur et al., 2017, Arora et al., 2018b, Ma et al., 2020, Kammonen et al., 2020].The comparison between these works and this paper is presented in Appendix A.Source: §2 · p. 2

Model and hypothesis classes

• The model is decomposed as f = ftop ◦ fbot, and inserting a zero-initialized residual block expands Hold to Hnew while recovering the trained old model at initialization.The inserted block is parameterized by θ with h0 = 0.Source: §3 · p. 3

Jumpboard models

• Population and empirical jumpboards use first-order descent directions to define margins ∆R and ∆train,S for the population-risk and direct train/test routes, respectively.The population route uses vpop, whereas the direct train/test route uses the training-set direction vS.Source: §3 · pp. 3–4

Analytic assumptions

• The framework assumes differentiability under expectation, Lipschitz-bounded loss and top maps, and width-independent gradient-covariance control for activation-gradient signals.These conditions support the first-order analysis and control the relevant gradient fluctuations.Source: §3 · p. 3

Normalized-network regularity

• Normalized residual networks satisfy branch Lipschitz, parameter norm, input norm, and non-degeneracy conditions that support norm-based generalization control.An engineering-ϵ alternative replaces the idealized non-degeneracy requirement with uniformly Lipschitz, bounded stabilized normalization operators and ϵeng-dependent constants.Source: §3 · p. 3

First-order non-degeneracy

• Strict improvement is claimed only when an inserted-block direction has negative first-order loss derivative; otherwise, the first-order mechanism makes no strict-improvement claim.Condition 1 characterizes the improvable regime, where new residual features are not orthogonal to the remaining activation-gradient signal, and need not hold for every model or insertion layer.Source: §3 · p. 4

Selection and optimization

• The selected expanded model is no worse on the training set than the comparison jumpboard, requiring only a weak fallback condition rather than optimizer convergence or algorithm-specific guarantees.The optimization gain is ∆ERM ≥ 0; positive ∆ERM represents additional empirical improvement beyond the jumpboard.Source: §3 · p. 4

Theorem 1: Population-risk mechanism

• If ΔR + ΔERM > 2ϵM, the selected expanded model has lower population risk than the old model.Theorem 1 assumes the expanded model improves training loss over the population-risk jumpboard by ΔERM ≥ 0 and establishes a deterministic positive ΔR.Source: §4 · p. 4

Theorem 2: Test-set mechanism

• Theorem 2 provides a high-probability test-set scaling guarantee through population-risk transfer under the same jumpboard and training-loss comparison.The guarantee uses deterministic η0 > 0 and ΔR > 0 and holds with probability at least 1 − 2δ − 6 exp(...).Source: §4 · pp. 4–5

Theorem 3: Direct train/test comparison

• If ΔR + ΔERM > 2(ϵM + ϵK), the direct train/test route gives lower test loss for the expanded model than for the old model.Theorem 3 uses a jumpboard built from a first-order descent direction and a random finite-test-set margin.Source: §4 · p. 5

Empirical alignment mechanism

• The probability of nonpositive training-to-test alignment decreases with training size, test size, and the width-dependent signal, scaling as O(1/MN + 1/KN) in the uniformly active regime.This probability bound holds up to a mixed covariance term; width increases test-set visibility rather than making the finite-test margin deterministic.Source: §4 · p. 5

Three proof ingredients

• Under Assumption 1 and Condition 1, the inserted block is not first-order dead, yielding a jumpboard model with strictly lower population risk.This establishes the positive population-risk margin ΔR > 0 used by the population-risk route.Source: §5 · p. 5

Route A: through population risk

• Route A transfers the population-risk improvement using Hoeffding, achieving a tighter total generalization price of 2ϵM when the margin ΔR > 0 is non-vanishing.Its proof appears in Appendix G.Source: §5 · pp. 5–6

Three proof ingredients

• Under Assumption 2 and either Condition 4 or 4ϵ, Lemma 1 uniformly controls the expanded loss class through ϵM and ϵK.With RMSNorm and engineering constant ϵ > 0, the ϵ-version removes the main text’s pre-normalization non-degeneracy requirement.Source: §5 · p. 5
• The optimization comparison connects the existential jumpboard model to the actual final model by requiring no worse training loss up to the gain term ΔERM.This is the step that converts representational existence into an achievable final-model guarantee.Source: §5 · p. 5

Route B: directly on train/test risks

• Route B bypasses population risk and Hoeffding, controlling train/test risk directly through ϵM + ϵK and remaining robust when ΔR is too small for Route A.Route A is sharper when ΔR is not too small and M ≫ K; together, the routes provide complementary scaling guarantees.Source: §5 · p. 6

Depth

• Depth creates new improving directions when an inserted residual block yields a nonzero first-order descent direction near zero initialization.This can produce a lower-risk jumpboard model and, under the main theorems, improved test performance.Source: §6 · p. 6
• Further depth eventually loses its first-order mechanism when Condition 1 fails at every admissible insertion layer.Near this deepest-model regime, the remaining signal weakens; at the limit, additional blocks may enlarge the hypothesis class formally without yielding meaningful strict improvement.Source: §6 · p. 6

Data

• Model expansion improves performance only when data grows fast enough to offset the statistical cost of the enlarged, norm-controlled hypothesis class.The generalization term follows an M^-1/2 rate, with architecture-dependent factors involving norm, depth, width, and normalization; the effective-dimension shorthand applies only when those factors are controlled or fixed.Source: §6 · p. 6
• The bound supports qualitative data–complexity coupling but does not predict the roughly 20-training-tokens-per-parameter Chinchilla ratio.The precise proportionality depends on norm amplification, objective, compute budget, and optimization setting.Source: §6 · pp. 6–7

Depth–width coupling

• As depth approaches saturation, width compensates by improving finite-sample observability of weak first-order gains, provided width and data scale sufficiently fast.In this regime, Route A suffers from a shrinking deterministic margin, while Route B must reliably detect the favorable direction at finite sample size.Source: §6 · p. 7

7.1 Empirical support for Assumption 3

• Covariance heatmaps and sampled off-diagonal distributions show activation-gradient covariances concentrated near zero, providing empirical support for the approximately diagonal, spectrally controlled condition in Assumption 3.The experiment supports the approximation underlying the width-alignment analysis without claiming that the covariance matrix is exactly diagonal.Source: Figure 1 · p. 7, §7.1 · p. 8

7.2 Decay of first-order activation-gradient signals with depth

• Activation-gradient norms decrease substantially with depth across widths, while normalized curves confirm depth-driven decay and wider models preserve larger, more stable absolute signals.The reduction is strongest during early deepening before flattening, indicating progressive depletion of the first-order improvement signal near the deepest-model regime.Source: Figure 2 · p. 8, §7.2 · pp. 8–9

7.3 Test loss under joint depth and width scaling

• Increasing depth and width often reduces test loss, but on CIFAR-10 loss rebounds at larger depths when width does not grow adequately; larger width stabilizes depth scaling.For CIFAR-100, no comparable rebound appears within the tested depth range. Width substantially reduces test loss in narrow-to-moderate regimes, then curves flatten as further widening yields marginal improvement.Source: §7.3 · p. 9

8 Discussion and Conclusion

• Model-dependent Assumptions G and G’ couple first-order improvement to excess-risk reduction, inducing a recursion that yields asymptotic power-law scaling.The qualitative mechanism therefore supports, but does not itself establish, the empirical power-law form.Source: §8 · pp. 9–10
• Depth expansion combines representational gain, optimization gain, and generalization transfer into population-risk and direct train/test routes for improving test risk.The framework identifies depth as creating improving directions, width as preserving their finite-sample observability, and data as controlling expansion’s statistical cost.Source: §8 · p. 10

A Related Work

• Unlike prior theory on global scaling laws, residual-network generalization, or deep-ResNet optimization, this work studies old-versus-new test-risk comparisons for normalized residual depth expansion.Its unified framework combines constructive representational gain, optimization comparison, and generalization transfer.Source: A Related Work · p. 12, Table 1 · p. 13

C Clarification of Condition 1: From Hidden-Space Descent to Residual-Block Realizability

• Condition 1 requires both a hidden-space descent direction and first-order realizability of that direction by the inserted residual block near zero initialization.These are logically separate requirements linking risk reduction in representation space to an implementable network expansion.Source: C Clarification of Condition 1: From… · p. 16

C.1 Hidden-space descent direction

• A sufficiently small hidden-space step along a suitable perturbation decreases population risk to first order, providing the mechanism behind the population jumpboard model.The same construction has an empirical training-set analogue for the direct train/test route.Source: §C.1 · p. 17

C.2 Residual-block realizability

• At zero residual initialization, the parameter Jacobian induces the hidden perturbation to first order, with a remainder vanishing as the parameter step tends to zero.This Jacobian-based realization is the parameter-space interpretation of Condition 1.Source: §C.2 · pp. 17–18

C.3 Equivalent Jacobian form of Condition 1

• Condition 1 is mild because the residual block need not span every hidden perturbation; its tangent class only needs nonzero projection onto the population activation-gradient signal.Equivalently, a parameter direction u must satisfy the first-order non-degeneracy condition expressed through Jθh0.Source: §C.3 · p. 18
• Condition 1 has population and empirical forms, using parameter directions that construct population and training-set jumpboard models, respectively.The empirical direction may depend on the training sample, while the population direction supports the population jumpboard.Source: §C.3 · pp. 18–19

C.4 Construction of the jumpboard model

• A population descent direction yields a jumpboard model with strictly lower population risk than the old model.The construction follows a sufficiently small parameter path under Condition 1’.Source: §C.4 · p. 19

C.5 Relation to standard zero-output residual blocks

• Standard zero-output residual blocks satisfy Condition 1 when residual features are not orthogonal to the relevant activation-gradient signal.Zeroing only the final output projection embeds the old model while preserving nontrivial first-order directions through that projection.Source: §C.5 · pp. 21–22

C.6 Why the derivative is taken with respect to parameters

• The relevant Jacobian is with respect to inserted-block parameters, because the jumpboard fixes the old representation and changes only the new residual block.An input Jacobian can vanish for a zero-last-layer block even though its parameter Jacobian is nonzero.Source: §C.6 · pp. 22–23

C.7 Why Condition 1 is mild.

• Condition 1 is mild: it fails only when the activation-gradient signal is orthogonal to every residual feature realizable by the inserted block.If the corresponding matrix is nonzero, choosing the sign of the output-projection perturbation gives strict first-order descent; the failure case is characterized as a degeneracy.Source: §C.7 · p. 23

Population Jumpboard Existence

• Under Assumptions 1 and 2(a), the expanded hypothesis class contains a model with population risk strictly smaller than the old model’s.This establishes the value of depth at the population level without requiring the optimizer to recover the auxiliary jump model exactly.Source: D Population Jumpboard Existence:… · pp. 23–25
• Starting from zero initialization, the inserted residual block vanishes at the origin, where a parameter direction produces a negative first-order population-risk derivative.The proof parameterizes a one-dimensional path θ(t) = tv and shows that population risk decreases to first order along it.Source: D Population Jumpboard Existence:… · p. 24
• Differentiability and negativity of the derivative imply that a sufficiently small positive step along the path gives a strictly better model in the expanded class.The argument uses differentiability at the origin to convert the local descent direction into strict population-risk improvement.Source: D Population Jumpboard Existence:… · p. 24

E Proof of Theorem 5

• Theorem 5 shows that the probability of positive alignment between the training-selected empirical direction and an independent test direction improves with width, training size, and test size.The proof combines training-direction concentration with independent test-direction fluctuation control.Source: E Proof of Theorem 5 · p. 25
• Theorem 5 decomposes alignment failure into training misalignment with the population direction and independent test-direction fluctuation.The argument separately bounds the event that the empirical direction is unreliable and the event that the test direction reverses the alignment.Source: E Proof of Theorem 5 · p. 25
• The proof bounds the training-sample failure term and combines it with the conditional independent-test bound to establish the theorem’s alignment guarantee.Conditioning fixes the training-selected direction while retaining randomness in the independent test direction.Source: E Proof of Theorem 5 · pp. 26–27

F Proof of Lemma 1

• Lemma 1 derives a norm-based empirical Rademacher complexity bound for normalized residual networks under Assumptions 2 and 4.The proof controls normalized representations, layerwise input sensitivity, parameter-space compactness, parameter Lipschitzness, covering numbers, and the loss-class complexity.Source: F Proof of Lemma 1 · pp. 27–30
• The resulting corollary gives a uniform generalization bound whose explicit factors depend on norms, depth, and normalization.The dominant complexity term is retained through the bound’s dependence on the network-class parameters.Source: F Proof of Lemma 1 · pp. 30–31
• Because the bound is uniform over H_new, it applies to data-dependent choices including the empirical jumpboard model and the final selected model.The result supplies the generalization quantities used in the main theorems.Source: F Proof of Lemma 1 · p. 31
• Lemma 1 controls generalization uniformly over a norm-bounded class rather than modeling training dynamics.Fixed-statistics BatchNorm can be incorporated into the constants, whereas mini-batch-dependent training-time BatchNorm requires additional assumptions.Source: F Proof of Lemma 1 · p. 31

Controlled-architecture regime

• When architecture factors are fixed or uniformly controlled, the required sample size is nearly linear in d up to logarithmic factors.In this restricted regime, Γarch(L, N, m) = O(log m).Source: §F.1 · p. 32

Joint scaling regime

• If depth L, width N, or normalization constants scale, Γarch includes genuine architecture-dependent amplification through Λmax.When Λmax grows exponentially in L, log Λmax contributes linearly in L.Source: §F.1 · p. 32
• With exponential growth of Λmax, the data requirement scales like log m + L + lower-order architecture terms rather than merely d log m/ε2.This shows why the controlled-architecture shorthand does not capture the full joint-scaling requirement.Source: §F.1 · p. 32

Architecture-dependent data requirement

• Data controls the statistical cost of depth and width expansion through both effective dimension d and architecture factor Γarch.An equivalent effective statistical dimension incorporates the architecture-dependent amplification factor.Source: §F.1 · p. 32

Step 2: Construct a deterministic positive margin at the population level

• A deterministic positive population-risk margin is constructed along a jump path, yielding R(˜fpop) < R(f∗old).The proof uses the population descent direction and chain rule to establish the strict inequality.Source: G Proof of Theorem 2: Population-Risk… · p. 33

Step 3: Transfer the population margin to the test set by Hoeffding

• Hoeffding’s inequality transfers the population margin to empirical and test risks for the fixed old and jumpboard models.The transfer uses bounded losses and a union bound over the relevant concentration events.Source: G Proof of Theorem 2: Population-Risk… · p. 33

Step 4: Move from the jumpboard model to the trained new model

• The trained expanded model is linked to the jumpboard model through the hypothesis-class comparison and the optimization assumption.Because both models lie in Hnew, Lemma 1 and the optimization condition provide the required bridge from the constructed model to fnew.Source: G Proof of Theorem 2: Population-Risk… · p. 34

Step 5: Close the whole chain

• Intersecting the concentration and optimization events closes the chain and yields the theorem’s final high-probability comparison.The proof also accounts for the total failure probability and notes that the two stated consequences follow immediately.Source: G Proof of Theorem 2: Population-Risk… · p. 34

H Proof of Theorem 3: Directly on train/test risks.

• Theorem 3 remains meaningful in the degenerate regime where the test-side jump margin vanishes.The zero residual block recovers the old model inside the expanded class, and the theorem still retains its stated inequality and strict-improvement claim when the margin is positive.Source: H Proof of Theorem 3: Directly on… · pp. 35–36
• Theorem 3 combines uniform train/test comparisons with optimization gain to prove a strict test-risk improvement for the expanded model.The proof applies the comparison bound to training and test samples, transfers the jumpboard gain to the optimized model, and combines the inequalities.Source: H Proof of Theorem 3: Directly on… · p. 35
• The direct guarantee holds with probability at least 1−2δ, or 1−3δ when the optional width-alignment event is included.The extra δ supports the nondegenerate interpretation involving a positive test-side jump margin.Source: H Proof of Theorem 3: Directly on… · p. 36
• For sufficiently small jump size, positive alignment µ^Tg implies a positive finite-test margin, with η<10^-3 identified as an empirically validated first-order regime.The margin has expansion ηµ^Tg+o(η), so its sign is determined by the alignment term in the small-step regime.Source: H Proof of Theorem 3: Directly on… · pp. 36–37
• Width alignment makes a training-improving jumpboard direction test-favorable, and the probability of a positive test-side jump increases with width N.When train-side and test-side first-order directions align, sufficiently small jumps yield lower test loss than the old model; simultaneous alignment and generalization events give probability 1−3δ.Source: H Proof of Theorem 3: Directly on… · p. 37

I Proof of Theorem 1

• Under the population condition, Theorem 1 supplies a population jumpboard model with strictly lower risk than the old model.The proof defines the deterministic population margin as R(f*old)−R(f̃pop)>0.Source: I Proof of Theorem 1 · pp. 37–38
• Uniform generalization transfers the population jumpboard advantage to the data-dependent final model, proving strict improvement with the theorem’s conservative probability 1−2δ.The uniform event applies simultaneously to the population comparison model and the final selected model, while the proof notes that 1−δ already suffices.Source: I Proof of Theorem 1 · p. 38

J Discussion: relation between Theorems 2 and 3

• Theorem 2 is sharper in regular regimes, while Theorem 3 remains meaningful when the population margin ΔR becomes degenerate.Theorem 2 pays only 2ϵM, whereas Theorem 3 pays 2(ϵM + ϵK) but requires only that ΔERM dominate the combined cost.Source: J Discussion: relation between… · p. 39

K Properties Near the Deepest-Model Regime

• Near the deepest model, depth’s remaining gain weakens, so width increasingly determines whether the first-order signal remains observable under finite-test sampling.The generalization cost is governed by the springboard subspace rather than full-network complexity, while larger width stabilizes gradient alignment.Source: K Properties Near the Deepest-Model… · p. 40

L Removing Assumption 4 via Engineering ε

• Engineering ε removes the normalization singularity, allowing Assumption 4(N) to be dropped from Lemma 1 with constants depending on εeng.The result applies to stabilized RMSNorm and LayerNorm; BatchNorm is covered only in fixed-statistics evaluation mode, not training-time mini-batch dynamics.Source: L Removing Assumption 4 via… · pp. 40–42

M From Qualitative Scaling to a Power-Law Form

• The power-law appendix does not derive scaling unconditionally; it adds model-dependent assumptions describing how first-order improvement decays near irreducible risk.The main theorems supply the improvement mechanism, while the added assumptions specify its decay shape.Source: M From Qualitative Scaling to a… · p. 43

M.1 Depth–width coupling and parameter count

• The depth–width coupling L = u(N) defines the reliably supported depth at width N and induces an eventually increasing parameter budget P(N), with supported depth L(P).No specific form of u is assumed beyond u(N) →∞.Source: §M.1 · p. 43

M.3 Additional assumptions for a power-law form

• Assumptions G and G’ link remaining excess risk to the weakening population gradient signal and convert that signal into a one-step depth-expansion risk decrease.Assumption G’ supplies a quadratic-gradient-scale gain up to curvature constant q.Source: §M.3 · p. 45

M.4 Recursion and its power-law upper envelope

• Under β = 0 the recursion gives exponential decay, whereas β > 0 yields a power-law upper envelope rather than an asymptotic equivalence without an additional gain lower bound.For polynomial sublinear coupling, the parameter-count exponent is a_PL(ν) = (1 − ν)/((3 − ν)β).Source: §M.4 · pp. 45–46

M.5 Reliability constraint on the coupling function

• The coupling function must satisfy finite-sample reliability constraints, because alignment noise must remain small relative to the decaying k-th expansion signal.Using Δk = O(k^-1/β), the observable signal may decay according to the corresponding power-law scale.Source: §M.5 · p. 47
• As supported depth u(N(P)) grows, width and data must grow sufficiently to keep the remaining first-order signal observable and the alignment-failure probability small.This width-supported condition is a finite-sample observability requirement, distinct from the population coupling used in the power-law recursion.Source: §M.5 · p. 47

M.6 Interpretation

• Theorems 2 and 3 give a qualitative mechanism in which depth expansion creates improving directions, optimization converts them into training gains, and generalization determines whether test gains survive.The parameter-count form applies these conclusions along coupled paths L(P) = u(N(P)), with separate population-risk and direct train/test guarantees.Source: §M.2 · p. 43, §M.6 · p. 47
• Under additional assumptions on gradient–risk coupling and depth–width coupling, the remaining excess risk obeys Δ_k = O(k−1/β), so the power-law exponent is not universal.The exponent depends on β and u; the main theorems establish the qualitative mechanism, while the power-law form requires the additional assumption-based route.Source: §M.6 · p. 47

N.1 Model Architecture

• The accompanying experiments use CIFAR-scale ResNets with depth d = 6n + 2 and systematically varied base channel width C, under specified SGD, deterministic-evaluation, and gradient-covariance settings.The architecture sweep covers depths from 8 to 104 and base channels from 4 to 64; the reported settings include momentum 0.9, weight decay 10−5, and fixed random seeds.Source: §N.1 · p. 48, §N.2 · p. 48, §N.3 · p. 48

O.1 Depth–width behavior in self-attention

• Levine et al. [Levine et al., 2020] provide indirect Transformer evidence that added depth becomes less effective without sufficient width, supporting coupled depth–width scaling rather than direct validation of the residual-network bounds.Their fit suggests about 30K width with depth around 95 for a projected 1-trillion-parameter self-attention model.Source: §O.1 · pp. 48–49

O.2 Data–parameter coupling in Transformer language models

• Hoffmann et al. [Hoffmann et al., 2022] find that compute-optimal Transformer scaling increases model parameters and training tokens in approximately equal proportions, consistent with data paying for increased statistical complexity.They report that Chinchilla, a 70B-parameter model trained on 1.4T tokens, outperforms Gopher and several larger models at approximately comparable training compute.Source: §O.2 · pp. 49–50

O.3 Model-growth evidence from LiGO

• LiGO shows that structured growth initialization can reuse a smaller pretrained Transformer, aligning with the theory’s structured jumpboard connection between old and expanded models.LiGO factorizes learned parameter mappings into width-growth and depth-growth operators, but it does not directly implement or verify the residual-block insertion theorem.Source: §O.3 · pp. 50–51