PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion

Zeyu Liu, Jinhao Zhang, Yunquan Zhang, Guangming Tan, Xiang Gao, Fangming Liu, Daning Cheng

arXiv:2608.14664v1cs.LG

TL;DR

The paper asks how to tell whether a trained network has exhausted the useful first-order value of additional residual depth. It develops an exact tangent-space criterion and finds that low activation-gradient signal aligns with diminishing benefits from further growth.

  • Problem

    Existing growth methods lack an exact criterion for deciding when additional residual depth should terminate.

  • Method

    The paper defines first-order saturation under fixed function-preserving growth and characterizes it through activation-gradient projection onto admissible residual tangent spaces.

  • Results

    Function-preserving growth reaches comparable converged training losses to scratch training, while low-signal regimes align with little or no further performance improvement.

  • Takeaways & Limitations

    Tangent-space projection provides an exact local criterion, while a stable low activation-gradient plateau is a conservative practical indicator of limited residual-growth value.

  • Takeaways & Limitations

    The analysis is restricted to the fixed protocol and excludes adaptive candidate or residual-family selection, Hessian-based updates, nonzero optimizer states, and stochastic escape.

Abstract

from arXiv · show

How can we determine whether a trained neural network is already deep enough? We study this under a fixed function-preserving residual-growth protocol specifying insertion locations, residual families, zero-output initializations, and zero-state first-order updates. We define first-order residual depth saturation as the absence of a strict local decrease from every admissible insertion. We prove residual non-degeneracy is necessary and sufficient: additional depth has first-order value exactly when conditional activation gradients have a nonzero projection onto at least one admissible residual tangent space. This boundary is shared by descent-compatible zero-state updates and invariant under regular local reparameterizations preserving that tangent space. Under residual-signal realizability, raw activation-gradient vanishing exactly certifies saturation. Across ResNets, GPT-2-style models, and continued-pretrained Pythia checkpoints, the maximum activation-gradient norm decreases toward a low-signal regime with depth. Function-preserving growth also achieves converged performance competitive with training from scratch. These results support activation-gradient magnitude as a conservative diagnostic of the remaining empirical first-order value of residual depth.

Introduction

The introduction frames residual depth saturation as a missing termination criterion for adaptive growth and proposes first-order residual directions as the diagnostic. It motivates a fixed, function-preserving protocol and evaluates activation-gradient norms as an upper bound on realizable residual-growth value.

  • Motivation: Increasing depth eventually yields little test-set gain, motivating a criterion for deciding when a trained network should stop growing.Globally zero-output residual blocks preserve the represented function, enabling marginal depth value to be studied at a common initialization.
  • Criterion: The study fixes admissible insertion candidates before observing gradients and introduces first-order residual depth saturation as an operational stopping notion.The protocol is designed to test whether any admissible first-order residual direction remains.
  • Experiments: Activation-gradient norms are measured across depth in ResNets, continued-pretrained Pythia checkpoints, and GPT-2-style models.Orthogonal projection cannot increase norm, so the sample-wise score upper-bounds empirical projected residual-growth value on each fixed sample.
  • Experiments: Additional residual blocks improve shallower ResNets, but realized gains disappear once the sample-wise score reaches a stable low-signal regime.The comparison inserts one globally zero-output block at a time and contrasts function-preserving growth with training identical final-depth architectures from random initialization.
  • Criterion: Residual non-degeneracy is characterized through the parameterization-invariant projection of the conditional activation gradient onto the residual tangent space.For standard zero-output blocks, the parameter criterion reduces to the cross-gradient between the activation gradient and residual features.

Related Work

Prior work develops function-preserving network growth, depth-growth policies, and local insertion scores, while studies of residual redundancy and zero-initialized branches motivate the protocol. This work isolates marginal first-order depth value at a common checkpoint and extends residual non-degeneracy from sufficiency to necessity.

  • Function-preserving operators widen or deepen trained networks without changing their represented function, supporting progressive stacking, parameter reuse, and other efficient Transformer expansion methods.
  • Constructive-learning research schedules depth growth using validation or fitting risk and selects insertions through splitting directions, gradient norms, bottlenecks, derivatives, and natural-expansion scores.
  • Residual-path redundancy, layer pruning, stochastic depth, and benign zero-initialized residual branches motivate studying additional depth through zero-output initialization.
  • The function-preserving setting measures marginal first-order depth value at a common checkpoint, unlike scaling-law comparisons of independently trained models.
  • Cheng et al. show residual non-degeneracy is sufficient for local improvement, whereas this work proves necessity under a fixed zero-state protocol and invariance across optimizers and tangent-space-preserving reparameterizations.

Problem Setup, Assumptions, and Notation

The framework fixes a function-preserving residual-growth protocol before observing activation-gradient signals, then defines first-order improvement and saturation through admissible residual insertions. Its analysis relies on differentiability, integrability, zero-state descent-compatible updates, and residual tangent spaces induced by candidate Jacobians.

  • Protocol: The residual-growth protocol fixes a finite set of insertion candidates, residual families, function-preserving origins, parameterizations, and optimizer direction rules in advance.These components cannot be changed after observing activation-gradient signals; internal direction-rule hyperparameters are fixed, while the positive local step size remains free.
  • Protocol: Each candidate is initialized with a residual function identically zero, so insertion preserves the reference model pointwise.The local coordinate origin θ_l = 0 need not mean every raw residual-branch parameter is numerically zero.
  • Assumptions: First-order analysis assumes measurable sample-wise and population differentiability, finite local objectives, differentiable residual maps, and integrability sufficient to interchange differentiation and expectation.A local integrable Lipschitz envelope is given as a standard sufficient condition for the required interchange.
  • Assumptions: The zero-state update has the form θ_l^+ = ηd_l(g), with d_l(0) = 0 and g^⊤d_l(g) < 0 for every nonzero gradient, ensuring feasible local descent directions.This class includes gradient descent, positive-definite preconditioned gradient descent, and the first bias-corrected Adam or AdamW update at zero state.
  • Notation: Residual tangent spaces are the ranges of candidate Jacobian operators mapping parameter directions into hidden-state perturbation functions, with orthogonal projections defined in population and empirical Hilbert spaces.For standard zero-output blocks h_l(z) = V_lψ_l(z), activating only V_l spans perturbations ΔV_lψ_l(z); varying U_l does not enlarge the first-order tangent space at V_l = 0.

First-Order Residual Depth Saturation

Under the fixed residual-growth protocol, first-order depth saturation occurs exactly when no admissible residual tangent direction can produce a strict local decrease. With residual-signal realizability, raw activation-gradient vanishing is equivalent to saturation, while the exact criterion remains invariant under tangent-space-preserving reparameterizations.

  • Necessary and sufficient characterization: Theorem 1 equates Condition 1, nonzero projected activation gradients, positive growth value, and a locally improving admissible residual insertion.These equivalent statements characterize the exact boundary between first-order residual depth with and without optimization value.
  • Necessary and sufficient characterization: Depth-wide first-order saturation occurs if and only if every candidate is population first-order saturated, equivalently Vdepth = 0.The same equivalence holds for the empirical counterparts at every fixed differentiable sample.
  • First-order scope: The optimizer-realization statement applies specifically to the fixed zero-state update direction and does not rule out stochastic, higher-order, or externally perturbed escape.Its reverse implication uses the zero-state property dl(0) = 0.
  • Parameterization invariance: The growth value depends only on the residual tangent subspace and is invariant under regular local reparameterizations preserving that subspace, unlike the coordinate-dependent Euclidean parameter-gradient norm.The projected activation gradient and residual-parameter gradient share the same zero-versus-nonzero boundary, although their norms generally differ.
  • Activation-gradient diagnostic: Under residual-signal realizability, raw activation-gradient vanishing is equivalent to candidate-wise and depth-wide first-order saturation.Without realizability, activation-gradient vanishing remains sufficient, but the reverse implication requires the realizability condition.

Experiments

Experiments evaluate activation-gradient decline across residual-network and language-model depth, then compare function-preserving growth with scratch training. The results show a persistent low-signal regime associated with minimal marginal depth benefit, while grown models remain competitive after convergence.

  • Activation-gradient signal: Experiments measure maximum per-example activation-gradient norms across ResNets, GPT-2-style models, and continued-pretrained Pythia checkpoints as depth increases.The evaluated datasets include CIFAR-10, CIFAR-100, ImageNet-100, and FineWeb-Edu.
  • Activation-gradient signal: AGmax exactly equals empirical first-order residual-growth value under sample-wise realizability, while otherwise serving as its upper bound.Because AGmax is the maximum across admissible insertion locations, a small value conservatively indicates small projected growth value at every candidate.
  • Depth saturation: AGmax decreases rapidly at shallow and intermediate depths, then enters a persistent low-signal plateau as loss improvements diminish.The transition is broadly consistent across controlled ResNet and GPT-2-style sweeps, with a sharper transition in Pythia and less monotone behavior on ImageNet-100.
  • Depth saturation: Approximately 5 × 10−3 marks the low-signal regime, beyond which further depth yields little or no additional performance improvement.Extended-depth results kept AGmax below approximately 5 × 10−3, while test performance fluctuated without systematic gains from added depth.
  • Function-preserving growth: Across four CIFAR-10 ResNet configurations, average training loss was 0.0066 for Growth and 0.0072 for Scratch.Growth had lower training loss in ResNet-10, ResNet-11, and ResNet-13, while ResNet-12 showed only a small increase relative to Scratch.
  • Function-preserving growth: Growth reduced average test loss from 0.2922 to 0.2576, an approximately 11.8% relative reduction, and improved test loss for all four final architectures.Absolute reductions ranged from 0.0242 for ResNet-12 to 0.0456 for ResNet-10.

Conclusion · A Complete Notation

The paper establishes a first-order criterion for deciding whether function-preserving residual growth remains locally useful: strict improvement occurs exactly when conditional activation gradients project nontrivially onto admissible residual tangent spaces. The notation section clarifies activation-gradient score normalization and its limits as an empirical growth-value proxy.

  • Conclusion: Additional residual depth yields a strict first-order improvement if and only if the conditional activation gradient has nonzero projection onto the corresponding residual tangent space.This criterion applies under the fixed function-preserving residual-growth protocol.
  • Conclusion: For standard zero-output residual blocks, the tangent-space condition reduces to an activation–feature cross-gradient criterion.
  • Conclusion: The framework evaluates whether additional residual depth remains locally useful under a fixed function-preserving growth protocol.
  • A Complete Notation: Tables 2–4 collect the symbols used throughout the main text and supplementary proofs.Dependence on the fixed reference model and residual-growth protocol is suppressed when no ambiguity arises.
  • A Complete Notation: Under finite-sample residual-signal realizability, AGl equals VS,l when both use the same sample normalization.The notation explicitly links the experimental activation-gradient score to the corresponding empirical residual-signal quantity.
  • A Complete Notation: The coordinate-normalized score f AGl preserves the zero-versus-nonzero boundary but can change candidate rankings when hidden dimensions differ.
  • A Complete Notation: The coordinate-normalized score should be treated as a descriptive visualization scale rather than the exact empirical growth value.

B Detailed Experimental Settings … C Preliminary Identities

The experiments vary residual or decoder depth while holding other within-sweep factors fixed, using activation-gradient norms as the primary diagnostic. Preliminary identities establish that directional growth value vanishes exactly when the relevant conditional gradient is orthogonal to the admissible tangent space.

  • B Detailed Experimental Settings: Depth is the controlled architectural variable, while widths, task heads, and remaining hyperparameters stay fixed within each sweep.This protocol is stated for the depth-scaling experiments overall.
  • B.1 Depth Variables and Gradient Measurement: ResNet sweeps add N blocks cyclically across three stages, producing 3+N total residual blocks without changing stage widths or the classification head.The CIFAR architecture uses a ResNet-8 backbone with stage widths 16, 32, and 64 and sweeps N = 1, . . . , 100.
  • B.4 GPT-2 Depth Sweep on Ascend: GPT-2 experiments vary only the number of decoder blocks, with L = 12 as baseline and all other architectural dimensions and per-update optimization settings fixed.The baseline follows GPT-2 Small dimensions, including hidden size 768, feed-forward size 3072, and 12 attention heads; FineWeb-Edu contains approximately 310,000,347 tokens.
  • B.1 Depth Variables and Gradient Measurement: Activation-gradient norms are computed from complete input-activation tensors, flattened per batch, and averaged over B analysis batches.Each analysis batch uses one forward and one backward pass; classifier and language-modeling heads are diagnostic endpoints, not residual-growth candidates.
  • B.2 CIFAR-10 and CIFAR-100: CIFAR-10 and CIFAR-100 use 50,000 training images and 10,000 test images each, with crop, flip, and dataset-specific normalization preprocessing.The tasks use 10 and 100 output classes, respectively.
  • B.3 ImageNet-100: ImageNet-100 retains the CIFAR-style three-stage ResNet-8 stem and initially sweeps N ∈{1, 5, 9, . . . , 49} with base channel count 32.These experiments omit the standard ImageNet 7 × 7 stride-2 convolution and max pooling, and run each configuration for 300 epochs.
  • B.5 Pythia Continued-Pretraining Analysis: The Pythia study measures frozen-model activation-gradient norms after continued pretraining from official step143000 checkpoints, providing cross-model rather than depth-only evidence.It covers six deduplicated pretrained configurations whose architectural dimensions vary in addition to depth.
  • C Preliminary Identities: The preliminary lemmas show that population and empirical directional derivatives vanish exactly when conditional or sample gradients project to zero in the relevant tangent space.Equivalently, the tangent-space growth values V_l and V_S,l vanish when the gradients are orthogonal to the corresponding tangent-operator ranges.

D Proof of the Main-Paper Necessary-and-Sufficient Saturation · D.1 Closed Forms for the Growth Values

The proof establishes that residual non-degeneracy is equivalent to strict first-order improvement from an admissible insertion, while depth-wide saturation occurs exactly when all candidate growth values vanish. Closed forms show these values depend on tangent-subspace geometry, remain invariant under range-preserving reparameterizations, and support only fixed-sample interpolation claims without additional rank-stability assumptions.

  • D Proof of the Main-Paper Necessary-and-Sufficient Saturation: Residual non-degeneracy is equivalent to a nonzero conditional activation gradient, because choosing v = −g yields g⊤v = −∥g∥2 < 0.Conversely, zero gradient prevents strict decrease under descent-compatible zero-state updates.
  • D Proof of the Main-Paper Necessary-and-Sufficient Saturation: Each nonzero candidate gradient yields strict local decrease for sufficiently small positive steps, whereas a zero gradient makes strict decrease impossible.This establishes the local necessary-and-sufficient criterion for each admissible insertion.
  • D Proof of the Main-Paper Necessary-and-Sufficient Saturation: Vdepth = 0 exactly when every candidate Vl = 0, which is equivalent to depth-wide first-order saturation.The same finite-candidate argument yields the empirical threshold result.
  • D Proof of the Main-Paper Necessary-and-Sufficient Saturation: At η = 1/Ll, ordinary gradient descent guarantees a one-step decrease of at least ∥g(l)∥2/(2Ll) under Ll-Lipschitz gradients.The analogous empirical statement uses the empirical smoothness constant LS,l.
  • D.1 Closed Forms for the Growth Values: The growth values admit Gram-matrix representations, with the empirical formula obtained through the corresponding finite-dimensional projection identity.These forms express growth through projection onto the admissible residual tangent space.
  • D.1 Closed Forms for the Growth Values: The geometric value Vl depends only on Range(Al), so regular local reparameterizations preserving that range leave V_l^2 unchanged.For θl = Tαl, the coordinate transformations Al 7→ AlT and Gl 7→ T⊤GlT preserve the represented tangent subspace, unlike ∥g(l)∥2.
  • D.1 Closed Forms for the Growth Values: Under fixed-M sample-wise tangent completeness, the interpolation identity lower-bounds expected residual error by V_l^2, but does not justify convergence as M →∞ with fixed feature dimension.The asymptotic conclusion requires additional rank-stability and complexity assumptions.

E Joint Insertion of Several Residual Blocks

For independently parameterized residual blocks inserted jointly at distinct locations, strict joint first-order descent exists exactly when at least one candidate is residual-nondegenerate. Under the stated differentiability, disjointness, and zero-output assumptions, the joint derivative concatenates the candidate activation-gradient representations.

  • Assumptions: Under joint Fréchet differentiability, disjoint active parameter blocks, and globally zero-output residuals, independently parameterized blocks can be analyzed at their joint origin.Each residual transformation equals the identity at its designated origin, so intermediate-segment differentiability is unnecessary for this pointwise identity.
  • Joint descent criterion: A strict joint first-order descent direction exists if and only if at least one candidate satisfies residual non-degeneracy.Thus joint insertion offers first-order value whenever any admissible insertion remains non-degenerate.
  • Derivative representation: The joint derivative is represented by the concatenated vector of candidate activation-gradient representations, and it has a negative direction if and only if at least one component is nonzero.This nonzero-component condition is equivalent to residual non-degeneracy for at least one candidate under the necessary-and-sufficient saturation theorem.

F Proof of the Main-Paper Activation-Gradient Characterization

The proof connects the activation-gradient characterization to residual tangent spaces at both population and empirical levels. It also shows that standard-block parameterization and approximate residual-signal realizability preserve the relevant first-order criterion.

  • Population and empirical equivalence: The saturation theorem’s population equivalence becomes empirical when qS,l lies in Range(AS,l), because ΠS,lqS,l = qS,l.The empirical equivalence does not require residual-signal realizability.
  • Tangent-space reduction: For the standard block, every matrix direction ∆Vl induces the tangent action Jl(z) vec(∆Vl) = ∆Vlψl(z).This identifies the residual tangent directions used in the characterization.
  • Tangent-space reduction: At Vl = 0, activating Ul adds no first-order directions because its derivative is VlDUlψl,Ul(z) = 0.Thus fixing Ul = U0,l and varying only Vl preserves the full tangent range.
  • Tangent-space characterization: A matrix belongs to the residual tangent range exactly when each row belongs to the row space of Ψ(l).The associated projector is unchanged by the positive rescaling in the tuple–matrix inner-product identification.
  • Approximate realizability: Approximate residual-signal realizability guarantees a nonzero tangent projection when distHl(ml, T (l) is sufficiently small relative to ∥ml∥Hl.The same conclusion holds on a fixed sample using empirical distance and norm, including when feature dimension is smaller than sample size.

G Supplementary Finite-Sample Consistency Result · H More experiments

The supplementary result establishes consistent finite-sample detection of first-order depth saturation under independent probing, while clarifying coordinate dependence and practical testing limitations. Additional experiments probe whether a zero-output residual block can produce an immediate loss reduction through a fitted terminal projection.

  • G Supplementary Finite-Sample Consistency Result: Theorem 3 establishes consistent finite-sample detection of first-order depth saturation under the stated independent finite-sample probing assumption.The detector is consistent relative to the fixed growth protocol.
  • G Supplementary Finite-Sample Consistency Result: The detector’s false-negative probability satisfies Pr(d Satn = 0) = Pr(bΓn > τn) ≤ 1/(nτn^2) under the stated concentration argument.The bound follows from Markov’s inequality applied to the squared norm and a union bound over the finite insertion set.
  • G Supplementary Finite-Sample Consistency Result: For sufficiently large n, choosing τn < Γ/2 separates the empirical detector from a nonzero population signal.On the event bΓn ≤ τn, the relevant empirical gradient norm is controlled below the population signal threshold.
  • G Supplementary Finite-Sample Consistency Result: The zero-versus-nonzero saturation boundary is invariant under regular reparameterization, but Euclidean gradient magnitudes and thresholds depend on coordinates and the fixed parameterization.Thus the consistency theorem should be interpreted relative to the specified parameterization.
  • G Supplementary Finite-Sample Consistency Result: Because exact saturation Γ = 0 is a knife-edge hypothesis, a separated test can compare H0 : Γ ≤ δ with H1 : Γ ≥ δ + 2t using threshold δ + t.When max_l ∥b g(l)∥2 < t, the test is correct under either hypothesis; its error bound requires variance control.
  • G Supplementary Finite-Sample Consistency Result: The consistency theorem requires an independent probe sample because reusing training data generally breaks the conditional i.i.d. argument without stability or sample-splitting assumptions.The fixed-sample equivalences in the main theorems are deterministic and may still be evaluated on training data.
  • H More experiments: On four converged CIFAR-10 ResNets, the experiment inserts a globally zero-output residual block and fits only its terminal projection to a small negative activation-gradient target.The test asks whether this first-order direction produces an immediate reduction in empirical task loss.

H.1 Local Effect of Activation-Gradient-Matched Insertions

Activation-gradient-matched residual insertions produced immediate training-loss reductions in all four converged CIFAR-10 ResNet checkpoints without task-loss fine-tuning. The small, depth-dependent reductions support the predicted local improvement and diminishing first-order value of added depth.

  • Construction: The experiment inserted one standard, zero-initialized residual block whose output matched a small negative activation-gradient perturbation at a candidate location.The reference model was frozen, hidden representations and activation gradients were cached, and only the feature-producing parameters were trained by regression.
  • Immediate effect: 7×10−4, 4×10−4, 2×10−4, and 2×10−4 lower training loss followed insertion for ResNet-8, ResNet-14, ResNet-20, and ResNet-34, respectively.Each fitted model was evaluated immediately, without classification-loss optimization or joint fine-tuning.
  • Immediate effect: The reductions consistently had the predicted sign because the target used activation gradients from the same training samples.This behavior is consistent with capturing a descending component within the tangent family generated by the fixed residual features.
  • Depth trend: The reduction decreased from 7 × 10−4 for ResNet-8 to 2 × 10−4 for ResNet-20 and ResNet-34, qualitatively consistent with diminishing empirical first-order value.The reductions were small, as expected from α = 10−3 and evaluation before task-loss fine-tuning; differing baseline losses and one insertion per checkpoint limit comparisons.
Loading 2608.14664v1…