PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth Laws for the Precision Floor of Trained Neural Networks: Amplification, Residual Scaling, and a Quantization-Aware Training Paradox

Ahmad S. Tarawneh

arXiv:2609.32060v1cs.LG

TL;DR

The paper asks how network depth changes the precision needed for accuracy to collapse, a question less studied than quantizer and QAT recipes. It relates the precision floor to full-precision predictive amplification and finds that their depth exponents agree across most trained architectures and GPT-2.

  • Problem

    How the precision a network needs depends on its architecture and depth is less understood than quantizer, calibration, and QAT methods.

    Source: §1 · p. 1

  • Method

    A first-order theory relates the precision floor to full-precision predictive amplification, with a linearity check for whether the law applies.

    Source: §3.2 · p. 6, §7 · pp. 21–22

  • Results

    The precision-floor and amplification exponents agree within their 95% intervals in 12 of 13 trained architectures and GPT-2 from 12 to 48 layers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Depth-scaled residual branches and pre-normalisation remove the depth penalty, while the architecture determines whether depth costs precision.

    Source: §7 · p. 21, §8 · p. 23

    The first-order law applies only when amplification is independent of probe-noise level; the two smallest Pythia models fail this check.

    Source: §7 · p. 22

Abstract

from arXiv · show

How many bits does a network need before its accuracy collapses, and how does this grow with depth? We study the precision floor, the perturbation level or bit-width at which accuracy falls halfway to chance, in MLPs, CNNs, Vision Transformers and nine pretrained language models, under post-training quantization (PTQ) and quantization- or noise-aware training (QAT). (i) A first-order theory sets the floor through one full-precision quantity, the predictive amplification $G$: $\eta_c=\Lambda/G$, and $G^2$ grows linearly in depth at a rate proportional to the squared residual branch scale. (ii) The predicted equality $\alpha_{PTQ}=\rho$ of depth exponents holds within 95% intervals in twelve of thirteen trained architectures and in GPT-2 from 12 to 48 layers, with $\Lambda=1.45\pm14\%$ across trained architectures. (iii) Residual branches scaled by $1/\sqrt{D}$ and pre-normalisation remove the depth penalty, and each quantizer turns noise into bits at a rate fixed by its step rule, giving $b_c=(\alpha/\gamma)\log_2 D+C$. (iv) A QAT paradox: noise-aware training roughly doubles the tolerable noise of shallow networks, but the gain decays with depth, so the depth law steepens ($\alpha_{QAT}/\alpha_{PTQ}=1.45$-$1.47$ on two datasets, ten seeds each). Decision margins, cross-layer error cancellation and heavy tails do not set the floor.

1 Introduction

The paper asks how a network’s precision requirement changes with depth and defines the precision floor as the perturbation level or bit-width where accuracy falls halfway from full-precision performance to chance. It links the floor to full-precision predictive amplification and quantizer step rules, then tests this account across architectures and training regimes.

1 Introduction

  • The precision floor is the perturbation level or bit-width where test accuracy falls halfway from its full-precision value to chance, measured across four model families.The study includes trained MLPs, CNNs, and Vision Transformers, plus nine pretrained language models that are only evaluated.Source: §1 · pp. 1–2
  • The floor follows a depth power law whose exponent is set by architecture, while the quantizer’s step rule sets the noise-to-bits conversion rate.The paper’s two-link account runs from architecture to amplification and noise tolerance, then from noise tolerance to bits.Source: §1 · p. 2, Figure 1 · p. 2
  • For small perturbations, predictive amplification G governs output-distribution change; G^2 grows approximately linearly with depth at a rate proportional to squared residual-branch scale, and ηc = Λ/G.The account is first-order and uses a collapse functional to relate amplification to the floor.Source: §1 · p. 2
  • In twelve of thirteen trained architectures and GPT-2 from 12 to 48 layers, floor and amplification exponents agree within 95% intervals; Λ = 1.45 ± 14% across trained architectures.Exponents span −0.57 to +0.32; GPT-2 has Λ = 1.01–1.13.Source: §1 · p. 2
  • Residual branches scaled by 1/√D and pre-normalisation remove the depth penalty in MLPs, CNNs, and ViTs; a fixed-scale control restores it.In a depth-scaled ViT, deeper networks need fewer bits.Source: §1 · p. 3
  • Noise-aware training roughly doubles shallow-network noise tolerance, but its gain shrinks with depth and steepens the QAT depth law relative to PTQ.The reported exponent ratio is approximately 1.5 on two datasets with ten seeds each; about half the decay in a CNN is a training-budget effect.Source: §1 · p. 3
  • Decision margins, cross-layer error cancellation, and activation heavy tails do not set the precision floor.The paper also reports that predictions used as tests were fixed before the corresponding experiment.Source: §1 · p. 3

2 Related work

Prior work connects sensitivity to precision allocation and studies depth-related quantization effects, signal propagation, and error accumulation. This paper’s distinct focus is how amplification and noise-aware training shape the depth scaling of the accuracy-defined precision floor.

Sensitivity and required precision

  • Sensitivity-based precision methods use back-propagated noise gains or Hessian information to choose per-network precision or allocate bits across layers; amplification G is a Monte-Carlo analogue of noise gains.The paper distinguishes its contribution as deriving and measuring how amplification and the floor scale with depth.Source: §2 · p. 3

Depth, quantization and signal propagation

  • Blumenfeld et al. (2019) derive an initialization trainability trade-off, while this paper studies the accuracy-defined floor of trained networks under PTQ and QAT.Both lines of work share the idea that depth multiplies per-layer error; the paper’s growth law also draws on signal-propagation reasoning (Poole et al., 2016; Schoenholz et al., 2017).Source: §2 · p. 3
  • Residual branch scaling is established for stability and infinite-depth limits; this paper tests its precision-floor consequence against a fixed-scale control.The related work contrasts the floor of trained classifiers with expressivity-focused work on depth and low precision.Source: §2 · pp. 3–4

Error accumulation in post-training quantization

  • Prior PTQ studies examine cross-layer error counteraction and propagation; this paper reports no such mechanism in its networks, and heavy tails do not make quantization disproportionately harmful.The paper observes depth-growing heavy tails in residual CNNs, but finds they do not by themselves increase quantization harm disproportionately.Source: §2 · p. 4

Training with noise

  • Earlier noise-aware-training work studies robustness, information recovery, or quantization noise, but does not report how it changes the depth exponent; this paper reports a steeper exponent under QAT.The text identifies this depth-exponent finding as new to its knowledge.Source: §2 · p. 4

3 A first-order theory of the precision floor

The theory links relative perturbations to a precision floor through predictive amplification, then relates that floor to depth and quantizer step rules. It also predicts how depth-dependent tolerance gains from noise-aware training affect the depth law.

3.1 Perturbation model and the precision floor

  • The precision floor is the perturbation level or bit-width where accuracy falls halfway from full precision to chance, with relative Gaussian noise providing a scale-free perturbation model.Noise is added at weight and activation sites; its magnitude is scaled by each tensor’s RMS.Source: §3.1 · pp. 4–5

3.2 Amplification

  • Under norm-preserving propagation, amplification aggregates site sensitivities, and residual networks with fixed branch scale have amplification squared growing with depth at a rate proportional to the squared scale.The coefficient also depends on architecture and width; the predicted exponent is width-stable in the reported MLP comparison.Source: §3.2 · pp. 5–7

3.3 From amplification to the noise floor

  • If logit perturbations are approximately isotropic Gaussian and normalized margins are depth-stable, the floor follows ηc = Λ/G and its depth exponent equals amplification’s exponent.The paper notes that the theory is first-order while collapse occurs at finite perturbation, and treats Λ as an empirical task constant.Source: §3.3 · p. 7

3.4 From noise to bits

  • Under the no-clipping and equal-impact assumptions, the bit-floor depth slope is α/γq, where γq is set by the quantizer’s step rule.Measured γ values are 0.53, 0.55, and 0.57 for LSQ-style MLPs, CNNs, and ViTs, and 1.04 for min–max scaling; real quantization error can be up to about 1.5 times as harmful as Gaussian noise in CNNs and ViTs.Source: §3.4 · pp. 7–8

3.5 Noise-aware training

  • The QAT tolerance gain summarizes the ratio of QAT to PTQ floors; when that gain decreases with depth, QAT steepens the depth law.The paper reports that the latter pattern is observed.Source: §3.5 · p. 8

4 Experimental protocol

The protocol varies model depth at fixed width across MLP and CNN architectures, with selected normalized CNN variants. It compares test-time perturbation or quantization against training with the same perturbation, and estimates floors and depth exponents from accuracy measurements.

Architectures

  • MLPs and CNNs are studied across depth at fixed width, including plain and residual CNNs and two separately tested normalized CNN variants.The CNN set also includes plain models with Dirac identity initialization; the main architectures omit normalization layers.Source: §4 · pp. 8–9

Architectures and preregistration

  • The protocol records the tested depth values for each architecture in Table 1, and the quantitative predictions and selected decision rule were preregistered before their corresponding runs.Source: Table 1 · p. 9, §4 · p. 10

Perturbations and training regimes

  • Gaussian perturbations and quantization act at weight and activation sites, while PTQ applies them only at test time and QAT applies them during training and testing.Noise-aware comparisons use ten MLP seeds; CNN results use three, except the two-seed 30-epoch control.Source: §4 · pp. 9–10

Measurements

  • Noise floors are estimated from accuracy-versus-noise measurements, bit floors by interpolation across tested bit-widths, and depth exponents by log-log slopes.Amplification is estimated from noise draws on 2,000 test inputs; the reported linear-regime check gives G(0.001)/G(0.01) of 0.98–1.06 across architecture means.Source: §4 · p. 9

5 Results

Across trained architectures, amplification generally predicts how the PTQ noise floor changes with depth; residual scaling and pre-normalisation yield nearly depth-independent floors, while the tested ViTs closely follow the same law.

5.1 Amplification follows the growth law

  • G^2 grows approximately linearly with depth in MLPs, with slope proportional to s^2; depth-scaled branches show no growth.The corresponding amplification exponents are 0.55 ± 0.05 for plain MLPs, 0.29 ± 0.07 for fixed-scale residual MLPs, and −0.06 ± 0.04 for depth-scaled MLPs.Source: §5.1 · p. 10, Figure 2 · p. 11

5.3 Residual scaling removes the depth penalty

  • Depth-scaled residual branches make the MLP noise floor nearly depth-independent, unlike the fixed-scale control.Under PTQ, the depth-scaled floor rises from ηc = 0.38 at D = 8 to 0.41 at D = 64; the fixed-scale floor falls from 0.38 to 0.20.Source: §5.3 · p. 11, Figure 4 · p. 13

5.2 Amplification predicts the depth exponent

  • In twelve of thirteen trained architectures and GPT-2 from 12 to 48 layers, the 95% intervals for αPTQ and ρ overlap; the width-32 plain MLP is the exception.The post-norm CNN agrees only within a wide interval because its deepest models are undertrained.Source: §5.2 · pp. 10–11, Figure 3 · p. 12

5.4 Normalisation

  • The pre-norm residual CNN has a flat PTQ floor and stable accuracy across depths, while the post-norm plain CNN attenuates slightly after training.The post-norm network is undertrained at large depth, with accuracy 0.54 at D = 32 and 0.27 at D = 64.Source: §5.4 · pp. 13–14

5.5 Transformers

  • All three ViT variants closely follow the amplification–floor law, and the depth-scaled ViT’s bit floor falls by 0.7 bits from D = 2 to D = 32.Accuracy remains stable across depth (0.76–0.84), avoiding the undertraining confound in these measurements.Source: §5.5 · p. 14

5.6 Pretrained language models

GPT-2 follows the predictive-amplification noise-floor law from 12 to 48 layers, while pretrained Pythia models reveal a linearity boundary and an unexplained outlier. Quantizer step rules set how noise tolerance translates into bits, and noise-aware training raises shallow-network tolerance but steepens its depth law through a depth-decaying gain.

5.6 Pretrained language models

  • GPT-2’s noise floor doubles from 0.146 to 0.301 as amplification halves from 7.3 to 3.4 across 12–48 layers, with Λ = 1.06, 1.13, 1.01, 1.02 and agreeing exponents.The prediction and measured floor coincide in Figure 5(a).Source: §5.6 · p. 14, Figure 5 · p. 15
  • Two smallest Pythia models fail the noise-independence check, so the law makes no claim about them; Pythia-410M remains an unexplained Λ outlier.Pythia-410M has Λ = 3.7 per tensor and 1.8 with per-channel perturbations.Source: §5.6 · p. 15

5.7 From noise to bits

  • Measured bits-to-noise rates are γ = 0.535 for LSQ-style MLPs, 0.550 for LSQ-style CNNs, 0.572 for signed LSQ-style ViTs, and 1.039 for min–max CNNs.The LSQ-style excess over predictions is attributed to clipping at low bit-widths.Source: §5.7 · p. 16
  • Calibrated MLP bit slopes are 1.24, 0.42, and −0.02 for plain, fixed-scale, and depth-scaled networks, respectively, broadly matching predictions.The unscaled residual CNN is an exception: its LSQ-style slope is +0.30 versus +0.04 predicted, falling to +0.08 after longer training.Source: §5.7 · p. 16

5.8 The QAT paradox

  • QAT raises shallow-network noise tolerance by about 1.8–2.2×, but its gain decays with depth and steepens the floor exponent, including in architectures without a PTQ depth penalty.For plain MLPs, the exponent ratio is 1.47 [1.37, 1.57] on Fashion-MNIST and 1.45 [1.37, 1.54] on MNIST; the ratio is not universal across architectures.Source: §5.8 · p. 16
  • Doubling the unscaled residual CNN training budget from 15 to 30 epochs raises its D = 64 QAT gain to 1.44× and reduces Δα from +0.33 to +0.14, leaving about half the steepening.A training-budget contribution at the largest depths cannot be excluded.Source: §5.8 · p. 18

5.9 Where the QAT paradox comes from

  • QAT lowers every layer’s sensitivity, but its amplification benefit is depth-independent; the depth-decaying collapse factor ΛQAT/ΛPTQ accounts for the gain decay.The collapse factor falls from 1.03 at D = 4 to 0.59 at D = 24; the layer-map factorisation is approximate because networks were trained at different noise levels.Source: §5.9 · p. 19

6 What does not set the floor

The tested alternatives do not account for the precision floor: changing decision margins has limited effect, measured cross-layer errors generally do not cancel, and heavy tails alone do not make quantization disproportionately harmful.

Decision margins

  • Changing median normalized margins by up to 5× moves the PTQ floor by at most 16% and leaves the QAT floor within ±12%; depth changes the floor by more.A margin-corrected predictor performs worse than 1/G alone in eleven of thirteen trained architectures.Source: §6 · pp. 19–20

Cross-layer error cancellation

  • Measured cross-layer cancellation is near zero in MLPs and mostly negative in CNNs, so counteraction does not explain the observed laws.In CNNs, new and inherited errors mostly add rather than cancel.Source: §6 · p. 20

Activation heavy tails

  • Residual-CNN activation kurtosis reaches 100–120 at D = 64, yet heavy tails alone are insufficient to cause disproportionate quantization harm.The excess bit-slope penalty appears only in the unscaled network and disappears with longer training; depth-scaled branches and pre-normalization keep tails lighter.Source: §6 · pp. 20–21

7 Discussion and limitations

At fixed width, architecture determines whether depth costs precision; where it does, full-precision predictive amplification predicts the floor. Depth-scaled residual branches and pre-normalisation remove the penalty, while the QAT depth effect remains tied to nonlinear collapse behavior.

What the laws say, and what they do not

  • ηc ≈ Λ/G estimates the noise floor from a full-precision measurement, with Λ ≈ 1.45 for trained networks and ≈ 1 for GPT-2.The growth exponent is architecture-specific, and Λ can differ across tasks while remaining constant across depth within a task and family.Source: §7 · pp. 21–22

The measurement rule and the validity condition

  • The first-order law applies only when amplification is independent of probe-noise level; the ratio G(0.002)/G(0.01) is a suggested diagnostic.The two smallest Pythia models fail this check, while Pythia-410M passes but has Λ ≈ 2–4 without an explanation.Source: §7 · p. 22

The plain-MLP exception

  • Plain width-32 MLPs have αPTQ = 0.70 versus ρ = 0.55, while at width 128 the gap shrinks to 0.53 versus 0.47.The authors attribute the narrower-network exception to larger output perturbations at the floor, where neglected second-order terms matter most.Source: §7 · p. 22

Towards a theory of the QAT paradox

  • QAT lowers every layer’s sensitivity by a depth-independent total factor, but its collapse constant Λ decreases with depth relative to PTQ.Because Λ is set at collapse, where the first-order expansion fails, explaining the paradox requires a theory of nonlinear breakdown.Source: §7 · p. 22

Relation to residual-free transformers

  • Depth-scaled residual branches and pre-normalisation make precision requirements depth-independent; pre-LN ViTs need no more bits at depth 32 than at depth 2.The authors distinguish this depth effect from the separate quantizer-level harm of heavy tails, seen mainly in undertrained models.Source: §7 · pp. 21–22

Limitations

  • The study’s trained models are small and briefly trained, and the QAT paradox is not established for large pretrained transformers.CNN and ViT results use three seeds, and several exponents use only four or five depths; the GPT-2 comparison also mixes depth and width.Source: §7 · pp. 22–23

8 Conclusion

The paper concludes that precision floors follow architecture-dependent depth laws that can be predicted from full-precision amplification, subject to a linearity check. Its results also identify depth-scaled residual branches and pre-normalisation as ways to remove the depth penalty, while QAT steepens depth dependence.

8 Conclusion

  • Across MLPs, CNNs, Vision Transformers, and pretrained GPT-2 models, the precision floor follows a depth power law whose exponent matches predictive-amplification growth.The collapse constant Λ = ηcG is nearly the same across architectures and depths, supporting prediction from full-precision measurement when the linearity check passes.Source: §8 · p. 23
  • Depth-scaled residual branches and pre-normalisation remove the depth penalty, while noise-aware training raises tolerable noise but steepens its depth dependence.The QAT paradox replicates on two datasets and in a Vision Transformer, survives in part a doubled training budget, and lies in the nonlinear collapse regime rather than amplification.Source: §8 · p. 23
  • The authors identify explaining nonlinear collapse and testing QAT depth laws on large transformers as next steps.Result files, training and measurement code, and figure-, table-, and number-generation scripts are available in the cited repository.Source: §8 · p. 23

Appendix A. Derivation of the growth law (Proposition 2)

The derivation models perturbations at residual-block weight and activation sites, then tracks their propagation through the stream to the logits. Under its stated assumptions, independent site contributions yield depth growth proportional to the squared residual-branch scale.

Appendix A. Derivation of the growth law (Proposition 2)

  • The derivation treats pre-activation residual networks, with plain networks handled as a special case.The stream is hl+1 = hl + s ul, with ReLU branch output and a head applied after the final block.Source: Appendix A · p. 23
  • A relative perturbation at either the activation or weight site changes the branch output by an amount whose expected squared norm is approximately proportional to the branch-output scale.The derivation assumes variance-preserving weights and uses an O(1) constant for the proportionality.Source: Appendix A · p. 23
  • Because branch output and stream are approximately uncorrelated at a variance-preserving operating point, stream norm grows by a factor approximately 1 + s^2r^2 per block.Here r^2 is the branch-output norm divided by the stream norm and is O(1).Source: Appendix A · p. 24
  • The relative perturbation injected into the stream at each block is scaled by the branch-to-stream relation, with squared size involving 1 + s^2r^2.The displayed expression is continued across the supplied derivation passages.Source: Appendix A · p. 24
  • Under Assumption 1, a relative stream perturbation reaches the logits with a block-independent factor, and independent site perturbations make contributions additive.The derivation explicitly relies on independence of the perturbations across sites.Source: Appendix A · p. 24
  • For plain networks without an identity path, the same summation gives G^2 = G0^2 + κD, equivalent to the residual formula with s = 1.The account says each layer re-normalises the signal and that a relative perturbation reaches the logits with factor ch.Source: Appendix A · p. 24
  • With depth-scaled residual branches, the growth term can remain independent of D; with fixed s, it grows linearly in D.The local exponent is obtained by differentiating one-half log G^2 with respect to log D.Source: Appendix A · p. 24

Appendix B. Measurement artefacts and how they were corrected

The appendix records measurement and training artefacts that produced apparent effects, alongside the corrections used for reported results. These include noisy amplification estimates, batch-dependent activation scales, a problematic step rule, undertraining, and an unstable exponent ratio.

Appendix B. Measurement artefacts and how they were corrected

  • Using five noise draws gave ±10–20% run-to-run scatter per model, comparable to the CNN depth trend of ≈25%; reported values instead use 50 draws at three noise levels.The linearity check G(0.001)/G(0.01) = 0.98–1.06 across architecture means supports the small-noise regime.Source: Appendix B · p. 24
  • Batch-specific activation steps made one image’s quantization depend on its batchmates and inflated plain-CNN bit slope (+0.16 versus +0.05 after calibration).Reported bit floors use steps calibrated once on training data.Source: Appendix B · p. 24
  • The LSQ-style mean-based step rule clips heavily on tensors with a few large entries, producing 8-bit relative error 0.11–0.33 instead of 0.03 for Dirac-initialised kernels.The authors therefore report no bit floor for that architecture.Source: Appendix B · p. 24
  • At 15 epochs, the deepest unscaled residual and Dirac CNNs were undertrained, with full-precision accuracy 0.61–0.63, inflating QAT steepening and unscaled-residual-CNN bit slope.A 30-epoch control quantifies both effects.Source: Appendix B · pp. 24–25
  • The ratio αQAT/αPTQ is numerically unstable when αPTQ is near zero, so the authors report Δα as the primary statistic.A preliminary CNN estimate of the ratio was ≈4.Source: Appendix B · p. 25

Appendix C. Width

The width study finds that the linear growth form and depth exponents persist across widths, while the slope κ depends on width and architecture. The plain-MLP agreement between αPTQ and ρ improves with width, and κ ≈2 at width 32 is not width-independent.

Appendix C. Width

  • The linear growth form and depth exponents persist at widths 128 and 512, but the slope κ falls steeply with width.The study repeated MLP measurements at widths 128 and 512; Table 5 reports 95% intervals.Source: Appendix C · p. 25, Table 5 · p. 25
  • For plain MLPs, αPTQ vs. ρ improves from 0.70 vs. 0.51 at width 32 to 0.53 vs. 0.47 at width 128.The authors characterize the plain-MLP exception as largely a narrow-network effect.Source: Appendix C · p. 25
  • The κ ≈2 measured at width 32 is a coincidence; κ is an architecture- and width-dependent constant.The width-32 value matched a naive count of two perturbation sites per layer, but that match does not generalize across widths.Source: Appendix C · p. 25

Related papers

Loading 2609.32060v1…