The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks

• Existing gradient diagnostics lack consensus because gradient length, component magnitudes, and Jacobian eigenvalues can motivate different architecture strategies.Source: §1 · p. 1
• The NLC is a gradient-based measurement of neural-network nonlinearity.Source: §1 · p. 1
• The NLC computed at random initialization predicts test error, with a right-sized value associated with optimal performance in fully connected feedforward networks.Source: §1 · p. 1
• Unlike existing gradient metrics, the NLC is designed to resist confounding changes in input scale, input bias, and input dimensionality.Source: §1 · p. 1
• The study evaluates 55,000 full training runs across a wide range of randomly sampled architectures.Source: §1 · pp. 1–2
• The NLC is tied to linearly approximable regions, activation-function linearity, and sensitivity to small random input perturbations.Source: §1 · p. 2

• The network maps an input vector x to an output vector f(x), with classification error minimized in expectation over the data distribution.Source: §2 · p. 2
• The Jacobian J is the derivative of the network output with respect to the input.Source: §2 · p. 2
• The analysis uses the data-input mean and covariance together with the induced network-output mean and covariance.Source: §2 · p. 2

• Linear models are globally determined by one Jacobian and output value, whereas nonlinear models have bounded regions where local linear approximations remain accurate.Source: §3 · p. 2
• Nonlinearity is formalized as the fraction of an input-direction distance over which a local linear approximation stays within a specified output tolerance.Source: §3 · p. 2
• The practical construction samples inputs from the data distribution, output directions from a unit Gaussian, and input displacements from a Gaussian fitted to input covariance.Source: §3 · p. 2
• The nonlinearity distribution assigns each sampled case the smallest factor C meeting the approximation criterion, with linear functions attaining C = 1.Source: §3 · p. 2
• The nonlinearity distribution is costly and numerically difficult to compute, motivating a scalar approximation.Source: §3 · pp. 2–3
• The NLC approximates nonlinearity through the ratio of Jacobian-induced output changes to distance from the output to the codomain boundary.Source: §3 · p. 3
• For a linear function Ax + b, the NLC equals 1 and can also be interpreted as normalized sensitivity to small random input changes.Source: §3 · p. 3
• The NLC can be computed simply and cheaply, with the full algorithm deferred to the appendix.Source: §3 · p. 3

4. On the predictive power of the NLC - large-scale empirical study

• The study sampled 750 architectures—250 each across MNIST, CIFAR10, and waveform-noise—by varying depth, initial weight and bias scales, activations, normalization, and skip connections.Each architecture was studied with a single random initialization, reflecting a preference for more architectures over multiple initializations under a limited computational budget.Source: §4 · p. 3
• The randomly initialized NLC closely matches the median nonlinearity distribution, supporting it as an accurate estimate of network nonlinearity.This comparison is shown in Figure 1.Source: §4 · pp. 3–4, Figure 1 · p. 3
• An initial NLC approximately between 1 and 3 is associated with optimal test error across MNIST, CIFAR10, and waveform-noise, although some architectures near this range still perform poorly.The relationship also closely matches results obtained with Adam and SGD on waveform-noise.Source: §4 · pp. 3–4
• Before-training NLC significantly predicts after-training NLC, but after-training NLC correlates less with test error; very high initial NLC often remains high.Unless the after-training NLC lies in a narrow range, test error is close to random.Source: §4 · p. 4
• High-NLC architectures can still reach zero or near-zero training error, indicating that NLC predicts generalization rather than necessarily trainability.Some trainable high-NLC networks required very small learning rates and parameter updates, including one learning rate of 4 ∗10−19.Source: §4 · p. 4

Further predictors: output bias and skip connections

• Low initial output bias is required for optimal test error, while skip connections lower NLC and test error at a given NLC level.The study reports that high output bias universally damages both trainability and generalization, whereas skip connections are associated with lower NLC values.Source: Further predictors: output bias and… · p. 5

5. On the linear approximability of activation functions

• Nonlinearity compounds exponentially across layers, and the network NLC closely tracks the activation function used.For most tested activations, 49-layer batch-normalized networks matched the exponentiated single-activation NLC.Source: §5 · p. 5
• Square and odd-square activations depart from the expected NLCdepth−1 relationship at high depth because they are unstable.Some inputs grow with depth while most collapse toward the zero vector.Source: §5 · pp. 5–6
• For ReLU, SELU, tanh, sigmoid, and Gaussian activations, linear approximation error is approximately NLCτ −1.The relationship breaks down for the three most nonlinear activations, but their approximation errors remain larger than those of the other five.Source: §5 · p. 6
• The NLCτ provides a meaningful activation-level nonlinearity measure that can calibrate an architecture through activation-function linear approximability.The study compares each activation's best linear fit under unit Gaussian input with its NLCτ.Source: §5 · pp. 5–6

6. On the robustness of the NLC vs other metrics - and related work

• Gradient-vector component size, Lipschitz-related measures, and Jacobian metrics are also presented as susceptible to confounders similar to those affecting GVCS.The paper positions the NLC as designed to avoid these confounders.Source: §6 · pp. 6–7
• Input scaling, loss scaling, and input dimensionality can change GVCS while preserving learning dynamics or test error, making it unreliable as a predictor.The paper reports these confounders in figures 4A–C.Source: §6 · pp. 6–7
• As plain He-initialized ReLU networks become deeper, NLC and test error increase while GVCS remains stable, showing that stable gradients do not ensure high performance.The authors attribute the degradation partly to decreasing Tr(Covf).Source: §6 · p. 6, Figure 4 · p. 7
• Shorter periods make sawtooth activations increasingly nonlinear: NLCτ diverges while GVCS remains stable because the derivative magnitude stays one.The corresponding two-layer experiments are shown in figure 4F.Source: Figure 5 · p. 6, §6 · p. 7
• Correlation-based metrics can be arbitrarily altered by adding an input bias without changing learning dynamics when batch normalization removes that bias.The paper illustrates this with five-layer ReLU batch-normalized networks in figure 4D.Source: §6 · p. 7
• Correlation depth scale can miss failures in finitely deep networks because sub-exponential asymptotic convergence may still be fast over the depths used in practice.Both plain ReLU and short-period sawtooth networks are described as lying at the edge of chaos despite poor finite-depth performance.Source: §6 · p. 7
• The study reports a positive correlation between depth and CIFAR10 test error, which the authors suspect is mainly due to larger NLC in deeper networks.The depth-versus-test-error comparison is shown in figure 6.Source: §6 · p. 8, Figure 6 · p. 8
• Compared with prior large-scale predictors, the NLC enables test-error prediction before training, when the eventual trained weights are unavailable.The authors describe this as both more practically useful and more difficult than post-training prediction.Source: §6 · p. 8

7. Discussion and conclusion

• The NLC is presented as a standalone predictor of test error that is conceptually grounded, stable during training, cheap to compute, and robust to common metric confounders.The authors propose using it to discard sub-optimal architectures before training in fully-connected feedforward networks.Source: §7 · p. 8
• Excessive output bias and skip connections contribute independently to performance alongside achieving a right-sized NLC.The conclusion treats these factors as distinct from the NLC criterion.Source: §7 · p. 8
• Overfitting is associated with nonlinearity rather than depth or parameter count, while very high output sensitivity can harm generalization without necessarily preventing trainability.The authors attribute the trainability distinction partly to extensive learning-rate search and 64-bit training.Source: §7 · p. 8
• The paper argues that exponential expressivity is largely harmful in its experiments because it produces poor generalization rather than greater trainability.The authors conjecture that great depth may provide little practical benefit in fully-connected networks.Source: §7 · p. 8
• The paper identifies unresolved scope questions, including whether the observed NLC range generalizes to larger datasets and convolutional or densely connected networks.It also reports that empirical NLC measurements were too noisy to conclusively detect an underfitting regime.Source: §7 · p. 8

A.1. 1D toy example

• In the 1D illustration, the NLC generalizes the number of local linear regions needed to cover the input domain, using covariance matrices and the Jacobian in multiple dimensions.The construction measures diameter ratios rather than volume ratios, with coverage scaling informally as NLC^d.Source: §A.1 · p. 11

A.2. 2D neural network example

• For batchnorm-ReLU networks, increasing depth and NLC makes outputs change more rapidly across the input sphere, implying smaller linearly approximable regions.Table 2 visualizes these functions using colors assigned to normalized three-dimensional outputs; purer colors correspond to more confident three-class predictions.Source: §A.2 · p. 11, Table 2 · p. 12

• The section provides additional results beyond the main empirical-study findings.It points readers to the detailed analyses that follow.Source: B. Large-scale empirical study -… · p. 12

• The study trained 750 architectures with 40 geometrically spaced starting learning rates each, using architecture-specific ranges intended to contain the ideal rate.The smallest rate was selected so updates remained meaningful in 32-bit precision.Source: §B.1 · p. 12
• The validation-error histogram shows a wide selected-index range but a margin of never-selected runs on both sides, supporting that the search usually found each generalizing architecture’s ideal learning rate.Architectures failing to achieve under-80% CIFAR10 test error were omitted from this check.Source: §B.1 · p. 12
• For waveform-noise retraining without early stopping, the selected learning-rate index range was wider, and three architectures required rates outside the original 40-run range.This experiment used 60 training runs and minimized training classification error.Source: §B.1 · p. 12
• Some architectures could be trained successfully only with learning rates as small as 4 ∗10^-19, especially when their randomly initialized NLC was high.The figure compares NLC with rates selected for minimum validation error on CIFAR10 or minimum training error on waveform-noise.Source: §B.1 · p. 12
• As NLC grows, the learning rate appears to decay approximately as the inverse square of NLC because larger NLC is associated with larger gradients requiring smaller bounded updates.The observation mirrors Philipp et al. (2018), who related update magnitude inversely to gradient growth.Source: §B.1 · p. 12

B.2. The importance of avoiding excessive output bias

• High output bias before training leads to high test error, and reducing it toward 1 is necessary for better-than-random CIFAR10 performance.The analysis uses output bias and NLC values computed on the training set.Source: §B.2 · p. 12, Figure 11 · p. 14
• The output-bias analyses use frequency plots of selected training runs for CIFAR10 validation error and waveform-noise training error, excluding architectures at or below better-than-random thresholds.The thresholds were 80% for CIFAR10 and 50% for waveform-noise.Source: Figure 9 · p. 13, Figure 10 · p. 13
• High output bias independently predicts test error and makes architectures completely untrainable on waveform-noise in the reported 60-rate retraining experiment.This contrasts with some high-NLC architectures that still achieved zero or near-zero training error.Source: §B.2 · p. 14
• The authors were developing an improved SGD method intended to train and generalize high-output-bias architectures, but that algorithm and related signals are outside this paper’s scope.This qualifies the reported training limitation as specific to the methods studied here.Source: §B.2 · p. 14

B.3. The value of using skip connections

• Architectures with skip connections exhibit lower NLC overall and tend to outperform non-skip architectures with similar NLC.Separate plots were provided because color differences were difficult to distinguish in the combined scatter plot.Source: §B.3 · p. 14

• Depth varied over odd values from 3 to 49, while width was chosen so each network had approximately 1 million trainable parameters.Source: C. Architecture sampling · p. 14
• Orthogonal weight initialization and signal-preserving scaling were used to avoid rapid forward-pass growth or decay, while additional variations broadened the NLC range.Source: C. Architecture sampling · pp. 15–16
• Architectures varied activation functions, normalization, bias initialization, activation dilation and shifts, and skip-connection presence, strength, and location.Source: C. Architecture sampling · pp. 15–16
• 250 architectures were sampled independently for each of three datasets, with results reported across multiple figures.Source: C. Architecture sampling · p. 16
• The loss input was normalized by its post-initialization activation scale so softmax cross-entropy dynamics would not confound architecture comparisons.Source: C. Architecture sampling · p. 16

D.1. Selection

• MNIST and CIFAR10 provide small image benchmarks, while waveform-noise adds a three-class dataset with 40 input features.MNIST and CIFAR10 have 10 classes; waveform-noise was selected after filtering UCI datasets for balance, size, feature count, and feature type.Source: §D.1 · pp. 16–17, §D.2 · p. 17

D.2. Processing

• CIFAR10 and MNIST are normalized, reduced by PCA to 810 and 334 dimensions, and mapped through random orthogonal projections to reduce first-layer computation.Preliminary experiments found this preprocessing faster and lower-error than training on raw data with only feature normalization.Source: §D.2 · p. 17
• The datasets use train-validation-test splits of 50,000/10,000/10,000 for MNIST, 40,000/10,000/10,000 for CIFAR10, and 3,000/1,000/1,000 for waveform-noise.Source: §D.2 · p. 17

E.1. NLC vs nonlinearity study (figure 1)

• NLC and median nonlinearity are estimated from random batches and Gaussian directions, using batch-aware derivatives and 1,000 sampled configurations per architecture.The reported value is the median across 10 batches and 10 random U and V draws, with batch size 250.Source: §E.1 · p. 17

E.2. Predictiveness study (figures 2, 3, 6, 9, 10, 11 and 12)

• NLC-versus-error analyses compare randomly initialized and trained networks across architectures, learning rates, optimizers, skip connections, output bias, and depth.Training uses independently tuned learning-rate schedules; additional waveform-noise runs use broader learning-rate ranges to include architectures with very high NLCs.Source: §E.2 · p. 18
• The learning-rate search is heuristic but was checked so that the best rate was not selected from the five smallest or largest candidates for any non-random-error architecture.This validation held across all architectures and datasets.Source: §E.2 · pp. 18–19

E.3. Error robustness study (figure 3)

• The error-robustness measure finds the largest perturbation radius whose pathwise test error stays within 5% of the original error.It uses 10 random batches and 10 Gaussian direction matrices, reporting the median over 100 values.Source: §E.3 · p. 19

E.4. Approximability study (table 1)

• Approximability values are reported as medians over 10 random initializations, with little initialization variation except for 49-layer square or odd-square activation networks.Source: §E.4 · p. 19

E.5. Confounder study (figure 4)

• The confounder study found unchanged test error and learning trajectories under input scaling, loss scaling, and additive input bias, with only slight change under replicated dimensions.When the first layer was frozen in the additive-bias condition, overall error was slightly elevated.Source: §E.5 · p. 19

F. Computing the NLC

• For networks without batch normalization, NLC is estimated by exact output-mean and covariance calculations plus stochastic Jacobian projections from random inputs and output directions.The procedure uses batches of 250 and 100 independently drawn batches for the stochastic average.Source: F. Computing the NLC · p. 20
• One-pass variance estimates can suffer numerical underflow when output bias is high, so accurate computation may require two passes and double precision.The study reports that architectures with output bias up to 10^16 require the two-pass method; batch normalization keeps output bias near 1 initially and small during training.Source: F. Computing the NLC · pp. 20–21, G. Computing output bias · p. 21
• With batch normalization, NLC is generalized to batches and therefore depends on batch size, which is fixed at B = 250 in these experiments.The generalized definition reduces to the original one when batch normalization is absent.Source: F. Computing the NLC · pp. 20–21