Sensitivity and Generalization in Neural Networks: an Empirical Study

Roman Novak; Yasaman Bahri; Daniel A. Abolafia; Jeffrey Pennington; Jascha Sohl-Dickstein

Source-linked AI summary

Sensitivity and Generalization in Neural Networks: an Empirical Study

Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein

arXiv:1802.08760v3Feb 23, 2018stat.MLcs.AIcs.LGcs.NE

TL;DR

The paper asks why larger over-parameterized neural networks can generalize better despite classical complexity measures favoring smaller models. It empirically studies sensitivity metrics across fully-connected image-classification networks and finds that robustness near the training data manifold, especially measured by the input-output Jacobian norm, correlates with generalization.

Problem
Larger over-parameterized neural networks often generalize better than smaller ones, challenging classical notions of function complexity.
Method
The study evaluates linear-region and input-output Jacobian sensitivity metrics across fully-connected networks, architectures, optimizers, hyperparameters, and image-classification settings.
Results
The input-output Jacobian norm correlates with generalization, with trained networks more robust near the training data manifold and sensitivity changes accompanying generalization changes.
Takeaways & Limitations
Sensitivity provides an empirical way to relate local function geometry and robustness to generalization in image classification.
Takeaways & Limitations
The investigation is limited to image classification and fully-connected architectures, with extension to more complex architectures left for future work.

Abstract

from arXiv · show

In practice it is often found that large over-parameterized neural networks generalize better than their smaller counterparts, an observation that appears to conflict with classical notions of function complexity, which typically favor smaller models. In this work, we investigate this tension between complexity and generalization through an extensive empirical exploration of two natural metrics of complexity related to sensitivity to input perturbations. Our experiments survey thousands of models with various fully-connected architectures, optimizers, and other hyper-parameters, as well as four different image classification datasets. We find that trained neural networks are more robust to input perturbations in the vicinity of the training data manifold, as measured by the norm of the input-output Jacobian of the network, and that it correlates well with generalization. We further establish that factors associated with poor generalization $-$ such as full-batch training or using random labels $-$ correspond to lower robustness, while factors associated with good generalization $-$ such as data augmentation and ReLU non-linearities $-$ give rise to more robust functions. Finally, we demonstrate how the input-output Jacobian norm can be predictive of generalization at the level of individual test points.

1 INTRODUCTION

The paper examines why larger over-parameterized networks can generalize better than smaller ones by studying sensitivity near the data manifold. It finds that input-output Jacobian norms correlate with generalization across varied experiments.

Large neural networks often generalize better than smaller ones, contrary to classical complexity measures and Occam’s razor.
Trained networks behave more robustly near the training data manifold than away from it.
The study analyzes sensitivity through two metrics and finds that the input-output Jacobian norm correlates with generalization across many scenarios.
The experiments compare sensitivity across training-data regions, single-hyperparameter changes, broad model settings, and individual test points.

2 RELATED WORK

Related work frames sensitivity as an alternative to parameter-count complexity and connects robustness to generalization, regularization, and optimization. The paper contributes large-scale empirical evidence for these connections in fully-connected networks.

The paper evaluates linear-region counts and input-output Jacobian norms as sensitivity measures for comparing fully-connected networks.
Sensitivity can characterize function complexity through input-output variation, complementing parameter counts and linear-region measures.
Extensive experiments show that improved generalization is consistently coupled with better robustness measured by the input-output Jacobian norm.
Random labels increase sensitivity, whereas ReLU networks tend to be more robust than saturating HardSigmoid networks.
Stochastic mini-batch optimization shows an inductive bias toward robustness relative to full-batch training.
The study evaluates the Jacobian metric at scale, including networks up to 65 layers deep and 216 units wide.

3 SENSITIVITY METRICS

The paper defines sensitivity for piecewise-linear fully-connected networks using how outputs respond to input perturbations and how the input space is partitioned into linear regions.

A piecewise-linear network partitions input space into disjoint regions, each implementing a single affine mapping.
The proposed measures capture sensitivity through linear-region structure and the input-output Jacobian of the network.
The metrics ask how network outputs change under perturbations within a linear region.

2. How likely is the linear region to change in response to change in the input?

The paper defines two sensitivity measures: local output change within a linear region and changes in the network’s piecewise-linear regions under input perturbations.

Local sensitivity: The Jacobian norm measures local sensitivity through the Frobenius norm of the class-probability Jacobian with respect to the input.It estimates average-case output sensitivity around a point under infinitesimal Gaussian perturbations and does not require labels.
Region changes: The transition metric detects when perturbations move inputs between distinct linear regions of a piecewise-linear network.Neuron activation codes identify regions; ReLU units use binary codes, while ReLU6 units use three-region codes.
Estimation: Transitions are estimated by sampling trajectories around test points, using horizontal image translations to remain near the data manifold.The same translation strategy is used for corresponding data augmentation experiments.
Curvature interpretation: For sufficiently large sample counts, curvature is approximated from changes in first derivatives along equidistant trajectory samples.This curvature estimate captures nonbinary changes between linear regions but is computationally intensive on densely sampled paths.
Interpretation: The Jacobian norm and transition count qualitatively track first- and second-order terms of the function’s Taylor expansion, respectively.The transition count parallels curvature, whereas the Jacobian norm captures local first-order sensitivity.

4 EXPERIMENTAL RESULTS

Experiments compare sensitivity near and away from the data manifold, across generalization-related factors, broad model sweeps, and individual test points. The input-output Jacobian norm consistently provides the strongest connection to generalization, while transition counts have important limitations.

Sensitivity on and off the training data manifold: Random ellipses probe unfamiliar input-space regions, while ellipses through same- or different-class training points compare sensitivity near and between data-manifold neighborhoods.Different-class interpolations are expected to lie farther from the natural image space in MNIST than same-class interpolations.
Sensitivity on and off the training data manifold: Trained networks are more robust near the training data manifold than away from it, as shown by Jacobian norm and transition density.Training points lie in regions of lower transition density after training.
Sensitivity and generalization factors: Lower sensitivity corresponds to smaller generalization gaps when networks differ through labels, augmentation, nonlinearities, or optimization settings.Jacobian norm shows this pattern consistently, while transitions correlate in almost all considered scenarios.
Sensitivity and generalization gap: The Jacobian norm has a strong relationship with generalization across broad sweeps of architectural and optimization hyper-parameters.The experiments compare many networks trained to perfect or near-perfect training accuracy.
Sensitivity and generalization gap: Transition counts are not sufficient for comparing networks of different sizes because neuron count strongly influences the count.This limitation contrasts with the broad Jacobian-norm relationship reported across datasets.
Sensitivity and per-point generalization: At individual test points, Jacobian norm predicts cross-entropy loss nonlinearly and imperfectly, with a similar but noisier trend for ℓ2-loss.Points with the highest Jacobian norms tend to be mostly misclassified, but some misclassified points have much smaller norms than correctly classified points.

5 CONCLUSION

The paper uses sensitivity measures to study trained fully-connected networks in image classification and reports that input-output Jacobian geometry predicts generalization across varied experiments. Sensitivity also changes substantially with distance from the training data manifold.

Conclusion: The study finds that input-output Jacobian geometry can predict generalization across many image-classification contexts.It also reports drastic variation in sensitivity depending on proximity to the training data manifold.

A.1.1 LINEAR REGION ENCODING

The linear-region encoding assigns codes to points in a network’s piecewise-linear regions, but exceptional configurations can produce duplicate codes.

Linear region encoding: Different linear regions receive different codes, although saturated or exactly cancelling neurons can cause duplicate codes for one region.The paper ignores this case because its probability drops exponentially with network width.

A.1.2 TRANSITION COUNTING

Transition counting relies on sufficiently dense sampling and an unlikely no-simultaneous-transition assumption. In experiments, transition counts generally fail to correlate with generalization across network sizes.

Transition counting: The discrete and continuous transition counts agree when sampling is dense enough to avoid missing narrow linear regions.The encoding must change at most once between consecutive sampled points.
Transition counting: For computational efficiency, the method assumes that no two neurons transition simultaneously.The paper considers simultaneous transitions extremely unlikely under random initialization and stochastic optimization.
Transition counting: Transition counts do not generally correlate with the generalization gap on CIFAR10 or CIFAR100.The plotted networks achieved 100% training accuracy on CIFAR10 and at least 99.9% on CIFAR100.
Transition counting: The transition-density metric uses interpolated horizontal translations to keep sampled points close to translation-augmented data.This construction provides a tractable estimate of transition density around the data manifold.

A.2 DO NEURAL NETWORKS DEFY OCCAM’S RAZOR?

The paper argues that parameter count does not adequately explain neural-network generalization: within the evaluated search space, the best generalization came from a model with about 10^4 times more parameters than the smallest fitting model. It proposes sensitivity-based hypothesis classes as an alternative basis for model comparison, while noting that the observed correlation does not explain why large networks find less-sensitive solutions.

Motivation: Occam’s razor favors simpler models, but neural networks can generalize best at much larger parameter counts than the smallest fitting model.Classical Bayesian reasoning links simplicity to evidence and often approximates it as inversely proportional to parameter count.
Empirical tension: 10^4 times as many parameters produced the best achieved generalization as the simplest fitting model within the evaluated search space.The comparison is explicitly limited to the models evaluated in Figure 1.
Caveat: The paper questions whether Laplace-based evidence and parameter-space volume arguments capture neural-network behavior under stochastic optimization and width growth.The authors identify these as potential issues for the classical Occam analysis.
Alternative complexity view: Sensitivity offers an alternative model-comparison basis because robustness to small input perturbations is easier to justify for natural-image classification than parameter count or Kolmogorov complexity.The paper interprets sensitivity-defined hypothesis classes as having a strongly non-uniform prior.
Interpretation: The authors conjecture that larger networks access more robust solutions in an underdetermined fitting problem, whereas smaller models may be overconstrained, but they state that this needs further investigation.This is presented as a hypothesis rather than an established explanation.

A.3 BOUNDING THE JACOBIAN NORM

This section relates the Jacobian of a network’s output probabilities to cross-entropy loss and derives approximate bounds for Jacobian norms under explicit assumptions. The resulting bounds are compared with empirical data across individual test points and multiple models and datasets.

Target class Jacobian: The analysis relates the derivative of the correct-class output probability to the per-point cross-entropy loss.The correct class is indexed using the target-class vector and the loss is l(x) = −log [fσ(x)]y(x).
Target class Jacobian: The Jacobian expression uses a Hadamard element-wise product between the output-probability term and the logit Jacobian.The notation is simplified by writing fσ(x) as σ and omitting the x argument.
Assumptions and bounds: The approximate relation assumes that individual logit-derivative magnitudes vary little across logits and over the input space.Under this assumption, the target-class relation is simplified and bounded on the probability simplex.
Full Jacobian: The full Jacobian norm can be bounded from below at test time because the target class is unknown, while the upper bound assumes a maximum-entropy distribution over incorrect classes.The assumption is σ_i ≈ (1 − σ_y)/(n − 1) for i ≠ y.
Empirical validation: Figure App.11 compares analytic bounds and a norm approximation with experimental Jacobian measurements across test inputs.The experiment uses 1000 CIFAR10 test inputs from a network with 100% training accuracy, with additional evaluations reported across multiple datasets and models.