PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting the Generalization Gap in Deep Networks with Margin Distributions

Yiding Jiang, Dilip Krishnan, Hossein Mobahi, Samy Bengio

arXiv:1810.00113v2stat.MLcs.LG

TL;DR

Deep networks can achieve very small training loss while generalizing poorly, raising the question of how to predict the gap from training data and network parameters. The paper proposes a margin-distribution predictor using normalized margins across multiple layers and finds strong correlation with generalization gap, while noting that compared theoretical bounds target provable upper bounds rather than low expected prediction error.

  • Problem

    Deep networks can fit randomly labeled or corrupted data with very small training loss yet poor held-out performance, making generalization gap prediction from training information an important open question.

  • Method

    The paper approximates decision-boundary distances at network layers, summarizes normalized positive margin distributions, and predicts the gap with a linear model using additive or log-transformed statistics.

  • Results

    The multilayer normalized margin-distribution predictor correlates strongly with generalization gap and significantly outperforms the spectral-complexity normalized-output-margin baseline and recently developed theoretical bounds.

  • Takeaways & Limitations

    Predictive accuracy improves when margins are normalized, characterized by distribution statistics, measured across layers, and combined multiplicatively through log space.

  • Takeaways & Limitations

    The compared theoretical bounds were designed as provable upper bounds rather than estimates with low expected error.

Abstract

from arXiv · show

As shown in recent research, deep neural networks can perfectly fit randomly labeled data, but with very poor accuracy on held out data. This phenomenon indicates that loss functions such as cross-entropy are not a reliable indicator of generalization. This leads to the crucial question of how generalization gap should be predicted from the training data and network parameters. In this paper, we propose such a measure, and conduct extensive empirical studies on how well it can predict the generalization gap. Our measure is based on the concept of margin distribution, which are the distances of training points to the decision boundary. We find that it is necessary to use margin distributions at multiple layers of a deep network. On the CIFAR-10 and the CIFAR-100 datasets, our proposed measure correlates very strongly with the generalization gap. In addition, we find the following other factors to be of importance: normalizing margin values for scale independence, using characterizations of margin distribution rather than just the margin (closest distance to decision boundary), and working in log space instead of linear space (effectively using a product of margins rather than a sum). Our measure can be easily applied to feedforward deep networks with any architecture and may point towards new training loss functions that could enable better generalization.

1 INTRODUCTION

The paper asks how to predict deep-network generalization gaps when training losses remain small despite poor held-out performance. It proposes a normalized, multilayer margin-based measure and reports strong empirical prediction on CIFAR-10 and CIFAR-100.

  • Motivation: Deep networks can fit corrupted data and still generalize poorly, so small cross-entropy loss does not reliably indicate the generalization gap.The gap is the difference between training and test accuracy.
  • Contribution: The proposed measure uses margin distributions from multiple layers of feedforward networks to predict generalization gap.The authors argue that single-layer margins are insufficient and multilayer information significantly improves prediction.
  • Empirical motivation: Normalized margin distributions shift right as test accuracy increases, despite nearly perfect training accuracy and near-zero training loss.Figure 1 compares CIFAR-10 networks with 55.2%, 70.6%, and 85.1% test accuracy; the lowest-accuracy network used 20% corrupted labels.
  • Evaluation: The measure significantly outperforms recently developed theoretical generalization bounds across extensive experiments on CIFAR-10 and CIFAR-100.The compared bounds were designed as provable upper bounds rather than low-error estimates.
  • Scope and design: Unlike many bounds, the proposed measure handles feedforward architectures broadly, including topology changes such as residual connections.The authors contrast this with bounds based primarily on weight norms and architecture-specific treatments.

2 RELATED WORK

Related work connects margin distributions and classifier complexity to generalization, while recent deep-network bounds motivate empirical prediction of the generalization gap. The paper builds on these ideas while addressing the limited assessment of predictive accuracy across practical settings.

  • Deep-network bounds: Recent studies proposed deep-network generalization bounds based on network complexity or noise stability, but broad empirical assessment of their predictive accuracy remains unavailable.This gap concerns prediction of generalization gap across varied practical settings.
  • Margin-based analysis: Margin distributions have been linked to generalization in linear models, including explanations of boosting and bagging effectiveness.Prior work also emphasized controlling classifier complexity when measuring margins, motivating normalization.

3 PREDICTION OF GENERALIZATION GAP

The paper predicts the generalization gap from normalized margin distributions across deep-network layers, compresses them into signatures, and fits a simple predictor evaluated on held-out models. Its design addresses nonlinear-distance intractability, scale dependence, distribution summarization, multilayer information, and feature-complexity effects.

  • 3.1 MARGIN APPROXIMATION: Deep-network distances to decision boundaries are approximated with a first-order Taylor method because exact distances to nonlinear surfaces are intractable.For p = 2, the approximation uses layer representations and class-score gradients.
  • 3.1 MARGIN APPROXIMATION: Margin distributions contain positive distances for correctly classified training points, with the ground-truth class compared against the highest competing class.Distances can be negative for misclassified points, but those points are excluded from the margin distribution.
  • 3.1 MARGIN APPROXIMATION: Dividing distances by the square root of total variation makes the margin distribution invariant to layer-wise scaling.Total variation is the sum of coordinate-wise activation variances and supplies the distribution’s scale factor.
  • 3.2 SUMMARIZING THE MARGIN DISTRIBUTION: Margin distributions are summarized using either the first five moments or five quartile-based statistics, with the quartile signature working slightly better.The quartile description uses Q1, Q2, Q3, and lower and upper fences based on the interquartile range.
  • 3.2 SUMMARIZING THE MARGIN DISTRIBUTION: The total signature concatenates margin signatures from all network layers because experiments find that every layer contributes to predicting the generalization gap.Using only input or output margins can miss opposing shifts across layers.
  • 3.3 EVALUATION METRICS: A linear predictor combines total-signature features either additively or multiplicatively through identity or log transforms and is assessed with held-out-model R^2.Parameters are fitted by minimizing mean squared error on pretrained models, then evaluated using 10-fold 90/10 validation; adjusted R^2 penalizes excessive feature counts.

4 EXPERIMENTS

Experiments evaluate the margin-distribution predictor across convolutional and residual networks on CIFAR-10 and CIFAR-100, including feature and baseline comparisons. The full multi-layer signature predicts generalization gap strongly, with performance varying by dataset and representation.

  • Ablations: Log-transformed features and normalized margin distributions were assessed alongside linear transforms, single-layer signatures, single-statistic signatures, and moment-based signatures.These comparisons were part of ablation experiments designed to quantify normalization, layer selection, and feature-transformation effects.
  • CIFAR-10 residual networks: The residual-network experiment on CIFAR-10 achieved R^2 = 0.87 across 216 models.A residual network with 91.7% test accuracy had larger normalized margins, especially at hidden layers, than a CNN with 87.2% test accuracy.

5 DISCUSSION

The proposed predictor uses normalized margin distributions across multiple network layers and combines distribution quartiles multiplicatively, with addition in log space. It shows stronger predictive power than the spectral-complexity baseline and applies to feedforward networks including ResNets.

  • Predictor design: The predictor uses normalized margin distributions across multiple layers and summarizes them with quartiles combined multiplicatively.The multiplicative combination is additive after a log transform.
  • Empirical comparison: The predictor has much higher predictive power than the spectral-complexity normalized-output-margin baseline.The paper presents this comparison as part of its empirical assessment of generalization-gap prediction.
  • Scope: The measure can be applied to any feedforward network, including ResNets.The discussion contrasts this scope with several generalization bounds that do not directly accommodate ResNet topologies.
  • Layer dependence: Hidden-layer information is crucial for the predictor’s predictive power.The paper identifies hidden layers as an important distinction from output-only margin approaches.
  • Implications: The findings are proposed as a starting point for new generalization theories and loss functions, alongside the released DEMOGEN dataset.The paper frames these as possible directions rather than established outcomes.

A APPENDIX: EXPERIMENTAL DETAILS

The appendix describes the experimental architectures and training configurations used to generate variation in generalization gaps. CNN and ResNet experiments vary architectural width, normalization, and other settings across hundreds of models.

  • CNN architecture: The base CNN architecture is similar to Network in Network but removes dropout and max pooling.This defines the starting architecture for the CNN experiments.
  • CNN training: The CNN configuration produces 216 different network architectures trained with SGD, momentum α = 0.9, minibatches of 128, and 380 epochs.The learning rate starts at 0.01 and decays by 10× every 100 epochs.
  • ResNet architecture: The ResNet-32 experiments use three stages and vary network width among 1×, 2×, and 4× channel configurations.The architecture modifications are used to create generalization-gap variation.
  • ResNet training: One ResNet configuration spans 216 architectures and uses SGD with momentum α = 0.9, minibatches of 128, and 380 epochs.The learning rate decays by 10× at 100-epoch intervals.
  • CIFAR-100 configuration: The CIFAR-100 ResNet configuration spans 324 architectures under the same 380-epoch training schedule.Its architecture variation includes the same 1×, 2×, and 4× width choices described for ResNet experiments.

B.1 ANALYSIS WITH NEGATIVE MARGINS

Additional experiments examine negative margins and prediction targets. Including negative margins generally makes test accuracy easier to predict than the generalization gap, and this analysis is restricted to linear features.

  • Negative-margin analysis: Including negative margins generally makes test accuracy easier to predict than the generalization gap.The comparison adds rows for predicting the gap and test accuracy.
  • Feature restriction: Negative margins can only be used with linear features.The table caption specifies this restriction for the additional analyses.
  • Experimental setup: The additional rows extend Table 1 with negative-margin generalization-gap prediction and test-accuracy prediction.The remaining Table 1 rows contain the original ablation and baseline results.

B.2 ANALYSIS FOR INDIVIDUAL LAYER’S MARGIN DISTRIBUTIONS

Individual hidden-layer margin distributions provide predictive information about the generalization gap, motivating comparison of predictive power across layers.

  • Comparing individual layers shows that hidden-layer margin distributions are important for predicting the generalization gap.
  • Table 4 compares single-layer predictors using quartile descriptions with a logarithmic transformation.

C APPENDIX: FURTHER ANALYSIS OF REGRESSION

The appendix evaluates regression residuals and feature significance across explanatory-variable settings, using the full quartile description.

  • Residuals are fairly evenly distributed around zero in two experimental settings, with the lower fence clipped at zero.
  • Using p = 0.05, all variables reject the null hypotheses in one reported significance analysis.
  • One experimental setting has less-even residuals, with clustering along the x axis and limited generalization-gap diversity proposed as a possible explanation.
  • Another analysis does not reject the null hypotheses for four variables, which the authors associate with insufficiently diverse generalization behavior.
  • A further analysis rejects the null hypotheses for every variable except h3 upper fence.
  • The regression uses the full quartile description rather than a reduced margin summary.

D.1 CROSS ARCHITECTURE COMPARISON

Cross-architecture analysis compares base CNNs and ResNet32 models trained on CIFAR-10, finding strong predictive consistency across architectures.

  • ¯R2 = 0.91 and the k-fold R2 = 0.88 for the cross-architecture regression analysis.The results suggest that the same coefficient works generally well across architectures trained on the same data.
  • The margin distributions at three network locations are comparable despite the architectures having vastly different depths.

D.2 CROSS DATASET COMPARISON

Cross-dataset and combined analyses report strong regression performance, including consistency across datasets, architectures, and their combination.

  • ¯R2 = 0.93 and the k-fold R2 = 0.93 when all experiment data are combined.The authors note that a common set of coefficients exists across both datasets and architectures.
Loading 1810.00113v2…