PaperlayerView exact revision on arXiv

Source-linked AI summary

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang

arXiv:1609.04836v2cs.LGmath.OC

TL;DR

Large-batch training can improve parallelism but is associated with worse generalization than small-batch training. This paper uses numerical experiments to examine the relationship between batch size and minimizer sharpness, finding that large-batch methods tend toward sharper minima while small-batch methods tend toward flatter ones.

  • Problem

    Large batches offer greater parallelization but are observed to produce worse test-set performance, motivating investigation of this generalization gap.

  • Method

    The paper uses one-dimensional parametric plots and perturbation-based sharpness measures across multiple deep-learning architectures to compare large- and small-batch minimizers.

  • Results

    Large-batch methods converge to sharper minimizers associated with poorer generalization, whereas small-batch methods converge to flatter minimizers.

  • Takeaways & Limitations

    Training with large batches without sacrificing generalization could enable scaling to more computational nodes and potentially reduce training time by orders of magnitude.

  • Takeaways & Limitations

    The paper provides numerical evidence for the sharp-minima explanation but does not prove that large-batch methods typically converge to sharp minimizers.

Abstract

from arXiv · show

The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.

1 INTRODUCTION

The paper examines why increasing batch sizes can harm generalization, linking large-batch training to sharp minimizers while motivating scalable methods that preserve generalization.

  • 1 INTRODUCTION: Deep learning trains non-convex neural-network objectives, with SGD and variants commonly used to optimize them.Each data-point loss captures prediction deviation, while the optimized weight vector defines the training objective.
  • 1 INTRODUCTION: Small-batch SGD uses noisy mini-batch gradients, typically with batch sizes of 32–512, and has succeeded across many applications.The batch is sampled from the dataset, and small-batch methods use |Bk| ≪ M.
  • 1 INTRODUCTION: Small batches limit parallelization, motivating larger batches to improve scalability across computing nodes.The sequential iteration and small batch sizes constrain speed-ups and scalability.
  • 1 INTRODUCTION: The paper reports that the generalization gap correlates with marked sharpness in minimizers obtained by large-batch methods.This motivates large-batch algorithms that scale to more nodes without sacrificing generalization performance.
  • 1 INTRODUCTION: The study presents numerical evidence and discusses possible modifications to reduce the large-batch generalization problem.Subsequent sections examine small-batch performance, theoretical connections, and open questions about sharp minima and training modifications.

2 DRAWBACKS OF LARGE-BATCH METHODS

Large-batch methods exhibit a generalization gap despite similar training-function values, converging to sharper, more sensitive minimizers than small-batch methods. Experiments indicate this gap is not caused by overfitting or overtraining and that sharpness is concentrated in a small subspace.

  • Core observation: The generalization gap occurs despite large-batch methods usually achieving training-function values similar to small-batch methods.The paper presents this as a drawback specific to generalization rather than comparable training-function performance.
  • Core observation: Large-batch methods tend to converge to sharp minimizers with many large positive Hessian eigenvalues, whereas small-batch methods converge to flat minimizers with numerous small eigenvalues.The paper links these qualitatively different minimizers to large-batch methods’ poorer generalization.
  • Alternative explanations: The experiments do not attribute the gap to over-fitting or over-training, because testing accuracy does not peak and then decay from learning training-data idiosyncrasies.Early stopping is therefore not presented as the explanation for the observed gap.
  • Numerical evidence: The LB minima are strikingly sharper than SB minima along a one-dimensional slice, and their relative sharpness remains evident along a curved path.These visualizations compare randomly selected SB and LB minimizers from the 5 trials used for Table 2.
  • Sharpness measurements: 1–2 orders of magnitude separate the sharpness-metric values for small-batch and large-batch regimes, reinforcing larger sensitivity for large-batch solutions.The metric is a computationally feasible sensitivity measure based on exploring a small neighborhood around the solution.
  • Geometry of sharpness: Sharp minimizers do not form cones: the loss rises steeply only along a small subspace, such as 5% of the whole space, and is relatively flat elsewhere.This qualifies the observed sharpness as directionally concentrated rather than rapid growth in most directions.

3 SUCCESS OF SMALL-BATCH METHODS

Small-batch methods are associated with better generalization because gradient noise helps move iterates away from sharp minimizers toward flatter ones. Increasing batch size beyond problem-dependent thresholds produces large testing-accuracy drops, while large-batch sharpness rises as loss decreases.

  • Batch-size thresholds: Beyond batch-size thresholds of approximately 15,000 for F2 and 500 for C1, testing accuracy drops sharply.Around these thresholds, the upward drift in sharpness is considerably reduced.
  • Noise and flatness: Gradient noise pushes small-batch iterates out of sharp-minimizer basins and toward flatter minimizers.When batch size exceeds the threshold, stochastic-gradient noise is insufficient to eject iterates from sharp-minimizer basins.
  • Batch-size thresholds: Figure 4 evaluates final testing accuracy and sharpness after 100 training epochs across batch sizes for the F2 and C1 networks.Sharpness is reported for ϵ values of 10^-3 and 5 · 10^-4.
  • Sharpness versus loss: As cross-entropy loss decreases, sharpness for large-batch iterates rapidly increases.Near the initial point, where loss is larger, small- and large-batch methods yield similar sharpness values.

4 DISCUSSION AND CONCLUSION

The experiments support the view that large-batch methods generalize poorly because they converge to sharp minimizers. The paper also reports preliminary attempts to remedy this issue and identifies several open questions about large-batch training.

  • 4 DISCUSSION AND CONCLUSION: Numerical experiments support that convergence to sharp minimizers gives rise to poor generalization in large-batch deep-learning methods.The evidence includes one-dimensional parametric plots and perturbation-based sharpness measures across varied deep-learning architectures.
  • 4 DISCUSSION AND CONCLUSION: Preliminary remedies explored for the large-batch generalization problem include data augmentation, conservative training, and robust optimization.These attempts are described in Appendix E.
  • 4 DISCUSSION AND CONCLUSION: The findings align with prior observations that deep-learning loss functions contain many local minimizers, often with similar loss values under regularity assumptions.The cited theoretical work concerns properties of neural-network loss surfaces.
  • 4 DISCUSSION AND CONCLUSION: The results motivate open questions about proving large-batch convergence to sharp minimizers, the relative density of minima, suitable architectures, and initialization.The paper emphasizes that its evidence for large-batch convergence to sharp minimizers is numerical rather than a proof.

A DETAILS ABOUT DATA SETS

The experiments use multiple data sets summarized in Table 5; TIMIT is pre-processed with Kaldi and trained using a fully connected network, while the remaining data sets are unprocessed.

  • A DETAILS ABOUT DATA SETS: The experiments use data sets summarized in Table 5, with TIMIT serving as a speech-recognition data set.TIMIT is pre-processed using Kaldi and trained using a fully connected network; the other data sets receive no pre-processing.

B ARCHITECTURE OF NETWORKS … B.4 NETWORKS C2 AND C4

The paper evaluates fully connected networks F1 and F2 alongside convolutional networks C1–C4, using batch normalization and ReLU activations throughout the described architectures. The networks differ in input/output dimensions, layer counts, convolutional configurations, pooling, and dense-layer sizes.

  • B.1 NETWORK F1: F1 uses a 784-dimensional input, five batch-normalized 512-neuron ReLU layers, and a 10-neuron softmax output.The architecture is described as a fully connected network.
  • B.2 NETWORK F2: F2 uses a 360-dimensional input, seven batch-normalized 512-neuron ReLU layers, and a 1973-neuron softmax output.Its architecture is described as similar to F1.
  • B.3 NETWORKS C1 AND C3: C1 is a modified AlexNet configuration with two [64, 5, 5, 2] convolutional sets, MaxPool(3), dense layers of sizes 384 and 192, and a 10-unit output.The passage also states that all layers use batch normalization and ReLU activations and that Dropout is used, but the provided text is truncated after this point.
  • B.3 NETWORKS C1 AND C3: C3 uses convolutional stacks of 2×[64, 3, 3, 1], 2×[128, 3, 3, 1], 3×[256, 3, 3, 1], and two 3×[512, 3, 3, 1] stacks.Each stack is followed by MaxPool(2).
  • B.3 NETWORKS C1 AND C3: C3’s convolutional stack is followed by a 512-dimensional dense layer and a 10-dimensional output layer.The passage states that the activation and layer properties follow the description in B.3.
  • B.4 NETWORKS C2 AND C4: C2 is a modified VGG configuration, while C4 is described as identical to C2 except for an omitted difference in the provided passage.The supplied text ends before specifying the exception.

C PERFORMANCE MODEL · D CURVILINEAR PARAMETRIC PLOTS

The performance model frames large-batch competitiveness as requiring both good generalization and sufficiently rapid convergence. It illustrates that, under specified efficiency and batch-size conditions, large-batch training must substantially reduce iteration counts to yield performance benefits.

  • C PERFORMANCE MODEL: Large-batch training could scale to more nodes and potentially reduce training time if it avoids the generalization gap.The paper presents an idealized performance model to demonstrate this goal.
  • C PERFORMANCE MODEL: Large-batch methods must reach comparably accurate, well-generalizing minimizers in a reasonably small number of iterations to compete with small-batch methods.The model denotes these iteration counts as I_s and I_ℓ and the batch sizes as B_s and B_ℓ.
  • C PERFORMANCE MODEL: The model compares small- and large-batch iteration requirements using their batch sizes, processor count, and small-batch parallel efficiency.It assumes P < B_ℓ and defines f_s(P) as the small-batch method’s parallel efficiency.
  • C PERFORMANCE MODEL: Large-batch training is faster than small-batch training only when its iteration reduction offsets the effects of batch size and parallel efficiency.The section formulates this requirement as a performance inequality.
  • C PERFORMANCE MODEL: 0.2 and 0.1 imply that large-batch training must converge in at most half as many iterations as small-batch training to provide benefits.This example uses f_s(P) = 0.2 and B_s/B_ℓ = 0.1.
  • D CURVILINEAR PARAMETRIC PLOTS: The D CURVILINEAR PARAMETRIC PLOTS section states that x⋆_s can be found in Figure 7.The supplied passage is a fragment referring to the figure.

E ATTEMPTS TO IMPROVE LB METHODS

This section evaluates strategies intended to improve large-batch methods’ poor generalization, using consistent batch-size and optimizer settings across experiments.

  • E ATTEMPTS TO IMPROVE LB METHODS: The experiments use 10% batch size for large-batch methods, 256 for small-batch methods, and ADAM for all conditions.These settings are held constant irrespective of batch size.

E.1 DATA AUGMENTATION

The section examines data augmentation as a domain-specific way to make the loss geometry more benign for large-batch methods. Aggressive augmentation yields large-batch accuracy comparable to small-batch training, but sharp minima persist, suggesting sensitivity to images.

  • Motivation: Data augmentation is considered as a way to modify loss geometry and make it more benign for large-batch methods.The loss depends on both the objective geometry and the training set’s size and properties.
  • Results: Large-batch accuracy becomes comparable to small-batch accuracy with augmented training, but sharp minima still exist, suggesting sensitivity to images.The comparison is reported for the four image-based convolutional networks in Table 6.

E.2 CONSERVATIVE TRAINING

Conservative training improves large-batch SGD by obtaining iterates from a proximal sub-problem that better utilizes each batch before advancing. The sub-problem is solved inexactly, and Li et al. report improved convergence and empirical performance on convex problems.

  • Conservative training: Large-batch SGD convergence can be improved by obtaining iterates through a proximal sub-problem.This strategy is attributed to Li et al. (2014).
  • Conservative training: The strategy aims to better utilize one batch before moving onto the next.The underlying idea is not specific to convex problems.
  • Conservative training: 3–5 iterations of gradient descent, co-ordinate descent or L-BFGS approximately solve the minimization problem.The minimization is solved inexactly using one of these methods.
  • Conservative training: Li et al. report improved SGD convergence and empirical performance on convex machine learning problems.Both improvements are reported as outcomes of the strategy.

E.3 ROBUST TRAINING

The section examines robust optimization as a way to avoid sharp minima, but notes that its direct application is prohibitively expensive. It also distinguishes data and solution robustness and reports that adversarial training did not improve generalization in the experiments.

  • Robust optimization: Robust optimization targets a worst-case cost rather than the nominal cost to avoid sharp minima.For ϵ > 0, these techniques formulate an optimization problem over worst-case perturbations.
  • Robust optimization: Geometrically, nominal optimization finds a valley’s lowest point, whereas robust optimization lowers an ϵ-disc over the loss surface.The section presents this geometric interpretation as the distinction between the two optimization approaches.
  • Robust optimization: Direct robust optimization is not feasible in this context because each iteration is prohibitively expensive.Each iteration involves solving a large-scale second-order conic problem.
  • Robustness in Deep Learning: Deep Learning involves inter-dependent robustness to data and robustness to the solution, with prior work proving equivalence between solution robustness and adversarial training.Data robustness treats f as a statistical model, while solution robustness treats f as a black-box function.
  • Adversarial training: Adversarial training did not improve generalization in the experiments.Unlike randomized data augmentation, it constructs new examples using the model’s sensitivity.
Loading 1609.04836v2…