PaperlayerView exact revision on arXiv

Source-linked AI summary

The Break-Even Point on Optimization Trajectories of Deep Neural Networks

Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, Krzysztof Geras

arXiv:2002.09572v1cs.LGstat.ML

TL;DR

The paper asks how early SGD choices shape the later optimization trajectory, addressing limited understanding of optimization geometry and generalization links. It models and tests a break-even point, finding that large early learning rates reduce gradient noise and improve conditioning, including in batch-normalized networks. The authors identify connecting these effects to generalization as future work.

  • Problem

    The connection between DNN optimization and generalization is not fully understood, including why large initial learning rates and batch normalization affect training and generalization.

    Source: §1 · p. 1

  • Method

    The paper develops a simplified break-even-point model and empirically studies how early SGD learning rates and batch sizes affect K and H along training trajectories.

    Source: §1 · p. 2, §4.2 · p. 7

  • Results

    Large early learning rates reduce gradient variance, improve covariance-of-gradients conditioning, and steer training toward better-conditioned loss-surface regions, including with batch normalization.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Key properties of the loss surface and gradient noise are strongly influenced by SGD hyperparameters used in the early training phase.

    Source: §5 · p. 10

    The conjectures apply only to learning rates and batch sizes that guarantee training convergence, and the paper leaves closer Hessian investigation for future work.

    Source: §3 · p. 5, §4.2 · p. 7

Abstract

from arXiv · show

The early phase of training of deep neural networks is critical for their final performance. In this work, we study how the hyperparameters of stochastic gradient descent (SGD) used in the early phase of training affect the rest of the optimization trajectory. We argue for the existence of the "break-even" point on this trajectory, beyond which the curvature of the loss surface and noise in the gradient are implicitly regularized by SGD. In particular, we demonstrate on multiple classification tasks that using a large learning rate in the initial phase of training reduces the variance of the gradient, and improves the conditioning of the covariance of gradients. These effects are beneficial from the optimization perspective and become visible after the break-even point. Complementing prior work, we also show that using a low learning rate results in bad conditioning of the loss surface even for a neural network with batch normalization layers. In short, our work shows that key properties of the loss surface are strongly influenced by SGD in the early phase of training. We argue that studying the impact of the identified effects on generalization is a promising future direction.

1 INTRODUCTION

The introduction frames early SGD hyperparameters as determinants of later optimization geometry. It proposes a break-even point and argues that larger early learning rates reduce gradient noise and improve conditioning, including with batch normalization.

Motivation

  • The paper studies how early-training SGD hyperparameters affect the entire optimization trajectory through gradient covariance K and Hessian H.K captures mini-batch gradient noise, while H captures local loss-surface curvature.Source: §1 · pp. 1–2

Empirical Illustration

  • Figure 1 compares η = 0.01 and η = 0.001 trajectories on CIFAR-10, showing that lower η reaches a region with larger λ1K after the break-even point at matched training accuracy.Points encode UMAP embeddings of test predictions; background color encodes the spectral norm of K.Source: Figure 1 · p. 2, §1 · p. 2

The Break-Even Point

  • The break-even point is where gradient descent begins oscillating along the loss surface’s most curved direction.The paper models early curvature as changing monotonically along the trajectory and presents empirical evidence for this point in actual DNNs.Source: §1 · p. 2

Main Conjectures

  • Early SGD hyperparameters are conjectured to control the spectral norms and conditioning of both K and H after the break-even point.These properties describe gradient noise and local curvature along the subsequent optimization trajectory.Source: §1 · p. 2
  • A larger learning rate before the break-even point reduces K’s spectral norm, thereby reducing mini-batch gradient variance.The paper links lower gradient variance to improved convergence speed from the optimization perspective.Source: §1 · p. 2

Batch Normalization

  • The analysis extends to batch-normalized networks, where a large learning rate is necessary to reach better-conditioned loss-surface regions rather than batch normalization alone producing that condition.This complements the paper’s broader claim that early SGD choices shape later optimization geometry.Source: §1 · p. 2

2 RELATED WORK

Related work connects optimization trajectories, early critical periods, gradient covariance, Hessian curvature, and implicit regularization to DNN optimization and generalization. The paper builds on these lines by focusing on how early SGD dynamics shape the full trajectory.

Implicit Regularization

  • Prior studies report that large initial learning rates can improve generalization, while parameter norms and whole optimization trajectories are used to study this connection.Trajectory-based analyses complement bounds based on properties of the final minimum.Source: §2 · pp. 2–3

Learning Dynamics

  • Earlier work identifies critical early learning periods in which example shuffling, regularization, or other interventions can affect the eventual solution.These studies motivate examining the early phase rather than only the final trained model.Source: §2 · p. 3

Gradient Covariance and Hessian

  • The Hessian measures local loss-surface curvature, and its largest eigenvalues can grow rapidly during early training.Prior work also studies how optimization hyperparameters influence Hessian properties at the final minimum.Source: §2 · p. 3
  • The Hessian can be decomposed into two terms, with uncentered gradient covariance G dominant at least near the end of training.This connects curvature analyses to gradient-statistics analyses.Source: §2 · p. 3
  • The covariance of gradients K captures variation and geometry across examples and has been related to generalization error.Related measures include cosine alignment between gradients computed on different examples.Source: §2 · p. 3

Learning Dynamics

  • The paper’s theoretical analysis is motivated by learning-dynamics studies reporting negative cosine similarity between consecutive SGD steps on classification benchmarks.It also draws on prior dynamical-stability analyses of neural-network training.Source: §2 · p. 3

3 THE BREAK-EVEN POINT AND THE TWO CONJECTURES ABOUT SGD

The paper models a break-even point where SGD stability changes along the most curved direction, then conjectures that early learning rate and batch size shape curvature, gradient-noise variance, and conditioning along the trajectory.

The Break-Even Point

  • The theoretical model assumes a quadratic loss along the top Hessian eigenvector, monotonic curvature behavior, and a batch size much smaller than the training set.These assumptions are used to generate empirically testable predictions about early training.Source: §3 · p. 4
  • A larger learning rate or smaller batch size is predicted to reach the break-even point earlier and determine the Hessian and gradient-covariance properties there.The prediction is made for trajectories initialized in a stable region and is formally stated for the model before empirical testing.Source: §3 · pp. 4–5
  • The break-even point is the first trajectory iteration where the stability condition becomes equality, and it is unique by definition.The model uses this point to separate the early phase from subsequent optimization behavior.Source: §3 · p. 4

The Two Conjectures About SGD

  • After the break-even point, the model predicts that SGD avoids regions where the largest eigenvalues of H or K exceed their break-even values.This post-break-even restriction follows from maintaining stability along the leading Hessian direction.Source: §3 · p. 5
  • The variance-reduction conjecture states that larger learning rates or smaller batches yield smaller maximum spectral values of the gradient covariance K.Lower λ1(K) can reduce the L2 distance between full-batch and mini-batch gradients.Source: §3 · p. 5
  • The pre-conditioning conjecture states that larger learning rates or smaller batches increase the smallest non-zero eigenvalues of K and H while reducing their maximum traces.The conjectures apply only to learning rates and batch sizes that guarantee convergence, and K has at most N − 1 non-zero eigenvalues.Source: §3 · p. 5

4 EXPERIMENTS

Experiments support the break-even framework across architectures and classification tasks: early SGD hyperparameters shape gradient covariance and loss-surface curvature throughout training. Higher learning rates generally reduce the largest covariance eigenvalue and improve conditioning, including in batch-normalized networks.

Experimental setup

  • Experiments span CIFAR-10, IMDB, ImageNet, and MNLI using CNN, ResNet, LSTM, DenseNet, and BERT architectures.Additional results cover MLPs, momentum, and learning-rate decay.Source: §4 · p. 5, §4.2 · p. 8

Covariance and curvature

  • Lower learning rates reach regions with larger λ1 K and worse conditioning of both K and H, including more spectral outliers.These effects are observed at comparable training accuracy and when the spectra attain their largest values.Source: §4.1 · p. 6
  • Higher learning rates reduce the maximum λ1 K along the optimization trajectory, while larger batch sizes increase it.The reported trend supports variance reduction and pre-conditioning effects of SGD.Source: §4.2 · pp. 7–8
  • For ResNet-32 and SimpleCNN, smaller learning rates or larger batch sizes coincide with larger maximum λ1 H.The Hessian trend complements the covariance results.Source: §4.2 · p. 8

Broader validation

  • The same conjectured learning-rate effects hold for BERT on MNLI, DenseNet on ImageNet, and networks with batch normalization.In batch-normalized networks, a high learning rate is necessary to improve loss-surface conditioning relative to the corresponding network without batch normalization.Source: §4.2 · p. 8, §4.3 · pp. 9–10

5 CONCLUSION

The paper argues that SGD's early-phase hyperparameters shape the spectral norms and conditioning of gradient covariance K and Hessian H after the break-even point. A large initial learning rate steers training toward better-conditioned loss-surface regions, while links to generalization remain future work.

5 CONCLUSION

  • Theoretical and empirical analysis identifies a break-even point after which early SGD hyperparameters control the spectral norms and conditioning of K and H.K describes mini-batch gradient noise, while H describes local loss-surface curvature.Source: §5 · p. 10
  • A large initial learning rate steers training toward better-conditioned regions of the loss surface, benefiting optimization.Source: §5 · p. 10
  • The paper proposes connecting these SGD-controlled measures to deep-network generalization as a future research direction.Source: §5 · p. 10
  • Figure 8 visualizes conditioning of K, Tr(K), and the spectral norm of H across analogous training trajectories.Source: Figure 8 · p. 14

B PROOFS

The proofs formalize how SGD stability along the most curved direction determines the break-even point and how learning rate and batch size affect when it is reached. They also establish corresponding curvature bounds for stable and unstable initialization settings.

B PROOFS

  • Theorem 1 defines the break-even point through the first trajectory iteration where the stability condition becomes equal to 1.Source: B PROOFS · p. 14
  • The derived threshold values of λ1 H are smaller for a larger η or a smaller S.Source: B PROOFS · pp. 14–15
  • The proof rewrites the stability inequality using gradient variance, with s(t) defined through the variance of Hessian-related quantities.Source: B PROOFS · p. 14
  • Theorem 2 extends the argument to training initialized in an unstable region along the leading Hessian eigenvector.Source: B PROOFS · p. 15
  • The proof compares trajectories with different learning rates or batch sizes to derive their relative break-even behavior.Source: B PROOFS · p. 15

C APPROXIMATING THE EIGENSPACE OF K AND H

This section develops computational approximations for the eigenspace of K using mini-batch gradients and Gram matrices, avoiding the infeasible quadratic cost of explicitly forming K. It also describes related Hessian estimates and a correlation check between full-data and mini-batch spectra.

C APPROXIMATING THE EIGENSPACE OF K AND H

D EXPERIMENTAL DETAILS FOR SEC. 4.2

The experiments cover convolutional, recurrent, multilayer-perceptron, transformer, and DenseNet models across CIFAR-10, IMDB, FashionMNIST, MNLI, and ImageNet. Training schedules, batch sizes, learning-rate settings, repetitions, and preprocessing are specified for each setup.

D EXPERIMENTAL DETAILS FOR SEC. 4.2

E ADDITIONAL EXPERIMENTS FOR SEC. 4.2.

Additional experiments extend the paper’s findings across architectures, datasets, and learning-rate or batch-size settings. The FashionMNIST MLP experiments report that all conclusions carry over to that setting.

Additional experiments

F ADDITIONAL EXPERIMENTS FOR SGD WITH MOMENTUM

The momentum experiments test whether the paper’s conjectures extend beyond ordinary SGD. They report analogous effects for varying momentum and confirm both conjectures with momentum held constant.

Varying momentum

Fixed momentum

G ADDITIONAL EXPERIMENTS FOR SGD WITH LEARNING RATE DECAY

Learning-rate decay experiments examine whether the conjectures persist under a schedule. The reported results indicate that both conjectures generalize when the learning rate is divided by 10 after the 100th epoch.

Learning-rate decay

H ADDITIONAL EXPERIMENTS FOR SIMPLECNN-BN

Additional SimpleCNN-BN experiments vary batch size to further examine conditioning of the loss surface. Overall, decreasing batch size has effects similar to increasing learning rate, with an exception at batch size 10.

SimpleCNN-BN

Related papers

Loading 2002.09572v1…