A Walk with SGD

• The motivating gap is that existing SDE accounts often assume small learning rates and obscure how learning rate, batch size, and gradient covariance differ in DNN optimization.The paper frames this gap against evidence that flatter regions generalize better and are associated with small batches and large learning rates.Source: §1 · pp. 1–2
• SGD’s loss interpolation between consecutive updates is roughly convex with an intermediate valley floor, indicating that optimization bounces between valley walls above the floor.This mechanism is observed as a recurring trajectory pattern rather than isolated behavior.Source: §1 · p. 2
• Learning rate controls the height above the valley floor, whereas batch size controls gradient stochasticity and exploration, giving them qualitatively different roles.Smaller batches produce larger parameter distances from initialization, consistent with stronger exploration.Source: §1 · p. 2
• Valley floors contain barriers, so large learning rates help SGD move above them instead of crossing them directly.The authors connect this above-floor motion to avoiding obstacles along the optimization path.Source: §1 · p. 2
• Experiments across datasets, architectures, and hyperparameters support the reported trajectory findings, while mini-batch stochasticity is needed for better optimization and generalization.Artificial isotropic noise without mini-batch stochasticity instead harms DNN optimization.Source: §1 · p. 2

• Related theory and empirical work links flatter or wider minima with better generalization, while other studies model DNN landscapes using spin glasses, connected level sets, or degenerate Hessians.These approaches motivate studying geometry but generally do not track the optimization path itself.Source: §2 · pp. 2–3
• Small-learning-rate SDE theories predict similar dynamics when learning rate and batch size change proportionally, but this framework may fail at the large learning rates common in DNN training.The paper therefore studies the distinct effects of large-learning-rate noise and small-batch noise.Source: §2 · pp. 2–3
• Contrary to diffusion-based claims that SGD crosses barriers to escape sharp minima, per-iteration interpolation suggests it almost never crosses significant barriers during most training.The paper instead attributes barrier avoidance to traveling above valley floors.Source: §2 · p. 3
• The paper’s Figure 1 relates interpolated loss minima, consecutive-gradient cosine, and distance from initialization to valley-like oscillation under full-batch GD.After roughly 15 iterations, consecutive gradients become nearly opposite while the valley floor decreases.Source: Figure 1 · p. 3
• The paper analyzes the DNN loss surface through SGD’s trajectory, tracking metrics to connect landscape geometry with hyperparameter-driven motion.This contrasts with studies that examine the loss surface in isolation from optimization dynamics.Source: §2 · p. 4

3.1 Optimization Trajectory

• SGD’s interpolated loss between consecutive updates is usually roughly convex with a minimum between parameters, indicating motion between valley walls above a valley floor.This pattern is observed across the analyzed optimization trajectory and supports the paper’s valley-bouncing interpretation.Source: §3.1 · p. 5
• SGD explores farther than full-batch GD, reaching parameter distance ∼1.7 versus GD’s ∼1.4 after 40 iterations, while its consecutive gradients are less negative.The valley floor also contains repeated ups and downs, interpreted as barriers along SGD’s path.Source: Figure 2 · p. 4, §3.1 · p. 5
• Across most training, SGD crosses no significant barriers, including experiments spanning VGG-11, Resnet-56, and MLP settings.Barriers crossed late in training are substantially smaller than the corresponding loss values.Source: §3.1 · p. 5, Table 1 · p. 5
• The Hessian spectral norm generally decreases during training as validation accuracy improves, then increases near the end alongside a mild validation-accuracy decline.The authors interpret the decreasing spectral norm as movement toward flatter regions.Source: §3.1 · p. 5, Figure 3 · p. 6

3.2 Qualitative Roles of Learning Rate and Batch Size

• Small batch sizes change gradient alignment and increase distance traveled from initialization, whereas changing learning rate at fixed batch size does not produce the same exploration effect.The authors associate mini-batch noise with exploration that may lead to better minima.Source: §3 · p. 4, §3.2 · p. 6
• Learning rate controls SGD’s height above the valley floor: smaller learning rates generally produce smaller heights, increasing exposure to barriers.The height is computed from endpoint losses and the interpolated minimum across sampled points between consecutive iterates.Source: §3.2 · p. 6, Table 2 · p. 6

Noise structure

• Mini-batch SGD gradients combine the full-data gradient with Gaussian noise whose covariance is C(θ) and whose scale depends on batch size B.This formulation isolates the covariance structure absent from full-batch gradient descent with isotropic noise.Source: §4 · p. 6
• Changing batch size changes the cosine between consecutive gradients, whereas changing learning rate has no significant effect, confirming their qualitatively different roles.Larger batch sizes make the cosine more negative, indicating more bouncing within the same region rather than farther travel.Source: Figure 4 · p. 7

Isotropic-noise comparison

• Structured mini-batch noise is important: adding isotropic noise to full-batch gradients makes convergence and generalization worse than noiseless gradient descent.The experiment treats isotropic noise as a null hypothesis for testing whether the covariance structure induced by mini-batches matters.Source: §4 · p. 7
• Parameter norms were 87 for GD and 82 for SGD, versus 369 and 443 under isotropic noise variances 0.014 and 0.028.The much larger isotropic-noise solutions support the finding that this noise both hinders optimization and harms generalization.Source: §4 · p. 7

Noise structure

• The Hessian and gradient covariance are related and nearly equal near minima, suggesting mini-batch noise aligns with sharper loss-landscape directions.The authors propose that this alignment discourages convergence in sharp directions unless a wider region is found, helping explain why SGD finds wider minima without relying on small-learning-rate SDE assumptions.Source: §4 · pp. 7–8

5 Discussion

• GD and SGD empirically move through valley-like landscapes by bouncing between valley walls, consistent with analogous quadratic-loss dynamics.The authors frame this as an empirical comparison with quadratic optimization rather than a theory under additional assumptions.Source: §5 · p. 8
• GD can oscillate between valley walls while its valley floor decreases smoothly during consecutive iterations.Source: §5 · p. 8
• Adding isotropic Gaussian noise to GD makes optimization stall and worsens validation performance relative to noise-free GD.Source: Figure 5 · p. 8
• SGD almost never crosses significant barriers; instead, a large learning rate lets it travel above valley-floor barriers and explore better regions.The authors relate this mechanism to the observed association between small learning rates and worse generalization.Source: §5 · p. 8
• The conclusions may differ for other datasets, architectures, or parameterizations, including choices such as ReLUs and batch normalization.Source: §5 · p. 8

A Optimization Trajectory

• Across datasets, architectures, epochs, batch sizes, and learning rates, the trajectory observations remain consistent in the reported experiments.The appendix covers ResNet-56, VGG-11, and MLP models across CIFAR-10, MNIST, and tiny ImageNet.Source: A Optimization Trajectory · p. 12
• For VGG-11 on tiny ImageNet, quantitative evaluation finds no barrier between any two consecutive iterations despite a visually unusual valley-like trajectory.Source: A Optimization Trajectory · p. 12
• The study also tracks Hessian spectral norm alongside validation accuracy during ResNet-56 training on CIFAR-10.Source: A Optimization Trajectory · p. 12

B Qualitative Roles of Learning Rate and Batch Size

• Smaller batch sizes produce larger angles between consecutive gradients, so optimization travels farther along the valley instead of oscillating within one region.Source: B Qualitative Roles of Learning Rate… · p. 12
• Changing the learning rate does not change the angles between consecutive gradients across the tested architectures and datasets.Source: B Qualitative Roles of Learning Rate… · p. 12

C Learning Rate Schedule

• The schedule guideline is motivated by maintaining a larger height above the valley floor to avoid barriers and facilitate exploration of flatter regions.Source: C Learning Rate Schedule · p. 12
• Methods that increase learning rate during training are slightly better than the tested alternatives, including stepwise annealing.The comparison uses trapezoid and cyclical-learning-rate-inspired schedules, with experiments on VGG-11 and ResNet-56 using batch size 100.Source: C Learning Rate Schedule · p. 13
• The proposed trapezoid schedule raises learning rate, holds it flat, then linearly anneals it to test whether repeated CLR cycles are necessary.Source: C Learning Rate Schedule · p. 13

• The analysis uses the negative log-likelihood form L_i(θ) = −log(p_i(θ)) to relate Hessian structure to gradient covariance.Source: D Importance of SGD Noise Structure · p. 13
• The derivation relies on properties specific to the chosen loss formulation.Source: D Importance of SGD Noise Structure · p. 13
• The passage introduces a detailed derivation rather than reporting an optimization experiment.Source: D Importance of SGD Noise Structure · p. 13

E Discussion

• Cyclic learning-rate schedules contain redundant cycles, as illustrated by comparisons with stepwise annealing and trapezoid schedules.Source: Figure 7 · p. 14, Figure 8 · p. 14
• Most training updates do not cross loss barriers, although the observed barrier heights are substantially smaller than the corresponding iteration losses.Source: E Discussion · p. 14
• Learning-rate changes alter the height above the valley floor, whereas batch-size changes alter gradient stochasticity and cosine similarity between consecutive gradients.Source: Figure 31 · p. 20, Figure 32 · p. 20, Figure 33 · p. 20
• For quadratic surfaces, convergence behavior depends on the learning rate relative to Hessian eigenvalues, with some directions becoming underdamped or overdamped.Source: E Discussion · p. 13, Figure 36 · p. 21