Source-linked AI summary
Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence
Aditya Golatkar, Alessandro Achille, Stefano Soatto
TL;DR
The paper asks when regularization affects deep-network generalization, beyond the usual focus on the loss landscape at convergence. By varying when weight decay or data augmentation is applied, it finds that regularization matters most during an initial transient, not near convergence.
Problem
The timing of regularization in deep-network training had not been addressed, despite prior work on which regularizers to use and their effects on the loss landscape.
Source: §1 · p. 1Method
The authors test how applying weight decay or data augmentation at different training phases affects generalization.
Source: §1 · p. 1Results
Weight decay or data augmentation during initial epochs affects generalization, whereas applying regularization only late has little effect and can leave generalization as poor as without regularization.
Source: §1 · p. 1Takeaways & Limitations
The findings support focusing analysis of deep-network regularization on transient learning dynamics rather than only on convergence.
Source: §1 · p. 2, §5 · p. 8The exploration is not exhaustive, and the conclusions are supported for the architectures, training sets, and learning rates tested.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Regularization is typically understood as improving generalization by altering the landscape of local extrema to which the model eventually converges. Deep neural networks (DNNs), however, challenge this view: We show that removing regularization after an initial transient period has little effect on generalization, even if the final loss landscape is the same as if there had been no regularization. In some cases, generalization even improves after interrupting regularization. Conversely, if regularization is applied only after the initial transient, it has no effect on the final solution, whose generalization gap is as bad as if regularization never happened. This suggests that what matters for training deep networks is not just whether or how, but when to regularize. The phenomena we observe are manifest in different datasets (CIFAR-10, CIFAR-100), different architectures (ResNet-18, All-CNN), different regularization methods (weight decay, data augmentation), different learning rate schedules (exponential, piece-wise constant). They collectively suggest that there is a ``critical period'' for regularizing deep networks that is decisive of the final performance. More analysis should, therefore, focus on the transient rather than asymptotic behavior of learning.
1 Introduction
The paper asks when regularization matters during deep-network training and finds that its timing, rather than simply its presence, is decisive. Regularization has its strongest effect during an initial critical period; later application does not improve generalization, while early removal can preserve it.
1 Introduction
- Regularization via weight decay or data augmentation has the same effect on generalization when applied only during the initial epochs.The authors contrast this with convex optimization, where the transient is irrelevant.Source: §1 · p. 1
- Weight decay or data augmentation applied beyond the initial transient does not improve generalization.The transient is described as decisive for asymptotic performance.Source: §1 · p. 2
- Applying regularization only during the final convergence phase does not improve, and can degrade, generalization.The authors argue this is not because regularization reshapes the loss landscape at convergence.Source: §1 · pp. 1–2
- A short sliding window of regularization has its strongest effect during a critical period of only a few epochs.The authors conclude that analysis should focus on transient rather than asymptotic learning dynamics.Source: §1 · p. 2
- The proposed explanation is that regularization biases the initial transient toward regions containing multiple solutions with good generalization properties.After entering such a region, changing the regularizer can move the solution without changing test accuracy.Source: §1 · p. 2
2 Related Work
Prior work largely examines regularization through the loss landscape and properties of solutions at convergence, while comparatively little addresses its timing. The paper relates timing to other regularization schedules and questions whether convergence-time flatness explains generalization.
2 Related Work
- Most deep-network regularization work studies convergence geometry, including flat minima and minima-versus-saddles, while timing has received limited attention beyond learning-rate schedules.The cited work also examines SGD dynamics and initialization.Source: §2 · p. 2
- In stereo reconstruction, regularization is introduced early and then removed to avoid over-smoothing, unlike machine learning accounts that treat it as part of the problem formulation.The passage contrasts transient optimization aid with a prior on the solution.Source: §2 · p. 2
- Prior work links Fisher information at convergence with generalization, but the authors say no causal link has been proven.They suggest the correlation may be an epiphenomenon: weight decay raises Fisher information during the transient, while asymptotic Fisher norm is not causative.Source: §2 · pp. 2–3
- Figure 1 reports that regularization timing affects final accuracy, with weight decay showing a more marked time dependence than data augmentation.Its panels show accuracy by removal epoch and onset, plus sensitivity to the onset of a 50-epoch window.Source: Figure 1 · p. 3
3 Preliminaries and notation
The preliminaries define the network’s output distribution and dataset loss, then describe weight decay and data augmentation as training procedures. They also preview how regularization timing relates to weight norms and training paths.
3 Preliminaries and notation
- The network output distribution p_w(y|x) is parameterized by weights w, and the cross-entropy loss is defined over dataset D.For discrete labels, the output distribution is usually softmax(f_w(x)).Source: §3 · p. 3
- Weight decay adds an L2 penalty to the dataset loss, while stochastic gradient descent updates weights using mini-batch gradient estimates.The regularized loss is L = L_D(w) + λ/2 ∥w∥².Source: §3 · p. 3
- Data augmentation expands the training set with randomly transformed inputs while retaining their labels.The transformations are sampled from a known distribution.Source: §3 · p. 3
- The figure relates weight norms over training epochs and projected training paths to regularization timing, showing that lower asymptotic norm need not mean better generalization.It also contrasts weight decay’s critical-period pattern with data augmentation, whose paths converge to a similar area.Source: Figure 2 · p. 4
- Experiments use CIFAR-10 and CIFAR-100 with ResNet-18 and All-CNN, typically training with momentum SGD and an exponentially decaying learning rate.The default learning rate starts at 0.1 and decays by a factor of 0.97 per epoch.Source: §3 · p. 4
4 Experiments
Experiments vary when regularization is applied, across architectures, datasets, regularizers, and learning-rate schedules. They find that early training is the decisive period: later regularization or its removal often changes the converged solution without improving generalization.
Regularization interrupted
- Regularization through the initial epochs achieves final test accuracy as good as regularizing throughout, while continuing it beyond the transient brings no measurable improvement.This holds for weight decay and data augmentation, including on CIFAR-100.Source: §4 · pp. 4, 6
Experimental scope
- Repeating the Figure 1 experiment on CIFAR-100 produced consistent results; piecewise-constant learning rates showed qualitatively similar behavior to exponential decay, while constant-rate results were inconclusive because the network did not converge.Source: Figure 3 · p. 5
Regularization interrupted
- Removing weight decay after late training changes weight norms, so optimization is not stuck; the changed convergence point does not improve test accuracy.The same experiments without batch normalization remain compatible with the timing effect, which is not solely due to batch-normalization interaction.Source: §4 · pp. 5, 7
Reshaping the loss landscape
- Early regularization appears to direct training toward regions containing different extrema with similar generalization, making convergence-landscape changes insufficient to explain the benefit.Late regularization can converge in a regularized landscape yet generalize poorly; early regularization can be stopped while the network converges in an unregularized landscape.Source: §4 · pp. 5–6
Regularization delayed
- Delaying regularization until around epoch 50 yields no benefit; delaying weight decay by 50 epochs raises test error from 5% to 7%.The delayed and continuously regularized runs share the same loss landscape at convergence.Source: §4 · p. 5
Sliding Window Regularization
- A 50-epoch sliding window is most effective early: it yields generalization almost as good as continuous regularization, while weight decay mainly affects initial epochs and augmentation remains influential later.The sensitivity profile depends on the regularizer; late-onset windows include 50 extra training epochs to ensure convergence.Source: §4 · p. 6
Effect of Batch-Normalization
- With batch normalization, weight decay's higher effective learning rate benefits generalization only when it changes dynamics during the critical period.Initial effective learning rate correlates more strongly with generalization than final effective learning rate.Source: §4 · p. 7
Weight decay, Fisher and flatness
- Weight decay increases Fisher Information Matrix curvature, especially during the transient, despite leading to better generalization.The peak FIM trace correlates with test accuracy (Pearson 0.92, p-value < 0.001), whereas the final trace does not (Pearson 0.29, p-value > 0.05).Source: §4 · pp. 7–8, Figure 5 · p. 7
5 Discussion and Conclusions
The discussion argues that regularization chiefly matters during early training: once a network reaches a well-generalizing region, switching regularization on or off changes the loss and optimization path but not test error in the tested settings. The authors connect this critical-period account to other regularizers and call for more study of transient dynamics.
5 Discussion and Conclusions
- DNNs can reach zero training error regardless of weight-norm constraints, so regularization does not impose the classical trade-off between regularization and training loss.The authors note that weight decay can work even when batch normalization makes the network invariant to weight scale.Source: §5 · p. 8
- Early regularization biases training toward regions with good generalization; after that, turning regularization on or off changes the loss and optimization path without affecting test error in the tested settings.The authors report this account across the architectures, training sets, and learning rates they tested.Source: §5 · p. 8
- The experiments cover two datasets, two architectures, and two learning-rate schedules, while the authors caution that the exploration is not exhaustive.They describe critical periods as dependent on the deficit, task, and architecture, and relate the idea to earlier work on humans, songbirds, and DNNs.Source: §5 · p. 8
- The critical-period account also applies to batch normalization, offering an explanation for why weight decay works despite scale-invariant activations.This challenges prior explanations of weight decay’s mechanism.Source: §5 · p. 8
- Regularization’s main effect is on transient dynamics, rather than solely on changing the loss landscape or gradient noise to prevent convergence to particular local minima.The authors argue that without regularization in the end, the network can still converge.Source: §5 · p. 8
- The authors say no viable theory of transient regularization currently exists and invite theoreticians to investigate the empirical findings.They contrast this gap with theoretical work focused on asymptotics and minima at convergence.Source: §5 · p. 8
- A possible interpretation is that increased noise from effective learning rate or sample variety helps networks cross beneficial loss-landscape bottlenecks while avoiding irreversibly suboptimal ones.The authors present this as a possible interpretation, not an established mechanism.Source: §5 · p. 9
A Details of the Experiments
The experiments use ResNet-18 and All-CNN with SGD, specifying training duration, batch size, learning-rate schedules, and architecture-specific settings.
A Details of the Experiments
- ResNet-18 experiments use SGD with momentum 0.9, batch size 128, and 200 epochs, with 50 extra epochs when regularization is applied late.The standard architecture is ResNet-18 [11], unless stated otherwise.Source: A Details of the Experiments · p. 11
- ResNet-18 uses initial learning rate 0.1 and weight decay 0.0005 with exponential decay 0.97 per epoch, while piece-wise decay drops by 0.1 every 60 epochs and constant-rate experiments use 0.001.These are the stated learning-rate settings for the ResNet-18 experiments.Source: A Details of the Experiments · p. 11
- All-CNN uses initial learning rate 0.05, weight decay 0.001, and 0.97-per-epoch learning-rate decay; it omits dropout and adds batch normalization to all layers.The All-CNN architecture is cited as [31].Source: A Details of the Experiments · p. 11
A.1 Path Plotting
The path-plotting procedure combines saved network weights from runs with different regularization durations and projects the resulting matrix onto its first two principal components.
A.1 Path Plotting
- Weights saved at regular training intervals across different regularization durations are combined into matrix M and projected onto its first two principal components.The procedure follows [23] and is used to plot training trajectories.Source: §A.1 · p. 11
- The projection is constructed from the combined weights matrix rather than from a single training run’s trajectory.The matrix pools weights from networks trained with varying durations of regularization.Source: §A.1 · p. 11
- Figure 6 presents the PCA-projected training paths for All-CNN on CIFAR-10.The caption identifies the architecture and dataset, but gives no path comparison outcome.Source: Figure 6 · p. 11
B Additional Experiments
The additional experiments examine weight-decay dynamics, learning-rate effects, and whether the reported patterns extend across architectures and datasets. The constant-rate comparison is inconclusive because its test error remains substantially higher than with the other schedules.
B Additional Experiments
- Figure 7 plots the trace of the Fisher information matrix across training epochs for delayed and sliding-window weight decay on ResNet-18 with CIFAR-10.The caption specifies the comparison but does not state an outcome.Source: Figure 7 · p. 11
- The constant-learning-rate experiment is inconclusive: its test error is almost twice that achieved with exponential or piece-wise constant schedules.The authors say careful tuning might reach baseline performance and still show critical-period phenomena.Source: Figure 8 · p. 12
- Weight decay from epochs 0–50 yields a 1.92% test-accuracy increase, versus 1.39% for epochs 25–75 and 0.24% for epochs 50–100.The early-window model has higher effective learning rate during the critical period but lower rate afterward than the later-window models.Source: Figure 9 · p. 12
- Initial effective learning rate correlates with final test accuracy at 0.96 (p-value < 0.001), compared with 0.85 (p-value < 0.001) for final effective learning rate.Initial and final means are computed over the first and final 100 epochs, respectively.Source: Figure 9 · p. 12
- Figure 10 reports similar results to Figure 2 across ResNet-18 and All-CNN and across CIFAR-10 and CIFAR-100.The caption identifies the plotted quantity as weight L2 norm over training epochs.Source: Figure 10 · p. 13