Source-linked AI summary
Early Stopping without a Validation Set
Maren Mahsereci, Lukas Balles, Christoph Lassner, Philipp Hennig
TL;DR
Overfitting makes generalization difficult to assess from finite training data, while validation-based stopping trades estimate reliability against data available for training. The paper proposes a gradient-statistics criterion that avoids a held-out set; experiments report misleading validation loss in scarce-data logistic regression and comparable network test losses with element-wise stopping.
Problem
Validation-based early stopping estimates generalization from held-out data, but small validation sets can be noisy and larger ones reduce the training data available.
Source: §1.1 · p. 2Method
The EB criterion uses local gradient statistics to detect when observed gradients reflect finite-sample noise rather than an informative direction, without requiring a held-out validation set.
Source: §1.1 · p. 3, §2.2 · p. 4Results
In scarce-data logistic regression, the 60-instance validation loss decreased throughout and failed to find a suitable stopping point, while element-wise stopping yielded comparable network test losses.
Source: §3.3 · p. 9, §3.6 · p. 11Takeaways & Limitations
The criterion applies to gradient descent and SGD with little computation, time, and memory overhead, allowing the optimizer to use all available training data.
Source: §4 · p. 12If the intermediate-weight assumption is fundamentally violated, EB stops too early; it detects convergence but does not explicitly encode loss curvature.
Source: §3.2 · p. 8, §5.3 · p. 14
Abstract
from arXiv · showhide
Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization. To find a good point to halt the optimizer, a common practice is to split the dataset into a training and a smaller validation set to obtain an ongoing estimate of the generalization performance. We propose a novel early stopping criterion based on fast-to-compute local statistics of the computed gradients and entirely removes the need for a held-out validation set. Our experiments show that this is a viable approach in the setting of least-squares and logistic regression, as well as neural networks.
1 Introduction
The introduction frames early stopping as a way to limit overfitting while preserving generalization, but validation-based stopping trades estimate reliability against the training data available. It introduces a criterion based on local gradient statistics that avoids a held-out validation set.
1.1 Overfitting, Regularization and Early-Stopping
- Overfitting arises when minimizing empirical risk on finite data yields poor test performance, motivating early stopping before training loss minimization passes the best-generalizing point.Early stopping complements model and data design rather than correcting poor choices of loss function.Source: §1.1 · p. 2
- Model selection, data augmentation, and regularization are other approaches to overfitting, respectively adjusting model complexity, enriching data, and constraining parameter magnitudes [19] [20] [7].Model selection and early stopping address different stages of fitting; early stopping does not undo poor model design.Source: §1.1 · p. 2
- Validation-based stopping estimates generalization by monitoring held-out loss, but small sets can mislead and larger sets leave less training data [14, 16, 15].Repeated validation evaluations can also be computationally expensive.Source: §1.1 · p. 2
- Maclaurin et al.'s early-stopping approach estimates marginal likelihood by tracking posterior entropy changes, but requires Hessian-diagonal estimates and incurs considerable computational overhead [11].Source: §1.1 · pp. 2–3
- The proposed criterion uses local gradient statistics without a held-out set, and Figure 1 sketches the gradient-based stopping regions.The approach is intended to let the optimizer use all available training data.Source: §1.1 · p. 3, Figure 1 · p. 3
2 Model
The section develops an early-stopping rule from the sampling uncertainty of gradients: halt when observed gradients are small enough to be consistent with noise rather than informative about the population gradient. It approximates the needed uncertainty from available gradients and extends the rule from full-batch gradient descent to mini-batch SGD.
2.1 Distribution of Gradient Estimators
- For independently sampled data, empirical losses and gradients estimate their population counterparts, are approximately normal, and have variances that shrink with sample size.The covariance of the gradient estimator scales inversely with the number of samples.Source: §2.1 · p. 3
2.2 When to stop? An Evidence-Based Criterion
- A finite training set yields one noisy realization of the population objective, and its minimizer can differ from the population minimizer, permitting overfitting.The section estimates gradient variance from elementwise squared deviations of per-example gradients around their sample mean.Source: §2.2 · pp. 3–4
- Because the population gradient is unknown, stop when the observed training gradients are sufficiently probable under a zero population gradient, suggesting they carry little information beyond finite-sample noise.The proposed criterion uses the probability, or evidence, of observing the empirical gradient under the zero-gradient model.Source: §2.2 · p. 4
2.3 Early Stopping Criterion for Gradient Descent
- For full-batch gradient descent, approximate the gradient distribution using the current gradient and estimated variance at each iterate, a choice considered sensible near convergence.The approximation uses the variance at the current position in place of the unknown variance at the true minimum.Source: §2.3 · p. 4
- The gradient-descent rule treats dimensions independently for scalable computation and stops when logarithmic gradient evidence exceeds its expected value.This diagonal simplification is combined with statistical hypothesis testing and scaled by the objective dimension.Source: §2.3 · pp. 4–5
- A single mean criterion assumes parameter dimensions converge on similar timescales; groups with different timescales may need separate stopping criteria.The section gives neural-network layers as an example of parameter groups that may converge at different speeds.Source: §2.3 · p. 5
2.4 Stochastic Gradients and Mini-batching
- For mini-batch SGD, replace the full-data gradient with a noisier mini-batch estimate and adjust the gradient uncertainty to account for subsampling.The resulting stopping rule extends the full-batch criterion to stochastic gradients.Source: §2.4 · p. 5
- The criterion requires an available gradient-variance estimate; computing it can increase a neural-network backward pass cost by about 1.25 times.The rule then uses elementwise division of squared gradients by variance, followed by aggregation across dimensions.Source: §2.4 · p. 5
3 Experiments
Across toy, synthetic, and applied experiments, the EB criterion often gives useful stopping decisions without a validation set, but its performance depends on the optimization setting and its assumptions. The experiments also examine subgroup and element-wise variants.
3.1 Linear Least-Squares as Toy Problem
- Both validation loss and the EB criterion find acceptable stopping points in the over-parameterized least-squares toy problem.The experiment uses 20 training samples and about 10 validation samples.Source: Figure 3 · p. 6, §3.1 · p. 7
3.2 Synthetic Large-Scale Quadratic Problem
- Across uniform, exponential, and structured eigen-spectra, the EB criterion gives meaningful stopping decisions on partially ill-conditioned synthetic quadratic problems.The result also holds across noise levels; the illustrated case uses Λ = 10 · I, with dense gradient covariance.Source: §3.2 · p. 7, Figure 4 · p. 7
- When initialization prevents overfitting, the EB criterion can stop sub-optimally because its assumptions are violated, while test loss merely flattens.In this setup, any early stopping decision leads to under-fitting.Source: §3.2 · p. 8, Figure 5 · p. 8
3.3 Logistic Regression on WDBC
- On WDBC, EB stopping is acceptable but not optimal, and using all training data yields lower test loss than any result attainable after a validation split.The validation loss on the small 60-instance set decreases throughout training and therefore gives no suitable stopping point.Source: Figure 2 · p. 6, §3.3 · p. 9
3.4 Multi-Layer Perceptron on MNIST
- On MNIST, EB stopping performs as well as or better than validation stopping in all three SGD configurations, but stops slightly too early under full-batch gradient descent.With full-batch gradient descent, test loss is 10^-1.04 for validation stopping and 10^-0.92 for EB stopping; EB stopping quality varies across configurations.Source: §3.4 · pp. 9–10, Figure 7 · p. 10
3 Experiments
- On SECTOR, EB stopping beats validation-based stopping for both architectures; on MNIST it matches or exceeds validation stopping in all three SGD runs, while WDBC’s validation loss fails to find a stopping point.On WDBC, EB stopping is not optimal but yields lower test loss than any point attainable after withholding validation data.Source: Figure 2 · p. 6, §3.4 · p. 9, §3.3 · p. 9
3.5 Logistic Regression and Shallow-Net on SECTOR
- On SECTOR, EB stopping beats validation-based stopping for both logistic regression and a shallow neural network, despite stopping as test loss begins to flatten.The dataset has 6412 training examples and 55,197 features per datapoint; the validation split reduces the training set to 5130 examples.Source: §3.5 · p. 10, Figure 8 · p. 11
3.6 Greedy Element-wise Stopping
- Greedy element-wise stopping shuts off 10% or more of weights, depending on training configuration, with the fraction increasing during training; despite reduced effective model complexity, test losses remain comparable to runs without it.Toward the end of optimization, switched-off parameters reach up to 80% in a single layer and around 50% across the whole network.Source: §3.6 · p. 11
4 Conclusion
The conclusion presents the EB-criterion as a gradient-statistics-based way to choose when to stop optimization without a held-out validation set. Experiments support its viability in least-squares, logistic regression, and a multilayer perceptron, while element-wise stopping remains preliminary.
4 Conclusion
- The EB-criterion uses fast-to-compute gradient statistics to determine when to stop, without a held-out validation set.This lets the optimizer use all available training data.Source: §4 · p. 12
- The criterion applies to gradient descent and stochastic gradient descent with little added computation, time, or memory overhead.Source: §4 · p. 12
- Experiments support the method's viability for linear least-squares fitting, logistic regression, and a multilayer perceptron.Source: §4 · p. 12
- Preliminary element-wise early-stopping findings suggest the possibility of monitoring and controlling model fitting in greater detail.Source: §4 · p. 12
5 Comparison to RMSPROP
The section compares EB-criterion and RMSPROP to clarify their different learning-rate behavior, not as direct competitors.
5.1 Non-Greedy Elementwise EB-Criterion
- The non-greedy EB-criterion allows learning rates to switch back on, unlike the greedy implementation, where switched-off rates stay off.This reversible behavior is allowed by Eq. 10.Source: §5.1 · p. 13
5.2 Learning Rate Damping in RMSPROP
- RMSPROP scales each parameter's learning rate using an exponential running average of gradient magnitudes; very small relative factors act like switching learning off.Its elementwise scaling factor is relative to the largest learning-rate factor.Source: §5.2 · p. 13
5.3 Connections and Differences
- RMSPROP encodes geometric information, while EB-criterion leaves active learning rates unchanged and does not improve SGD's convergence during training.EB-criterion can set rates to zero for converged parameters but does not explicitly encode curvature or other loss geometry.Source: §5.3 · p. 14
- EB-criterion uses a statistically justified stopping threshold; RMSPROP only shortens uncertain directions, and its damping does not prevent overfitting.The section contrasts EB-criterion's stopping decision with RMSPROP's milder scaling of uncertain gradients.Source: §5.3 · pp. 14–15
- EB-criterion's local variance computations are unsmoothed; RMSPROP's smoothing can bias steps upward for high-variance samples.The paper says non-smoothed local computations are essential to the stopping decision because large gradient samples are often associated with large variances.Source: §5.3 · p. 15
5.4 Empirical Comparison
- Test losses for vanilla SGD and SGD+EB-criterion are almost identical; RMSPROP converges a bit faster, while EB-criterion switches off about 10–20% of rates initially, rising to about 50%.Figure 10 compares the methods on MNIST; the EB-criterion's switched-off fraction rises gradually.Source: §5.4 · p. 15, Figure 10 · p. 16