PaperlayerView exact revision on arXiv

Source-linked AI summary

Do We Need Zero Training Loss After Achieving Zero Training Error?

Takashi Ishida, Ikko Yamane, Tomoya Sakai, Gang Niu, Masashi Sugiyama

arXiv:2002.08709v2cs.LGstat.ML

TL;DR

Overparameterized networks may continue driving training loss toward zero after memorizing the data, leaving open whether zero training loss is necessary after zero training error. The paper introduces flooding, which keeps loss around a small positive level by reversing the optimization direction below that level. Experiments report improved test accuracy across synthetic and benchmark datasets, alongside a double descent curve for test loss.

  • Problem

    After achieving zero training error, whether overparameterized models need to continue reducing training loss toward zero remains an open issue.

  • Method

    Flooding keeps training loss around a user-chosen positive flood level by using gradient descent above it and gradient ascent below it.

  • Results

    Flooding improved test accuracy across synthetic and benchmark datasets and produced a double descent curve for test loss as a byproduct.

  • Takeaways & Limitations

    Flooding provides a direct way to avoid zero training loss while often preserving zero training error and maintaining memorization.

  • Takeaways & Limitations

    Flooding's level can be difficult to set without expert knowledge of the domain or task, so it may need validation-based hyperparameter selection.

Abstract

from arXiv · show

Overparameterized deep networks have the capacity to memorize training data with zero \emph{training error}. Even after memorization, the \emph{training loss} continues to approach zero, making the model overconfident and the test performance degraded. Since existing regularizers do not directly aim to avoid zero training loss, it is hard to tune their hyperparameters in order to maintain a fixed/preset level of training loss. We propose a direct solution called \emph{flooding} that intentionally prevents further reduction of the training loss when it reaches a reasonably small value, which we call the \emph{flood level}. Our approach makes the loss float around the flood level by doing mini-batched gradient descent as usual but gradient ascent if the training loss is below the flood level. This can be implemented with one line of code and is compatible with any stochastic optimizer and other regularizers. With flooding, the model will continue to "random walk" with the same non-zero training loss, and we expect it to drift into an area with a flat loss landscape that leads to better generalization. We experimentally show that flooding improves performance and, as a byproduct, induces a double descent curve of the test loss.

1. Introduction

Overparameterized models can continue reducing training loss toward zero after reaching zero training error, raising an open question about whether zero training loss is necessary. The paper proposes flooding, which keeps training loss around a user-chosen positive level and reports improved test performance.

  • Results: In CIFAR-10 learning curves, flooding avoids near-zero training loss and leads to decreasing test loss again.The paper presents this comparison in Figure 1(c)–(d).
  • Motivation: After zero training error, overparameterized models can continue reducing training loss toward near-zero, but whether this is necessary remains open.The paper distinguishes achieving zero training error from continuing optimization until training loss is also near-zero.
  • Proposed method: When the learning objective exceeds the flood level, flooding follows gradient descent; below it, the gradient reverses and produces gradient ascent.This creates gravity above the flood level and buoyancy below it.
  • Proposed method: Flooding makes training loss repeatedly cross the flood level while the model random-walks at non-zero loss and is expected to move toward flatter loss regions.The paper reports increased loss-function flatness during this random-walk period.
  • Implementation: Flooding can be added with one code line after evaluating the original objective and used with mini-batches, stochastic optimizers, and other regularizers.The implementation applies the transformed objective before optimization.
  • Proposed method: Flooding directly prevents further training-loss reduction near zero by maintaining loss around a small user-chosen flood level.The flood level is the positive loss value the user wants to keep.

2. Backgrounds

The background reviews regularization methods, their assumptions and scope, and flooding’s distinction: directly targeting a chosen nonzero training-loss level. It also situates flooding alongside double descent and related risk-control techniques.

  • 2.1. Regularization Methods: Unlike methods based on assumptions such as parameters near zero or sparsity, flooding makes no assumptions about the optimal model parameters.The paper presents this as useful for complex problems where optimal parameters need not be close to zero or sparse.
  • 2.1. Regularization Methods: Regularization includes methods intended to reduce generalization error without reducing training error, including parameter penalties, dropout, label smoothing, early stopping, and data augmentation.The paper adopts this broader meaning of regularization while noting that some methods are task- or model-specific.
  • 2.1. Regularization Methods: Flooding is presented as domain-, task-, and model-independent, complementing established regularization tools that may be task- or model-specific.The paper characterizes existing methods as effective in practice while positioning flooding as an alternative with broader applicability.
  • 2.1. Regularization Methods: Flooding differs from other regularizers by allowing the user to target a specific training-loss level other than zero.Other regularizers are described as indirect ways to control training loss, making a preset level difficult to achieve.
  • 2.2. Double Descent Curves with Overparametrization: Flooding produced an epoch-wise double descent curve for test loss in experiments, reportedly not observed previously and appearing after about 100 epochs.The paper identifies connections to earlier double descent results as an important future direction.
  • 2.3. Avoiding Over-Minimization of the Empirical Risk: Early stopping is sensitive to optimization and mini-batch randomness, so the best epoch from one training path may not transfer reliably to another.This sensitivity makes hyperparameter selection methods requiring repeated retraining difficult to use.

3. Flooding: How to Avoid Zero Training Loss

Flooding replaces the original empirical risk with an objective that keeps training loss around a chosen positive flood level. It reverses the update direction below that level, while remaining compatible with mini-batched stochastic optimization and hyperparameter search.

  • Algorithm: Training loss and training error are distinct: flooding keeps loss positive while still allowing memorization when the flood level is not too large.Training and test loss use a surrogate loss, whereas training and test error use zero-one loss.
  • Algorithm: Flooding defines a modified empirical risk that prevents flexible models from driving surrogate training loss to zero.The flood level b sets the target around which the empirical risk is maintained.
  • Algorithm: Above the flood level, optimization uses gradient descent; below it, optimization uses gradient ascent toward the flood level.The modified objective has the same gradient direction as the original risk above b and the opposite direction below b.
  • Implementation: The flood level can be selected through exhaustive hyperparameter search when its optimal value is unknown.The search can be performed in parallel using validation-based evaluation.
  • Implementation: Mini-batched optimization applies the flooded objective to each batch and minimizes an upper bound of the corresponding full-batch objective.The upper-bound relation follows from convexity of the absolute value function and Jensen’s inequality.

4. Does Flooding Generalize Better?

Experiments on synthetic and benchmark datasets evaluate flooding with and without early stopping and alongside other regularizers. Flooding often improves or preserves test accuracy, and benchmark results produce a double descent curve for test loss.

  • Synthetic Datasets: Synthetic experiments use Gaussian, sinusoid, and spiral datasets to study flooding in a controlled setup.The experiments use specified training, validation, and test sample sizes for these synthetic datasets.
  • Synthetic Results: Flooding often improves test accuracy over the baseline without flooding, including when the final epoch is used.The authors report that flooding makes late-stage training beneficial in these experiments.
  • Synthetic Results: Early stopping improves the no-flood baseline in many cases, while flooding combined with early stopping is often close to early stopping alone.The results indicate that longer training without flooding was harmful in these experiments.
  • Benchmark Experiments: Benchmark experiments compare flooding with early stopping, weight decay, data augmentation, batch normalization, and learning-rate decay.The benchmark setup covers MNIST, Kuzushiji-MNIST, SVHN, CIFAR-10, and CIFAR-100.
  • Benchmark Results: Every best-performing benchmark setup uses flooding, while combining it with other regularizers usually has complementary effects and yields a double descent curve for test loss.The double descent curve is reported as a byproduct using relatively few epochs.

5. Why Does Flooding Generalize Better?

The experiments examine whether flooding improves generalization by preserving memorization, altering gradient behavior, and increasing loss flatness. Across these analyses, flooding is associated with lower test loss, smaller test-loss gradient amplitude, and flatter loss during training.

  • 5.1. Memorization: Flooding preserves zero training error at flood levels selected using validation accuracy, although higher flood levels eventually reduce memorization.The validation-selected marks are often at zero training error, while training accuracy declines as the flood level increases.
  • 5.2. Performance and gradients: Flooded models often have larger training-loss gradient amplitude than unflooded models, implying that flooding prevents them from staying at a local minimum.In test-loss gradient plots, flooding improves performance as gradient amplitude becomes smaller.
  • 5.2. Performance and gradients: Flooding produces lower test loss with a double descent curve in most learning-curve settings.Figure 2 compares the no-flooding baseline with multiple flood levels, including settings with data augmentation and learning-rate decay.
  • 5.4. Theoretical Insight: During flooding, test loss becomes lower and flatter while training loss continues floating around the flood level until training ends.The authors expect this behavior to reflect a random walk that escapes sharp loss-landscape regions.
  • 5.4. Theoretical Insight: The flooded risk estimator has lower mean squared error than the original estimator when the flood level lies between the original training and test losses.Theorem 1 states the comparison under the stated flood-level condition, with strict improvement under an additional positive-probability condition.

6. Conclusion

The paper concludes that flooding keeps training loss near a small positive constant while often preserving zero training error. Experiments report improved test accuracy across synthetic and benchmark datasets, and flooding also produces a double descent test-loss curve.

  • 6. Conclusion: Flooding keeps training loss around a small constant to avoid zero training loss while often preserving zero training error.The method forces loss to remain positive without necessarily making training error positive.
  • 6. Conclusion: Flooding improves test accuracy across the reported synthetic and benchmark datasets, while theory shows reduced MSE under certain conditions.The conclusion combines the experimental accuracy finding with the theorem's conditional MSE result.
  • 6. Conclusion: Flooding produces a double descent curve for test loss as a byproduct.The paper identifies studying its relationship with double descent from previous work as an important future direction.

A. Proof of Theorem 1

The proof compares the MSE of the original and flooded risk estimators by decomposing their difference according to whether the original empirical risk is above or below the flood level. Under the theorem's conditions, the flooded estimator has no larger MSE and can have strictly smaller MSE.

  • Case decomposition: The proof rewrites the MSE difference as an expectation of a quantity B determined by the max-based flooded estimator.The estimator is re-expressed using A = 2 max(flooded risk, flood level), after which the MSE difference is analyzed through B.
  • Case decomposition: When the original risk is at least the flood level, the corresponding contribution to B is zero.The piecewise expression distinguishes this case from the case where the original risk falls below the flood level.
  • Case decomposition: When the original risk is below the flood level, B is positive if the flood level remains below the original risk.The proof identifies positivity in the relevant intermediate regime.
  • Conclusion of proof: The MSE difference is nonpositive under the theorem's condition, and it is strictly positive for original minus flooded MSE when the additional probability condition holds.Equivalently, the flooded estimator has strictly smaller MSE in that case.

B. Benchmark Datasets

The benchmark evaluation uses six image datasets spanning handwritten and cursive digits, house-number images, and object-recognition tasks. The datasets vary in class count, image format, and training/test sample size.

  • Digit datasets: MNIST and Kuzushiji-MNIST are 10-class datasets of 28 × 28 grayscale images with 60,000 training and 10,000 test samples.MNIST contains handwritten digits, whereas Kuzushiji-MNIST contains cursive Japanese characters.
  • Natural-image datasets: SVHN contains 10-class 32 × 32 × 3 RGB house-number images, with 73,257 training and 26,032 test samples.The dataset is derived from Google Street View imagery.
  • Natural-image datasets: CIFAR-10 contains 10 object classes with 5,000 training and 1,000 test images per class.Each image is a 32 × 32 × 3 RGB image.
  • Natural-image datasets: CIFAR-100 contains 100 object classes, with 500 training and 100 test samples per class.It is also described as a subset of the 80 million tiny images dataset.

C. Learning Curves

This section presents additional test-loss learning curves and identifies the corresponding CIFAR-10 ResNet-18 examples shown in Figure 1.

  • Figure 6 provides additional test-loss learning curves beyond the main-paper results.
  • Figure 1(c) shows the first 80 epochs for CIFAR-10 without flooding using ResNet-18.
  • Figure 1(d) shows ResNet-18 learning curves with flooding at flood level 0.18.

D. Additional Figures for Section 5.2 and Section 5.3

The additional figures examine test-loss learning curves, loss–gradient-amplitude relationships, and one-dimensional flatness visualizations for models trained with and without flooding.

  • Figures 7 and 8 provide results for datasets not included in the main paper.
  • Figure 6 compares baseline and flooding test-loss learning curves across flooding levels b ∈{0.01, 0.02, . . . , 0.10}.The figure includes KMNIST, CIFAR-10, and CIFAR-100 results.
  • Figure 7 relates test or training loss to gradient amplitude across epochs for models with and without flooding.The plots exclude the first 10 epochs and mark early-stopping epochs with large black symbols.
  • Figure 8 visualizes training and test loss under perturbations for models selected before flooding, at flooding’s end, and without flooding.
Loading 2002.08709v2…