PaperlayerView exact revision on arXiv

Source-linked AI summary

Early Stopping Against Label Noise Without Validation Data

Suqin Yuan, Lei Feng, Tongliang Liu

arXiv:2502.07551v1cs.LG

TL;DR

Label-noise learning complicates model selection: holding out validation data reduces training data, and noisy labels make validation unreliable. Label Wave tracks changes in training-set predictions to choose an early stopping point; experiments report effectiveness across settings and improved generalization in noisy-label methods.

  • Problem

    Hold-out validation reduces the data available for training, while label noise makes both learning from training data and selecting a model with validation data difficult.

    Source: §1 · p. 1

  • Method

    Label Wave tracks prediction changes on the noisy training set to identify the early stopping point without separate hold-out data.

    Source: §1 · p. 3, §3.3 · p. 6

  • Results

    Experiments show Label Wave is effective across various settings and improves generalization performance in methods for learning with noisy labels.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    Label Wave can select a stopping point without reserving separate hold-out data, leaving sufficient training data to produce an effective model.

    Source: §1 · p. 3

    The transition at Point 2 from misguided generalization to overfitting is not satisfactorily explained by the predefined stability and variability metrics alone.

    Source: A DISCUSSION ON learning confusing… · p. 14

Abstract

from arXiv · show

Early stopping methods in deep learning face the challenge of balancing the volume of training and validation data, especially in the presence of label noise. Concretely, sparing more data for validation from training data would limit the performance of the learned model, yet insufficient validation data could result in a sub-optimal selection of the desired model. In this paper, we propose a novel early stopping method called Label Wave, which does not require validation data for selecting the desired model in the presence of label noise. It works by tracking the changes in the model's predictions on the training set during the training process, aiming to halt training before the model unduly fits mislabeled data. This method is empirically supported by our observation that minimum fluctuations in predictions typically occur at the training epoch before the model excessively fits mislabeled data. Through extensive experiments, we show both the effectiveness of the Label Wave method across various settings and its capability to enhance the performance of existing methods for learning with noisy labels.

1 INTRODUCTION

The introduction motivates early stopping as a way to limit fitting mislabeled data, while noting that hold-out validation consumes training data and is unreliable under label noise. Label Wave instead tracks training-set prediction changes; the paper identifies their local minimum as the stopping point before fitting mislabeled examples harms performance.

1 INTRODUCTION

  • Hold-out validation can reduce training data, while label noise makes both training and validation data unreliable for learning and selecting a model.The introduction frames these as challenges for choosing an early stopping point.Source: §1 · p. 1
  • The local minimum of prediction changes marks the turning point between Stages 1 and 2 and is identified as the early stopping point.The PC curve matches the test-error curve across these stages.Source: §1 · p. 2
  • In Stage 2, fitting mislabeled examples raises prediction fluctuations and test error while harming the model’s fit to correct training patterns.The paper terms this phenomenon “learning confusion patterns,” reflecting misinterpretation of the training-data distribution.Source: §1 · pp. 2–3
  • Label Wave identifies the transition from primarily fitting clean examples to mislabeled ones without prior knowledge of which examples are mislabeled or separate hold-out validation data.The paper says this leaves sufficient training data to produce an effective model.Source: §1 · p. 3
  • Label Wave uses changes in training-set predictions to identify an early stopping point, building on the observed correlation between fitting and generalization performance.Prediction changes count examples whose predictions shift between successive epochs.Source: §1 · pp. 2–3

2 RELATED WORK

Related work addresses memorization and forgetting, indirect model selection, learning stages, and example-level learning dynamics. The paper distinguishes its focus on selecting an early stopping point from approaches centered on other selection criteria or individual-example dynamics.

Memorization & Forgetting

  • Prior work studies forgetting events to understand how training data and network structures influence generalization (Toneva et al., 2019; Feldman & Zhang, 2020).The paper observes that forgetting increases when the model starts fitting mislabeled examples, then decreases after a turning point.Source: §2 · p. 3

Model Selection

  • Unlike indirect model-selection methods, Label Wave selects an early stopping point during training without additional or hold-out data or preprocessing.The related-work discussion lists approaches based on marginal likelihood, gradients, unlabeled data, noise stability, generalization gaps, loss distributions, and training speed.Source: §2 · p. 3

Learning Stages

  • Prior accounts describe underfitting and overfitting or simple-pattern learning followed by memorization, while other work identifies additional stages under specific conditions.These studies primarily classify stages by generalization performance, overlooking variations in model fitting performance.Source: §2 · p. 3

Learning Dynamics

  • Learning-dynamics methods classify example hardness for noisy-label or long-tail settings, but focus on individual examples for sample selection rather than overall training stages.Examples include Late Stopping (Yuan et al., 2023), FSLT&SSFT (Maini et al.), Self-Filtering (Wei et al., 2022b), SELFIE, and RoCL.Source: §2 · p. 3

Memorization & Forgetting

3 DETERMINING EARLY STOPPING BY TRACKING FITTING PERFORMANCE

The section links training-set fitting changes to generalization shifts and develops Label Wave, an early-stopping method that does not require hold-out validation data or prior knowledge of mislabeled examples.

3.1 METRICS FOR TRACKING FITTING PERFORMANCE

  • Point 1 is the test-error global minimum; after it, the model begins significantly fitting mislabeled examples, which the authors associate with impaired generalization.Figure 2 tracks test error alongside training error on mislabeled examples.Source: §3.1 · p. 4, Figure 2 · p. 4
  • The section measures fitting changes with k-epoch learning for prediction stability and prediction changes for variability between successive epochs.Higher k-epoch learning indicates greater recent stability, while lower prediction changes indicate fewer fluctuations.Source: §3.1 · pp. 4–5

3.2 FITTING MISLABELED EXAMPLES IMPAIRS THE MODEL’S FITTING PERFORMANCE

  • Fluctuations in predictions decrease before Point 1, increase after Point 1, then decrease toward a steady state after Point 2.The authors report that stability and variability turning points correlate with critical shifts in generalization performance.Source: §3.2 · p. 5

3.3 LABEL WAVE METHOD

  • Label Wave smooths prediction changes across recent epochs and selects their first local minimum as the stopping point, without validation data or prior knowledge of training labels.The method uses moving averages and patience to make stopping more robust to training uncertainty.Source: §3.3 · p. 6

4 EXPERIMENTS

Across noisy-label settings, Label Wave selected models at or near peak test accuracy and enhanced existing noisy-label methods. It also outperformed hold-out validation across tested validation-set sizes and noise rates.

4 EXPERIMENTS

  • The experiments assess Label Wave across settings, its enhancement of noisy-label learning methods, and its model-selection association with test accuracy.The hold-out comparison considers both selection precision and the trade-off between validation-set size and training data.Source: §4 · p. 7, §4.3 · p. 9

4.1 EFFECTIVENESS OF LABEL WAVE

  • Across varied architectures, datasets, noise conditions, parameters, and optimizers, Label Wave selected models at or near the global maximum in test accuracy.The tested settings included instance-dependent, symmetric, and real-world noise, multiple vision and text datasets, and class-imbalanced data.Source: §4.1 · p. 7

4.2 ENHANCING EXISTING LEARNING WITH NOISY LABEL METHODS

  • Label Wave consistently outperformed conventional validation methods using 5% to 30% hold-out data on CIFAR-10 and CIFAR-100 with 40% symmetric noise.The evaluations used ResNet-18 on CIFAR-10 and ResNet-34 on CIFAR-100.Source: §4.2 · p. 8, Table 2 · p. 8, Table 3 · p. 8

4.3 CORRELATION WITH TEST ACCURACY

  • Label Wave selected models more precisely and yielded better test accuracy than hold-out validation, including when computed on a smaller subset.The comparison varied subset sizes from 250 to 16,000 examples and noise rates from 20% to 60%; gains were especially notable at higher noise and larger set sizes.Source: §4.3 · p. 9, Figure 4 · p. 9

5 CONCLUSION

The conclusion presents Label Wave as an early-stopping method for label noise that uses prediction changes instead of separate hold-out data. It reports effectiveness across settings and identifies learning confusing patterns and improved prediction-change metrics as directions for further work.

5 CONCLUSION

  • Label Wave uses prediction changes to track fitting performance and select an early-stopping point without separate hold-out data.The conclusion reports improved generalization performance in existing methods for learning with noisy labels.Source: §5 · p. 9
  • The conclusion identifies learning confusing patterns as a transitional stage in learning with noisy labels.Source: §5 · p. 9
  • Future work could develop a prediction-change metric that surpasses the current one and investigate the transitional stage more deeply.The paper suggests these directions may enhance Label Wave and provide insight into how deep neural networks learn with noisy labels.Source: §5 · p. 9

A DISCUSSION ON learning confusing patterns

The section identifies Point 2 as a turning point in fitting performance and introduces learning confusing patterns between Points 1 and 2. It describes the stage as a shift toward misguided generalization, while noting that the transition at Point 2 remains unexplained by the predefined metrics.

A DISCUSSION ON learning confusing patterns

  • Figure 5 proposes learning confusing patterns as a transitional stage between Points 1 and 2, based on tracked generalization and fitting metrics.Panel a tracks the metrics between the points; panel b depicts the proposed stage.Source: Figure 5 · p. 14
  • At Point 2, prediction fluctuations switch from increasing to decreasing and eventually stabilize, marking a turning point in fitting performance similar to early-stopping Point 1.Before Point 1, declining fluctuations accompany improving generalization; after Point 2, their decline reflects a shift toward fitting individual labels and memorization.Source: A DISCUSSION ON learning confusing… · p. 14
  • Between Points 1 and 2, learning confusing patterns describes a stage where fitting and generalization decline as the model adapts to mislabeled examples.The authors contrast this with learning simple patterns before Point 1 and describe the stage as learning from misled patterns.Source: A DISCUSSION ON learning confusing… · p. 14
  • The predefined stability and variability metrics do not satisfactorily explain the marked transition at Point 2 from misguided generalization to overfitting.The authors identify explaining this transition as an open question for future investigation.Source: A DISCUSSION ON learning confusing… · p. 14

B DETAILED OVERVIEW OF BASIC MODEL IN OUR EXPERIMENTS

The appendix specifies a basic ResNet-18 training setup and a noisy CIFAR-10 setting, then describes experiments across varied environments to assess Label Wave.

B DETAILED OVERVIEW OF BASIC MODEL IN OUR EXPERIMENTS

  • The basic model uses PyTorch 1.11.0, ResNet-18 without dropout, learning rate 0.01, SGD with momentum 0.9, and CrossEntropyLoss.These settings are listed as the basic experimental model configuration.Source: B DETAILED OVERVIEW OF BASIC MODEL IN… · p. 15
  • The CIFAR-10 setup holds out 20% of training data for validation and applies normalization, random crops, and rotations limited to ±15 degrees.Images are normalized to [0,1]; random 32x32 crops may follow padding.Source: B DETAILED OVERVIEW OF BASIC MODEL IN… · p. 15

Label Noise

  • The Label Noise setting adds 40% symmetric noise to 80% of CIFAR-10 training data, affecting 40,000 examples.Source: Label Noise · p. 15

C EFFECTIVENESS OF LABEL WAVE

  • Effectiveness experiments vary architectures, datasets, noise types and levels, parameters, and optimizers across settings commonly used in noisy-label learning.The appendix says specific results for each setting appear in Table 1 of Section 4.1.Source: C EFFECTIVENESS OF LABEL WAVE · p. 16

C.1 NETWORK ARCHITECTURES

The architecture experiments assess Label Wave using several established convolutional network families, including multiple ResNet variants.

C.1 NETWORK ARCHITECTURES

  • The authors describe these architecture choices as a test of Label Wave’s versatility and effectiveness.Source: §C.1 · p. 16
  • The architecture set includes ResNet-18, -34, -50, and -101, VGG-16, Inception-v3, and DenseNet-121.Source: §C.1 · p. 16

C.2 DATASETS EMPLOYED

The experiments span image and text datasets, including benchmarks with real-world label noise and class-imbalanced settings.

C.2 DATASETS EMPLOYED

  • To assess Label Wave’s versatility, the study employed several datasets, including CIFAR-N’s “Random 1, 2, 3” setting, marked by approximately 20% real-world label noise, plus Clothing1M, WebVision, and Food101.Source: §C.2 · pp. 16–17
  • The passage identifies CIFAR-10 and CIFAR-100 as established benchmarks.Source: §C.2 · p. 16
  • NEWS extends the evaluation beyond images, using pretrained word embeddings and a three-layer MLP with Softsign activation.Source: §C.2 · p. 16
  • The class-imbalanced setting uses Imbalance Factor 0.1 and symmetric Noise Ratio 0.4, testing both CID-CE and CID-LDAM.Source: §C.2 · p. 17

C.3 LABEL NOISE

The experiments examine Label Wave across varied noise, optimization, and regularization settings, and report that its selected models remain close to global-maximum test accuracy. The method is not applicable in very low or no label noise.

C.3 LABEL NOISE

  • Evaluations span 20%–80% symmetric and instance-dependent noise, CIFAR-N real-world noise, and 10% symmetric and instance-dependent noise.Source: §C.3 · p. 17
  • Label Wave is not applicable in very low or no label noise, and may not identify an appropriate stopping point when learning confusion patterns are absent.Examples include perfect data or robust regularization; in these scenarios, early stopping to improve generalization may not be necessary.Source: §C.3 · p. 17

C.4 PARAMETERS AND OPTIMIZATION

  • The parameter and optimization experiments vary batch size, learning rate, and optimizer, including SGD with momentum, RMSProp, and Adam.Source: §C.4 · p. 18

C.5 REGULARIZATION TECHNIQUES

  • The regularization evaluation includes Mixup, Batch Normalization, dropout, and data augmentation.The supplied table caption identifies test accuracy (mean±std) on CIFAR-10 with 40% symmetric label noise.Source: §C.5 · p. 18, Table 5 · p. 18

C.6 QUANTITATIVE COMPARISON BETWEEN LABEL WAVE AND GLOBAL MAXIMUM TEST ACCURACY

  • Across diverse settings, Label Wave selections have consistently slight test-accuracy disparities from global-maximum models, aligning closely with or matching the optimal models.A smaller Difference indicates performance nearer the global maximum.Source: §C.6 · p. 19

D ENHANCING EXISTING LEARNING WITH NOISY LABEL METHODS

The section evaluates Label Wave alongside noisy-label learning methods across CIFAR settings. The supplied passages specify the evaluation configurations and table scopes, but do not provide cell values or comparative outcomes.

D ENHANCING EXISTING LEARNING WITH NOISY LABEL METHODS

E SENSITIVITY ANALYSIS OF k VALUE IN MOVING AVERAGES

The sensitivity analysis varies the moving-average window for prediction changes (PC) and compares averaged PC with test accuracy. The inverse relationship remains across tested windows, although its strength varies.

E SENSITIVITY ANALYSIS OF k VALUE IN MOVING AVERAGES

  • The analysis varies k, the number of epochs in the PC moving average, and calculates the Pearson correlation between averaged PC and test accuracy.The method uses moving averages of PC for early stopping with noisy labels.Source: E SENSITIVITY ANALYSIS OF k VALUE IN… · p. 21

E.1 OBSERVATIONS •

  • The strongest negative correlation occurs at k = 3, while k = 5 and k = 10 show a slight decrease in strength.Source: §E.1 · p. 21

E.2 CONCLUSION

  • Changing k affects correlation strength but does not significantly alter the overall negative relationship between moving-average PC and test accuracy.Source: §E.2 · p. 21

Related papers

Loading 2502.07551v1…