PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Improving Early Stopping for Learning with Noisy Labels

Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, Tongliang Liu

arXiv:2106.15853v2cs.LG

TL;DR

The paper addresses the problem that whole-network early stopping does not account for different layers’ sensitivity to noisy labels. It proposes progressive early stopping, which trains earlier parts longer and later parts for fewer epochs with preceding parts fixed, and reports improved, stable performance including state-of-the-art benchmark results.

  • Problem

    Whole-network early stopping may antagonistically affect DNN layers because later layers are more sensitive to noisy labels than former layers.

  • Method

    Progressive early stopping separates a DNN into parts, trains former layers for more epochs, and progressively trains latter layers for fewer epochs with preceding layers fixed.

  • Results

    PES consistently outperforms baselines across synthetic CIFAR-10 and CIFAR-100 settings and improves performance over existing methods on real-world Clothing-1M.

  • Takeaways & Limitations

    PES can improve the quality of selected confident examples and obtain substantial performance improvements compared with current state-of-the-art baselines.

  • Takeaways & Limitations

    Splitting a DNN into parts introduces additional hyperparameters that require careful tuning.

Abstract

from arXiv · show

The memorization effect of deep neural network (DNN) plays a pivotal role in many state-of-the-art label-noise learning methods. To exploit this property, the early stopping trick, which stops the optimization at the early stage of training, is usually adopted. Current methods generally decide the early stopping point by considering a DNN as a whole. However, a DNN can be considered as a composition of a series of layers, and we find that the latter layers in a DNN are much more sensitive to label noise, while their former counterparts are quite robust. Therefore, selecting a stopping point for the whole network may make different DNN layers antagonistically affected each other, thus degrading the final performance. In this paper, we propose to separate a DNN into different parts and progressively train them to address this problem. Instead of the early stopping, which trains a whole DNN all at once, we initially train former DNN layers by optimizing the DNN with a relatively large number of epochs. During training, we progressively train the latter DNN layers by using a smaller number of epochs with the preceding layers fixed to counteract the impact of noisy labels. We term the proposed method as progressive early stopping (PES). Despite its simplicity, compared with the early stopping, PES can help to obtain more promising and stable results. Furthermore, by combining PES with existing approaches on noisy label training, we achieve state-of-the-art performance on image classification benchmarks.

1 Introduction

Deep networks can overfit noisy labels, making robust learning and the choice of training duration important. The paper motivates progressive early stopping by showing that different layers respond differently to noise and should therefore be trained with different schedules.

  • Motivation: Deep networks can memorize noisy labels and overfit, causing poor generalization when annotations contain mistakes.This motivates methods that remain robust to noisy annotations.
  • Related Work: Existing noisy-label methods either model label noise explicitly or exploit the tendency to learn clean patterns before noisy ones.Noise-transition approaches can be fragile for heavily noisy data and difficult to scale to many classes.
  • Early Stopping: Stopping too early can underfit clean labels, while stopping too late can overfit noisy labels, making the stopping epoch critical.Conventional methods select one stopping point for the network as a whole.
  • Layer Sensitivity: Experiments indicate that later layers reach their best performance at fewer epochs and then suffer stronger performance drops than earlier layers.The analysis evaluates representations from multiple layers after training on noisy data for different epoch counts.
  • Proposed Method: PES trains earlier layers longer, then progressively trains later layers for fewer epochs with preceding layers fixed.The method is designed to exploit memorization while counteracting different layers’ sensitivity to noisy labels; the paper reports promising and stable results with only slightly greater training time.

2 Proposed Method

The method addresses noisy-label overfitting by assigning different training durations to different DNN parts, then uses confident examples and semi-supervised learning for refinement. PES progressively fixes earlier parts while training later parts for fewer epochs, improving accuracy and stability over traditional early stopping.

  • Motivation: Noisy-label training requires balancing clean-label fitting against eventual memorization of incorrect labels.Early stopping can prevent noise fitting but may underfit clean labels; continuing optimization can memorize more noisy labels.
  • Motivation: Traditional early stopping selects one stopping epoch for the whole network, although different DNN layers respond differently to noisy labels.This can cause layers to be antagonistically affected during joint optimization.
  • Progressive Early Stopping: PES first trains former DNN parts for more epochs, then reinitializes and trains successive later parts with preceding parts fixed.The method progressively optimizes each later part while retaining already optimized preceding parameters.
  • Progressive Early Stopping: PES uses decreasing training durations across parts, satisfying T1 ≥ T2 ≥ · · · ≥ TL because later parts are more sensitive to noisy labels.The final network combines the progressively optimized parameters.
  • Empirical Comparison: On CIFAR-10 experiments with different noise types, PES generally achieves higher accuracy and smaller variation than traditional early stopping.Figure 2 reports means over five runs, while the accompanying analysis states the general performance improvement.
  • Learning with Confident Examples: The refinement procedure selects confident examples when predictions agree with given labels, using averaged predictions from two augmentations.Confident examples can support further classifier training, but class-count differences motivate a weighted classification loss.
  • Combining with Semi-Supervised Learning: To use remaining data, the method treats confident examples as labeled and other noisy examples as unlabeled, then applies MixMatch.Algorithm 1 extracts both sets and trains with a MixMatch loss.

3 Experiments

Experiments evaluate PES on synthetic and real-world noisy-label datasets, comparing it with baselines, examining extracted confident examples, sensitivity, and training time. PES generally improves accuracy and stability, consistently outperforms baselines in reported settings, and remains computationally efficient.

  • Datasets and implementation: Experiments cover CIFAR-10, CIFAR-100, and Clothing-1M under synthetic and real-world label noise.The synthetic settings include symmetric, pairflip, and instance-dependent noise.
  • Preliminary experiments: PES generally improves test accuracy, label precision, and label recall over traditional early stopping across most CIFAR-10 cases.For instance-dependent noise, PES achieves higher recall with comparable label precision.
  • Classification accuracy evaluation: PES consistently outperforms all compared baselines across synthetic CIFAR-10 and CIFAR-100 settings without semi-supervised learning.The comparison reports results across various noise settings using mean and standard deviation over five runs.
  • Classification accuracy evaluation: PES outperforms all baselines across the reported semi-supervised CIFAR settings, exceeding state-of-the-art methods by over 8% for pairflip noise on both datasets.Tables 3 and 4 report mean accuracy and standard deviation for the evaluated noise types and levels.
  • Sensitivity analysis: PES performs best with T2=7 and T3=5, and both hyperparameters remain robust across different noise types.Performance varies less with T3 than with T2; the T2 trend generally rises and then falls as T2 increases.
  • Training time comparison: PES with cross-entropy loss has the fastest reported baseline training time at about 1 hour, while its MixMatch variant takes slightly over half DivideMix’s time.The comparison uses a single Nvidia V100 GPU; ELR+ uses fewer training epochs than PES.

4 Related work

Related work on learning with noisy labels divides into model-based methods that correct losses using noise models and model-free methods that exploit training dynamics. The latter use the memorization effect to identify or emphasize likely clean examples.

  • Model-based methods: Model-based methods estimate a noise transition matrix and use it to construct loss corrections.With a perfectly estimated matrix, corrected-loss models can approximate models trained with clean labels.
  • Model-free methods: Model-free methods exploit deep networks’ tendency to fit clean patterns before overfitting noisy patterns.This memorization effect motivates optimization-policy methods for noisy-label learning.
  • Model-free methods: Co-teaching selects confident examples using small-loss samples from two differently initialized networks.The method uses paired networks to operationalize the memorization effect.

5 Conclusion

The conclusion presents progressive early stopping as a layer-wise response to differing sensitivity to noisy labels. Experiments across synthetic and real-world datasets report substantial improvements over current state-of-the-art baselines, while additional hyperparameters remain a limitation.

  • Conclusion: PES separates a DNN into parts and progressively trains them to address different noisy-label impacts across layers.Former layers are treated as less affected than latter layers, motivating layer-specific training.
  • Conclusion: Experiments across multiple synthetic and real-world noisy datasets show substantial performance improvements over current state-of-the-art baselines.The reported evaluation covers both synthetic and real-world noisy datasets.
  • Conclusion: Splitting a DNN into parts introduces additional hyperparameters that require careful tuning.The authors identify this tuning burden as the method’s main limitation.

A Training details

The training-details section introduces the three adopted kinds of noisy labels and indicates that preprocessing and hyperparameter settings are specified for the experiments.

  • Training details: The experiments adopt three kinds of noisy labels.The section states that the three noise types are described before preprocessing and hyperparameter settings.
  • Training details: Data preprocessing is part of the reported experimental setup.The passage places preprocessing details after the description of label-noise types.
  • Training details: The experimental setup includes specified hyperparameter settings.These settings are presented alongside the preprocessing details.

A.1 Definition of noise

The paper distinguishes symmetric class-dependent, pairflip class-dependent, and instance-dependent label noise. It defines a transition matrix as the conditional process mapping clean labels to observed noisy labels.

  • Noise types: The paper considers symmetric class-dependent, pairflip class-dependent, and instance-dependent label noise.These categories are distinguished by how noisy labels correlate with clean labels.
  • Transition matrix: A transition matrix T(x) models noisy-label generation through conditional flip probabilities.Its entries satisfy T_ij(x) = Pr(Ŷ = j|Y = i, X = x).
  • Noise types: Symmetric class-dependent noise uses a common flip rate with diagonal and off-diagonal transition probabilities determined by the number of classes.For noise excluding true labels, the diagonal is 1 − α and off-diagonal entries are α/(c − 1).
  • Noise types: Pairflip noise simulates mistakes between similar adjacent classes using pairflip transition matrices.The construction is intended to model fine-grained classification errors.
  • Noise types: Instance-dependent noise generates flip rates from image features and sampled variables.Algorithm 2 assigns instance-specific rates and controls the diagonal entry of each transition-matrix row.

A.2 Data preprocessing and experimental settings

Experiments use standard augmentation and dataset-specific training settings, with PES hyperparameters and semi-supervised loss weights specified for benchmark evaluations.

  • Data preprocessing: CIFAR-10/100 experiments use random cropping and horizontal flips, while semi-supervised settings additionally use MixUp.
  • Data preprocessing: Clothing-1M images are resized to 256 × 256, randomly cropped to 224 × 224, and randomly horizontally flipped.
  • PES hyper-parameters: PES uses Adam for T2 and T3, with each selected from {2, 5, 7}; total training epochs include T1 but exclude T2 and T3.
  • Semi-supervised learning: Semi-supervised experiments fix most hyperparameters across noise levels and adjust λu, selected from {5, 15, 25, 50, 75, 100}.
  • Hyper-parameter references: Tables 7 and 8 report training hyperparameters for CIFAR-10/100 and Clothing-1M, and semi-supervised loss weights for CIFAR-10/100.

B Additional experiments

Additional experiments on CIFAR-100 and Fashion-MNIST test whether noisy labels affect later network layers more severely and compare the method with ensemble-based baselines. The results show larger degradation in later layers and comparable single-network performance against two-network ensemble baselines.

  • Baseline comparisons: The section adds comparisons with methods using ensemble networks, alongside cross-entropy and MixUp single-network baselines.Table 9 reports comparisons on CIFAR-10 and CIFAR-100 under symmetric label noise at different noise levels.
  • Experimental setup: The experiments extend evaluation to CIFAR-100 with ResNet-34 and Fashion-MNIST with LeNet.CIFAR-100 uses a deeper network and more classes, while Fashion-MNIST uses 60,000 28x28 training images and a five-layer LeNet.
  • Layer sensitivity: Figures 4 and 5 evaluate noisy-label effects on representations from multiple intermediate and final layers.ResNet-34 representations are examined at layers 17, 33, and the final layer; LeNet representations are examined at layers 1, 4, and the final layer.
  • Layer sensitivity: The final layer shows the largest performance drop, the 33-th layer declines gradually, and the 17-th layer remains relatively stable during training.These observations on CIFAR-100 are reported as similar to those previously observed on CIFAR-10.
  • Baseline comparisons: The method’s single-network results are comparable to state-of-the-art baselines that use two-network ensembles.The comparison is based on results from the original papers, with the highest results reported for all methods.
Loading 2106.15853v2…