PaperlayerView exact revision on arXiv

Source-linked AI summary

Sample Selection with Uncertainty of Losses for Learning with Noisy Labels

Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, Masashi Sugiyama

arXiv:2106.00445v1cs.LG

TL;DR

Sample selection often treats current small losses as evidence of clean labels, although losses are uncertain and can exclude clean examples from underrepresented classes. This paper combines robust loss estimates across training iterations with confidence-bound-based conservative selection, and reports superior robustness to state-of-the-art methods across synthetic balanced and imbalanced, and real-world noisy datasets.

  • Problem

    Current small-loss selection is uncertain and can exclude clean examples with large losses, including examples from non-dominant classes.

    Source: §1 · p. 2

  • Method

    The method robustly estimates losses across training iterations and uses confidence bounds to conservatively select less-selected examples.

    Source: §1 · p. 2, §2.3 · p. 5

  • Results

    The method achieves superior robustness to state-of-the-art methods on synthetic noisy balanced and imbalanced datasets and real-world noisy datasets.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that the approach opens possibilities for sample selection, especially for improving robustness on imbalanced noisy datasets.

    Source: §4 · p. 12

    Averaging training losses can be suboptimal when mislabeled-data outliers bias the mean and thereby affect sample selection.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

In learning with noisy labels, the sample selection approach is very popular, which regards small-loss data as correctly labeled during training. However, losses are generated on-the-fly based on the model being trained with noisy labels, and thus large-loss data are likely but not certainly to be incorrect. There are actually two possibilities of a large-loss data point: (a) it is mislabeled, and then its loss decreases slower than other data, since deep neural networks "learn patterns first"; (b) it belongs to an underrepresented group of data and has not been selected yet. In this paper, we incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses, where lower bounds of the confidence intervals of losses derived from distribution-free concentration inequalities, but not losses themselves, are used for sample selection. In this way, we also give large-loss but less selected data a try; then, we can better distinguish between the cases (a) and (b) by seeing if the losses effectively decrease with the uncertainty after the try. As a result, we can better explore underrepresented data that are correctly labeled but seem to be mislabeled at first glance. Experiments demonstrate that the proposed method is superior to baselines and robust to a broad range of label noise types.

1 Introduction

The introduction argues that current-loss sample selection is uncertain at both ends and can exclude clean examples from underrepresented classes. It proposes using robust loss estimates across training iterations and confidence bounds to address this uncertainty.

1 Introduction

  • The introduction contrasts sample selection with noise-transition-matrix estimation, which can be difficult to estimate accurately, especially with many classes.Sample selection instead chooses possibly clean mini-batch examples for training.Source: §1 · p. 2
  • Small-loss selection is common because deep networks are assumed to learn clean-label patterns first, making small-loss examples likely to be clean.Prior methods select small-loss examples using the current network’s predictions in each iteration.Source: §1 · p. 2
  • Current-loss selection can miss clean examples when predictions are unstable or when imbalanced data receive large losses and remain unselected.The latter cases can be important for generalization, particularly for non-dominant labels.Source: §1 · p. 2, Figure 1 · p. 3
  • The method extends loss estimation across training iterations, uses robust mean estimators, and applies confidence bounds to conservatively try less-selected examples.The robust estimators use soft or hard truncation; the confidence bounds account for how often an example was selected.Source: §1 · pp. 2–3
  • The authors report theoretical analyses and superior robustness on synthetic balanced and imbalanced datasets and real-world noisy datasets.They present these experiments as validation of the proposed criteria against state-of-the-art methods.Source: §1 · p. 3

2 Method

The method uses losses across training iterations to estimate loss uncertainty and select examples, including those selected less often. Two networks select examples for peer-network updates. [p08, p27, p40]

2.1 Preliminaries

  • The task is to learn a classifier that predicts clean test labels from training data drawn from a corrupted distribution; the paper models each example’s losses across iterations as a Markov process.The loss at each iteration is computed using the classifier’s prediction and the noisy label.Source: §2.1 · p. 4

2.2 Extended Time Intervals

  • Averaging losses across training iterations can mitigate instability in noisy-class posterior estimates, but high-uncertainty outliers can bias the mean.The paper identifies this bias as a possible source of suboptimal classification performance.Source: §2.2 · p. 4

2.3 Robust Mean Estimation and Conservative Search

  • Soft estimation uses an influence function to reduce extreme-loss effects, while hard estimation removes potential outliers with KNN before averaging retained losses.The paper notes that other outlier-detection algorithms could also be used.Source: §2.3 · p. 5
  • The selection criteria use concentration-inequality lower bounds and selection counts to encourage choosing examples selected less often.For rarely selected examples, fixed-length loss intervals are used because replacing the iteration count with the selection count can change the bounds drastically.Source: §2.3 · p. 6
  • CNLCU uses two networks to select a small proportion of each mini-batch and send selected examples to the peer network for updates.The selected proportion decreases as training progresses.Source: §2.3 · p. 7

3 Experiments

Across balanced synthetic datasets, CNLCU methods generally outperform the sample-selection baselines, and they retain clear advantages on synthetic imbalanced data and Clothing1M. The imbalanced experiments also show that the methods make better use of examples from imbalanced classes, while the Clothing1M comparison is against JoCor.

3.1 Experiments on Synthetic Balanced Noisy Datasets

  • On MNIST, CNLCU-S and CNLCU-H achieve the best results in most cases; on F-MNIST and CIFAR-10, they lead the baselines with greater robustness across noise types.S2E outperforms the proposed methods in some MNIST cases, including asymmetric noise; Co-teaching works in some CIFAR-10 cases but does not handle varied noise types.Source: §3.1 · p. 8, Table 1 · p. 9, Table 2 · p. 9
  • CNLCU-S and CNLCU-H are robust to the tested time-interval choices, with soft truncation using shorter intervals than hard truncation because it does not remove losses.The reported ranges are 3–8 for CNLCU-S and 10–15 for CNLCU-H; the interval sensitivity results are in Appendix B.4.Source: §3.1 · p. 9

3.2 Experiments on Synthetic Imbalanced Noisy Datasets

  • The imbalanced experiments reduce examples and labels for classes “0” through “4” to 1% of their previous counts and use asymmetric noise.Test accuracy and the selected ratio of examples with imbalanced classes are measured.Source: §3.2 · p. 10
  • On IM-MNIST, CNLCU-S and CNLCU-H clearly outperform baselines, with nearly a 30% lead under Asym. 10%; on IM-F-MNIST, they lead by about 5%.All baselines show serious early overfitting on IM-MNIST; the caption identifies Figure 2 as test accuracy across epochs for both imbalanced datasets.Source: §3.2 · p. 10, Figure 2 · p. 11
  • The selected ratio for imbalanced classes remains far below the 10% test class prior, but the authors report that a small ratio improvement can accompany considerable test-accuracy gains.They conclude that improving selection of imbalanced-class training examples is challenging but promising for generalization.Source: §3.2 · p. 11, Table 4 · p. 11

3.3 Experiments on Real-world Noisy Datasets

  • On Clothing1M, CNLCU-S and CNLCU-H outperform JoCor by +1.28% and +0.99% on Best, and by +1.01% and +0.54% on Last.The authors report lower results than some state-of-the-art methods using ResNet-50 or additional techniques, while their comparison uses ResNet-18 and focuses on sample selection.Source: Table 5 · p. 12, §3.3 · p. 12

4 Conclusion

The paper addresses uncertainty in losses during noisy-label sample selection. It uses losses across training iterations and confidence-interval estimation, and reports benchmark effectiveness, especially on imbalanced noisy datasets.

4 Conclusion

  • The method uses training losses across iterations and confidence-interval estimation to address uncertainty in sample selection for noisy labels.The conclusion reports benchmark effectiveness and highlights potential for improving model robustness on imbalanced noisy datasets.Source: §4 · p. 12

A.1 Proof of Theorem 1

The appendix derives probability bounds for robust loss estimates and uses concentration results to obtain lower bounds for sample selection. The proof also explains how the soft estimator limits the influence of large potential outliers.

A.1 Proof of Theorem 1

  • The proof establishes probability bounds for the robust estimate, with each bound holding at least 1 − ϵ.It then combines results to obtain a bound that holds with probability at least 1 − 2ϵ.Source: §A.1 · pp. 13–14
  • The selection count n_t denotes how many times an example was selected during the time intervals.The proof incorporates this count into its bound.Source: §A.1 · p. 14
  • The soft estimator’s influence function reduces the negative impact of large potential outliers on mean estimation.The appendix illustrates this mechanism with ψ(x) = log(1 + x + x^2/2).Source: §A.1 · p. 14

A.2 Proof of Theorem 2

  • Theorem 2’s proof applies concentration inequalities to bounded losses, removes potential outliers, and uses the resulting lower bound.The text notes that the loss bound L can be identified by observing loss distributions on noisy datasets.Source: §A.2 · pp. 15–16

B Complementary Experimental Analyses

Table 6 summarizes the synthetic datasets used in the experiments.

  • Table 6 provides a summary of the synthetic datasets used in the experiments.Source: Table 6 · p. 16

B.1 The Details of Datasets and Generating Noisy Labels

The appendix specifies the synthetic noise settings and reports supplementary comparisons with other baseline types. On MNIST and F-MNIST, the proposed methods outperform APL and CDR in classification performance.

B.1 The Details of Datasets and Generating Noisy Labels

  • The appendix states that important dataset statistics are summarized in Table 6.Source: §B.1 · p. 16
  • The experiments use five synthetic noise types across class-dependent and instance-dependent settings.The specified types are symmetric, asymmetric, pairflip, tridiagonal, and instance noise.Source: §B.1 · pp. 16–17
  • The MNIST illustration shows synthetic class-dependent transition matrices at a 20% noise rate.Source: Figure 4 · p. 17, §B.1 · p. 17

B.2 Comparison with Other Types of Baselines

B.3 Experiments on Synthetic CIFAR-100

On CIFAR-100, the proposed method outperforms all baselines in the reported experiments.

B.3 Experiments on Synthetic CIFAR-100

  • The proposed method outperforms all baselines on CIFAR-100.The results are reported as test accuracy over the last ten epochs; the experiments use a 7-layer CNN.Source: Table 9 · p. 18, §B.3 · p. 18

B.4 Experiments for Ablation Study

The ablation study finds that both uncertainty concerns improve robustness, and that combining them and using robust mean estimation performs better. CNLCU-S and CNLCU-H are also robust to choices of time-interval length.

B.4 Experiments for Ablation Study

  • CNLCU-S and CNLCU-H are robust to choices of time-interval length.The study reports this sensitivity result in Figures 5 and 6, whose shaded error bars show standard deviations.Source: §B.4 · p. 18, Figure 5 · p. 18, Figure 6 · p. 19
  • The ablation experiments use MNIST and F-MNIST, with two networks teaching each other; omitting uncertainty reduces the methods to Co-teaching.Source: §B.4 · p. 19
  • The ablations remove either concern about large-loss examples or concern about small-loss examples; Co-teaching-M uses a non-robust mean.Without large-loss concern, networks are not encouraged to choose less-selected examples; without small-loss concern, selection uses only current-network predictions.Source: §B.4 · p. 19
  • Both uncertainty concerns improve robustness, their combination better combats noisy labels, and robust mean estimation outperforms the non-robust mean.The ablation compares the proposed methods with settings that remove one uncertainty concern or use a non-robust mean.Source: §B.4 · p. 19

C Complementary Explanation for Network Structures

The network-structure appendix describes CNNs used across several datasets, including their LReLU slope, and notes that these architectures support direct comparison with prior approaches.

C Complementary Explanation for Network Structures

Loading 2106.00445v1…