PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning with Instance-Dependent Label Noise: A Sample Sieve Approach

Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, Yang Liu

arXiv:2010.02347v2cs.LGstat.ML

TL;DR

Instance-dependent label noise remains difficult because existing theory-supported approaches often require estimating noise rates. CORES2 uses confidence regularization and a dynamic sample sieve to separate clean from corrupted examples without those estimates, with theoretical guarantees and separate training treatments. The analysis assumes Bayes optimal labels equal clean labels, and the authors identify extensions beyond that assumption as future work.

  • Problem

    Instance-dependent label noise, whose errors depend on features and labels, is difficult to handle because existing theory-supported approaches commonly require estimating noise rates.

  • Method

    CORES2 learns a confidence-regularized classifier and dynamically sieves examples by comparing regularized losses with thresholds, without extra noise-rate estimates.

  • Results

    Under its stated conditions, the sample sieve guarantees that clean examples are not identified as corrupted and, with infinite capacity and sufficiently many examples, achieves vn = 1(yn = ˜yn) for all examples.

  • Takeaways & Limitations

    The clean–corrupted split supports training clean examples with standard loss and corrupted examples with an unsupervised consistency loss.

  • Takeaways & Limitations

    The analysis assumes Bayes optimal labels equal clean labels; extending it to cases where they differ remains future work.

Abstract

from arXiv · show

Human-annotated labels are often prone to noise, and the presence of such noise will degrade the performance of the resulting deep neural network (DNN) models. Much of the literature (with several recent exceptions) of learning with noisy labels focuses on the case when the label noise is independent of features. Practically, annotations errors tend to be instance-dependent and often depend on the difficulty levels of recognizing a certain task. Applying existing results from instance-independent settings would require a significant amount of estimation of noise rates. Therefore, providing theoretically rigorous solutions for learning with instance-dependent label noise remains a challenge. In this paper, we propose CORES$^{2}$ (COnfidence REgularized Sample Sieve), which progressively sieves out corrupted examples. The implementation of CORES$^{2}$ does not require specifying noise rates and yet we are able to provide theoretical guarantees of CORES$^{2}$ in filtering out the corrupted examples. This high-quality sample sieve allows us to treat clean examples and the corrupted ones separately in training a DNN solution, and such a separation is shown to be advantageous in the instance-dependent noise setting. We demonstrate the performance of CORES$^{2}$ on CIFAR10 and CIFAR100 datasets with synthetic instance-dependent label noise and Clothing1M with real-world human noise. As of independent interests, our sample sieve provides a generic machinery for anatomizing noisy datasets and provides a flexible interface for various robust training techniques to further improve the performance. Code is available at https://github.com/UCSC-REAL/cores.

1 INTRODUCTION

The paper addresses the difficulty of learning from instance-dependent label noise, where annotation errors depend on features and labels. CORES2 provides a theoretically guaranteed sample sieve without estimating noise rates, then trains on clean and corrupted examples separately.

  • Research gap: Instance-dependent annotation noise makes avoiding overfitting especially challenging because errors depend on both features and true labels.Human annotations can vary in error rates with task difficulty.
  • Research gap: Existing theory-supported approaches for instance-dependent noise commonly require estimating noise rates, while distribution learning and threshold-based distillation are also difficult or sensitive with limited data.Learning an accurate noisy distribution for each example is non-trivial, and distilled-example quality depends on thresholds.
  • Approach: CORES2 uses a theoretically guaranteed sample sieve to split clean and corrupted examples without specifying or estimating noise rates.The method focuses on learning the underlying clean distribution rather than noisy distributions or rates.
  • Approach: The method trains clean examples with standard loss and corrupted examples with an unsupervised consistency loss, treating the two groups separately.The paper presents this separation as providing performance improvements.

2 CORES2: CONFIDENCE REGULARIZED SAMPLE SIEVE

CORES2 combines confidence regularization with a dynamic sample sieve for instance-dependent noisy-label learning. Its sieve evaluates regularized example losses against thresholds, while the resulting clean and corrupted subsets receive different training treatments.

  • 2.1 CONFIDENCE REGULARIZATION: Under stated conditions, minimizing the confidence-augmented loss encourages predictions toward confident class assignments rather than locally optimal interior probabilities.In the binary case, the prediction approaches one class probability equal to 1.
  • 2.1 CONFIDENCE REGULARIZATION: The confidence-regularized loss subtracts the expected cross-entropy against labels sampled from the noisy dataset.The regularizer is ℓCR(f(xn)) := −β · E[ℓ(f(xn), eY)], with positive β.
  • 2.1 CONFIDENCE REGULARIZATION: The paper distinguishes ℓCR from entropy regularization because ℓCR is built on ln p and is presented with theoretical and experimental advantages for instance-dependent noise.Entropy regularization instead uses −p ln p.
  • 2.2 CONFIDENCE REGULARIZED SAMPLE SIEVE: CORES2 dynamically alternates confidence-regularized model updates with sample-sieve updates, representing clean status with vn and using a non-backpropagating model copy.The dynamic procedure addresses the difficulty of directly solving the combinatorial optimization.
  • 2.2 CONFIDENCE REGULARIZED SAMPLE SIEVE: The sieve compares each example’s regularized loss with a closed-form threshold αn,t obtained from the current model and example, requiring no extra estimation.The threshold controls which examples are sieved out.
  • 2.2 CONFIDENCE REGULARIZED SAMPLE SIEVE: With ℓCR, CIFAR-10 loss distributions form distinct clean and corrupted clusters separable at threshold 0, whereas standard cross-entropy can overfit during late training.The comparison covers both symmetric and instance-based noise visualizations.

3 THEORETICAL GUARANTEES OF CORES2

CORES2 provides theoretical guarantees for handling instance-dependent label noise without requiring feature-independence or explicit noise-rate estimation. Its confidence-regularized loss supports clean-distribution recovery and a progressively safer sample sieve, while finite-sample and extreme-noise conditions limit the guarantees.

  • The sample sieve does not identify a clean example as corrupted when its model prediction is better than random guess.This condition yields progressive filtering and improves the quality of the clean/corrupted division.
  • CORES2 addresses general instance-based noise transition matrices, including settings where noise depends on features rather than assuming feature-independent noise.
  • Confidence-regularized CE can be equivalent to clean-distribution CE under the stated assumptions, controlling noise-related terms through the regularization design.The decoupling separates clean-loss, class-prior, and corrupted-example effects into three additive terms.
  • Under infinite capacity, sufficient data, and the theorem’s assumptions, CORES2 achieves an effectively clean set of sieved clean examples.The accompanying argument states that the sieve achieves 100% precision on both clean and corrupted examples under these idealized conditions.
  • Theoretical guarantees may fail for extreme-noise instances when no suitable β exists, requiring auxiliary methods such as sample pruning.With finite data, direct ERM can also be sub-optimal because noise reduces loss discrimination and increases selection variance.
  • Sieved clean examples can be used for ERM alone, while the clean/corrupted split also supports robust techniques such as semi-supervised learning.

4 EXPERIMENTS

CORES2 is evaluated on synthetic symmetric, asymmetric, and instance-dependent noise across CIFAR-10 and CIFAR-100, plus real human noise in Clothing1M. Its sieve separates examples for supervised training and supports consistency-based use of filtered examples.

  • Evaluation settings: The experiments cover symmetric, asymmetric, and instance-dependent label noise, with instance-dependent noise identified as the more challenging setting.Instance-dependent noise is generated following Xia et al. (2020).
  • Post-sieve training: After sieving, clean examples guide training with cross-entropy, while filtered examples are dropped from supervised training and used for prediction-consistency regularization.The consistency loss minimizes KL divergence between predictions on original and augmented features.
  • Sample-sieve quality: F-score measures the sieve’s clean-example quality through the harmonic mean of precision and recall.Figure 3 compares CORES2 with Co-teaching and Co-teaching+ over training epochs.
  • Sample-sieve quality: CORES2 and Co-teaching maintain consistently high F-scores under symmetric noise, while CORES2 performs better under instance-dependent noise, especially at 60% noise.The other two methods have low F-scores at the 60% instance-dependent noise rate.
  • Evaluation settings: CORES2 is evaluated on CIFAR-10, CIFAR-100, and Clothing1M using synthetic and real-world label noise.ResNet-34 is used for CIFAR-10 and CIFAR-100, and ResNet-50 for Clothing1M.
  • Test accuracy: Across CIFAR benchmarks, CORES2 works consistently well under instance-based noise, and consistency training further improves results; it also performs fairly well on Clothing1M.The comparisons include symmetric, asymmetric, and instance-based settings.

5 CONCLUSIONS

The conclusion presents CORES2 as robust to general instance-dependent label noise without explicit noise-rate knowledge, while identifying an assumption that limits the current analysis.

  • Conclusion: CORES2 is guaranteed to sieve corrupted examples under general instance-dependent label noise without explicit knowledge of label-noise rates.The analysis assumes that Bayes-optimal labels equal clean labels.
  • Limitations and future work: Extending the analysis to cases where Bayes-optimal labels differ from clean labels remains a future direction.The paper also proposes exploring alternative robust training designs with sieved examples.

CONDITIONS REQUIRED FOR THEOREM 1

Theorem 1 is developed within a proof framework that uses stated model, optimization, and smoothness assumptions, alongside a decoupling analysis of the regularized CE loss.

  • Assumptions: The theorem analysis assumes infinite model capacity, gradient-descent updates, and a smooth network function without singular points.The assumptions are stated as A1–A3 in the theorem-development framework.
  • Context: The related-work discussion contrasts loss correction, bounded-loss designs, and sample selection with the paper’s instance-dependent-noise setting.Prior correction methods commonly require estimating a transition matrix or noise rates, while sample-selection methods mainly target feature-independent noise.
  • Loss decomposition: Theorem 3 states that the expected regularized CE loss decouples into three additive terms.The proof proceeds through the expected noisy CE loss, the expected confidence-regularization loss, and their combined form.
  • Loss decomposition: The decomposition introduces class-dependent quantities involving transition probabilities and conditional expected losses.The notation includes Δ_j, U_ij(X), and an indicator-based expectation term.

B.2 PROOF FOR THEOREM 1

Theorem 1’s proof argues that confidence-regularized training creates a decreasing direction away from fully positive predictions on mislabeled examples, while the sieve preserves sufficiently confident clean examples.

  • Theorem 1 argument: For any example whose predicted probabilities are positive for all classes, the confidence-regularized objective admits a decreasing direction and therefore is not locally optimal there.The proof states that this conclusion holds for any data distribution, including the noisy distribution.
  • Theorem 1 argument: The proof constructs label-wise variations around a feature point and scales positive and negative components to preserve the probability-sum constraint.A continuous neighborhood construction preserves continuity of the network outputs.
  • Optimization dynamics: The dynamical analysis models output changes through network-parameter updates under gradient descent and smoothness assumptions.The network output before softmax is denoted by g_y(x).
  • Sieve guarantee: The sample sieve guarantees that a clean example is not identified as corrupted when the model predicts its true label better than random guessing.The condition is f_xn[y_n] > 1/K.

B.4 PROOF FOR THEOREM 4

The proof establishes conditions under which confidence-regularized cross-entropy on the noisy distribution has the same minimizer as clean cross-entropy. It does so by analyzing label shift and bounding the regularization strength.

  • Regularization effect: Adding the second regularization term preserves the Bayes-optimal label because the class-separation error satisfies ϵX < 0.5.The proof defines ϵX through the probability of matching the Bayes-optimal label.
  • Label-shift analysis: Label shift preserves the Bayes-optimal label under a sufficient separation condition on the feature-dependent class probabilities.The shifted distribution preserves P(X|Y) while allowing class priors to change.
  • Theorem 4: Under Assumptions 1 and 2, minimizing noisy confidence-regularized CE is equivalent to minimizing clean CE.This is the stated robustness guarantee of Theorem 4.
  • Regularization effect: The regularization coefficient β must be large enough to encourage confident predictions but not so large that it biases the objective away from clean results.The proof uses an upper bound on β to maintain unbiasedness with respect to clean outcomes.

C.1 COMPARING ℓCR WITH ENTROPY REGULARIZATION

This section contrasts confidence regularization with entropy regularization. The paper argues that confidence regularization produces stronger gradients and reports higher CIFAR-10 accuracy in the comparison experiment.

  • Regularizer comparison: Confidence regularization uses ln p, whereas entropy regularization uses −p ln p for a class prediction probability p.The two regularizers therefore have different functional forms.
  • Gradient analysis: Confidence regularization has larger gradients than entropy regularization for 0 < p < 0.5.The proposition concludes that its gradient-based regularization ability is stronger.
  • Gradient analysis: The compared regularization functions are symmetric around p = 0.5, so the proof analyzes only 0 < p < 0.5.Both gradients are positive in this interval.
  • Empirical comparison: The larger gradient of confidence regularization results in more confident predictions, according to the paper’s analysis.Figure 4 visualizes the regularization values as functions of p.
  • Empirical comparison: The CIFAR-10 comparison experiment reports higher accuracy for the proposed regularization than for the entropy term.The supplied text does not provide the numerical values from Table 4.

C.3 ANALYSIS FOR THE RISK BOUND

The risk analysis treats the sieved clean subset as potentially biased under instance-dependent noise. It introduces importance weighting to recover the clean-distribution risk and states a corresponding risk bound.

  • Importance weighting: Importance weights γ(X) correct the selection bias so the clean risk equals the weighted risk over the sieved clean distribution.The weight is defined as γ(X) := P_D(X) / P_eDL∗(X).
  • Limitations: Choosing the kernel Φ(·) for importance-weight estimation is non-trivial, especially for complicated features.This is an implementation boundary of the proposed correction.
  • Risk bound: Corollary 2 provides a high-probability risk bound when γ · ℓ is [0, b]-valued.The bound holds with probability at least 1 − δ.
  • Selection bias: The sieved clean examples may exhibit covariate-shift bias because selection depends on instance-based label noise.The paper therefore does not treat the sieved subset as automatically representative of the clean distribution.
  • Limitations: Estimating γ(X) can degrade performance, creating a trade-off between correcting selection bias and errors in the estimated weights.The paper notes that kernel mean matching and DNN adaptations can estimate γ(X).

D.2 IMPLEMENTATION DETAILS AND MORE ANALYSIS

The implementation section specifies training and noise-generation procedures, then examines how β affects the sieve. The reported comparisons describe CORES2 as competitive across additional benchmarks.

  • CIFAR implementation: CIFAR-10 and CIFAR-100 use SGD with batch size 64, initial learning rate 0.1, momentum 0.9, and 100 training epochs.The learning rate decays by 0.1 at epoch 50, with standard augmentation and weight decay 0.0005.
  • CORES2⋆: CORES2⋆ uses the epoch-40 sieve, although the sieve may not yet have reached its highest test accuracy.The authors report that the confidence regularizer’s division property works well at that stage.
  • β analysis: Small β weakens the division property, whereas large β biases training toward the CE term and causes clean and corrupted examples to overlap.Figure 6 visualizes the effect using β = 0.5, 2, and 10.
  • Noise generation: Instance-dependent noise is generated by sampling per-example flip rates, transforming features with random weights, and assigning wrong labels with probability q_n.The global noise rate ε controls the truncated-normal distribution from which the instance-specific rates are sampled.
  • Benchmark comparisons: CORES2 achieves competitive results against recent benchmark methods on CIFAR-10 and Tiny-ImageNet.The supplied passage summarizes Tables 5 and 6 without reporting their numerical accuracies.

D.5 ABLATION STUDY

The ablation study evaluates CORES2 and its components on CIFAR-10 using ResNet-34. Both dynamic sample sieving and confidence-regularized model updates improve final accuracy, while CORES2 remains strongest without consistency training.

  • Without consistency training: Without consistency training, CORES2 still achieves the best performance among the compared noise-robust methods.The model can be forced to concentrate only on clean examples by optimizing its loss.
  • Component analysis: The CIFAR-10 ablation compares traditional CE, sample sieving with CE, typical CORES2, and CORES2⋆ using ResNet-34.The table analyzes the contributions of data selection and confidence regularization.
  • Component analysis: Both dynamic sample sieving and confidence-regularized model updates show positive effects on final accuracy.The comparison supports the inclusion of both components in CORES2.
Loading 2010.02347v2…