Source-linked AI summary
Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise
Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, Pheng-Ann Heng
TL;DR
The paper addresses the reliance on CCN despite evidence that real-world label noise depends on instances. It tests and formalizes IDN, proposes SEAL, and reports effectiveness across synthetic and real-world noise, including a 1.56% absolute Clothing1M improvement. Its experiments retrain networks from scratch each iteration, increasing computational cost, although practical savings may be possible.
Problem
Real-world label noise is unlikely to satisfy CCN, motivating study of the more practical instance-dependent noise setting.
Method
The paper formalizes controllable IDN and proposes SEAL, which averages DNN predictions for each instance over training before retraining on soft labels.
Results
SEAL is effective under synthetic IDN and real-world noise, improving Clothing1M test accuracy by 1.56% absolute with normal cross-entropy.
Takeaways & Limitations
The theoretical analysis provides rigorous motivation for studying IDN, an important topic for future research.
Takeaways & Limitations
Experiments retrain networks from scratch at each SEAL iteration, resulting in scaled computational cost, though reserving the best model and training fewer epochs may reduce it.
Abstract
from arXiv · showhide
Supervised learning under label noise has seen numerous advances recently, while existing theoretical findings and empirical results broadly build up on the class-conditional noise (CCN) assumption that the noise is independent of input features given the true label. In this work, we present a theoretical hypothesis testing and prove that noise in real-world dataset is unlikely to be CCN, which confirms that label noise should depend on the instance and justifies the urgent need to go beyond the CCN assumption.The theoretical results motivate us to study the more general and practical-relevant instance-dependent noise (IDN). To stimulate the development of theory and methodology on IDN, we formalize an algorithm to generate controllable IDN and present both theoretical and empirical evidence to show that IDN is semantically meaningful and challenging. As a primary attempt to combat IDN, we present a tiny algorithm termed self-evolution average label (SEAL), which not only stands out under IDN with various noise fractions, but also improves the generalization on real-world noise benchmark Clothing1M. Our code is released. Notably, our theoretical analysis in Section 2 provides rigorous motivations for studying IDN, which is an important topic that deserves more research attention in future.
1 Introduction
Existing label-noise work commonly relies on CCN, although samples sharing a class can have different mislabeling probabilities. The paper motivates IDN and proposes SEAL to correct instance-dependent noise.
- Most prior theoretical analyses and noise-synthesis experiments use the CCN assumption, where noise is independent of input features given the true label.
- Samples sharing a class can differ substantially, making equal mislabeling probabilities across the class problematic.
- The paper tests CCN on Clothing1M, demonstrates the need to study IDN, and characterizes IDN as challenging through theoretical and empirical evidence.
- SEAL averages each instance’s DNN predictions over training and retrains using the resulting soft labels for instance-dependent correction.
- SEAL improves generalization on Clothing1M, achieving a 1.56% absolute improvement with normal cross-entropy and improving DMI initialization.
2 From CCN to IDN - Theoretical Evidence
The paper contrasts CCN, whose fixed transition matrix ignores input features, with IDN, whose transition matrix depends on the instance. A Clothing1M hypothesis test rejects CCN, while theory shows that clean-sample selection can fail under IDN.
- CCN model: Under CCN, the observed noisy label is conditionally independent of the input given the true label and is generated by a fixed transition matrix M.
- CCN hypothesis testing: The Clothing1M test yields a contradiction with probability below 10^-21250, implying that CCN does not hold for this dataset.
- CCN hypothesis testing: The observed error is lower than the CCN-derived error because the network learned feature-dependent noise that generalized to the noisy validation set.
- IDN model: Under IDN, the noise transition matrix M(X) is a function of the input, with CCN as the special case in which all instances share the same matrix.
- IDN challenges: Existing robust-loss guarantees generally assume CCN, while identifying clean samples becomes more difficult under IDN because noise correlates with confusing input features.
- IDN challenges: Optimal sample selection exists theoretically under CCN but may fail under IDN because selected clean samples can have an input distribution differing from the original distribution.
3 A typical controllable IDN
The paper proposes a controllable instance-dependent noise generator based on DNN prediction errors and characterizes how DNNs train under IDN versus CCN. Experiments show that IDN is easier to fit, degrades early generalization, and weakens but does not eliminate memorization effects; instance-level memorization also varies substantially.
- 3.1 Enabling controllable experiments: The IDN generator flips the labels of the p% instances with the highest mislabeling scores, producing noise tied to DNN prediction errors.It uses predictions across training epochs to identify difficult instances and their likely alternative labels.
- 3.1 Enabling controllable experiments: The generated noise is instance-dependent because it comes from prediction errors and is challenging because it targets classes that the DNN finds difficult to distinguish.The authors also report that examples of the resulting noisy samples are semantically meaningful.
- 3.2 Characterizations of training with IDN: Under matched noise fractions, IDN yields higher training accuracy than CCN, indicating that DNNs fit IDN more easily.The feature-dependent noisy labels can mislead DNNs and contribute to overfitting.
- 3.2 Characterizations of training with IDN: Under IDN, peak testing accuracy is lower and early generalization degrades, with testing accuracy decreasing from very early training on MNIST.The paper relates this behavior to DNNs fitting IDN easily and overfitting feature-dependent noise.
- 3.2 Characterizations of training with IDN: The memorization effect remains under IDN but is less significant than under CCN, with lower peak testing accuracy and CSR and a smaller peak-to-converged accuracy gap.Testing accuracy can still peak before all training samples are memorized, while CSR increases during training.
- 3.2 Characterizations of training with IDN: For individual instances under 20% IDN, DNNs eventually memorize wrong labels, while true-label outputs can remain activated with oscillation and differ in timing and intensity across instances.Figure 3 plots the noisy-label and latent true-label softmax outputs across training epochs.
4 SEAL: a primary attempt to combat IDN
SEAL performs instance-dependent label correction by averaging a DNN’s predictions throughout training and retraining on the resulting soft labels. Iterative correction is supported by theoretical intuition and experiments on synthetic IDN and Clothing1M.
- Method: SEAL averages each instance’s predictions over training, then retrains a classifier using the averaged soft labels for instance-dependent correction.Multiple iterations reuse the previous soft labels as input, forming the paper’s self-evolution procedure.
- Method: SEAL’s averaged label correction is expected to improve on noisy labels while reducing variance, and repeated iterations are expected to approach optimal labels.The averaging is motivated by DNN prediction oscillations before memorizing noise.
- Method: Unlike existing pseudo-labeling methods, SEAL avoids specifically tuning when the network has learned useful features but before it fits too much noise.The paper states that canonical training hyperparameters can be selected by examining training accuracy on noisy data.
- Empirical evaluation: Across MNIST and CIFAR-10 experiments with varying IDN fractions, SEAL is evaluated against CE, Forward, Co-teaching, GCE, DAC, DMI, and other baselines.The experiments use CNNs for MNIST and Wide ResNet 28×10 for CIFAR-10.
- Empirical evaluation: In Figure 5, iterative SEAL shifts noisy-instance distance distributions toward lower values, while Figure 6 shows examples of identified and corrected noisy labels.The distance is normalized to [0, 1], and the figures examine correction across iterations and individual instances.
- Empirical evaluation: 1.56% absolute improvement is reported for SEAL with cross-entropy on Clothing1M, and SEAL also improves DMI when used as initialization.Clothing1M contains 1M noisy training instances and clean validation and testing sets.
5 Conclusion
The paper argues that label noise should be studied beyond the CCN assumption and presents IDN and SEAL as steps toward that goal. It concludes that IDN merits substantially more research attention.
- Theoretical analysis justifies moving beyond the class-conditional noise assumption toward instance-dependent noise.
- The paper formalizes controllable IDN generation and characterizes IDN as semantically meaningful and challenging.
- SEAL is proposed as a primary method for combating IDN and is reported effective on synthetic IDN and real-world noise.
- Learning with IDN is identified as an important topic deserving more research attention.
A Examples of noisy samples
Figure 7 illustrates that the IDN labeler produces instance-dependent noise even among examples sharing the same true class. Each row contains mostly mislabeled instances alongside two correct ones.
- Each row represents instances from the same true class under the IDN labeler.
- Within each row, two instances retain correct labels while eight instances are mislabeled.
- The figure uses “airp.” and “auto.” as abbreviations for airplane and automobile.
B More details on experiments
The experiments use different network architectures, training schedules, optimizers, and datasets for MNIST, CIFAR-10, and Clothing1M. These settings follow dataset-specific configurations, including a pretrained ResNet-50 and clean validation data for Clothing1M.
- MNIST: MNIST uses a four-layer CNN with 28×28 inputs, 50 training epochs, batch size 64, and SGD momentum 0.5.
- CIFAR-10: CIFAR-10 uses Wide ResNet 28×10 for 150 epochs with batch size 128, SGD momentum 0.9, and weight decay 5 × 10^-4.
- Clothing1M: Clothing1M uses ImageNet-pretrained ResNet-50, a 14K clean validation set, and 10 epochs on 1M noisy training instances.