Open-set Label Noise Can Improve Robustness Against Inherent Label Noise

1 Introduction

• The paper frames its question against prior approaches that treat open-set noise as harmful and mitigate it through detection, loss correction, or filtering.The motivating concern is that imperfect labeling methods can produce noise that DNNs memorize, harming generalization.Source: §1 · p. 1
• Open-set noises can become harmless as their sample count grows and can improve robustness to inherent noisy labels, unlike the closed-set counterpart.The paper offers an insufficient-capacity interpretation: fitting more open-set samples slows fitting inherent noise.Source: §1 · p. 2
• ODNL assigns open-set samples independently sampled random labels that change each epoch, consuming extra network capacity to prevent overfitting inherent noisy labels.The method is presented as a complementary regularization for noisy-label training.Source: §1 · p. 2
• ODNL-induced SGD noise is random-direction, conflict-free, and biased, properties linked respectively to flat minima, training stability, and conservative OOD predictions.The paper connects these properties to robustness against noisy labels and OOD detection.Source: §1 · p. 2
• Experiments on simulated and real-world CIFAR-10, CIFAR-100, and Clothing1M datasets report that ODNL enhances existing robust algorithms and improves OOD detection with noisy training labels.The authors describe the approach as applicable across a wide range of existing algorithms.Source: §1 · p. 2

2.1 Preliminaries: learning with inherent noisy labels

• The setup treats observed training labels as potentially different from ground truth and defines open-set auxiliary instances as OOD samples disjoint from the training distribution.The setting uses a DNN classifier with a softmax output and standard cross-entropy training.Source: §2.1 · pp. 2–3

2.2 Motivation: the role of open-set label noise in generalization and robustness

• With inherent noisy labels on CIFAR-10, increasing open-set auxiliary-set size can improve generalization beyond standard training, whereas closed-set auxiliary noise remains harmful.Figure 1 compares accuracy under inherent noise across auxiliary-set sizes and reports standard training for comparison.Source: Figure 1 · p. 3, §2.2 · p. 3
• On clean CIFAR-10 labels, open-set auxiliary noise becomes gradually harmless as its dataset grows, while closed-set noise consistently harms generalization and yields unstable training.Figure 1 reports accuracy and standard-deviation error bars across auxiliary-set sizes.Source: Figure 1 · p. 3, §2.2 · p. 3
• The observational study compares fixed random labels on open-set and closed-set auxiliary datasets, varying their sizes and evaluating CIFAR-10 with clean or 40%-symmetric-noise labels.It uses WRN-40-2, 300K Random Images as open-set data, and CIFAR-5m as closed-set data.Source: §2.2 · p. 3
• Training-loss plots support the interpretation that increasing the number of open-set auxiliary samples slows fitting inherent training-set noise; larger auxiliary datasets also require more time to memorize open-set noises.Source: §2.2 · pp. 3–4

3 Method: Open-set regularization with Dynamic Noisy Labels

• ODNL uses open-set auxiliary samples as a task that consumes extra network capacity without interfering with clean-data patterns, helping prevent overfitting noisy data.The method is motivated by the finite capacity of neural networks and the risk of memorizing inherent noisy labels.Source: §3 · p. 4
• Dynamic Noisy Labels independently sample each open-set instance's label uniformly from the class set and change those labels across epochs.Resampling makes the finite auxiliary task continuously difficult to memorize, unlike fixed random labels.Source: §3 · p. 4
• ODNL combines the original-data cross-entropy loss with an auxiliary open-set loss weighted by η.The original training objective uses standard cross-entropy; the auxiliary objective enforces consistency with dynamically sampled labels.Source: §3 · p. 4
• The regularization can be incorporated into sample-selection, loss-correction, and robust-loss methods by adding it to their original learning objectives.The paper characterizes ODNL as a general, complementary method.Source: §3 · p. 4

4 Understanding from SGD noise

• ODNL's random-direction noise is uniformly sampled from open-set noisy-sample gradients and may help models escape local minima.The paper links this noise to difficulty converging and escaping local minima.Source: §4 · pp. 4–5
• Because open-set samples make the induced noise independent of original-example gradients, closed-set samples may instead create conflicting noise and unstable training.Synthetic auxiliary data experiments report better robustness to inherent noise when using open-set auxiliary data.Source: §4 · p. 5, Figure 3 · p. 6
• ODNL's nonzero expected noise biases predictions toward more conservative outputs on OOD samples, supporting improved OOD detection.The paper relates this expected bias to OE's regularization effect on OOD examples.Source: §4 · pp. 5, 7
• ODNL and SLN lead to flat minima on CIFAR-10 with symmetric-40% noisy labels, while Standard and OE lead to sharp minima.The landscapes use the same-scale z-axis, with loss distributions shown separately by color bars.Source: Figure 2 · p. 6
• ODNL further improves SLN across varied label-noise types, while OE lacks the dynamic noise needed to improve robustness against noisy labels.The paper reports ODNL and OE have similar regularization effects on OOD examples because their induced biases are equal in expectation.Source: Figure 3 · p. 6, §4 · pp. 6–7
• SLN induces Gaussian SGD noise with adjustable standard deviation, and the paper reports that ODNL further improves SLN's performance across varied label-noise types.The authors characterize SLN and ODNL as different SGD-noise variants.Source: §4 · p. 6

5.1 Setups

• The benchmark setup covers symmetric, asymmetric, instance-dependent, and open-set synthetic noise on CIFAR-10/100, alongside real-world Clothing1M noise.CIFAR-10 open-set noise replaces 40% of training images with CIFAR-100 images; CIFAR-10/100 training uses WRN-40-2 and 300K Random Images as the auxiliary dataset.Source: §5.1 · p. 7

5.2 CIFAR-10 and CIFAR-100

• ODNL consistently improves existing robust training methods across noisy-label types on CIFAR-10 and CIFAR-100, including open-set noise where most methods alone do not help.The CIFAR-100 table reports average test accuracy and standard deviation across five trials; the supplied text gives no numerical cell values.Source: §5.2 · p. 8, Table 2 · p. 8
• For symmetric-40% noisy labels on CIFAR-10, larger η nearly eliminates the post-140th-epoch decline in test accuracy.Figure 4 presents the sensitivity analysis across η values.Source: §5.2 · p. 8, Figure 4 · p. 8

5.3 Clothing1M

• On Clothing1M, vanilla ODNL outperforms simple baselines and further improves SLN and DivideMix, supporting its applicability to large-scale real-world noisy-label scenarios.The experiments use a class-balanced training subset each epoch, pretrained ResNet-50, and ImageNet auxiliary data with clothing-related classes removed.Source: Table 3 · p. 8, §5.3 · p. 9

5.4 OOD detection with noisy labels

• With symmetric-40% noisy labels on CIFAR-10, ODNL improves test accuracy and OOD detection, whereas OE does not improve generalization performance.Detection performance is averaged over three noise types and seven OOD datasets in the prose; Table 4 describes averages over ten test datasets.Source: §5.4 · p. 9, Table 4 · p. 9

• Noisy-label methods include sample selection, clean-sample weighting, loss correction, robust losses, regularization, and semi-supervised training strategies.Source: §6 · p. 9
• Auxiliary datasets have been used for representation learning, OOD detection, and adversarial robustness; unlike cited adversarial-robustness work requiring in-distribution unlabeled data, this method uses out-of-distribution instances.Source: §6 · p. 10
• OOD detection work includes softmax confidence baselines, synthesized or unlabeled auxiliary data, and energy scores; the paper also reports OOD-detection performance under noisy training labels.Source: §6 · p. 10

7 Conclusion and Discussion

• ODNL enhances existing robust training methods against inherent noisy labels and also improves OOD detection.The conclusion describes the method as computationally inexpensive and applicable to large-scale real-world scenarios.Source: §7 · p. 10
• The experiments consider only instance-independent open-set noisy labels; results may differ when open-set labels depend on the instance.The paper notes that real open-set samples can resemble clean samples while still belonging to another class.Source: §7 · p. 10
• The method uses realistic open-set auxiliary data, while whether synthesized data can help remains an open question.Adversarial examples with dynamic noisy labels are offered as a possible synthesized-data direction.Source: §7 · p. 10

A Proofs

• Proposition 4 states that Eq. (6) induces noise z under cross-entropy loss.The proof introduces the noisy gradient and specifies cross-entropy as ℓ(f, y) = −ey log f.Source: A Proofs · p. 16
• The proof simplifies the noise expression by using the one-hot target vector, whose only nonzero entry is at the target class.It then denotes the i-th component of the noise as z_i.Source: A Proofs · p. 16
• For cross-entropy, Proposition 3 states that each open-set sample induces a bias term on the gradient of the loss.The expression includes element-wise division, as specified in the proposition.Source: A Proofs · p. 16

B.1 Datasets

• CIFAR-10 and CIFAR-100 experiments use 300K Random Images as open-set auxiliary data, while Clothing1M experiments use ImageNet 2012 with clothing classes removed.The 300K Random Images dataset excludes CIFAR, Places, and LSUN classes and images with divisive metadata; CIFAR-5m supplies closed-set samples for selected figures.Source: §B.1 · p. 17
• OOD evaluation covers artificial distributions and real datasets including SVHN, Places365, LSUN-Crop, LSUN-Resize, and iSUN.CIFAR-100 is also used as an OOD test dataset for CIFAR-10 experiments.Source: §B.1 · p. 17

B.2 Setups

• Training protocols vary by experiment: WRN-40-2 is used for several figures, WRN-28-2 for the SLN comparison, and pretrained ResNet-50 for Clothing1M.The compared methods are reimplemented with a shared backbone for fair comparison, while Clothing1M follows DivideMix settings.Source: §B.2 · p. 17
• η is tuned by binary search using validation accuracy: a late-stage decline indicates overfitting and η below the best value.The appendix reports vanilla ODNL’s best η values in Table 5.Source: §B.2 · pp. 17–18, Table 5 · p. 18

C.1 Our method also works on other architectures.

• ODNL improves test accuracy with ResNet-18 and VGG-11 under noisy-label settings beyond WRN architectures.Table 6 reports CIFAR-10 test accuracy under symmetric-40% noise across architectures.Source: Table 6 · p. 18, §C.1 · p. 18

C.2 Different choices of open-set auxiliary datasets.

• 50,000 examples from 300K Random Images achieve comparable performance to using the full auxiliary dataset.The authors report that simple Gaussian noise also improves robustness, but less than real data: 80.24% accuracy versus 82.03% on CIFAR-10 with symmetric-40% noise.Source: §C.2 · p. 18
• Different CIFAR-100 auxiliary subsets achieved very close results, with 81.56% test accuracy, despite differing numbers of classes.The authors report that auxiliary-data diversity was not important for noisy-label robustness in this experiment.Source: §C.2 · p. 18

C.3 Detailed results for OOD detection.

• Under symmetric-40% noisy labels, ODNL outperforms OE on CIFAR-10 OOD detection across the reported test datasets.Source: §C.3 · p. 18, Table 7 · p. 19

D Discussion

• Unlike OE and OAT’s uniform softmax regularization, ODNL induces dynamic noise; the discussion links its rationale to open-set noise that can benefit robustness against inherent noisy labels.Source: D Discussion · p. 19

C.3 Detailed results for OOD detection.

• With clean labels, ODNL achieves comparable OOD detection performance to OE on CIFAR-10.Source: §C.3 · p. 18, Table 8 · p. 20