Source-linked AI summary
Dimensionality-Driven Learning with Noisy Labels
Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah M. Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, James Bailey
TL;DR
Noisy labels make accurate DNN training and generalization difficult. The paper analyzes the dimensionality of learned representation subspaces, uses LID to monitor training, and adapts the loss through D2L. It reports strong classification performance under high proportions of noisy labels while learning low-dimensional representations that capture the data distribution.
Problem
DNNs may generalize poorly on datasets with high proportions of incorrect class labels, creating a need to understand and address this behavior.
Method
The paper uses Local Intrinsic Dimensionality to monitor representation subspaces and adapts the loss when training shifts from dimensionality compression to expansion.
Results
D2L shows strong classification performance across scenarios with high proportions of noisy labels and learns low-dimensional subspaces capturing the underlying data distribution.
Takeaways & Limitations
Dimensionality-based analysis provides a perspective for understanding noisy-label learning and supports a training strategy that suppresses subsequent dimensionality expansion.
Takeaways & Limitations
The effects of adversarial or corrupted inputs and asymmetric label noise on local subspace dimensionality and DNN learning remain open issues.
Abstract
from arXiv · showhide
Datasets with significant proportions of noisy (incorrect) class labels present challenges for training accurate Deep Neural Networks (DNNs). We propose a new perspective for understanding DNN generalization for such datasets, by investigating the dimensionality of the deep representation subspace of training samples. We show that from a dimensionality perspective, DNNs exhibit quite distinctive learning styles when trained with clean labels versus when trained with a proportion of noisy labels. Based on this finding, we develop a new dimensionality-driven learning strategy, which monitors the dimensionality of subspaces during training and adapts the loss function accordingly. We empirically demonstrate that our approach is highly tolerant to significant proportions of noisy labels, and can effectively learn low-dimensional local subspaces that capture the data distribution.
1. Introduction
This paper explains noisy-label generalization through changes in the dimensionality of DNN representation subspaces and proposes a loss-adaptation strategy based on that analysis.
- Noisy class labels can cause DNNs to generalize poorly, motivating training methods that remain effective under variable noise levels.
- DNNs exhibit distinctive dimensionality-based learning styles when trained with clean versus noisy labels.
- LID identifies the transition from early dimensionality compression to later dimensionality expansion during noisy-label training.
- D2L modifies the loss function at this transition to avoid the later expansion stage and reduce overfitting to noisy labels.
- On MNIST, SVHN, CIFAR-10, and CIFAR-100, D2L learns low-dimensional subspaces capturing the data distribution, simpler hypotheses, and high-quality deep representations.
2. Related Work
Prior work studied noisy-label learning through memorization, noise modeling, relabeling, and regularization, but this paper adds a local-subspace dimensionality perspective and uses it to regulate training.
- Earlier studies reported that noisy labels require greater capacity and longer training, produce more complex hypotheses, and can lead DNNs to memorize training labels.
- The paper identifies an incomplete account of how clean-label and noisy-label learning differences relate to DNN generalization.
- Existing robust-training approaches include expert relabeling, noise-aware models, and loss-correction methods based on explicit or implicit noise formulations.
- D2L is a loss-correction method that regulates learning with estimates of local intrinsic dimensionality and is empirically compared with other loss-correction strategies.
- Unlike prior dimensionality-based regularization focused on global data characteristics, this work monitors local dimensional characteristics to modify DNN learning under noisy labels.
3. Dimensionality of Deep Representation Subspaces
The paper uses Local Intrinsic Dimensionality (LID) to assess the dimensional complexity of deep representation subspaces, estimating it from local neighbor distances after network transformation. Experiments show that clean and noisy labels produce distinct dimensionality dynamics, including compression followed by expansion under label noise.
- 3.1. Local Intrinsic Dimensionality (LID): Local Intrinsic Dimensionality (LID) measures the expansion-based intrinsic dimensionality of a local data subspace or submanifold.LID reflects how rapidly the cumulative distance distribution grows as distance from a reference sample increases.
- 3.1. Local Intrinsic Dimensionality (LID): LID is estimated using the k nearest-neighbor distances around a reference point, with the estimate distinguished from the underlying distributional quantity.The nearest-neighbor distances are treated as extreme events from the lower tail of the induced distance distribution.
- 3.2. LID Estimation through Batch Sampling: The method estimates LID within a randomly selected batch because computing neighborhoods over the full dataset can be prohibitively expensive.The transformed reference point and batch neighbors are compared after applying the network's second-to-last-layer mapping g.
- 3.3. Dimensionality Dynamics: Figure 1 compares average LID scores with train and test accuracy across training for CNNs on CIFAR-10 and MNIST under clean and noisy labels.Scores are computed at layer 11 for CIFAR-10 and layer 4 for MNIST; clean-label results occupy the left subfigures and noisy-label results the right.
- 3.4. Two-Stage of Learning of DNNs on Noisy Labels: With noisy labels, training follows early dimensionality compression and later dimensionality expansion as the network overfits the noisy data.The same dimensionality trends were also observed for CNNs on SVHN and CIFAR-100.
- 3.4. Two-Stage of Learning of DNNs on Noisy Labels: Incorrectly labeled points migrate toward subspaces associated with their assigned labels, becoming local outliers that increase the surrounding dimensionality.Stopping earlier preserves neighborhoods closer to the original subspace and supports learning the true data distribution with improved test generalization.
4. Dimensionality-Driven Learning Strategy
D2L monitors LID during training to detect the shift from dimensionality compression to expansion, then adapts supervision to limit overfitting and test-accuracy loss.
- D2L targets overfitting and test-accuracy loss associated with dimensionality expansion under noisy labels.
- LID scores are estimated from nearest-neighbor distances within randomly selected batches and averaged across batches for each epoch.The method uses the second-to-last network layer; experiments later use m = 10 batches.
- The adaptive labels reduce the effect of noisy labels on learning the true data distribution by using a LID-based factor updated each epoch.The weighting λ = i/T reflects decreasing confidence in raw labels as LID begins to increase.
- D2L balances subspace dimensionality and prediction performance through an exponential interpolation between raw and predicted labels.As the ratio of current to historically lowest average LID exceeds 1, the decay factor favors predicted labels; LID estimation costs approximately 1–2% of training time.
- The turning point is flagged when the current LID exceeds the preceding window's mean by two standard deviations after an initialization window.The window reduces stochastic-optimization variation and allows network initialization.
- After detecting the turning point, D2L rolls the model back one epoch and begins interpolating raw and predicted label assignments.Before the turning point, α remains 1, so the loss is equivalent to cross-entropy; afterward, α decays according to dimensional expansion.
5. Experiments
Experiments evaluate D2L's effects on representation subspaces, hypothesis complexity, visual representations, hyperparameter sensitivity, and robustness across noisy-label benchmarks. D2L stabilizes accuracy, learns lower-dimensional subspaces and simpler hypotheses, and outperforms competitors across datasets and noise rates.
- Subspace Learning: D2L stabilizes test accuracy after around 60 epochs at 40% and 60% noise, while competitors experience substantial accuracy decreases.The result indicates that D2L limits overfitting to noisy labels.
- Subspace Learning: D2L learns representation subspaces with significantly lower dimensionality than other models, alongside better generalization and higher test accuracy.These observations support learning a low-dimensional representation of the true data distribution.
- Hypothesis Learning: D2L learns significantly simpler hypotheses than its competitors, and the paper associates this with better generalization and smoother decision boundaries.Hypothesis complexity is measured using Critical Sample Ratio (CSR).
- Representation Learning: t-SNE visualizations compare 500 randomly selected samples from each of two CIFAR-10 classes under cross-entropy and D2L with 60% incorrect labels.The experiment examines the quality of learned representations using two-dimensional embeddings.
- Parameter Sensitivity: D2L is relatively insensitive to neighborhood size k and batch count m for LID estimation on CIFAR-10, with similar behavior on the other three datasets.The sensitivity analysis varies both hyperparameters across noise rates.
- Robustness against Noisy Labels: D2L outperforms competitors consistently across MNIST, SVHN, CIFAR-10, and CIFAR-100 and across all tested noise rates.The reported means and standard deviations are computed over 5 repetitions, with noise rates from 0% to 60%.
- Robustness against Noisy Labels: As noise increases from 20% to 60%, D2L has the smallest accuracy drop and maintains relatively high classification accuracy even with 60% label noise.The performance gap between D2L and competitors also increases over this noise range.
6. Discussion and Conclusion
The paper explains noisy-label generalization through intrinsic dimensionality, identifying early compression followed by expansion during overfitting. It proposes D2L to suppress that expansion and reports strong classification performance under high label noise, while leaving several extensions open.
- Discussion and Conclusion: DNN learning shows early dimensional compression followed by dimensional expansion as training begins to overfit.This pattern is used to characterize generalization behavior for noisy labels.
- Discussion and Conclusion: D2L identifies the transition between compression and expansion using LID, then suppresses subsequent dimensional expansion to avoid overfitting.The strategy is described as a dimensionality-driven learning approach.
- Discussion and Conclusion: D2L delivers very strong classification performance across scenarios with high proportions of noisy labels.The conclusion summarizes the method's reported performance across varied settings.
- Discussion and Conclusion: Future work includes theoretical formulations, effects of data augmentation and regularization, and other noise types such as adversarial, corrupted-input, and asymmetric label noise.The paper identifies these as open research issues rather than resolved parts of the present study.