Source-linked AI summary
Selective-Supervised Contrastive Learning with Noisy Labels
Shikun Li, Xiaobo Xia, Shiming Ge, Tongliang Liu
TL;DR
Expensive clean labels support strong representations, whereas noisy labels corrupt pair construction and impair generalization. Sel-CL selects confident pairs without a noise-rate prior, and extensive experiments report state-of-the-art performance on multiple noisy datasets.
Problem
Expensive high-quality labels motivate robust representation learning because noisy labels corrupt supervised contrastive pairs and representations.
Method
Sel-CL first builds pairs from confident examples, then uses their representation-similarity distribution to select additional confident pairs without knowing noise rates.
Results
Experiments on multiple noisy datasets demonstrate state-of-the-art performance and robust representations with Sel-CL.
Takeaways & Limitations
Sel-CL provides a pair-selection approach for using supervised contrastive learning with noisy labels.
Takeaways & Limitations
Performance relies on adequate augmentation and many negative samples, while KNN selection increases computational consumption.
Abstract
from arXiv · showhide
Deep networks have strong capacities of embedding data into latent representations and finishing following tasks. However, the capacities largely come from high-quality annotated labels, which are expensive to collect. Noisy labels are more affordable, but result in corrupted representations, leading to poor generalization performance. To learn robust representations and handle noisy labels, we propose selective-supervised contrastive learning (Sel-CL) in this paper. Specifically, Sel-CL extend supervised contrastive learning (Sup-CL), which is powerful in representation learning, but is degraded when there are noisy labels. Sel-CL tackles the direct cause of the problem of Sup-CL. That is, as Sup-CL works in a \textit{pair-wise} manner, noisy pairs built by noisy labels mislead representation learning. To alleviate the issue, we select confident pairs out of noisy ones for Sup-CL without knowing noise rates. In the selection process, by measuring the agreement between learned representations and given labels, we first identify confident examples that are exploited to build confident pairs. Then, the representation similarity distribution in the built confident pairs is exploited to identify more confident pairs out of noisy pairs. All obtained confident pairs are finally used for Sup-CL to enhance representations. Experiments on multiple noisy datasets demonstrate the robustness of the learned representations by our method, following the state-of-the-art performance. Source codes are available at https://github.com/ShikunLi/Sel-CL
1. Introduction
High-quality labels enable strong deep-network representations but are expensive, while noisy labels corrupt representations and damage generalization. Sel-CL addresses this by selecting confident pairs for supervised contrastive learning without requiring known noise rates.
- High-quality annotated labels support strong latent representations, but large-scale collection is extremely expensive.
- Noisy labels provide incorrect representation-learning signals, producing inaccurate decisions and poor generalization.
- Sup-CL can learn better representations than Uns-CL, but noisy supervised information creates misleading training pairs.
- Sel-CL selects confident pairs from noisy pairs and uses them for supervised contrastive learning to obtain robust representations.
- Sel-CL identifies confident examples first, then uses their representation-similarity distribution to select additional pairs without estimating noise rates.
- Experiments on synthetic and real-world noisy datasets report better performance than state-of-the-art methods, with ablation studies and discussions.
2. Related Works
Related work covers broad noisy-label strategies and contrastive representation learning. Prior Sup-CL-based methods combine supervised contrastive pretraining with regularization, pseudo-labeling, or reliable-data fine-tuning to address noisy labels.
- Noisy-label methods include transition-matrix estimation, example reweighting, confident-example selection, robust losses, regularization, pseudo-labeling, and combined techniques.
- Unsupervised contrastive learning maximizes similarity for positive same-instance views and minimizes similarity for negative pairs.
- Sup-CL-based methods use supervised information to learn latent representations and handle noisy labels through one- or two-stage designs.
- Two-stage approaches pre-train with Sup-CL, reduce noisy-label effects using general-purpose techniques, and fine-tune on reliable data.
3. Selective-Supervised Contrastive Learning
Sel-CL progressively identifies confident examples and pairs from noisy labels, then uses those pairs for supervised contrastive learning and robust representation learning.
- Selecting Confident Examples: Sel-CL first warms up representations with unsupervised contrastive learning before identifying confident examples.The warm-up obtains low-dimensional representations used later for confidence detection.
- Selecting Confident Examples: Confident examples are selected by comparing noisy labels with pseudo-labels formed from the dominant labels among top-K representation neighbors.The experiments use K=250 neighbors, and pseudo-labels approximate clean class posterior probabilities.
- Selecting Confident Examples: Class-specific thresholds use per-class agreement fractiles to produce a class-balanced confident-example set T.The resulting set is described as less noisy and more reliable than the original noisy dataset.
- Selecting Confident Pairs: Confident examples with matching noisy labels form reliable pairs G′, while additional positive pairs G′′ are selected using a dynamic similarity threshold.The threshold is based on a fractile of representation similarities in G′, avoiding noise-rate estimation; the final set is G = G′ ∪ G′′.
- Representation Learning with Selected Pairs: At each epoch, selected confident pairs are used for supervised contrastive learning, while other examples use unsupervised contrastive learning.Mixup, classification learning with confident examples, and a similarity-label objective are also included in the representation-learning framework.
- Classification Fine-tuning: The method alternates pair selection and representation learning, creating a positive cycle in which improved representations support better pair selection.Sel-CL+ retains the pretrained encoder and adds a new classifier head for fine-tuning.
4. Experiments
Experiments evaluate Sel-CL on simulated and real-world noisy datasets, testing representation quality, classification accuracy, selection behavior, component choices, and robustness across training settings.
- Datasets and evaluation: Sel-CL is evaluated on CIFAR-10, CIFAR-100, and WebVision-50 using simulated or real-world noisy labels.The experiments include symmetric and asymmetric noise on CIFAR datasets and realistic web-label noise on WebVision-50.
- Representation learning evaluations: 55.58% versus 56.23%: under 80% symmetric noise, MOIT’s weighted KNN accuracy falls below Uns-CL.Sel-CL consistently outperforms the baselines across evaluated noisy cases.
- Representation learning evaluations: Sel-CL progressively improves selection and representations through a positive cycle between confident examples, confident pairs, and representation quality.The process is illustrated using CIFAR-100 with 20% symmetric noise.
- Results on simulated noisy datasets: Sel-CL+ achieves the best performance over baselines for asymmetric noise and competitive performance for symmetric noise on CIFAR-10 and CIFAR-100.The comparison reports test accuracy across multiple noise levels and datasets.
- Results on the real-world noisy dataset: Sel-CL+ achieves the best top-1 and top-5 accuracy on both WebVision and ImageNet validation sets among the compared state-of-the-art methods.The model is trained on WebVision-50.
5. Limitations
The approach has computational and hardware demands from its contrastive-learning design and KNN-based selection.
- Performance relies on adequate data augmentation and large numbers of negative samples.Larger batch sizes or a memory bank increase storage demands on computing devices.
- The KNN algorithm increases computational consumption.Faster KNN algorithms are used to alleviate this issue for large-scale datasets.
6. Conclusion
The paper proposes Sel-CL for robust pre-trained representations under noisy labels by selecting confident pairs without a noise-rate prior. Extensive experiments on multiple noisy datasets report state-of-the-art performance, while future work includes extending the method to object detection and text matching.
- Sel-CL handles noisy training labels by learning robust pre-trained representations.It uses contrastive learning's pair-wise characteristic to enhance network robustness.
- Confident pairs are selected from noisy pairs for supervised contrastive learning without a noise-rate prior.
- Extensive experiments on multiple noisy datasets demonstrate state-of-the-art performance for Sel-CL.
- Future work will consider extending Sel-CL to object detection and text matching.
A. Hyperparameter Sensitivity Analysis
The sensitivity analyses examine the similarity-loss weight and noise-detection fractiles. The reported results indicate robustness to the tested choices of these hyperparameters.
- Analysis of λs: λs balances the added similarity loss LSIM and is set to 0.01 in all experiments.The approach is reported to be robust to selection of λs in Tab. 1.
- Analysis of α and β: Noise-detection fractiles α and β determine dynamic thresholds for selecting confident examples and pairs.For CIFAR-10 with simulated label noise, the settings are α = 50% and β = 25%.
- Analysis of α and β: The approach is reported to be robust to choices of α and β.The sensitivity analysis uses CIFAR-10 datasets and reports results in Tables 2 and 3.
B. Implementation Details For Baselines
Baseline implementations use published code or recommended and tuned hyperparameters, with training schedules and settings varying across methods and datasets.
- Baselines are obtained from related work or published codes using recommended or well-tuned hyperparameters.
- Uns-CL is trained for 1000 epochs on CIFAR-10/100 and WebVison-50 with specified augmentation, temperature, batch-size, and learning-rate settings.
- GCE uses dataset- and noise-condition-specific q values across CIFAR-10 and CIFAR-100.The schedules include 120 epochs for CIFAR-10 and 150 epochs for CIFAR-100.
- GCE with Uns-CL initialization uses condition-specific q values and 120- or 150-epoch training schedules.
- ELR and ELR with Uns-CL initialization use condition-specific λ and β settings with 250 training epochs.
- MOIT+ uses recommended hyperparameters for pre-training and fine-tuning, with one CIFAR-100 exception where semi-supervised training is omitted.
- DivideMix uses the C2D implementation with its recommended hyperparameter setting.
- ELR+ with Uns-CL initialization is trained for 250 epochs using ELR's λ and β settings.
C. Relations with MOIT
Sel-CL differs from MOIT by selecting confident pairs rather than only confident examples, and by applying contrastive learning directly to those selected pairs. This design is presented as improving robustness more directly than MOIT’s regularization-based selection.
- Selection targets: Sel-CL selects confident pairs, whereas MOIT performs point-wise selection of confident examples.Sel-CL can use pairs with either correct or incorrect class labels.
- Selection roles: MOIT uses selected examples for semi-supervised classification during pre-training, while contrastive learning still operates on all noisy pairs.MOIT’s selection therefore serves a regularization role in representation learning.
- Selection roles: Sel-CL performs supervised contrastive learning only on selected pairs, improving representation robustness more directly and effectively than MOIT.The paper reports experiments verifying Sel-CL’s advantages over MOIT.
D. Visualization Results
The visualization compares representations learned by Cross-Entropy and Sel-CL using t-SNE. Sel-CL obtains more robust representations and better combats noisy labels in this comparison.
- t-SNE visualization compares representations learned by Cross-Entropy and Sel-CL.
- Sel-CL obtains more robust representations than Cross-Entropy in the visualization.
- The comparison indicates that Sel-CL better combats noisy labels.
E. Pseudo-code of the Proposed Sel-CL
Algorithm 1 initializes Sel-CL’s training settings and iteratively trains an encoder, beginning with warm-up learning before selecting confident examples and pairs. It then applies supervised contrastive and Mixup classification learning until returning the learned encoder.
- Algorithm specification: Algorithm 1 defines Sel-CL’s inputs, including noise-detection, Mixup, temperature, loss-weight, warm-up, and maximum-epoch parameters, and outputs the learned encoder f.The listed parameters include α, β, αm, τ, λc, λs, Twarm, and Tmax.
- Warm-up training: During warm-up epochs, Sel-CL trains the deep encoder with either unsupervised or supervised contrastive learning.This stage runs while t ≤ Twarm.
- Selective training: After warm-up, Sel-CL selects confident examples from representation-label agreement, selects confident pairs using similarity distributions, and trains on them with contrastive learning and Mixup classification.The loop continues through Tmax epochs before returning f.