Source-linked AI summary
Co-learning: Learning from Noisy Labels with Self-supervision
Cheng Tan, Jun Xia, Lirong Wu, Stan Z. Li
TL;DR
Noisy labels can cause deep networks to overfit misleading information and degrade generalization. Co-learning combines supervised and self-supervised views through a shared encoder, and experiments on synthetic and real-world noisy datasets show robust performance against competing methods.
Problem
Noisy labels from inexpensive annotation alternatives can cause neural networks to overfit and substantially degrade generalization performance.
Method
Co-learning uses a shared feature encoder with supervised and self-supervised heads, combining label-dependent and feature-dependent information with intrinsic and structural similarity constraints.
Results
Co-learning outperforms other co-training-based and state-of-the-art methods across synthetic and real-world noisy datasets, remaining robust under high-level label noise.
Takeaways & Limitations
Self-supervised learning can assist robust learning with noisy labels, while structural similarity regularizes supervised training against noisy-label overfitting.
Takeaways & Limitations
Prior small-loss methods discard difficult or noisy samples and require noisy-rate estimates and carefully chosen hyperparameters, limiting their real-world applicability.
Abstract
from arXiv · showhide
Noisy labels, resulting from mistakes in manual labeling or webly data collecting for supervised learning, can cause neural networks to overfit the misleading information and degrade the generalization performance. Self-supervised learning works in the absence of labels and thus eliminates the negative impact of noisy labels. Motivated by co-training with both supervised learning view and self-supervised learning view, we propose a simple yet effective method called Co-learning for learning with noisy labels. Co-learning performs supervised learning and self-supervised learning in a cooperative way. The constraints of intrinsic similarity with the self-supervised module and the structural similarity with the noisily-supervised module are imposed on a shared common feature encoder to regularize the network to maximize the agreement between the two constraints. Co-learning is compared with peer methods on corrupted data from benchmark datasets fairly, and extensive results are provided which demonstrate that Co-learning is superior to many state-of-the-art approaches.
1 INTRODUCTION
Noisy labels from inexpensive annotation sources can cause neural networks to overfit and generalize poorly. Co-learning addresses this by cooperatively combining supervised and self-supervised views, and experiments report superiority to state-of-the-art methods on simulated and real-world noisy datasets.
- Motivation: Inexpensive annotation sources such as web crawling and crowd-sourcing can produce noisy labels that degrade generalization.Deep neural networks can overfit these misleading labels.
- Motivation: Prior approaches use supervised and unsupervised views to exploit label-dependent and label-independent information when learning with noisy labels.The supplied related-work passage describes these as two complementary views.
- Co-learning: Co-learning uses a shared feature encoder with self-supervised and noisily-supervised heads that constrain each other.The projection head learns intrinsic similarity, while structural similarity regularizes the classifier against noisy-label bias.
- Co-learning: Unlike several co-training-like methods, Co-learning does not require noise rates, data distributions, or additional clean samples.This avoids the associated hyperparameters.
- Results: Experiments on CIFAR-10, CIFAR-100, Animal-10N, and Food-101N report that Co-learning is superior to state-of-the-art methods and robust to high-level label noise.The evaluation includes simulated and real-world noisy datasets.
2 RELATED WORK
Related work addresses noisy labels through transition-matrix estimation, sample selection, semi-supervised learning, and self-supervised representation learning. Co-learning differs by aligning feature representations from two heads sharing one encoder rather than relying primarily on two classifiers and small-loss selection.
- Learning with noisy labels: Transition-matrix methods correct predictions using estimated label-transition structure, but estimating that matrix can be challenging or infeasible in real-world settings.Forward correction multiplies softmax outputs by the estimated matrix, while Gold loss correction uses trusted labels.
- Learning with noisy labels: Sample-selection methods identify presumed clean instances using teacher networks, peer networks, disagreement, or small-loss criteria.These approaches differ in how selected samples guide subsequent training.
- Learning with noisy labels: Semi-supervised approaches either divide noisy data into labeled and unlabeled sets or apply unsupervised learning to noisy training data.Both directions use unsupervised information to assist supervised learning.
- Self-supervised learning: Self-supervised learning derives pretext-task labels from data itself and learns representations through tasks including contrastive learning.Contrastive learning groups similar samples closer and diverse samples farther apart using intrinsic similarity.
- Relations to other approaches: Small-loss-based methods discard remaining samples and may require noisy rates and carefully chosen hyperparameters, creating difficulties in real-world applications.Their selection strategy does not use all available data.
- Relations to other approaches: Co-learning maximizes agreement in the shared feature encoder, combining feature-dependent and label-dependent information rather than only comparing predictions.The shared encoder also reduces computation cost.
3 METHODS
Co-learning jointly trains supervised classification and self-supervised representation learning through a shared encoder, using intrinsic and structural similarity constraints to regularize learning with noisy labels.
- Co-learning: Co-learning combines a supervised classifier with an auxiliary contrastive self-supervised task that jointly learn the encoder and classifier.The framework uses a shared feature encoder with separate classifier and projection heads.
- Loss design: The complete objective linearly combines supervised loss, intrinsic similarity loss, and structural similarity loss.The supervised component uses cross-entropy on corrupted labels, while the other terms impose feature-dependent constraints.
- Optimization: MixUp augmentation slows feature-encoder convergence so supervised learning does not quickly overfit noisy labels.New samples are created by linearly interpolating examples randomly selected from the same mini-batch.
- Intrinsic similarity: Contrastive prediction treats transformed views of the same example as positive pairs and views from different examples as negative pairs.The projection head learns intrinsic similarity from multiple transformed versions of each training example.
- Structural similarity: Structural similarity constrains pairwise relationships between classifier outputs and corresponding projection-head outputs because both heads share the encoder.The method converts distances into similarity metrics and defines their agreement with a KL-divergence loss.
4 EXPERIMENTS
Experiments on synthetic and real-world noisy-label datasets compare Co-learning with established baselines. Co-learning performs especially well under high noise and shows less overfitting, with ablations attributing robustness to balanced supervised and self-supervised training.
- Experimental settings: Experiments use CIFAR-10, CIFAR-100, Animal-10N, and Food-101N, covering synthetic, human-labeled, and webly noisy data.CIFAR datasets are manually corrupted, while Animal-10N and Food-101N provide real-world noisy-label settings.
- Experimental settings: Co-learning is compared with standard cross-entropy, Decoupling, Co-teaching, Co-teaching+, JoCoR, and APL under unified experimental settings.The comparisons use common implementations and benchmark protocols, with specified network and optimizer settings.
- Comparison with the State-of-the-Arts: Co-learning performs best in all six CIFAR-10 cases, while standard cross-entropy increasingly overfits as noise rates rise.The CIFAR-10 results are reported in Figure 5 and Table 2.
- Comparison with the State-of-the-Arts: Under CIFAR-100 Symmetry-50% and Symmetry-80% noise, Co-learning works significantly better and produces more stable learning curves with less overfitting.The paper reports the same overall pattern as CIFAR-10, with stronger advantages at high noise rates.
- Ablation study: Ablations show that self-supervision alone helps only slightly, whereas MixUp and structural similarity together support balanced training and reduce overfitting under higher noise.Removing structural similarity causes overfitting under Symmetry-50%, while the full method maintains a more noise-tolerant procedure.
5 CONCLUSION
Co-learning combines supervised and self-supervised information through two heads sharing a feature encoder, using intrinsic and structural similarity to regularize learning with noisy labels.
- Co-learning uses supervised learning for label-dependent information and self-supervised learning for feature-dependent information.
- Two exclusive heads share a feature encoder instead of training two separate classifiers.
- The model is regularized with both intrinsic similarity and structural similarity.
- Extensive synthetic and real-world experiments show Co-learning outperforms other co-training-based methods.
A.1 Dataset
The appendix describes the datasets and transformation settings used in the paper, including strong and weak augmentation pipelines.
- Table 6 summarizes the datasets used in the paper.
- The strong transformation T combines geometric and color transformations.
- T uses RandomResizedCrop, RandomHorizontalFlip, ColorJitter, and RandomGrayscale with specified scales and probabilities.
- The weak transformation T′ contains only RandomResizedCrop and RandomHorizontalFlip as geometric transformations.
A.3 Training details
The training details specify a unified comparison framework using ResNet-18, Adam, fixed batch size and epoch settings, and learning-rate decay for CIFAR datasets.
- Co-training-based noisy-label methods are implemented in a unified framework for fair comparisons.
- ResNet-18 is used for CIFAR-10 and CIFAR-100 experiments.
- Training uses Adam with momentum=0.9, an initial learning rate of 0.001, and batch size 128.
- Models run for 200 epochs, with the learning rate linearly decayed to zero from epochs 80 to 200.
B.1 Comparison with loss-design methods
Co-learning is compared with loss-design methods GCE and SCE, showing faster convergence and the highest accuracy across four conditions.
- Co-learning converges faster than GCE and SCE under all four tested conditions.
- Co-learning achieves the highest accuracy across the four conditions, while GCE and SCE have smoother convergence curves.
B.2 MixUp v.s. Weighted supervised loss
Co-learning uses MixUp to slow overfitting on noisy labels, while weighted supervised loss can achieve similar results under suitable weights. MixUp is preferred because it is less sensitive to hyperparameter choice.
- MixUp slows overfitting on noisy labels during Co-learning training.
- Weights of 0.001 and 0.01 produce smooth convergence curves for the supervised loss.
- Larger supervised-loss weights fail, whereas small weights achieve results similar to MixUp.
- MixUp is preferred because its performance is more insensitive to hyperparameters.