PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning with Neighbor Consistency for Noisy Labels

Ahmet Iscen, Jack Valmadre, Anurag Arnab, Cordelia Schmid

arXiv:2202.02200v2cs.CVcs.LG

TL;DR

To address label noise in costly-to-collect supervised datasets, the paper introduces Neighbor Consistency Regularization, which encourages examples with similar feature representations to have similar predictions. It reports competitive or state-of-the-art accuracies across synthetic and realistic noise scenarios.

  • Problem

    Large supervised datasets are expensive to obtain, while modern sources such as social media and webpages provide noisy labels that standard unsupervised and semi-supervised approaches cannot use.

    Source: §1 · p. 1

  • Method

    Neighbor Consistency Regularization adds a loss that encourages each example’s prediction to match a similarity-weighted combination of its neighbors’ predictions in feature space.

    Source: §1 · p. 1

  • Results

    NCR achieves competitive or state-of-the-art accuracies on synthetic CIFAR-10 and CIFAR-100 and realistic mini-WebVision, mini-ImageNet-Red, and Clothing1M noise scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    NCR is a comparatively simple, single-stage training procedure that adds an extra loss to the objective optimized in stochastic gradient descent.

    Source: §6 · p. 8

    NCR assumes access to an adequate feature representation; affinity-based label propagation requires a fixed feature space, potentially learned from noisy data.

    Source: §3 · p. 4, §6 · p. 8

Abstract

from arXiv · show

Recent advances in deep learning have relied on large, labelled datasets to train high-capacity models. However, collecting large datasets in a time- and cost-efficient manner often results in label noise. We present a method for learning from noisy labels that leverages similarities between training examples in feature space, encouraging the prediction of each example to be similar to its nearest neighbours. Compared to training algorithms that use multiple models or distinct stages, our approach takes the form of a simple, additional regularization term. It can be interpreted as an inductive version of the classical, transductive label propagation algorithm. We thoroughly evaluate our method on datasets evaluating both synthetic (CIFAR-10, CIFAR-100) and realistic (mini-WebVision, WebVision, Clothing1M, mini-ImageNet-Red) noise, and achieve competitive or state-of-the-art accuracies across all of them.

1. Introduction

Large supervised datasets are costly to obtain, and modern data sources often contain noisy labels. The paper introduces NCR, a neighbor-based consistency regularizer, and reports competitive or state-of-the-art accuracy across synthetic and realistic noise benchmarks.

1. Introduction

  • Deep learning’s demand for large supervised datasets is costly, while images from social media and queried webpages can carry noisy labels.Source: §1 · p. 1
  • Many recent noisy-label methods use model predictions to reject or modify examples, risking overfitting and often requiring multiple models or alternating updates.Source: §1 · p. 1
  • Unlike label-propagation methods applied to a fixed dataset, NCR performs propagation within mini-batches during SGD and produces a model for unseen examples.The resulting training procedure is single-stage.Source: §1 · p. 1
  • NCR adds a loss that encourages examples with similar feature representations to have similar predictions.It penalizes divergence from a similarity-weighted combination of neighbors’ predictions.Source: §1 · p. 1
  • NCR is complementary to mixup [49].Source: §1 · p. 2
  • NCR achieves competitive or state-of-the-art accuracies on CIFAR-10, CIFAR-100, mini-WebVision, mini-ImageNet-Red, and Clothing1M.The paper evaluates synthetic and realistic noise scenarios.Source: §1 · p. 2

2. Related work

Prior work addresses noisy labels through regularization, prediction-based relabeling, and neighbor-based filtering or propagation. NCR differs by using neighbor consistency to transfer supervision directly to mislabeled examples.

2. Related work

  • Mixup augmentation [49] interpolates examples and labels, and has been shown robust to synthetic label corruption on CIFAR-10.Regularization can limit a high-capacity model’s ability to fit arbitrary labels.Source: §2 · p. 2
  • Neighbor-based noisy-label methods often identify, reweight, or discard suspected errors; NCR instead uses neighbors to modify each example’s supervision.Examples include local-density methods [42] [12], graph filtering [43], k-NN disagreement [2], and MOIT [31].Source: §2 · p. 2
  • Iscen et al. [17] propagate labels from trusted examples, whereas NCR needs no clean set; method [47] filters a global graph once, unlike NCR’s training-time consistency loss.Source: §2 · p. 2
  • Consistency methods use predictions across transformed images or unlabeled examples, while neighbor consistency transfers supervision directly to mislabeled examples.Examples include ICT [39], Mixmatch [3], Unsupervised Data Augmentation [45], and noisy-label consistency [10].Source: §2 · p. 3

3. Preliminaries

The task is to learn accurate predictions for unknown true labels when training labels may be noisy. The section introduces label propagation’s graph objective and its transductive and fixed-feature limitations, motivating an inductive regularizer.

3. Preliminaries

  • The goal is high accuracy on unknown true labels despite an unknown number of noisy training labels.Source: §3 · p. 3
  • Label propagation spreads labels through a dataset graph and minimizes an objective combining label fitting with smoothness between nearby predictions.The affinity matrix represents example similarities; the fitting term uses one-hot labels and the smoothing term encourages nearby outputs to agree.Source: §3 · pp. 3–4
  • Label propagation is transductive: classifying a new test example requires constructing a new graph, making inference inefficient.Source: §3 · p. 4
  • Computing label-propagation affinities requires a fixed feature space, and alternating feature learning with propagation does not directly enforce smoothness during optimization.Source: §3 · p. 4
  • NCR generalizes label propagation by enforcing smoothness as a regularizer during optimization, avoiding an explicit graph and supporting inference on unseen examples.Source: §3 · p. 4

4. Method

Neighbor Consistency Regularization (NCR) adds a feature-space neighborhood consistency loss to supervised training to reduce the impact of noisy labels. It performs label propagation locally and online as features are learned, and relates to bootstrapping, label smoothing, and Mixup.

4.1. Neighbor Consistency Regularization

  • NCR adds a loss that encourages examples with similar feature representations to have similar predictions, regardless of their labels.The method assumes overfitting is less dramatic before the classifier; its neighborhood constraint acts on features as they are learned, without requiring a global graph or a feature representation learned from noisy examples.Source: §4.1 · pp. 4–5
  • NCR minimizes divergence between an example’s prediction and a similarity-weighted combination of its nearest neighbors’ predictions, then combines this loss with supervised classification.The similarity is cosine similarity; the objective uses KL divergence, temperature T = 2, and a hyperparameter α to control the two loss terms.Source: §4.1 · p. 4
  • NCR adds O(m2(d + c)) computation for similarity and weighted predictions, which the authors describe as relatively fast for moderate batch sizes on modern GPUs.Here m is batch size, d feature dimension, and c number of classes.Source: §4.1 · p. 5

4.2. Relation to other methods

  • NCR bootstraps from neighborhood structure rather than model predictions, modifies label smoothing by replacing the uniform distribution with a neighbor-defined distribution, and targets existing examples unlike Mixup.NCR avoids dependence on classifier parameters W; it reduces to label smoothing when neighbors are random or temperature T is high.Source: §4.2 · p. 5

5. Experiments

Experiments evaluate NCR on synthetic and realistic label noise, including CIFAR, mini-ImageNet, WebVision, and Clothing1M. NCR achieves competitive or state-of-the-art results, with gains over baselines and improved feature separation in several settings.

5.1. Experimental setup

  • The experiments cover uniform synthetic noise on CIFAR and mini-ImageNet-Blue, more realistic mini-ImageNet-Red noise, and organic noise on WebVision and Clothing1M, with clean evaluation data.Source: §5.1 · p. 5

5.2. Ablation study

  • At nonzero noise, α=0.9 and smaller k (e.g. 10) improve accuracy; e=0 performs better at 40% and 80%, indicating early NCR helps prevent memorization.Source: §5.2 · pp. 5–6, Figure 2 · p. 6

5.3. Baseline comparison

  • Up to 4.9% accuracy improvement over the standard baseline is reported across noise ratios; mixup further improves results in some scenarios, and gains are larger when labels are noisy.NCR also improves accuracy at 0% noise, suggesting a general regularization effect.Source: §5.3 · p. 6, Table 1 · p. 7
  • Against an oracle trained only on known clean examples, NCR performs better on Red except at 40% noise, but worse on Blue, where noisy examples are often dissimilar to clean ones.Source: §5.3 · p. 6, Table 1 · p. 7
  • After training, the baseline assigns high confidence to clean and noisy labels, whereas NCR more often lowers confidence on mislabeled examples; at 80% Blue noise, underfitting leaves both near zero.Source: §5.3 · p. 6, Figure 3 · p. 7

5.5. State-of-the-art comparison

  • NCR outperforms GJS’s augmentation-only consistency approach, supporting an additional benefit from consistency across neighboring examples.Source: §5.5 · p. 7

5.4. Effect on feature embeddings

  • NCR’s gradients through feature similarities improve class separation; on Blue, separation holds up to 40% noise, while on Red, same-class mislabeled examples remain unseparated.Source: §5.4 · p. 6, Figure 4 · p. 8

5.5. State-of-the-art comparison

  • On mini-WebVision, NCR improves the best previous result by 1.2%; it is competitive on WebVision and 0.2% below state-of-the-art on Clothing1M, while leading or placing second on CIFAR.Source: §5.5 · p. 7, Table 2 · p. 8, Table 3 · p. 8

6. Conclusion

NCR is presented as an effective, simple strategy for learning with label noise, with state-of-the-art results on several synthetic and realistic noise scenarios. The conclusion notes a feature-representation assumption and broader concerns about bias and privacy when learning from scraped data.

6. Conclusion

  • NCR achieved state-of-the-art results on CIFAR-10, CIFAR-100, and mini-WebVision noise scenarios.The method adds an extra loss optimized in stochastic gradient descent rather than requiring a multi-stage procedure.Source: §6 · p. 8
  • NCR assumes access to an adequate feature representation, addressed in practice by training for e epochs before applying its loss.Removing this additional training hyperparameter is identified as future work.Source: §6 · p. 8
  • Learning from automatically scraped web data may amplify existing data biases and may use images without their owners’ consent.The passage identifies possible privacy infringement as a concern.Source: §6 · p. 8
  • Future directions include coupling NCR with out-of-distribution example rejection and applying it as a regularizer in semi-supervised learning.Source: §6 · p. 8

Supplementary Material

The supplementary material includes a table of network hyperparameters used in the experiments.

Supplementary Material

  • Table 4 lists the network hyperparameters used in the experiments.Source: Table 4 · p. 11

A. Training details

The training details specify model architectures and dataset-specific protocols, augmentation, and NCR hyperparameter selection. They also describe a held-out validation set for mini-ImageNet and a batch-size study on WebVision.

A. Training details

  • The experiments use ResNet-18 and ResNet-50; Clothing1M fine-tunes a pre-trained ResNet-50 for 80 epochs of 1,000 mini-batches.CIFAR follows the ELR protocol, while Clothing1M follows the protocol in [23].Source: A. Training details · p. 11
  • Training uses random crops and 224 × 224 images, except CIFAR uses 32×32 images with reduced strides.A single Nvidia V100 GPU is used per model.Source: A. Training details · p. 11
  • NCR hyperparameters α, k, and e are swept using validation accuracy, separately for each noise ratio.The listed settings are in Table 5.Source: A. Training details · p. 11, Table 5 · p. 11
  • Table 6 examines the effect of batch size on NCR for WebVision’s 1,000 classes.Source: Table 6 · p. 11
  • Mini-ImageNet hyperparameters are selected using a held-out set of clean Red examples absent from the noisy splits, then reused for Blue.Source: A. Training details · p. 11

B. Dataset details

The dataset details span synthetic and realistic noise settings, including web-sourced images and labels. The datasets differ in how noise is introduced and in their size and class coverage.

B. Dataset details

  • Mini-ImageNet-Red has 50,000 training and 5,000 validation examples, with realistic noise from text-to-image and image-to-image search outside the training-class vocabulary.Some clean examples are replaced by retrieved noisy images according to the noise ratio.Source: B. Dataset details · p. 11
  • Mini-ImageNet-Blue has 60,000 training examples and synthetic noise that independently and uniformly changes labels to other classes in the training set.Source: B. Dataset details · p. 11
  • WebVision contains 2.4 million images across 1,000 classes collected through Google and Flickr search, with imbalanced class sizes.Mini-WebVision uses the first 50 Google-image classes, comprising 65,944 training images and 2,500 validation images.Source: B. Dataset details · p. 11
  • Clothing1M contains one million web-collected images in 14 categories, with noisy labels assigned from surrounding text.The experiments do not use its clean human-verified training subset.Source: B. Dataset details · p. 11

C. Effect of batch size and number of labels

The section examines how batch size affects accuracy on WebVision, a 2.4M-image dataset with 1K classes. Accuracy rises with batch size and plateaus once batch size exceeds the number of classes.

C. Effect of batch size and number of labels

Related papers

Loading 2202.02200v2…