PaperlayerView exact revision on arXiv

Source-linked AI summary

FINE Samples for Learning with Noisy Labels

Taehyeon Kim, Jongwoo Ko, Sangwook Cho, Jinhwan Choi, Se-Young Yun

arXiv:2102.11628v3cs.LG

TL;DR

Noisy labels can hinder DNN generalization, and existing noise detectors may rely on classifier outputs biased by corrupted labels. FINE detects noise through latent-representation alignment using Gram-matrix eigenvectors, and its applications consistently improve generalization across three noisy-label learning approaches.

  • Problem

    Noisy labels can hinder DNN generalization, while common loss- or gradient-based detectors may be biased by a corrupted classifier.

    Source: §1 · pp. 1–2

  • Method

    FINE uses eigenvectors of latent-representation Gram matrices to score alignment and detect clean instances without estimating noise rates.

    Source: §1 · p. 2

  • Results

    FINE consistently improves generalization in sample selection, semi-supervised learning, and collaboration with noise-robust losses.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors see FINE as a route to explainable noisy-label detection that could reduce labor-intensive label-quality checking.

    Source: §5 · p. 10

    The theoretical analysis assumes separate Gaussian clean and noisy clusters with features aligned to respective unit vectors.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Modern deep neural networks (DNNs) become frail when the datasets contain noisy (incorrect) class labels. Robust techniques in the presence of noisy labels can be categorized into two folds: developing noise-robust functions or using noise-cleansing methods by detecting the noisy data. Recently, noise-cleansing methods have been considered as the most competitive noisy-label learning algorithms. Despite their success, their noisy label detectors are often based on heuristics more than a theory, requiring a robust classifier to predict the noisy data with loss values. In this paper, we propose a novel detector for filtering label noise. Unlike most existing methods, we focus on each data's latent representation dynamics and measure the alignment between the latent distribution and each representation using the eigendecomposition of the data gram matrix. Our framework, coined as filtering noisy instances via their eigenvectors (FINE), provides a robust detector with derivative-free simple methods having theoretical guarantees. Under our framework, we propose three applications of the FINE: sample-selection approach, semi-supervised learning approach, and collaboration with noise-robust loss functions. Experimental results show that the proposed methods consistently outperform corresponding baselines for all three applications on various benchmark datasets.

1 Introduction

Label noise can undermine DNN generalization, while common detectors rely on classifier outputs that may be biased by corrupted labels. FINE instead uses latent-representation alignment and is applied to three noisy-label learning approaches.

1 Introduction

  • Existing loss- or gradient-based detectors may be biased by a corrupted classifier, while nearest-neighbor filtering can be sensitive to clustering hyperparameters under severe noise.The paper motivates FINE as a detector that avoids reliance on the linear classifier's posterior information.Source: §1 · p. 2
  • FINE is presented with theoretical guarantees and does not require estimating noise rates.The paper also reports theoretical and experimental support for the principal components used by the detector.Source: §1 · p. 2
  • FINE detects clean instances using eigen decomposition of latent-representation Gram matrices to define a boundary in high-order topological space.It measures each representation's alignment toward a class-representative vector derived from the Gram matrix.Source: §1 · p. 2, Figure 1 · p. 2
  • FINE-based sample selection achieves consistently superior detection quality and higher test accuracy than Co-teaching, TopoFilter, and CRUST alternatives.The comparison is reported for the sample-selection approach.Source: §1 · p. 2
  • FINE consistently improves generalization across sample selection, semi-supervised learning, and collaboration with noise-robust losses.The paper reports these three applications in experiments.Source: §1 · p. 3

2 Related Works

Related work spans noise-cleansing detectors, noise-robust training methods, and dataset-resampling approaches. The supplied FINE sample-selection procedure builds class-specific representation statistics and selects examples by alignment scores.

Noise-Cleansing-based Approaches

  • Noise-cleansing methods include co-teaching, disagreement-based selection, training-filtering-training, nearest-neighbor filtering, low-rank Jacobian subset selection, and SSL-based filtering.The SSL approach separates a labeled clean set from an unlabeled noisy set and uses MixMatch.Source: §2 · p. 3

Noise-Robust Models

  • Robust-training work includes MAE, GCE, RCE, SCE, early-learning regularization, and prediction-level regularization, with reported performance limitations as noise rates increase.SCE is described as a weighted sum of CE and RCE.Source: §2 · p. 3

Dataset Resampling

  • Dataset resampling extracts a helpful subset from the original distribution, with prior examples addressing class imbalance, representation bias, and spurious artifacts.The passage also connects label-noise detection to anomaly and out-of-distribution detection, while noting this paper focuses on noisy image classification.Source: §2 · pp. 3–4

FINE Sample Selection

  • FINE sample selection accumulates class-specific Gram matrices, extracts the leading eigenvector, scores representations by squared alignment, and uses GMM clean probabilities to collect examples.The algorithm takes noisy data, a feature extractor, class count, and clean-probability threshold, and outputs collected clean data.Source: §2 · p. 4

3 Method

FINE detects clean instances by measuring representation alignment with the leading eigenvector of each class’s feature Gram matrix. The section motivates this detector with a perturbation analysis and evaluates its detection behavior, scalability, and use in sample selection.

3 Method

  • FINE scores each representation by squared alignment to its class’s leading Gram-matrix eigenvector, then treats the higher-alignment cluster as clean.Alignment clusterability is assessed by fitting a Gaussian mixture model to the alignment distribution.Source: §3 · p. 4

3.1 Alignment Analysis for Noisy Label Detector

  • Under two-Gaussian clusters with mean directions v and w, the perturbation bound tightens as clean data increase, the noise ratio falls, and the directions approach orthogonality.The derived lower bounds for precision and recall also increase as v and w become more orthogonal.Source: §3.1 · pp. 5–6
  • FINE’s leading eigenvector nearly maximizes Eq. (2), and its perturbation remains small even at severe noise.Figure 3 visualizes Eq. (2), perturbation, and principal-component similarity under the stated noise settings.Source: Figure 3 · p. 6, §3.1 · p. 6
  • Using just 1% of the data yields a 0.99 cosine similarity to FINE’s original eigenvector, with little computation time.This approximates the eigenvector without using the whole dataset for eigendecomposition.Source: §3.1 · p. 6
  • Across symmetric and asymmetric noise, FINE maintains higher sample-selection F-scores than Co-teaching and TopoFilter on CIFAR-10/100.Figure 4 tracks these comparisons during training.Source: §3.1 · pp. 6–7, Figure 4 · p. 7
  • FINE consistently outperforms the Mahalanobis-distance estimator in both precision and recall, yielding a better F-score.The comparator selects clean samples using pre-logit Mahalanobis distance with a minimum covariance determinant estimator.Source: §3.1 · p. 7

4 Experiment

Section 4 evaluates FINE across sample selection, semi-supervised learning, and noise-robust-loss collaborations, including benchmark and real-world settings. The reported results show gains over competing sample-selection methods, improved performance over DivideMix, and useful performance on Clothing1M.

4.1 Experimental Settings

  • The experiments use symmetric and asymmetric synthetic label noise on CIFAR-10 and CIFAR-100, plus inherently noisy Clothing1M data.For Clothing1M, training uses a pseudo-balanced 120,000-image subset and evaluation uses 10,000 verified-clean images.Source: §4.1 · p. 7

4.2.1 Sample Selection-Based Approaches

  • After warmup, FINE selects clean samples each epoch using class-wise representation Gram-matrix eigenvectors, and the selected samples train the networks.The study also substitutes FINE for the sample-selection state in the Co-teaching family.Source: §4.2.1 · pp. 7–8
  • FINE consistently outperforms competing sample-selection methods across noise rates, datasets, and noise distributions.The authors report that filtering clean instances without losing essential information supports training a robust network.Source: Table 1 · p. 8, §4.2.1 · p. 8

4.2.2 SSL-Based Approaches

  • F-DivideMix achieves higher performance than DivideMix and comparable performance to DST and LongReMix; clean and noisy samples are well classified under extreme noise.FINE replaces DivideMix’s loss-based filtering method.Source: §4.2.2 · p. 8, Figure 6 · p. 8, Table 2 · p. 9

4.2.3 Collaboration with Noise-Robust Loss Functions

  • FINE facilitates generalization when combined with GCE, SCE, and ELR, particularly under severe noise rates.Figure 7 reports test accuracies for CIFAR-10 and CIFAR-100 across noisy types and fractions, with and without FINE.Source: §4.2.3 · p. 9, Figure 7 · p. 9

4.3 Experiments on Real-World Dataset

  • FINE and F-DivideMix perform fairly well on Clothing1M compared with other approaches under the same reproduced settings.Table 3 reports test accuracy on Clothing1M.Source: Table 3 · p. 9, §4.3 · p. 9

5 Conclusion

The conclusion presents FINE as a representation-based label-noise detector with theoretical and experimental support across three learning applications. The appendix describes a concentration-inequality proof strategy and introduces spectral-norm and SVD preliminaries.

5 Conclusion

  • FINE uses principal components of latent representations to detect label noise and reports strong results across standard benchmarks and a real-world dataset.The conclusion describes applications to sample selection, semi-supervised learning, and collaboration with noise-robust loss functions.Source: §5 · p. 9

A Theoretical Guarantees for FINE Algorithm

  • The appendix derives Theorem 1 and lower bounds for precision and recall using concentration inequalities.The supplied passage identifies the proof approach but does not state the bounds themselves.Source: A Theoretical Guarantees for FINE… · p. 14

A.1 Preliminaries

  • The preliminaries introduce spectral norm and SVD notation, including orthogonal U and V matrices and ordered singular values.The singular values are described as the diagonal entries of a diagonal matrix.Source: §A.1 · p. 14

A.2 Proof of Theorem 1

The proof section combines matrix perturbation and concentration arguments to establish the stated bounds. Its supplied steps include an epsilon-net bound and a sub-Gaussian empirical-Gram-matrix result.

A.2 Proof of Theorem 1

  • The proof organizes intermediate steps around orthogonal matrices, SVD properties, and a reduction to a sufficient condition.The supplied fragments include a lemma for orthogonal matrices and refer to a k-th singular value.Source: §A.2 · pp. 14–16
  • The proof invokes the David–Kahan sin theorem for eigenvector changes under a symmetric-matrix perturbation and notes that a scalar shift preserves eigenvectors and eigenvalue gaps.The proof also refers to singular-vector arguments and orthogonal invariance.Source: §A.2 · pp. 14–16
  • An epsilon-net argument bounds the spectral norm of a symmetric matrix by its maximum quadratic form over the net, scaled by 1/(1−2ϵ).The supplied proof applies the bound with ϵ = 1/4.Source: §A.2 · pp. 16–17
  • A concentration result assumes i.i.d. σ-sub-Gaussian vectors with covariance Σ and gives a high-probability bound for their empirical Gram matrix.The proof proceeds through standard concentration tools and a sub-exponential tail bound, then inverts the bound.Source: §A.2 · p. 17

A.2.1 Proof of Theorem 1

The proof represents a vector using components parallel and orthogonal to v, then uses projection matrices for clean and whole instances to establish its bound. The conclusion applies when τ is sufficiently small.

A.2.1 Proof of Theorem 1

  • The proof decomposes w into components along v and its orthogonal unit vector v⊥, parameterized by θ.It states w = cos θ · v + sin θ · v⊥, with −π/2 ≤ θ ≤ π/2.Source: §A.2.1 · p. 17
  • Projection matrices for clean instances and all instances set up the Davis–Kahan sin theorem comparison.The proof then bounds the difference between the first and second eigenvalues of the clean-instance Gram matrix.Source: §A.2.1 · pp. 17–18
  • When τ is sufficiently small, the proof combines earlier equations to obtain the stated bound.The passage points to Eqs. (14), (16), and (17), together with Lemma 3.Source: §A.2.1 · pp. 18–19

A.3 Additional Theorem

This theorem gives high-probability lower bounds for FINE’s precision and recall after projection, under a two-Gaussian feature-distribution model. The bounds improve with the separation between the projected means of clean and noisy instances.

A.3 Additional Theorem

  • The theorem bounds precision and recall after projecting features onto a principal component, assuming the projected distribution is a two-Gaussian mixture.Under the stated LDA assumption and ζ = 0.5, the decision boundary is the average of the two cluster means.Source: §A.3 · p. 19
  • The theorem defines z_i as the linear projection of x_i onto an arbitrary vector.Source: §A.3 · p. 19
  • With probability 1 − δ, the theorem provides lower bounds for both precision and recall.The bounds are given in Eqs. (21) and (22).Source: §A.3 · p. 19
  • A larger difference between the projected clean and noisy means, ∆, yields larger lower bounds for both recall and precision.The passage identifies ∆ > 0 as important to computing the bounds.Source: §A.3 · pp. 20–21

B Implementation Details for section 4

The experiments compare image-classification methods on CIFAR-10, CIFAR-100, and Clothing-1M, using validation data for CIFAR datasets without predefined validation sets. Data augmentation and reporting choices vary by the stated evaluation protocol.

B Implementation Details for section 4

  • The study reports either accuracy from a model selected using an additional validation set or both best and last test accuracy without validation.The passages associate these reporting approaches with [14] [29] and [25], respectively.Source: B Implementation Details for section 4 · p. 21
  • Experiments compare image-classification methods on CIFAR-10, CIFAR-100, and Clothing-1M.For CIFAR-10 and CIFAR-100, 10% of the training data is held out for validation because these datasets lack predefined validation sets.Source: B Implementation Details for section 4 · p. 21
  • Training data are normalized and augmented with random crops and horizontal flips.Crop size is 32 for CIFAR datasets and 224 for Clothing-1M.Source: B Implementation Details for section 4 · p. 21

B.1 Sample-Selection Approaches

The appendix describes FINE’s use in sample selection, semi-supervised learning, and noise-robust-loss experiments, alongside analyses of its eigenvector choice and sensitivity. These passages specify some experimental settings and comparisons, but do not provide the corresponding table values.

B.1 Sample-Selection Approaches

  • FINE replaces Topofilter for sample selection in experiments with varied noise settings, using ResNet-34 and periodic data selection after warmup.The detector is applied every 10 epochs after a 40-epoch warmup.Source: §B.1 · p. 21

B.2 Semi-Superivsed Approaches

  • In DivideMix, FINE substitutes for the co-divide noise detector, filtering noisy instances each training epoch.DivideMix uses two networks: each divides samples into clean and noisy subsets for the other, followed by semi-supervised learning on labeled and unlabeled data.Source: §B.2 · p. 21

B.3 Collaboration with Noise-Robust Loss Functions

  • For collaboration with noise-robust losses, FINE is run every 10 epochs after 50 epochs of naive training.The experiments use CE, GCE, SCE, and ELR, with ResNet-34 models.Source: §B.3 · p. 23

C.1 Degree of Alignment

  • The first eigenvector is described as more robust to noisy vectors than other eigenvectors, while replacing it with the mean vector degrades accuracy as noise increases.Table 4 compares perturbations between vectors, and Table 5 reports CIFAR-10 test accuracies.Source: §C.1 · p. 23, Table 4 · p. 23, Table 5 · p. 23

C.2 Detailed values for Figure 7

  • Table 6 reports accuracies for noise-robust-loss approaches on CIFAR-10 and CIFAR-100 across noise types and fractions.The passage says the table reports averages and standard deviations over three trials, but supplies no values.Source: §C.2 · p. 23, Table 6 · p. 24

C.3 Hyperparameter sensitivity towards ζ

  • On CIFAR-10 with 80% symmetric noise, performance changes little for ζ between 0.4 and 0.6.The sensitivity experiment reports test accuracy and F1-score across ζ values.Source: §C.3 · p. 24, Table 7 · p. 24

C.4 Feature-dependent Label Noise

  • Additional FINE experiments use feature-dependent label noise at rates of 20% and 40%, following CORES settings.The passage begins to describe a comparison with CORES but does not state its outcome.Source: §C.4 · p. 24

C.5 Filtering Time Analysis

Filtering-time experiments compare FINE with other filtering-based methods and examine how the amount of data used for eigendecomposition affects runtime. The authors report little runtime difference across data amounts and no efficiency degradation.

C.5 Filtering Time Analysis

  • Filtering time is compared with other filtering-based methods per epoch on one Nvidia GeForce RTX 2080.Table 9 reports the CIFAR-10 filtering-time analysis.Source: §C.5 · p. 24, Table 9 · p. 25
  • Changing the number of instances used for eigendecomposition makes little difference to computational time.Source: §C.5 · p. 24
  • The authors report that FINE's computational efficiency is obtained without degradation issues.Source: §C.5 · p. 24

C.6 Other Real-world Noisy Datasets

On mini-WebVision, FINE outperforms CRUST in the reported test accuracy comparison. The authors note that the Dividemix comparison is marginal under constrained batch size and untuned hyperparameters.

C.6 Other Real-world Noisy Datasets

  • 75.24% for FINE versus 72.40% for CRUST on the mini-WebVision test dataset.Table 10 covers top-1 and top-5 accuracy on the WebVision and ImageNet ILSVRC12 validation sets.Source: §C.6 · p. 25, Table 10 · p. 25
  • The difference between Dividemix and F-Dividemix is marginal in the reported comparison.Source: §C.6 · p. 25, Table 10 · p. 25
  • The authors attribute the marginal comparison to reduced batch size from GPU limits and the inability to tune weight decay or learning rate.They state that a final version supplementing this issue is expected to improve performance.Source: §C.6 · p. 25
Loading 2102.11628v3…