PaperlayerView exact revision on arXiv

Source-linked AI summary

A Second-Order Approach to Learning with Instance-Dependent Label Noise

Zhaowei Zhu, Tongliang Liu, Yang Liu

arXiv:2012.11854v2cs.LGcs.AI

TL;DR

Instance-dependent label noise can unevenly down-weight examples and create imbalances that first-order methods do not capture. The paper introduces covariance-assisted second-order statistics and a loss transformation to reduce this setting to class-dependent noise, with evaluations on synthetic and real-world benchmarks supporting the approach.

  • Problem

    Human annotation errors can depend on instance difficulty, making instance-dependent noise a challenging setting beyond methods designed for class-dependent noise.

  • Method

    The paper uses covariance terms between instance-dependent noise and Bayes optimal labels to construct a loss whose noisy risk corresponds to a class-dependent-noise risk.

  • Results

    The approach is evaluated on CIFAR10, CIFAR100, and Clothing1M, with the paper reporting theoretical guarantees and empirical tests under synthetic and real-world label noise.

  • Takeaways & Limitations

    Second-order statistics can capture imbalances caused by instance-dependent noise and enable the use of existing methods for class-dependent noise.

  • Takeaways & Limitations

    Imperfect estimation of the covariance terms creates a performance gap from the ideal case with perfect covariance knowledge.

Abstract

from arXiv · show

The presence of label noise often misleads the training of deep neural networks. Departing from the recent literature which largely assumes the label noise rate is only determined by the true label class, the errors in human-annotated labels are more likely to be dependent on the difficulty levels of tasks, resulting in settings with instance-dependent label noise. We first provide evidences that the heterogeneous instance-dependent label noise is effectively down-weighting the examples with higher noise rates in a non-uniform way and thus causes imbalances, rendering the strategy of directly applying methods for class-dependent label noise questionable. Built on a recent work peer loss [24], we then propose and study the potentials of a second-order approach that leverages the estimation of several covariance terms defined between the instance-dependent noise rates and the Bayes optimal label. We show that this set of second-order statistics successfully captures the induced imbalances. We further proceed to show that with the help of the estimated second-order statistics, we identify a new loss function whose expected risk of a classifier under instance-dependent label noise is equivalent to a new problem with only class-dependent label noise. This fact allows us to apply existing solutions to handle this better-studied setting. We provide an efficient procedure to estimate these second-order statistics without accessing either ground truth labels or prior knowledge of the noise rates. Experiments on CIFAR10 and CIFAR100 with synthetic instance-dependent label noise and Clothing1M with real-world human label noise verify our approach. Our implementation is available at https://github.com/UCSC-REAL/CAL.

1. Introduction

The paper addresses instance-dependent label noise, which can mislead deep networks because difficult examples are more prone to incorrect annotation. It proposes using second-order statistics to improve robustness beyond first-order approaches.

  • Instance-dependent noise makes some difficult examples more likely to be mislabeled, creating hidden imbalances that can harm training outcomes.
  • Prior theory-supported methods mainly use loss correction or regularization based on first-order statistics of model predictions.
  • The paper proposes a second-order approach using covariance terms to improve robustness under instance-dependent label noise.
  • With covariance terms, the proposed loss transforms instance-dependent noise into class-dependent noise, enabling existing methods for the easier setting.
  • The approach is evaluated on CIFAR10, CIFAR100, and Clothing1M with synthetic and real-world human label noise.

2. Preliminaries

The paper formalizes learning from noisy data while distinguishing clean labels from Bayes optimal labels. Its goal is to learn a classifier from the noisy distribution that also performs optimally with respect to Bayes labels.

  • The problem uses noisy examples drawn from a distribution with instance-dependent label noise, while clean labels remain unobservable.
  • The Bayes optimal label is the most probable label given a feature, and its distribution can differ from the clean distribution when labels are probabilistic.
  • The paper focuses on the Bayes optimal distribution because its associated second-order statistics can be estimated from noisy data.
  • The noise transition matrix T(X) describes label corruption conditional on features, with T_i,j(X) denoting its matrix elements.
  • The learning objective is to use the noisy distribution while minimizing classification error against Bayes optimal labels.

3. Insufficiency of First-Order Statistics

First-order robust losses can be insufficient for instance-dependent noise because heterogeneous error rates down-weight examples unevenly. The analysis motivates second-order statistics that capture these induced imbalances.

  • 3.2. Peer Loss with IDN: The analysis assumes closed-set noise and presents binary formulations, while noting that most discussions generalize to multiclass settings.
  • 3.1. Using First-Order Statistics in Peer Loss: Peer loss and related regularizers use first-order statistics of model predictions, but first-order information does not capture the distribution of feature-dependent noise transitions.
  • 3.1. Using First-Order Statistics in Peer Loss: Although a confidence regularizer has theoretical robustness guarantees, reaching its guaranteed optimum is difficult in highly non-convex experiments.
  • 3.2. Peer Loss with IDN: Theorem 1 links peer-loss error to the mean and variation of instance-dependent error rates, as well as imbalance in the Bayes optimal distribution.
  • 3.2. Peer Loss with IDN: Peer loss is invariant to class-dependent noise, but heterogeneous instance-dependent noise breaks the direct applicability of that property.
  • 3.3. Down-weighting Effect of IDN: Instance-dependent noise down-weights examples with higher noise rates, potentially biasing the learned classifier toward a distorted distribution.

4. Covariance-Assisted Learning (CAL)

CAL uses covariance terms to correct the imbalance induced by instance-dependent label noise, transforming the noisy-risk problem into one with class-dependent noise and estimating the correction from an approximate Bayes-optimal dataset.

  • Motivation: Instance-dependent noise automatically assigns different weights to examples, causing imbalances that motivate balancing effective example weights with second-order statistics.The covariance terms capture additional contributions arising from differences in instance-level noise rates.
  • 4.1. Extracting Covariance from IDN: Theorem 2 decomposes expected peer loss into an average class-dependent-noise effect plus covariance terms that grow with more heterogeneous noise rates.The covariance terms characterize the extra contribution of examples whose noise rates differ from one another.
  • 4.2. Using Second-Order Statistics: The covariance-assisted loss extends to K classes under an expected-transition-matrix condition that makes off-diagonal transition rates equal across source classes.The multi-class result is stated as Corollary 1.
  • 4.2. Using Second-Order Statistics: CAL subtracts covariance terms from peer loss so its minimizer matches the Bayes-optimal risk minimizer under the ideal distribution.The construction is motivated by Theorem 2 and is expressed through the covariance-assisted loss.
  • 4.2.1 Constructing ˆD: CAL estimates covariance terms using an approximate Bayes-optimal dataset constructed from noisy data through sample-selection procedures and thresholded adjusted losses.The procedure uses sample sieves and related selection methods, then estimates transition and covariance quantities during SGD.
  • 4.3. CAL with Imperfect Covariance Estimates: Theorem 4 shows covariance assistance improves the worst-case accuracy guarantee when the proxy dataset contains correct examples with probability at least 0.5.The proxy quality controls the scale of the worst-case error upper bound.

5. Experiments

Experiments on CIFAR10, CIFAR100, and Clothing1M evaluate CAL against robust-loss, correction, and sample-selection baselines. CAL remains strongest under high synthetic noise, performs well on real human noise, and benefits from combining covariance and peer terms.

  • 5.1. Experimental Setup: Experiments use ResNet34 for CIFAR datasets and ResNet50 for Clothing1M, with comparisons spanning correction, selection, and noise-robust loss methods.Cross-entropy is included as a common baseline, alongside generalized cross-entropy and other robust approaches.
  • 5.2. Implementation Details: CIFAR training constructs an estimated Bayes-optimal dataset before CAL retraining, using empirically selected thresholds and scheduled optimization.The reported CIFAR setting uses Lmin = Lmax = −8.3 for constructing the proxy dataset.
  • 5.3.1 CIFAR: CAL achieves the best performance as CIFAR label-noise rates become high, while most competing methods suffer severe degradation.Similar observations hold on CIFAR100, although Reweight-R can match or exceed CAL in some severe-noise trials.
  • 5.3.2 Clothing1M: On Clothing1M, CAL performs well under real-world human label noise using a class-balanced dataset constructed from noisy examples.The method selects 18,976 noisy examples per class before continued training.
  • 5.4. Ablation Study: The covariance and peer terms each work individually, but their combination significantly improves performance on CIFAR10.The covariance term improves accuracy from 84.41% to 85.55% when combined with the peer term at η = 0.4.

6. Conclusions

The paper concludes that second-order covariance information can transform instance-dependent label noise into a class-dependent setting that existing methods can address. It estimates this information through sample selection while identifying more accurate covariance estimation as future work.

  • 6. Conclusions: CAL transforms challenging instance-dependent label noise into class-dependent label noise, enabling the use of existing methods for the latter setting.The conclusion frames this transformation as the paper’s central outcome.
  • 6. Conclusions: The current covariance-term estimates rely on a sample-selection method, and future work will explore alternative estimation techniques.The paper also proposes combining second-order information with other robust learning techniques.

A.2. Proof for Lemma 2

The proof establishes that the optimal peer classifier on the Bayes optimal distribution also minimizes its 0-1 loss. Unlike the clean class-balanced case, this result does not require a balanced Bayes-label prior.

  • Lemma 2: The optimal peer classifier defined in Lemma 2 also minimizes expected 0-1 loss on the Bayes optimal distribution.The proof argues by contradiction from the assumed non-optimality of the Bayes classifier.
  • Lemma 2: The lemma holds without requiring the Bayes-optimal class prior p* to equal 0.5.This differs from the earlier clean-distribution result, which assumes a class-balanced dataset.

B.1. Proof for Theorem 1

The proof expands peer loss under instance-dependent noise into Bayes-distribution loss, class-dependent terms, and covariance corrections. It then relates these components through the noise-rate variables and bounded-variance assumptions.

  • Theorem 1: The covariance corrections capture dependencies between instance-dependent noise rates and classification-error indicators.The proof writes covariance expressions as centered products and uses them in the risk expansion.
  • Theorem 1: The proof assumes bounded variance in the error rates before deriving the final comparison involving Bayes-optimal loss and covariance terms.The assumption is introduced immediately before the key inequality chain.
  • Theorem 1: The expected peer loss decomposes into a Bayes-optimal loss term, error-rate-weighted classification terms, and covariance terms.The decomposition introduces Term-A and Term-B using instance-dependent error rates e+(X) and e−(X).
  • Theorem 1: Replacing instance-dependent error rates by their expectations yields class-dependent terms plus the covariance residuals.The resulting expression separates expected noise-rate effects from instance-specific dependence.

B.4. Proof for Theorem 4

The proof relates covariance terms estimated on noisy and clean subsets, using the expected correct-example ratio τ to bound their discrepancy.

  • The expected correct-example ratio τ defines the probability that an example in ˆDτ belongs to D*; the complementary probability is 1 − τ.
  • The argument assumes D*, ˆDτ, and ˜D share the same feature set.
  • The covariance discrepancy is bounded by 2(1 − τ)(ε+ + ε−), linking estimation error to the incorrect-example fraction.
  • The proof concludes by applying the proof strategy of Theorem 1 to the stated covariance relationships.

C. Proof for Corollaries

The corollaries analyze two strategies for constructing and weighting feasible examples, then describe instance-dependent noise generation used in experiments.

  • Strategy-1: Lmin < Lmax: Strategy 1 may increase the proportion of Bayes-optimal labels by dropping ambiguous examples, but requires importance re-weighting to correct covariate shift.
  • Strategy-1: Lmin < Lmax: With suitable thresholds and optimal weights γ*(X), the covariance can be guaranteed to be optimal when every feasible example is re-weighted.
  • Strategy selection: Strategy 2 sets Lmin = Lmax, avoids estimating γ(X), and is selected because it reduces computation, potential errors, and hyperparameter tuning.
  • Instance-dependent label noise: The noise generator samples instance flip rates from a truncated normal distribution and uses a random matrix to produce instance-dependent class probabilities.
  • Instance-dependent label noise: When η > 0.5, the generator caps incorrect-class probabilities at 0.9·Tii(X) to maintain an informative dataset.

D.3. Performance without Data Augmentations

This section reports performance comparisons without data augmentations, using the experimental setup associated with Table 4.

  • The comparison reproduces a recent instance-dependent-label-noise method using the same data augmentations and noise file before removing augmentations for Table 4.
  • Table 4 presents performance comparisons without data augmentations.

D.4. More Implementation Details on Clothing1M

The Clothing1M implementation first constructs a feasible subset from noisy data, then continues training with the CAL loss under a separate schedule.

  • Construct ˆD: The network is initially trained for 120 epochs on 1 million noisy images with balanced noisy labels, mixup, and a learning-rate decay schedule.
  • Train with CAL: After constructing ˆD, training switches to CAL for 120 additional epochs without mixup, using an initial learning rate of 10^-5.
  • Train with CAL: Retraining with ˆD was also tested and achieved an accuracy of 73.56.
Loading 2012.11854v2…