PaperlayerView exact revision on arXiv

Source-linked AI summary

Provably End-to-end Label-Noise Learning without Anchor Points

Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, Masashi Sugiyama

arXiv:2102.02400v4cs.LG

TL;DR

Label-noise learning needs a transition matrix for statistically consistent classification, but existing estimators rely on anchor points that may not exist. The paper introduces VolMinNet, which jointly fits noisy posteriors and minimizes transition-matrix simplex volume, and proves recovery under sufficiently scattered clean posteriors while reporting robust experiments.

  • Problem

    Transition-matrix estimation is generally unidentifiable without assumptions, while existing consistent estimators rely on anchor points that may be violated.

  • Method

    VolMinNet jointly optimizes noisy-label cross-entropy fitting and the volume of a trainable diagonally dominant column-stochastic transition matrix.

  • Results

    Under the sufficiently scattered assumption and sufficiently many noisy data, the estimated transition matrix converges to T and the learned posterior converges to P(Y |X).

  • Takeaways & Limitations

    The anchor-point assumption is a special case of sufficiently scattered posteriors, and VolMinNet is reported to be robust on synthetic and real-world datasets.

  • Takeaways & Limitations

    The method assumes sufficiently scattered clean class-posteriors and diagonal dominance of the true transition matrix.

Abstract

from arXiv · show

In label-noise learning, the transition matrix plays a key role in building statistically consistent classifiers. Existing consistent estimators for the transition matrix have been developed by exploiting anchor points. However, the anchor-point assumption is not always satisfied in real scenarios. In this paper, we propose an end-to-end framework for solving label-noise learning without anchor points, in which we simultaneously optimize two objectives: the cross entropy loss between the noisy label and the predicted probability by the neural network, and the volume of the simplex formed by the columns of the transition matrix. Our proposed framework can identify the transition matrix if the clean class-posterior probabilities are sufficiently scattered. This is by far the mildest assumption under which the transition matrix is provably identifiable and the learned classifier is statistically consistent. Experimental results on benchmark datasets demonstrate the effectiveness and robustness of the proposed method.

1. Introduction

Label noise arises when inexpensive annotation replaces accurate labeling, undermining test performance and motivating consistent learning methods. Without anchor points, existing transition-matrix estimators lack theoretical guarantees, while VolMinNet uses simplex volume to recover the matrix under sufficiently scattered posteriors.

  • Cheap but less accurate annotation methods introduce label noise, and deep models can overfit noisy labels, significantly degrading test performance.
  • Transition matrices support statistically consistent classifiers by connecting noisy and clean class-posterior distributions.
  • Without additional assumptions, transition matrices are unidentifiable, and violating the anchor-point assumption can produce poorly learned matrices and degenerated classifiers.
  • Noisy class-posterior probabilities are convex combinations of transition-matrix columns, so estimating the matrix becomes recovering the simplex enclosing those probabilities.
  • VolMinNet jointly optimizes noisy-posterior prediction and transition-matrix simplex volume, identifying the matrix and yielding a statistically consistent classifier when clean posteriors are sufficiently scattered.
  • The paper reports experiments on synthetic and real-world datasets after presenting the proposed framework and theory.

2. Label-Noise Learning with Anchor Points

The section formalizes class-dependent label noise and explains why transition-matrix estimation is difficult without assumptions. Anchor points make the matrix recoverable from extreme noisy posteriors, but their absence motivates an alternative consistent algorithm.

  • The noisy dataset contains corrupted labels generated from clean examples through a noise transition matrix T(x).
  • The transition matrix entries specify the probability of each noisy label given a clean label and instance, and the matrix is generally non-identifiable without additional assumptions.
  • Under class-dependent noise, the transition matrix is independent of the instance, and this work additionally assumes diagonal dominance.
  • The anchor-point assumption requires an instance for every class whose clean class-posterior probability for that class equals one.
  • With anchor points, transition-matrix estimation reduces to finding class-specific points and using their noisy class-posterior probabilities.
  • When anchor points do not exist, extreme-posterior points are not anchors, so this procedure cannot consistently estimate the matrix or guarantee a statistically consistent classifier.

3. Volume Minimization Network

VolMinNet is an end-to-end framework that learns a clean-posterior network and a constrained transition matrix together. Its objective balances noisy-label fit against the volume of the transition-matrix simplex.

  • VolMinNet removes anchor-point identification and separate loss correction by jointly learning the classifier and transition matrix end to end.
  • The classifier maps inputs to the clean class-posterior simplex, while a trainable diagonally dominant column-stochastic matrix models noisy class posteriors.
  • Training minimizes a distribution-fitting loss for the predicted noisy posterior together with a volume penalty for the simplex formed by the estimated transition-matrix columns.
  • The Lagrangian objective combines simplex volume with expected cross-entropy risk on noisy examples.
  • The method uses log det for volume measurement because determinant-based volume is numerically unstable and computationally difficult to optimize.
  • The transition matrix parameterization uses sigmoid off-diagonal entries and column normalization to enforce differentiability, diagonal dominance, and column stochasticity.

4. Theoretical Results

The paper replaces anchor points with the sufficiently scattered assumption, under which minimum-volume simplex recovery identifies the transition matrix and clean classifier.

  • Sufficiently Scattered Assumption: The sufficiently scattered assumption requires clean class-posteriors whose cone contains R and is not contained in the cone of any non-permutation unitary transform.These are the two formal conditions defining sufficient scattering.
  • Relationship to Anchor Points: The sufficiently scattered assumption can hold even when the anchor-point assumption is violated.The paper illustrates this relationship through examples where only sufficient scattering is satisfied.
  • Relationship to Anchor Points: For C > 2, the anchor-point assumption is sufficient but not necessary for sufficient scattering.Thus, sufficient scattering includes cases where anchor points do not exist.
  • Relationship to Anchor Points: The anchor-point assumption is recovered as a special case because identity posteriors satisfy the geometric conditions for sufficient scattering.The paper's proposition establishes this implication formally.
  • Consistency Result: Under sufficient scattering, sufficiently many noisy data force the optimal transition estimate to equal T and the learned classifier to equal P(Y |X = x).Theorem 1 gives these identifiability and statistical-consistency guarantees for the optimal solutions of Eq. (5).
  • Consistency Result: Minimum-volume data-enclosing convex-hull recovery explains why the method recovers the ground-truth transition matrix and clean posterior.Sufficiently scattered clean posteriors make the noisy posteriors sufficiently spread inside the simplex formed by T's columns.

5. Related Works

Related work includes heuristic, loss-correction, jointly trained, and semi-supervised approaches to label-noise learning, with differing theoretical guarantees and practical sensitivities.

  • Heuristic Algorithms: Heuristic methods select reliable samples or correct labels and may work empirically, but they lack theoretical guarantees.Examples include sample-selection and label-correction strategies.
  • Statistically Consistent Algorithms: Statistically consistent methods commonly use loss correction, where estimating the noise transition matrix is central to risk consistency.Some approaches first estimate the transition matrix and then modify the loss in a two-stage procedure.
  • Statistically Consistent Algorithms: Existing transition-matrix estimation methods generally rely on anchor points for each class.The anchor-point assumption requires instances whose clean class posterior for a class equals one.
  • Joint and One-Step Methods: Other methods model the noise transition matrix with constrained or nonlinear layers, or use total-variation regularization in one-step training.These approaches target transition estimation or neural-network overconfidence through architectural or regularization choices.
  • Semi-Supervised Methods: Semi-supervised methods such as SELF and DivideMix improve noise robustness but are sensitive to hyperparameters and changes in data or noise types.The cited methods combine techniques including augmentations and multiple networks.

6. Experiments

Experiments evaluate VolMinNet through transition-matrix estimation and classification accuracy on synthetic and real-world noisy datasets. VolMinNet generally outperforms baselines and shows robustness across noise settings, datasets, and evaluation conditions.

  • Experimental setup: Experiments measure transition-matrix estimation error and classification accuracy on MNIST, CIFAR-10, CIFAR-100, and Clothing1M.Synthetic datasets use symmetry-flipping and pair-flipping noise; Clothing1M contains real-world noisy labels.
  • Transition Matrix Estimation: VolMinNet consistently achieves lower transition-matrix estimation error than baseline methods across noise settings on MNIST, CIFAR-10, and CIFAR-100.The reported error is ||T − T_hat − ΔT_hat||_1 / ||T||_1, with lower values preferred.
  • Transition Matrix Estimation: Around 0.25 estimation error on CIFAR-100 with Flip-0.45 compares with around 0.75 for baseline methods.These results are reported as establishing state-of-the-art transition-matrix estimation performance.
  • Classification accuracy Evaluation: VolMinNet outperforms baseline methods on almost all synthetic-noise classification settings, with differences marginal under symmetric noise.With Pair-45% noise, it exceeds baselines by over 10% and has much smaller standard deviations.
  • Classification accuracy Evaluation: On Clothing1M, VolMinNet outperforms previous transition-matrix-based and heuristic methods using only the dataset’s 1M noisy examples for training and validation.The reported performance also shows certain robustness against instance-dependent noise.

7. Discussion and Conclusion

The paper addresses label-noise learning without anchor points by introducing VolMinNet and proving consistency under the sufficiently scattered assumption. Experiments demonstrate robustness, while improving noisy-posterior estimation remains identified as future work.

  • 7. Discussion and Conclusion: VolMinNet relaxes the anchor-point assumption for label-noise learning and has theoretically proved consistency for the transition matrix and learned classifier under sufficient scattering.The paper concludes that experimental results demonstrate the method’s robustness.
  • 7. Discussion and Conclusion: Improving estimation of the noisy class posterior is identified as future work because it is considered the method’s bottleneck.This scope boundary is stated in the conclusion.

A.1. Proof of Proposition 1

The proof shows that anchor points imply the sufficiently scattered condition, but are not necessary for it. It uses non-anchor posterior geometry to establish the broader condition.

  • A.1. Proof of Proposition 1: The proof establishes that the anchor-point assumption is sufficient for the sufficiently scattered assumption.It verifies both conditions required by sufficient scattering.
  • A.1. Proof of Proposition 1: Under anchor points, the selected clean-posterior matrix H equals the identity matrix.Each anchor point has clean posterior equal to the corresponding basis vector.
  • A.1. Proof of Proposition 1: Because cone{H}=cone{I}, the second sufficiently scattered condition holds for any non-permutation unitary matrix Q.The identity cone cannot be contained in cone{Q} for such Q.
  • A.1. Proof of Proposition 1: The first sufficiently scattered condition follows by showing that vectors in the cone R are nonnegative and therefore lie in the nonnegative orthant.The proof normalizes vectors and verifies each coordinate is nonnegative.
  • A.1. Proof of Proposition 1: Sufficient scattering can hold without anchor points when a set of clean posteriors covers the nonnegative orthant except for regions along the coordinate axes.This construction is described for C > 2.

A.2. Proof of Theorem 1

The theorem proof analyzes any feasible alternative factorization through an invertible transformation and compares simplex volumes. Sufficient scattering and diagonal dominance force the alternative solution to equal the ground truth.

  • A.2. Proof of Theorem 1: The proof begins with a feasible solution (T_star, h_theta_star) and relates it to the ground-truth posterior matrix H through an invertible matrix A.Full rank of T, T_star, and H supports this representation.
  • A.2. Proof of Theorem 1: Cone inclusion follows by expressing vectors in cone{H} through A^−⊤ and nonnegative transformed coefficients.This yields cone{H} ⊆ cone{A^−⊤}.
  • A.2. Proof of Theorem 1: Equality in the volume comparison forces A^−⊤ to be a permutation matrix up to diagonal scaling, and the criterion forces that scaling to be the identity.The sufficient-scattering condition restricts the transformation to permutation-diagonal form.
  • A.2. Proof of Theorem 1: Diagonal dominance eliminates nonidentity permutations, yielding T_star=T and h_theta_star=P(Y|X) as the unique optimal solution.The conclusion identifies both the transition matrix and clean posterior.

B. Experiments on datasets where possible anchor points are manually removed.

The experiments remove likely anchor points from benchmark datasets and evaluate transition-matrix estimation and classification accuracy on the resulting /NA variants. The proposed method outperforms baseline methods in these settings.

  • MNIST/NA removes 40% of instances with the largest estimated class-posterior probabilities per class, while CIFAR-10/NA and CIFAR-100/NA remove 10%.
  • Figure 5 reports transition-matrix estimation error for MNIST/NA, CIFAR-10/NA, and CIFAR-100/NA, with lower values indicating better estimation.
  • Table 3 reports classification accuracy percentages on the original datasets and their /NA variants.
  • The experimental performance shows that the proposed method outperforms the baseline methods.
Loading 2102.02400v4…