Source-linked AI summary
Are Anchor Points Really Indispensable in Label-Noise Learning?
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, Masashi Sugiyama
TL;DR
Label-noise learning depends on transition matrices, yet anchor-point assumptions may fail and make existing matrices inaccurate. The paper proposes T-Revision, which initializes the matrix from anchor-like examples and revises it jointly with a classifier using noisy data; experiments report improved label-noise learning performance.
Problem
Without anchor points, existing methods can inaccurately estimate transition matrices, degrading current consistent algorithms.
Method
T-Revision initializes a transition matrix from examples with high learned noisy class posterior probabilities, then learns a slack-variable revision with the classifier using noisy data.
Results
Experiments on synthetic and real-world label-noise data show that T-Revision can significantly boost label-noise learning performance.
Takeaways & Limitations
T-Revision maintains a risk-consistent learning paradigm without using exact anchor points and tunes transition matrices closer to their ground truths.
Takeaways & Limitations
The theoretical development assumes an instance-independent, class-dependent transition matrix and includes an importance-reweighting stability condition.
Abstract
from arXiv · showhide
In label-noise learning, \textit{noise transition matrix}, denoting the probabilities that clean labels flip into noisy labels, plays a central role in building \textit{statistically consistent classifiers}. Existing theories have shown that the transition matrix can be learned by exploiting \textit{anchor points} (i.e., data points that belong to a specific class almost surely). However, when there are no anchor points, the transition matrix will be poorly learned, and those current consistent classifiers will significantly degenerate. In this paper, without employing anchor points, we propose a \textit{transition-revision} ($T$-Revision) method to effectively learn transition matrices, leading to better classifiers. Specifically, to learn a transition matrix, we first initialize it by exploiting data points that are similar to anchor points, having high \textit{noisy class posterior probabilities}. Then, we modify the initialized matrix by adding a \textit{slack variable}, which can be learned and validated together with the classifier by using noisy data. Empirical results on benchmark-simulated and real-world label-noise datasets demonstrate that without using exact anchor points, the proposed method is superior to the state-of-the-art label-noise learning methods.
1 Introduction
Label-noise learning addresses increasingly large datasets whose labels are expensive to annotate accurately. Consistent methods use the noise transition matrix, but existing transition-matrix learning can fail without anchor points, motivating T-Revision.
- Large-scale datasets are often annotated with noisy labels because accurate annotation is expensive.
- Label-noise methods include heuristic approaches that select, reweight, or correct examples and statistically consistent approaches.
- The noise transition matrix records probabilities that clean labels flip into noisy labels and supports risk- and classifier-consistent algorithms.
- Existing transition-matrix learning methods rely on anchor points, so inaccurate matrices without anchors can degrade consistent classifiers.
- T-Revision initializes the matrix from examples resembling anchor points, then learns a slack-variable revision with the classifier using noisy data.
2 Label-Noise Learning with Anchor Points
This section defines the noisy-label setting, transition matrices, consistent algorithms, and anchor points. It explains that anchor-point assumptions support transition-matrix learning but that their absence can produce inaccurate matrices and degraded classifiers.
- The clean data distribution contains features X and clean labels Y, while observed training examples contain independently flipped noisy labels.
- A transition matrix gives P(¯Y = j|Y = i, X = x), linking clean and noisy labels; the paper studies class-dependent, instance-independent matrices.
- Consistent algorithms use the transition matrix to modify losses for risk consistency or correct hypotheses for classifier consistency.
- An anchor point for class i is an instance with P(Y = i|X = x) equal to one or close to one.
- Anchor points enable transition-matrix estimation from noisy class posteriors, but their absence can yield inaccurate matrices and degraded consistent algorithms.
Learning without Anchor Points
T-Revision learns transition matrices without exact anchor points by initializing them from high noisy-posterior examples and revising them with a learned slack variable. Its inverse-free risk-consistent estimator tunes the matrix and classifier jointly using noisy data.
- Motivation: Anchor-point methods can inaccurately estimate transition matrices when selected examples are not truly anchor points or when anchor points are absent.Using a non-identity clean-posterior matrix L yields an estimated matrix TL rather than the true matrix T.
- Motivation: A small transition-matrix error can change the inferred clean posterior enough to produce an incorrect class prediction.In the five-class example, changing two entries causes the predicted class to switch from the fourth to the third.
- Risk-consistent estimator: The proposed estimator avoids explicitly inverting the transition matrix by directly estimating clean posteriors and modeling their noisy counterparts through Tᵀg(x).Importance reweighting rewrites clean-data risk using noisy-data observations and does not require the inverse transition matrix.
- T-Revision method: T-Revision initializes the transition matrix using examples with the highest estimated noisy class posterior probabilities, then adds a learnable slack variable.The classifier and slack variable are optimized in a second training stage using a weighted risk-consistent loss.
- Validation and guarantees: The method uses noisy validation data to assess the learned slack variable and transition revision, while its generalization analysis bounds the gap between training and test risk.The paper reports superior classification accuracy to state-of-the-art methods in experiments.
4 Experiments
Experiments evaluate T-Revision on synthetic and real-world noisy-label datasets, comparing anchor availability, estimator type, revision, noise level, and transition-matrix estimation error.
- Datasets and setup: Experiments use MNIST, CIFAR-10, CIFAR-100, and Clothing1M, with synthetic corruption for the first three and real-world noisy labels for Clothing1M.The synthetic experiments are repeated five times, while Clothing1M includes 1M noisy images plus clean data for training, validation, and testing.
- Importance of anchor points: On MNIST, methods without removed possible anchors do not uniformly outperform alternatives because dataset properties influence methods independent of transition matrices.“-N/A” methods work better on MNIST, whereas “-A” methods work better on CIFAR-10 and CIFAR-100 for the compared inconsistent methods.
- Importance of anchor points: Without possible anchor points, Forward and Reweight show clear degeneration, with CIFAR-100 accuracy dropping at least 4%.Possible anchors are removed using instances with high estimated clean class posterior probabilities.
- Different noise levels: The proposed method outperforms baselines more significantly as MNIST noise increases from the standard setting to 60%, 70%, and 80%.These results are reported in Table 3 under otherwise unchanged experiment settings.
- Risk-consistent versus classifier-consistent estimators: Reweight is slightly better than or comparable to Forward, while avoiding the inverse of the transition matrix.Forward is classifier-consistent; Reweight is risk-consistent and does not involve transition-matrix inversion.
- T-Revision results: T-Revision significantly outperforms the other compared methods and improves classification without removing possible anchor points.The proposed Reweight-R also performs significantly better than the Forward-R baseline.
- Transition-matrix estimation: Risk-consistent-estimator-based revision produces smaller transition-matrix estimation errors than the classifier-consistent Forward-R approach across the reported cases.Figure 3 reports estimator error under sym-20 and sym-50 label noise, with shaded standard-deviation error bars.
5 Conclusion
The paper reframes anchor points as unnecessary for consistent label-noise learning by introducing T-Revision and a risk-consistent estimator that avoids transition-matrix inversion.
- T-Revision revises an estimated transition matrix and validates the revision using noisy data.
- Consistent learning methods estimate label noise through a transition matrix and apply statistical or algorithmic correction using that estimate.
- The paper’s novelty is relaxing the requirement for anchor points when estimating the transition matrix for consistent learning.
- Methods that select or correct labels do not estimate the transition matrix and therefore do not require anchor points.
- For transition-matrix-based label-noise learning, anchor points were previously necessary but are no longer required by the proposed approach.
B How consistent algorithms work
Consistent algorithms use the transition matrix to connect noisy and clean label posteriors or losses, but existing risk-consistent estimators rely on matrix inversion and can be inefficient or degenerate.
- A risk-consistent method estimates clean risk from noisy data, while a classifier-consistent method converges toward the clean-data optimal classifier.
- The learned network output g(X) approximates the noisy class posterior P(¯Y|X), which relates to the clean posterior through P(¯Y|X) = T^⊤P(Y|X).
- Noise-adaptation layers modify the network’s softmax output so noisy-data training can produce classifier-consistent predictions.
- When T is invertible, clean posteriors can be recovered as P(Y|X) = (T^⊤)^−1P(¯Y|X), motivating transition-adjusted loss functions.
- Current risk-consistent estimators require transition-matrix inversion, making parameter tuning inefficient and causing performance degeneration, especially for non-invertible matrices.
C Proof of Theorem 1
Theorem 1 combines concentration, complexity, and neural-network bounds to control the learned weighted empirical risk for the revised transition matrix and classifier.
- Lemma 1 bounds the deviation between expected and empirical weighted risk for the learned slack variable and classifier under a valid revised transition matrix.
- The complexity argument uses Rademacher-complexity bounds for the deep-network function class induced by h.
- The neural-network bound assumes bounded Frobenius norms, 1-Lipschitz positive-homogeneous activations, and bounded input norm.
C.1 Proof of Lemma 1
Lemma 1 controls generalization of the weighted loss using bounded differences and McDiarmid’s inequality under validity constraints on the revised transition matrix.
- The proof checks the bounded-difference property of Φ(S) and applies McDiarmid’s concentration inequality.
- Validity requires every revised transition entry to be nonnegative and each diagonal entry to exceed the corresponding off-diagonal entries.
- Under these validity conditions, the weighted loss is bounded by CM.
C.2 Proof of Lemma 2
The proof establishes boundedness properties for valid revised transition matrices and uses them to handle the supremum argument underlying Lemma 2.
- Bound for valid revisions: For any valid revised matrix, the relevant transformed quantity is bounded above by C.The bound follows through the minimum diagonal entry of the revised transition matrix.
- Supremum argument: The proof constructs two transition revisions and classifiers for every ϵ > 0 to approach the supremum.This invokes the definition of the supremum directly.
- Conclusion: The final inequality holds because the transformed quantity remains bounded for every valid revised matrix.Since the inequality holds for arbitrary ϵ, the desired conclusion follows.
C.3 Proof of Lemma 3
The proof of Lemma 3 establishes Lipschitz continuity of the loss with respect to each model output and uses rank preservation and contraction arguments.
- Lipschitz property: The loss function is shown to be 1-Lipschitz-continuous with respect to each h_i(X).The proof differentiates the loss with respect to h_i(X) for cases where i differs from the noisy label.
- Rank-preserving transformation: Softmax preserves the rank of the logits, allowing the classifier constraint to be expressed through the corresponding h_i(X) values.The proof also notes that arg max and max impose the same constraint on h_i.
- Noise model: The symmetry-flipping transition matrix is defined using C classes and a noise rate ϵ.Its diagonal entries are 1 − ϵ, while off-diagonal entries are ϵ/(C−1).
- Empirical comparison: Figure 4 compares transition-matrix estimation errors for classifier-consistent and risk-consistent estimators under sym-20 and sym-50 noise.The standard-deviation error bars are shown as shaded regions.
E More discussions about Figure 3
The appendix discussion compares transition-matrix learning with and without exact anchor points and explains why risk-consistent revision is more effective.
- Anchor-point initialization: At epoch 0, the estimation error represents the transition matrix obtained by identifying anchor points with the traditional method.Methods marked “-N/A” operate on modified datasets excluding instances with large clean class posterior probabilities.
- Classifier-consistent revision: Classifier-consistent revision can fail to learn a good transition matrix because its objective does not directly target transition-matrix quality.The discussion gives Forward-N/A-R as an example, including behavior shown in Figure 1(a).
- Risk-consistent revision: Risk-consistent methods consistently produce smaller transition-matrix estimation errors than the corresponding classifier-consistent methods.This comparison is reported for Reweight-A-R and Reweight-N/A-R against Forward-A-R and Forward-N/A-R.