Source-linked AI summary
Masking: A New Perspective of Noisy Supervision
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, Masashi Sugiyama
TL;DR
Noisy-label learning must estimate an unknown transition matrix, but two-step and finite-data end-to-end approaches have practical estimation difficulties. Masking incorporates human cognition of invalid transitions into a structure-aware probabilistic model, reducing the probabilities that must be estimated. Experiments report significantly improved classifier robustness across structured and agnostic noise settings.
Problem
Unknown noise transition matrices are difficult to estimate accurately because two-step methods are indirect and end-to-end methods face finite-data limitations.
Method
Masking uses human cognition of invalid class transitions as a structure prior in a structure-aware probabilistic model, addressing structure extraction and alignment.
Results
Masking significantly improves classifier robustness across CIFAR-10, CIFAR-100, and Clothing1M experiments, with performance nearly comparable to CLEAN for tri-diagonal and block-diagonal noise.
Takeaways & Limitations
By estimating only unmasked noise transition probabilities, Masking substantially reduces the estimation burden while improving robustness to noisy labels.
Takeaways & Limitations
The approach assumes a human-specified noise structure, and future work must address correcting an incorrectly initialized structure.
Abstract
from arXiv · showhide
It is important to learn various types of classifiers given training data with noisy labels. Noisy labels, in the most popular noise model hitherto, are corrupted from ground-truth labels by an unknown noise transition matrix. Thus, by estimating this matrix, classifiers can escape from overfitting those noisy labels. However, such estimation is practically difficult, due to either the indirect nature of two-step approaches, or not big enough data to afford end-to-end approaches. In this paper, we propose a human-assisted approach called Masking that conveys human cognition of invalid class transitions and naturally speculates the structure of the noise transition matrix. To this end, we derive a structure-aware probabilistic model incorporating a structure prior, and solve the challenges from structure extraction and structure alignment. Thanks to Masking, we only estimate unmasked noise transition probabilities and the burden of estimation is tremendously reduced. We conduct extensive experiments on CIFAR-10 and CIFAR-100 with three noise structures as well as the industrial-level Clothing1M with agnostic noise structure, and the results show that Masking can improve the robustness of classifiers significantly.
1 Introduction
Noisy labels degrade classifier accuracy, while existing transition-matrix estimators struggle with heuristic two-step estimation or finite-data end-to-end learning. Masking uses human-specified invalid transitions to constrain estimation and improves robustness.
- Motivation: Noisy labels systematically corrupt training data, degrading classifier accuracy and deep models’ generalization because they can memorize the noise.Such labels occur in web queries, social-network tagging, crowdsourcing, medical images, and financial analysis.
- Existing approaches: Sample-selection methods and regularization techniques address label noise but introduce difficult-to-characterize selection or permanent regularization biases.Examples include small-loss or disagreement samples, manifold regularization, virtual adversarial training, temporal ensembling, and mean teacher.
- Transition-matrix estimation: Two-step transition-matrix estimation is limited by heuristic accuracy, whereas brute-force end-to-end estimation is inexact with finite datasets.The two-step approach uses limited estimation data, while end-to-end methods offer a unified framework.
- Masking: Masking conveys human cognition of invalid class transitions and uses that structure as a constraint on estimating noise transition probabilities.Estimating only probabilities along the supplied structure substantially reduces the estimation burden.
- Method and evaluation: Masking derives a structure-aware probabilistic model with a structure prior and addresses structure extraction and alignment through a tempered sigmoid and a GAN variant.Experiments cover CIFAR-10, CIFAR-100, and Clothing1M, including three structured noise types and agnostic noise.
2 A new perspective of noisy supervision
Human-task interactions can create structured label noise because cognition masks invalid transitions and highlights plausible ones. Masking represents these patterns as transition-matrix structures, then estimates probabilities within the resulting structure.
- Human cognition: Human cognition can mask invalid class transitions and highlight valid ones during interactions such as social-network tagging and crowdsourcing.The paper links this behavior to the structure of the noise transition matrix.
- Representative structures: In social-network tagging, scenery labels can be confused with foreground animals, yielding a column-diagonal noise structure.The dog and cat classes form the column lines in the example.
- Representative structures: Fine-grained annotation errors between adjacent similar classes are represented by a tri-diagonal noise structure.The example uses several closely related terrier classes and highlights neighboring valid transitions.
- Representative structures: Hierarchical-grained errors are represented by a block-diagonal matrix, with each block corresponding to a super-class.Transitions among sub-classes are treated as possible, while transitions across super-classes are treated as impossible.
- Structure-guided estimation: Once the noise structure is known, Masking focuses estimation on the noise transition probabilities compatible with that structure.Figure 1 encodes ground-truth labels vertically, noisy labels horizontally, and valid versus invalid transitions with white versus black blocks.
3 Learning with Masking
Masking incorporates human-provided noise-structure information into an end-to-end probabilistic model, reducing estimation to plausible transitions while addressing structure extraction and alignment.
- 3.1 Deficiency of benchmark models: Benchmark methods estimate the noise transition matrix before or jointly with classifier learning, but finite or agnostic noisy data make accurate estimation difficult.Two-step estimation is described as heuristic for agnostic noise, while brute-force joint estimation is inexact with finite datasets.
- 3.2 Does structure matter?: Human cognition supplies constraints on invalid class transitions, allowing the end-to-end model to estimate only noise-transition probabilities along the permitted structure.The constrained model reduces the estimation burden, making a good local minimum easier to find.
- 3.2.1 Straightforward dilemma: A direct regularizer is rejected because it requires an unjustified distance measure and parameter, cannot reliably support noisy validation correction, and demands costly tuning with clean validation data.The paper specifically notes that the inverse of an estimated transition matrix cannot be accurately computed and that repeated training can consume substantial resources.
- 3.2 Does structure matter?: Masking represents the noise transition matrix with a random variable and controls the prior of a corresponding structure variable derived through a mapping function.The model assumes s_o = f(s), where s is the transition matrix and s_o describes its structure.
- 3.2.2 When structure meets generative model: Masking instead derives an end-to-end ELBO whose first term models noisy-label likelihood and whose second term measures mismatch between learned and human-specified structure distributions.The variational distribution Q(s_o) is compared with the structure prior P(s_o); the framework avoids tuning a regularization parameter.
- 3.2.3 Towards principled realization: The approach uses implicit neural distributions and a GAN-style generator–discriminator game to implement Q(s), Q(s_o), and the divergence from P(s_o).This avoids specifying explicit distributions, while the structure prior may still be sparse and difficult to optimize.
4 Related literature
Related work addresses noisy labels through statistical learning, deep learning, and noise-transition-matrix estimation. Existing matrix-estimation methods trade limited estimation data for heuristics or unified learning for finite-data inaccuracy.
- Statistical learning studies surrogate losses, noise-rate estimation, and probabilistic modeling for noisy labels.Examples include unbiased noise-corrected losses, robust non-convex losses, class-probability estimators, and annotator models.
- Deep-learning approaches include knowledge distillation, label-cleaning networks, and joint optimization for learning from noisy labels.
5 Experiments
Experiments evaluate MASKING on benchmark datasets with three structured noise types and on Clothing1M with agnostic noise. MASKING consistently outperforms the compared noisy-label methods and approaches CLEAN in two structured cases.
- Datasets: Experiments use CIFAR-10 and CIFAR-100 with column-diagonal, tri-diagonal, and block-diagonal noise structures, plus Clothing1M with agnostic noise.CIFAR datasets contain 50k training and 10k test samples; Clothing1M contains 1000k training and 1k test samples.
- Baselines and measurements: MASKING is compared with F-correction, S-adaptation, NOISY, and CLEAN using test accuracy on clean test data.CLEAN serves as an oracle or upper bound.
- Benchmark results: MASKING persistently outperforms F-correction, S-adaptation, and NOISY on benchmark datasets.In tri-diagonal and block-diagonal cases, its performance is nearly comparable to CLEAN.
- Matrix estimation: MASKING estimates the tri-diagonal noise-transition matrix better than F-correction and S-adaptation, which severely fail in that case.F-correction pre-estimates a non-ideal matrix, while S-adaptation worsens it.
- Clothing1M results: On Clothing1M, MASKING still outperforms other methods despite an unknown noise model and an approximated structure prior.The authors attribute the marginal advantage over F-correction and S-adaptation partly to an inaccurate prior and complex ground-truth structure.
6 Conclusions
Masking conveys human cognition of invalid class transitions to speculate the structure of the noise transition matrix. The resulting structure-aware probabilistic model improves classifier robustness, while correcting incorrectly specified noise structures remains future work.
- Masking conveys human cognition of invalid class transitions and speculates the structure of the noise transition matrix.
- The model incorporates a structure prior to guide estimation of the noise transition matrix.
- Empirical results demonstrate that Masking can improve classifier robustness.
- Future work will investigate whether Masking can self-correct an incorrectly specified initial noise structure using finite data.
A Deduce ELBO
The model objective is derived using variational inference. A variational distribution approximates the posterior of the noise transition matrix, and Jensen’s Inequality is applied to the data log-likelihood.
- The objective is derived from a variational-inference perspective.
- A variational distribution Q(s) approximates the posterior of the noise transition matrix s.
- Jensen’s Inequality is applied to the data log-likelihood to obtain the model objective.
- The mapping function f(·) transforms the noise transition matrix s into its structure s_o.
B Network structures and training settings
The GAN-like implementation uses generator, discriminator, and reconstructor modules, with the generator and discriminator trained at a fixed learning rate of 3e−4.
- The GAN-like structure consists of generator, discriminator, and reconstructor modules.
- Figure 5 presents the network configuration of the generator module.
- Figure 6 presents the network configuration of the discriminator module.
- The generator and discriminator use a fixed learning rate of 3e−4.
- Figure 7 presents the network configuration of the reconstructor module.