PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Inference via Generative Classifiers for Handling Noisy Labels

Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, Jinwoo Shin

arXiv:1901.11300v2stat.MLcs.LG

TL;DR

Noisy labels can cause modern DNNs to generalize poorly, motivating an inference method that avoids retraining. RoG fits a robust generative classifier to fixed hidden features using MCD estimation and reports improved accuracy across noisy-label settings, including 43.02% versus 33.34% on CIFAR-100 with 45% noise.

  • Problem

    Modern DNNs may generalize poorly when trained on large datasets containing incorrect machine-generated labels, while existing remedies require costly retraining and tuning.

  • Method

    RoG post-processes fixed hidden features with an LDA-based generative classifier whose parameters are estimated using the minimum covariance determinant estimator.

  • Results

    43.02% test accuracy versus 33.34% for Han et al. (2018b) on CIFAR-100 with 45% noisy labels, with improvements also reported across architectures, tasks, and noisy-label types.

  • Takeaways & Limitations

    A simple generative classifier can be substantially more robust to labeling noise without much sacrifice of discriminative performance on clean-label data.

  • Takeaways & Limitations

    The theoretical superiority result depends on stated assumptions, including Gaussian features and conditions such as sufficiently large N_c; relaxing them remains future work.

Abstract

from arXiv · show

Large-scale datasets may contain significant proportions of noisy (incorrect) class labels, and it is well-known that modern deep neural networks (DNNs) poorly generalize from such noisy training datasets. To mitigate the issue, we propose a novel inference method, termed Robust Generative classifier (RoG), applicable to any discriminative (e.g., softmax) neural classifier pre-trained on noisy datasets. In particular, we induce a generative classifier on top of hidden feature spaces of the pre-trained DNNs, for obtaining a more robust decision boundary. By estimating the parameters of generative classifier using the minimum covariance determinant estimator, we significantly improve the classification accuracy with neither re-training of the deep model nor changing its architectures. With the assumption of Gaussian distribution for features, we prove that RoG generalizes better than baselines under noisy labels. Finally, we propose the ensemble version of RoG to improve its performance by investigating the layer-wise characteristics of DNNs. Our extensive experimental results demonstrate the superiority of RoG given different learning models optimized by several training techniques to handle diverse scenarios of noisy labels.

1. Introduction

Noisy labels can make modern DNNs generalize poorly, while retraining-based remedies impose cost and reproducibility concerns. RoG instead post-processes fixed hidden features with a robust generative classifier, achieving stronger results across noisy-label settings.

  • Motivation: Machine-generated labels enable large-scale datasets but may be incorrect, causing modern deep architectures to generalize poorly from noisy data.The paper cites the Open Images Dataset V4 as containing 70 million machine-generated labels.
  • Motivation: Existing noisy-label training strategies can require expensive retraining, additional time, hyperparameter tuning, and may suffer reproducibility issues.RoG is presented as complementary to these methods rather than a replacement for them.
  • Approach: RoG induces a robust generative classifier on fixed features from any discriminative neural classifier pre-trained on noisy data, without retraining.The approach assumes meaningful feature patterns remain available even when labels are noisy.
  • Approach: The method uses LDA with minimum covariance determinant estimation to obtain robust parameters, with an ensemble extension motivated by layer-wise feature characteristics.Theoretical support is provided under Gaussian assumptions, where RoG has smaller parameter-estimation errors under stated conditions.
  • Results: 43.02% test accuracy versus 33.34% for Han et al. (2018b) on CIFAR-100 with 45% noisy labels demonstrates RoG’s reported improvement over a strong training method.The paper also reports gains across image classification and natural language processing, plus semantic and open-set noisy-label settings.

2. Related work

Prior work mainly addresses noisy labels during training by correcting labels or modifying objectives. Generative models on hidden features have also been used, but this paper distinguishes RoG through robust parameter estimation for noisy-label inference.

  • Training-based methods: Earlier methods estimate or correct labels during training, including bootstrapping with model predictions and loss or posterior modifications.These approaches intervene in the training procedure rather than only post-processing a pre-trained model.
  • Generative feature models: Generative models layered on hidden features have supported speech recognition and novelty detection, but prior approaches used naive parameter estimation under clean-data assumptions.RoG targets robustness to noisy labels through a different parameter-estimation strategy.

3. Robust Inference via Generative Classifiers

RoG replaces naive generative-parameter estimation on hidden DNN features with an MCD-based robust classifier, without retraining the network. It further combines classifiers from multiple layers and is theoretically supported under Gaussian assumptions.

  • Generative classifier and MCD estimator: RoG induces a generative classifier from hidden features of a pretrained softmax model and estimates its parameters robustly using the MCD estimator.The classifier uses an LDA assumption with tied covariance, while MCD selects a subset whose covariance determinant is minimized.
  • Generative classifier and MCD estimator: The LDA-based generative classifier models each class with a Gaussian distribution sharing covariance Σ and uses a Bernoulli class prior.Its posterior is obtained from the Gaussian class-conditional distributions and class priors through Bayes' rule.
  • Generative classifier and MCD estimator: MCD improves robustness by excluding outliers from parameter estimation, whereas naive sample estimates can be strongly influenced by noisy-label samples.For each class, MCD selects a subset with minimum covariance determinant before estimating means and covariance.
  • Theoretical properties: Under the paper’s assumptions, MCD has a near-optimal breakdown value of approximately 50% and yields smaller parameter-estimation errors than the sample estimator.The theoretical result supports a better generalization bound for the MCD-based classifier than for the sample-estimator baseline.
  • Theoretical properties: The theoretical analysis assumes Gaussian hidden-feature distributions, equal class sizes and outlier fractions, and constraints on the outlier and MCD-selected fractions.The authors identify relaxing most of these assumptions as future work.
  • Ensemble of generative classifiers: The ensemble version induces MCD-based generative classifiers at multiple hidden layers and combines their posterior distributions using learned layer weights.The method uses features from low-level layers as well as penultimate features; weights are tuned using validation data.

4. Experiments

Experiments evaluate RoG across vision and NLP models, datasets, noise types, covariance assumptions, and training strategies. RoG generally improves noisy-label classification, with ensemble and robust estimation choices supporting its performance.

  • Experimental settings: RoG is evaluated on DenseNet, ResNet, and other neural networks across CIFAR, SVHN, Twitter POS, and Reuters datasets with random, semantic, and open-set noise.The experiments include both convolutional and fully connected models, and cover uniform, flip, machine-generated semantic, and out-of-distribution label noise.
  • Ablation study: LDA-based generative classifiers generalize better than softmax and identity-covariance alternatives under uniform label noise.The comparison uses features from the penultimate layer of a ResNet-34 trained on CIFAR-10; LDA remains comparable to softmax on clean data.
  • Ablation study: MCD estimation further improves the LDA-based classifier by selecting training samples with clean labels rather than relying only on naive sample estimates.The reported improvement is attributed to the robust parameter estimation provided by the minimum covariance determinant estimator.
  • Ablation study: Low-level feature classifiers are more stable as noise increases, enabling the ensemble version of RoG to significantly improve classification accuracy.The paper relates this behavior to lower feature dimensionality and the inverse relationship between MCD’s breakdown point and feature dimension.
  • Compatibility and comparison: RoG improves softmax accuracy when combined with single-network noisy-label training methods, with gains reported as more significant than those from the specialized training methods.The comparison uses ResNet-44 under uniform noise and includes bootstrapping, forward/backward correction, and D2L.
  • Compatibility and comparison: RoG with Co-teaching achieves the best performance in all tested cases, while also improving results on NLP tasks without using the trusted labels available to competing methods.The Co-teaching comparison covers CIFAR-10 and CIFAR-100 with uniform and flip noise; the NLP comparison covers Reuters and Twitter POS.
  • Semantic and open-set noisy labels: RoG achieves comparable or significantly better test accuracy than softmax under open-set noise, where out-of-distribution images retain labels from the target classes.The open-set benchmark replaces some CIFAR-10 training samples with out-of-distribution images while preserving class labels and per-class image counts.

5. Conclusion

The paper proposes RoG, an inference method that induces a generative classifier on fixed features from a pre-trained model to handle noisy labels. It argues that this approach can be more robust to labeling noise without substantially sacrificing clean-label discriminative performance.

  • RoG induces a generative classifier on fixed features from a pre-trained model for inference under noisy labels.
  • Generative classifiers, often outperformed by discriminative deep classifiers in fully supervised settings, can nevertheless be significantly more robust to labeling noise.

Supplementary Material:

The preliminaries contrast discriminative and generative classifiers and specialize the latter to Gaussian discriminant analysis with a shared covariance matrix. The section also describes the robustness rationale for the minimum covariance determinant estimator.

  • A discriminative classifier directly models the posterior P(y|x), whereas a generative classifier specifies P(x|y) and P(y) to define the posterior indirectly.
  • Under the Gaussian assumption, the generative classifier models class-conditional features with multivariate Gaussians and class priors with a Bernoulli distribution.
  • LDA is the GDA special case in which all classes share one covariance matrix, canceling the quadratic term in the posterior representation.
  • The resulting generative posterior is equivalent to a softmax classifier after expressing its Gaussian parameters as softmax weights and biases.
  • The MCD estimator is motivated by high breakdown points, measuring how much arbitrary contamination can drive an estimator beyond all bounds.

B. Experimental setup

The experiments evaluate RoG across neural architectures, image and language datasets, noisy-label training methods, and open-set noise scenarios. They also examine layer-wise generative classifiers and robust validation-based ensemble weighting.

  • Datasets and models: The study uses DenseNet and ResNet architectures for CIFAR-10, CIFAR-100, and SVHN image classification, with softmax models trained by cross-entropy.
  • Datasets and models: The NLP evaluation uses Twitter part-of-speech tagging and Reuters text categorization, with 14,468 Twitter training examples across 19 retained classes.
  • Noise scenarios: Open-set noise replaces CIFAR-10 training images with external CIFAR-100 and downsampled ImageNet images while retaining the original CIFAR-10 labels.
  • Noise-handling methods: The evaluation includes bootstrapping, loss-correction, dimensionality-based, decoupling, MentorNet, Co-teaching, and conventional cross-entropy training methods.
  • Layer-wise analysis: Layer-wise experiments find architecture-dependent patterns, including meaningful DenseNet features after the 20th basic block.

D. Proof of Theorem 1

The proof of Theorem 1 establishes two asymptotic statements concerning estimation error and estimated error ratio as the number of training samples tends to infinity.

  • Theorem 1 is proved through separate statements for the limit of estimation error and the estimated error ratio.
  • The proof invokes the Continuous Mapping Theorem while taking the number of training samples N to infinity for the stated convergences.
  • The theorem proof addresses the two claims one by one as separate components.

D.1. Proof of the limit of estimation error (3)

This lemma analyzes the asymptotic behavior of sample and MCD estimators for hidden features containing a fixed fraction of outliers. It establishes the convergence statements used in the estimation-error analysis.

  • The proof considers N-dimensional? No; it considers N d-dimensional training samples containing outliers with fixed fraction δout < 1.The clean and outlier samples are modeled separately, with the clean distribution characterized by mean µ and covariance σ2I.
  • As N →∞, the sample estimator’s mean and covariance converge almost surely to their corresponding mixture-distribution quantities.
  • As N →∞, the MCD estimator’s mean and covariance converge almost surely to their robust limiting quantities.
  • The proof relies on expressing the covariance determinant as a polynomial in the outlier ratio under the stated assumptions.
  • The argument extends the single-distribution convergence result to multiple class-conditional hidden-feature distributions.

D.2. Proof of the limit of estimated error ratio (4)

This proof derives the asymptotic ratio of estimation errors by analyzing tied covariance estimators and a function of their condition number. The resulting limit establishes the stated error-ratio bound.

  • The classwise covariance assumptions imply tied covariance matrices formed by aggregating the class-specific sample and MCD covariance estimates.
  • The tied sample and MCD covariance matrices are shown to converge under the preceding estimator-convergence results.
  • The proof defines a function of the tied covariance matrix and rewrites it using the condition number t of bΣ−1.
  • For t ∈[1, ∞), φ(t) = 4t (1+t)2 is monotonically decreasing, so its maximum occurs at t = 1.
  • The covariance limits and the resulting φ-ratio limit establish the theorem’s stated mean-distance and error-ratio relationship.

D.3. Proof of Lemma 1

The proof of Lemma 1 compares sample and MCD estimators through asymptotic mixture distributions. It shows that the MCD-selected subset converges to the clean distribution under the stated trimming condition.

  • The sample estimator converges almost surely to the mean and covariance of the mixture distribution formed by clean and outlier samples.This follows from the fixed outlier fraction and the Strong Law of Large Numbers.
  • For MCD subsets containing outlier fraction q, the subset estimators converge almost surely to the corresponding mixture parameters µq and Σq.
  • MCD selects a subset of fixed size whose covariance determinant is minimal among the candidate subsets.
  • Under δmcd ≤ 1 −δout, the clean subset exists and det(Σq) ≥ det(Σ0), with equality only at q = 0.
  • Therefore, the MCD mean and covariance converge almost surely to the clean distribution’s µ and σ2I.

E. Comparison with robust clustering methods

The paper compares MCD with trimmed K-means as alternative robust procedures for estimating generative-classifier parameters from hidden features with noisy labels. The supplied table caption specifies the evaluation setting and highlighting rule.

  • Trimmed K-means is evaluated as a robust-clustering baseline alongside the MCD estimator.Noisy labels initialize the clusters, and each trained cluster receives a majority label.
  • The comparison uses tied covariance across clusters for the generative classifiers.
  • Table 8 reports ResNet test accuracy (%) on CIFAR-10 with uniform noise.
  • In Table 8, bold highlights indicate gains greater than 1%.
Loading 1901.11300v2…