PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-training with Noisy Student improves ImageNet classification

Qizhe Xie, Minh-Thang Luong, Eduard Hovy, Quoc V. Le

arXiv:1911.04252v4cs.LGcs.CVstat.ML

TL;DR

The paper addresses how to exploit abundant unlabeled images to improve supervised ImageNet models. Noisy Student Training iteratively pseudo-labels unlabeled data with a teacher and trains an equal-or-larger student under noise. It reaches 88.4% ImageNet top-1 accuracy and substantially improves robustness on ImageNet-A, ImageNet-C, and ImageNet-P.

  • Problem

    Supervised vision models rely on large labeled corpora, leaving much larger collections of unlabeled images underused for improving accuracy and robustness.

  • Method

    Noisy Student Training iteratively trains a teacher on labeled images, pseudo-labels unlabeled images, and trains an equal-or-larger student on both datasets with augmentation and model noise.

  • Results

    88.4% ImageNet top-1 accuracy is 2.0% better than the previous state of the art, while robustness improves substantially on ImageNet-A, ImageNet-C, and ImageNet-P.

  • Takeaways & Limitations

    Self-training with added student noise can leverage unlabeled images at scale to advance both ImageNet accuracy and robustness.

  • Takeaways & Limitations

    Robustness comparisons may benefit from augmentation overlap with ImageNet-C corruptions, and adversarial results are not directly comparable because input resolution is 800x800.

Abstract

from arXiv · show

We present Noisy Student Training, a semi-supervised learning approach that works well even when labeled data is abundant. Noisy Student Training achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2. Noisy Student Training extends the idea of self-training and distillation with the use of equal-or-larger student models and noise added to the student during learning. On ImageNet, we first train an EfficientNet model on labeled images and use it as a teacher to generate pseudo labels for 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the learning of the student, we inject noise such as dropout, stochastic depth, and data augmentation via RandAugment to the student so that the student generalizes better than the teacher. Models are available at https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet. Code is available at https://github.com/google-research/noisystudent.

1. Introduction

Noisy Student Training uses abundant unlabeled images to improve ImageNet accuracy and robustness beyond supervised learning. It combines larger students, pseudo-labeling, and noise during training.

  • Supervised vision models require many labeled images, limiting their use of much larger unlabeled image collections.
  • Noisy Student Training iteratively trains a teacher on labeled images, generates pseudo labels for unlabeled images, and trains a student on both datasets.The student is subsequently used as the teacher for relabeling and further training.
  • Equal-or-larger students and injected noise improve self-training and distillation by encouraging learning from pseudo labels.Noise includes RandAugment, dropout, and stochastic depth.
  • 88.4% ImageNet top-1 accuracy is 2.0% better than the previous state of the art using 3.5B weakly labeled Instagram images.The result uses 300M unlabeled images with EfficientNet.
  • Robustness also improves: ImageNet-A top-1 accuracy rises from 61.0% to 83.7%, while ImageNet-C mCE falls from 45.7 to 28.3 and ImageNet-P mFR from 27.8 to 12.2.

2. Noisy Student Training

Noisy Student Training extends self-training by generating pseudo labels with a clean teacher and training an equal-or-larger noisy student. Iteration, augmentation, model noise, filtering, and balancing support learning from labeled and unlabeled images.

  • The algorithm trains a teacher on labeled images, generates soft or hard pseudo labels for clean unlabeled images, and trains a noisy student on both datasets.
  • Iterative training reuses the student as a teacher to generate new pseudo labels.
  • The student is equal to or larger than the teacher, distinguishing the method from distillation that often uses a smaller student.The paper characterizes this as knowledge expansion.
  • Student noise combines RandAugment input augmentation with dropout and stochastic depth model noise.
  • Noise enforces prediction consistency across augmented images and makes the student mimic a more powerful teacher ensemble.
  • Teacher-confidence filtering removes likely out-of-domain images, while class balancing aligns unlabeled and labeled distributions.
  • Soft and hard pseudo labels both work well, with soft labels slightly better for out-of-domain unlabeled data.

3. Experiments

Experiments evaluate Noisy Student Training on ImageNet, robustness benchmarks, and adversarial attacks using EfficientNet models and large unlabeled datasets. The method improves both standard accuracy and robustness, reaching 88.4% ImageNet top-1 accuracy and substantial gains across robustness evaluations.

  • Experiment details: The unlabeled data comes from JFT, containing around 300M images whose original labels are ignored, with ImageNet validation images filtered out.Teacher-confidence filtering and class balancing produce 130M training images, including duplicated examples.
  • Experiment details: Experiments use EfficientNet baselines, including the larger and deeper EfficientNet-L2, whose training takes approximately five times as long as EfficientNet-B7.EfficientNet-L2 uses lower resolution and more parameters to accommodate large amounts of unlabeled data.
  • Experiment details: The best model uses three iterations, beginning with EfficientNet-B7 as teacher and subsequently using EfficientNet-L2 models as teachers for new students.The process repeatedly promotes the student to teacher while training on labeled and pseudo-labeled data.
  • ImageNet results: 88.4% top-1 accuracy is achieved on ImageNet, exceeding EfficientNet’s previous 85.0% by 3.4 percentage points; Noisy Student contributes +2.9% versus +0.5% from scaling.The method also exceeds the 86.4% FixRes ResNeXt-101 WSL result, which uses 3.5B weakly labeled Instagram images.
  • Robustness results: Robustness improves substantially: ImageNet-A top-1 accuracy rises from 61.0% to 83.7%, while ImageNet-C mCE falls from 45.7 to 28.3 and ImageNet-P mFR reaches 12.2 at 299×299 resolution.The robustness comparisons use models trained with 3.5B weakly labeled images as the previous state of the art.
  • Adversarial robustness results: Noisy Student Training also improves adversarial accuracy under PGD with 10 iterations at ϵ = 16, raising EfficientNet-L2 performance from 1.1% to 4.4%.The model was not optimized specifically for adversarial robustness.

4. Ablation Study

The ablations identify noise, iterative teacher–student training, and key data, model, and pseudo-label choices that improve Noisy Student performance. Iterations raise accuracy progressively, while several design choices determine how effectively students exploit unlabeled data.

  • Noise: Noise from stochastic depth, dropout, and data augmentation consistently improves student performance over removing the noise.Removing noise lowers performance, although with 130M unlabeled images the noiseless model still reaches 84.3% versus 84.0% for the supervised baseline.
  • Noise: A noised teacher lowers accuracy, supporting the use of a powerful unnoised teacher to generate pseudo labels.
  • Iterative training: 87.6% accuracy is reached after the first iteration and 88.1% after the second, with the final iteration reaching 88.4%.The final iteration uses a larger unlabeled-to-labeled batch-size ratio.
  • Design choices: A large teacher, abundant unlabeled data, and a large student model each support better performance.The ablations also find that soft pseudo labels can help out-of-domain data in certain cases.
  • Design choices: Joint training on labeled and unlabeled data outperforms pretraining on unlabeled data followed by labeled-data fine-tuning.
  • Design choices: Training from scratch can outperform teacher initialization, while initialized students still require many epochs to perform well.Data balancing is useful for small models, and a larger unlabeled-to-labeled batch-size ratio enables longer unlabeled-data training.

5. Related works

Noisy Student Training builds on self-training, robustness research, semi-supervised learning, and distillation while differing in how it uses noise, unlabeled data, and model scaling. The related work positions the method as a self-training approach aimed at improving both accuracy and robustness.

  • Self-training: Self-training uses labeled data to train a teacher, pseudo-label unlabeled data, and jointly train a student; Noisy Student adds student noise and larger-or-equal students.
  • Self-training: Prior self-training improved ResNet-50 from 76.4% to 81.2% top-1 accuracy but did not show significant robustness gains on ImageNet-A, C, and P.
  • Distillation and related methods: Unlike approaches that strengthen teachers through transformed-image ensembles, Noisy Student weakens the student with injected noise.
  • Semi-supervised learning: Co-training obtains noise from disjoint feature partitions, whereas Noisy Student injects noise into the student model for ImageNet.
  • Knowledge distillation: Knowledge distillation typically makes students smaller for compression, while Noisy Student uses unlabeled data and aims to improve the student beyond the teacher.
  • Robustness: Noisy Student improves accuracy and general robustness without directly optimizing robustness, unlike works that optimize adversarial robustness on unlabeled data.

6. Conclusion

The paper shows that unlabeled images can substantially advance state-of-the-art ImageNet accuracy and robustness without billions of weak labels. Noisy Student improves self-training by adding student noise to learn beyond the teacher’s knowledge.

  • Unlabeled images significantly advance state-of-the-art ImageNet accuracy and robustness, whereas prior weakly supervised methods required billions of weakly labeled data.
  • 88.4% accuracy is 2.9% higher than without Noisy Student Training and 2.0% better than the previous best method using an order of magnitude more weakly labeled data.
  • Noisy Student Training boosts robustness on ImageNet-A, ImageNet-C, and ImageNet-P.

A.1. Architecture Details

EfficientNet-L2 is a scaled architecture whose width and depth increase capacity for unlabeled data, while its larger size substantially increases training cost. The architecture table specifies scaling and train/test resolution factors.

  • EfficientNet-L2 training takes around five times as long as EfficientNet-B7.
  • Table 8 specifies EfficientNet width, depth, and training and testing resolutions for the paper’s models.

A.2. Ablation Studies

Ablations show that Noisy Student benefits from larger teachers and students, more unlabeled data, joint training, and careful pseudo-label choices. These effects vary with model capacity and the teacher used.

  • Unlabeled Data Size: Performance remains similar at 1/16 of the unlabeled data, or 8.1M duplicated images, but drops with further reduction.The baseline accuracy in this study is 83.2%, and models are trained without iterative training.
  • Hard Pseudo-Label vs. Soft Pseudo-Label on Out-of-domain Data: Soft pseudo labels remain robust on out-of-domain unlabeled images, whereas hard pseudo labels can hurt performance.Both label types significantly improve results for high-confidence, in-domain images; the comparison uses 1.3M images per confidence interval.
  • Student Model’s Capacity: A larger student consistently performs better with the same teacher, supporting larger student capacity as an important design choice.The comparison uses EfficientNet students ranging from B0 to B7 and teachers from B0 to B7 or an L2-trained teacher.
  • Joint Training and Data Balancing: Joint training significantly outperforms pseudo-label pretraining followed by fine-tuning, while data balancing helps smaller models more than larger ones.Pretraining only on pseudo-labeled data performs below supervised learning on labeled data, and the balancing gap narrows for EfficientNet-B2 and B3.
  • Teacher Initialization: Warm-starting from the teacher can trap the student at the teacher’s accuracy, so the authors train from scratch despite the added training time.With an 86.4% teacher, warm-starting remained at 86.4% after 210 epochs, while training from scratch reached 86.9%.

A.3. Results with a Different Architecture and Dataset

Noisy Student Training improves performance beyond EfficientNet on ResNet-50 and improves a smaller SVHN benchmark using extra unlabeled data.

  • Results with ResNet-50: 1.3% improvement over the ResNet-50 baseline shows that Noisy Student Training benefits an architecture other than EfficientNet.The experiment uses ImageNet labeled data, 130M JFT unlabeled images, and RandAugment noise.
  • Results on SVHN: 98.6% accuracy improves on the 98.1% SVHN baseline and exceeds the previous state of the art achieved by RandAugment with Wide-ResNet-28-10.The SVHN experiment uses a 73K-image core set and a 531K-image extra set as unlabeled data.

A.4. Results on YFCC100M

Noisy Student Training improves results with the public YFCC100M dataset, although performance is better with JFT, probably because JFT is larger.

  • Results on YFCC100M: Noisy Student Training produces significant improvements with YFCC100M, but achieves better performance using JFT.The experiments use the same model as teacher and student and the same hyperparameters for both unlabeled datasets.

A.5. Details of Robustness Benchmarks

The robustness benchmarks measure performance under restricted classes, corruptions, and perturbations. Noisy Student’s gains remain significant relative to a supervised baseline that also uses RandAugment.

  • ImageNet-A: ImageNet-A reports top-1 and top-5 accuracy on the 200 classes included in the benchmark.Those classes are mapped to the original ImageNet classes.
  • ImageNet-C: ImageNet-C mCE averages corruption error across corruptions, normalized using AlexNet’s error rate as the baseline.Top-1 accuracy is separately averaged across corruptions and severity levels.
  • ImageNet-P: ImageNet-P mFR averages the probability that top-1 predictions change across perturbations, using AlexNet’s flip probability as a baseline.Reported top-1 accuracy is averaged across all images in ImageNet-P.
  • Influence of RandAugment: Noisy Student improves ImageNet-C and ImageNet-P relative to a supervised EfficientNet-L2 baseline that also uses RandAugment.The authors note that overlap between RandAugment transformations and the benchmark corruptions and perturbations is small.
Loading 1911.04252v4…