PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta Pseudo Labels

Hieu Pham, Zihang Dai, Qizhe Xie, Minh-Thang Luong, Quoc V. Le

arXiv:2003.10580v4cs.LGstat.ML

TL;DR

Pseudo-labeling can reinforce inaccurate labels, limiting the student’s improvement over the teacher. Meta Pseudo Labels adapts the teacher using feedback from the student’s labeled-data performance, achieving 90.2% top-1 ImageNet accuracy, 1.6% above the previous state of the art.

  • Problem

    Inaccurate pseudo labels can make the student learn from inaccurate data, producing confirmation bias and limiting improvement over the teacher.

  • Method

    Meta Pseudo Labels trains the student on teacher-generated pseudo labels while updating the teacher from the student’s performance on labeled data.

  • Results

    90.2% top-1 accuracy on ImageNet is 1.6% better than the previous state of the art.

  • Takeaways & Limitations

    Consistent gains across low-resource benchmarks and ImageNet support the benefit of using student feedback to train the teacher.

  • Takeaways & Limitations

    Hard pseudo labels require a modified REINFORCE procedure to obtain the approximated teacher gradient.

Abstract

from arXiv · show

We present Meta Pseudo Labels, a semi-supervised learning method that achieves a new state-of-the-art top-1 accuracy of 90.2% on ImageNet, which is 1.6% better than the existing state-of-the-art. Like Pseudo Labels, Meta Pseudo Labels has a teacher network to generate pseudo labels on unlabeled data to teach a student network. However, unlike Pseudo Labels where the teacher is fixed, the teacher in Meta Pseudo Labels is constantly adapted by the feedback of the student's performance on the labeled dataset. As a result, the teacher generates better pseudo labels to teach the student. Our code will be available at https://github.com/google-research/google-research/tree/master/meta_pseudo_labels.

1. Introduction

Meta Pseudo Labels addresses confirmation bias in pseudo-labeling by adapting the teacher using feedback from the student's labeled-data performance. It achieves strong semi-supervised results, including 90.2% top-1 accuracy on ImageNet.

  • Pseudo-labeling can suffer confirmation bias when inaccurate pseudo labels cause the student to learn from inaccurate data.
  • Meta Pseudo Labels adapts the teacher by using feedback from the student to correct pseudo-labeling bias.
  • The student learns from teacher-annotated unlabeled minibatches while the teacher learns from the student’s labeled-data performance as a reward signal.
  • 90.2% top-1 accuracy on ImageNet is 1.6% better than the previous record of 88.6%.
  • Small-scale experiments on CIFAR-10-4K, SVHN-1K, and ImageNet-10% outperform recently proposed methods including FixMatch and Unsupervised Data Augmentation.

2. Meta Pseudo Labels

Meta Pseudo Labels turns pseudo-labeling into an alternating optimization process: the student learns from teacher-generated labels, while the teacher is updated according to student performance on labeled data. Practical training uses approximations and hard-label gradients to make this feasible at scale.

  • Unlike fixed-teacher Pseudo Labels, Meta Pseudo Labels trains the teacher along with the student and feeds student performance back to the teacher.
  • Pseudo Labels: Pseudo Labels trains a student on unlabeled data using pseudo targets produced by a fixed, pretrained teacher.
  • Meta objective: The teacher parameter is optimized because the student’s labeled-data loss depends on teacher-generated pseudo targets.
  • Practical approximation: A one-step student gradient update approximates the otherwise multi-step optimization needed to construct the practical teacher objective.
  • Practical approximation: Hard pseudo labels reduce computational-graph size for large-scale experiments but require a modified REINFORCE method to estimate the teacher gradient.
  • Optimization procedure: Training alternates between updating the student with pseudo-labeled unlabeled data and updating the teacher using the reused student update.
  • Implementation: The implementation additionally trains the teacher with supervised and UDA semi-supervised objectives, and can finetune the converged student on labeled data.

3. Small Scale Experiments

Small-scale experiments examine Meta Pseudo Labels on TwoMoon and three semi-supervised benchmarks, using controlled architecture and baseline comparisons. The method improves over UDA and achieves strong results against same-architecture and representative baselines.

  • Experiment scope: The experiments cover a visualized TwoMoon study and benchmark comparisons on CIFAR-10-4K, SVHN-1K, and ImageNet-10%.The study also includes experiments with standard ResNet architectures.
  • TwoMoon experiment: Only 6 of 2,000 TwoMoon examples are labeled, with the remaining examples unlabeled and separated using the clustering assumption.The dataset contains two clusters of 1,000 examples each, with 3 labeled examples per cluster.
  • TwoMoon experiment: Meta Pseudo Labels has a much higher success rate of finding the correct TwoMoon classifier than Supervised Learning and Pseudo Labels.In the described outcome, Pseudo Labels misclassifies half the data, while Meta Pseudo Labels finds a good classifier using student-loss feedback.
  • Benchmark setup: The benchmark comparisons use WideResNet-28-2 for CIFAR-10-4K and SVHN-1K and ResNet-50 for ImageNet-10%, with matching architectures for fair comparisons.The experimental setup also uses the same protocols for the direct Supervised Learning and UDA baselines.
  • Benchmark results: Meta Pseudo Labels improves ImageNet-10% top-1 accuracy over the UDA teacher from 68.07% to 73.89%.The paper reports improvement over UDA under strictly fair comparisons and identifies the ImageNet gain as particularly significant.
  • Benchmark results: Meta Pseudo Labels achieves the best reported accuracies among the same model architectures on CIFAR-10-4K, SVHN-1K, and ImageNet-10%.The paper reports almost 10% relative error reduction on CIFAR-10-4K and SVHN-1K and a 2.19% top-1 accuracy advantage over SimCLR on ImageNet-10%.
  • Comparison scope: Some higher reported benchmark results use larger models, stronger regularization, or additional distillation procedures than the compared setup.The paper notes that these enhancements could also be applied to Meta Pseudo Labels.
  • ResNet-50 experiment: On ImageNet with ResNet-50, Meta Pseudo Labels raises top-1 accuracy from 76.9% to 83.2%, outperforming UDA and Noisy Student.It also outperforms Billion-scale SSL in top-1 accuracy.

4. Large Scale Experiment: Pushing the Limits of ImageNet Accuracy

The large-scale experiment applies Meta Pseudo Labels to EfficientNet architectures and large labeled and unlabeled datasets, requiring specialized parallelism for training. It achieves state-of-the-art ImageNet performance and also performs strongly on ImageNet-ReaL.

  • Setup: The experiment uses EfficientNet-L2 with ImageNet as labeled data and JFT as unlabeled data.EfficientNet-L2 is selected for its higher capacity and prior ImageNet performance; the JFT data are used as unlabeled images.
  • Training infrastructure: A hybrid model-data parallelism framework is used because storing both teacher and student networks exceeds accelerator memory.Training runs across 2,048 TPUv3 cores with synchronized data parallelism and within-replica model parallelism.
  • Training infrastructure: Training lasts 1 million steps, followed by 20,000 labeled-data finetuning steps.The reported training time is about 11 days for EfficientNet-L2 and 10 days for EfficientNet-B6-Wide.
  • Results: 90.2% top-1 accuracy on ImageNet establishes a new state-of-the-art, improving 1.6% over the previous record.The improvement is reported relative to the previous state-of-the-art and is described as significant compared with recent gains.
  • Results: 91.02% Precision@1 on ImageNet-ReaL is 0.4% better than Vision Transformer.The additional evaluation is used to test whether the model simply overfits the ImageNet ILSVRC 2012 validation set.
  • Lite variant: Reduced Meta Pseudo Labels reaches 86.9% top-1 accuracy with EfficientNet-B7 while avoiding two networks in memory.This lite version uses ImageNet labels and YFCC100M unlabeled data instead of proprietary JFT data.

5. Related Works

The related-work discussion positions Meta Pseudo Labels against self-training, other semi-supervised objectives, distillation, and bi-level optimization. Its distinguishing design uses labeled-data performance feedback to adapt the teacher while keeping the student trained from pseudo labels.

  • Pseudo Labels: Vanilla Pseudo Labels keep a pretrained teacher fixed, whereas Meta Pseudo Labels adapts the teacher using the student’s labeled-data performance.The paper identifies inaccurate pseudo labels and confirmation bias as the motivation for teacher adaptation.
  • Other SSL approaches: Meta Pseudo Labels differs from common semi-supervised methods because the student never learns directly from labeled data.The paper states that this design helps avoid overfitting when labeled data are limited.
  • Other SSL approaches: The teacher’s feedback from student performance on labeled data is presented as a novel use of labeled data.Other approaches commonly combine supervised and unsupervised losses in a single model objective.
  • Knowledge distillation: The teacher uses softmax predictions as soft labels, connecting the method to knowledge distillation and label-smoothing techniques.The discussion distinguishes these manually designed or distilled soft labels from the method’s feedback-based teacher adaptation.
  • Bi-level optimization: The teacher update is derived from a bi-level optimization problem related to methods that optimize a model’s learning process.The paper relates this formulation to learning-rate schedules, architecture design, and correction of wrong training labels.

6. Conclusion

Meta Pseudo Labels trains the student from teacher-generated pseudo labels and updates the teacher from the student’s performance. Across low-resource benchmarks and ImageNet, the method reports consistent gains, including a 90.2% ImageNet top-1 accuracy.

  • Method: Meta Pseudo Labels updates the student from pseudo-labeled data and the teacher from the student’s performance.These two updates define the method’s learning process.
  • Results: Experiments on CIFAR-10-4K, SVHN-1K, and ImageNet-10% outperform many existing semi-supervised learning methods.The conclusion summarizes these as standard low-resource benchmarks.
  • Results: 90.2% top-1 accuracy on ImageNet is 1.6% better than the previous state-of-the-art.The conclusion presents this result as evidence that the method scales to large problems.
  • Takeaway: The consistent gains are reported to confirm the benefit of student feedback to the teacher.This conclusion connects the benchmark improvements to the method’s feedback mechanism.

A. Derivation of the Teacher’s Update Rule

The derivation formalizes teacher learning as a gradient through the student’s expected update: pseudo labels update the student, and labeled-data performance supplies the teacher’s feedback. The implementation combines this feedback with the teacher’s labeled and UDA gradients.

  • Pipeline: The teacher samples pseudo labels for unlabeled examples, and the student updates its parameters using those sampled labels.The student’s expected updated parameter is taken over the teacher’s pseudo-label distribution.
  • Pipeline: The teacher is updated to minimize the student’s cross-entropy on labeled data after the student’s pseudo-label-driven update.This creates the bi-level dependency underlying the teacher’s feedback signal.
  • Gradient derivation: The derivation applies the chain rule to propagate labeled-data performance through the student update into the teacher parameters.The first factor is computed by back-propagation, while subsequent terms account for the teacher’s influence through pseudo labels.
  • Gradient derivation: REINFORCE is used because the student-feedback term depends on teacher parameters through the sampled pseudo label.The resulting expression includes the gradient of the pseudo-label log probability with respect to teacher parameters.
  • Approximation: Monte Carlo approximation uses sampled pseudo labels to estimate the expected student update and the teacher gradient.The student update is θ′_S = θ_S − η_S · ∇_θS CE(ŷ_u, S(x_u; θ_S)).
  • Algorithm: The extended objective applies UDA to the teacher while the student continues learning only from teacher-provided pseudo-labeled data.Algorithm 1 includes separate teacher gradients from student feedback, labeled data, and the UDA loss.

C.1. Dataset Splits

The experiments define dataset splits, augmentation modifications, optimization settings, and controlled analyses of Meta Pseudo Labels as a supervised-learning regularizer.

  • Dataset splits: CIFAR-10-4K uses 4,000 labeled images after removing the first 5,000 images reserved for validation.
  • Dataset splits: ImageNet data subsets use successive shard counts for 10%, 20%, and larger labeled-data fractions, with the last 20 shards reserved for hyper-parameter tuning.
  • Augmentation: For SVHN, rotations and horizontal translations are removed from RandAugment because they create incorrect digit invariances.
  • Optimization: Training uses Nesterov Momentum for standard ResNet-based experiments and RMSProp with exponential decay for EfficientNet-L2 and EfficientNet-B6-Wide ImageNet+JFT experiments.
  • Component analysis: On ImageNet-10%, UDA improves over Supervised+MPL+Finetune by 6.05% top-1 accuracy, while UDA+MPL adds 5.25% over UDA.
  • Regularization analysis: Meta Pseudo Labels can act as adaptive Label Smoothing when labeled data is also treated as unlabeled data, because its teacher produces adaptive soft labels.

D.3. Meta Pseudo Labels Is a Mechanism to Addresses the Confirmation Bias of Pseudo Labels

Meta Pseudo Labels provides evidence that student feedback can address confirmation bias by changing teacher and student training dynamics relative to supervised learning.

  • The observed training-accuracy patterns are presented as empirical evidence that Meta Pseudo Labels helps address teacher confirmation bias.
  • On CIFAR-10-4K, both Meta Pseudo Labels teacher and student converge much more slowly than the corresponding supervised model.
  • On ImageNet-10%, the Meta Pseudo Labels student has low training accuracy while the teacher’s accuracy rises faster than supervised learning.

D.4. Meta Pseudo Labels with Different Training Techniques for the Teacher

Meta Pseudo Labels remains effective when the teacher is trained with different techniques, including Pseudo Labels, Mixup, and RandAugment.

  • Meta Pseudo Labels improves over teachers trained with Pseudo Labels, Mixup, and RandAugment on CIFAR-10-4K.
  • The results indicate that Meta Pseudo Labels works with all tested teacher-training techniques.
  • The experiments suggest that better teacher-training techniques tend to produce better students.

D.5. Meta Pseudo Labels with Different Amounts of Labeled Data

Meta Pseudo Labels delivers larger gains when labeled data is scarce, but its improvement plateaus as the amount of labeled data increases.

  • Meta Pseudo Labels is evaluated with 10%, 20%, 40%, 80%, and 100% of ImageNet’s labeled examples.
  • The method delivers substantial gains with less labeled data compared with supervised learning and RandAugment.
  • Meta Pseudo Labels’ gains plateau as more labeled data becomes available.

E. Results with An Economical Version of Meta Pseudo Labels

Reduced Meta Pseudo Labels addresses Meta Pseudo Labels’ memory demands by precomputing teacher targets and training a small reduced teacher with the student. Across large labeled-set experiments, it outperforms NoisyStudent on CIFAR-10 and ImageNet while matching it on SVHN, with gains depending on unlabeled-data quality.

  • Meta Pseudo Labels exceeds 16G of accelerator memory for large architectures because it stores both teacher and student models.Hybrid data-model parallelism enables these models but makes training slow and expensive.
  • Reduced Meta Pseudo Labels precomputes target distributions with a converged large teacher before loading the student, then trains a small reduced teacher alongside it.The reduced teacher can be a small multilayer perceptron, avoiding the original method’s large memory footprint.
  • The economical method evaluates large labeled sets for CIFAR-10, SVHN, and ImageNet using out-of-domain unlabeled data for CIFAR-10 and ImageNet.Reduced Meta Pseudo Labels is used because its memory footprint permits these large-scale experiments.
  • Reduced Meta Pseudo Labels outperforms NoisyStudent on CIFAR-10 and ImageNet and is on-par with NoisyStudent on SVHN.The comparison uses NoisyStudent because the methods differ only in whether the teacher adapts to the student’s learning state.
  • 86.87% top-1 accuracy on ImageNet is 1.06% better than NoisyStudent, while CIFAR-10 improves 0.34% in accuracy, corresponding to a 19% error reduction.ImageNet uses EfficientNet-B7; CIFAR-10 results are reported against the NoisyStudent baseline.
  • Reduced Meta Pseudo Labels gains less on SVHN because NoisyStudent is already highly accurate and its unlabeled images are high-quality, unlike noisier ImageNet data.The paper states that low-quality or out-of-domain ImageNet images make adaptive adjustment more crucial for student performance.
Loading 2003.10580v4…