PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Pre-training and Self-training

Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D. Cubuk, Quoc V. Le

arXiv:2006.06882v2cs.CVcs.LGstat.ML

TL;DR

The paper asks whether self-training can use additional data more effectively than ImageNet pre-training, especially on COCO where pre-training may fail. It evaluates both approaches across data regimes, augmentation strengths, tasks, and data sources, finding self-training broadly helpful and able to improve pretrained models.

  • Problem

    ImageNet pre-training is widely used in computer vision, yet evidence shows limited COCO detection benefits, motivating comparison with self-training for exploiting additional data.

  • Method

    The study compares supervised and self-supervised ImageNet pre-training with self-training across labeled-data amounts, augmentation strengths, tasks, architectures, and unlabeled data sources.

  • Results

    Self-training helps across low- and high-data regimes and augmentation strengths, improves pre-trained models, and reaches 54.3AP on COCO and 90.5AP on PASCAL segmentation.

  • Takeaways & Limitations

    Self-training is general and flexible across tasks, architectures, data sources, and settings where pre-training either fails or succeeds.

  • Takeaways & Limitations

    The paper does not study targeted pre-training in depth because it is expensive, not scalable, and can fail even when source and target tasks match.

Abstract

from arXiv · show

Pre-training is a dominant paradigm in computer vision. For example, supervised ImageNet pre-training is commonly used to initialize the backbones of object detection and segmentation models. He et al., however, show a surprising result that ImageNet pre-training has limited impact on COCO object detection. Here we investigate self-training as another method to utilize additional data on the same setup and contrast it against ImageNet pre-training. Our study reveals the generality and flexibility of self-training with three additional insights: 1) stronger data augmentation and more labeled data further diminish the value of pre-training, 2) unlike pre-training, self-training is always helpful when using stronger data augmentation, in both low-data and high-data regimes, and 3) in the case that pre-training is helpful, self-training improves upon pre-training. For example, on the COCO object detection dataset, pre-training benefits when we use one fifth of the labeled data, and hurts accuracy when we use all labeled data. Self-training, on the other hand, shows positive improvements from +1.3 to +3.4AP across all dataset sizes. In other words, self-training works well exactly on the same setup that pre-training does not work (using ImageNet to help COCO). On the PASCAL segmentation dataset, which is a much smaller dataset than COCO, though pre-training does help significantly, self-training improves upon the pre-trained model. On COCO object detection, we achieve 54.3AP, an improvement of +1.5AP over the strongest SpineNet model. On PASCAL segmentation, we achieve 90.5 mIOU, an improvement of +1.5% mIOU over the previous state-of-the-art result by DeepLabv3+.

1 Introduction

The paper compares self-training with ImageNet pre-training for using additional data in COCO detection and related vision tasks. Self-training remains helpful under stronger augmentation and larger labeled-data regimes where pre-training can fail, while also improving pretrained models when pre-training helps.

  • ImageNet pre-training does not improve COCO accuracy, challenging its common use for initializing detection and segmentation backbones.
  • Self-training generates pseudo labels on ImageNet with a COCO-trained detector, then jointly trains a new model on human and pseudo labels.
  • -1.0AP: the strongest data augmentation significantly hurts pre-training accuracy, whereas self-training under the same augmentation yields +1.3AP.
  • Self-supervised SimCLR pre-training performs similarly to supervised ImageNet pre-training, and both hurt COCO performance with high data and strong augmentation.
  • +1.3% mIOU: self-training improves a pretrained model on PASCAL segmentation even when ImageNet pre-training itself helps.
  • 54.3AP: self-training on COCO with Open Images data exceeds the strongest SpineNet model by +1.5AP, while PASCAL segmentation reaches 90.5AP.

2 Related Work

Prior work established that ImageNet pre-training is widespread but can fail across tasks, especially under strong regularization or augmentation. This paper extends the analysis across augmentation strengths, pre-training types, checkpoint qualities, and targeted-pre-training boundaries.

  • ImageNet pre-training is widely used, but prior studies report limited or negative benefits for COCO detection, CityScapes segmentation, and medical image classification.
  • The paper studies pre-training more broadly using stronger augmentation, supervised and self-supervised methods, and checkpoints with different qualities.
  • Targeted pre-training is not studied in depth because it is expensive, not scalable, and can still hurt performance even when source and target tasks match.
  • Self-training is positioned as a scalable and general approach with prior applications in image classification, machine translation, and speech recognition.

3 Methodology

The methodology varies augmentation strength and pre-trained checkpoint quality while evaluating pre-training and self-training across detection and segmentation. Experiments use teacher-generated pseudo labels and multiple labeled or unlabeled datasets.

  • Data Augmentation: Four increasingly strong augmentation policies—FlipCrop, AutoAugment, higher-jitter AutoAugment, and higher-jitter RandAugment—support controlled comparisons.They are named Augment-S1 through Augment-S4.
  • Pre-training: Two EfficientNet-B7 ImageNet checkpoints isolate pre-training quality while holding architecture constant: 84.5% and 86.9% top-1 accuracy.The checkpoints are trained with AutoAugment and Noisy Student, respectively.
  • Self-training: Self-training trains a teacher on labeled data, generates pseudo labels on unlabeled data, and jointly trains a student on human and pseudo labels.
  • Object Detection: COCO detection uses 118k supervised images with ImageNet and OpenImages as unlabeled sources, using RetinaNet with an EfficientNet-B7 backbone.
  • Semantic Segmentation: PASCAL segmentation uses 1.5k supervised images and tests augmented PASCAL, COCO, and ImageNet as unlabeled sources with NAS-FPN backbones.

4 Experiments

Experiments show that stronger augmentation and more labeled COCO data diminish or reverse the value of ImageNet pre-training, whereas self-training improves performance across augmentation strengths and dataset sizes. Self-training also remains beneficial when combined with pre-training and yields strong COCO and PASCAL results.

  • The effects of augmentation and labeled dataset size on pre-training: Pre-training helps with standard augmentation but diminishes as augmentation strength increases, ultimately hurting performance by -1.0 AP.The experiments use COCO object detection and vary both augmentation strength and pre-trained checkpoint quality.
  • The effects of augmentation and labeled dataset size on pre-training: Pre-training is helpful in the 20% labeled-data regime but neutral or harmful with high COCO dataset sizes.Checkpoint quality correlates with final performance in the low-data regime, where ImageNet++ performs best.
  • The effects of augmentation and labeled dataset size on self-training: More than +1.3AP self-training gain occurs under strong augmentation, where pre-training hurts by -1.0AP.The benefit appears on the full COCO dataset across Augment-S2, Augment-S3, and Augment-S4, with similar results on ResNet-101.
  • The effects of augmentation and labeled dataset size on self-training: Self-training improves object detectors across all labeled dataset sizes and model initializations, including the 100% regime where pre-training hurts.In the 20% regime, self-training provides a +3.4AP gain over Rand Init, exceeding the +2.6AP gain from ImageNet Init.
  • The effects of augmentation and labeled dataset size on self-training: Combining ImageNet pre-training and self-training produces additive gains across dataset sizes, including an additional +2.7AP in the 20% regime.The additive benefit is observed even when pre-training and self-training use the same ImageNet data source.
  • Self-supervised pre-training also hurts when self-training helps in high data/strong augmentation regimes: SimCLR pre-training hurts COCO performance by -0.7AP, while self-training improves performance by +0.8AP in the same high-data, strong-augmentation setting.Both supervised and self-supervised pre-trained models decrease performance relative to random initialization.
  • Exploring the limits of self-training and pre-training: Self-training reaches 54.3AP on COCO, improving by +1.5AP over the strongest 52.8AP SpineNet model.The COCO self-training experiment uses SpineNet models and Open Images Dataset as unlabeled data.
  • Exploring the limits of self-training and pre-training: Self-training reaches 90.5% mIOU on PASCAL VOC 2012, exceeding the previous 89% mIOU state of the art.The result uses ImageNet and PASCAL segmentation data, while the method achieves 41.5% mIOU without pre-training.

5 Discussion

The discussion argues that self-training is more task-aligned, scalable, and broadly useful than pre-training across the studied settings, while pre-training remains valuable for faster fine-tuning and low-data applications.

  • The benefit of joint-training: +3.3AP is achieved by combining pre-training, joint-training, and self-training on the same ImageNet data source.The reported gains are additive in the comparison on COCO.
  • The importance of task alignment: Self-training can outperform pre-training when ImageNet labels are un-targeted or additional labels are noisy, because pseudo labels align with the task of interest.On PASCAL, adding noisy labels hurts under strong augmentation, whereas using the same data for self-training improves accuracy.
  • Limitations: Pre-training remains useful because fine-tuning is faster, with speedups from 1.3x to 8x, and because low-data applications may require strong pretrained models.The paper specifically identifies PASCAL segmentation as a low-data setting where good pretrained models are needed.
  • Limitations: Self-training requires more compute than fine-tuning a pretrained model, limiting its practical advantage despite its broader experimental effectiveness.The discussion frames compute cost as a limitation of current self-training techniques.
  • The scalability, generality and flexibility of self-training: Self-training improves performance whether pre-training succeeds or fails and remains effective across data regimes, augmentation strengths, architectures, data sources, and tasks.The paper reports effectiveness for object detection and segmentation using multiple architectures and datasets.

Broader and Social Impact

The paper presents self-training as a computer-vision method with potential relevance beyond the studied applications, while distinguishing it from related pseudo-label and consistency-training approaches.

  • Broader and Social Impact: Self-training is studied for object detection and segmentation and is presented as potentially applicable to speech recognition, NLP, bioinformatics, and other fields.The datasets are described as generic and publicly available rather than tied to a specific application.
  • Broader and Social Impact: Self-training differs from consistency training by using two models, and this work applies it to detection and segmentation rather than only image classification.The passage also relates the method to pseudo labels and consistency training.

B Loss Normalization Analysis

The loss-normalization analysis addresses instability in self-training and shows that its weighting parameter should be tuned to augmentation strength and training duration rather than fixed universally.

  • B Loss Normalization Analysis: Loss Normalization rescales human and pseudo-label losses using their moving averages to stabilize self-training as α varies.The moving averages use an exponential decay rate of 0.9997.
  • B Loss Normalization Analysis: Loss Normalization performs better in almost all tested settings and helps avoid instability when α is large.The experiments vary augmentation strength, training iterations, learning rate, and α on COCO detection with ImageNet self-training.
  • B Loss Normalization Analysis: The optimal α decreases as augmentation strength increases and increases as training iterations increase.The study evaluates multiple α values across augmentation and training-duration settings.
  • B Loss Normalization Analysis: Initializing the student from teacher weights produces slightly better performance than random initialization, with gains of 0.3-0.4 AP across four augmentation regimes.The comparison is summarized in Table 10.

D Further Study of Augmentation, Supervised Dataset Size, and Pre-trained Model Quality

Further experiments show that stronger augmentation and more supervised data reduce the value of pre-training, while checkpoint quality matters most in the small-data PASCAL setting.

  • D Further Study of Augmentation, Supervised Dataset Size, and Pre-trained Model Quality: Stronger augmentation and more supervised data diminish the value of pre-training, and stronger augmentation can make pre-training hurt performance.Figure 3 examines the interaction among augmentation strength, COCO size, and checkpoint quality.
  • D Further Study of Augmentation, Supervised Dataset Size, and Pre-trained Model Quality: Checkpoint quality significantly improves supervised PASCAL segmentation results when training uses the 1.5k-image PASCAL train dataset.The gain from checkpoint quality diminishes as augmentation strength increases.
  • D Further Study of Augmentation, Supervised Dataset Size, and Pre-trained Model Quality: Performance on PASCAL VOC is correlated with the performance of the ImageNet checkpoint used for initialization.This correlation is reported alongside the effects of checkpoint quality and augmentation strength.

E ResNet-101 Self-training Performance on COCO

ResNet-101 self-training improves COCO object detection across all four augmentation strengths, with gains over supervised training under the same protocol.

  • Self-training improves COCO performance across all four augmentation strengths with a ResNet-101 backbone.The comparison uses RetinaNet and the same training protocol as the EfficientNet experiments, with minor stability-related changes.

F The Effects of Unlabeled Data Sources on Self-Training

Self-training performance depends on the source of unlabeled data. On PASCAL segmentation, changing the source from ImageNet to COCO improves results, while the smaller in-domain PASCAL aug set performs best.

  • The source comparison evaluates different self-training datasets under varying augmentation strengths on COCO object detection.
  • Changing the additional data source from ImageNet to COCO improves PASCAL segmentation performance across all augmentation strengths.
  • PASCAL aug is the best self-training dataset, outperforming COCO despite containing only about 9k images versus COCO’s 240k.

G Visualization of Pseudo Labels in Self-training

The visualizations show pseudo labels generated on PASCAL aug and ImageNet examples, including cases where pseudo labels outperform noisy human annotations or map absent ImageNet classes to PASCAL classes.

  • Selected PASCAL aug examples show pseudo labels that are more accurate than noisy human labels.
  • ImageNet classes absent from PASCAL VOC can be predicted as one of PASCAL’s 20 classes, such as saw and lizard being predicted as bird.
  • Although these pseudo labels are noisy, they still improve the student model’s accuracy.
  • ImageNet visualizations contain pseudo segmentation masks generated on randomly selected images.

H Optimal Model Training Iterations and Alpha Weighting

The experiments select training iterations and self-training loss weights by validation performance, with reported settings varying across setups, models, augmentation strengths, and datasets.

  • Training continues until validation performance no longer improves, and self-training searches alpha values from 0.25 to 3.0.The listed iteration and alpha settings are intended to support reproducibility.
  • 41.9 COCO AP is achieved with random initialization and self-training, compared with 41.1 AP from random initialization without self-training.
  • 54.3 AP is achieved by SpineNet-190 with self-training, compared with 52.8 AP for the corresponding SpineNet-190 model.
  • 90.5 test mIOU is achieved by EfficientNet-L2 with self-training from an ImageNet++ initialization.
  • Self-training improves results across all four augmentation strengths for EfficientNet-B7 with either teacher or random initialization.
  • On COCO, self-training with ImageNet or OID improves performance over supervised training across Augment-S1 through Augment-S4.
Loading 2006.06882v2…