PaperlayerView exact revision on arXiv

Source-linked AI summary

Big Self-Supervised Models are Strong Semi-Supervised Learners

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton

arXiv:2006.10029v2cs.LGcs.CVstat.ML

TL;DR

The paper asks whether task-agnostic self-supervised pretraining can effectively use abundant unlabeled data when only a few labels are available for ImageNet. It combines large-model SimCLRv2 pretraining, supervised fine-tuning, and task-specific distillation, achieving strong accuracy with 1% and 10% of labels while transferring performance to smaller networks.

  • Problem

    Semi-supervised learning seeks to learn from few labeled examples while making effective use of abundant unlabeled data.

  • Method

    The method uses big ResNet SimCLRv2 self-supervised pretraining, supervised fine-tuning, and distillation on unlabeled examples.

  • Results

    73.9% ImageNet top-1 accuracy is achieved with ResNet-50 using 1% of labels, and 77.5% using 10% of labels.

  • Takeaways & Limitations

    Task-agnostic unlabeled-data use is a strong semi-supervised baseline on ImageNet, while task-specific distillation transfers knowledge to smaller networks.

  • Takeaways & Limitations

    The effect of deeper projection heads is smaller for bigger ResNets, possibly because those models already use relatively wide projection heads.

Abstract

from arXiv · show

One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Although this paradigm uses unlabeled data in a task-agnostic way, in contrast to common approaches to semi-supervised learning for computer vision, we show that it is surprisingly effective for semi-supervised learning on ImageNet. A key ingredient of our approach is the use of big (deep and wide) networks during pretraining and fine-tuning. We find that, the fewer the labels, the more this approach (task-agnostic use of unlabeled data) benefits from a bigger network. After fine-tuning, the big network can be further improved and distilled into a much smaller one with little loss in classification accuracy by using the unlabeled examples for a second time, but in a task-specific way. The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge. This procedure achieves 73.9% ImageNet top-1 accuracy with just 1% of the labels ($\le$13 labeled images per class) using ResNet-50, a $10\times$ improvement in label efficiency over the previous state-of-the-art. With 10% of labels, ResNet-50 trained with our method achieves 77.5% top-1 accuracy, outperforming standard supervised training with all of the labels.

1 Introduction

The paper investigates task-agnostic self-supervised pretraining followed by supervised fine-tuning for semi-supervised ImageNet learning, finding that larger models and task-specific distillation make this approach highly effective.

  • 1 Introduction: Self-supervised pretraining followed by supervised fine-tuning is investigated as a semi-supervised learning paradigm for ImageNet.The approach uses unlabeled images without class labels during pretraining and labeled examples only during fine-tuning.
  • 1 Introduction: The framework uses unlabeled data again after fine-tuning to improve task-specific predictions and distill them into a smaller network.Distillation trains a student network to mimic the fine-tuned teacher’s predictions on unlabeled examples.
  • 1 Introduction: The proposed method combines unsupervised pretraining, supervised fine-tuning, and unlabeled-data distillation using the SimCLRv2 framework.The method evaluates this three-step procedure on ImageNet with 1% and 10% of labeled images available.
  • 1 Introduction: Larger self-supervised models become more label-efficient as the number of labeled examples decreases.The paper reports that bigger models perform significantly better when fine-tuned on only a few labeled examples.
  • 1 Introduction: 73.9% ImageNet top-1 accuracy is achieved with 1% of labels using a distilled ResNet-50.With 10% of labels, the same procedure reaches 77.5% top-1 accuracy, while fully supervised ResNet-50 training with all labels reaches 76.6%.

2 Method

The framework uses unlabeled data first for task-agnostic representation learning and later for task-specific self-training or distillation. It combines SimCLRv2 pretraining, supervised fine-tuning, and teacher-guided student training, with larger models and modified projection-head fine-tuning improving performance.

  • Framework: Unlabeled data is used twice: task-agnostically for general visual representations and task-specifically for prediction refinement or model compression.The first use is unsupervised pretraining; the second trains students on teacher-imputed labels.
  • Self-supervised pretraining: SimCLRv2 improves SimCLR through larger ResNet models, a deeper projection head, and fine-tuning from an intermediate projection-head layer.The largest model is a 152-layer ResNet with 3× wider channels and selective kernels.
  • Self-supervised pretraining: Scaling from ResNet-50 to ResNet-152 (3×+SK) yields a 29% relative improvement in top-1 accuracy when fine-tuned on 1% of labeled examples.The comparison concerns model scaling within the proposed pretraining approach.
  • Fine-tuning: Fine-tuning adapts the task-agnostically pretrained network by retaining part of the MLP projection head rather than discarding it entirely.The model is fine-tuned from a middle projection-head layer instead of the projection-head input layer.
  • Self-training / knowledge distillation: Distillation uses a fine-tuned network as a fixed teacher whose predictions supervise a student on unlabeled examples without real labels.The student minimizes a distillation loss, and the student architecture may match the teacher or be smaller.
  • Self-training / knowledge distillation: When labeled examples are plentiful, the distillation loss can also be combined with ground-truth examples through a weighted combination.The paper focuses on distillation using only unlabeled examples, while describing this combination as an option.

3 Empirical Study

The empirical study evaluates SimCLRv2 on ImageNet across label fractions, model sizes, projection-head settings, and distillation targets. Larger self-supervised models improve label efficiency, while unlabeled-data distillation transfers task-specific performance to smaller networks.

  • Settings and Implementation Details: Only 1% or 10% of ImageNet labels are used for semi-supervised evaluation, while linear evaluation also measures representations with all labels.The dataset contains approximately 1.28 million images, with 12,811 or 128,116 labeled examples in the reduced-label settings.
  • Bigger Models Are More Label-Efficient: Increasing width, depth, and selective kernels improves self-supervised performance, with differences reaching 17% for fine-tuning on 1% of labeled images.The largest model, ResNet-152 (3×+SK), is only marginally better than ResNet-152 (2×+SK) despite nearly twice the parameters.
  • Bigger Models Are More Label-Efficient: 10× larger SimCLRv2 models improve label efficiency by 10×, from ResNet-50 to ResNet-152 (2×).Figure 4 compares supervised and semi-supervised top-1 accuracy across model sizes and label fractions.
  • Bigger/Deeper Projection Heads Improve Representation Learning: Deeper projection heads improve fine-tuning when the optimal middle layer is used, typically the first projection-head layer, especially with fewer labels.The benefit of deeper heads is smaller for bigger ResNets, where projection heads are also wider.
  • Distillation Using Unlabeled Data Improves Semi-Supervised Learning: Distillation using unlabeled examples improves fine-tuned models through both self-distillation and transfer from a larger teacher to a smaller student.With small labeled fractions, the distillation loss alone works almost as well as combining distillation and label losses.
  • Overall Comparison: The approach greatly improves previous semi-supervised results on ImageNet for both small and big ResNet variants.The reported comparison covers the paper’s best models against previous state-of-the-art methods.

4 Related work

Related work distinguishes task-agnostic representation learning from task-specific use of unlabeled data. The paper combines self-supervised SimCLRv2 pretraining with supervised fine-tuning and subsequent distillation.

  • Task-agnostic use of unlabeled data: Unsupervised pretraining followed by supervised fine-tuning has been widely used in natural language processing but only recently produced promising computer-vision results.The paper builds on recent contrastive-learning methods for visual representations.
  • Task-specific use of unlabeled data: Task-specific semi-supervised methods include pseudo-labeling, self-training, and label-consistency regularization.Unlike these approaches, the paper’s teacher begins with SimCLRv2 pretraining and fine-tuning, and its students may be smaller than the teacher.

5 Discussion

The paper presents a three-step semi-supervised ImageNet framework and finds that larger models help with scarce labels, while task-specific distillation transfers knowledge into compact networks. It also identifies dataset scope and limited understanding of the model-size effect as boundaries for interpretation.

  • The framework combines unsupervised pretraining, supervised fine-tuning, and distillation with unlabeled data for semi-supervised ImageNet classification.
  • Bigger models produce larger improvements with fewer labeled examples, although the authors primarily study this phenomenon on ImageNet.Similar results are also observed on CIFAR-10, while the underlying phenomenon remains insufficiently understood.
  • Task-agnostically learned representations can be distilled into a more specialized, compact network using unlabeled examples for a specific classification task.The process imputes labels with a teacher and can consistently improve performance when teacher and student architectures are similar.

6 Broader Impact

The findings may improve computer-vision accuracy where labeling is more expensive or difficult than training larger models, but they also carry social risks.

  • The approach could benefit computer-vision applications where obtaining additional labels is more expensive or difficult than training larger models.The paper gives medical imaging and agriculture as potentially beneficial application areas.
  • The approach could also contribute to harmful surveillance systems and reduce short-term income for some human labeling workers.

A When Do Bigger Models Help More?

Increasing model size improves performance across supervised and SimCLRv2-based semi-supervised settings, with larger benefits under stronger regularization or unlabeled pretraining. Parameter efficiency also matters, including through SK.

  • Bigger models improve relative top-1 performance in supervised and SimCLRv2 semi-supervised learning, with larger gains under regularization or unlabeled pretraining.The gains may diminish near a performance ceiling.
  • With SK, better top-1 accuracy can be achieved at the same parameter count, while larger models remain better overall.The study does not use group convolution and only uses 3 × 3 kernels for SK.

C The Correlation Between Linear Evaluation and Fine-tuning

Linear evaluation correlates with fine-tuning, more strongly when fine-tuning begins in the projection head’s middle layer, while projection-head depth and memory affect performance under particular settings.

  • Fine-tuning and linear evaluation show an overall linear correlation, which is stronger when fine-tuning starts from a middle projection-head layer.The correlation slope decreases as the number of labeled fine-tuning images increases.
  • Memory provides about a 1% top-1 advantage for linear evaluation and 1%-label fine-tuning, likely because the models already use a batch size of 4096.
  • With few labeled examples, a deeper projection head and fine-tuning from a middle layer improve semi-supervised performance, especially for smaller models.The comparison covers 2- and 3-layer projection heads across model sizes.

F Further Distillation Ablations

Distillation performance depends on the temperature setting, while teacher and student task performance are strongly correlated.

  • Temperature 0.1 and 1.0 perform similarly, but temperature 2.0 is significantly worse.The default is 0.1 when the teacher is fine-tuned and 1.0 otherwise.
  • Teacher task performance is strongly correlated with student task performance.

G CIFAR-10

Experiments on CIFAR-10 reproduce the ImageNet trend that larger pretrained models perform well with few labels, with further gains from fine-tuning augmentations and distillation.

  • CIFAR-10 experiments with six ResNet depths test whether the ImageNet findings extend to a smaller dataset.The models range from ResNet-18 through ResNet-200, using Selective Kernel and a width multiplier of 1× by default.
  • Big pretrained models can perform well, often better, with a few labeled examples on CIFAR-10.
  • 96.4% top-1 accuracy is the best CIFAR-10 result, achieved by ResNet-101 (+SK) and ResNet-152 (+SK).ResNet-200 (+SK) is slightly worse at 96.2%.
  • Better fine-tuning augmentations and an extra distillation step further improve the results.

H Extra Results

The extra results report top-5 accuracy for fine-tuned SimCLRv2 models across label fractions and model sizes, alongside linear-classifier and supervised baselines.

  • Top-5 accuracy is evaluated for fine-tuned SimCLRv2 models under varied label fractions and model sizes.
  • The comparison includes linear classifiers on ResNet outputs and supervised models trained from scratch using all labels.
  • Fine-tuning models with 1% and 10% labeled examples include additional nonlinear projection layers that increase parameter counts.The added count is 4M for 1× models and 17M for 2× models.
Loading 2006.10029v2…