PaperlayerView exact revision on arXiv

Source-linked AI summary

S4L: Self-Supervised Semi-Supervised Learning

Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, Lucas Beyer

arXiv:1905.03670v2cs.CVcs.LG

TL;DR

The paper addresses semi-supervised image classification when labeled data are scarce and costly. It introduces self-supervised semi-supervised learning, deriving methods that use self-supervised targets with labeled and unlabeled images. These methods are competitive with existing approaches, and their combination with semi-supervised techniques achieves state-of-the-art performance on semi-supervised ILSVRC-2012.

  • Problem

    Semi-supervised image classification seeks to learn effective classifiers from limited labeled data together with more abundant unlabeled images.

  • Method

    S4L bridges self-supervised representation learning and semi-supervised learning by deriving semi-supervised losses from self-supervised targets.

  • Results

    S4L methods perform competitively with semi-supervised methods, and combining them with existing techniques yields state-of-the-art performance on semi-supervised ILSVRC-2012.

  • Takeaways & Limitations

    Self-supervised targets can serve as the basis for semi-supervised image-classification methods and can complement existing semi-supervised techniques.

  • Takeaways & Limitations

    Results are less clear with 1 % of ILSVRC-2012 labels, where 13 labeled examples per class may make the setting resemble few-shot learning.

Abstract

from arXiv · show

This work tackles the problem of semi-supervised learning of image classifiers. Our main insight is that the field of semi-supervised learning can benefit from the quickly advancing field of self-supervised visual representation learning. Unifying these two approaches, we propose the framework of self-supervised semi-supervised learning and use it to derive two novel semi-supervised image classification methods. We demonstrate the effectiveness of these methods in comparison to both carefully tuned baselines, and existing semi-supervised learning methods. We then show that our approach and existing semi-supervised methods can be jointly trained, yielding a new state-of-the-art result on semi-supervised ILSVRC-2012 with 10% of labels.

1. Introduction

S4L addresses the scarcity and cost of labeled data by combining self-supervised representation learning with semi-supervised image classification. The resulting methods use unlabeled images alongside limited labels and can be combined with existing semi-supervised techniques.

  • Motivation: Limited labeled data constrains computer-vision systems because annotation is expensive and real-world categories can differ from benchmark datasets.The paper frames learning new concepts from only a small number of labeled examples as an important research challenge.
  • Approach: S4L bridges self-supervised and semi-supervised learning by deriving semi-supervised losses from self-supervised targets.The framework is intended to turn self-supervision methods into semi-supervised learning algorithms.
  • Results: Combining S4L methods with existing semi-supervised regularizations improves results because the approaches are complementary.The paper experimentally investigates this joint training and reports improved results.
  • Results: S4L methods outperform carefully tuned labeled-only baselines and achieve performance competitive with existing semi-supervised learning techniques.The comparison is made for natural-image classification using the paper’s semi-supervised evaluation setup.
  • Results: The paper reports new state-of-the-art performance on semi-supervised ILSVRC-2012 when S4L methods are combined with existing semi-supervised techniques.The benchmark includes settings using 10 % or 1 % of the labels.

2. Related Work

Related work spans semi-supervised methods that exploit unlabeled-data structure and self-supervised methods that learn representations through pretext tasks. The paper follows standard partial-label evaluation while targeting the more challenging ILSVRC-2012 setting.

  • Semi-supervised Learning: Semi-supervised learning uses both labeled and unlabeled samples, with methods differing in what structure they extract from unlabeled data.The standard assumption is that labeled and unlabeled samples come from the same or similar distributions.
  • Semi-supervised Learning: The standard evaluation protocol retains a portion of labels from a labeled dataset and treats the remainder as unlabeled data.The paper follows this protocol despite noting that it may not reflect realistic semi-supervised settings.
  • Semi-supervised Learning: Deep semi-supervised learning includes generative models, consistency regularization, pseudo-labeling, and conditional entropy minimization.Consistency methods compare predictions on perturbed unlabeled examples, while pseudo-labeling imputes approximate classes.
  • Evaluation Setting: Semi-supervised algorithms are usually evaluated on small datasets, whereas relatively few studies address larger and more challenging ILSVRC-2012.The paper identifies Mean Teacher as a contemporary state-of-the-art result on ILSVRC-2012 with 10 % of labels.
  • Self-supervised Learning: Self-supervised learning formulates surrogate tasks using only unsupervised data, designed so that solving them requires learning useful image representations.Examples include predicting patch arrangements, colorizing grayscale images, and predicting image-rotation angles.
  • Self-supervised Learning: Representation-focused self-supervised methods also impose constraints such as augmentation invariance or additivity in the representation space.The exemplar loss encourages invariance to heavy image augmentations, while another method relates patch and whole-image representations.

3. Methods

S4L combines supervised classification with self-supervised losses on labeled and unlabeled images, instantiated through rotation and exemplar objectives. The section also contrasts these methods with established semi-supervised baselines, including VAT, entropy minimization, and pseudo-labeling.

  • General framework: S4L combines labeled-image cross-entropy with an unsupervised loss weighted by a non-negative scalar, and can incorporate multiple unsupervised losses.The objective uses model parameters θ and applies the supervised loss to labeled data while defining Lu over unsupervised images.
  • S4L instantiations: S4L-Rotation predicts one of four rotation degrees for each transformed image, applying the rotation loss to all four rotated versions in each minibatch.The four rotations are 0°, 90°, 180°, and 270°, producing a 4-class classification problem.
  • Training losses: The self-supervised loss may be applied to labeled images as well as unlabeled images, while labeled transformed images also receive classification supervision.The paper studies experimentally whether including labeled images in the self-supervised loss affects performance.
  • S4L instantiations: S4L-Exemplar creates eight transformed instances per image and uses a batch-hard triplet loss to align views of the same image while separating views of different images.The transformations include Inception cropping, horizontal mirroring, and HSV-space color randomization.
  • Baselines: The compared semi-supervised baselines include VAT, entropy minimization combined with VAT, and pseudo-labeling.VAT encourages prediction robustness to local perturbations, EntMin encourages confident predictions, and pseudo-labeling retrains on confident predicted classes.

4. ILSVRC-2012 Experiments and Results

The experiments evaluate semi-supervised methods on ILSVRC-2012 using carefully tuned supervised baselines and controlled comparisons. S4L methods outperform tuned labeled-only baselines, are competitive with prior semi-supervised methods, and improve further when combined with them.

  • Experimental setup: The experiments use ILSVRC-2012 with class-balanced labels available for 10% or 1% of the dataset, while hyperparameters are selected on a custom training-validation split.The custom split contains 1,231,121 training images and 50,046 validation images.
  • Plain supervised learning: The tuned ResNet50v2 supervised baselines achieve 80.43% top-5 accuracy with 10% labels and 48.43% with 1% labels.The corresponding top-1 accuracies are 56.35% and 25.39%, respectively.
  • Pseudo-labeling: Pseudo-label retraining consistently improves results for both the 10% and 1% label settings.The method assigns pseudo-labels to the full dataset and retrains a ResNet50v2 from scratch on the resulting dataset.
  • Semi-supervised baselines: Adding entropy minimization consistently improves VAT performance, particularly within the evaluated semi-supervised baseline comparisons.The paper then extends the co-training setup to include the self-supervised rotation loss.
  • Main comparisons: S4L techniques outperform carefully tuned no-unlabeled-data baselines and achieve performance competitive with existing semi-supervised methods.The paper evaluates S4L-Rotation and S4L-Exemplar alongside pseudo-labeling, VAT, and VAT+EntMin.
  • Robustness: The proposed method is not sensitive to random seed or labeled-subset split, according to experiments varying both sources of randomness.The authors report these checks in Appendix B.
  • Additional capacity: 86.41% top-5 accuracy with 10% labels and 57.50% with 1% labels are achieved by a wider, more powerful S4L-Rotation model with extensive tuning.The architecture is a 2× wider ResNet152v2 trained with a larger computational budget.

5. Semi-supervised Learning is Complementary to S4L

The paper combines S4L with existing semi-supervised regularization through a three-step Mix Of All Models pipeline. This combination yields a new state-of-the-art result on ILSVRC-2012 with 10% labels.

  • Step 1: Rotation+VAT+EntMin: MOAM first jointly trains S4L-Rotation with VAT and entropy minimization in a wider model.This first step combines the self-supervised rotation loss with VAT and EntMin losses.
  • Step 2: Retraining on Pseudo Labels: The second step generates pseudo-labels by averaging predictions across five crops and four rotations, then retrains the network using all losses.The retrained network is initialized from the first-step weights and every example receives a label.
  • Step 3: Fine-tuning: The third step fine-tunes the second-step model on the original 10% labeled set only.Fine-tuning uses the original labels rather than the pseudo-labels.
  • Results: 91.23% top-5 accuracy is achieved by the final MOAM model, setting a new state-of-the-art on the 10%-label benchmark.Table 2 compares MOAM with prior methods using architectures larger than those in Table 1.
  • Results: MOAM exceeds the fully supervised baseline in the 100%-label setting by +0.87% top-5 accuracy and +1.6% top-1 accuracy.The authors describe this as promising performance in the high-data regime.

6. Transfer of Learned Representations

The paper evaluates whether S4L representations transfer to Places205 through linear logistic regression. S4L representations converge faster during evaluation and slightly outperform a fully supervised representation despite using fewer labeled data.

  • Evaluation protocol: Transfer evaluation freezes learned representations and trains a linear logistic regression classifier on Places205 features.The representation is extracted from the pre-logits layer under a protocol following prior self-supervised work.
  • Learning curves: S4L-Rotation reaches a good separating hyperplane in very few epochs, whereas the plain self-supervised rotation representation requires a much longer schedule.The comparison uses models pretrained on ILSVRC-2012 with 10% labels.
  • Transfer result: 83.3% accuracy is obtained by the MOAM representation on transfer, compared with 83.1% for a fully supervised baseline trained with ten times more labeled data.The authors describe the MOAM representation as transferring slightly better.

7. Is a Tiny Validation Set Enough?

Although semi-supervised learning typically uses a large labeled validation set for model selection, the authors find that a tiny validation set can identify the best hyperparameters comparably.

  • Using a large labeled validation set for model selection conflicts with the practical goal of semi-supervised learning, which assumes few labeled examples.
  • A 1000-image validation set is sufficient for selecting hyperparameters in the studied ILSVRC-2012 setting.The authors also evaluated a 5000-image set against the full 50 046-image validation set.
  • The best model selected on a tiny validation set is also the best model selected on the full validation set.
  • Performance scores on 1000- and 5000-image validation sets show very strong correlation with scores on the full validation set.

8. Discussion and Future Work

The paper frames S4L as a bridge between self-supervised representation learning and semi-supervised learning, with complementary methods achieving state-of-the-art performance when combined.

  • S4L turns any self-supervision method into a semi-supervised learning algorithm through a framework of self-supervised losses.
  • S4L-Rotation and S4L-Exemplar perform competitively with existing semi-supervised methods on ILSVRC-2012.
  • Combining S4L with existing semi-supervised techniques improves results because the methods are complementary.
  • MOAM, the proposed combination of S4L and existing semi-supervision techniques, achieves state-of-the-art performance.
  • Results are promising with 10 % of ILSVRC-2012 labels, but the picture is much less clear with 1 %, where only 13 labeled examples per class are available.

A. Detailed Results of the Supervised Baselines

The supervised-baseline study uses hypersweep curves to analyze hyperparameter effects, finding that weight decay and training duration matter most while larger models remain advantageous on limited data.

  • Hypersweep curves plot fully trained models sorted by accuracy to assess hyperparameter sensitivity beyond the best model.
  • The rightmost curve point identifies the best hyperparameter value, while curve separation and similarity indicate sensitivity and average robustness.
  • A 1000-image validation set is sufficient for determining the best hyperparameters, supporting a more realistic tuning protocol.
  • Weight decay and training duration are the two most important factors when training with only a fraction of ILSVRC-2012.
  • Deeper and wider models outperform shallower and thinner counterparts even with 1 % of the training data, contrary to the expectation that reducing capacity helps.
  • Color augmentation helps at 1 % of the data but does not help when added to the best models trained with 10 %.
  • Nearly all performance gains could have been obtained with a few dozen trials focused on weight decay and training epochs, rather than thousands of models.
  • The tuned baselines perform almost 20 % absolute better than previously reported baselines in the literature.

B. Randomness of S4L

The authors assess S4L robustness across random labeled subsets and random training seeds using nine runs and report top-5 accuracy as mean±standard deviation.

  • Top-5 accuracies are reported as mean±standard deviation for the nine runs.
  • Nine S4L models are trained with random image subsets and random seeds to estimate performance variability.
  • The observed standard deviation is fairly small across subsets and runs, supporting robust empirical comparisons among techniques.

C. More Results in the Transfer Setup

The transfer evaluation measures how well representations learned under different label budgets and methods support Places205 classification with linear models on frozen features. S4L and especially MOAM show strong transfer, with MOAM (full) on 10% labels slightly exceeding a wider fully supervised baseline.

  • Transfer evaluation: Table 4 evaluates each method by training linear models on frozen representations and measuring accuracy on Places205.All methods use the same plain ResNet50v2 base model, except models marked with ∗, which use a 4× wider network.
  • Transfer evaluation: The transfer protocol follows [17], but uses 30 training epochs with learning-rate decays at epochs 10 and 20 for labeled representations.The shorter schedule reflects faster convergence of representations learned with labels; 0%-label models use 520 epochs for fair comparison.
  • Results: MOAM clearly outperforms all other 10%-label models in transfer, and MOAM (full) is slightly better than the 100%-supervised 4× wider baseline.The result suggests that learning with multiple losses may produce representations that generalize better to unseen tasks.
Loading 1905.03670v2…