Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples

Mahmoud Assran; Mathilde Caron; Ishan Misra; Piotr Bojanowski; Armand Joulin; Nicolas Ballas; Michael Rabbat

Source-linked AI summary

Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples

Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, Michael Rabbat

arXiv:2104.13963v3Apr 28, 2021cs.CVcs.AIcs.LGeess.IV

TL;DR

Semi-supervised learning needs methods that use scarce labels efficiently without the computational burden of conventional self-supervised pre-training. PAWS assigns soft pseudo-labels from labeled support samples and matches them across augmented views, reaching state-of-the-art ImageNet accuracy with much less training.

Problem
Self-supervised pre-training for semi-supervised learning requires substantial computation and does not use available labeled data during pre-training.
Method
PAWS non-parametrically compares unlabeled view representations with labeled support representations to produce soft pseudo-labels and minimizes consistency between augmented views.
Results
PAWS outperforms other semi-supervised methods across architectures and reaches 75% top-1 accuracy with 10% labels and 66% with 1% labels on ImageNet using a ResNet-50.
Takeaways & Limitations
PAWS achieves competitive semi-supervised accuracy with significantly less training and provably avoids collapsing representations.
Takeaways & Limitations
The authors leave open whether competitive representations can be learned using only instance supervision and more flexible memory representations.

Abstract

from arXiv · show

This paper proposes a novel method of learning by predicting view assignments with support samples (PAWS). The method trains a model to minimize a consistency loss, which ensures that different views of the same unlabeled instance are assigned similar pseudo-labels. The pseudo-labels are generated non-parametrically, by comparing the representations of the image views to those of a set of randomly sampled labeled images. The distance between the view representations and labeled representations is used to provide a weighting over class labels, which we interpret as a soft pseudo-label. By non-parametrically incorporating labeled samples in this way, PAWS extends the distance-metric loss used in self-supervised methods such as BYOL and SwAV to the semi-supervised setting. Despite the simplicity of the approach, PAWS outperforms other semi-supervised methods across architectures, setting a new state-of-the-art for a ResNet-50 on ImageNet trained with either 10% or 1% of the labels, reaching 75.5% and 66.5% top-1 respectively. PAWS requires 4x to 12x less training than the previous best methods.

1 Introduction

PAWS addresses semi-supervised learning by using labeled data throughout self-supervised-style training while assigning pseudo-labels non-parametrically to unlabeled views. It achieves strong ImageNet accuracy with substantially less training than prior approaches.

Semi-supervised learning remains challenging because self-supervised pre-training followed by fine-tuning requires substantial computation and does not use available labels during pre-training.
100 epochs of PAWS training take less than 8.5 hours on 64 NVIDIA V100-16G GPUs while achieving higher accuracy than previous work with fewer epochs.
PAWS uses labeled support samples throughout training to generate soft pseudo-labels for views of unlabeled images.Pseudo-labels are based on similarities between view representations and randomly sampled labeled representations.
PAWS extends the distance-metric loss used by BYOL and SwAV to semi-supervised learning through non-parametric incorporation of labeled samples.
75% top-1 accuracy is achieved with 10% labels and 66% with 1% labels on ImageNet using a ResNet-50, establishing a new state-of-the-art.The result uses 200 epochs of training, four times fewer than the previous best method.

2 Related Work

Prior semi-supervised methods combine supervised and unsupervised objectives, generate pseudo-labels through self-training, or learn metric spaces for few-shot prediction. PAWS is also related to self-supervised multi-view representation learning and its computational costs.

Semi-supervised methods commonly add an unsupervised regularizer or self-supervised pretext loss to supervised cross-entropy on labeled data.
Self-training methods generate pseudo-labels for unlabeled samples and optimize predictions against both ground-truth labels and pseudo-labels.
Teacher-student approaches use a teacher network to assign pseudo-labels that train a student network.
Few-shot methods such as Matching Networks and Prototypical Networks learn metric spaces and use differentiable nearest-neighbour classifiers with labeled support examples.
Self-supervised representation learning aligns representations of differently augmented views, but contrastive methods can require 800–1000 ImageNet pre-training epochs.

3 Methodology

PAWS learns representations from unlabeled images and a small labeled support set by matching soft pseudo-labels across augmented views. Its objective combines sharpened consistency targets with mean entropy maximization to prevent collapse and balance class usage.

PAWS uses a large unlabeled dataset and a small annotated support set during pre-training, then fine-tunes the learned representations on the support labels.
For each unlabeled image, random augmentations produce an anchor view and a positive view whose predictions are trained to agree.
A differentiable soft nearest-neighbours classifier compares each view representation with labeled support representations and outputs a soft class distribution.
The support set supplies pseudo-labels for unlabeled views rather than directly predicting support labels, distinguishing PAWS from few-shot classifiers.
Target sharpening encourages confident predictions and is sufficient to eliminate collapsing solutions in the PAWS framework.Training without sharpening can result in collapsing solutions empirically.
ME-MAX maximizes the entropy of the average sharpened prediction while individual predictions remain confident, encouraging use of the support set’s full class range.
For more than two views, PAWS sums the loss across views and uses the average prediction from the other views as the target.

4 Theoretical Guarantees

PAWS theoretically avoids trivial representation collapse under balanced support sampling and non-uniform sharpened targets. Collapse produces uniform predictions, which cannot match the sharpened targets and therefore is not stationary.

Under class-balanced support sampling and non-uniform target sharpening, collapsed representations have a nonzero gradient of the cross-entropy objective.
When all representations collapse, equal similarities to support samples make the similarity classifier output the uniform distribution over sampled classes.
Because sharpening makes the targets non-uniform, collapsed predictions differ from targets and collapse is not a stationary point of training dynamics.
The target-sharpening assumption may require a tie-breaking mechanism, although the authors report it is unnecessary in practice when sharpening starts at the beginning of training.

5 Implementation Details

PAWS pre-trains representations with labeled support samples and unlabeled image views, then evaluates them through fine-tuning or nearest-neighbour classification.

PAWS pre-trains representations before fine-tuning a classifier using only the labeled samples.The paper also evaluates the pre-trained representations directly with nearest-neighbour classification.
The pre-training setup uses LARS with momentum 0.9, weight decay 10^-6, temperature τ = 0.1, and batch-size 4096.The learning rate warms from 0.3 to 6.4 for 10 epochs, then follows cosine decay.
Each unlabeled image produces two 224 × 224 crops and six 96 × 96 crops, while each support mini-batch samples 6720 labeled images from 960 classes.The support sample comprises 7 images per class; small-crop targets average the large-crop predictions.
The encoder is a ResNet trunk with a 3-layer MLP projection head and, by default, a 2-layer MLP prediction head before cosine-similarity computation.Ablations show that PAWS also works well without the prediction head.
The evaluation protocol fine-tunes encoder and classifier weights simultaneously with labeled samples and supervised cross-entropy.Table 1 distinguishes PAWS fine-tuning from PAWS-NN, which uses nearest-neighbour classification directly on PAWS-pretrained representations.

6 Main Results

On ImageNet, PAWS outperforms comparable semi-supervised and self-supervised methods across label fractions and architectures while using substantially fewer training epochs, though some comparisons exclude distillation-based results.

Evaluation scope: The reported baseline comparisons focus on methods using the same architectures and exclude results that distill from larger teacher models.The best cited distilled ResNet-50 results are 73.9% and 77.5% top-1 for 1% and 10% labels.
Computational efficiency: 100 epochs of PAWS training take less than 8.5 hours on 64 NVIDIA V100-16G GPUs in the 1% label setting.The figure compares top-1 validation accuracy with training epochs and reports higher accuracy than previous work using fewer epochs.
Self-supervised and semi-supervised comparisons: 75% and 66% top-1 accuracy are achieved with 10% and 1% labels, respectively, after 200 PAWS training epochs.These results set a new state-of-the-art for a ResNet-50 on ImageNet in the reported label settings.
Self-supervised and semi-supervised comparisons: PAWS outperforms other self-supervised approaches while using roughly 10× fewer pre-training epochs.With 100 epochs, PAWS surpasses the reported self-supervised state of the art; with 200 epochs, it reaches the reported 75% and 66% top-1 accuracies.
Evaluation variants: PAWS-NN surpasses other self-supervised methods, while fine-tuning increases top-1 accuracy by 1–3%.Because fine-tuned PAWS consistently performs better than PAWS-NN, subsequent results report PAWS rather than PAWS-NN.
Computational efficiency: PAWS takes 8.2 hours for 100 epochs versus 49.6 hours for SwAV at 800 epochs, with +9.9% and +3.7% top-1 improvements in the 1% and 10% settings.PAWS also avoids the additional momentum encoder or memory buffer used by SimCLRv2 and BYOL.
Self-supervised and semi-supervised comparisons: PAWS exceeds MPL, the reported state-of-the-art semi-supervised method, while requiring significantly fewer training epochs.MPL simultaneously trains student and teacher networks for 800 epochs.
Architecture scaling: 69.9% and 79.0% top-1 accuracy are achieved by a 4× wider ResNet-50 with 1% and 10% labels, respectively, after 200 epochs.The reported larger-architecture results are consistent with increasing model capacity improving semi-supervised performance.

7 Ablation Study

The ablations show that PAWS generally reaches useful performance with short training, benefits from larger and more diverse support sets, and remains effective without a prediction head or with small batches.

Longer Training: More than 200 pre-training epochs is generally unnecessary across the tested ResNet architectures and label settings, yielding only marginal improvements.This pattern is reported for ResNet-50 and ResNet-50 (2×) with both 1% and 10% labels.
Learning During Pre-training: During training, cross-entropy, instance-discrimination, and classification losses decrease, while target-prediction confidence increases.The auxiliary instance-discrimination and classification losses are reported for analysis only and do not receive gradients.
Support Set: Larger support sets improve performance, while sampling more classes with fewer images per class is preferable when the number of instances is fixed.The support-set ablation uses ResNet-50 on ImageNet for 100 epochs.
Prediction Head: PAWS still converges without a prediction head, unlike self-supervised methods that collapse without one.The prediction head is marginally better omitted during PAWS pre-training.
ME-Max Regularization: ME-MAX regularization is especially helpful with 1% labeled data but provides only a marginal improvement with 10%.The regularizer maximizes the entropy of average predictions across unlabeled samples.
Small Batch Training: PAWS achieves good performance after 100 epochs with small batches on 8 NVIDIA V100-16G GPUs.The small-batch experiment uses an unsupervised batch size of 256 and disables ME-MAX regularization.

8 Discussion

PAWS combines a labeled support set with unlabeled data during pre-training and achieves competitive accuracy with substantially less training, while leaving instance-only supervision and more flexible memory representations for future work.

Method: PAWS leverages a small labeled support set during pre-training to learn from both labeled and unlabeled data.The method can also be interpreted as using an external memory represented by labeled support examples.
Results: PAWS achieves competitive semi-supervised accuracy and requires significantly less training than previous works.The paper also states that PAWS provably avoids collapsing solutions, a common challenge in self-supervised approaches.
Future Directions: Future work will investigate competitive representations using only instance supervision and more flexible memory representations.The stated practical advantage of the supervised support set is efficient learning.

A Implementation Details

PAWS uses sampled labeled support examples to produce soft nearest-neighbor pseudo-labels for augmented unlabeled views, with multicrop augmentation and standard optimization components completing the implementation.

Support Sampling: Each iteration samples a support mini-batch from labeled data to determine soft pseudo-labels for unlabeled image views.Classes are sampled first, followed by an equal number of images from each sampled class.
Projection and Prediction Heads: The projection head is a three-layer MLP, while the prediction head is a two-layer MLP with the stated hidden and output dimensions.Both heads use ReLU activations and batch normalization in the specified layers.
Fine-tuning: Fine-tuning optimizes supervised cross-entropy on the available labeled samples using the encoder and a linear classifier.The classifier is initialized with zero weights and weight decay is not used during fine-tuning.
Nearest-Neighbor Evaluation: PAWS also supports nearest-neighbor evaluation without fine-tuning by classifying representations using the available labeled training samples.The predicted class is the label with the highest probability under the similarity classifier.
Nearest-Neighbor Classifier: The soft nearest-neighbor classifier assigns larger weights to labels of representations more similar to the queried representation.Its output is a weighted average of labeled representations’ class labels.
Multi-Crop: PAWS generates two large 224 × 224 crops and six small 96 × 96 crops for each unlabeled image.Small crops use the two large crops as positive views, while each large crop uses the other large crop as its positive view.

B Comparison to Supervised Learning

PAWS is compared with fully supervised models using the same architectures, despite accessing only a small fraction of ImageNet labels, and matches fully supervised performance with wider ResNets.

Comparison: PAWS matches fully supervised learning for ResNet-50 (2×) and ResNet-50 (4×) using only 10% of ImageNet labels.The supervised baseline uses all ImageNet labels and the same architecture, with the best result selected across training and augmentation settings.
Baseline: The comparison uses supervised models trained with standard cross-entropy on the full ImageNet label set.The reported supervised baselines are taken from SimCLR and selected over specified epoch and augmentation sweeps.

C Additional Experiments — CIFAR10

On CIFAR10, PAWS is evaluated with 4000 labels and shows similar observations to ImageNet, including strong nearest-neighbour performance with less training. The experiments also examine implementation choices and representation-collapse guarantees.

C Additional Experiments — CIFAR10: The CIFAR10 setup uses multi-crop augmentation with two large views and six small views generated from each unlabeled image.Large crops are 32 × 32 and small crops are 18 × 18.
C Additional Experiments — CIFAR10: PAWS pre-training works well without a prediction head on both ImageNet and CIFAR10.The CIFAR10 encoder uses a WideResNet-28-2 trunk with a three-layer MLP projection head and no prediction head.
C Additional Experiments — CIFAR10: PAWS-NN matches the state-of-the-art on CIFAR10 with significantly less training when evaluated using 4000 labeled images as support.The evaluation uses the pretrained representations directly in a nearest-neighbour classifier rather than fine-tuning a linear classifier.
C Additional Experiments — CIFAR10: PAWS assigns soft pseudo-labels by comparing anchor and positive-view representations with labeled support samples, then minimizes their cross-entropy.The similarity classifier is a differentiable Soft Nearest Neighbours strategy.
C Additional Experiments — CIFAR10: Target sharpening prevents trivial representation collapse because collapsed representations produce high-entropy predictions while targets remain low-entropy.The theoretical guarantee identifies sharpening as sufficient to make collapsed representations non-stationary.

D.1 Semi-Supervised Prediction

The semi-supervised prediction analysis provides an alternative non-collapse guarantee when labeled samples appear in each mini-batch. This strategy is theoretical rather than experimental, because the experiments use sharpened positive-view predictions instead.

D.1 Semi-Supervised Prediction: Using a labeled sample's class label as the prediction target provides a theoretical alternative to using the positive-view prediction.The alternative is analyzed under Proposition 2.
D.1 Semi-Supervised Prediction: Each mini-batch of image views is assumed to contain at least one labeled sample.This is the stated semi-supervised image-view assumption.
D.1 Semi-Supervised Prediction: Under the stated assumptions, collapsed representations have a nonzero cross-entropy gradient and are therefore non-stationary.Proposition 2 gives this guarantee for the semi-supervised setting.
D.1 Semi-Supervised Prediction: The labeled-target strategy is not used in experiments; PAWS instead uses the sharpened positive-view prediction as the target.The paper presents Proposition 2 only as a theoretical alternative for preventing collapse.
D.1 Semi-Supervised Prediction: Adding entropy minimization is another possible non-collapse strategy, but the paper uses target sharpening because it is simpler and sufficient under Proposition 1.The entropy term would penalize the high-entropy predictions associated with collapsed representations.

E Ethical Considerations

The paper frames data-efficient recognition as a way to reduce reliance on larger models and longer training, while cautioning that computationally intensive pipelines carry environmental and access-related costs.

E Ethical Considerations: Increasing model and dataset sizes can improve image-recognition performance but may increase environmental footprints and exclusionary effects.The paper connects computationally expensive training with both environmental and participation concerns.
E Ethical Considerations: PAWS matches current state-of-the-art data-efficient image recognition using smaller models and fewer training epochs.The paper also reports that smaller-model performance is not yet saturated.
E Ethical Considerations: Greater computational effort does not always correspond linearly to larger models or better accuracy.For example, the paper contrasts a wider ResNet-50 trained for 100 epochs with a smaller ResNet-50 trained for 200 epochs and reports different top-1 accuracies.