Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples

• Semi-supervised learning remains challenging because self-supervised pre-training followed by fine-tuning requires substantial computation and does not use available labels during pre-training.Source: §1 · p. 1
• 100 epochs of PAWS training take less than 8.5 hours on 64 NVIDIA V100-16G GPUs while achieving higher accuracy than previous work with fewer epochs.Source: Figure 1 · p. 1
• PAWS uses labeled support samples throughout training to generate soft pseudo-labels for views of unlabeled images.Pseudo-labels are based on similarities between view representations and randomly sampled labeled representations.Source: §1 · p. 1
• PAWS extends the distance-metric loss used by BYOL and SwAV to semi-supervised learning through non-parametric incorporation of labeled samples.Source: §1 · p. 1
• 75% top-1 accuracy is achieved with 10% labels and 66% with 1% labels on ImageNet using a ResNet-50, establishing a new state-of-the-art.The result uses 200 epochs of training, four times fewer than the previous best method.Source: §1 · p. 2

• Semi-supervised methods commonly add an unsupervised regularizer or self-supervised pretext loss to supervised cross-entropy on labeled data.Source: §2 · p. 2
• Self-training methods generate pseudo-labels for unlabeled samples and optimize predictions against both ground-truth labels and pseudo-labels.Source: §2 · p. 2
• Teacher-student approaches use a teacher network to assign pseudo-labels that train a student network.Source: §2 · p. 2
• Few-shot methods such as Matching Networks and Prototypical Networks learn metric spaces and use differentiable nearest-neighbour classifiers with labeled support examples.Source: §2 · p. 2
• Self-supervised representation learning aligns representations of differently augmented views, but contrastive methods can require 800–1000 ImageNet pre-training epochs.Source: §2 · p. 2

• PAWS uses a large unlabeled dataset and a small annotated support set during pre-training, then fine-tunes the learned representations on the support labels.Source: §3 · p. 3
• For each unlabeled image, random augmentations produce an anchor view and a positive view whose predictions are trained to agree.Source: §3.1 · p. 3
• A differentiable soft nearest-neighbours classifier compares each view representation with labeled support representations and outputs a soft class distribution.Source: §3.1 · p. 3, §3.2 · p. 3
• The support set supplies pseudo-labels for unlabeled views rather than directly predicting support labels, distinguishing PAWS from few-shot classifiers.Source: §3.1 · p. 3
• Target sharpening encourages confident predictions and is sufficient to eliminate collapsing solutions in the PAWS framework.Training without sharpening can result in collapsing solutions empirically.Source: §3.2 · p. 4
• ME-MAX maximizes the entropy of the average sharpened prediction while individual predictions remain confident, encouraging use of the support set’s full class range.Source: §3.2 · p. 4
• For more than two views, PAWS sums the loss across views and uses the average prediction from the other views as the target.Source: §3.2 · p. 4

• Under class-balanced support sampling and non-uniform target sharpening, collapsed representations have a nonzero gradient of the cross-entropy objective.Source: §4 · p. 5
• When all representations collapse, equal similarities to support samples make the similarity classifier output the uniform distribution over sampled classes.Source: §4 · p. 5
• Because sharpening makes the targets non-uniform, collapsed predictions differ from targets and collapse is not a stationary point of training dynamics.Source: §4 · p. 5
• The target-sharpening assumption may require a tie-breaking mechanism, although the authors report it is unnecessary in practice when sharpening starts at the beginning of training.Source: §4 · p. 5

• PAWS pre-trains representations before fine-tuning a classifier using only the labeled samples.The paper also evaluates the pre-trained representations directly with nearest-neighbour classification.Source: §5 · p. 5
• The pre-training setup uses LARS with momentum 0.9, weight decay 10^-6, temperature τ = 0.1, and batch-size 4096.The learning rate warms from 0.3 to 6.4 for 10 epochs, then follows cosine decay.Source: §5 · p. 5
• Each unlabeled image produces two 224 × 224 crops and six 96 × 96 crops, while each support mini-batch samples 6720 labeled images from 960 classes.The support sample comprises 7 images per class; small-crop targets average the large-crop predictions.Source: §5 · p. 5
• The encoder is a ResNet trunk with a 3-layer MLP projection head and, by default, a 2-layer MLP prediction head before cosine-similarity computation.Ablations show that PAWS also works well without the prediction head.Source: §5 · p. 5
• The evaluation protocol fine-tunes encoder and classifier weights simultaneously with labeled samples and supervised cross-entropy.Table 1 distinguishes PAWS fine-tuning from PAWS-NN, which uses nearest-neighbour classification directly on PAWS-pretrained representations.Source: §5 · p. 5, Table 1 · p. 6

Evaluation scope

• The reported baseline comparisons focus on methods using the same architectures and exclude results that distill from larger teacher models.The best cited distilled ResNet-50 results are 73.9% and 77.5% top-1 for 1% and 10% labels.Source: §6 · p. 6

Computational efficiency

• 100 epochs of PAWS training take less than 8.5 hours on 64 NVIDIA V100-16G GPUs in the 1% label setting.The figure compares top-1 validation accuracy with training epochs and reports higher accuracy than previous work using fewer epochs.Source: Figure 3 · p. 6

Self-supervised and semi-supervised comparisons

• 75% and 66% top-1 accuracy are achieved with 10% and 1% labels, respectively, after 200 PAWS training epochs.These results set a new state-of-the-art for a ResNet-50 on ImageNet in the reported label settings.Source: §6 · p. 6
• PAWS outperforms other self-supervised approaches while using roughly 10× fewer pre-training epochs.With 100 epochs, PAWS surpasses the reported self-supervised state of the art; with 200 epochs, it reaches the reported 75% and 66% top-1 accuracies.Source: §6 · pp. 6–7

Evaluation variants

• PAWS-NN surpasses other self-supervised methods, while fine-tuning increases top-1 accuracy by 1–3%.Because fine-tuned PAWS consistently performs better than PAWS-NN, subsequent results report PAWS rather than PAWS-NN.Source: §6 · p. 7

Computational efficiency

• PAWS takes 8.2 hours for 100 epochs versus 49.6 hours for SwAV at 800 epochs, with +9.9% and +3.7% top-1 improvements in the 1% and 10% settings.PAWS also avoids the additional momentum encoder or memory buffer used by SimCLRv2 and BYOL.Source: §6 · p. 7

Self-supervised and semi-supervised comparisons

• PAWS exceeds MPL, the reported state-of-the-art semi-supervised method, while requiring significantly fewer training epochs.MPL simultaneously trains student and teacher networks for 800 epochs.Source: §6 · p. 7

Architecture scaling

• 69.9% and 79.0% top-1 accuracy are achieved by a 4× wider ResNet-50 with 1% and 10% labels, respectively, after 200 epochs.The reported larger-architecture results are consistent with increasing model capacity improving semi-supervised performance.Source: §6 · p. 7

Longer Training

• More than 200 pre-training epochs is generally unnecessary across the tested ResNet architectures and label settings, yielding only marginal improvements.This pattern is reported for ResNet-50 and ResNet-50 (2×) with both 1% and 10% labels.Source: Table 3 · p. 7, §7 · p. 7

Learning During Pre-training

• During training, cross-entropy, instance-discrimination, and classification losses decrease, while target-prediction confidence increases.The auxiliary instance-discrimination and classification losses are reported for analysis only and do not receive gradients.Source: §7 · p. 7, Figure 4 · p. 8

Support Set

• Larger support sets improve performance, while sampling more classes with fewer images per class is preferable when the number of instances is fixed.The support-set ablation uses ResNet-50 on ImageNet for 100 epochs.Source: §7 · p. 7, Table 4 · p. 8

Prediction Head

• PAWS still converges without a prediction head, unlike self-supervised methods that collapse without one.The prediction head is marginally better omitted during PAWS pre-training.Source: Table 5 · p. 8, §7 · p. 8

ME-Max Regularization

• ME-MAX regularization is especially helpful with 1% labeled data but provides only a marginal improvement with 10%.The regularizer maximizes the entropy of average predictions across unlabeled samples.Source: Table 6 · p. 8, §7 · p. 8

Small Batch Training

• PAWS achieves good performance after 100 epochs with small batches on 8 NVIDIA V100-16G GPUs.The small-batch experiment uses an unsupervised batch size of 256 and disables ME-MAX regularization.Source: §7 · pp. 8–9, Table 7 · p. 9

Method

• PAWS leverages a small labeled support set during pre-training to learn from both labeled and unlabeled data.The method can also be interpreted as using an external memory represented by labeled support examples.Source: §8 · p. 9

Results

• PAWS achieves competitive semi-supervised accuracy and requires significantly less training than previous works.The paper also states that PAWS provably avoids collapsing solutions, a common challenge in self-supervised approaches.Source: §8 · p. 9

Future Directions

• Future work will investigate competitive representations using only instance supervision and more flexible memory representations.The stated practical advantage of the supervised support set is efficient learning.Source: §8 · p. 9

Support Sampling

• Each iteration samples a support mini-batch from labeled data to determine soft pseudo-labels for unlabeled image views.Classes are sampled first, followed by an equal number of images from each sampled class.Source: A Implementation Details · p. 11

Projection and Prediction Heads

• The projection head is a three-layer MLP, while the prediction head is a two-layer MLP with the stated hidden and output dimensions.Both heads use ReLU activations and batch normalization in the specified layers.Source: A Implementation Details · p. 11

Fine-tuning

• Fine-tuning optimizes supervised cross-entropy on the available labeled samples using the encoder and a linear classifier.The classifier is initialized with zero weights and weight decay is not used during fine-tuning.Source: A Implementation Details · p. 11

Nearest-Neighbor Evaluation

• PAWS also supports nearest-neighbor evaluation without fine-tuning by classifying representations using the available labeled training samples.The predicted class is the label with the highest probability under the similarity classifier.Source: A Implementation Details · p. 11

Nearest-Neighbor Classifier

• The soft nearest-neighbor classifier assigns larger weights to labels of representations more similar to the queried representation.Its output is a weighted average of labeled representations’ class labels.Source: Figure 5 · p. 11

Multi-Crop

• PAWS generates two large 224 × 224 crops and six small 96 × 96 crops for each unlabeled image.Small crops use the two large crops as positive views, while each large crop uses the other large crop as its positive view.Source: A Implementation Details · p. 12

Comparison

• PAWS matches fully supervised learning for ResNet-50 (2×) and ResNet-50 (4×) using only 10% of ImageNet labels.The supervised baseline uses all ImageNet labels and the same architecture, with the best result selected across training and augmentation settings.Source: B Comparison to Supervised Learning · p. 13, Figure 6 · p. 13

Baseline

• The comparison uses supervised models trained with standard cross-entropy on the full ImageNet label set.The reported supervised baselines are taken from SimCLR and selected over specified epoch and augmentation sweeps.Source: B Comparison to Supervised Learning · p. 13

C Additional Experiments — CIFAR10

• The CIFAR10 setup uses multi-crop augmentation with two large views and six small views generated from each unlabeled image.Large crops are 32 × 32 and small crops are 18 × 18.Source: C Additional Experiments — CIFAR10 · p. 14
• PAWS pre-training works well without a prediction head on both ImageNet and CIFAR10.The CIFAR10 encoder uses a WideResNet-28-2 trunk with a three-layer MLP projection head and no prediction head.Source: C Additional Experiments — CIFAR10 · p. 14
• PAWS-NN matches the state-of-the-art on CIFAR10 with significantly less training when evaluated using 4000 labeled images as support.The evaluation uses the pretrained representations directly in a nearest-neighbour classifier rather than fine-tuning a linear classifier.Source: C Additional Experiments — CIFAR10 · pp. 13–14
• PAWS assigns soft pseudo-labels by comparing anchor and positive-view representations with labeled support samples, then minimizes their cross-entropy.The similarity classifier is a differentiable Soft Nearest Neighbours strategy.Source: C Additional Experiments — CIFAR10 · p. 14, D Alternative Strategies for… · p. 15
• Target sharpening prevents trivial representation collapse because collapsed representations produce high-entropy predictions while targets remain low-entropy.The theoretical guarantee identifies sharpening as sufficient to make collapsed representations non-stationary.Source: D Alternative Strategies for… · p. 15

D.1 Semi-Supervised Prediction

• Using a labeled sample's class label as the prediction target provides a theoretical alternative to using the positive-view prediction.The alternative is analyzed under Proposition 2.Source: §D.1 · p. 15
• Each mini-batch of image views is assumed to contain at least one labeled sample.This is the stated semi-supervised image-view assumption.Source: §D.1 · p. 15
• Under the stated assumptions, collapsed representations have a nonzero cross-entropy gradient and are therefore non-stationary.Proposition 2 gives this guarantee for the semi-supervised setting.Source: §D.1 · p. 15
• The labeled-target strategy is not used in experiments; PAWS instead uses the sharpened positive-view prediction as the target.The paper presents Proposition 2 only as a theoretical alternative for preventing collapse.Source: §D.1 · p. 15
• Adding entropy minimization is another possible non-collapse strategy, but the paper uses target sharpening because it is simpler and sufficient under Proposition 1.The entropy term would penalize the high-entropy predictions associated with collapsed representations.Source: §D.2 · p. 15

E Ethical Considerations

• Increasing model and dataset sizes can improve image-recognition performance but may increase environmental footprints and exclusionary effects.The paper connects computationally expensive training with both environmental and participation concerns.Source: E Ethical Considerations · p. 15
• PAWS matches current state-of-the-art data-efficient image recognition using smaller models and fewer training epochs.The paper also reports that smaller-model performance is not yet saturated.Source: E Ethical Considerations · p. 15
• Greater computational effort does not always correspond linearly to larger models or better accuracy.For example, the paper contrasts a wider ResNet-50 trained for 100 epochs with a smaller ResNet-50 trained for 200 epochs and reports different top-1 accuracies.Source: E Ethical Considerations · p. 15