Source-linked AI summary
When Does Self-supervision Improve Few-shot Learning?
Jong-Chyi Su, Subhransu Maji, Bharath Hariharan
TL;DR
The paper asks whether self-supervision improves few-shot learning when labeled and unlabeled data are limited. It combines supervised meta-learning with auxiliary self-supervision, studies domain shift, and selects compatible images from generic pools. SSL improves few-shot performance, especially on smaller or harder tasks, but mismatched domains can hurt; automatic domain-based selection provides further gains.
Problem
The paper addresses the limited evidence on whether self-supervised learning helps few-shot learning with small datasets and scarce unlabeled images.
Method
The paper combines supervised few-shot meta-learning with self-supervised losses, analyzes domain shifts, and uses a domain classifier to select similar-domain unlabeled images.
Results
Self-supervision improves few-shot learning across domains, with larger benefits on more challenging tasks; domain-mismatched images can hurt performance, while selected similar-domain images improve results further.
Takeaways & Limitations
Images within small datasets can support self-supervision, while additional unlabeled data is most useful when it comes from the same or a similar domain.
Takeaways & Limitations
The method assumes that small or fine-grained domains may have limited labeled base data, making generalizable representation learning challenging.
Abstract
from arXiv · showhide
We investigate the role of self-supervised learning (SSL) in the context of few-shot learning. Although recent research has shown the benefits of SSL on large unlabeled datasets, its utility on small datasets is relatively unexplored. We find that SSL reduces the relative error rate of few-shot meta-learners by 4%-27%, even when the datasets are small and only utilizing images within the datasets. The improvements are greater when the training set is smaller or the task is more challenging. Although the benefits of SSL may increase with larger training sets, we observe that SSL can hurt the performance when the distributions of images used for meta-learning and SSL are different. We conduct a systematic study by varying the degree of domain shift and analyzing the performance of several meta-learners on a multitude of domains. Based on this analysis we present a technique that automatically selects images for SSL from a large, generic pool of unlabeled images for a given dataset that provides further improvements.
1 Introduction
The paper examines whether self-supervision improves few-shot learning when labeled and unlabeled data are limited, and how the self-supervision domain affects performance. It finds benefits across domains and harder tasks, but domain mismatch can reduce performance.
- Few-shot representation learning can discard semantic information useful for novel classes, especially with small base datasets or challenging class distinctions.
- Self-supervised tasks improve existing few-shot techniques across benchmarks spanning multiple domains without additional training data.The auxiliary task is combined with supervised learning as a data-dependent regularizer.
- Self-supervision provides greater benefits as tasks become harder, including with smaller training sets, low-resolution inputs, or greyscale images.
- Additional unlabeled images improve performance only when they come from the same domain as the base classes; mismatched images can hurt the learner.
- A domain classifier selects similar-domain unlabeled images from a generic pool, improving over self-supervision using images already within the dataset.
2 Related Work
Prior work studies few-shot meta-learning, self-supervised representation learning, multi-task learning, and domain selection. This paper differs by using self-supervision to augment supervised few-shot transfer without external data and by selecting compatible unlabeled images.
- Few-shot methods include optimization-based, distance-based, and feed-forward approaches for adapting representations or classifiers from limited examples.
- Prior self-supervised learning typically replaces supervised representation learning with large-scale unlabeled-data pretraining, whereas this work augments supervised few-shot transfer in a low-data regime.
- The paper extends related few-shot self-supervision work to harder fine-grained datasets, deeper models, domain-shift analysis, and automatic image selection.
- Multi-task learning can help when objectives are compatible, but training multiple tasks together often hurts individual-task performance.
- Unlike earlier domain-selection methods, the proposed approach does not assume that the source domain is labeled.
3 Method
The method combines supervised few-shot meta-learning with self-supervised objectives through a shared feature backbone. It supports same- or different-domain SSL data and studies how to choose effective auxiliary images.
- The supervised model maps images through a feature function f and classifier g while minimizing empirical loss with regularization.
- Self-supervised learning adds a predictor h that learns automatically generated labels from transformed images, using jigsaw permutations or rotation angles.
- The final objective combines supervised and self-supervised losses, making SSL a data-dependent regularizer for representation learning.
- Prototypical networks sample episodic N-way K-shot tasks and classify query examples using distances to class prototypes; MAML and standard cross-entropy baselines are also evaluated.
- When supervised and self-supervised domains differ, the method uses a separate SSL batch and combines gradients after separate forward passes.
4 Experiments
Experiments across diverse few-shot benchmarks show that self-supervised auxiliary tasks improve meta-learning, with larger gains on harder or smaller-data settings. The study also finds that SSL depends on domain alignment and introduces domain-weighted image selection to exploit generic unlabeled pools.
- Self-supervised learning improves few-shot learning: Rotation prediction improves ProtoNet on most datasets, while combining jigsaw and rotation can help on some datasets.Rotation is ineffective on aircrafts and flowers, where image structure may make the task too hard or too trivial.
- Improvements generalize to other meta-learners: Jigsaw auxiliary learning improves average 5-way 5-shot accuracy for softmax, MAML, and ProtoNet from 85.5%, 82.6%, and 88.5% to 86.6%, 83.8%, and 90.4%.ProtoNet with self-supervision is the best model across all five fine-grained datasets.
- Self-supervision alone is not enough: SSL alone substantially underperforms supervised learning, achieving 32.9% or 33.7% versus 85.5% average 5-way 5-shot accuracy.The SSL-only figures use jigsaw or rotation, respectively, across five fine-grained datasets.
- Effect of domain shift: Increasing SSL data helps with same-domain images, whereas domain shift makes SSL less effective and can make fivefold expansion worse than using 20% in-domain data.The experiments vary unlabeled-set size and replace in-domain images with images from other datasets.
- Domain-weighted image selection: A domain classifier selects similar-domain images from a generic pool using importance weights, and the selected pool improves over no SSL, in-dataset SSL, and random-selection baselines.The evaluated pool combines Open Images V5 and iNaturalist 2018; the oracle uses held-out same-distribution images.
5 Conclusion
Self-supervision improves few-shot learning across domains, including challenging settings with small datasets, while additional unlabeled images help only when they come from the same or similar domains. The paper proposes automatically identifying such images from a larger pool, while leaving alternative self-supervised tasks for future work.
- Self-supervision improves few-shot learning across a range of different domains.
- The benefits are greater for more challenging problems, especially when few images are available for self-supervision.
- Additional unlabeled images improve performance only when they come from the same or similar domains.
- A simple approach automatically identifies similar-domain images from a larger pool when unlabeled data is limited.
- Future work could examine other self-supervised tasks and generalization across self-supervised and supervised tasks.
A Appendix
The appendices provide numerical figure results, additional classification findings, model visualization, and implementation details.
- The appendices separately provide the numbers underlying Figures in Sections 4.1 and 4.3.
- Additional appendices examine SSL for traditional fine-grained classification, visualize its model, and describe implementation details.
A.1 Results on Few-shot Learning
Self-supervision improves ProtoNet performance across datasets and yields larger gains on harder or smaller-data settings, with rotation prediction outperforming jigsaw supervision in coverage.
- 2% to 21% improvements occur across all seven datasets when ProtoNet predicts rotations instead of using random initialization.
- Jigsaw-puzzle supervision improves performance only on the aircrafts and flowers datasets relative to random initialization.
- SSL gains are higher on degraded datasets and when training uses only 20% of base-category images.
A.2 Results on Selecting Images for SSL
The image-selection procedure compares random pool sampling with weighted sampling that favors images from related domains.
- “Pool (random)” samples images uniformly in proportion to each dataset’s size, whereas “pool (weight)” favors related domains.
A.3 Results on Standard Fine-grained Classification
On standard fine-grained classification, self-supervised losses improve models trained from scratch using only the dataset’s training images. Rotation prediction helps most on several datasets, while jigsaw loss leads on others.
- Self-supervision improves fine-grained classification when a ResNet-18 is trained from scratch using only labeled training images.The study uses standard training and test splits and compares against supervised training.
- The evaluation addresses standard fine-grained classification, where test images are novel examples from classes seen during training.
- 4.1%, 3.1%, and 3.0% improvements result from rotation prediction on birds, cars, and dogs, respectively.
- 0.9% and 3.6% improvements result from the jigsaw puzzle loss on aircrafts and flowers, respectively.
A.4 Visualization of Learned Models
The visualization analysis examines which image pixels support correct classification and suggests that self-supervision shifts attention toward foreground object regions. The reported tables also show stronger gains for jigsaw loss on more challenging settings and benefits from selecting similar unlabeled images.
- Saliency maps compute each pixel’s gradient magnitude for the correct-class logit as a proxy for pixel importance.
- Jigsaw-loss improvements are higher when inputs are degraded or only 20% of base-set images are available.
- Selecting similar unlabeled images by importance weights improves performance on all five datasets, whereas random additions often hurt.
- Self-supervised models tend to focus more on foreground regions within the object bounding box.
- The authors hypothesize that self-supervised tasks reduce reliance on background features accidentally correlated with class labels.
A.5 Experimental Details
The experiments combine supervised and self-supervised objectives with task-specific optimization choices, architectures, and training procedures. Implementation details include loss weighting, batch-normalization handling, episode queries, domain classification, and self-supervised task heads.
- For jigsaw training, batch-normalization statistics are estimated independently for each batch because full images and patches may have different statistics.
- The total loss is L = (1 − λ)Ls + λLss, with λ = 0.5 usually and λ = 0.3 for jigsaw on mini- and tiered-ImageNet.
- Training episodes use 16 query images per class, reduced to 5 when only 20% of labeled data are used; MAML uses 10 queries.
- The domain classifier uses 2048-dimensional penultimate-layer features from ImageNet-pretrained ResNet-101 and logistic regression to distinguish labeled from unlabeled-pool images.
- Standard classification trains ResNet-18 from scratch with ADAM at learning rate 0.001 for 600 epochs and batch size 16.
- The rotation head maps ResNet-18’s 512-dimensional output through fully connected layers to four rotation-angle predictions with ReLU and 0.5 dropout.