Source-linked AI summary
Boosting Few-Shot Visual Learning with Self-Supervision
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, Matthieu Cord
TL;DR
Few-shot learning must recognize novel classes from very few labeled examples, while self-supervision learns representations from annotation-free tasks. The paper combines them by adding self-supervision as an auxiliary loss during few-shot training and extends the framework to diverse unlabeled data. Across architectures, datasets, and self-supervision techniques, the approach consistently improves few-shot performance and achieves state-of-the-art results.
Problem
Few-shot models must learn to recognize novel classes from very few examples, while deep visual representation learning traditionally depends on large amounts of labeled data.
Method
The method adds an auxiliary self-supervised loss to the first-stage few-shot training objective, optionally using diverse unlabeled data alongside annotated data.
Results
Experiments across architectures, datasets, and self-supervision techniques consistently improve few-shot classification performance and achieve state-of-the-art results.
Takeaways & Limitations
Self-supervision can be integrated with few-shot recognition and used to exploit unlabeled data in semi-supervised and unsupervised few-shot settings.
Abstract
from arXiv · showhide
Few-shot learning and self-supervised learning address different facets of the same problem: how to train a model with little or no labeled data. Few-shot learning aims for optimization methods and models that can learn efficiently to recognize patterns in the low data regime. Self-supervised learning focuses instead on unlabeled data and looks into it for the supervisory signal to feed high capacity deep neural networks. In this work we exploit the complementarity of these two domains and propose an approach for improving few-shot learning through self-supervision. We use self-supervision as an auxiliary task in a few-shot learning pipeline, enabling feature extractors to learn richer and more transferable visual representations while still using few annotated samples. Through self-supervision, our approach can be naturally extended towards using diverse unlabeled data from other datasets in the few-shot setting. We report consistent improvements across an array of architectures, datasets and self-supervision techniques.
1. Introduction
Few-shot learning seeks recognition from very few labeled examples, while self-supervision learns transferable representations without annotations. This paper combines them by adding self-supervision to few-shot training and extending it to unlabeled data.
- Few-shot visual learning targets recognition models that learn new classes from only one or a few examples per class.
- Few-shot systems typically learn transferable representations from annotated base classes before adapting to unseen novel classes with few examples.
- Self-supervised representation learning uses annotation-free pretext tasks to learn image features transferable to downstream vision tasks.
- The proposed method adds a self-supervised loss to the first-stage few-shot objective, jointly training the feature extractor with supervised recognition.Figure 1 illustrates image rotation prediction as the auxiliary task, using four rotations and optionally separate unlabeled images.
- Self-supervision permits semi-supervised and unsupervised few-shot regimes by incorporating diverse unlabeled data during the first learning stage.The authors report that both regimes can be used for few-shot recognition.
- Experiments on MiniImagenet, CIFAR-FS, and tiered-MiniImagenet found improved few-shot performance and state-of-the-art results, with further gains in semi-supervised settings.
2. Related work
Related work spans few-shot learning methods and self-supervised representation learning. The paper positions its contribution as auxiliary self-supervision for improving the main few-shot task.
- Few-shot learning: Few-shot learning includes gradient-based adaptation, metric learning, memory-based methods, and approaches that generate classifiers or neural-network weights.
- Few-shot learning: The paper uses Prototypical Networks and Cosine Classifiers as simple, flexible metric-learning approaches, while stating that auxiliary self-supervision is more broadly compatible.
- Self-supervised learning: Self-supervised learning defines annotation-free pretext tasks that provide surrogate supervision for feature learning, including colorization, patch-position, rotation, and image-completion prediction.
- Self-supervised learning: This work trains the backbone with joint supervision from the supervised end task and an auxiliary self-supervised task, optimizing the main task rather than both tasks equally.
3. Methodology
The methodology combines standard few-shot learning with self-supervised objectives during the first learning stage to improve feature transfer to novel classes. It evaluates Prototypical Networks and Cosine Classifiers with rotation and patch-location tasks, including unlabeled-data extensions.
- Few-shot learning setup: Few-shot learning uses base classes in a first stage to learn transferable representations, then adapts to disjoint novel classes with Nn-way K-shot data.Benchmarks use K = 1 or 5 samples per novel class.
- Explored few-shot methods: Prototypical Networks form class prototypes by averaging support features, whereas Cosine Classifiers jointly train a feature extractor and base-class cosine classifier.In the novel-class stage, the feature extractor is frozen and prototypes are computed from novel-class samples.
- Auxiliary self-supervision: The proposed method adds a self-supervised loss to the first-stage few-shot objective, with α controlling its importance, while retaining either the PN or CC few-shot loss.The experiments use α = 1.0.
- Unlabeled-data extension: Extra unlabeled images can contribute to self-supervised training without sharing classes with the labeled base dataset, enabling a more flexible semi-supervised regime.The paper also considers an unsupervised regime that removes base classes from the first learning stage.
4. Experimental Results
Experiments evaluate auxiliary self-supervision, semi-supervised training with unlabeled data, and few-shot recognition as a way to compare self-supervised methods across datasets and architectures. Self-supervision consistently improves few-shot performance, with stronger gains for high-capacity networks and state-of-the-art comparisons.
- Experimental setup: Experiments use MiniImageNet, tiered-MiniImageNet, and CIFAR-FS, evaluating 1-shot and 5-shot 5-way classification over sampled novel-class tasks.The standard evaluation uses 15-way? No: M = 15 sampled tasks, Nn = 5 novel classes, and K = 1 or 5 examples per class.
- Experimental setup: The study evaluates Conv-4-64, Conv-4-512, and WRN-28-10 feature extractors with Cosine Classifiers and Prototypical Networks.Rotation prediction uses a convolutional auxiliary network, while relative patch location uses a network operating on concatenated patch features.
- Auxiliary self-supervision: Adding rotation prediction improves few-shot classification for Cosine Classifiers and Prototypical Networks, with larger gains for high-capacity architectures such as WRN-28-10.For Prototypical Networks, rotation augmentation without the auxiliary task degraded performance and was therefore not used.
- Auxiliary self-supervision: Relative patch location prediction also improves few-shot classification, although its gains are smaller than those from rotation prediction and more pronounced for high-capacity networks.Its assessment is restricted to Cosine Classifiers on MiniImageNet because CIFAR-FS images are too small for meaningful patch extraction.
- Comparison with prior work: Across MiniImageNet, CIFAR-FS, and tiered-MiniImageNet, the approach achieves state-of-the-art results and surpasses prior methods by significant margins.On MiniImageNet, it exceeds LEO by about 1.3 percentage points in 1-shot and 2.3 percentage points in 5-shot settings.
- Semi-supervised training: Unlabeled images improve semi-supervised few-shot classification, including cross-dataset unlabeled data and a WRN-28-10 model reaching 63.77% and 80.70% MiniImageNet accuracy in 1-shot and 5-shot settings.With 20% annotations, the Conv-4-64 model reaches 51.21% and 68.89%, compared with 50.41% and 64.39% for a method using 40% annotations.
- Evaluating self-supervised representations: Few-shot recognition can evaluate self-supervised methods by training on unlabeled base-class images and measuring transfer to novel-class classification.The framework uses only the self-supervised loss in the first stage, then compares resulting few-shot performance.
5. Conclusions
The paper adds self-supervision as an auxiliary loss to few-shot recognition, improving novel-class classification across several datasets and enabling use of diverse unlabeled data.
- Adding self-supervision to few-shot recognition models significantly improves novel-class classification performance.
- Experiments on MiniImagenet, CIFAR-FS, and tiered-MiniImagenet produce state-of-the-art results for the employed few-shot models.
- The annotation-free self-supervised loss supports semi-supervised training with diverse unlabeled data, further improving classification performance.
- The framework can also evaluate self-supervised or unsupervised methods through few-shot object recognition.
A.1. Rotation prediction self-supervision: Impact of rotation augmentation
Ablations show that rotation prediction benefits more from removing rotation augmentation, while the auxiliary patch classification loss contributes little compared with relative patch-location self-supervision.
- Rotation prediction self-supervision yields more significant improvements when baselines are trained without rotation augmentation.
- In some cases, rotation augmentation reduces few-shot classification performance.
- The auxiliary patch-based object classification loss provides small or nonexistent gains compared with relative patch-location self-supervision.
B.1. Network architectures
The experiments use three feature-extractor architectures and report ablations for rotation augmentation and patch-based classification alongside the architecture configurations.
- Conv-4-64 has four 64-channel convolutional blocks and produces a 1600-dimensional feature vector.
- Conv-4-512 increases block widths to 96, 128, 256, and 512 channels, producing a 12,800-dimensional feature vector.
- WRN-28-10 is a 28-layer Wide Residual Network with width factor 10 and a 640-dimensional pooled feature vector.
- The rotation-augmentation and patch-classification ablations report average 5-way novel-class test accuracies with 95% confidence intervals.
- Rotation prediction uses a convolutional or residual prediction network over Fθ feature maps, while patch-location prediction uses concatenated patch features and fully connected layers.
B.2. Incorporating self-supervision during training
Self-supervision is incorporated during training by generating transformed images or patches and applying auxiliary tasks alongside few-shot classification.
- Rotation prediction creates four rotated copies of each mini-batch image and applies the self-supervised loss to them.
- The object-classification loss uses all rotated images with rotation augmentation, but only upright images otherwise.
- Relative patch-location training combines images with patches sampled from a 3 × 3 grid after resizing and optional grayscale conversion.
- Each patch is randomly sampled at 24 × 24 pixels from a 32 × 32 grid region.
B.3. Training routine for first learning stage
The training routine uses momentum SGD with scheduled learning-rate decay, dataset- and architecture-specific training settings, and mixed labeled–unlabeled mini-batches for semi-supervised experiments.
- Training uses mini-batch SGD with momentum 0.9, weight decay 5e−4, and an initial learning rate of 0.1.
- MiniImageNet and CIFAR-FS models train for 60 epochs with learning-rate reductions every 20 epochs, while tiered-MiniImageNet models train for 100 epochs with reductions every 40 epochs.
- Semi-supervised mini-batches combine labeled and unlabeled images, using architecture- and labeling-fraction-specific batch compositions.
B.4. Assessing self-supervised representations based on the few-shot object recognition task
The few-shot object-recognition evaluation largely follows the other CC-based experiments, but removes object supervision and validation-based early stopping during the first learning stage.
- During the first learning stage, the evaluation uses no object-based supervision and therefore no Lfew loss.
- The evaluation omits early stopping based on a validation set during the first learning stage.
- Apart from these first-stage changes, the implementation details remain the same as in the other CC-based experiments.