Source-linked AI summary
Leveraging the Feature Distribution in Transfer-based Few-Shot Learning
Yuqing Hu, Vincent Gripon, Stéphane Pateux
TL;DR
Few-shot classification must work with limited labelled data, while extracted backbone features may have complex, non-Gaussian distributions. The paper preprocesses those features toward Gaussian-like distributions and applies an optimal-transport-inspired class-association algorithm, achieving state-of-the-art accuracy across varied benchmarks, backbones, and few-shot settings.
Problem
Few-shot classification uses few labelled examples, while backbone features may be complex or non-Gaussian despite common Gaussian-like distribution assumptions.
Method
The method applies skew-reducing power transforms and unit-variance projection, then uses a maximum-a-posteriori, optimal-transport-inspired algorithm for transductive classification.
Results
The method achieves state-of-the-art accuracy across standardized vision benchmarks, varied backbone architectures, and some inductive and most transductive settings.
Takeaways & Limitations
The proposed pipeline brings gains in 1-shot and 5-shot classification and across a variety of feature extractors, with few hyperparameters.
Takeaways & Limitations
The optimal-transport-inspired algorithm is used in the transductive setting, while the evaluation covers only some inductive settings.
Abstract
from arXiv · showhide
Few-shot classification is a challenging problem due to the uncertainty caused by using few labelled samples. In the past few years, many methods have been proposed to solve few-shot classification, among which transfer-based methods have proved to achieve the best performance. Following this vein, in this paper we propose a novel transfer-based method that builds on two successive steps: 1) preprocessing the feature vectors so that they become closer to Gaussian-like distributions, and 2) leveraging this preprocessing using an optimal-transport inspired algorithm (in the case of transductive settings). Using standardized vision benchmarks, we prove the ability of the proposed methodology to achieve state-of-the-art accuracy with various datasets, backbone architectures and few-shot settings.
1 Introduction
Few-shot learning addresses classification when large labelled datasets are costly or impossible to create. This paper proposes transforming backbone features toward Gaussian-like distributions and using them in a transfer-based method evaluated across settings and benchmarks.
- Few-shot learning addresses classification with few labelled examples when acquiring or annotating large datasets is costly.
- Inductive few-shot predicts test inputs independently, whereas transductive few-shot uses the joint distribution of an unlabeled test batch.
- Transfer architectures use backbones trained once on a large dataset to extract features for novel tasks.
- The proposed strategy preprocesses backbone features toward Gaussian-like distributions, then applies a maximum-a-posteriori and optimal-transport-based algorithm in transductive settings.
- Standardized benchmarks show state-of-the-art accuracy across varied problems and backbone architectures, including some inductive and most transductive settings.
2 Related work
Related work spans meta-learning and transfer-based approaches, with transfer methods typically performing best on standardized benchmarks. The paper focuses on the comparatively underexplored distribution of extracted features and proposes preprocessing and class-association improvements.
- Meta-learning methods train optimizers or augment data to adapt classification systems from limited examples.
- Transfer-based methods train feature extractors to segregate novel classes, supporting classifiers based on distances, class centers, graphs, or iterative adjustment.
- Transfer-based techniques typically reach the best performance on standardized benchmarks.
- Few studies examine feature distributions experimentally, motivating transforms that reduce skew and a Wasserstein-based method for aligning features with class distributions.
3 Methodology
The method trains a backbone, preprocesses its nonnegative feature embeddings toward Gaussian-like distributions, and estimates class centers using Wasserstein-based soft assignments in transductive settings.
- Problem setting: The problem uses disjoint base and novel datasets, with labelled support examples and unlabelled query examples from w new classes.The task is parameterized by w-way, s-shot, and q unlabelled examples per class.
- Feature extraction: A backbone trained only on the base dataset extracts embeddings intended to generalize to novel classes.The experiments use the penultimate-layer output, whose feature components are nonnegative because of ReLU.
- Feature preprocessing: Power transformation and unit-variance projection reduce skew and prevent high-variance features from dominating, often producing close-to-Gaussian features.The transform uses ε = 1e−6 for strict positivity and β as a hyper-parameter; β = 0.5 gave the most consistent experimental results.
- Feature preprocessing: The preprocessing step can be applied in both inductive and transductive few-shot settings.Transductive prediction uses unlabelled samples and priors about their relative proportions.
- Iterative center estimation: Assuming Gaussian-like class distributions, the method iteratively estimates class centers through soft allocations of unlabelled features.It initializes centers from labelled samples, computes a Sinkhorn mapping, then re-estimates centers by weighted averaging labelled and allocated unlabelled features.
- Sinkhorn mapping: The class mapping minimizes Wasserstein distance using a Sinkhorn-inspired objective with entropy regularization.The transport cost is based on Euclidean distances between unlabelled examples and class centers, while soft mapping allows an example to be allocated across classes.
4 Experiments
Experiments evaluate the method across standardized few-shot datasets, backbones, inductive and transductive settings, cross-domain transfer, and ablations. Results show competitive or state-of-the-art performance, gains from Power Transform and MAP, and robustness to several experimental variations.
- Experimental setup: The method is evaluated on standardized few-shot classification datasets using multiple backbone architectures, with novel-class testing after base-class training and validation-based hyperparameter tuning.Experiments include WRN, ResNet18, ResNet12 and pretrained DenseNet backbones, with 1-shot and 5-shot settings.
- Comparison with state-of-the-art methods: The proposed method reaches state-of-the-art performance in most inductive and transductive benchmark cases and improves accuracy across datasets and backbones.The reported evaluation covers miniImageNet, tieredImageNet and other standardized benchmarks.
- Cross-domain settings: The method achieves the best accuracy in both 1-shot and 5-shot cross-domain classification from miniImageNet-trained features to CUB novel classes.The cross-domain evaluation uses WRN as the backbone.
- Ablation study: The MAP procedure substantially outperforms standard K-Means, indicating better class-center estimation in the evaluated inductive setting.The comparison initializes K-Means centroids with available labelled samples for each class.
- Effect of Power Transform: Power Transform yields about 6% accuracy gain for 1-shot and 4% for 5-shot transductive classification, while reshaping feature distributions toward Gaussian-like forms.The distributional effect is visualized for five randomly selected novel classes, with similar behavior observed on other datasets.
- Other experiments: Transductive accuracy quickly reaches a near-asymptotic plateau as unlabelled samples increase and remains higher than inductive performance when class proportions are only approximately known.The class-imbalance experiment fixes the total number of query inputs at 100 and considers a mixed estimate q = [30, 30].
5 Conclusion
The paper presents a pipeline that Gaussian-aligns backbone features before using an optimal-transport inspired algorithm to estimate class centers. Experiments on standard vision benchmarks report state-of-the-art accuracy, gains in 1-shot and 5-shot settings, and applicability across feature extractors with few hyperparameters.
- The proposed pipeline preprocesses raw backbone features toward a Gaussian distribution, then iteratively estimates class centers with an optimal-transport inspired algorithm.
- The method reaches state-of-the-art accuracy on standard vision benchmarks, with important gains in both 1-shot and 5-shot classification.
- The approach brings gains across a variety of feature extractors while using few hyperparameters.
6 ADDITIONAL EXPERIMENTS
Additional experiments show that PT-MAP remains effective across differently trained pretrained backbones and can benefit from concatenating features from multiple backbones. Direct multi-backbone concatenation yields about a 1% gain in both 1-shot and 5-shot settings.
- Effect of PT-MAP across pretrained backbones: PT-MAP produces large accuracy increases across pretrained backbones regardless of their training procedure.
- Effect of PT-MAP on multi-backbones: The multi-backbone evaluation combines WRN, ResNet18 and ResNet12 features under different concatenation configurations.
- Effect of PT-MAP on multi-backbones: Direct concatenation of features from multiple backbones can improve PT-MAP performance by about 1% in both 1-shot and 5-shot settings.