Source-linked AI summary
Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learning
Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, Kwang-Ting Cheng
TL;DR
Few-shot learning must recognize unseen classes from limited labeled data, yet transferring all base-model knowledge can retain biased or harmful information. P-Transfer selectively freezes or fine-tunes layers with searched learning rates, and experiments report strong improvements across meta-learning, non-meta methods, and conventional transfer settings.
Problem
Few-shot learning transfers from base classes to unseen classes with limited support data, but complete transfer can include biased or harmful knowledge because base and novel classes do not overlap.
Method
P-Transfer uses evolutionary search to select backbone layers for freezing or fine-tuning and determine individual layer learning rates for novel-class adaptation.
Results
P-Transfer consistently outperforms complete and hand-crafted transfer strategies and improves both meta-based and non-meta few-shot methods across datasets.
Takeaways & Limitations
Partial transfer provides a flexible way to preserve useful base knowledge while adapting the backbone with few novel-class support samples.
Takeaways & Limitations
The authors caution that failures in applications such as medical imaging could produce unreliable conclusions, including misclassified diseases.
Abstract
from arXiv · showhide
The goal of few-shot learning is to learn a classifier that can recognize unseen classes from limited support data with labels. A common practice for this task is to train a model on the base set first and then transfer to novel classes through fine-tuning (Here fine-tuning procedure is defined as transferring knowledge from base to novel data, i.e. learning to transfer in few-shot scenario.) or meta-learning. However, as the base classes have no overlap to the novel set, simply transferring whole knowledge from base data is not an optimal solution since some knowledge in the base model may be biased or even harmful to the novel class. In this paper, we propose to transfer partial knowledge by freezing or fine-tuning particular layer(s) in the base model. Specifically, layers will be imposed different learning rates if they are chosen to be fine-tuned, to control the extent of preserved transferability. To determine which layers to be recast and what values of learning rates for them, we introduce an evolutionary search based method that is efficient to simultaneously locate the target layers and determine their individual learning rates. We conduct extensive experiments on CUB and mini-ImageNet to demonstrate the effectiveness of our proposed method. It achieves the state-of-the-art performance on both meta-learning and non-meta based frameworks. Furthermore, we extend our method to the conventional pre-training + fine-tuning paradigm and obtain consistent improvement.
1. Introduction
Few-shot learning transfers knowledge from abundant base classes to unseen classes with limited labeled data, but whole-model transfer can preserve knowledge that mismatches novel classes. P-Transfer searches layer-specific fine-tuning strategies to balance reuse of base knowledge with adaptation to novel data.
- Few-shot learning aims to classify unseen classes from only a few labeled samples, motivating transfer from abundant base-class data.
- Limited novel-class support makes training from scratch unstable and fine-tuning all layers prone to poor performance and overfitting.
- Because base and novel classes differ, completely freezing the backbone can transfer biased or harmful knowledge and suffer from domain discrepancy.
- P-Transfer uses evolutionary search to determine which backbone layers to freeze or fine-tune, assigning different learning rates to selected layers.
- The layer-wise search takes approximately 6 hours with Conv6 and one day with ResNet-12 on one V100 GPU, on average.
- P-Transfer outperforms complete and hand-crafted transfer strategies by a remarkable margin across few-shot learning methods and datasets.
2. Background
Few-shot learning generalizes from labeled base classes to novel classes with few labeled examples, using meta-learning or non-meta approaches. P-Transfer differs from architecture search by evolving fine-tuning schemes and supports both meta and non-meta settings.
- Few-shot learning generalizes a network trained on abundant labeled base examples to classify new classes using few labeled samples.
- Meta-learning methods learn models or optimizers that rapidly adapt to unseen tasks, while non-meta methods train base feature extractors and predict novel-class classifier weights.
- P-Transfer uses evolutionary search to find fine-tuning schemes rather than neural architectures, distinguishing it from evolutionary neural architecture search.
3. Methodology
P-Transfer searches for a layer-wise strategy that selectively freezes or fine-tunes the pretrained feature extractor when transferring from base to novel classes. Its three-step framework combines base-model training, evolutionary strategy search, and partial transfer for both baseline and meta-learning methods.
- Framework: The framework has three stages: base-class pre-training, evolutionary search using accuracy, and partial transfer with the searched strategy.It applies the searched configuration to the novel support set instead of freezing the backbone or using only conventional transfer operations.
- Problem definition: Few-shot classification uses abundant base-class data and limited novel-class support data to recognize query images from unseen classes.The base and novel label sets are disjoint, and an N-way K-shot task contains N classes with K support images and Q query images per class.
- Framework: P-Transfer searches which feature-extractor layers to freeze or fine-tune and assigns layer-wise learning rates during transfer.A learning rate of zero represents freezing a layer, while nonzero values control fine-tuning.
- Framework integration: P-Transfer can be incorporated into cosine-distance baseline++ and meta-learning frameworks by adjusting the backbone feature extractor during fine-tuning.The method searches transfer strategies for both non-meta and meta-based classification pipelines and tunes the model end-to-end.
- Search space: The search space combines layer-level freezing or fine-tuning with learning-rate assignment, yielding m^K configurations for m choices across K layers.For example, four learning-rate choices produce 4^6 configurations for a Conv6 structure.
- Evolutionary search: Evolutionary search initializes strategy vectors, evaluates them after mini-fine-tuning, and iteratively improves them through mutation, crossover, and TopK selection.The final strategy is the vector with the highest validation accuracy.
4. Experiments
Experiments on mini-ImageNet and CUB evaluate searched partial-transfer schemes across non-meta, meta-learning, and cross-domain settings. The evolutionary strategy generally outperforms fixed or manually designed fine-tuning, with deeper or more discrepant domains requiring broader adaptation.
- Experimental setup: Experiments use mini-ImageNet and CUB for generic, fine-grained, and cross-domain few-shot classification.The study evaluates both meta and non-meta methods, including transfer from mini-ImageNet to CUB.
- Ablation study: Table 2 compares fixed, manually designed, and evolutionary-searched fine-tuning schemes for Conv6.Fixed fine-tunes only the fully connected layer, Manual fine-tunes the last convolutional layer, and Searched follows the evolutionary scheme.
- Ablation study: The evolutionary strategy generally achieves better accuracy than fixing the backbone or using a human-defined strategy.This comparison includes the non-meta Conv6 and ResNet-12 evaluations.
- Ablation study: 40.84±0.8% and 50.95±0.9% are obtained for 1-shot and 5-shot mini-ImageNet when fine-tuning all layers.These values are reported as a baseline for the all-layer fine-tuning setting.
- Searched schemes: Deeper networks and larger base-to-novel domain differences require fine-tuning more layers.Figure 4 visualizes searched schemes, with colored boxes denoting fine-tuned layers and grey boxes denoting frozen layers.
- Final results: Partial transfer consistently outperforms other state-of-the-art methods in both 1-shot and 5-shot settings.The improvement is reported even without DropBlock and label smoothing.
- Traditional transfer learning: Partial reinitialization and fine-tuning of selected layers achieves higher accuracy than inheriting all weights and fine-tuning in ImageNet-to-CUB transfer.This extends the method to the conventional pre-training plus fine-tuning paradigm.
5. Discussions
The discussion argues that complete transfer can retain unnecessary or harmful base-class information, especially when novel data are scarce or domains differ. Partial transfer addresses this by selecting layers and learning rates according to the transfer setting.
- The Essential Role of Transferring in Few-shot: Few-shot transfer must provide suitable knowledge from base to novel classes in both meta and non-meta learning.The discussion frames transferability as a central requirement because novel data are limited.
- The Essential Role of Transferring in Few-shot: Cross-domain mini-ImageNet-to-CUB evaluation finds that domain differences require fine-tuning more layers, while GroupNorm outperforms BatchNorm.Table 4 reports results for both meta and non-meta methods.
- The Essential Role of Transferring in Few-shot: Complete transfer may retain unnecessary or harmful base-class information because novel data are few and sensitive to the feature extractor.The paper presents partial transfer as a response to this issue.
- Why Is Partial Better Than All in Few-shot Learning?: When source and target domains differ in content, conventional transfer learning is no longer applicable in the discussed setting.The paper describes diverse layer-wise learning rates as suitable for this situation.
- Why Is Partial Better Than All in Few-shot Learning?: Fixed transfer is a special case of partial transfer, while diverse learning rates allow different layers to be adapted selectively.The method therefore covers both fully frozen transfer and more flexible layer-wise adaptation.
6. Conclusion
The paper introduces P-Transfer, a partial-transfer approach for few-shot classification that searches how knowledge should be transferred. It reports substantial gains for meta- and non-meta methods in 1-shot and 5-shot settings.
- P-Transfer searches which backbone layers to freeze or fine-tune during few-shot transfer.The search strategy automatically determines the layers requiring adaptation.
- The method improves both meta-learning and non-meta-based methods by a large margin under 1-shot and 5-shot conditions.The reported benefit comes from flexibly adjusting backbone parameters using few support examples.
- The authors identify partial transfer as a promising direction for few-shot classification and traditional transfer learning.
Potential Ethical Impact
The work targets data-scarce applications by using prior labeled knowledge to generalize to new data with few annotations. Its authors caution that failures could produce unreliable conclusions, including misleading medical decisions.
- Few-shot transfer may help build systems where collecting labeled data is costly, including medical imaging and wildlife applications.
- Misclassified medical images could produce unreliable conclusions and mislead doctors.