Source-linked AI summary
Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, Xiaolong Wang
TL;DR
Few-shot learning research has not clearly established whether meta-learning is more effective than whole-classification, despite meta-learning's training–testing alignment and whole-classification's competitive embeddings. The paper introduces Meta-Baseline, which meta-learns on the evaluation metric of a whole-classification-pre-trained model, and finds competitive performance on standard benchmarks while identifying a trade-off between their objectives.
Problem
The effectiveness of meta-learning relative to whole-classification in few-shot learning remains unclear because their edge is underexplored.
Method
Meta-Baseline first trains a Classifier-Baseline on all base classes, removes its last fully connected layer, and then meta-learns with the cosine nearest-centroid evaluation metric.
Results
Meta-Baseline achieves competitive performance to state-of-the-art methods on standard benchmarks and shows a potential trade-off between meta-learning and whole-classification objectives.
Takeaways & Limitations
The results suggest that class transferability and dataset similarity to novel classes should receive more attention when comparing few-shot learning objectives.
Takeaways & Limitations
When base classes sufficiently cover the novel-class distribution, novel-class generalization may converge to base-class generalization and favor meta-learning.
Abstract
from arXiv · showhide
Meta-learning has been the most common framework for few-shot learning in recent years. It learns the model from collections of few-shot classification tasks, which is believed to have a key advantage of making the training objective consistent with the testing objective. However, some recent works report that by training for whole-classification, i.e. classification on the whole label-set, it can get comparable or even better embedding than many meta-learning algorithms. The edge between these two lines of works has yet been underexplored, and the effectiveness of meta-learning in few-shot learning remains unclear. In this paper, we explore a simple process: meta-learning over a whole-classification pre-trained model on its evaluation metric. We observe this simple method achieves competitive performance to state-of-the-art methods on standard benchmarks. Our further analysis shed some light on understanding the trade-offs between the meta-learning objective and the whole-classification objective in few-shot learning.
1. Introduction
Few-shot learning addresses the challenge of learning new concepts from few labeled examples, but the relative effectiveness of meta-learning and whole-classification remains unclear. The paper introduces Meta-Baseline, which combines whole-classification pre-training with meta-learning on its evaluation metric, and analyzes their trade-off.
- Few-shot learning aims to train networks to understand new concepts from a few labeled examples.
- Meta-learning samples N-way K-shot tasks and optimizes performance on query samples given support samples, aligning training and testing objectives.Each task contains N classes, K support samples per class, and Q query samples per class.
- Whole-classification can produce embeddings comparable to or better than many meta-learning algorithms, leaving meta-learning's effectiveness unclear.
- Meta-Baseline performs meta-learning over a converged whole-classification model using its cosine nearest-centroid evaluation metric.The method resembles MatchingNet or ProtoNet with an additional classification pre-training stage.
- The analysis suggests a trade-off: meta-learning may favor N-way K-shot performance, whereas whole-classification may favor stronger class transferability.The paper identifies improving class transferability as a likely advantage of whole-classification pre-training before meta-learning.
- Meta-Baseline achieves competitive performance to state-of-the-art methods on standard benchmarks without complex design choices.
2. Related Work
Prior few-shot learning work is dominated by meta-learning, while whole-classification models have challenged its effectiveness. This paper positions Meta-Baseline at their intersection and uses it to study their objective trade-off.
- Meta-learning approaches include memory-based, optimization-based, and metric-based methods for adapting to novel few-shot tasks.Examples include LSTM-based meta-learners, MAML, MetaOptNet, and Prototypical Networks.
- Whole-classification methods train on the complete base-class label set and have achieved competitive performance against meta-learning methods.Cosine classifier and Baseline++ adapt whole-classification models to novel-class few-shot tasks using nearest centroids or fine-tuning.
- The effectiveness of meta-learning relative to whole-classification remains unclear because their boundary is underexplored.
- The paper reports a potential trade-off between meta-learning and whole-classification objectives to clarify their comparison.
- Meta-Baseline applies meta-learning to a converged Classifier-Baseline using the cosine nearest-centroid evaluation metric.Classifier-Baseline trains with a linear top layer, removes that class-dependent layer, and performs cosine nearest-centroid classification at test time.
3. Method
The method combines whole-classification pretraining with episodic meta-learning evaluated through cosine nearest-centroid classification. Meta-Baseline further optimizes the pretrained encoder on few-shot tasks, while scaling cosine logits with a learnable scalar.
- Classifier-Baseline: Classifier-Baseline trains on all base classes, removes the final fully connected layer, and uses the resulting encoder for few-shot classification.At test time, support embeddings form class centroids, and queries are assigned using cosine nearest-centroid classification.
- Classifier-Baseline: Support samples are averaged within each class to produce the centroid used for query classification.
- Meta-Baseline: Meta-Baseline samples N-way K-shot tasks after classification pretraining and optimizes query cross-entropy using support-derived centroids.Each batch can contain several tasks, with the average task loss used for training.
- Meta-Baseline: Meta-Baseline further optimizes a converged Classifier-Baseline on its evaluation metric rather than replacing the whole-classification stage.
- Meta-Baseline: A learnable scalar τ scales cosine similarity before Softmax because cosine logits lie in [−1, 1].
- Motivation: The method is designed to test whether the meta-learning objective remains effective over a whole-classification model.
4. Results on Standard Benchmarks
Meta-Baseline is evaluated on miniImageNet, tieredImageNet, and ImageNet-800, where it achieves competitive performance with a simple design. The meta-learning stage improves Classifier-Baseline in some settings, especially 1-shot evaluation.
- The experiments use average 5-way accuracy, with 95% confidence intervals reported for benchmark comparisons.Testing uses consistent sampling, including a fixed set of 800 testing tasks when confidence intervals are omitted.
- Meta-Baseline achieves competitive performance to state-of-the-art methods on miniImageNet and tieredImageNet.The comparison reports average 5-way accuracy with 95% confidence intervals.
- The Classifier-Baseline also performs competitively with meta-learning methods, especially in 5-shot tasks.
- On tieredImageNet, the gap between Meta-Baseline and Classifier-Baseline is smaller, and meta-learning does not improve 5-shot performance.
- On ImageNet-800, Meta-Baseline improves Classifier-Baseline in 1-shot but not in 5-shot.This experiment uses the larger ImageNet-800 dataset; freezing Batch Normalization is beneficial in this setting.
5. Observations and Hypothesis
The experiments indicate a trade-off between meta-learning and whole-classification: meta-learning can improve base-task performance while reducing novel-class generalization, whereas whole-classification strengthens class transferability. Meta-Baseline balances these objectives by calibrating a whole-classification embedding with meta-learning while retaining its metric.
- Objective discrepancy in meta-learning: During meta-learning, increasing base class generalization can coincide with decreasing novel class generalization.This pattern is observed through evaluations that separately measure base-class and novel-class few-shot tasks.
- Objective discrepancy in meta-learning: The discrepancy is attributed to class-level overfitting caused by different base-class and novel-class distributions, rather than sample-level overfitting.
- Trade-off between objectives: Whole-classification pre-training provides stronger class transferability that significantly helps novel class generalization.Meta-Baseline trained without classification pre-training has higher base-class generalization but much lower novel-class generalization.
- Effect of dataset properties: Meta-learning is likely most effective after whole-classification training when novel classes are similar to base classes.Shuffling tieredImageNet classes to bring base and novel distributions closer makes the meta-learning improvement significantly larger.
- Effect of dataset properties: Meta-learning improves less in 5-shot than in 1-shot, while larger datasets also show smaller improvements.The authors hypothesize that averaging more support embeddings reduces the potential benefit of meta-learning and that whole-classification transferability becomes stronger at larger scale.
- The trade-off between meta-learning and whole-classification: Meta-Baseline balances the trade-off by calibrating whole-classification embeddings with meta-learning and inheriting a metric with high initial performance.The paper suggests that retaining a good metric limits later modifications to the pre-trained embedding and helps preserve class transferability.
- The trade-off between meta-learning and whole-classification: The trade-off may weaken when base classes sufficiently cover or resemble the target novel-class distribution.In that setting, novel-class generalization should converge toward base-class generalization, and meta-learning methods may outperform whole-classification baselines again.
6. Additional Results on Meta-Dataset
Additional Meta-Dataset experiments evaluate Classifier-Baseline and Meta-Baseline under variable-way, variable-shot tasks, including ILSVRC-only and multi-dataset training settings.
- Meta-Dataset evaluates models with variable numbers of ways and shots across diverse datasets.The experiments use the Meta-Dataset benchmark setting and sample 1000 tasks for evaluating Classifier-Baseline or Meta-Baseline.
- Under ILSVRC-only training, Meta-Baseline does not significantly improve Classifier-Baseline, possibly because the average number of shots is high.
- The multi-dataset setting trains Classifier-Baseline as an encoder with multiple dataset-specific fully connected layers.
- Datasets less relevant to ILSVRC show greater preference for meta-learning than datasets with similar properties to ILSVRC.The authors suggest that sampling ILSVRC with 0.5 probability makes meta-learning harder to improve on ILSVRC-like datasets.
7. Conclusion and Discussion
The paper presents Meta-Baseline as a simple, competitive approach and identifies a potential objective discrepancy between meta-learning and whole-classification. Dataset similarity may influence which objective is preferred.
- Meta-Baseline is competitive with state-of-the-art methods on standard benchmarks without additional parameters or complex design choices.
- Experiments indicate that better generalization to unseen tasks from base classes might coincide with worse performance on tasks from novel classes.
- Meta-learning remains effective when applied over whole-classification models, despite reports that whole-classification alone can suffice.
- The preference between meta-learning and whole-classification changes with class similarity and other dataset factors.The authors suggest these factors deserve more attention in future model comparisons.
A. Details of ResNet-12
The ResNet-12 backbone uses four residual blocks followed by global average pooling to produce a 512-dimensional feature vector, while omitting some optional backbone enhancements for simplicity.
- The ResNet-12 backbone has four residual blocks, each containing three convolutional layers with 3 × 3 kernels, batch normalization, and Leaky ReLU.The blocks use 64, 128, 256, and 512 channels, with 2×2 max-pooling after each block.
- A final 5 × 5 global average pooling layer produces a 512-dimensional feature vector.
- The architecture omits modifications such as DropBlock and wider channels to preserve simplicity.The authors note that these modifications may increase performance.
B. Training plot of Meta-Baseline without classification training stage
Training Meta-Baseline without classification pre-training produces higher base-class generalization but poorer peak novel-class generalization, especially after learning-rate decay.
- When the learning rate decays, novel-class generalization quickly begins to decrease during training from scratch.
- The from-scratch model achieves higher base-class generalization than Meta-Baseline with classification training, but its highest novel-class generalization is much worse.
- The result suggests whole-classification training may provide representations with extra class transferability.
C. Comparison to cosine classification training
The comparison finds that cosine versus linear classification training has dataset- and shot-dependent effects, while stronger base-class generalization can coincide with weaker novel-class generalization.
- Cosine outperforms Linear in 1-shot on miniImageNet but performs worse in 5-shot.
- Linear outperforms Cosine in both 1-shot and 5-shot on tieredImageNet.
- The authors choose the linear layer because it is more common and works better in more cases.
- On ImageNet-800, base-class generalization increases during training while novel-class generalization quickly decreases for both ResNet-18 and ResNet-50.
E. Comparison of the Classifier-Baseline and Baseline++ [1]
The Classifier-Baseline ablation connects it to Baseline++ and shows that simple nearest-centroid classification with cosine distance can outperform fine-tuning, while optimizer choice substantially affects performance.
- Fine-tuning is outperformed by the simple nearest-centroid method with cosine metric.
- Using a standard ImageNet-like optimizer significantly improves the whole-classification method for few-shot learning.
- Table 10 compares the Classifier-Baseline with Baseline++ [1].