Source-linked AI summary
Incremental Few-Shot Learning with Attention Attractor Networks
Mengye Ren, Renjie Liao, Ethan Fetaya, Richard S. Zemel
TL;DR
Incremental few-shot learning asks how to add novel classes from few examples while preserving performance on pre-trained base classes without revisiting the original data. The paper uses an attention-attractor regularizer learned through recurrent back-propagation while solving each episodic classifier to convergence, and reports state-of-the-art results on mini-ImageNet and tiered-ImageNet.
Problem
Incremental few-shot learning must add novel classes from few examples while preserving base-class performance without access to the original training data.
Method
The Attention Attractor Network meta-learns a regularizer over a converged episodic classifier by attending to base classes according to novel-class interference.
Results
The method produces state-of-the-art results on incremental few-shot learning tasks using mini-ImageNet and tiered-ImageNet.
Takeaways & Limitations
Recurrent back-propagation provides an effective modular tool for learning regularizers in meta-learning settings with iterative few-shot optimization.
Takeaways & Limitations
The model assumes the optimal parameters W* are unique and achievable.
Abstract
from arXiv · showhide
Machine learning classifiers are often trained to recognize a set of pre-defined classes. However, in many applications, it is often desirable to have the flexibility of learning additional concepts, with limited data and without re-training on the full training set. This paper addresses this problem, incremental few-shot learning, where a regular classification network has already been trained to recognize a set of base classes, and several extra novel classes are being considered, each with only a few labeled examples. After learning the novel classes, the model is then evaluated on the overall classification performance on both base and novel classes. To this end, we propose a meta-learning model, the Attention Attractor Network, which regularizes the learning of novel classes. In each episode, we train a set of new weights to recognize novel classes until they converge, and we show that the technique of recurrent back-propagation can back-propagate through the optimization process and facilitate the learning of these parameters. We demonstrate that the learned attractor network can help recognize novel classes while remembering old classes without the need to review the original training set, outperforming various baselines.
1 Introduction
Incremental few-shot learning combines pre-trained knowledge of common classes with few-shot adaptation to novel classes, without harming recognition of the original classes. The paper proposes an attention-attractor regularizer and evaluates it on mini-ImageNet and tiered-ImageNet.
- Motivation: Few-shot learning uses a few labeled examples for unseen classes but typically ignores common classes already available in large datasets.This limits practical usability when systems must recognize both existing and newly added concepts.
- Problem setting: Incremental few-shot learning adds novel classes without catastrophic forgetting, often without access to the original training data.The setting combines incremental learning for preserving base classes with few-shot learning for data-scarce novel classes.
- Method: The proposed method meta-learns a regularizer that reduces catastrophic forgetting during incremental few-shot learning.The regularizer is inspired by attractor networks and acts as a memory of base classes adapted to novel classes.
- Results: The method is empirically evaluated on incremental few-shot learning tasks using mini-ImageNet and tiered-ImageNet.The paper reports state-of-the-art results on both benchmarks.
2 Related Work
Related work spans metric-learning and adaptive meta-learning approaches, while incremental few-shot learning addresses adding classes to a pre-trained classifier. The paper instead learns a converged episodic classifier with an attention-attractor regularizer and recurrent back-propagation.
- Few-shot learning: Metric-learning methods map inputs into embeddings where same-category examples are close and different-category examples are far apart.Examples include Deep Siamese, Matching, and Prototypical Networks.
- Meta-learning: Adaptive meta-learning methods learn parameter updates, initialization, or memory mechanisms across episodic tasks.MAML back-propagates through gradient steps, while other methods use recurrent learners or augmented memory.
- Meta-learning: Meta-learning methods can suffer from short-horizon bias when test-time training runs for more steps than during meta-training.Logistic regression offers a closed-form alternative, whereas recurrent back-propagation supports more general continuous optimizers.
- Incremental learning: Incremental learning methods address catastrophic forgetting through stored examples, parameter regularization, or generative models, but commonly start from scratch.These approaches often perform worse than models trained with all available classes.
- Incremental few-shot learning: Prior incremental few-shot methods start from pre-trained base-class networks and augment them with novel classes using gradients, prototypes, hallucination, or attention.The paper differs by directly solving a randomly initialized classifier to convergence and adding a learned regularizer.
3 Model
The model learns novel-class classifiers episodically while preserving base-class performance through an attention-based attractor regularizer. Meta-learning optimizes this regularizer through the converged few-shot optimization using recurrent back-propagation.
- Incremental Few-Shot Learning: The method first pretrains a fixed feature representation and base classifier, then learns novel-class classifiers in episodic few-shot tasks.Base classifier parameters remain fixed after pretraining, while episode-specific fast weights are learned from novel-class support examples.
- Incremental Few-Shot Learning: Joint evaluation combines base and novel classes, testing whether a classifier trained only on novel support examples retains performance on base classes.A base-class mini-batch is added to the novel query set to form the joint evaluation set.
- Learning via Recurrent Back-Propagation: Recurrent back-propagation differentiates through the fast-weight optimization so meta-parameters can be updated after the novel-class weights converge.For logistic regression, convexity of the regularized objective ensures a unique achievable optimum under the stated formulation.
- Attention Attractor Networks: Direct few-shot optimization can cause catastrophic forgetting because novel-class weights may dominate base classes in joint prediction.The attractor regularizer is introduced specifically to address this failure mode.
- Attention Attractor Networks: The regularizer penalizes distance between each novel classifier weight and an attractor, biasing learning toward information encoded from base classes.The attractor regularization can be extended layer-wise for multilayer perceptrons.
- Attention Attractor Networks: Each novel class attends to base classes using support-set representations and cosine similarity, producing episode-dependent attractors from a learned memory matrix.The attention weights are normalized with a softmax and combine base-class memory vectors with a learned bias embedding.
4 Experiments
Experiments evaluate incremental few-shot learning on mini-ImageNet and tiered-ImageNet, comparing the proposed attractor-based models with prior methods and ablations. Results show advantages for learned regularization, attention attractors, recurrent back-propagation, and attractor dynamics.
- Datasets and evaluation: Experiments use modified mini-ImageNet and tiered-ImageNet benchmarks, with 64+5-way and 200+5-way results, respectively.mini-ImageNet uses 64, 16, and 20 training, validation, and testing classes; tiered-ImageNet uses categorical splits and is harder for generalization.
- Datasets and evaluation: The evaluation reports Base, Novel, and Both accuracies, plus Δa, Δb, and their average Δ for degradation under joint prediction.The average Δ is defined as 1/2(Δa + Δb).
- Comparisons: The proposed models outperform prior prototype-based methods on both incremental few-shot benchmarks.The comparison includes Prototypical Networks, Weights Imprinting, and Learning without Forgetting.
- Ablation studies: Learned regularization improves joint base-and-novel prediction and reduces degradation relative to manually chosen classifier weight decay.This pattern holds across the ablation studies on mini-ImageNet and tiered-ImageNet.
- Ablation studies: Attention attractors outperform static attractors on mini-ImageNet, while the attention model predicts attractors from episode representations.The attention attractor controls weights below the final layer in MLP models through an MLP applied to average episodic representations.
- Optimization comparison: RBP models remain effective when the inner optimization reaches convergence, whereas T-BPTT performance drops significantly at test-time convergence.T-BPTT can match RBP with early stopping, but choosing the stopping point is difficult; RBP supports full episodic training efficiently.
5 Conclusion
The paper concludes that incremental few-shot learning improves model flexibility by jointly handling predefined and novel concepts. It presents an attention attractor model and reports advantages for convergent iterative learning and recurrent back-propagation over stated baselines.
- Incremental few-shot learning jointly predicts predefined and additional novel concepts, supporting more flexible and usable machine-learning models.
- The attention attractor model regulates each episode’s training objective by attending to the base classes.
- The iterative model, which solves the few-shot objective to convergence, performs better than baselines using one-step inference.
- Recurrent back-propagation is effective for learning in a general meta-learning setting, whereas truncated back-propagation through time fails to learn well-converging functions.
A Regular Few-Shot Classification
On regular 5-way few-shot classification without base classes, the full model performs similarly to standard competitive approaches. The results indicate that its learned regularizer primarily addresses interference between base and novel classes.
- The full model shows similar performance to competitive meta-learning approaches on regular few-shot classification.
- The learned regularizer mainly solves interference between base and novel classes.
- Regular 5-way few-shot classification is evaluated on mini-ImageNet without base classes.
B Visualization of Few-Shot Episodes
The paper includes additional few-shot episode visualizations comparing its method with Dynamic Few-Shot Learning without Forgetting.
- Additional visualizations of few-shot episodes highlight differences between the proposed method and Dynamic Few-Shot Learning without Forgetting.
C Visualization of Attention Attractors
Across 20 mini-ImageNet episodes, learned attention attractors form rough semantic clusters, while a static attractor remains centered among them.
- Across 20 episodes and five semantic mini-ImageNet classes, learned attention attractors roughly form semantic clusters.
- The static attractor remains at the center of all attractors.
D Dataset Statistics
This section provides additional details about the datasets used in the experiments.
- The experiments use dataset-specific splits and statistics described in the following sections.These details support evaluation of the incremental few-shot learning setting.
D.1 Validation and testing splits for base classes
Unlike standard few-shot learning, incremental evaluation requires additional validation and test splits for assessing base-class predictions. The paper reports these splits and related statistics for mini-ImageNet and tiered-ImageNet.
- Incremental evaluation requires additional validation and test splits from the meta-training set to evaluate base-class predictions.Standard few-shot learning instead uses disjoint meta-training, validation, and test class sets.
- Mini-ImageNet provides Train-Val and Train-Test images for evaluating the training set, while tiered-ImageNet reserves approximately 20% of images for base-class validation and testing.
- Split statistics for mini-ImageNet and tiered-ImageNet are listed in Table 9, with full ablation results reported separately in Tables 7 and 8.
D.2 Novel classes
The experiments emulate novel-class learning differently across mini-ImageNet and tiered-ImageNet. Mini-ImageNet masks five classes among 64 base classes, whereas tiered-ImageNet separates 200 base classes from 151 novel classes for meta-learning.
- mini-ImageNet: Mini-ImageNet masks five base classes in Wa, effectively training a 59+5 classification task with the same training set for Da and Db.The authors report limited second-stage meta-learning progress because all classes were previously seen.
- tiered-ImageNet: Tiered-ImageNet uses 200 Train-A base classes and 151 Train-B novel classes to emulate second-stage novel-class learning.These splits are used only for meta-learning, while first-stage classifier training uses Train-A-Train data.
- tiered-ImageNet: Each tiered-ImageNet meta-learning episode samples few-shot examples from Train-B and a query base set from Train-A-Val.