Source-linked AI summary
Dynamic Few-Shot Visual Learning without Forgetting
Spyros Gidaris, Nikos Komodakis
TL;DR
The paper addresses how vision systems can learn novel categories from only a few test-time examples without forgetting base categories. It introduces an attention-based classification-weight generator and a cosine-similarity ConvNet classifier. The approach achieves state-of-the-art few-shot results while maintaining base-category recognition and also performs strongly on a second benchmark.
Problem
Few-shot vision systems need to learn novel categories from few examples while retaining accurate recognition of base categories without retraining.
Method
The system generates novel-category classification weights with attention over base-category weights and uses cosine similarity between features and classification weights.
Results
The approach achieves excellent few-shot recognition accuracy that surpasses prior state-of-the-art approaches and maintains high recognition accuracy on base categories.
Takeaways & Limitations
Cosine-similarity classification unifies base- and novel-category recognition and produces feature representations with better generalization capabilities.
Takeaways & Limitations
With only a single training example, simple averaging cannot infer an accurate novel-category classification weight vector.
Abstract
from arXiv · showhide
The human visual system has the remarkably ability to be able to effortlessly learn novel concepts from only a few examples. Mimicking the same behavior on machine learning vision systems is an interesting and very challenging research problem with many practical advantages on real world vision applications. In this context, the goal of our work is to devise a few-shot visual learning system that during test time it will be able to efficiently learn novel categories from only a few training data while at the same time it will not forget the initial categories on which it was trained (here called base categories). To achieve that goal we propose (a) to extend an object recognition system with an attention based few-shot classification weight generator, and (b) to redesign the classifier of a ConvNet model as the cosine similarity function between feature representations and classification weight vectors. The latter, apart from unifying the recognition of both novel and base categories, it also leads to feature representations that generalize better on "unseen" categories. We extensively evaluate our approach on Mini-ImageNet where we manage to improve the prior state-of-the-art on few-shot recognition (i.e., we achieve 56.20% and 73.00% on the 1-shot and 5-shot settings respectively) while at the same time we do not sacrifice any accuracy on the base categories, which is a characteristic that most prior approaches lack. Finally, we apply our approach on the recently introduced few-shot benchmark of Bharath and Girshick [4] where we also achieve state-of-the-art results. The code and models of our paper will be published on: https://github.com/gidariss/FewShotWithoutForgetting
1. Introduction
The paper targets dynamic few-shot recognition: learning novel categories from a few test-time examples while retaining recognition of base categories without retraining. It combines an attention-based weight generator with cosine-similarity classification and reports state-of-the-art results across evaluated benchmarks.
- Motivation: Few-shot systems must learn novel categories quickly while preserving recognition accuracy on previously trained base categories.The paper frames this as dynamic few-shot learning without forgetting, using base-category data as the sole input during the initial training phase.
- Method: The attention-based generator uses a few examples of a novel category to produce its classification weight vector.It incorporates attention over base-category classification weight vectors to exploit acquired visual knowledge, especially with a single example.
- Method: The cosine-similarity classifier unifies recognition of base and novel categories and improves feature generalization to novel categories over dot-product classification.The paper identifies this redesign as necessary for incorporating generated novel-category weights into the recognition system.
- Contributions: The proposed system dynamically learns novel categories from few examples without forgetting the base categories on which it was trained.This is presented as the paper’s central object-recognition contribution.
- Evaluation: The approach is evaluated on Mini-ImageNet and the Bharath–Girshick few-shot benchmark, achieving state-of-the-art results.The paper also reports that cosine-similarity-trained feature extractors surpass prior approaches when used for image matching.
2. Related work
Prior few-shot learning work includes meta-learning, metric-learning, and feature-regularization approaches. These methods learn task-specific learners, similarity-preserving representations, or features intended to generalize to unseen categories.
- Meta-learning based approaches: Meta-learning methods train a meta-learner to quickly learn a learner model from a few examples of a new classification task.The cited example uses an LSTM-based meta-learner to generate parameter updates for the learner.
- Metric-learning based approaches: Metric-learning methods learn representations that preserve class-neighborhood structure, placing same-object features closer than different-object features.Examples formulate one-shot recognition as image matching or use differentiable nearest-neighbor classification.
- Other approaches: Bharath and Girshick use an l2 regularization loss on feature representations to improve generalization to unseen categories.The paper contrasts this with its cosine-similarity classifier, which also unifies base- and novel-category recognition.
3. Methodology
The framework combines a ConvNet recognition model with a few-shot weight generator to incorporate novel categories at test time while recognizing base and novel categories together. A cosine-similarity classifier aligns base and generated novel weights, improving generalization to novel categories.
- Framework overview: The system learns base categories from a dataset with substantial training data, then dynamically learns novel categories from a few test-time examples without forgetting base categories.Its two main components are a ConvNet-based recognition model and a few-shot classification weight generator.
- Recognition model: The ConvNet extracts a feature vector and applies category-specific classification weights to produce probability scores.During training, it learns the feature-extractor parameters and base-category weight vectors.
- Few-shot weight generation: The weight generator receives few-shot novel-category features and base-category weights, producing novel weights that are added to the classifier.Using W* = Wbase ∪ Wnovel enables unified recognition of base and novel categories.
- Cosine-similarity classifier: The dot-product classifier is problematic because base weights are gradually learned while novel weights are predicted from input features, causing incompatible magnitudes and raw scores.The method replaces dot products with cosine similarity to address this mismatch.
- Cosine-similarity classifier: Cosine similarity supports unified base-and-novel recognition and produces feature representations that generalize better to novel categories than dot-product features.The learned features form more compact, distinctive category-specific clusters, and the objective resembles metric-learning objectives.
- Few-shot weight generation: Averaging novel examples alone cannot fully exploit the ConvNet’s learned visual knowledge and may be inaccurate with a single example.The attention-based generator instead can compose novel weights from visually related base-category weight vectors.
4. Experimental results
Experiments evaluate the system on Mini-ImageNet and an ImageNet-based benchmark, testing novel-category few-shot recognition alongside retention of base-category accuracy. Ablations show that cosine similarity and attention-based weight generation improve novel-category performance while preserving base-category recognition.
- Evaluation setting: The evaluation measures both few-shot recognition of novel categories and retention of performance on the 64 base categories.Mini-ImageNet uses 64 training, 16 validation, and 20 test categories, with separate evaluation of novel and base categories.
- Ablation study: Cosine-similarity classifiers substantially improve few-shot recognition over dot-product classifiers and generalize better to unseen categories.This comparison is made in the ablation study, including settings where novel-category weights are estimated by feature averaging without additional few-shot-task training.
- Ablation study: Removing the final ReLU improves novel-category accuracy, whereas retaining it increases base-category accuracy.The reported trade-off is based on comparisons between cosine classifiers with and without the final ReLU unit.
- Ablation study: The attention-based weight generator performs especially well in the 1-shot setting, exceeding the averaging-based generator by more than 3 percentage points.Both weight generators significantly boost few-shot recognition, while the generator does not change base-category accuracy, which remains around 70.50%.
- Comparison with state-of-the-art: 74.92% is achieved by the full system, improving on the prior state-of-the-art result of 72.67% in the reported comparison.The full system combines the cosine classifier with the attention-based few-shot weight generator.
- ImageNet-based benchmark: The approach achieves superior performance to prior methods on the ImageNet-based Bharath and Girshick benchmark, exceeding Prototype Matching Net approaches in all but one case.The benchmark evaluation reports mean accuracies and 95% confidence intervals over 100 repeated samples of novel-category training images.
5. Conclusions
The proposed system quickly learns novel categories while retaining recognition of base categories. Its attention-based weight generator and cosine-similarity classifier support unified recognition and strong few-shot performance.
- The system quickly learns novel categories without forgetting the base categories used during training.
- The attention-based weight generator and cosine-similarity classifier unify recognition of novel and base categories.
- The ResNet-10 feature extractor omits its final ReLU non-linearity.
- The approach achieves few-shot recognition accuracy on novel categories that surpasses prior state-of-the-art approaches by a significant margin.
Appendix A. Implementation details of training procedure followed during the 2nd training stage
During the second training stage, the model trains its few-shot weight generator through episodes containing simulated novel categories and remaining base categories. It then jointly classifies test examples from both groups using generated and base-category weights.
- Each training episode samples Knovel fake novel categories with N′ training examples per category.
- Episodes also include Tnovel test examples from fake novel categories and Tbase test examples from the remaining base categories.
- The generator infers a fake novel classification weight vector from each category’s N′ training examples.
- The model classifies T = Tnovel + Tbase test examples using both generated novel-category weights and weights for the remaining base categories.
- Stochastic-gradient-descent routines train the weight generator and base-category classification vectors using batched episodes.