Source-linked AI summary
Few-Shot Image Recognition by Predicting Parameters from Activations
Siyuan Qiao, Chenxi Liu, Wei Shen, Alan Yuille
TL;DR
Few-shot recognition must handle many novel categories with very few examples while retaining performance on established categories. The paper learns a category-agnostic mapping from activations to classifier parameters, allowing novel-category adaptation without training. On full ImageNet and MiniImageNet, it reports strong novel-category performance while maintaining comparable large-scale-category performance.
Problem
Few-shot learning requires adapting to novel categories with fewer than six examples while preserving large-scale-category performance and minimizing adaptation cost.
Method
A category-agnostic mapping predicts final fully connected-layer parameters from category activation statistics, including novel categories after learning on large-scale categories.
Results
The method achieves state-of-the-art novel-category accuracy by a significant margin on full ImageNet while maintaining comparable large-scale-category performance, and strongly outperforms previous state-of-the-art methods on MiniImageNet.
Takeaways & Limitations
Once learned, the activation-to-parameter mapping enables novel-category parameters to be predicted by a simple forward pass instead of retraining or enumerating the training set.
Takeaways & Limitations
The method's inference formulation assumes a linear parameter mapping to compute an otherwise exponentially large expectation efficiently.
Abstract
from arXiv · showhide
In this paper, we are interested in the few-shot learning problem. In particular, we focus on a challenging scenario where the number of categories is large and the number of examples per novel category is very limited, e.g. 1, 2, or 3. Motivated by the close relationship between the parameters and the activations in a neural network associated with the same category, we propose a novel method that can adapt a pre-trained neural network to novel categories by directly predicting the parameters from the activations. Zero training is required in adaptation to novel categories, and fast inference is realized by a single forward pass. We evaluate our method by doing few-shot image recognition on the ImageNet dataset, which achieves the state-of-the-art classification accuracy on novel categories by a significant margin while keeping comparable performance on the large-scale categories. We also test our method on the MiniImageNet dataset and it strongly outperforms the previous state-of-the-art methods.
1. Introduction
Few-shot learning seeks effective adaptation to novel categories from very few examples without substantially harming established categories. The paper proposes predicting novel-category classifier parameters from category activations, enabling training-free adaptation and fast inference.
- Problem: Few-shot learning aims to classify novel categories from fewer than six examples while preserving performance on large-scale categories and enabling efficient adaptation.The paper frames these as three desired properties: reasonable novel-category accuracy, limited degradation on established categories, and little or zero adaptation training.
- Challenge: Parametric methods often struggle with sparse, imbalanced novel-category data, whereas non-parametric methods require carefully designed distance metrics.This motivates an approach that avoids both direct few-shot parameter learning and empirical metric design.
- Approach: The method re-parameterizes the final fully connected layer by predicting each novel category's parameters from its category-specific activations.For category y, the method uses activations Ay from images belonging to y to predict the fully connected-layer parameters wy.
- Motivation: The approach is motivated by similar spatial structures between category activation means and final-layer parameter vectors.The paper uses t-SNE visualizations to compare average activations and category parameters across mammals, birds, buses, and home appliances.
- Approach: A category-agnostic feedforward mapping learned on large-scale categories predicts parameters for novel categories without retraining the classifier.The same mapping is intended to generalize from Clarge to Cfew after learning on Dlarge.
- Evaluation: Evaluation covers MiniImageNet and full ImageNet, including a 1000-way setting with 900 large-scale and 100 few-shot categories.The full ImageNet evaluation is larger than prior few-shot settings described in the paper.
2. Model
The model learns a category-agnostic predictor that maps activation statistics to classifier parameters, then uses these predictions to incorporate novel categories efficiently. Training samples both individual and mean activations, while inference uses expectation-based parameter prediction and a linear mapping for efficiency.
- Parameter prediction: φ maps activation statistics to fully connected-layer parameters using the same category-agnostic function for large-scale and novel categories.The mapping is learned on Clarge and is expected to generalize to Cfew because activations and parameters share structure.
- Learning parameter predictor: The predictor is trained from classification supervision on Dlarge with a regularized classification loss.The objective includes a regularizer on φ.
- Learning parameter predictor: Training samples either a category mean activation or an individual activation, with probabilities pmean and 1 − pmean, to model few-shot statistics.Each category receives one sampled statistic from Ay ∪ ¯ay.
- Inference: During inference, predicted parameters for all categories are used to classify images, while novel categories evaluate responses from each corresponding sample and retain the maximal response.The category set is C = Clarge ∪ Cfew, and probabilities are computed through SoftMax.
- Inference: Assuming a linear predictor makes the expectation over the exponentially large statistic space computationally efficient.For linear φ, the predictor is represented by a matrix Φ and ES[sy] can be pre-computed; adapting novel categories only updates the corresponding expectation.
- Inference: The predicted category parameter is ˆwy = Φ · sy, and the resulting score is h(sy, a(x)) = ˆwy · a(x) = Φ · sy · a(x).With normalized activations and Φ set to the identity, this score becomes cosine similarity; learning Φ produces a more general similarity metric.
- Implementation details: Experiments use ILSVRC 2015 with 900 categories in Dlarge and 100 categories in Dfew, training a 50-layer ResNet and precomputing image and mean activations.The activation a(x) is taken from the global average pooling layer.
- Implementation details: The implementation compares one-layer and two-layer predictors, with the nonlinear φ2 yielding slight accuracy improvements and φ2∗ showing strong bias toward Clarge.φ2 uses two 2048-dimensional fully connected layers, with ReLU after the first; outputs are normalized.
3. Related Work
Few-shot learning developed alongside increasingly large image datasets but remains difficult because limited examples constrain learning. Prior methods include Bayesian, meta-learning, metric-based, and fast concept-learning approaches, with some lacking flexibility across large-scale and few-shot settings.
- Dataset evolution: Modern image datasets grew from tens of thousands of images to millions of detailed annotations, including ImageNet and MS COCO.Earlier representative datasets include Caltech-101, Caltech-256, Pascal VOC, and CIFAR-10/100.
- Few-shot learning: Few-shot learning remains at an early stage because of its inherent difficulty, with early work using variational Bayesian and hierarchical Bayesian approaches.A model-agnostic meta-learning method was also proposed for one-shot character recognition.
- Prior methods: Existing state-of-the-art methods on specialized few-shot tasks may not flexibly support both large-scale and few-shot learning because k and m are fixed in their architectures.The paper compares against these methods on their tasks for fair comparisons.
- Prior methods: Metric learning followed by nearest-neighbor classification is applicable but not necessarily optimal for unified large-scale and few-shot learning, while Learning like a Child targets fast adaptation with hundreds of examples per category.Nearest-neighbor methods require carefully designed distance metrics, which can be difficult and empirical; Learning like a Child is described as less effective in the broader setting.
4. Results
Experiments evaluate performance on few-shot and large-scale categories, adaptation cost, inference speed, and activation-parameter relationships. The method achieves strong few-shot accuracy while largely preserving large-scale performance, with fast adaptation and inference across ImageNet and MiniImageNet.
- Experimental setup: The evaluation compares a pre-trained 50-layer ResNet, Learning like a Child, Siamese-Triplet Network, nearest neighbor, and the proposed method.The baseline classifiers start from a 900-category classifier trained on Dlarge, while nearest neighbor uses its pre-trained deep features.
- Full ImageNet Classification: The ImageNet experiments measure 1000-way accuracy using 1, 2, or 3 examples per novel category and assess both Cfew and Clarge.Table 1 varies the percentage of Dlarge used and the number of Dfew samples per category.
- Full ImageNet Classification: Our method shows state-of-the-art accuracies on Cfew without compromising too much the performances on Clarge.Triplet Network sacrifices Clarge accuracy, whereas training-based methods preserve Clarge accuracy but perform poorly on Cfew.
- Full ImageNet Classification: Modeling using Eq. 2 and Eq. 1 shows a tradeoff between Clarge and Cfew.φ2* is biased toward Clarge because training samples only mean activations, yet it still outperforms other baseline methods on Cfew.
- Efficiency Analysis: Our method adapts novel categories in 0.683s and reaches about 6.83ms per image, while nearest-neighbor-style inference requires 37.867ms after optimization.Fine-tuned ResNet and Learning like a Child require about 1.8 hours for one retraining epoch on 4 GPUs; unoptimized nearest-neighbor inference takes 2.3 hours per image.
- MiniImageNet Classification: The proposed method makes no assumptions about the number of reference categories or images, unlike several prior methods excluded from full ImageNet evaluation.The paper reports good results on both full ImageNet and MiniImageNet tasks.
5. Conclusion
The paper proposes predicting novel-category classifier parameters from activations, enabling a unified approach to large-scale and few-shot learning. Experiments on full ImageNet and MiniImageNet report strong improvements while preserving comparable performance on large-scale classes.
- A category-agnostic activation-to-parameter mapping predicts novel-category parameters in a simple forward pass.This avoids retraining parametric models or enumerating training sets in non-parametric approaches.
- The method targets a unified approach that works for both large-scale and few-shot learning.
- On full ImageNet, the method achieves state-of-the-art accuracy on novel categories while maintaining comparable performance on large-scale classes.The full ImageNet setting contains 1000 categories and fewer than 4 training samples for few-shot categories.
- On MiniImageNet, the method outperforms previous state-of-the-art methods by a large margin.