Source-linked AI summary
Dense Classification and Implanting for Few-Shot Learning
Yann Lifchitz, Yannis Avrithis, Sylvaine Picard, Andrei Bursuc
TL;DR
Few-shot learning aims to recognize unseen classes from very few examples, where deep networks face overfitting and generalization challenges. The paper proposes dense classification over feature maps and neural implants for task-specific adaptation, reporting improvements over prior few-shot methods on miniImageNet.
Problem
Few-shot learning seeks recognition of previously unseen classes from very few annotated examples, but low-data training creates overfitting and generalization challenges.
Method
The method applies classification at all spatial locations during base-class training, then freezes the embedding and trains new task-specific implant layers on novel-class support data.
Results
Dense classification and implanting both improve the cosine-classifier baseline and outperform previous few-shot learning methods on 5-way miniImageNet classification.
Takeaways & Limitations
The resulting architecture uses localized supervision and limited new parameters to adapt a pretrained embedding to new tasks.
Takeaways & Limitations
The described implant-training setup does not apply to 1-shot learning because it requires at least two support samples per class.
Abstract
from arXiv · showhide
Training deep neural networks from few examples is a highly challenging and key problem for many computer vision tasks. In this context, we are targeting knowledge transfer from a set with abundant data to other sets with few available examples. We propose two simple and effective solutions: (i) dense classification over feature maps, which for the first time studies local activations in the domain of few-shot learning, and (ii) implanting, that is, attaching new neurons to a previously trained network to learn new, task-specific features. On miniImageNet, we improve the prior state-of-the-art on few-shot classification, i.e., we achieve 62.5%, 79.8% and 83.8% on 5-way 1-shot, 5-shot and 10-shot settings respectively.
1. Introduction
Few-shot learning addresses recognition of unseen classes from very few annotations, where standard deep networks face low-data challenges. The paper proposes retaining a conventional parametric pipeline while adding dense classification and neural implants.
- Deep neural networks power many vision tasks but typically require vast amounts of annotated data to fit.
- Few-shot learning targets recognition of previously unseen classes using very few annotated examples.
- A conventional pipeline using all available classes and data with a parametric classifier is presented as effective and appealing.
- Dense classification applies supervision over feature maps to address the spatial information discarded by flattening or pooling.
- Neural implants attach layers to an already trained network so it can adapt to new tasks with few examples.
2. Problem formulation and background
The problem is formulated as learning a base-class representation and then adapting a classifier to novel classes from limited support data. The background covers spatial embeddings, confidence-based classifiers, episodic evaluation, and prototype- or weight-based adaptation.
- Stage 1 learns a domain representation from examples and labels belonging to a set of base classes.
- In stage 2, a classifier learns to map new query examples to labels from novel classes using few support examples.
- Few-shot evaluation commonly uses c′-way tasks with k support examples per novel class and randomly sampled queries.
- The embedding network maps inputs to spatial feature tensors, with r denoting spatial dimensions and d denoting feature dimensions.
- A classifier applied to the embedding outputs confidence scores over base or novel classes, and prediction selects the class with maximum confidence.
- Prototypical networks classify queries by comparing them with one averaged support-set prototype per class.
- Imprinted weights replace the classifier weights with base and prototype weights, then fine-tune using the entire base training data.
- Few-shot learning without forgetting stores base-class weights rather than the entire training data while using a parametric classifier in both stages.
3. Method
The method preserves spatial structure during base-class training through dense classification, then adapts to novel few-shot tasks by training implanted layers while keeping the learned embedding fixed. Dense classification supervises each feature-map location, whereas inference pools support embeddings for prototype construction.
- Dense classification: Dense classification applies shared class weights independently at every spatial location instead of flattening or globally pooling the embedding.The embedding is treated as vectors in R^d across spatial locations, with each location compared to the same class weights.
- Dense classification: Each spatial location incurs a classification loss, encouraging correct predictions across the feature map rather than only at the strongest response.The assumption is that the image-level label applies to all locations because the feature map has low resolution and large receptive fields.
- Dense classification: Dense classification produces smoother, more object-aligned activation maps and encourages the embedding to identify multiple object parts.The authors describe this as implicit augmentation from exhaustive shifts and crops represented by locations on a dense grid.
- Implanting: The method trains implants as new layers for novel tasks while freezing the base embedding network and recomputing prototypes using the implanted representation.Implants add task-specific features through a parallel processing stream connected to the base network.
- Implanting: The implant training setup does not apply to 1-shot tasks because prototype-based training requires at least two support samples per class.During training, each example is treated as a query while the remaining examples form the support set.
- Inference: At inference, support embeddings are globally pooled to compute novel-class prototypes, and query predictions average classification scores across spatial locations.The implanted and non-implanted models use the same prototypical-network inference procedure.
4. Related work
The paper situates its parametric-classifier approach among metric-learning, meta-learning, and network-adaptation methods, while relating neural implants to widening networks with frozen prior weights.
- Metric-learning approaches modify softmax or compare samples, whereas this work uses a parametric classifier related to prototypical and matching networks.
- Meta-learning methods include optimization-, memory-, data-generation-, and parameter-generation-based approaches for adapting models to few-shot tasks.
- Implants operate at multiple depth levels, using previously learned features while learning task-specific features during fine-tuning.
- Neural implants add new neurons in parallel to existing ones and freeze the old weights, relating them to prior network-adaptation methods.
5. Experiments
Experiments on miniImageNet and FC100 evaluate dense classification and implanting with ResNet-12 under standardized few-shot protocols. Dense classification improves stage-1 training, implanting adds further gains on miniImageNet, while gains are smaller on FC100.
- Dense classification: Dense classification improves ResNet-12 accuracy on novel classes, whereas it is detrimental for the smaller C128F network.The authors attribute this contrast to differences in depth and receptive-field size.
- Dense classification: Dense classification during stage 1 outperforms global average pooling by a large margin across the evaluated pooling strategies.The comparison is conducted on 5-way 5-shot miniImageNet classification with ResNet-12.
- Implanting: Average pooling for both support examples and queries is the best stage-2 pooling choice and is used in subsequent experiments.This result comes from evaluating pooling strategies with implanting on miniImageNet.
- Ablation study: Implanting further improves dense-classification results, while widening the last residual block by 16 channels does not improve novel-class performance.The widening baseline tests whether gains arise merely from more parameters or feature dimensionality.
- Comparison with the state-of-the-art: The best solutions outperform prior methods by a large margin on miniImageNet 1-, 5-, and 10-shot classification and exceed TADAM by at least 3% in all settings.On FC100, the model also outperforms TADAM, but by a smaller margin; the authors associate the reduced difference with CIFAR-100’s lower resolution.
6. Conclusion
The paper combines dense classification with neural implants to improve few-shot adaptation. Its resulting architecture outperforms previous methods by a large margin and sets a new state of the art on standard benchmarks.
- Conclusion: Dense classification applies supervision independently at each spatial location, improving activation-map distribution and performance on new tasks.The reported benefit comes with deeper architectures and high-dimensional embeddings.
- Conclusion: Neural implants adapt the network to new tasks by adding limited new parameters without changing the original embedding.The implants are trained for task-specific features from limited novel-class examples.
- Conclusion: The combined architecture outperforms previous methods by a large margin and sets a new state of the art on standard benchmarks.