Source-linked AI summary
TapNet: Neural Network Augmented with Task-Adaptive Projection for Few-Shot Learning
Sung Whan Yoon, Jun Seo, Jaekyun Moon
TL;DR
Few-shot learning remains difficult when models must handle unseen tasks from only a few examples. TapNet combines episodic meta-learning of an embedding network and class references with an episode-specific linear projection, and reports state-of-the-art accuracies on Omniglot, miniImageNet, and tieredImageNet.
Problem
Few-shot learning seeks reliable performance on previously unseen tasks from only a small number of examples, but remains challenging when labeled data are scarce or expensive.
Method
TapNet learns an embedding network and per-class reference vectors across episodes, then linearly projects features into a task-specific space for distance-based classification.
Results
TapNet reports state-of-the-art few-shot classification accuracies on Omniglot, miniImageNet, and tieredImageNet across various scenarios.
Takeaways & Limitations
Combining across-task learning with quick task-adaptive projection provides excellent generalization to new tasks and data.
Takeaways & Limitations
Direct comparison with a recently reported method is not made because that method uses a substantially larger Wide ResNet-28-10 extractor and separate pre-training.
Abstract
from arXiv · showhide
Handling previously unseen tasks after given only a few training examples continues to be a tough challenge in machine learning. We propose TapNets, neural networks augmented with task-adaptive projection for improved few-shot learning. Here, employing a meta-learning strategy with episode-based training, a network and a set of per-class reference vectors are learned across widely varying tasks. At the same time, for every episode, features in the embedding space are linearly projected into a new space as a form of quick task-specific conditioning. The training loss is obtained based on a distance metric between the query and the reference vectors in the projection space. Excellent generalization results in this way. When tested on the Omniglot, miniImageNet and tieredImageNet datasets, we obtain state of the art classification accuracies under various few-shot scenarios.
1. Introduction
TapNet addresses the difficulty of few-shot learning through episodic meta-learning and explicit task-dependent conditioning. It combines learned embeddings and per-class references with a task-specific linear projection, achieving state-of-the-art classification accuracies across three image datasets.
- Motivation: Few-shot learning targets previously unseen tasks using only a small number of relevant examples, but remains unreliable despite strong performance with abundant labeled data.The paper emphasizes applications where labeled data are scarce or expensive.
- Background: Episodic meta-learning exposes a model to widely varying tasks containing few labeled examples, preparing it to adapt to new data.Episodes are presented sequentially during meta-training.
- Approach: TapNet adds explicit task-dependent conditioning by linearly projecting embedded features into a new classification space for each episode.Classification then uses distances from per-class reference vectors in the projected space.
- Approach: The embedding network and per-class reference vectors are learned across varying episodes, while the projected classification space is constructed anew for each episode.The references are stand-alone vectors rather than outputs of an embedding function.
- Results: TapNet achieves state-of-the-art few-shot classification accuracies on Omniglot, miniImageNet, and tieredImageNet.The reported generalization results cover various few-shot scenarios.
2. Task-Adaptive Projection Network
TapNet combines an embedding network and learned per-class references with a projection space constructed from each episode’s support examples. The projection aligns matching features and references, while episodic distance-based updates train the network and references across tasks.
- Model Description: TapNet has three elements: an embedding network fθ, per-class reference vectors Φ, and a task-dependent projection M into a new classification space.An episode contains support images with labels and query images with labels.
- Projection Construction: Given a support set, M is constructed so matching embedded features and class references align closely after projection.The projection is computed using the support-set embeddings and the learned references.
- Few-Shot Testing: At test time, M is recomputed for the presented shots while the learned network and references remain fixed, and queries are classified by projected distances.Training can use more classes per episode than the final few-shot task, requiring reference selection at test time.
- Overall Design: The learned network and references provide across-task learning, while episode-specific M supplies quick task-dependent conditioning for new data.The paper attributes excellent generalization to this combination.
- Projection Construction: The projection seeks separation from nonmatching references while using a simple linear mapping that requires no additional learning.Class-average support embeddings are compared with matching and nonmatching reference vectors.
- Projection Construction: Linear nulling finds M by making projected error vectors zero, with SVD providing a solution based on the matrix of classwise errors.M is a matrix whose columns span the projection space.
- Projection Construction: The projection dimension D can be smaller than the original feature dimension, and the required SVD computation is small compared with typical model complexity.The passage gives complexity O(LN_c^2) for obtaining the projection.
- Episodic Training: During episodic training, class support and query samples are drawn without overlap, query-reference distances in M define the loss, and θ and Φ are updated.The process repeats across episodes with new image classes and queries.
3. Related Work
This section situates TapNet among metric-based, optimization-based, and memory-augmented few-shot learners. It emphasizes TapNet’s task-dependent projection and its relationship to learned reference vectors and external-memory formulations.
- Metric-Based Meta-Learners: Matching Networks and Prototypical Networks classify queries using distances or similarities in learned embedding spaces, with class references derived from labeled examples.Matching Networks use separate query and labeled-sample embeddings, whereas Prototypical Networks use a single embedding function.
- Metric-Based Meta-Learners: TapNet adds explicit task-dependent conditioning by projecting embedded features into a new classification space before measuring distances to per-class references.Unlike prior metric-based learners, its conditioning is performed through a task-specific linear projection.
- Metric-Based Meta-Learners: TapNet uses one embedding network, learnable stand-alone reference vectors, and a task-conditioning projection, rather than treating embedded support examples as class references.The references are not directly coupled to input images, unlike the class representations in Matching Networks and Prototypical Networks.
- Metric-Based Meta-Learners: TADAM also uses task-dependent conditioning, but TapNet computes its projection directly from features of the new task and current references rather than using extra fully connected networks.The comparison concerns how task conditioning is constructed, not the overall episodic-learning framework.
- Memory-Augmented Neural Networks: TapNet is closely connected to MANN because cosine-based distances between projected queries and references yield a form analogous to MANN’s similarity computation with learnable per-class weights.This correspondence identifies TapNet’s reference matrix with MANN’s learned weight matrix and relates embedded image features to MANN’s key vector.
- Optimization-Based Meta-Learners: Optimization-based meta-learners instead rapidly adapt the embedding network through fine-tuning, as in meta-learner LSTM, MAML, Reptile, Platipus, and LEO.These approaches prepare or optimize parameters so that a learner can adapt to a new task with few updates.
4. Experiment Results
Experiments evaluate TapNet across Omniglot, miniImageNet, and tieredImageNet using episodic training and distance-based classification. TapNet achieves leading few-shot results, while projection-space analyses examine reference separation, alignment, and dimensionality.
- Datasets and evaluation: Experiments use Omniglot, miniImageNet, and tieredImageNet benchmarks with few-shot classification accuracies reported in Tables 1 and 2.Omniglot contains 1623 handwritten-character classes; miniImageNet uses 64 training, 16 validation, and 20 test classes; tieredImageNet contains 608 classes grouped into 34 categories.
- Training protocol: TapNet is meta-trained with higher-way episodes before testing on smaller-way tasks, selecting the required subset of learned reference vectors.Training uses 60-way episodes for 20-way Omniglot and 20-way episodes for 5-way miniImageNet and tieredImageNet.
- Classification results: TapNet achieves the best reported performance on both 1-shot and 5-shot Omniglot and tieredImageNet tasks.On miniImageNet, TapNet has the best 1-shot accuracy and 5-shot accuracy comparable to TADAM-TC because the confidence intervals overlap.
- Projection-space analysis: In the projection space, images lie close to matching reference vectors and tend to remain separated from non-matching references.The non-matching references are included in the modified reference vectors used to define the error vector nulled during projection.
- Projection-space dimensionality: Test accuracy rises rapidly near projection dimensionality 50, peaks slightly around 200, and motivates using D = 200 for 5-way 5-shot miniImageNet.The full dimensionality is 492 during training and 507 during testing, but the reported exception uses D = 200.
5. Conclusions
TapNet combines task-specific null-space projection with an embedding network and learned per-class references for few-shot learning. This combination generalizes to new tasks and achieves state-of-the-art accuracies on standard image datasets.
- TapNet uses a linear transformer to perform task-specific null-space projection of network outputs.
- The embedding network and per-class references are learned across episodes, while the projection is specific to each episode.
- The resulting combination shows excellent generalization to new tasks.
- TapNet achieves state-of-the-art few-shot classification accuracies on standard image datasets.
Few-Shot Learning: Supplementary Material
The supplementary material lists Sung Whan Yoon, Jun Seo, and Jaekyun Moon as authors.
- Sung Whan Yoon is listed as an author.
- Jun Seo is listed as an author.
- Jaekyun Moon is listed as an author.
1. Architecture Details
TapNets use ResNet-12 as their feature extractor, with four residual blocks and four max-pooling layers.
- TapNets use ResNet-12, a residual network with 12 convolutional layers, as the feature extractor.
- ResNet-12 contains four residual blocks and four max-pooling layers.
- Each residual block has three 3 × 3 convolutional layers, followed by batch normalization and ReLU activation.
2. Ablation Study
The ablations examine episode composition, optimization, regularization, and reference-vector settings, with results showing sensitivity to reference construction and initialization choices.
- Ablation Study: The ablation studies cover training-episode composition, learning-rate optimization, regularization hyperparameters, and reference-vector settings.
- Episode Composition: 20 training classes with 8 query samples per class and 25 training classes with 6 query samples per class produce the best accuracy; TapNet experiments use 20/8.
- Reference Vectors: 75.53% accuracy using original references for projection-space construction is lower than 76.36% using modified references.
- Reference Selection: Random reference selection reaches 75.78% accuracy, slightly below the main selection based on references closest to class averages.
- Reference Initialization: Initializing reference vectors with a norm as large as fully meta-trained vectors lowers performance slightly to 75.78%.
3. Experimental Results for Varying Network Sizes
Across varying embedding-network sizes, TapNet generally achieves the best or near-best few-shot accuracy on miniImageNet and tieredImageNet, with performance depending on model complexity and training setup.
- Embedding networks: Conv4 is a four-block CNN with 3×3 convolutions, batch normalization, ReLU activations, and 2×2 max-pooling.This architecture is denoted Conv4 and is widely used in prior few-shot learning work.
- Embedding networks: ResNet-12-small uses four residual blocks with 64, 96, 128, and 256 channels, each built from three 3×3 convolutional layers.A 1×1 convolutional layer provides the residual connection for each block.
- Network-size comparison: Model size and complexity produce significant performance differences among the evaluated few-shot learners.The miniImageNet results are grouped by Conv4, ResNet-12-small, and ResNet-12 feature extractors.
- 5-way miniImageNet: TapNet consistently achieves the best or comparable 5-way miniImageNet accuracy when evaluated with the same base network.It is best for 1-shot classification across the considered network groups and comparable to TADAM-TC for 5-shot ResNet-12 results.
- 5-way tieredImageNet: TapNet achieves the best 5-shot and second-best 1-shot accuracy among methods using the same Conv4 embedding on tieredImageNet.The experiment adds average pooling, and 4-shot meta-training is used for 1-shot classification.
4. Number of Network Parameters
The paper analyzes learner complexity by examining the number of network parameters and other directly related learnable components.
- Parameter accounting: The parameter analysis focuses on convolutional layers and other important learnable parts directly related to network learning effort.These parts include fully connected layers and stand-alone linear weights for classification when present.
- Parameter accounting: The comparison considers fully connected layers in some methods and stand-alone linear classification weights as additional learnable components.The passage indicates that these components are counted alongside convolutional parameters.