Source-linked AI summary
TADAM: Task dependent adaptive metric for improved few-shot learning
Boris N. Oreshkin, Pau Rodriguez, Alexandre Lacoste
TL;DR
Few-shot learning seeks models that generalize from small labeled sample sets, but similarity metrics and task adaptation remain central design challenges. The paper combines metric scaling, task-conditioned feature extraction, and auxiliary task co-training, achieving strong results across few-shot image-classification benchmarks. Its scaling analysis also identifies a task-, dataset-, and metric-dependent optimal scaling value.
Problem
Few-shot learning requires models to generalize from small labeled sample sets, while suitable similarity metrics and task conditioning remain important open design issues.
Method
The paper scales the similarity metric, conditions the feature extractor on a task representation from the sample set, and uses auxiliary task co-training to learn the resulting task-dependent metric space.
Results
76.7% accuracy is reported on 5-shot, 5-way mini-Imagenet, with improvements of up to 8.5% over Snell et al. and 4.8% over the state of the art.
Takeaways & Limitations
Scaled cosine similarity can perform at par with Euclidean distance, and scaling changes the parameter-update regime under softmax and categorical cross-entropy.
Takeaways & Limitations
The optimal scaling value depends on the dataset, metric, and task, and must be selected empirically because large scaling can reduce the effective batch size.
Abstract
from arXiv · showhide
Few-shot learning has become essential for producing models that generalize from few examples. In this work, we identify that metric scaling and metric task conditioning are important to improve the performance of few-shot algorithms. Our analysis reveals that simple metric scaling completely changes the nature of few-shot algorithm parameter updates. Metric scaling provides improvements up to 14% in accuracy for certain metrics on the mini-Imagenet 5-way 5-shot classification task. We further propose a simple and effective way of conditioning a learner on the task sample set, resulting in learning a task-dependent metric space. Moreover, we propose and empirically test a practical end-to-end optimization procedure based on auxiliary task co-training to learn a task-dependent metric space. The resulting few-shot learning model based on the task-dependent scaled metric achieves state of the art on mini-Imagenet. We confirm these results on another few-shot dataset that we introduce in this paper based on CIFAR100. Our code is publicly available at https://github.com/ElementAI/TADAM.
1 Introduction
Few-shot learning trains models to generalize across tasks from small labeled sample sets, commonly using learned embeddings and similarity-based classification. This paper studies metric scaling, task conditioning, and auxiliary co-training to improve few-shot performance.
- Background: Few-shot learning uses support examples to classify query instances and evaluates meta-learned models on new tasks.Meta-learning exposes models to multiple training episodes, each providing a task-specific classifier.
- Background: Matching Networks and Prototypical Networks embed support and query examples, then classify using nearest-neighbor similarity.The literature therefore treats the learned similarity metric as a central component of few-shot classification.
- Metric scaling: Metric scaling improves similarity-based few-shot learning by exploiting the interaction between the similarity metric and the cost function.The paper analyzes how scaling changes objective-function parameter updates and reports positive empirical effects.
- Results: 76.7% accuracy is reached on 5-shot, 5-way mini-Imagenet, with improvements of up to 8.5% over Snell et al. and 4.8% over the state of the art.The reported task-dependent scaled metric achieves the best-reported accuracy on that dataset.
- Task conditioning: Task conditioning adapts the feature extractor to each task by using a task representation derived from the task sample set.The approach uses a task encoding network and feature-wise linear modulation to influence the extractor.
- Auxiliary task co-training: Auxiliary task co-training reduces training complexity and improves generalization while learning the task-dependent metric.The architecture is evaluated across multiple few-shot benchmarks and shot settings, including mini-Imagenet and a CIFAR100-based dataset.
2 Model Description
The model combines a scaled similarity metric with task-conditioned feature extraction and auxiliary co-training. Its optimization behavior depends on the scaling regime, while auxiliary classification supports training the conditioned feature extractor.
- Metric Scaling: Metric scaling changes how the similarity metric contributes to parameter updates through overall gradient scaling and weighting sharpness.The analysis examines the α-normalized gradient as α approaches 0 and infinity.
- Metric Scaling: Small α separates prototype distributions, whereas large α emphasizes correcting samples assigned to the closest incorrect prototype.The two regimes therefore favor different update behaviors: distribution-overlap minimization versus sample-wise cluster correction.
- Metric Scaling: As accuracy improves in the large-α regime, more correctly classified samples produce zero derivatives, reducing the effective batch size.The authors consequently hypothesize and empirically demonstrate an optimal α for a given dataset, metric, and task.
- Task Conditioning: The feature extractor is conditioned on each task through a task representation and predicted layer-wise scale and shift parameters.The task representation is the mean of task class prototypes, encoded by a task embedding network to produce γ and β vectors.
- Architecture: The architecture uses a ResNet-12 backbone with four residual blocks, max pooling after each block, and no projection layers around the backbone.Task-conditioning layers are inserted after convolutional and batch-normalization blocks.
- Auxiliary Co-training: Auxiliary co-training adds a classification head whose sampling probability is annealed over episodes to make joint optimization of the backbone and task embedding network tractable.The authors attribute the benefit over simple pre-training to regularization from auxiliary-task gradients.
3 Experimental Results
Experiments evaluate metric scaling, task conditioning, auxiliary co-training, and feature-extractor choices across mini-Imagenet and Fewshot-CIFAR100. Scaling improves cosine similarity substantially, while task conditioning requires auxiliary co-training to achieve state-of-the-art performance.
- Experimental setup and datasets: The experiments use mini-Imagenet and the newly introduced Fewshot-CIFAR100, with the latter designed to test whether results generalize beyond mini-Imagenet.Fewshot-CIFAR100 also uses superclass-based splits to reduce information overlap and create a more challenging few-shot setting.
- On the similarity metric: 68.9% and 67.7%: the re-implemented Prototypical Networks model achieved these accuracies in the 20-way and 5-way training scenarios, respectively.The improvement followed increasing training from 20K to 40K steps.
- On the similarity metric: 14 percentage points: scaled cosine similarity improves mini-Imagenet accuracy over its non-scaled version and closely matches Euclidean performance.Similar results were obtained on FC100; multiple initial learning rates produced similar accuracy, controlling for scaling effects on learning rate.
- Ablation study: An optimal metric scale parameter exists for each dataset-and-metric combination, producing inverse U-shaped cross-validation curves.The scale parameter was cross-validated on the validation set.
- Ablation study: Task-conditioned ResNet-12 does not improve without auxiliary co-training, because jointly learning TEN and feature-extractor representations can become stuck in local extrema.Auxiliary co-training eases initial convergence and regularizes the few-shot task through a decoupled supervised classification objective.
- Ablation study: The TEN-plus-auxiliary-co-training model outperforms all baselines and achieves state-of-the-art results.The reported comparison appears in the last rows of Tables 1 and 3.
4 Conclusions and Future Work
The paper validates metric scaling and task representation conditioning as improvements for few-shot learning, while identifying loss-scaling behavior and richer task representations as future directions.
- Scaled cosine similarity performs at par with Euclidean distance, unlike its unscaled counterpart.
- Scaling the similarity metric produces completely different parameter-update regimes under softmax and categorical cross-entropy.
- The optimal performance occurs between two asymptotic softmax regimes, motivating research on loss functions and α schedules for few-shot learning.
- Task representation conditioning improves the feature extractor for few-shot classification and suggests higher-order class-embedding statistics as a future representation direction.
- Experimental results on two independent challenging datasets validate the proposed improvements.
A Proof of Lemma 1
The proof analyzes the softmax-gradient limit as the scaling parameter α grows, showing that only the closest prototype can retain a non-zero contribution under the stated assumptions.
- As α →∞, denominator terms with positive exponential rates make their corresponding ratios converge to zero under assumption A2.
- The only non-zero limiting contribution occurs when the corresponding prototype is closest to the query point.
- Defining the closest prototype index as j∗ enables the proof to identify the surviving term in the large-α limit.
S1 Architecture details
The architecture uses a ResNet-12 backbone and TEN conditioning blocks, with implementation details specifying block structure, filter widths, regularization, and TEN generation networks.
- The ResNet-12 feature extractor comprises four residual blocks followed by global average pooling.
- Each residual block contains three convolutional blocks followed by 2x2 max-pooling, with batch normalization and swish-1 after each convolution.
- The first ResNet block uses 64 filters, doubled after each max-pool block, with an L2 regularizer weight of 0.0005 per layer.
- TEN uses two separate three-layer fully connected residual networks to generate γ and β vectors at each conditioned convolutional layer width.
- Figure 2 depicts the TEN block architecture.
S2 Few-shot CIFAR100 details
The CIFAR100 few-shot dataset divides superclasses into train, validation, and test splits while sampling tasks uniformly within each split.
- The train split contains 12 CIFAR100 superclasses, including fish, flowers, food containers, vehicles, and trees.
- The validation split contains four superclasses: large carnivores, large omnivores and herbivores, non-insect invertebrates, and small mammals.
- The test split contains four superclasses: aquatic mammals, insects, medium mammals, and people.
- Tasks are sampled uniformly within each train, validation, and test subset, so tasks usually contain classes from several superclasses.
S3 Training procedure details
The training procedure composes few-shot batches from multiple fixed five-class tasks, co-trains with an auxiliary 64-way classification task, and evaluates accuracy across repeated random trials. Optimization uses SGD with scheduled learning-rate reductions, while reproductions follow the original setup and architecture.
- Episode composition: Each few-shot task contains a fixed selection of 5 classes, and training batches combine several tasks with deployment-matched shot counts.The empirically best tasks per batch were 2 for 5-shot, 1 for 10-shot, and 5 for 1-shot training.
- Auxiliary classification task: Auxiliary co-training uses uniformly sampled batches of 64 images from the usual 64-way mini-Imagenet classification task.Its learning-rate annealing schedule is synchronized with the main few-shot task.
- Optimization, scheduling and learning rate: Training uses SGD with momentum 0.9 and initial learning rate 0.1, with reductions by a factor of 10 during optimization.Auxiliary-task training runs for 30,000 mini-Imagenet episodes and 10,000 FC100 episodes; without co-training, runs use twice as many episodes.
- Classification accuracy evaluation: Accuracy is measured over 10 random optimization restarts and 500 randomly generated tasks, each with 100 random query samples.
- Reproducing results: Reproductions of prior results use exactly the original paper’s reported setup and network architecture.