PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Learning Through an Information Retrieval Lens

Eleni Triantafillou, Richard Zemel, Raquel Urtasun

arXiv:1707.02610v2cs.LG

TL;DR

Few-shot learning must operate when only a few labelled examples are available for each new concept. The paper treats batch points as retrieval queries and optimizes mean Average Precision over their jointly predicted rankings using structured prediction. It reports competitive standard few-shot classification results, superiority over a strong baseline for few-shot retrieval, and a limitation for the siamese baseline when K exceeds one.

  • Problem

    Few-shot learning seeks to classify new concepts from few labelled examples, a regime where methods relying on large datasets can overfit.

  • Method

    The method treats each batch point as a query and uses structured prediction with SSVM and DLM to optimize mean Average Precision across batch rankings.

  • Results

    The method is competitive with state-of-the-art standard few-shot classification results and superior to a strong baseline on few-shot retrieval.

  • Takeaways & Limitations

    Jointly optimizing all relative batch orderings supports few-shot classification and enables the proposed few-shot retrieval task.

  • Takeaways & Limitations

    The siamese baseline does not naturally exploit all K representatives during classification, so the paper reports it only for one-shot learning.

Abstract

from arXiv · show

Few-shot learning refers to understanding new concepts from only a few examples. We propose an information retrieval-inspired approach for this problem that is motivated by the increased importance of maximally leveraging all the available information in this low-data regime. We define a training objective that aims to extract as much information as possible from each training batch by effectively optimizing over all relative orderings of the batch points simultaneously. In particular, we view each batch point as a `query' that ranks the remaining ones based on its predicted relevance to them and we define a model within the framework of structured prediction to optimize mean Average Precision over these rankings. Our method achieves impressive results on the standard few-shot classification benchmarks while is also capable of few-shot retrieval.

1 Introduction

The paper frames few-shot learning as learning new concepts from limited labelled examples, then proposes an information-retrieval objective that jointly exploits all batch relationships. It introduces few-shot retrieval alongside competitive classification results.

  • Few-shot learning classifies new concepts from only K examples per class, a setting where insufficient data can cause overfitting.The paper emphasizes that obtaining many examples for every new class is not always realistic or preferred.
  • The model treats each batch point as a query that ranks the remaining points by predicted relevance.The objective uses structured prediction to optimize mean Average Precision over these rankings.
  • The training objective simultaneously optimizes all relative orderings within each batch to extract more information when class data are scarce.Figure 1 illustrates this by maximizing AP across the rankings induced by the batch queries.
  • The paper introduces few-shot retrieval, in which a query image must retrieve identically labelled candidates from previously unseen classes.It also reports competitive results against state of the art on standard few-shot classification benchmarks and superiority over a strong baseline for few-shot retrieval.

2 Related Work

The related work spans metric learning, neural pairwise and triplet methods, and meta-learning. The paper distinguishes its approach by constructing query/support relationships across the entire batch for few-shot learning.

  • Metric learning: Metric-learning methods learn embeddings or metrics whose relative positions represent similarity relationships for tasks such as nearest-neighbor classification.Examples include NCA, LMNN, and their deep extensions.
  • Metric learning: Siamese and triplet networks use parallel, identically parameterized embeddings while operating on pairs or triplets, respectively.The cited applications include signature and face verification.
  • Few-shot metric learning: Several few-shot methods learn pairwise similarities or compare pairs through recurrent glimpses, but the paper notes they lack a natural mechanism for using K representatives in K-shot classification.The paper positions its own similarity metric as specifically tailored to few-shot learning.
  • Paper positioning: The paper can be viewed as constructing all query/support sets within each batch to fully exploit batch information.This contrasts with approaches that do not naturally aggregate all K class representatives for a reference.
  • Meta-learning: Meta-learning methods learn how to update or initialize a learner from support sets so it can generalize to query sets.Examples include Meta-Learner LSTM, MAML, and methods using external memory.

3 Background

The method defines mAP by averaging query-level AP scores over batch points and formulates ranking as structured prediction. SSVM and DLM provide loss-based optimization procedures for the AP ranking objective.

  • Mean Average Precision: For each query, relevant points are same-class batch members, and Average Precision averages precision at ranks containing relevant points.Mean Average Precision then averages AP across the batch points.
  • Structured prediction: Structured prediction is appropriate because ranking produces inter-dependent structured outputs, and SSVM supports custom task losses through loss-augmented optimization.The margin-rescaled SSVM optimizes an upper bound on task loss using a scoring function and ground-truth output.
  • Direct Loss Minimization: Direct Loss Minimization provides a perceptron-like update that directly optimizes the task loss rather than an upper bound.Its theorem yields updates when a scoring function and standard and loss-augmented inference procedures are defined.
  • Direct Loss Minimization: The positive DLM update moves away from a high-scoring, high-loss solution, whereas the negative update uses a best solution.The positive update resembles the loss-informed update of margin-rescaled SSVM, although the methods score different predictions.
  • Ranking representation: The ranking output represents pairwise orderings, with each y_ij indicating whether point i is ranked above or below point j.The scoring function uses a learned similarity between each point and the query.
  • mAP optimization: The paper extends AP optimization to mAP by accounting for every possible query in a batch while preventing improvement for one query from harming another.Loss-augmented inference uses a dynamic-programming algorithm from prior work.

4 Few-Shot Learning by Optimizing mAP

The method formulates few-shot learning as structured prediction over batch-wide rankings, treating each point as a query and optimizing mean Average Precision across those rankings. Its scoring and inference framework decomposes query-specific orderings so they can be optimized independently and in parallel.

  • Approach: The approach extends AP optimization to mean Average Precision by accounting for every possible query among the batch points.The central challenge is ensuring that improving one query’s ranking does not harm another’s.
  • Inference: Independent query orderings can be optimized without sacrificing correctness, enabling efficient parallel inference.The framework uses dynamic programming for loss-augmented inference and combines the independent query results into the overall score.
  • Optimization: The paper provides two optimization variants, mAP-DLM and mAP-SSVM, implemented through the corresponding gradient-update frameworks.Algorithm 1 outlines the inference, gradient computation, and weight-update process for both variants.
  • Approach: Each batch point acts as a query that ranks remaining points, with pairwise outputs representing whether a positive point ranks above a negative point.The model uses neural embeddings and cosine similarity to score point pairs, then represents rankings through binary pairwise comparisons.
  • Objective: The overall scoring function sums query-specific scores, while the ground-truth output assigns relevance to same-class versus different-class point pairs.For each query, the ground-truth relation is represented by an appropriately sized matrix of ones for positive-negative comparisons.
  • Objective: Average Precision is computed separately for each query ranking, and the mAP loss averages those AP losses across all query points.Precision at rank j is defined as the percentage of relevant points among the top-ranked j points.

5 Evaluation

The evaluation tests few-shot classification and retrieval on Omniglot, mini-ImageNet, and CUB, using episode-based protocols and comparisons with competitive baselines. Results show state-of-the-art-level classification, strong retrieval performance, and an advantage from the structured objective when fewer examples are available.

  • Evaluation tasks: Few-shot classification episodes use N classes, K representatives per class, and classify the remaining images across 1000 Omniglot or 600 mini-ImageNet test episodes.The retrieval protocol uses 10 images per class; each image queries the remaining 10N − 1 images, with mAP measuring whether the nine relevant images precede irrelevant ones.
  • Evaluation tasks: The model classifies an image by selecting the class whose representatives yield the highest average precision in the induced relevance ordering.For class i, its K representatives are relevant and the K(N − 1) representatives from other classes are irrelevant.
  • Benchmark results: On Omniglot, the method is among the top-performing classification approaches and performs equally well or better than the siamese network in few-shot retrieval.The siamese network remains a strong classification baseline and outperforms several recent few-shot methods on some classification tasks.
  • Benchmark results: On mini-ImageNet, the method outperforms recent metric-learning and meta-learning approaches, sets the state-of-the-art for 1-shot 5-way classification, and exceeds the siamese retrieval baseline.The benchmark uses 64 training, 16 validation, and 20 testing classes from 100 total classes.
  • Benchmark results: On CUB, the method also outperforms the siamese network, while overall classification performance is on par with state-of-the-art results and retrieval is superior to a strong baseline.The paper compares metric-learning and meta-learning families, finding no clear winning category across the compared directions.
  • Learning efficiency: With smaller shots, the structured objective has a clear advantage over the all-pairs siamese, and mAP-DLM reaches higher performance with smaller batch sizes.The method also learns faster when evaluation tasks have larger way, even when final accuracy or mAP becomes comparable.

6 Conclusion

The paper presents an information-retrieval-based few-shot learning approach that exploits information in each training batch by simultaneously optimizing all relative point orderings. Experiments support learning efficiency and show promising results on two standard few-shot learning datasets, while future work includes improved batch construction and alternative retrieval metrics.

  • The approach formulates few-shot learning as information retrieval and simultaneously optimizes all relative orderings within each training batch.The objective uses mean Average Precision as its training target.
  • Experiments support the method’s learning-efficiency claims and show promising results on two standard few-shot learning datasets.
  • Future work includes constructing batches to better leverage training-set information and exploring retrieval metrics beyond mean Average Precision.Examples mentioned include ROC curve and discounted cumulative gain.

A Independence of Individual Query Rankings

The appendix argues that batch mean Average Precision can be optimized by independently optimizing the Average Precision of each query ranking. A proof sketch examines which binary rankings involving same-class and different-class points contribute to the objective.

  • The batch mAP objective can be decomposed into independent optimization of each query’s AP ranking.
  • The proof sketch considers two points from one class and a third point from another class to analyze relevant binary rankings.
  • Only binary rankings placing two distinct same-class points in the relevant positions and a different-class point in the contrasting position are computed.
  • Table 3 records the relative positions of points i, k, and j for each pair of variable values.

B Controlling the influence of loss-augmented inference on the loss gradient

The method introduces α to control the relative contribution of loss-augmented and standard scores in the gradient update. Experiments on Omniglot and miniImageNet find significant gains for α > 1, with little sensitivity among values above 1; α = 10 outperforms α = 1 on miniImageNet.

  • Controlling the influence of loss-augmented inference on the loss gradient: α controls the contribution of the loss-augmented score relative to the standard or ground-truth score in mAP-DLM and mAP-SSVM updates.For mAP-DLM, the comparison is between F(X, ydirect, w) and F(X, yw, w); for mAP-SSVM, it is relative to F(X, yGT, w).
  • Exploring different values of α: On Omniglot and mini-ImageNet, α > 1 produces significant gains, while performance does not significantly vary among values greater than 1.Figures 3 and 4 average performance over 20 randomly sampled held-out-data episodes.
  • Exploring different values of α: α = 10 produces significantly superior mini-ImageNet results compared with α = 1 for both mAP-SSVM and mAP-DLM.The α = 1 setting harms mAP-DLM more than mAP-SSVM.
  • Relationship between α and ϵ: ϵ and α regulate different trade-offs: ϵ affects task-loss weighting when computing ydirect, whereas α affects score weighting in the next update.
  • Conjecture on why α > 1 may help: The authors conjecture that α > 1 may strengthen learning when ydirect and yw have similar scores, leaving a weak task-loss signal despite incomplete optimization.

C Additional Experiments on the Caltech-UCSD Birds (CUB) 200-2011 dataset

Additional experiments evaluate the method on the fine-grained CUB 200-2011 dataset using the same test procedure and architecture as the other datasets. With α = 10, the approach improves by 3% relative to a strong siamese baseline.

  • CUB 200-2011 contains 11,788 images spanning 200 bird species, split into 100 training, 50 validation, and 50 testing species.The dataset is intended for fine-grained classification, making few-shot learning challenging because classes are similar.
  • The CUB evaluation uses the same test procedure and architecture as Omniglot and mini-ImageNet, producing a 1024-dimensional embedding space.
  • The CUB experiments use α = 10 for both mAP-DLM and mAP-SSVM, with the positive update for mAP-DLM.
  • 3% improvement relative to the siamese network is reported as a significant win against a strong baseline.The baseline outperforms recent few-shot methods on mini-ImageNet and performs comparably with state of the art on Omniglot.

D Training Details

The paper reports dataset-specific optimization settings for Omniglot and mini-ImageNet, with hyperparameters selected using validation performance. The mini-ImageNet schedule additionally decays learning rates during training.

  • Omniglot: Omniglot uses learning rates of 0.001 for mAP-DLM and mAP-SSVM, and 0.1 for the siamese network.mAP-DLM sets ϵ = 1, both mAP models use α = 10, and all three models train for 18K updates.
  • Omniglot: Omniglot hyperparameters were selected based on validation-set performance.
  • mini-ImageNet: On mini-ImageNet, mAP-DLM and mAP-SSVM start at 0.001 and decay every 2K steps from update 2K by multiplying the rate by 0.75.The siamese network starts at 0.01 and follows the same decay schedule.
Loading 1707.02610v2…