PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Metric Learning for Few-Shot Image Classification: A Review of Recent Developments

Xiaoxu Li, Xiaochen Yang, Zhanyu Ma, Jing-Hao Xue

arXiv:2105.08149v2cs.CVcs.LG

TL;DR

Few-shot metric learning methods face challenges in learning features that generalize across tasks and domains while avoiding underfitting and vulnerability. This paper reviews recent methods, organizes them by three metric-learning stages, and identifies challenges and future directions.

  • Problem

    Generic image embeddings may not be sufficiently discriminative for novel classes, while domain shifts create discrepancies between feature distributions.

  • Method

    The paper reviews recent few-shot deep metric learning methods, provides an evaluation framework, and categorizes representative approaches by feature embeddings, class representations, and distance measures.

  • Results

    The review identifies the novelties and problems of representative methods and summarizes main challenges for few-shot image classification.

  • Takeaways & Limitations

    Future work should explore directions addressing episodic-training limitations, robustness vulnerabilities, and cross-domain metric-space differences.

  • Takeaways & Limitations

    Episodic training can underfit the base dataset and suffer from catastrophic forgetting across continually sampled tasks.

Abstract

from arXiv · show

Few-shot image classification is a challenging problem that aims to achieve the human level of recognition based only on a small number of training images. One main solution to few-shot image classification is deep metric learning. These methods, by classifying unseen samples according to their distances to few seen samples in an embedding space learned by powerful deep neural networks, can avoid overfitting to few training images in few-shot image classification and have achieved the state-of-the-art performance. In this paper, we provide an up-to-date review of deep metric learning methods for few-shot image classification from 2018 to 2022 and categorize them into three groups according to three stages of metric learning, namely learning feature embeddings, learning class representations, and learning distance measures. With this taxonomy, we identify the novelties of different methods and problems they face. We conclude this review with a discussion on current challenges and future trends in few-shot image classification.

1 The Framework of Few-Shot Image Classification

Few-shot classification learns to classify novel classes from small support sets, using separate base and novel datasets and repeated episodic evaluation. The framework defines task variants and benchmarks spanning general, fine-grained, and cross-domain settings.

  • 1.1 Notation and definitions: Base and novel datasets have disjoint label spaces, and the novel dataset is split into support and query sets for training and testing.The base dataset transfers knowledge to facilitate learning on the novel dataset.
  • 1.1 Notation and definitions: Few-shot classification learns a classifier for novel query instances from a support set containing only 1–5 labeled examples per class.A C-way K-shot task has C classes and K labeled examples per class; one-shot classification uses one example per class.
  • 1.1 Notation and definitions: Cross-domain few-shot classification uses base and novel datasets from different domains, while generalized few-shot classification evaluates classes from both datasets.Domain differences are defined through feature spaces or marginal distributions.
  • 1.2 Evaluation procedure of few-shot classification: Each evaluation episode randomly selects C classes, K support samples per class, and M query samples from the remaining samples of those classes.The classifier predicts query labels for each episode, and performance is the mean classification accuracy across E episodes.

2 Few-Shot Deep Metric Learning Methods

Few-shot deep metric learning classifies novel queries by comparing them with support examples or learned class representations under a metric learned from base data. The review organizes methods by feature embeddings, class representations, and distance or similarity measures.

  • 2 Few-Shot Deep Metric Learning Methods: Deep metric learning learns distances that are small for semantically similar instances and large for dissimilar instances, then applies distance-based classification to novel queries.In few-shot classification, the metric is learned on the base dataset and used to compare query images with novel support images.
  • 2 Few-Shot Deep Metric Learning Methods: The review’s taxonomy covers learning feature embeddings, learning class representations, and learning distance or similarity measures.Methods contributing to two metric-learning aspects can appear twice in the taxonomy.
  • 2 Few-Shot Deep Metric Learning Methods: Deep metric learning often separates feature-embedding learning from distance-measure learning to capture nonlinear data structure and produce discriminative representations.Methods may compare queries with individual samples or with class representations such as prototypes and subspaces.

2.1 Learning feature embeddings

Feature-embedding methods learn task-agnostic or task-specific representations to separate few-shot classes, with augmentation, local structure, and adaptation addressing data scarcity and task variation.

  • Learning task-agnostic features: Task-agnostic embeddings aim to generalize across novel classes, while recent methods adapt representations to the particular task using base-dataset learning without novel-data retraining.Task-specific adaptation can improve discrimination while avoiding direct retraining on novel classes.
  • Learning task-agnostic features: The Siamese Network introduced shared-weight twin sub-networks for image comparison, using high-level features and a weighted L1 distance.Its weights and component-wise distance weights are trained with mini-batch gradient descent.
  • Learning task-agnostic features: Matching Network introduced episodic training and support-set-dependent embeddings for support and query images.Bidirectional and attention-based LSTMs encode relationships within the support set and between queries and supports.
  • Learning task-agnostic features: Domain generalization remains difficult because methods designed for same-domain novel data degrade when novel data comes from different domains.Proposed responses simulate varied feature distributions or distill a universal multi-domain representation.
  • Learning task-agnostic features for fine-grained image classification: Fine-grained classification motivates local, multi-scale, and second-order features because subcategories differ subtly while instances vary in pose, scale, and rotation.DN4 pools local descriptors by class and compares query descriptors with class-specific nearest neighbors; related methods add learned thresholds, weights, or multiple scales.
  • Learning task-specific features: Task-specific methods include TADAM’s conditional batch-normalization transformations, LGM-Net’s generated learner weights, and task-specific feature masks.TADAM represents tasks using the mean of class prototypes and adds standard base-class training as an auxiliary task.

2.2 Learning class representations

Class-representation methods replace scarce support examples with prototypes or richer class distributions, improving robustness by modeling class structure, uncertainty, or multiple modes.

  • Learning class representations: Early methods compare queries directly with scarce support samples, whereas prototype-based methods summarize each novel class with reference vectors.Prototypes can be simple or weighted averages, or learned functions of class features.
  • Learning class representations: Prototypical Network classifies queries by Euclidean distance to class prototypes defined as means of support embeddings.It assumes each class forms a cluster around a single prototype in the learned embedding space.
  • Learning class representations: Semi-supervised Prototypical Network incorporates unlabeled samples through soft k-means assignment scores and weighted prototype means.The weighted samples contribute according to their estimated class assignments.
  • Learning class representations: Multiple-prototype and query-dependent approaches address multimodality, support relevance, outliers, and label noise.IMP models class clusters as Gaussian distributions, while attentive prototypes weight support samples according to query similarity; RapNets further learn attention weights from correlation features.
  • Learning class representations: Other representations include learnable shot-free prototype functions, task-specific projected prototypes, iteratively refined hierarchical prototypes, and probabilistic prototype distributions.Bayesian variants represent classes through distributions and can estimate assignment confidence; DSN instead represents each class with a low-dimensional subspace and reports greater robustness than Prototypical Network.

2.3 Learning distance or similarity measures

Distance-learning methods go beyond fixed Euclidean or cosine measures by tuning scaling, adapting covariance-based metrics, designing dissimilarities, or learning neural and graph-based similarities.

  • Learning distance or similarity measures: After learning embeddings or class representations, many methods use fixed Euclidean distance or cosine similarity, motivating learned distance and similarity measures.The review treats distance learning as the third metric-learning stage.
  • Learning distance or similarity measures: Metric scaling methods tune loss scaling parameters, including separate ground-truth and other-class parameters that enlarge their distance contrast.Some approaches gradually tune these parameters over training episodes.
  • Learning distance or similarity measures: Simple CNAPS and TEAM learn task-specific metrics using covariance information, with Simple CNAPS applying regularized Mahalanobis distance to queries and prototypes.Simple CNAPS estimates class- and task-specific covariance matrices as convex combinations of sample covariances regularized toward identity.
  • Learning distance or similarity measures: SEN combines Euclidean distance with L2-norm differences, while DeepEMD combines dense structural distance, Earth Mover’s Distance, and convolutional feature embeddings.These measures introduce normalization or dense-image matching into few-shot comparison.
  • Learning distance or similarity measures: Relation Network learns pairwise similarity with an embedding module and relation module, while SAML computes local-feature relations and feeds them to an MLP.Both replace a solely fixed distance with a learned similarity score.
  • Learning distance or similarity measures: GNN-based methods model sample relations through graph structure, progressing from node or edge labeling to distribution-level propagation over dual complete graphs.EGNN explicitly clusters through edge similarities and dissimilarities, while DPGN propagates between sample-level and distribution-level graphs.

3 Challenges and Future Directions

Despite promising performance, few-shot metric learning still needs better ways to learn generalizable features from very limited labeled data.

  • Challenges and Future Directions: Few-shot metric learning has achieved promising performance but retains important challenges for future research.The review identifies unresolved issues rather than claiming the problem is solved.
  • Challenges and Future Directions: Existing methods commonly rely on attention mechanisms, data augmentation, and multi-task learning to learn discriminative features from few labeled examples.The review calls for new evaluation and feature-learning approaches that improve generalization ability.

1. Improving generalized feature learning on few samples

Episodic training can mismatch evaluation conditions, update on low-quality episodes, and potentially underfit the base dataset.

  • 1. Improving generalized feature learning on few samples: Episodic training rigidly matches each training episode’s classes and images to the evaluation episode.The review notes that training with more classes can be beneficial.
  • 1. Improving generalized feature learning on few samples: Episode-level updates ignore episode quality, making methods vulnerable to poorly sampled images such as outliers.
  • 1. Improving generalized feature learning on few samples: The review identifies possible underfitting of the base dataset as another consequence of episodic training.

2. Rethinking the use of episodic training strategy

The review highlights catastrophic forgetting as a limitation of continual learning across episodes and calls for methods that fit both base and novel datasets.

  • 2. Rethinking the use of episodic training strategy: Continual learning across tasks sampled from the base dataset can cause catastrophic forgetting.Knowledge learned from previous tasks may be supplanted after learning a new task.
  • 2. Rethinking the use of episodic training strategy: Avoiding forgetting while improving metric-learning models’ fitting ability on base and novel datasets remains an open challenge.

3. Enhancing stability to support samples and robustness to adversarial perturbations and distribution shifts

Despite improving classification accuracy, few-shot classification methods remain vulnerable to outliers, adversarial perturbations, and distribution shifts, limiting safety-critical use.

  • 3. Enhancing stability to support samples and robustness to adversarial perturbations and distribution shifts: Few-shot classification methods remain non-robust to input or label outliers.
  • 3. Enhancing stability to support samples and robustness to adversarial perturbations and distribution shifts: Adversarial perturbations targeting support or query images can fool existing few-shot classifiers into incorrect predictions.
  • 3. Enhancing stability to support samples and robustness to adversarial perturbations and distribution shifts: These vulnerabilities hinder deployment in safety-critical applications such as medical image analysis.

4. Developing metric learning methods for cross-domain few-shot classification

Cross-domain few-shot classification remains underexplored, and large domain shifts can make simple transductive fine-tuning outperform meta-trained methods.

  • 4. Developing metric learning methods for cross-domain few-shot classification: Only a few studies currently address cross-domain few-shot classification.
  • 4. Developing metric learning methods for cross-domain few-shot classification: Under large domain shifts, simple transductive fine-tuning outperforms all reviewed meta-trained methods.The reported setting trains on natural images and evaluates on agriculture and satellite images.

4 Conclusions

The paper reviews recent few-shot deep metric learning methods, provides a general evaluation framework, categorizes representative approaches, and identifies challenges that motivate future directions.

  • The review defines few-shot learning and establishes a general evaluation framework for comparing methods.
  • It categorizes and reviews representative approaches in few-shot deep metric learning.
  • The paper summarizes the main challenges in existing methods and uses them to identify directions for future exploration.
Loading 2105.08149v2…