PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Few-Shot Visual Classification

Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, Leonid Sigal

arXiv:1912.03432v3cs.CV

TL;DR

Few-shot learning seeks to reduce reliance on exhaustively labeled data, while prior approaches have emphasized increasingly complex feature extractors and classifier adaptation. This paper incorporates Mahalanobis distance into CNAPS and learns adaptive features that support covariance estimation from few samples, yielding Simple CNAPS with fewer parameters and stronger reported benchmark performance.

  • Problem

    Few-shot learning aims to address computer-vision settings where data is scarce or difficult to label extensively, while metric choice in embedding-space classification has received comparatively little attention.

  • Method

    Simple CNAPS replaces CNAPS’s adaptive classifier with deterministic class-covariance estimation and Mahalanobis-distance computations over task-adapted feature vectors.

  • Results

    6.1% improvement on average over SoTA (CNAPS) is reported, while Simple CNAPS removes 788,485 parameters from the original CNAPS architecture.

  • Takeaways & Limitations

    Learned adaptive feature extractors can produce embeddings that allow useful high-dimensional covariance estimates even when few labeled instances are available per task and class.

Abstract

from arXiv · show

Few-shot learning is a fundamental task in computer vision that carries the promise of alleviating the need for exhaustively labeled data. Most few-shot learning approaches to date have focused on progressively more complex neural feature extractors and classifier adaptation strategies, as well as the refinement of the task definition itself. In this paper, we explore the hypothesis that a simple class-covariance-based distance metric, namely the Mahalanobis distance, adopted into a state of the art few-shot learning approach (CNAPS) can, in and of itself, lead to a significant performance improvement. We also discover that it is possible to learn adaptive feature extractors that allow useful estimation of the high dimensional feature covariances required by this metric from surprisingly few samples. The result of our work is a new "Simple CNAPS" architecture which has up to 9.2% fewer trainable parameters than CNAPS and performs up to 6.1% better than state of the art on the standard few-shot image classification benchmark dataset.

1. Introduction

Few-shot image classification addresses learning new classes from very few labeled support images. This paper argues that the classification metric matters substantially and introduces Simple CNAPS, which combines task-adapted embeddings with Mahalanobis distance.

  • Motivation: Few-shot learning targets models that adapt to previously unseen classes using only a few labeled examples per class.The paper focuses on automatically adapting classification methods to new image-classification tasks at test time.
  • Approach: Mahalanobis distance incorporates each class’s feature-space distribution and can produce improved nonlinear decision boundaries.The task-adaptation mechanism is trained end-to-end from the Mahalanobis-based classification loss, encouraging informative covariance estimates.
  • Approach: Simple CNAPS uses a partially adapted feature extractor and a fixed Mahalanobis metric for final classification.Its organization differs from methods that adapt both the feature extractor and the final classifier.
  • Motivation: Metric choice has received less attention than feature extraction and classifier adaptation, despite the paper finding it important in practice.The authors contrast common assumptions about flexible nonlinear mappings with their empirical finding that the distance metric affects performance.
  • Contributions: 6.1% average improvement over SoTA CNAPS was obtained using a test-time-estimated class-covariance-based distance metric.The result is reported for few-shot image classification.
  • Contributions: Up to 9.2% fewer trainable parameters are used than in CNAPS while replacing learned classifier components with deterministic covariance estimation and Mahalanobis computations.The architecture removes 788,485 parameters, equivalent to 3.2%-9.2% of the original total.

2. Related Work

Few-shot methods differ mainly in how they form embeddings and compute classification distances. Relative to CNAPS and prototypical networks, this work keeps task-adapted feature extraction but uses regularized full covariance estimates with a fixed Mahalanobis classifier.

  • Method axes: Few-shot approaches can be organized by image-to-embedding transformation and by the distance used to assign labels.Figure 2 presents these as the two principal axes for differentiating methods.
  • Prior approaches: Earlier methods used nearest-neighbor or prototype-based classification with distances such as weighted L1, cosine similarity, or squared Euclidean distance.These methods vary in whether examples remain individual references or are summarized by class prototypes.
  • CNAPS: CNAPS adapts a pre-trained feature extractor with FiLM layers and also adapts a final dot-product classifier to each test-time task.Its task representations generate task-specific adaptation parameters.
  • Metric choice: The method improves on squared Euclidean distance by using the squared Mahalanobis distance, which also has theoretical connections to Gaussian mixture models through Bregman-divergence properties.This extends the theoretical motivation previously used for squared Euclidean distances in prototypical networks.
  • Simple CNAPS: Simple CNAPS shows that adapting the classifier is unnecessary for good performance when task-specific feature vectors support regularized class-covariance estimation.The classifier instead uses a fixed Mahalanobis-distance scheme.
  • Covariance estimation: Unlike an approach using neural networks for diagonal covariance estimates, Simple CNAPS generates regularized full covariance estimates from an end-to-end trained adaptation network.The distinction concerns the flexibility of the covariance representation and how it is learned.

3. Formal Problem Definition

The paper frames few-shot image classification as amortized classification over many sampled tasks. A classifier is trained to maximize expected query performance after receiving each task’s support set.

  • Task construction: Few-shot image classification is framed as an amortized classification task built from a large labeled image dataset.The dataset is denoted D and contains image-label pairs.
  • Task construction: Classification tasks are repeatedly sampled from the dataset, with each task identified by τ and represented by a support set S_τ.The sampling procedure constructs many tasks from the available labeled data.
  • Objective: The objective is to find classifier parameters θ that maximize expected query performance across tasks.The expectation is taken over the sampled task distribution.
  • Task constraints: Few-shot task construction constrains query labels to be a subset of support-set labels.The paper notes that practical task sampling imposes structured relationships between support and query image-label pairs.

4. Method

Simple CNAPS retains CNAPS’s task-adapted feature extractor but replaces its adapted linear classifier with deterministic Mahalanobis-distance classification. It estimates regularized class- and task-specific covariances from support embeddings, enabling covariance-aware classification with fewer classifier parameters.

  • Classifier design: Simple CNAPS shares CNAPS’s feature-adaptation architecture but replaces its adaptive classifier with a fixed, parameter-free distance computation.Both methods use task-adapted feature extraction, while CNAPS uses a trained linear classifier and Simple CNAPS computes distances deterministically.
  • Complexity: Simple CNAPS removes 778k classifier parameters relative to CNAPS while preserving task-specific feature adaptation.Its classifier is fully deterministic, unlike CNAPS’s adapted classifier.
  • Feature adaptation: Both models adapt a pre-trained ImageNet ResNet18 with FiLM layers whose scale and shift parameters are generated from a permutation-invariant support-set representation.The feature-adaptation module encodes the support set and produces FiLM parameters for each ResNet block.
  • CNAPS baseline: CNAPS classification uses adapted weights and biases produced from support-derived class means by a classifier adaptation network.The class mean is obtained by mean-pooling support feature vectors extracted by the adapted feature extractor.
  • Covariance estimation: Simple CNAPS estimates each task-and-class covariance from support embeddings using a regularized convex combination of class-within-task and all-classes-in-task covariance matrices.The all-classes covariance uses every support example, while the class covariance uses examples belonging to the relevant class.
  • Covariance estimation: The covariance weighting shifts toward class-level estimates as the number of shots increases, while one-shot classes receive only partial class-covariance support.The deterministic weighting scheme reflects that class covariance estimates improve with more shots and was selected over considered alternatives.

5. Theory

The theory interprets the classification probabilities through exponential-family and Gaussian-mixture perspectives. It explains why Euclidean distance can be restrictive and why Mahalanobis distance incorporates class covariance into decision boundaries.

  • Probabilistic interpretation: Equation 1 corresponds, as λ → 0, to equally weighted exponential-family mixture responsibilities associated with a regular Bregman divergence.The divergence is generated by a differentiable, strictly convex function F.
  • Mahalanobis distance: The squared Mahalanobis distance is a Bregman divergence generated by F(x) = 1 2 xT Σ−1 x and corresponds to the multivariate normal exponential family.When all class covariance estimates approximate the task covariance plus βI, the class probabilities can be viewed as Gaussian-mixture responsibilities.
  • Metric assumptions: Euclidean distance is the squared Mahalanobis distance with Qτ_k = I, implicitly assuming each feature cluster follows a unit normal.Mahalanobis distance instead incorporates cluster covariance when computing distances to class centers.

6. Experiments

Experiments on Meta-Dataset and mini/tieredImageNet evaluate Simple CNAPS across domains, distance metrics, support-set sizes, and task sizes. Simple CNAPS generally improves accuracy over CNAPS and other baselines, while using fewer parameters and remaining effective with very few examples.

  • Datasets and evaluation: Meta-Dataset evaluation covers in-domain and out-of-domain datasets, with additional comparisons on mini/tieredImageNet.Meta-Dataset uses held-out classes and datasets for in-domain and out-of-domain evaluation; mini/tieredImageNet provides fixed-shot, fixed-way comparisons.
  • In-domain performance: 73.8% average in-domain accuracy gives Simple CNAPS a 4.2% gain over CNAPS.Simple CNAPS outperforms baselines on 6 of 8 datasets and matches performance on Fungi and DTD.
  • Out-of-domain performance: 69.7% average out-of-domain accuracy gives Simple CNAPS an 8.2% improvement over the state-of-the-art baseline.Simple AR-CNAPS also exceeds the baseline, but its lower performance than Simple CNAPS suggests possible domain overfitting in the autoregressive adaptor.
  • Overall performance: 72.2% overall accuracy makes Simple CNAPS the best-performing variant, while its performance is statistically indistinguishable from Simple AR-CNAPS.The paper recommends Simple CNAPS because it has fewer parameters.
  • Distance metrics: Mahalanobis distance produces the best in-domain, out-of-domain, and overall average performance across the evaluated datasets.The comparison substitutes Mahalanobis distance with L1, L2, cosine similarity, and negative dot-product metrics.
  • Sensitivity analyses: Simple CNAPS remains effective with as few as four labeled instances per class and improves as the number of shots increases.Across tasks, all methods benefit from more support examples, but Simple CNAPS improves substantially as shots increase.
  • Sensitivity analyses: Accuracy improvements over CNAPS and the L2 variant persist irrespective of the number of classes in a task.Figure 7 groups test tasks by their number of classes and averages accuracy for each class count.
  • Mini/tieredImageNet: Simple CNAPS outperforms recent baselines on all standard 1-shot and 5-shot 5-way mini/tieredImageNet tasks.These experiments use an ImageNet-pretrained ResNet18 and train specifically for the evaluated shot/way configurations.

7. Discussion

The paper concludes that replacing CNAPS’s adaptive classifier with deterministic Mahalanobis-distance classification can establish a new state-of-the-art benchmark while estimating useful class covariances from few labeled examples.

  • Simple CNAPS estimates and uses Mahalanobis distances over task-adjusted, class-specific feature vectors for classification.The authors report that training produces embeddings supporting useful covariance estimates even with few labeled instances per task and class.
  • The method establishes a new state-of-the-art performance benchmark by simplifying the current state-of-the-art architecture.
  • The effectiveness of Mahalanobis distance in feature space suggests connections to hierarchical regularization schemes that could enable performance improvements.

A. Experimental Setting

The experiments evaluate task-adaptive architectures on variable few-shot tasks, including autoregressive feature adaptation and FiLM-conditioned residual networks.

  • A. Experimental Setting: Meta-Dataset evaluation samples test tasks with varying numbers of shots and ways, as illustrated by the test-time frequency distributions.The same task-sampling procedure is used during training and testing.
  • A. Experimental Setting: AR-CNAPS extends CNAPS with block-level set encoders that summarize support representations after preceding adapted blocks.These summaries are passed to later adaptation networks alongside the overall task representation.
  • A. Experimental Setting: The autoregressive adaptation procedure dynamically conditions each block on changes produced by earlier adaptation modules.The passage reports performance gains in certain domains, while the supplied figure identifies the pre-block encoder structure.
  • A. Experimental Setting: FiLM layers scale and shift feature channels inside residual blocks using channel-specific γ and β parameters.In this work, the FiLM parameters are conditioned on task information, allowing task-dependent feature-space transformation.

B.3. Network Architectures

The architecture simplifies CNAPS by retaining task and feature adaptation while removing classifier adaptation, and its evaluation includes cross-validation across diverse domains.

  • B.3. Network Architectures: Autoregressive set encoders receive outputs from corresponding adapted residual blocks, so they do not require additional convolutional layers.Their inputs have already been processed by convolutional filters.
  • B.3. Network Architectures: The task encoder mean-pools convolutionally filtered support examples to produce a task representation, while adaptation modules use FiLM-based architecture.The supplied figure description also identifies a three-layer MLP with ELU and a residual connection for the classifier adaptor architecture.
  • B.3. Network Architectures: Simple CNAPS does not use CNAPS’s classifier adaptation network.The supplied architectural description identifies classifier adaptation as the component removed from the simplified model.
  • B.3. Network Architectures: 788,485 fewer parameters characterize Simple CNAPS relative to CNAPS while the simplified model outperforms CNAPS by considerable margins.
  • C. Cross Validation: Cross-validation shows Simple CNAPS outperforming CNAPS on 7–8 of 8 datasets in all four folds.The evaluation uses diverse visual domains and reports in-domain, out-of-domain, and overall averages.

D. Ablation study of the Feature Extractor Adaptation Network

Ablations show that feature-extractor adaptation is important for Mahalanobis-based classification, while adding a projection network generally reduces performance, especially out of domain.

  • D. Ablation study of the Feature Extractor Adaptation Network: Without feature-extractor adaptation, models lose approximately 15, 5, and 12 percentage points in in-domain, out-of-domain, and overall accuracy.Simple CNAPS nevertheless retains the lead, particularly for out-of-domain classification.
  • D. Ablation study of the Feature Extractor Adaptation Network: End-to-end learning of feature adaptation with the Mahalanobis distance helps produce a feature space suited to squared Mahalanobis classification.The ablation attributes this observation to the adapted feature space rather than to classifier adaptation.
  • D. Ablation study of the Feature Extractor Adaptation Network: Feature adaptation can reduce the importance of task regularization by decorrelating features and normalizing their variance.When adaptation is absent, an all-classes task-level covariance estimate becomes important for maintaining performance.
  • E. Projection Networks: Simple CNAPS +P generally performs worse than Simple CNAPS after adding a projection network.In out-of-domain evaluation, average accuracy is 67.1±0.8 for Simple CNAPS +P versus 69.7±0.8 for Simple CNAPS.
Loading 1912.03432v3…