PaperlayerView exact revision on arXiv

Source-linked AI summary

Matching Feature Sets for Few-Shot Image Classification

Arman Afrasiyabi, Hugo Larochelle, Jean-François Lalonde, Christian Gagné

arXiv:2204.00949v1cs.CV

TL;DR

Few-shot image classification commonly represents each image with a single feature vector, despite the goal of transferring knowledge from base to novel classes. SetFeat instead extracts feature sets and compares them with set-to-set metrics; it outperforms the state of the art in almost all reported cases.

  • Problem

    Few-shot image classification commonly uses a single feature vector per image to transfer knowledge from abundant base classes to novel classes with few examples.

    Source: §1 · p. 1

  • Method

    SetFeat inserts shallow self-attention mappers into existing feature extractors to produce feature sets, then classifies using set-to-set matching.

    Source: §4.1 · p. 3, §4.2 · p. 3

  • Results

    SetFeat outperforms the state of the art in almost all cases, with a 1.83% 1-shot accuracy gain on miniImageNet.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Set-based representations improve few-shot classification without increasing the total number of network parameters.

    Source: §1 · p. 1

    Experiments use a fixed set of 10 mappers; using more was dismissed because it would require fewer filters and could cause underfitting.

    Source: §7 · p. 8

Abstract

from arXiv · show

In image classification, it is common practice to train deep networks to extract a single feature vector per input image. Few-shot classification methods also mostly follow this trend. In this work, we depart from this established direction and instead propose to extract sets of feature vectors for each image. We argue that a set-based representation intrinsically builds a richer representation of images from the base classes, which can subsequently better transfer to the few-shot classes. To do so, we propose to adapt existing feature extractors to instead produce sets of feature vectors from images. Our approach, dubbed SetFeat, embeds shallow self-attention mechanisms inside existing encoder architectures. The attention modules are lightweight, and as such our method results in encoders that have approximately the same number of parameters as their original versions. During training and inference, a set-to-set matching metric is used to perform image classification. The effectiveness of our proposed architecture and metrics is demonstrated via thorough experiments on standard few-shot datasets -- namely miniImageNet, tieredImageNet, and CUB -- in both the 1- and 5-shot scenarios. In all cases but one, our method outperforms the state-of-the-art.

1. Introduction

The paper replaces single-vector image representations with sets of features for few-shot classification, aiming for richer transfer to novel classes. It positions SetFeat among inductive few-shot approaches while distinguishing its per-image feature sets and set-to-set matching from related methods.

1. Introduction

  • Few-shot classification commonly transfers features learned from abundant base classes to novel classes using a single feature vector per image.The paper seeks a richer representation that captures different image characteristics and scales.Source: §1 · p. 1
  • SetFeat inserts shallow, independent attention-based mappers at multiple backbone scales and classifies by set-to-set distances between query and support features.It uses classification loss at each mapper during pretraining, followed by meta-training; three set-to-set metrics are proposed.Source: §1 · p. 1, §2 · p. 2
  • SetFeat outperforms the state of the art in almost all cases, with 1-shot accuracy gains of 1.83%, 1.42%, and 1.83% on miniImageNet, tieredImageNet, and CUB, respectively.The reported gains are over baselines; the method uses set representations without increasing total network parameters.Source: §1 · p. 1

2. Related work

  • SetFeat uses inductive few-shot classification rather than transductive methods that exploit structure across the entire novel set, and requires no extra data beyond base classes.The paper also relates its feature sets to multi-scale and deep-set approaches while applying them to few-shot image classification.Source: §2 · p. 2
  • Unlike methods that use attention to adapt episode features or support prototypes, SetFeat extracts a feature set for each query and support example.Its sets remain multi-element even in the 1-shot case.Source: §2 · p. 2

3. Preliminaries

The preliminaries define few-shot classification as predicting a query label from a small labeled support set. SetFeat replaces the usual single feature vector with a set extracted by mappers.

3. Preliminaries

  • In N-way K-shot classification, the query is classified among N classes using K labeled support examples per class.The support set is divided into class-specific sets of example-label pairs.Source: §3 · p. 2
  • The feature extractor is a convolutional network with B blocks, each parameterized separately, and intermediate block features are denoted z_b.A block groups convolutional layers, possibly with skip links, and typically ends with spatial downscaling such as pooling.Source: §3 · p. 2
  • SetFeat extracts M feature vectors per image through shallow self-attention mappers and uses set-to-set metrics to compare images.This contrasts with the common single-vector representation.Source: §3 · p. 2

4. Set-based few-shot image classification

SetFeat adapts image encoders to produce feature sets through attention mappers, then uses set-to-set metrics for few-shot classification. The section describes its extractor, matching metrics, inference, and two-stage training procedure.

4.1. Set-feature extractor

  • Each shallow, single-head mapper extracts a separate feature representation from a backbone block, and averaging its patches yields the mapper feature vector.For ResNet, the attention output is added residually, with a 1 × 1 convolution used when dimensions mismatch.Source: §4.1 · p. 3

4.2. Set-to-set matching metrics

  • SetFeat compares query and support feature sets with three set-to-set metrics, unlike FEAT and MatchingNetwork, which form sets over the support set directly.The method retains multiple feature elements even in the 1-shot case and uses non-parametric metrics for inference.Source: §4.2 · pp. 3–4

4.3. Inference

  • Inference follows Prototypical Networks by assigning a query to the support class with the smallest set-to-set distance.The method models query-class probabilities using one of its set metrics.Source: §4.3 · p. 4

4.4. Training procedure

  • Training first uses class-label cross-entropy to train each mapper independently, then episodic training simulates few-shot tasks and updates the network using query losses.The first stage adds fully connected layers to map mapper features to class logits; these layers are discarded for the second stage.Source: §4.4 · p. 4

5. Evaluation

Evaluation tests SetFeat with conventional few-shot backbones and compares its set-matching metrics across several datasets. Results report gains over baselines on miniImageNet, tieredImageNet, and CUB, with one stated exception.

5.2. Datasets and implementation details

  • The experiments evaluate SetFeat and its set-matching metrics across miniImageNet, tieredImageNet, and CUB, including fine-grained classification on CUB.The evaluation uses four backbones and three datasets.Source: §5 · p. 5, §5.2 · p. 5

5.1. Backbones

  • The evaluation uses Conv4-64, Conv4-512, and ResNet12 backbones, with 10 mappers distributed across blocks in a 1-2-3-4 pattern.SetFeat4-512 and SetFeat12 have slightly fewer parameters than their counterparts; SetFeat4-64 has more because reducing its parameters collapses training.Source: §5.1 · p. 5

5.3. Quantitative and comparative evaluations

  • SetFeat4-64 (sum-min) gains 1.83% and 1.4% over MELR on miniImageNet in 1- and 5-shot, except that sum-min does not outperform the state of the art for SetFeat12 in 5-shot.Across the evaluated metrics, sum-min otherwise outperforms the other proposed metrics and the state of the art.Source: Table 1 · p. 5, §5.3 · p. 6
  • SetFeat’s sum-min metric improves over Distill by 1.42% and 0.51% on tieredImageNet in 1- and 5-shot, respectively.The cited evaluation notes that Distill, MELR, and FEAT contain more parameters than ResNet12 and SetFeat12.Source: §5.3 · p. 6, Table 2 · p. 6
  • On CUB, SetFeat4-64 (min-min) gains 1.83% and 2.04% over MELR in 1- and 5-shot, while parameter-matched SetFeat12* sets a new state of the art against ResNet18.SetFeat12* has 11.466M parameters versus 11.511M for ResNet18.Source: §5.3 · p. 6, Table 3 · p. 6

6. Ablation

The ablations examine mapper placement, whether gains come from added parameters, mapper use, and visual specialization. They find advantages for distributed multi-scale sets and for aggregating more mappers, while each mapper is used across validation classes.

6.1. Mapper configurations

  • Distributed mapper layouts outperform placing all ten mappers at the final layer; progressive growth generally matches or beats other distributed layouts.The comparison uses miniImageNet validation accuracy with min-sum; the number of mappers is held constant, and concatenating their outputs as one vector cancels the gain.Source: §6.1 · pp. 6–7, Table 4 · p. 7

6.2. Over-parameterization of SetFeat4-64

  • At matched parameter counts, augmented Conv4-64 improves over the baseline but falls short of SetFeat4-64, so added parameters alone do not explain SetFeat’s gains.The comparison uses 0.239M parameters for augmented Conv4-64 and 0.238M for SetFeat4-64.Source: §6.2 · p. 7, Table 5 · p. 7

6.3. Probing the activation of mappers

  • All ten mappers are selected across miniImageNet validation classes, although earlier mappers are active more often.The activation comparison uses SetFeat12 over 600 5-way, 1-shot episodes.Source: §6.3 · p. 7, Figure 3 · p. 7

6.4. Top-m analysis

  • Classification results progressively improve as the top-m aggregation moves toward sum-min, which uses all mappers.The ablation compares top-m variants of min-sum on CUB validation.Source: §6.4 · pp. 7–8, Table 6 · p. 8

6.5. Visualizing mappers saliency

  • SetFeat saliency maps cover more image regions than the single-feature baseline, with its mappers jointly highlighting all three dogs in one example.The figure compares input images, baseline saliency maps, and mapper-specific maps on training and validation examples.Source: §6.5 · p. 8

7. Discussion

SetFeat extracts and matches feature sets using attention-based mappers at different backbone stages. The discussion reports state-of-the-art results on three datasets and notes that experiments fixed the mapper count at ten.

7. Discussion

  • SetFeat achieves state-of-the-art results on miniImageNet, tieredImageNet, and CUB across four adaptations of two backbones.The adapted backbones’ parameter counts are reduced to account for the mapper parameters.Source: §7 · p. 8
  • The saliency comparison presents input images, baseline maps, and SetFeat maps for training and validation examples.The figure includes three training examples and four miniImageNet validation examples.Source: Figure 5 · p. 8
  • The authors evaluated a fixed M = 10 mappers; adding more could require fewer backbone filters and risk underfitting.They identify testing more mappers, potentially with larger backbones, as future work.Source: §7 · p. 8

Dataset and backbone specifications

The supplementary tables document dataset specifications and compare SetFeat parameter counts with commonly used backbones.

Dataset and backbone specifications

Ablation with more ways and cross-domain results from miniImageNet 7→CUB

SetFeat is evaluated in 5-shot, higher-way, and cross-domain settings. It reaches state-of-the-art results in 5-, 10-, and 20-way 5-shot classification, while its cross-domain CUB result remains below MixtFSL.

Ablation with more ways and cross-domain results from miniImageNet 7→CUB

  • SetFeat12* has 11.466M parameters, compared with 11.511M for its ResNet18 counterpart.The table reports 5-way, 10-way, and 20-way comparisons of SetFeat12* and SetFeat12 with ResNet18 and ResNet12, respectively.Source: Ablation with more ways and… · p. 13
  • 6.18% and 2.84%: SetFeat12* and SetFeat12 gain over MixtFSL, respectively, in 5-way 5-shot classification.SetFeat with sum-min also achieves state-of-the-art results in 5-way, 10-way, and 20-way 5-shot classification.Source: Ablation with more ways and… · p. 13, Table 9 · p. 13
  • 0.92%: SetFeat12* is below MixtFSL when pre-trained on miniImageNet and tested on CUB.This is the cross-domain adaptation result reported in the last column of Table 9.Source: Ablation with more ways and… · p. 13, Table 9 · p. 13
  • Mapper saliency maps are shown alongside ResNet12 baseline maps for training and validation examples from miniImageNet.The figure includes ten mapper maps and five examples from each split.Source: Figure 6 · p. 14

Visualizing mappers saliency

The saliency maps compare mapper attention across SetFeat and baseline encoders, with the two SetFeat variants focusing on different input regions.

Class structure in cluster

Mapper-wise tSNE visualizations show class structure in train and validation sets, while a Hausdorff-distance variant is compared with Sum-min on miniIN and CUB.

Class structure in cluster

Hausdorff distance ablation

Related papers

Loading 2204.00949v1…