Source-linked AI summary
Revisiting Local Descriptor based Image-to-Class Measure for Few-shot Learning
Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, Jiebo Luo
TL;DR
Few-shot classification has limited class evidence, yet existing approaches commonly rely on image-level measures that may lose discriminative local information. DN4 learns deep local descriptors and classifies with a k-nearest-neighbor image-to-class measure in an end-to-end episodic framework. Across benchmark datasets, it consistently outperforms related state-of-the-art methods, with a largest absolute improvement of 17% over the next best method.
Problem
Few-shot examples may not represent class distributions effectively, while image-level final-classification measures can lose discriminative information under example scarcity.
Method
DN4 uses episodic end-to-end training to learn deep local descriptors and replaces image-level classification measures with a k-nearest-neighbor image-to-class measure.
Results
DN4 consistently outperforms related state-of-the-art methods on benchmark datasets, with a largest absolute improvement of 17% over the next best method.
Takeaways & Limitations
Deep local descriptors and image-to-class measurement are reported as more suitable for few-shot classification than image-level features and image-to-image measurement.
Abstract
from arXiv · showhide
Few-shot learning in image classification aims to learn a classifier to classify images when only few training examples are available for each class. Recent work has achieved promising classification performance, where an image-level feature based measure is usually used. In this paper, we argue that a measure at such a level may not be effective enough in light of the scarcity of examples in few-shot learning. Instead, we think a local descriptor based image-to-class measure should be taken, inspired by its surprising success in the heydays of local invariant features. Specifically, building upon the recent episodic training mechanism, we propose a Deep Nearest Neighbor Neural Network (DN4 in short) and train it in an end-to-end manner. Its key difference from the literature is the replacement of the image-level feature based measure in the final layer by a local descriptor based image-to-class measure. This measure is conducted online via a $k$-nearest neighbor search over the deep local descriptors of convolutional feature maps. The proposed DN4 not only learns the optimal deep local descriptors for the image-to-class measure, but also utilizes the higher efficiency of such a measure in the case of example scarcity, thanks to the exchangeability of visual patterns across the images in the same class. Our work leads to a simple, effective, and computationally efficient framework for few-shot learning. Experimental study on benchmark datasets consistently shows its superiority over the related state-of-the-art, with the largest absolute improvement of $17\%$ over the next best. The source code can be available from \UrlFont{https://github.com/WenbinLee/DN4.git}.
1. Introduction
Few-shot learning is difficult because a few examples poorly represent each class, while existing methods commonly use image-level measures that may discard useful local information. DN4 addresses this issue with a deep local-descriptor image-to-class measure and reports stronger benchmark performance.
- Motivation: Few-shot learning must generalize to unseen classes from only one or a few examples per class.The limited examples make effective class-distribution representation difficult.
- Motivation: Existing methods emphasize knowledge transfer, representation, or relation learning but commonly use pooled image-level features for final classification.The paper argues that this practice may be inappropriate under few-shot example scarcity.
- Proposed approach: DN4 replaces the image-level final measure with a local-descriptor image-to-class measure computed by k-nearest-neighbor search and trained end-to-end.The framework follows episodic training and learns deep local descriptors through convolutional neural networks.
- Results: DN4 improves miniImageNet 1-shot accuracy from 50.44% to 51.24% and 5-shot accuracy from 66.53% to 71.02%.On fine-grained datasets, its largest absolute improvement over the next-best method is 17%.
2. Related Work
Related few-shot methods use either meta-learning architectures or learned similarity metrics. DN4 belongs to metric learning but differs by classifying with deep local descriptors and an image-to-class measure rather than image-level features.
- Meta-learning methods: Meta-learning methods train across-task meta-learners, often using recurrent or memory-based architectures to generalize to unseen tasks.These methods learn mechanisms for storing knowledge or updating learner parameters.
- Meta-learning methods: Complicated memory-addressing architectures can be difficult to train because of temporally linear hidden-state dependencies.DN4 is described as easier to train end-to-end from scratch with a common CNN.
- Metric-learning methods: Metric-learning methods learn informative similarity metrics and representations for few-shot classification, including through episodic training.Representative approaches combine discriminative embeddings, attention, or memory mechanisms.
- DN4's distinction: DN4 differs from related metric-learning methods by using deep local descriptors and an image-to-class measure instead of image-level features.The paper reports that DN4 outperforms several state-of-the-art metric-learning methods.
3. The Proposed Method
DN4 formulates few-shot classification around episodic support-query tasks and replaces image-level comparison with a non-parametric image-to-class measure over learned local descriptors. Its CNN embedding and k-NN modules are trained end-to-end, while the local-descriptor search is designed to remain efficient in few-shot settings.
- Problem formulation: A C-way K-shot task classifies query images using a support set containing C classes and K labeled samples per class.An auxiliary set with disjoint labels supplies transferable knowledge through episodic training.
- Problem formulation: Episodic training repeatedly samples support and query sets from the auxiliary data to simulate few-shot tasks.The training episodes generally match the test task's numbers of ways and shots.
- Motivation from NBNN: NBNN motivates retaining local descriptors because image-level pooling can lose discriminative information and image-to-image similarity may not generalize beyond training images.An image-to-class measure instead aggregates local features across images in the same class.
- DN4 framework: DN4 integrates a deep embedding module Ψ with an image-to-class module Φ in a unified network trained end-to-end from scratch.The embedding module learns local descriptors, while the measure module compares query descriptors with class-specific descriptor spaces.
- DN4 framework: Each image is represented by an h×w×d tensor of m=hw local descriptors, and the class similarity is computed by k-NN over descriptors from all training images in that class.For an 84×84 image in the experiments, h=w=21 and d=64, yielding 441 descriptors.
- DN4 framework: The image-to-class similarity sums cosine similarities between each query descriptor and its k nearest class descriptors.The measure is non-parametric and can use other similarity or distance functions.
- Efficiency: Few-shot scarcity reduces the nearest-neighbor search pool, weakening a major computational obstacle associated with NBNN.DN4 also benefits from deep feature representations rather than NBNN's handcrafted features.
- Network architecture: The standard Conv-64F embedding uses four convolutional blocks with batch normalization and Leaky ReLU, plus max pooling in the first two blocks.During inference, the largest component of the C-dimensional similarity vector determines the predicted class.
4. Experimental Results
Experiments across generic and fine-grained few-shot benchmarks show that DN4’s end-to-end local descriptor image-to-class measure consistently outperforms related methods, while retaining efficient episodic evaluation.
- Generic few-shot classification: DN4 gains 15.82%, 10.42%, 7.91% and 4.05% over SNAIL, Meta-Learner LSTM, MAML and MM-Net in the 5-way 5-shot setting.The framework uses one unified network rather than additional complicated memory-addressing architectures.
- Fine-grained few-shot classification: 17% is DN4’s largest absolute improvement over the second-best method, GNN, on Stanford Cars under the 5-shot fine-grained setting.DN4 performs less well under the 1-shot setting because its k-nearest-neighbor performance depends on available examples.
- Backbone networks: DN4 reaches 54.37 ± 0.36% for 5-way 1-shot and 74.44 ± 0.29% for 5-way 5-shot classification with the ResNet-256F backbone.With the same backbone, DN4 gains 4.31% over Dynamic-Net under the 5-shot setting.
- Runtime: DN4 trains in 0.31s and 0.38s per 5-way 1-shot or 5-shot episode and tests in 0.18s per episode on the reported hardware.The authors note that optimized parallel implementation could further improve efficiency.
5. Conclusions
The paper proposes DN4, a few-shot learning framework centered on learnable deep local descriptors and a local descriptor-based image-to-class measure.
- DN4 revisits the local descriptor-based image-to-class measure for few-shot learning.
- Learnable deep local descriptors are emphasized as more suitable than image-level features for the few-shot problem.
- The image-to-class measure is reported as superior to the image-to-image measure because of visual-pattern exchangeability within classes.