Source-linked AI summary
RepMet: Representative-based metric learning for classification and one-shot object detection
Leonid Karlinsky, Joseph Shtok, Sivan Harary, Eli Schwartz, Amit Aides, Rogerio Feris, Raja Giryes, Alex M. Bronstein
TL;DR
Few-shot recognition and detection require effective class modeling when only a few examples are available, with detection also needing object localization amid background. RepMet jointly learns a backbone, embedding, and multi-modal class representatives, and uses them as a DML classifier head for classification and detection. It improves DML classification performance across fine-grained datasets, reports strong few-shot detection results on ImageNet-LOC, and introduces an episodic benchmark.
Problem
Few-shot learning must classify categories from limited examples, while few-shot detection additionally must recognize and localize objects amid background.
Method
RepMet jointly learns backbone parameters, an embedding space, and multi-modal class representatives, then uses a DML subnet as a classifier head for detection.
Results
RepMet improves state-of-the-art DML classification results on fine-grained datasets and achieves the best reported ImageNet-LOC few-shot detection results compared with strong baselines.
Takeaways & Limitations
The method supports introducing unseen detection categories using one or a few examples and provides a 500-episode ImageNet-LOC benchmark for future evaluation.
Takeaways & Limitations
The current implementation assumes isotropic Gaussian mixtures with fixed variance and does not learn mixture coefficients or covariances.
Abstract
from arXiv · showhide
Distance metric learning (DML) has been successfully applied to object classification, both in the standard regime of rich training data and in the few-shot scenario, where each category is represented by only a few examples. In this work, we propose a new method for DML that simultaneously learns the backbone network parameters, the embedding space, and the multi-modal distribution of each of the training categories in that space, in a single end-to-end training process. Our approach outperforms state-of-the-art methods for DML-based object classification on a variety of standard fine-grained datasets. Furthermore, we demonstrate the effectiveness of our approach on the problem of few-shot object detection, by incorporating the proposed DML architecture as a classification head into a standard object detection model. We achieve the best results on the ImageNet-LOC dataset compared to strong baselines, when only a few training examples are available. We also offer the community a new episodic benchmark based on the ImageNet dataset for the few-shot object detection task.
1. Introduction
The paper addresses few-shot recognition and detection by jointly learning a DML embedding, backbone parameters, and multi-modal class representations, then using those representations to introduce unseen categories with few examples.
- Few-shot learning addresses classification when only a few training samples are available per class.
- Few-shot object detection additionally requires recognizing and localizing objects amid image background, a relatively underexplored problem.
- RepMet represents each class with a multi-modal mixture and jointly learns its representatives, embedding space, and backbone parameters end-to-end.
- The method replaces a detector’s classifier head with a DML subnet that computes class posteriors for each region of interest.
- New categories are introduced by replacing learned representatives with embedding vectors computed from foreground regions in their few training examples.
- RepMet improves state-of-the-art DML classification results across fine-grained datasets and achieves favorable few-shot detection results compared with prior work and baselines.
2. Related work
Prior DML methods learn semantically meaningful embeddings for classification and retrieval, while few-shot methods use episodic or prototype-based representations; applying them to detection remains difficult.
- Distance Metric Learning: Deep DML typically learns embeddings where same-category objects are close under L2 distance and different-category objects are far apart.
- Distance Metric Learning: Class posteriors are commonly computed with k-NN or parametric models, often assuming uni-modal category distributions in the embedding space.
- Few-shot Learning: Matching Networks organize training and testing into episodes containing multiple categories, several examples per category, and a query image.
- Few-shot Learning: Prototype-based methods represent each class by the centroid of its batch elements, while later work includes meta-learning and synthesis methods.
- Few-shot Learning: Few-shot detection is harder because detector batches often contain one image with highly unbalanced foreground and background regions, limiting existing few-shot learners.
3. RepMet Architecture
RepMet jointly learns an embedding, class-specific multi-modal representatives, and posterior computations in one subnet attached to a backbone. Its training combines classification cross-entropy with a margin loss, while detection uses representative-based foreground and background probabilities.
- RepMet Architecture: The subnet jointly trains a DML embedding with a multi-modal mixture distribution for every category and can serve as a classifier head for classification or detection backbones.The embedding-producing backbone is trained jointly with the DML subnet.
- Embedding and representatives: Each pooled backbone feature X is mapped to an embedding E, while K learned representatives Rij encode the modes of each class mixture.The embedding dimension e is typically much smaller than the backbone feature dimension f.
- Class posterior: Distances from E to every representative determine the probability of each class mode, and the class posterior is the maximum probability across that class’s modes.This approximation is an upper bound on the actual class posterior and avoids learning mixture coefficients when representatives are replaced by novel-class examples.
- Class posterior: The model assumes class mixtures of isotropic multivariate Gaussians with variance σ^2 and does not learn their mixing coefficients.The fixed mixture assumption supports representative-based posterior computation for novel classes.
- Background handling: Detection estimates an open-background posterior from foreground class probabilities rather than modeling the background distribution directly.The background probability is obtained through a lower-bound construction based on foreground probabilities.
- Training losses: Training sums cross-entropy with a margin loss requiring the correct class’s closest representative to be at least α nearer than the closest wrong-class representative.The loss is applied to the DML subnet and backbone, using image features or region-of-interest features.
4. Implementation details
The implementation uses different backbone and posterior configurations for classification and few-shot detection. Classification employs an InceptionV3-based subnet, whereas detection replaces the Faster-RCNN classifier head on a deformable FPN and trains the full model end to end.
- Overview: The implementation details include separate configurations for DML-based classification and few-shot detection, with code intended for release upon acceptance.The classification and detection setups are described in separate sections.
- DML-based classification: Classification attaches the DML subnet before InceptionV3’s final fully connected layer, using two fully connected embedding layers followed by L2 normalization.The embedding layers have sizes 2048 and 1024, with batch normalization and ReLU on the first layer.
- DML-based classification: Because standard classification benchmarks lack a background category, classification uses equally weighted mixture modes and a softer normalized class posterior.The softer posterior was experimentally found more beneficial for DML-based classification.
- Few-shot detection: For few-shot detection, the DML subnet replaces the RCNN classification head on a deformable-convolution FPN backbone and the entire network is trained end to end.The implementation also uses online hard example mining and SoftNMS.
5. Results
The proposed DML approach improves fine-grained classification and supports few-shot detection through jointly learned embeddings and representatives. Experiments also evaluate attribute organization, hyperparameter robustness, unseen-category detection, and benchmark design.
- Fine-grained classification: The method outperforms state-of-the-art DML classification approaches on all tested fine-grained datasets except Oxford Flowers.The evaluated datasets include Stanford Dogs, Oxford-IIIT Pet, Oxford 102 Flowers, and ImageNet Attributes.
- Attribute distribution: Improved attribute precision shows that images with similar attributes become closer in the embedding space despite attribute annotations being unused during training.The evaluation uses ImageNet Attributes training data and Object Attributes annotations across varying neighborhood sizes.
- Hyperparameter robustness – ablation study: The method is more robust than the Magnet loss to changes in embedding architecture and the number of representatives per class.Robustness is measured by the standard deviation of classification error across architectures and representative counts during training.
- Few-shot detection benchmark: The proposed benchmark uses 500 random ImageNet-LOC episodes, with consistent episodes across 1-, 5-, and 10-shot experiments.Each n-shot, m-way episode contains n training examples per class and 10·m random query images.
- Few-shot object detection: Few-shot detection evaluates unseen ImageNet-LOC categories using representatives replaced by embeddings computed from the few available examples.The evaluation reports mAP on 5-way detection tasks across 500 test episodes.
- Few-shot object detection: Joint end-to-end training is reported as crucial because externally trained embeddings use limited background-ROI sampling and produce higher false-positive rates.The external baseline sampled 200 ROIs per image, reducing its ability to reject background regions.
- Few-shot object detection: The modified detector is slightly inferior to vanilla FPN-DCN on Pascal VOC but compares favorably on the 100 first ImageNet-LOC categories.This comparison tests whether replacing the RCNN classifier with the DML subnet harms performance on seen classes.
6. Summary & Conclusions
The paper presents a DML method that achieves state-of-the-art object classification and compares favorably in few-shot detection. It also introduces an ImageNet-LOC benchmark and identifies representative prediction and sample synthesis as future directions.
- 6. Summary & Conclusions: The proposed DML method achieves state-of-the-art object-classification performance against other DML-based approaches.The paper also reports a favorable comparison with few-shot detection state of the art.
- 6. Summary & Conclusions: The work introduces one of the first few-shot detection approaches and a benchmark based on ImageNet-LOC with multiple random episodes.The benchmark is intended to encourage further investigation of few-shot object detection.
- 6. Summary & Conclusions: Future work includes predicting mixture coefficients and covariances from the input and generating samples automatically to enrich representatives.