Source-linked AI summary
Learning and aggregating deep local descriptors for instance-level recognition
Giorgos Tolias, Tomas Jenicek, Ondřej Chum
TL;DR
Global descriptors deteriorate on very large collections, while existing learned local descriptors can require more memory and search time. This paper learns local descriptors through image-level metric learning and combines them with ASMK; the proposed approach outperforms existing global and local descriptors on two landmark recognition tasks while using less memory than DELF.
Problem
Global descriptors deteriorate on very large collections, and learned local descriptors such as DELF can require more memory and search time than compact global descriptors.
Source: §1 · pp. 1–2Method
The method trains a sum-pooled global descriptor through image-level metric learning and uses internal network activations as local descriptors designed for ASMK.
Source: §1 · p. 2, §4.1 · p. 6Results
The proposed descriptors outperform existing global and local descriptors on two landmark recognition tasks, and HOW performs better than DELF with less memory.
Source: §1 · p. 2, §5.4 · p. 13Takeaways & Limitations
At large scale, local representations are essential, while global descriptors can perform well enough at small scale.
Source: §5.4 · p. 13
Abstract
from arXiv · showhide
We propose an efficient method to learn deep local descriptors for instance-level recognition. The training only requires examples of positive and negative image pairs and is performed as metric learning of sum-pooled global image descriptors. At inference, the local descriptors are provided by the activations of internal components of the network. We demonstrate why such an approach learns local descriptors that work well for image similarity estimation with classical efficient match kernel methods. The experimental validation studies the trade-off between performance and memory requirements of the state-of-the-art image search approach based on match kernels. Compared to existing local descriptors, the proposed ones perform better in two instance-level recognition tasks and keep memory requirements lower. We experimentally show that global descriptors are not effective enough at large scale and that local descriptors are essential. We achieve state-of-the-art performance, in some cases even with a backbone network as small as ResNet18.
1 Introduction
Instance-level recognition combines search over many classes with relatively small intra-class variation, but global descriptors deteriorate on very large collections. This work learns local features and descriptors through image-level metric learning, pairing them with ASMK for landmark classification and search.
1 Introduction
- Instance-level recognition commonly uses search and nearest-neighbor classification, with local-feature BoW approaches also enabling spatial verification for small objects.Strong performance is also reported without spatial verification.Source: §1 · p. 1
- Global descriptors deteriorate on very large image collections, while learned local descriptors such as DELF have higher memory and search costs than compact global descriptors.DELF features can also be redundant because deep-network activations are spatially correlated.Source: §1 · pp. 1–2
- Image-level metric learning trains local features and descriptors for ASMK, which can perform instance-level recognition without precise localization or spatial verification.Similar responses from nearby locations are averaged rather than suppressed.Source: §1 · p. 2
- The proposed deep local detector and descriptor with ASMK outperforms existing global and local descriptors on landmark classification and search.The ablation study also reports reduced ASMK memory requirements from the proposed components.Source: §1 · p. 2
2 Related work
Related work covers deep global descriptors, learned local features for instance-level recognition, and local feature methods developed for image registration. The paper distinguishes its large-scale recognition focus from registration work that typically requires dense correspondences and many features.
Global descriptors
- Deep global descriptors pool fully convolutional network feature maps; approaches vary in pooling, training, and attention, including SPoC, CroW, and NetVLAD.CroW uses activation magnitudes as spatial weights, while a related attention map is discussed in this paper.Source: §2 · p. 3
Local features and descriptors for instance-level recognition
- DELF learns attention over feature-map locations using category labels and classification loss, then selects high-attention locations whose activation vectors serve as local descriptors.Prior work also jointly learns DELF-like local and global descriptors, while another approach extends to image registration.Source: §2 · p. 3
Local features and descriptors for registration
- Registration methods target denser point correspondences and typically require many features, whereas this work focuses on large-scale instance recognition where memory matters.Some registration approaches use pre-trained networks without learning; others learn with or without local-level labels.Source: §2 · p. 3
3 Background
SMK compares quantized local descriptors through a thresholded match kernel, while ASMK aggregates descriptors assigned to each visual word. These match-kernel representations support efficient image similarity computation and retrieval.
3 Background
- SMK represents an image as local descriptors quantized to nearest codebook visual words, with each descriptor encoded by the sign of its residual from its assigned word.The image's descriptors are grouped by their assigned visual word.Source: §3 · p. 4
- SMK estimates image similarity by cross-matching local descriptors, but only pairs assigned to the same visual word contribute.The match kernel uses a threshold parameter, and the similarity has a normalization ensuring unit self-similarity.Source: §3 · p. 4
- ASMK aggregates descriptors assigned to each visual word into a single binary vector, and matching can be restricted to visual words shared by both images.ASMK is computationally and memory-wise more efficient than SMK and handles burstiness better in practice.Source: §3 · pp. 4–5
- The paper omits IDF weighting, pre-binarization random projections, and median-value thresholding because they were found unnecessary.Only binarized SMK and ASMK variants are considered; the original asterisk notation is omitted.Source: §3 · pp. 4–5
4 Method
The method trains a global sum-pooled descriptor with image-level pair labels, using its equivalence to local cross-matching to learn descriptors for ASMK. It adds descriptor transforms, fixed attention and local smoothing, then selects and ranks local descriptors at test time.
4 Method
- Global sum-pooled image similarity is equivalent to local descriptor cross-matching, enabling image-level metric learning as a surrogate for local-descriptor optimization.The training framework is designed to avoid end-to-end ASMK learning with large codebooks and hard visual-word assignment.Source: §4 · p. 5, §4.1 · p. 5
4.1 Derivation of the architecture
- The global objective implicitly suppresses background features, strengthens foreground features, draws corresponding local descriptors together, and separates non-corresponding ones.The authors present these effects as the reason global-descriptor optimization can train local descriptors for efficient match kernels such as ASMK.Source: §4.1 · p. 6
- Fixed L2-norm attention weights feature contributions, and fixed M×M average pooling smooths activations to improve ASMK aggregation.Weak features have less influence during training because only the strongest features are selected at test time.Source: §4.1 · pp. 6–7
- Mean subtraction captures negative evidence, while PCA whitening and dimensionality reduction transform descriptors through a fixed linear projection initialized from training-set descriptors.The transform combines mean subtraction, whitening, and dimensionality reduction in a single function.Source: §4.1 · p. 7
- Contrastive loss pulls positive image-pair descriptors together and pushes negative pairs apart, using image-level labels without local correspondences.The optimized descriptors are generated from dense FCN activations, each associated with a keypoint corresponding to its receptive field or a fraction of it.Source: §4.1 · pp. 5, 7
- At test time, the method keeps the strongest attention-ranked local descriptors and merges features from multiple image scales for ASMK search.Multi-scale extraction is used only at testing; descriptors from all resolutions are ranked jointly.Source: Figure 2 · p. 6, §4.1 · p. 7, Figure 3 · p. 8
4.2 Relation to prior work
- Unlike DELF’s learned attention and classification-loss training, the proposed descriptors use fixed attention and metric learning; experiments report superior performance with ASMK.The paper also contrasts its in-network whitening and reduction with post-processing approaches, while comparing learned and fixed attention in ablations.Source: §4.2 · p. 8
5 Experiments
Experiments evaluate landmark and building recognition through instance-level search and classification. They compare descriptor performance and memory, including ablations and large-scale benchmarks.
5.2 Implementation details
- Training uses SfM120k image pairs, ImageNet-pretrained ResNet18 or ResNet50 variants, and ASMK validation to select the best checkpoint every five epochs.The test setup uses 128-dimensional descriptors and multi-scale extraction; training runs for 20 epochs.Source: §5.1 · p. 9, §5.2 · p. 10
5.1 Datasets
- SfM120k supplies same-model overlapping positive pairs and different-model negatives; evaluation covers ROxford/RParis search and GLD2 classification without training on GLD2.The SfM120k validation set has 719 queries and 12,441 database images; the search benchmarks include one million distractors.Source: §5.1 · p. 9
5.3 Ablation experiments
- Ablations show that local smoothing reduces ASMK memory and can improve performance with the last ResNet block removed; fixed PCA-initialized h and contrastive loss are also beneficial.Internal local descriptors perform better than the optimized global descriptor.Source: §5.3 · pp. 11–12
5.4 Large-scale instance-level search
- On ROxford and RParis, the proposed local descriptors outperform competing approaches at large scale; ResNet18 ranks second in all ROxford cases and hard RParis +R1M.The authors also report that HOW performs better than DELF with less memory.Source: Table 2 · p. 13, §5.4 · p. 13
5.5 Large-scale instance-level classification
- On GLD2, the proposed local descriptors outperform other methods while using less memory than raw 2048D global descriptors.DELF, R18how, and −R50how store 347, 252, and 423 vectors per image on average, respectively.Source: Table 3 · p. 14, §5.5 · p. 14
6 Conclusions
The proposed architecture pairs deep local features with ASMK, with its state-of-the-art performance arising from their synergy. Local-feature methods outperform global descriptors on large-scale problems, and the approach remains competitive with a less powerful backbone.
6 Conclusions
- The proposed method consistently outperforms other methods on standard benchmarks, including with a less powerful backbone.The authors attribute state-of-the-art performance to the synergy between the proposed local feature detector and ASMK.Source: §6 · p. 14
- On large-scale problems, methods based on local features outperform global descriptors, and the proposed method outperforms other local feature detectors combined with ASMK.Source: §6 · p. 14
- The authors demonstrate that image-level supervision alone can produce an architecture effective at learning image similarity from local features.Source: §6 · p. 14