Source-linked AI summary
Large-Scale Image Retrieval with Attentive Deep Local Features
Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, Bohyung Han
TL;DR
Large-scale image retrieval remains difficult under clutter, occlusion, and varied imaging conditions, while existing datasets provide limited scale and diversity. The paper proposes DELF, an attention-guided CNN local descriptor trained with image-level labels, and evaluates it on Google-Landmarks. DELF outperforms existing global and local descriptors by substantial margins in the large-scale setting.
Problem
Existing global descriptors struggle with clutter, occlusion, and viewpoint or illumination variation, while prevailing retrieval datasets are small and limited in diversity.
Method
DELF is a weakly supervised CNN local descriptor coupled with attention-based semantic feature selection, trained using image-level labels and integrated with Google-Landmarks evaluation.
Results
DELF outperforms existing global and local descriptors by substantial margins in the large-scale Google-Landmarks setting.
Takeaways & Limitations
A single forward pass obtains DELF keypoints and descriptors, supporting large-scale image retrieval with attention-based local feature selection.
Takeaways & Limitations
Google-Landmarks ground truth is noisy because GPS errors, distant photography, ambiguous landmarks, and multiple instances complicate annotation.
Abstract
from arXiv · showhide
We propose an attentive local feature descriptor suitable for large-scale image retrieval, referred to as DELF (DEep Local Feature). The new feature is based on convolutional neural networks, which are trained only with image-level annotations on a landmark image dataset. To identify semantically useful local features for image retrieval, we also propose an attention mechanism for keypoint selection, which shares most network layers with the descriptor. This framework can be used for image retrieval as a drop-in replacement for other keypoint detectors and descriptors, enabling more accurate feature matching and geometric verification. Our system produces reliable confidence scores to reject false positives---in particular, it is robust against queries that have no correct match in the database. To evaluate the proposed descriptor, we introduce a new large-scale dataset, referred to as Google-Landmarks dataset, which involves challenges in both database and query such as background clutter, partial occlusion, multiple landmarks, objects in variable scales, etc. We show that DELF outperforms the state-of-the-art global and local descriptors in the large-scale setting by significant margins. Code and dataset can be found at the project webpage: https://github.com/tensorflow/models/tree/master/research/delf .
†POSTECH, Korea
The supplied passage contains contact information for the authors.
- The passage lists the authors’ contact email address.
1. Introduction
The paper addresses limitations of existing large-scale image retrieval methods and datasets by proposing DELF and Google-Landmarks. DELF uses attention to select semantically useful local features, while the dataset enables challenging large-scale evaluation.
- Motivation: Large-scale retrieval is challenged by clutter, occlusion, viewpoint, and illumination changes that hinder global descriptors.Global descriptors lack patch-level matching, making partial matching difficult under clutter and occlusion.
- Motivation: Existing retrieval datasets are often small, use few queries, and provide limited diversity in landmark locations and types.The paper motivates a larger dataset for more comprehensive and statistically meaningful evaluation.
- Contribution: Google-Landmarks contains more than 1M landmark images, almost 13K landmarks, and an additional 100K diverse queries, including unmatched queries.This design evaluates robustness when queries do not necessarily depict landmarks.
- Contribution: DELF is a CNN-based local feature with weakly supervised attention trained from image-level labels to select semantically useful features.The attention model shares most network layers with the descriptor and requires no object- or patch-level annotations.
- Contribution: DELF achieves state-of-the-art retrieval performance with significant margins over existing global and local descriptors.The proposed system is presented as a large-scale retrieval framework based on a novel CNN-based feature descriptor.
2. Related Work
Prior work spans hand-crafted local features, global aggregation and CNN descriptors, CNN-based local features, and visual attention. The paper positions attention-based semantic feature selection as underexplored for image retrieval.
- Instance Retrieval: Early instance-retrieval systems combined hand-crafted local features with approximate nearest-neighbor search and geometric re-ranking.These feature-based techniques remained strong when high precision was required.
- Global Descriptors: VLAD and Fisher Vector aggregate local features into compact global descriptors for high-performance image retrieval.
- Global Descriptors: CNN-based global descriptors use pretrained or learned networks and are commonly trained with triplet loss to preserve relevant-image rankings.Some methods use deep local features within aggregation techniques such as VLAD or Fisher Vector.
- Local Features: CNN-based local-feature methods address keypoint detection, orientation estimation, patch representation, and metric learning.The related work includes methods such as MatchNet, DeepCompare, and LIFT.
- Visual Attention: Visual attention is widely used in recognition tasks but had not been actively explored for learning visual features in image retrieval.
3. Google-Landmarks Dataset
Google-Landmarks is a geographically broad, large-scale retrieval dataset designed around diverse landmarks, challenging image conditions, and noisy or irrelevant queries. Its ground truth uses visual features and GPS coordinates but can contain GPS-related annotation noise.
- Dataset Scale and Coverage: Google-Landmarks contains 1,060,709 images from 12,894 landmarks and 111,036 additional query images.Images were captured at various locations worldwide and associated with GPS coordinates.
- Dataset Challenges: The dataset includes foreground and background clutter, occlusion, partially out-of-view objects, and other wild variations.Example images include small objects and multiple landmarks.
- Dataset Challenges: Some personal-photo queries contain no landmarks and serve as distractors for evaluating robustness to irrelevant and noisy queries.
- Ground Truth: Ground truth clusters database images using visual features and GPS coordinates, then assigns landmark identity by a physical-distance threshold.
- Ground Truth: The ground-truth construction is noisy because GPS errors, distant photography, ambiguous landmarks, and multiple instances complicate annotation.The authors report few incorrect annotations when checking a subset with a 25km threshold.
4. Image Retrieval with DELF
DELF combines dense CNN feature extraction with attention-based keypoint selection, dimensionality reduction, and large-scale indexed retrieval. The attention model learns relevance scores from image-level classification supervision, selecting discriminative features for efficient matching and geometric verification.
- System overview: The retrieval pipeline comprises dense localized feature extraction, keypoint selection, dimensionality reduction, and indexing and retrieval.These four blocks define the system's main processing stages.
- Dense localized feature extraction: Dense local descriptors are extracted with a fully convolutional ResNet50 using an image pyramid to handle scale changes.The conv4_x feature maps form a grid of descriptors localized by their receptive fields.
- Descriptor learning: Image-level landmark classification training improves local descriptor relevance without object- or patch-level labels.The network is fine-tuned with cross-entropy loss, producing representations suited to landmark retrieval.
- Attention-based keypoint selection: Attention scores are learned from weighted feature pooling and a softmax landmark classifier, using a score function for each local feature.The attention model uses a non-negative two-layer CNN score function with softplus activation and shares the descriptor network's feature representation.
- Training strategy: The model uses two-step training because jointly learning descriptors and attention by backpropagation produced weak models in practice.Descriptors are fine-tuned first, followed by learning the attention score function; random rescaling is used during attention training.
- Attention-based keypoint selection: Keypoint selection follows descriptor extraction, prioritizing semantically discriminative features rather than relying only on repeatable low-level characteristics.This ordering contrasts with SIFT and LIFT and targets higher-level recognition needs.
- Indexing and retrieval: Selected descriptors are reduced to 40 dimensions, encoded with 50-bit product quantization, and retrieved with KD-tree, inverted-index, and geometric-verification stages.RANSAC uses the number of inliers as the retrieval score, while the index requires less than 8GB for 1 billion descriptors and has under 2-second search latency in the reported setup.
5. Experiments
Experiments evaluate DELF against global and local descriptors on large-scale and existing retrieval datasets. DELF substantially outperforms alternatives in the challenging large-scale setting, with attention, local matching, and complementary information driving its practical advantages.
- Quantitative Results: The large-scale experiment compares DELF with recent global and local descriptors on the Google-Landmarks dataset.Evaluated methods include DIR, siaMAC, CONGAS, and LIFT; LIFT extraction was infeasible at this scale.
- Quantitative Results: DELF clearly outperforms all other techniques, while DIR degrades with query expansion and CONGAS remains substantially worse.The evaluation uses a full precision-recall curve with unnormalized recall and threshold-dependent operating points.
- Quantitative Results: Both fine-tuning and attention modeling substantially improve performance, with attention more important than fine-tuning.Attention learns to select discriminative retrieval features even when descriptors use ImageNet-pretrained features without fine-tuning.
- Quantitative Results: DELF, CONGAS, and DIR require approximately 8GB to index the dataset, despite differences in descriptor representation.DELF and CONGAS use the same feature dimensionality and maximum number of features per image, whereas DIR stores 8KB per image.
- Qualitative Results: DELF’s local-region matches help identify the same object across imaging conditions, while global descriptors confuse similar scenes and small objects.DELF can also fail when localized patterns such as floor tiling or vegetation recur across landmarks.
- Qualitative Results: Compared with CONGAS, DELF retrieves more relevant landmarks and successfully matches challenging pairs involving occlusion, distracting objects, and clutter.Large receptive fields may center features outside the landmark, while still incorporating discriminative neighboring regions.
- Results in Existing Datasets: On Oxford and Paris datasets and their distractor extensions, combining DELF with DIR nontrivially improves accuracy although DELF alone is not best.The result indicates that DELF encodes information complementary to global descriptors.
6. Conclusion
DELF is a weakly supervised local feature descriptor with attention-based semantic feature selection, designed for large-scale image retrieval. Evaluated on Google-Landmarks, it substantially outperforms existing global and local descriptors.
- DELF is a local feature descriptor designed specifically for large-scale image retrieval applications.
- DELF uses image-level labels only and couples descriptor learning with an attention mechanism for semantic feature selection.
- A single network forward pass obtains both keypoints and descriptors in the proposed CNN-based model.
- The Google-Landmarks evaluation uses more than 1M database images, 13K unique landmarks, and 100K query images.
- DELF outperforms existing global and local descriptors by substantial margins in the large-scale setting.