Large-Scale Image Retrieval with Attentive Deep Local Features

• Global CNN descriptors struggle with patch-level partial matches under clutter, occlusion, viewpoint, and illumination variation, while proposed CNN local features lack semantic selection and have limited accuracy.Source: §1 · p. 1
• Prior retrieval evaluations included only 55 queries in and 500 in, with limited landmark diversity, motivating more statistically meaningful evaluation.Source: §1 · pp. 1–2
• Google-Landmarks contains more than 1M images across almost 13K landmarks plus 100K diverse queries, including no-match images to assess retrieval robustness.Source: §1 · p. 2
• DELF uses image-level weak supervision and coupled attention to select relevant local features and extract descriptors and keypoints in one pass; the authors report significant gains over existing global and local descriptors.Source: §1 · p. 2

Retrieval datasets

• Oxford5k, Paris6k, and Holidays are small datasets with few queries, limiting generalization assessment; Pitts250k is larger but specialized to repetitive visual places.Oxford5k and Paris6k are often augmented with Flickr100k distractors to form Oxford105k and Paris106k.Source: §2 · p. 2

Traditional local-feature retrieval

• Hand-crafted local features with approximate-nearest-neighbor search and geometric re-ranking remain strong when retrieval systems require high precision.The passage also points to [43] as a recent survey of instance retrieval.Source: §2 · p. 2

Feature aggregation

• VLAD and Fisher Vector [19] aggregate local features into global descriptors, with compact indexing as their main advantage.Source: §2 · p. 2

CNN-based retrieval descriptors

• CNN global descriptors use pretrained [4] [34] or learned networks [11], most commonly trained with triplet loss to preserve relevant-versus-irrelevant image ranking.Deep local features have also served as drop-in replacements for hand-crafted features in VLAD or FV [24], while other works proposed aggregation methods [21].Source: §2 · p. 2

CNN local features

• CNN-based local-feature methods have learned keypoint detection [37], canonical orientation [41], patch representations and metrics [42], and end-to-end detection, orientation, and description [40].The paper distinguishes these methods from its work because they are not designed for retrieval or trained to select semantically meaningful features.Source: §2 · p. 2

Visual attention

• Deep-network visual attention has been used in detection, segmentation [14], captioning [38], and visual question answering [39], but not actively to learn features for image retrieval.Source: §2 · p. 2

3. Google-Landmarks Dataset

• Compared with existing image-retrieval datasets, the new dataset is much larger, contains diverse landmarks, and includes 1,060,709 images from 12,894 landmarks plus 111,036 additional query images.Source: §3 · p. 2
• Landmarks in the dataset span 4,872 cities in 187 countries.Source: Figure 3 · p. 3
• Images include foreground/background clutter, occlusion, partially out-of-view objects, and other variations that challenge retrieval.Examples also show small objects and multiple landmarks.Source: §3 · p. 3, Figure 2 · p. 3
• Some personal-photo queries contain no landmark and are designated distractors to test robustness to irrelevant, noisy queries.Source: §3 · p. 3
• Ground truth clusters database images using visual features and GPS, then matches queries by distance to a cluster center, but GPS errors and ambiguous landmarks can make annotations noisy.Manual checks found few incorrect annotations with a 25 km threshold; the authors report that these minor errors do not undermine relative evaluation.Source: §3 · p. 3

4.1. Dense Localized Feature Extraction

• ImageNet-pretrained ResNet50 conv4 x features form a dense descriptor grid across image-pyramid scales, and landmark-image fine-tuning improves local descriptors without object- or patch-level labels.Feature locations use receptive-field centers; at the original image scale, receptive fields are 291 × 291 pixels.Source: §4.1 · pp. 3–4

4.2. Attention-based Keypoint Selection

• Dense features include irrelevant clutter, so DELF selects a subset of keypoints to improve retrieval accuracy and computational efficiency.Source: §4.2 · p. 4

4.2.1 Learning with Weak Supervision

• A landmark classifier learns local-feature relevance scores by weighted-sum pooling, then trains a softmax classifier on the resulting image embedding using image-level labels.The score function uses a two-layer CNN with 1×1 filters and a non-negative softplus output.Source: §4.2.1 · p. 4

4.2.2 Training Attention

• Joint descriptor-and-attention training produced weak models, so DELF first fine-tunes descriptors and then learns attention scores with descriptors fixed.Random rescaling during attention training is used to support effective scores across feature scales.Source: §4.2.2 · p. 4

4.2.3 Characteristics

• DELF selects keypoints after descriptor extraction, learning semantic feature representations and discriminative selection for instance recognition, while implicitly learning pose and viewpoint invariance.Source: §4.2.3 · pp. 4–5

4.3. Dimensionality Reduction

• Selected features are L2-normalized, reduced to 40 dimensions by PCA, and L2-normalized again to balance compactness and discriminativeness.Source: §4.3 · p. 5

4.4. Image Retrieval System

• Retrieval uses attention-ranked local features, approximate nearest-neighbor search, per-image match aggregation, and RANSAC geometric verification, scoring images by inlier count.Descriptors are encoded as 50-bit PQ codes; an inverted index, KD-tree partitioning, and locally optimized product quantization support search.Source: §4.4 · p. 5
• Indexing 1 billion descriptors requires less than 8GB memory, and nearest-neighbor search takes less than 2 seconds on one CPU under the reported setup.Source: §4.4 · p. 5

5.1. Implementation Details

• DELF uses seven pyramid scales from 0.25 to 2.0, is trained for attention on LF and fine-tuned for retrieval on LC, and extracts up to 1,000 40-dimensional features per image.LF contains 140,372 images and LC 35,382 images, each covering 586 landmarks; features use the top 60 nearest neighbors, and receptive-field size varies inversely with scale.Source: §5.1 · p. 5

5.2. Compared Algorithms

• Comparisons include 2,048-dimensional DIR, 512-dimensional siaMAC, 40D CONGAS, and 128-dimensional LIFT features.DIR and siaMAC are global descriptors; CONGAS is a hand-engineered local feature, and LIFT jointly learns keypoint detection, orientation estimation, and description.Source: §5.2 · pp. 5–6

5.3. Evaluation

• Evaluation uses threshold-based precision and unnormalized recall across queries, considering only the top-scoring image per landmark and reporting a full precision–recall curve.This modified precision/recall approach addresses distractor queries by using absolute retrieval scores to estimate image relevance, rather than summarizing performance in a single mAP value.Source: §5.3 · p. 6

5.4. Quantitative Results

• DELF+FT+ATT significantly outperforms the compared methods on Google-Landmarks; DIR with query expansion degrades, while CONGAS remains substantially worse.LIFT results are omitted because feature extraction takes approximately 2 min/image on a GPU, making the large-scale experiment infeasible. The figure caption also identifies fine-tuning and attention as critical to DELF’s improvement.Source: §5.4 · p. 6, Figure 5 · p. 6
• Both fine-tuning and attention substantially improve retrieval, with attention more important; attention also helps even when features are pretrained on ImageNet.DELF and CONGAS each require approximately 8GB of memory; DIR descriptors use 8KB per image and approximately 8GB to index the dataset.Source: §5.4 · p. 6, Figure 5 · p. 6

5.5. Qualitative Results

• DELF’s local matches help distinguish specific instances and small landmarks, while similar patterns such as floor tiling or vegetation can cause false matches.DIR can confuse similar objects or scenes, whereas DELF retrieves more relevant landmarks than CONGAS; examples show DELF matching despite clutter and partial occlusion, where CONGAS fails.Source: §5.5 · pp. 6–7, Figure 6 · p. 7, Figure 7 · p. 7
• Attention-based keypoint scores qualitatively disregard clutter more effectively than pretrained or fine-tuned feature-norm scores.The L2 norm scores of fine-tuned features differ only marginally from those without fine-tuning.Source: §5.5 · p. 7, Figure 9 · p. 8

5.6. Results in Existing Datasets

• DELF improves accuracy nontrivially when fused with DIR on Oxf5k, Par6k, Oxf105k, and Par106k, although DELF alone is not the best-performing method.Fusion uses a weighted mean of normalized scores, with DELF weighted at 0.25; the reported results indicate DELF encodes information complementary to global descriptors.Source: §5.6 · p. 7, Table 1 · p. 8

6. Conclusion

• DELF outperforms existing global and local descriptors by substantial margins in large-scale image retrieval evaluation.The evaluation uses the introduced Google-Landmarks dataset.Source: §6 · p. 8
• Google-Landmarks contains more than 1M database images, 13K unique landmarks, and 100K query images.Source: §6 · p. 8
• DELF is a local feature descriptor learned from image-level labels only and coupled with attention for semantic feature selection.One CNN forward pass obtains both keypoints and descriptors.Source: §6 · p. 8
• On existing datasets, DELF achieves excellent performance when combined with global descriptors.Source: §6 · p. 8