PaperlayerView exact revision on arXiv

Source-linked AI summary

Detect-to-Retrieve: Efficient Regional Aggregation for Image Search

Marvin Teichmann, Andre Araujo, Menglong Zhu, Jack Sim

arXiv:1812.01584v2cs.CV

TL;DR

Cluttered-scene retrieval can miss small relevant objects during initial filtering, while regional search can add irrelevant regions and memory and latency costs. This paper builds a manually boxed landmark dataset and uses detector-guided regional aggregation; its full system reports state-of-the-art results on Revisited Oxford and Paris.

  • Problem

    Relevant small objects in cluttered images may fail to reach re-ranking because their initial representations do not score highly during filtering, while existing regional methods incur memory and latency costs for small accuracy gains.

    Source: §1 · p. 2

  • Method

    The paper uses a trained landmark detector to select regions and a regional aggregated match-kernel representation, supported by a manually boxed landmark dataset.

    Source: §1 · p. 2

  • Results

    The full system improves mean average precision by 9.3% absolute on Revisited Oxford-Hard and 1.9% on Revisited Paris-Hard over previously published results.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The regional aggregated representation outperforms systems that index image regions independently while requiring only one descriptor per image.

    Source: §1 · p. 2

    The described kernel applies regional aggregation asymmetrically to database images, a setup especially relevant when queries are well-localized regions of interest.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

Retrieving object instances among cluttered scenes efficiently requires compact yet comprehensive regional image representations. Intuitively, object semantics can help build the index that focuses on the most relevant regions. However, due to the lack of bounding-box datasets for objects of interest among retrieval benchmarks, most recent work on regional representations has focused on either uniform or class-agnostic region selection. In this paper, we first fill the void by providing a new dataset of landmark bounding boxes, based on the Google Landmarks dataset, that includes $86k$ images with manually curated boxes from $15k$ unique landmarks. Then, we demonstrate how a trained landmark detector, using our new dataset, can be leveraged to index image regions and improve retrieval accuracy while being much more efficient than existing regional methods. In addition, we introduce a novel regional aggregated selective match kernel (R-ASMK) to effectively combine information from detected regions into an improved holistic image representation. R-ASMK boosts image retrieval accuracy substantially with no dimensionality increase, while even outperforming systems that index image regions independently. Our complete image retrieval system improves upon the previous state-of-the-art by significant margins on the Revisited Oxford and Paris datasets. Code and data available at the project webpage: https://github.com/tensorflow/models/tree/master/research/delf.

1. Introduction

Cluttered scenes can cause relevant objects to be missed during filtering, while existing regional methods incur costs from irrelevant regions. The paper introduces landmark boxes and detector-guided regional aggregation, reporting substantial retrieval gains over prior results.

1. Introduction

  • Image retrieval first filters database images by query similarity, then inspects and re-ranks a small set of top-ranked images.CNN-based global representations can support filtering, while CNN local representations can support re-ranking through spatial matching and geometric verification.Source: §1 · pp. 1–2
  • Regional aggregation describes local features per region with VLAD, then sum-pools and normalizes by visual word before combining the representation with selective match kernels.The method is called regional aggregated selective match kernels (R-ASMK) and does not increase dimensionality relative to original ASMK [38].Source: Figure 1 · p. 1
  • Relevant objects in cluttered database images may fail to reach re-ranking because their initial representations score too low during filtering.Existing regional selection can produce many irrelevant regions, bringing high memory and latency costs for only small accuracy gains.Source: §1 · p. 2
  • The full system improves mean average precision by 9.3% absolute on Revisited Oxford-Hard and 1.9% on Revisited Paris-Hard over previously published results.The contribution reports these gains as absolute mAP improvements.Source: §1 · p. 2
  • The proposed framework trains landmark detectors from 86k manually boxed images spanning 15k landmarks, then uses detected regions for more efficient search and aggregation.R-ASMK combines detected regions into a discriminative image representation that outperforms independently indexed regional systems while storing one descriptor per image.Source: §1 · p. 2

2. Related Work

Prior retrieval work uses regional selection either to search subimages independently or to improve whole-image representations. The paper positions its detector-based approach as using fewer regions and a different aggregation formulation.

2. Related Work

  • The landmark-box dataset uses human annotations for 86k images from 15k landmarks, compared with SIFT-matching-derived boxes for 49k images from 586 landmarks.OpenImages has generic object boxes across 9M images, but landmark-like classes make up a small fraction [19].Source: §2 · p. 2
  • Regional methods divide into regional search, which encodes selected regions independently, and regional aggregation, which uses regions to improve image representations.These approaches serve different purposes: retrieving subimages versus improving whole-image representations.Source: §2 · p. 2
  • Earlier regional search used grids, thousands of Selective Search regions, or MSER proposals, with prior evaluations reporting substantial memory and complexity costs.Other grid-based systems used 14 or 30 regions per image.Source: §2 · p. 2
  • Prior regional aggregation used roughly 20 grid regions or hundreds of proposals per image and typically pooled regions, whereas this work uses fewer than five and formulates aggregation as a match kernel.Gordo et al. used semi-automatic bounding-box annotations to train a region proposal network.Source: §2 · p. 2

3. Google Landmark Boxes Dataset

Google Landmark Boxes adds manually curated landmark boxes to the Google Landmarks data. Annotators box the most prominent landmark, while the dataset construction balances class coverage and excludes ambiguous cases.

3. Google Landmark Boxes Dataset

  • The dataset draws on Google Landmarks, whose training set has 1.2M images across 15k landmarks, including both man-made and natural landmarks.The annotated examples include varied landmarks, such as man-made and natural objects.Source: §3 · p. 2, Figure 2 · p. 3
  • Annotators drew at most one box around the main object or object set showcased in each image, following its single landmark label.The intended target was the most prominent landmark in the image.Source: §3 · pp. 2–3
  • The final dataset contains 54k training and 32k validation images after removing about 8% of initially selected images with no identifiable prominent landmark.The removed cases included occluded landmarks or images showing their surroundings.Source: §3 · p. 3
  • To balance landmark coverage, the authors randomly selected four training and four validation images per landmark, yielding 58k and 36k initially boxed images, respectively.For about 40% of landmarks, all available images were annotated.Source: §3 · p. 3

4. Regional Search and Aggregation

The section reviews aggregated local-feature representations, then develops detector-based regional search and aggregation methods that use landmark-predicted regions with deep local features and ASMK.

4.1. Background

  • The reviewed framework represents an image with local descriptors quantized to codebook visual words and aggregates descriptors assigned to each word.The framework encompasses Bag-of-Words, VLAD, and ASMK.Source: §4.1 · p. 3

4.2. Regional Search

  • Regional search stores descriptors for detector-predicted subimages independently and scores each database image by max- or average-pooling regional similarities.The original image is also treated as a valid region.Source: §4.2 · p. 4

4.3. Regional Aggregated Match Kernels

  • Regional aggregation uses detected boxes to improve database-image representations without separately storing each region’s descriptor.The method extends the aggregated match kernel framework.Source: §4.3 · p. 4
  • R-VLAD implements average pooling of regional VLAD similarities with aggregated regional descriptors, avoiding storage of each region’s representation.Naive-R-ASMK similarly aggregates ASMK residuals without selectivity, then applies selectivity when matching.Source: §4.3 · p. 4
  • R-ASMK addresses the downweighting of visual-word residuals when many regions lack features for those words.The authors identify this weakness in R-VLAD and Naive-R-ASMK and propose R-ASMK to fix it, drawing on ASMK’s changes to VLAD.Source: §4.3 · pp. 4–5

5. Experiments

The experiments validate landmark detection and evaluate how detected regions affect image retrieval through regional search and aggregation. The best system improves on prior state of the art across Revisited Oxford and Paris protocols, with similar gains at large scale.

5.1. Landmark Detection

  • The Faster R-CNN and SSD detectors reach 84.37% and 85.61% AP, respectively, on validation; their runtimes are 89 ms and 27 ms per image.Detection AP is measured at 50% Intersection-over-Union.Source: §5.1 · p. 5

5.2. Image Retrieval

  • ROxf and RPar each have 70 queries; the experiments report Medium and Hard protocols using mAP and mP@10, plus large-scale tests with R1M.The R1M distractor set contains 1,001,001 images.Source: §5.2 · p. 5

5.2.1 Regional Search

  • 40.65% mAP with 4.05 regions/image is 2.31% above single-image ASMK⋆; on RPar-Hard, regional search gains 2.5% mAP at 3.9 regions/image.Across ASMK variants, D2R regions raise mAP by 3.23% at an average of 4.05 regions per image.Source: §5.2.1 · pp. 6–7

5.2.3 Comparison Against State-of-the-Art

  • 8.2% mAP on ROxf-Medium, 1.8% on RPar-Medium, 9.3% on ROxf-Hard, and 1.9% on RPar-Hard improve prior state of the art, with similar large-scale gains.The table compares methods under Medium and Hard protocols, across the datasets and their large-scale extensions.Source: Table 2 · p. 7, §5.2.3 · p. 8

5.2.2 Regional Aggregated Match Kernels

  • D2R-R-ASMK and D2R-R-ASMK⋆ improve their baselines by 3.85% and 3.65% absolute mAP, respectively; RMACB and Selective Search gain at most 1.5%.Regional aggregation produces an image descriptor with the same dimensionality as the single-image baseline.Source: Figure 4 · p. 6, §5.2.2 · p. 7
  • D2R-R-ASMK⋆ beats the best regional-search variant by 1.3% mAP on ROxf-Hard and 0.1% on RPar-Hard, with 4.1× and 3.9× storage savings.The table reports retrieval mAP and relative database size for both datasets.Source: Table 1 · p. 6, §5.2.2 · p. 7
  • At a detection threshold of 0.01, R-ASMK⋆ reaches 41.6% mAP with 10.2 regions/image, while R-VLAD and Naive-R-ASMK degrade as more regions are used.R-VLAD falls to 16.46% mAP and Naive-R-ASMK to 9.2% under that threshold.Source: §5.2.2 · p. 7

5.2.4 Discussion

  • Separate regional indexing can retrieve small objects in clutter, but also risks matching localized regions from a different landmark; aggregation can boost object features across several boxes.The discussion illustrates cases where detected regions do not precisely cover the query object, while features inside and outside the main box contribute to the representation.Source: §5.2.4 · p. 8

6. Conclusions

The paper combines landmark detection and regional aggregation to improve image retrieval accuracy and efficiency, reporting state-of-the-art performance on two retrieval datasets.

6. Conclusions

  • The complete system achieves state-of-the-art performance by a large margin on the ROxford and RParis image retrieval datasets.Source: §6 · p. 8
  • The proposed regional search uses trained landmark detectors to improve retrieval performance and efficiency over existing regional methods.Source: §6 · p. 8
  • The paper introduces a landmark bounding-box dataset and shows that landmark detectors can be trained and used to extract regional representations.Source: §6 · p. 8
  • The proposed regional aggregated match kernel further boosts retrieval accuracy.Source: §6 · p. 8

Appendix A. Discussion: Detection Helps Finding Relevant Features

Predicted landmark boxes identify features that are more likely to match, and the paper uses this signal to re-weight local features for retrieval.

Appendix A. Discussion: Detection Helps Finding Relevant Features

  • Relevant local features are measured by matching same-landmark image pairs and identifying inliers to a high-confidence geometric transformation.Source: Appendix A · p. 9
  • The analysis uses DELF features and a Faster-RCNN landmark detector, with 10k image pairs and nearest-neighbor matching followed by affine RANSAC verification.Source: Appendix A · p. 9
  • 3 to 4× higher relevance probability is observed for a feature inside a predicted box than outside it at the same attention score.Source: Appendix A · p. 9
  • R-ASMK⋆ re-weights local features to favor those inside detected regions, and experiments on ROxford and RParis show that this improves retrieval performance.Source: Appendix A · p. 9

Appendix B. Detection Experiments

The appendix reports detector learning curves and examples, alongside a comparison of feature relevance inside and outside predicted boxes.

Appendix B. Detection Experiments

  • The SSD model slightly outperforms the Faster-RCNN baseline despite its smaller or weaker feature extractor.The authors conjecture that SSD's multi-scale feature map captures landmarks at different scales better than Faster-RCNN's single feature map.Source: Appendix B · p. 9
  • The relevance-probability plot compares DELF features inside predicted boxes with those outside them as attention scores vary.Source: Figure 6 · p. 9
  • Both detectors converge to around 85% mAP within 600k training steps, while MobileNetV2 SSD trains faster than ResNet-50 Faster-RCNN.Figure 7 compares mean average precision at IoU=0.5 across training steps; the models use batch sizes of 32 and 1, respectively.Source: Figure 7 · p. 9, Appendix B · p. 9
  • Detection examples span landmark scales, occlusion, and lighting, while failure cases include ambiguous boundaries causing double detections and hard-to-distinguish objects being missed.Only detections above confidence 0.2 are shown.Source: Appendix B · p. 9

Appendix C. Region Selection Comparison

The comparison shows that trained landmark detectors focus on prominent landmark regions, unlike content-independent RMAC grids, while selective-search boxes may not correspond to landmarks.

Appendix C. Region Selection Comparison

  • The comparison uses roughly similar region counts per image: about 4 for D2R, 9 for two-level RMAC, and 6 for selective search.Source: Appendix C · p. 10
  • The trained detector focuses on prominent landmark regions, whereas RMAC's fixed grid often does not overlap well with landmarks and selective search may select non-landmark objects.Source: Appendix C · p. 10
Loading 1812.01584v2…