PaperlayerView exact revision on arXiv

Source-linked AI summary

Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval

Tobias Weyand, Andre Araujo, Bingyi Cao, Jack Sim

arXiv:2004.01804v2cs.CV

TL;DR

Existing retrieval and recognition benchmarks do not adequately capture the scale and real-world difficulty of modern applications. The paper introduces GLDv2, a large worldwide landmark dataset with challenging class imbalance, out-of-domain queries, and intra-class variation, and shows that it supports competitive transfer learning and public benchmarking.

  • Problem

    Larger, more challenging datasets are needed to evaluate retrieval and instance recognition under practical conditions, including out-of-domain queries and scale beyond small city-based benchmarks.

  • Method

    The paper constructs GLDv2 from Wikimedia Commons with millions of landmark images, fine-grained instance labels, task-specific splits, and ground truth for recognition and retrieval.

  • Results

    Learned embeddings trained on GLDv2 achieve competitive retrieval performance on independent datasets, and the dataset supports public challenges for both recognition and retrieval.

  • Takeaways & Limitations

    GLDv2 provides a large-scale benchmark intended to advance methods for instance recognition and image retrieval under extreme class imbalance and out-of-domain test conditions.

  • Takeaways & Limitations

    GLDv2 does not target unseen-class generalization or image-level retrieval ground truth, using overlapping training and index classes and class-level relevance instead.

Abstract

from arXiv · show

While image retrieval and instance recognition techniques are progressing rapidly, there is a need for challenging datasets to accurately measure their performance -- while posing novel challenges that are relevant for practical applications. We introduce the Google Landmarks Dataset v2 (GLDv2), a new benchmark for large-scale, fine-grained instance recognition and image retrieval in the domain of human-made and natural landmarks. GLDv2 is the largest such dataset to date by a large margin, including over 5M images and 200k distinct instance labels. Its test set consists of 118k images with ground truth annotations for both the retrieval and recognition tasks. The ground truth construction involved over 800 hours of human annotator work. Our new dataset has several challenging properties inspired by real world applications that previous datasets did not consider: An extremely long-tailed class distribution, a large fraction of out-of-domain test photos and large intra-class variability. The dataset is sourced from Wikimedia Commons, the world's largest crowdsourced collection of landmark photos. We provide baseline results for both recognition and retrieval tasks based on state-of-the-art methods as well as competitive results from a public challenge. We further demonstrate the suitability of the dataset for transfer learning by showing that image embeddings trained on it achieve competitive retrieval performance on independent datasets. The dataset images, ground-truth and metric scoring code are available at https://github.com/cvdfoundation/google-landmark.

1. Introduction

GLDv2 addresses the need for larger, more challenging benchmarks for image retrieval and instance recognition under realistic conditions. It introduces a worldwide landmark dataset with extreme scale, difficult test distributions, and demonstrated transfer-learning utility.

  • Motivation: Existing retrieval benchmarks are small and geographically narrow, while many recognition datasets omit out-of-domain queries and false-positive evaluation.Oxford5k and Paris6k have only 55 query images each, and both draw from a single city.
  • Dataset: GLDv2 contains over 5M images and over 200k landmark classes sourced from Wikimedia Commons.It includes human-made and natural landmarks contributed by local experts.
  • Dataset: The benchmark includes 4M labeled training images, 762k retrieval index images, and 118k test queries with ground truth for both tasks.Only 1% of test images are in-domain landmarks, while 99% are out-of-domain.
  • Challenges: Its design combines extreme class imbalance, many out-of-domain test images, and high intra-class variability to simulate real-world conditions.The same class can include indoor and outdoor views or indirectly relevant images such as museum paintings.
  • Uses: GLDv2 recognition labels support descriptor training and pre-training, with learned embeddings achieving competitive retrieval performance on independent datasets.The dataset was also used in public challenges for instance recognition and image retrieval.

2. Related Work

Prior landmark datasets range from small, city-specific retrieval benchmarks to larger multi-location collections. GLDv2 extends this line of work with substantially greater scale and worldwide coverage for landmark recognition and retrieval.

  • Recognition scope: Instance-level recognition identifies a single occurrence of an entity, making it more fine-grained than basic categorization or broader fine-grained recognition.Examples include recognizing a specific cat, shoe model, or cathedral.
  • Existing datasets: Oxford, Paris, and Rome 16k are largely city-specific landmark datasets, with Oxford and Paris originally containing tens of queries and thousands of index images.Oxford and Paris have remained widely used retrieval benchmarks for more than a decade.
  • Existing datasets: More recent datasets broaden location coverage, but vary in scale, annotation scope, and test-set availability.European Cities covers multiple cities, while some other collections provide only training data or focus on famous landmarks.
  • GLDv2: GLDv2 surpasses existing datasets in the number of images and landmarks and uses worldwide, crowdsourced and semi-automatic data collection.The comparison table identifies GLDv2 as stable, meaning it can be retained indefinitely.
  • Related domains: Related instance-level datasets also cover domains such as logos and cars, while other recognition benchmarks target categorization or fine-grained categories.GLDv2 focuses specifically on individual human-made and natural landmarks.

3. Dataset Overview

GLDv2 is designed as a world-spanning benchmark that combines large scale with long-tailed classes, high intra-class variation, and predominantly out-of-domain queries. It provides defined recognition and retrieval metrics alongside freely licensed, geographically and categorically diverse data.

  • Scale: GLDv2 contains over 5M images and over 200k distinct instance labels across query, training, and index subsets.The dataset includes 118k query images, 4.1M training images, and 762k index images.
  • Challenges: 57% of classes have at most 10 images and 38% have at most 5 images, producing an extremely long-tailed distribution.The classes range from world-famous landmarks to lesser-known local ones.
  • Challenges: Only 1.1% of query images depict landmarks, while 98.9% are out-of-domain images for which no result is expected.This distribution emphasizes robustness in practical instance recognition systems.
  • Metrics: Recognition uses micro Average Precision (µAP), while retrieval uses mean Average Precision@100 (mAP@100).µAP penalizes landmark predictions on out-of-domain queries, and mAP@100 considers only the top 100 ranked images.
  • Data distribution: The dataset covers landmarks from 246 of 249 ISO 3166-1 countries, but image counts are shaped by local Wikimedia Commons activity rather than population representation.Its category distribution is led by churches, followed by parks and museums; among categories exceeding 25k images, roughly 28% are natural and 72% human-made.
  • Licensing: All images use Creative Commons or Public Domain licenses, enabling indefinite retention and reproduction in publications with attribution where required.This licensing policy prevents the dataset from shrinking as images are deleted.

4. Dataset Construction

GLDv2 combines Wikimedia Commons mining with license-based partitioning, targeted query collection, and human-assisted ground-truth correction. Its re-annotation workflow uses model proposals to make fine-grained landmark labeling tractable at scale.

  • Data sources: Wikimedia Commons supplies landmark photos contributed by photographers, libraries, archives, museums, and other organizations under reusable licenses.The repository provides broad coverage of human-made and natural landmarks.
  • Training and index sets: The mining pipeline uses Google Knowledge Graph entities, linked Wikipedia articles, and corresponding Wikimedia Commons categories to collect landmark images.Each mined image was restricted to association with a single Wikimedia category to avoid ambiguities.
  • Query set: Positive queries were collected by asking crowdsourcing operators to photograph selected landmarks, while negative queries were mined from non-landmark Knowledge Graph terms.Negative images with near-duplicates in the training or index sets were removed.
  • Dataset partitioning: Training and index images were separated by license, yielding a large but incomplete overlap between their label spaces.CC0 and Public Domain photos formed the index, while eligible Creative Commons By photos formed the training set.
  • Test set re-annotation: Human re-annotation addressed missing or incorrect ground truth by presenting raters with model-proposed labels from top challenge submissions.The interface reframed landmark identification as a yes-or-no decision about whether a query matches a displayed landmark class.
  • Test set re-annotation: The re-annotation process sent approximately 10k recognition queries and 90k retrieval queries to three human raters per image.Grouping queries by landmark class and selecting high-confidence candidates improved annotation efficiency.

5. Experiments

The experiments establish baseline performance for GLDv2 recognition and retrieval, compare training datasets, and evaluate challenge systems and re-annotation effects. GLDv2-trained models perform strongly for retrieval, while recognition benefits from local and combined feature methods; substantial room for improvement remains.

  • 5.1. Training Set Pre-Processing: 1.6M training images and 81k labels remain after cleaning GLDv2-train, while GLDv2-train-no-tail retains roughly the same image count as GLDv1-train but twice as many classes.GLDv2-train-clean contains 38% of the training images and 40% of the labels.
  • 5.2. Comparing Training Datasets: Training on GLDv2-train-clean improves ROxf and RPar retrieval mAP by up to 10% over GLDv1-train.The comparison uses ResNet-101 models with GeM pooling and ArcFace loss on independent landmark retrieval datasets.
  • 5.3. Benchmarking: GLDv2 training performs better for retrieval, whereas GLDv1 training performs better for recognition relative to training on Landmarks-full/clean.This cross-task difference is reported across the baseline comparisons.
  • 5.3. Benchmarking: For GLDv2 recognition, combining local and global features or using local feature aggregation outperforms systems based only on global descriptors.For retrieval, the GLDv2-trained global descriptor approach outperforms the other compared methods.
  • 5.4. Challenge Results: Successful challenge systems begin with global feature similarity search and additionally use local re-ranking, ensembling, detectors or classifiers, and query or database expansion.The challenge results use complex combinations of global and local features and expansion techniques.
  • 5.4. Challenge Results: Even complex challenge methods leave substantial room for improvement in both recognition and retrieval.The authors conclude that landmark recognition and retrieval are far from solved.
  • 5.5. Effect of Re-annotation: Re-annotation clearly improves recognition µAP, but retrieval mAP remains almost unchanged because class-level relevance raises measured precision while lowering measured recall.Precision@100 consistently increases after re-annotation for retrieval.

6. Conclusion

The paper presents GLDv2 as a large-scale benchmark for image retrieval and instance recognition. Its extreme class imbalance and out-of-domain test images introduce real-world challenges intended to support further progress.

  • GLDv2 is presented as a large-scale benchmark for image retrieval and instance recognition.
  • The benchmark includes real-world challenges absent from previous datasets, including extreme class imbalance and out-of-domain test images.
  • The authors hope GLDv2 will advance the state of the art and foster research addressing these challenges.

Appendix A. Comparison of Retrieval Subset with Oxford and Paris Datasets

GLDv2-retrieval and ROxford/RParis capture complementary aspects of instance-level retrieval. GLDv2 emphasizes worldwide scale, practical query diversity, and a single positive/negative evaluation protocol, while the older datasets emphasize cropped-region matching and offer separate protocols.

  • Scale: 101k landmarks worldwide and 1.1k queries make GLDv2-retrieval broader than ROxford/RParis, which cover 11 landmarks each and 70 queries.GLDv2-retrieval has 762k landmark index images; ROxford/RParis have 5k/6k landmark index images plus an optional 1M-image distractor set.
  • Scale: GLDv2-retrieval is expected to be more representative of practical applications, while ROxford/RParis remain more accessible under limited computational resources.The smaller datasets have shorter evaluation turnaround times, and recent work has used small GLDv2 subsets to improve feasibility.
  • Evaluation protocol: GLDv2 uses validation and testing splits to reduce overfitting, whereas ROxford/RParis lack a separate validation split.The older datasets are often tuned on small-scale versions that overlap conceptually with testing, making overfitting harder to detect.
  • Challenges: GLDv2 queries are uncropped and span easy-to-hard matching cases, whereas ROxford/RParis use cropped regions that stress scale- and perspective-invariant matching.GLDv2 does not explicitly target small-region retrieval but includes both full-landmark and architectural-detail queries.
  • Challenges: GLDv2 has one positive/negative protocol and includes easy queries, while ROxford/RParis use Easy, Medium, and Hard protocols based on graded relevance.Local-feature aggregation methods that excel on ROxford can perform worse than simple embeddings on GLDv2.
  • Application: GLDv2 treats any image associated with the same landmark as relevant, even when its view does not overlap the query; ROxford/RParis require the same instance and view.The datasets therefore reflect different application definitions of retrieval relevance.
  • Conclusion: The authors conclude that GLDv2-retrieval and ROxford/RParis should all be included for comprehensive assessment because they capture complementary retrieval characteristics.Their recommendation follows from the datasets’ differing scale, protocols, query construction, and relevance definitions.

Appendix B. Preventing Unintended Methods

The competition removed image metadata and addressed license-related information leakage to prevent participants from using unintended methods.

  • Preventing unintended methods: Metadata stripping was insufficient because CC-BY licenses could reveal author and original-image information.The competition therefore required additional design choices to prevent cheating through attached or externally found metadata.

Appendix C. Sample Images from the Dataset

The paper provides a qualitative sample of dataset images and directs readers to a web interface for exploring the collection.

  • Sample images: A sample of dataset images illustrates the collection qualitatively, while the dataset website offers an interface for further exploration.The passage does not specify particular visual patterns in the sample.

C.1 Intra-Class Variation in Training Set

GLDv2’s training set spans substantial visual variation within landmark classes, including viewpoint, illumination, weather, season, image quality, and historical appearance changes.

  • Intra-class variation: Training images cover widely different viewing angles, lighting and weather conditions, and seasons for each place.This broad coverage is shown across more than 200k training classes.
  • Intra-class variation: Historical archive photos add changes in photo quality and appearance over time to the training variation.The paper identifies these images as useful for making trained models robust to temporal and quality changes.

C.2 Retrieval Ground Truth

The retrieval ground truth is defined at the class level, creating realistic variation between query images and relevant index images. This requires retrieval methods to handle substantial visual and domain differences.

  • Class-level ground truth means relevant index images do not necessarily share content with the query image.
  • The retrieval task includes variation in viewpoint, occlusions, lighting, and weather.
  • Retrieval methods must be invariant to image-domain differences.

C.3 Test Set

The test set combines realistic landmark queries with a much larger collection of out-of-domain images. Its figures illustrate training classes, retrieval examples, and test-set samples.

  • C.3 Test Set: 1.1% of test images show natural or human-made landmarks captured by smartphones through crowdsourcing.
  • C.3 Test Set: 98.9% of test images are out-of-domain examples collected from Wikimedia Commons.
  • C.3 Test Set: Some landmark images lack ground-truth labels because their landmarks are absent from the training or query sets.
  • C.3 Test Set: Figures 7 and 8 show sample classes from the training set.
  • C.3 Test Set: Figures 9–11 show retrieval queries alongside samples of relevant index images, while Figure 12 shows test-set samples.
Loading 2004.01804v2…