Source-linked AI summary
Self-supervising Fine-grained Region Similarities for Large-scale Image Localization
Yixiao Ge, Haibo Wang, Feng Zhu, Rui Zhao, Hongsheng Li
TL;DR
Retrieval-based image localization must learn discriminative representations from noisy GPS labels and difficult positive pairs. The paper self-supervises fine-grained image-to-region similarities and iteratively refines them across training generations. It reports state-of-the-art localization results and strong generalization across image-retrieval datasets.
Problem
Weak GPS supervision and image-level similarities make it difficult to learn discriminative features for noisy, partially overlapping positive pairs in image-based localization.
Method
The method self-supervises image-to-image and image-to-region similarities, using refined soft labels to train the retrieval network across generations.
Results
The method outperforms state-of-the-art methods on standard localization benchmarks and generalizes strongly to image-retrieval datasets.
Takeaways & Limitations
Self-enhanced image-to-region similarities provide training supervision for difficult positives and improve discriminative localization features without additional parameters or manual annotations.
Takeaways & Limitations
Image-level soft supervision still gives the same target score to all spatial regions, limiting learning of discriminative local features for non-overlapping regions.
Abstract
from arXiv · showhide
The task of large-scale retrieval-based image localization is to estimate the geographical location of a query image by recognizing its nearest reference images from a city-scale dataset. However, the general public benchmarks only provide noisy GPS labels associated with the training images, which act as weak supervisions for learning image-to-image similarities. Such label noise prevents deep neural networks from learning discriminative features for accurate localization. To tackle this challenge, we propose to self-supervise image-to-region similarities in order to fully explore the potential of difficult positive images alongside their sub-regions. The estimated image-to-region similarities can serve as extra training supervision for improving the network in generations, which could in turn gradually refine the fine-grained similarities to achieve optimal performance. Our proposed self-enhanced image-to-region similarity labels effectively deal with the training bottleneck in the state-of-the-art pipelines without any additional parameters or manual annotations in both training and inference. Our method outperforms state-of-the-arts on the standard localization benchmarks by noticeable margins and shows excellent generalization capability on multiple image retrieval datasets.
1 Introduction
Image-based localization must learn discriminative image representations despite weak, noisy GPS supervision and difficult positive pairs. The paper proposes self-enhanced image-to-region similarities as fine-grained training supervision, iteratively improving retrieval features.
- Image-based localization estimates a query’s location by retrieving reference images captured at the same places from a geo-tagged database.
- Weak GPS labels make geographically close images unreliable positives because they may depict different scenes or viewpoints.
- Training only with easiest top-1 positives improves confidence but leaves features insufficiently robust to challenging conditions and difficult positives.
- Lower-ranking positives can be false positives without overlapping regions, causing feature-learning deterioration and possible network collapse.
- The method estimates self-enhanced soft image-to-image similarities and trains the retrieval network across generations to improve difficult-positive supervision.
- Image-level supervision can force non-overlapping regions toward identical scores, so query-to-region similarities provide finer guidance for local-feature learning.
- The proposed system self-enhances top-ranking similarities, adds region-level supervision, and reports improved localization and retrieval generalization without additional parameters or manual annotations.
2 Related Work
Related work places image-based localization among retrieval, 2D-3D registration, and position-classification approaches. Retrieval-based methods build on traditional place recognition and deep feature aggregation, while this paper targets their weakly supervised training bottleneck.
- Image-based localization methods are grouped into image retrieval, 2D-3D registration, and per-position classification approaches.
- This work focuses on the training bottleneck of weakly supervised image-retrieval-based localization.
- Retrieval-based localization is closely related to place recognition, which evolved from SIFT, BoW, VLAD, and Fisher Vector descriptors.
- NetVLAD advanced localization by aggregating dense CNN features through a learnable VLAD layer with learnable semantic centers.
3 Method
The method addresses noisy GPS-derived positives in retrieval-based image localization by progressively self-supervising query-to-gallery and query-to-region similarities. It uses difficult candidate positives and spatial sub-regions to refine training signals across generations and improve local feature discrimination.
- Motivation: Retrieval-based IBL is weakly supervised because geographically nearby images may face different directions and depict non-overlapping scenes.
- Motivation: Training only with easiest positive images limits robustness to viewpoint, camera-pose, focal-length, and other challenging positive variations.
- Self-supervising Query-gallery Similarities: The method self-supervises relative similarities among top-ranking gallery candidates, using soft labels to exploit difficult positives while mitigating inevitable label noise.The first-generation network estimates similarities that supervise the next generation through a soft cross-entropy objective combined with the original hard loss.
- Multi-generation Training: The network is trained in generations: each generation estimates refined similarities that become additional supervision for the next, with performance generally increasing and saturating around the fourth generation.Reported temperatures for successive generations are 0.07, 0.06, and 0.05; hard and soft losses are jointly optimized.
- Self-supervising Fine-grained Image-to-region Similarities: Image-to-region similarities decompose each plausible positive into four halves and four quarters, providing fine-grained supervision for discriminative local features.The region similarities are progressively refined through multi-generation training rather than assigning identical image-level targets to every spatial region.
- Self-supervising Fine-grained Image-to-region Similarities: Image-to-region supervision is preferred over region-to-region supervision because it balances robustness and granularity while avoiding superficially easy matches such as corresponding sky regions.
4 Experiments
Experiments evaluate localization performance, ablations, qualitative retrieval behavior, and cross-dataset generalization using models trained on Pitts30k-train. The method improves benchmark recall, benefits from self-enhanced fine-grained similarities, and generalizes across several retrieval datasets.
- Comparison with State-of-the-arts: 90.7% rank-1 recall on Pitts250k-test exceeds SARE’s 89.0% by 1.7%.
- Comparison with State-of-the-arts: 85.4% rank-1 recall on Tokyo 24/7 improves performance against SARE by up to 5.7%.The model is trained only on Pitts30k-train and evaluated directly on Tokyo 24/7.
- Ablation Studies: Adding noisy difficult positives alone lowers Tokyo 24/7 rank-1 recall from 80.6% to 76.2%, while self-enhanced soft supervision restores it to 80.6%.The soft objective improves “Baseline w/ top-k positives” from 76.1% to 80.6%.
- Ablation Studies: Removing quarter regions reduces Tokyo 24/7 rank-1 recall from 85.4% to 84.4%, and removing all sub-regions reduces it to 80.6%.The framework uses four half regions and four quarter regions to model partial overlap caused by camera-pose variation.
- Ablation Studies: K-reciprocal nearest-neighbor refinement further raises Tokyo 24/7 rank-1 recall by 1.6% beyond the variant without ranking refinement.The comparison supports more accurate identification of difficult positive images.
- Qualitative Evaluation: Qualitative heatmaps show greater attention to shop signs and less attention to misleading trees than SARE in illustrated Tokyo 24/7 examples.In a third example, both methods fail, but the proposed method retrieves a top-1 image with similar wall patterns.
- Generalization on Image Retrieval Datasets: The learned embeddings outperform other competitors on most standard image-retrieval datasets and test settings without fine-tuning.Holidays remains difficult for all compared methods because its natural scenery differs from the street-view training data.
5 Conclusion
The paper addresses image-based localization under weak and noisy supervision by self-supervising image-to-region similarities. It reports noticeable improvements over state-of-the-art methods on standard localization benchmarks.
- The method targets image-based localization by recognizing nearest reference images to estimate a query image’s geographical location.
- Self-supervised image-to-region similarities are used to address weak and noisy supervision.
- The method outperforms state-of-the-art methods on standard localization benchmarks by noticeable margins.
A Related Work (Cont.)
IBL research includes weakly supervised methods using geographical tags and fully supervised methods using 6DoF camera poses. Fully supervised datasets are difficult to collect and scale because they require additional capture and processing costs.
- Fully supervised IBL research uses datasets containing 6DoF camera pose information.
- Such datasets are difficult to collect and scale because they require dense image capture and post-processing for mapping and SfM.
- Weakly supervised methods, including the authors’ work, form another stream of IBL research.
B Progressively Refined Supervisions
On Tokyo 24/7, retrieval accuracy increases across training generations and saturates after the fourth generation. This pattern indicates progressive refinement of the self-predicted soft supervisions.
- Retrieval accuracies on Tokyo 24/7 gradually increase as the network generation proceeds.The reported metrics are Recall@1/5/10 (%).
- Performance saturates after the 4th generation on Tokyo 24/7.
- The generational trend indicates that self-predicted soft supervisions are progressively refined through training.