Semantic Pose Verification for Outdoor Visual Localization with Self-supervised Contrastive Learning

1. Introduction

• The system localizes standard-field-of-view query images against perspective views generated from panoramic Google Street View images.The panorama database provides a wider field of view to help when standard-field-of-view cameras have non-overlapping views.Source: §1 · p. 1
• Semantic information is used as a side modality at pose verification to check retrievals produced using RGB image features.The motivation is that semantic content is more robust to long-term changes, including weather, seasonal, illumination, and structural changes.Source: §1 · p. 2
• Self-supervised contrastive semantic similarity improves a state-of-the-art CNN-based visual localization method when used to update RGB-only retrieval scores.The approach trains on unlabeled semantic masks and uses a limited labeled localization dataset for fine-tuning and testing.Source: §1 · p. 2
• Pixel-wise semantic similarity also improves RGB-only results, but its gain is marginal compared with self-supervised learning using a large dataset.The paper treats pixel-wise similarity as a naive, easy-to-implement baseline.Source: §1 · p. 2
• The paper applies contrastive learning to semantic masks and uses the resulting similarity estimates to improve an existing image-based localization method.This differs from prior uses of semantic information for feature-point elimination or direct localization from semantic content.Source: §1 · p. 2

2. Related Work

• CNN-based retrieval methods include NetVLAD and SFRS [12], which uses image-to-region similarities to handle limited overlap and GPS position errors.The passage reports that SFRS outperformed previous works on visual localization.Source: §2 · p. 2
• Prior semantic localization includes 3D structure-based methods and 2D image-retrieval methods, with the latter retrieving the most similar image to a query.The paper characterizes the 2D approach as less expensive and argues it performs comparably to 3D-based methods.Source: §2 · p. 2
• Other 2D semantic approaches use semantic layouts, semantic edges, attention to stable regions, or semantic weighting and feature-point filtering.Examples include Singh and Koˇseck´a, Yu et al. [48], Seymour et al., Mousavian et al., and Naseer et al..Source: §2 · pp. 2–3
• Unlike prior approaches that localize from semantic labels or use them to focus on stable regions and eliminate unstable ones, this work checks semantic content to validate image-based retrievals.The related-work discussion distinguishes semantic localization from semantic pose verification.Source: §2 · p. 3
• Self-supervised contrastive learning trains representations by bringing similar sample pairs together and separating dissimilar ones; some methods use negative examples while others use only positive examples.MoCo [14] and SimCLR [5] use positive and negative examples, whereas BYOL and SimSiam [6] use positive examples only.Source: §2 · p. 3
• The paper trains a CNN to learn similarity between semantic masks using augmented versions of an anchor as positives and images from different scenes as negatives.A large dataset of masks with unknown locations supports self-supervised training where labeled localization data are limited; the paper mainly uses SimCLR [5].Source: §2 · p. 3, Figure 2 · p. 3

3.1. Dataset

• The dataset pairs 492 non-overlapping 90° query images from 123 locations with 2,674 gnomonic images from 222 locations, covering older queries and newer panoramas.Queries predate the 2019 panoramas and include seasonal, structural, and illumination changes.Source: §3.1 · p. 3

3.2. Searching perspective query images in a panoramic image database

• Twelve consecutive gnomonic views per panorama increase the chance of overlap, but query and database views can still differ by up to 15° in viewpoint.SFRS retrieves candidates, and semantic similarity is used to verify those retrievals.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3. Computing Semantic Similarity

• Because same-scene semantic masks can differ with camera position and viewpoint, the method learns a trainable semantic feature extractor; pixel-wise matching serves as a baseline.The CNN is trained with self-supervised contrastive learning on semantic masks, then can be fine-tuned using labeled query-database pairs.Source: §3.3 · pp. 4–5

3.4. Updating Retrieval Results with Semantic Similarity

• The method combines normalized RGB-only and semantic similarity scores to update only the top 10 candidates per query, also updating neighboring views from the same panorama.The weight is selected on a validation set, separately for pixel-wise similarity and the trainable feature extractor.Source: §3.4 · p. 5

4. Experimental Results

• Figure 5 illustrates a case where combining RGB and semantic scores moves the correct database match to the top rank after RGB-only retrieval fails.Source: Figure 5 · p. 6
• RGB-only retrieval, pixel-wise semantic scoring, and SimCLR-based semantic scoring are compared, with semantic scores updating RGB-only retrieval results.Fine-tuned models are also evaluated in the ablation study.Source: §4 · p. 6
• 2%: SimCLR-based semantic pose verification improves RGB-only Recall@1, and the learned extractor continues to outperform RGB-only and pixel-wise verification across increasing distance thresholds.Recall@N counts a query as localized when a retrieved image is within the threshold; experiments use 5 meters.Source: §4 · p. 6, Figure 7 · p. 7

4.1. Ablation Study

• None of the three fine-tuning schemes improves on self-supervised training; retraining all layers retains similar performance.The labeled set comprises 227 query masks and their corresponding database masks, versus 3,484 unlabeled training images.Source: §4.1 · pp. 6–7, Table 1 · p. 8
• Semantic similarity can move correct candidates higher when query and database images share semantic content; distinctive objects can help, and improvement can occur despite mask-labeling errors.Source: Figure 6 · p. 7
• SimSiam performs worse than SimCLR in the evaluated self-supervised training comparison.Source: §4.1 · p. 8
• Recall@1 is highest at a minimum crop ratio of 0.6, with performance gradually dropping when the ratio increases or decreases.The authors relate this pattern to the trade-off between views being too similar or too different.Source: §4.1 · p. 8, Table 2 · p. 8
• Pose verification works equally well for semantic-score weights from 0.25 to 0.40, so success is not specific to W = 0.25.Source: §4.1 · p. 8, Table 3 · p. 8

• Semantic similarity at pose verification contributed to the performance of a state-of-the-art visual-localization method [12].The setting localizes perspective query images in a geo-tagged database of panoramic images.Source: §5 · p. 8
• Self-supervised contrastive-learning CNN pose verification performs better than pixel-wise similarity between masks.Source: §5 · p. 8
• Computing semantic similarity directly from panoramic semantic masks is proposed as future work.The paper notes that other works search query images within panoramic images [16].Source: §5 · p. 8