Source-linked AI summary
Map-free Visual Relocalization: Metric Pose Relative to a Single Image
Eduardo Arnold, Jamie Wynn, Sara Vicente, Guillermo Garcia-Hernando, Áron Monszpart, Victor Adrian Prisacariu, Daniyar Turmukhambetov, Eric Brachmann
TL;DR
Conventional relocalization depends on extensive scene mapping, while a single reference image leaves metric scale unresolved. The paper introduces a one-reference-image task and a diverse benchmark, then evaluates feature matching with depth and relative pose regression; existing approaches show some success, but only in a narrow range of situations.
Problem
Conventional visual relocalization requires hundreds of mapping images and scale calibration, while a single reference image leaves feature-matched relative pose without absolute scale.
Source: §1 · pp. 1–3Method
The paper defines metric relocalization from one reference image, constructs a benchmark of 655 places, and evaluates relative pose regression and feature matching with single-image depth.
Source: §1 · pp. 2–3Results
Feature matching with single-image depth achieves the highest relative pose accuracy, while existing approaches address the task with some success only in a narrow range of situations.
Source: §1 · p. 3, §6 · p. 14Takeaways & Limitations
The benchmark provides a test bed for relative pose estimation and related problems such as wide-baseline matching, robust estimation, and single-image depth prediction.
Source: §1 · p. 3, §6 · p. 14RPR methods do not predict confidence for detecting spurious pose estimates, and feature-matching confidence is not always reliable.
Source: §5.2 · p. 13
Abstract
from arXiv · showhide
Can we relocalize in a scene represented by a single reference image? Standard visual relocalization requires hundreds of images and scale calibration to build a scene-specific 3D map. In contrast, we propose Map-free Relocalization, i.e., using only one photo of a scene to enable instant, metric scaled relocalization. Existing datasets are not suitable to benchmark map-free relocalization, due to their focus on large scenes or their limited variability. Thus, we have constructed a new dataset of 655 small places of interest, such as sculptures, murals and fountains, collected worldwide. Each place comes with a reference image to serve as a relocalization anchor, and dozens of query images with known, metric camera poses. The dataset features changing conditions, stark viewpoint changes, high variability across places, and queries with low to no visual overlap with the reference image. We identify two viable families of existing methods to provide baseline results: relative pose regression, and feature matching combined with single-image depth prediction. While these methods show reasonable performance on some favorable scenes in our dataset, map-free relocalization proves to be a challenge that requires new, innovative solutions.
1 Introduction
Visual relocalization can estimate camera pose from an image, but conventional systems require extensive scene scanning and mapping. The paper proposes map-free relocalization from one reference image, introduces a diverse benchmark, and evaluates pose regression and feature matching with depth.
1 Introduction
- A single reference image replaces extensive scene scanning in map-free relocalization, which seeks metric camera poses for new query images.Conventional methods gather images from hundreds of viewpoints and often require calibration or other measurements to recover metric poses.Source: §1 · pp. 1–2
- Feature matching yields relative pose without absolute scale, making scale recovery the central challenge for metric map-free relocalization.The paper identifies heuristics or world knowledge as necessary to resolve that ambiguity when using a single reference and query.Source: §1 · p. 3
- The paper evaluates metric relative pose regression and feature matching combined with single-image depth prediction as map-free baselines.The latter family combines feature matching with single-image depth prediction.Source: §1 · p. 3
- The benchmark covers 655 places with metric-pose queries and varied conditions, including stark viewpoint changes and little or no reference-image overlap.The test set contains queries from 130 places; the dataset is intended to support relative-pose estimation and related subproblems.Source: §1 · p. 3
- Existing methods achieve acceptable precision only in a narrow range of situations, motivating further work on map-free relocalization.The paper also reports additional experiments and ablations on ScanNet and 7Scenes.Source: §1 · p. 3
2 Related Work
Prior relocalization systems use explicit reconstructions, learned scene representations, or relative pose estimates tied to posed database images. Map-free relocalization instead uses one reference image, with estimated depth supplying metric scale when pairwise matching alone cannot.
Scene Representations in Visual Relocalization
- Traditional systems represent scenes with SfM or SLAM reconstructions, while learning-based systems can encode scenes in network weights trained on posed mapping images.The latter includes scene-coordinate regression and absolute pose regression.Source: §2 · pp. 3–4
- Some relative pose regression systems retrieve a posed database image and estimate the query-to-database pose, but still rely on obtaining posed images.Even sparse representations such as four reference images require continuous pose tracking during recording, unlike map-free relocalization's single identity-pose reference.Source: §2 · p. 4
Relative Pose by Matching Features
- Feature matching and robust essential-matrix estimation recover relative rotation and translation only up to scale.The literature improves this pipeline through learned features, matchers, and robust estimators, including work on small-overlap regimes.Source: §2 · p. 4
- With only a reference and query image, map-free relocalization cannot triangulate query pose from multiple references, so estimated depth is used to recover scale.Existing relocalization methods have used multiple scaleless pairwise poses to triangulate a metric pose.Source: §2 · p. 4
Relative Pose Regression and Single-image Depth Prediction
- Relative pose regression predicts pose directly from two images, with methods varying in whether they estimate metric or scaleless translation and in their rotation parameterization.Single-image depth prediction can be relative or metric; generalizing metric depth across domains, resolutions, aspect ratios, and focal lengths is challenging.Source: §2 · pp. 4–5
Single-image Depth Prediction
- Prior work also uses predicted depth in planar rectification, SLAM, and visual odometry, whereas this paper uses depth to scale two-image relative pose.The cited SLAM and odometry approaches use predicted depth as a soft constraint in multi-image problems.Source: §2 · p. 5
3 Map-free Relocalization
The task estimates a query camera's metric pose relative to a single RGB reference image with known intrinsics. The paper presents two routes: feature matching plus estimated depth, and direct relative pose regression.
3 Map-free Relocalization
- With known image intrinsics, the query's absolute pose is estimated as a scaled relative pose from the reference image, whose coordinate system is the world frame.The query pose is parameterized by rotation and translation mapping world points into the query camera's coordinate system.Source: §3 · p. 5
3.1 Feature Matching and Scale from Estimated Depth
- Feature matching estimates an essential-matrix pose up to scale, and monocular depth estimates provide the missing scale.The methods use SIFT, SuperPoint plus SuperGlue, or LoFTR; indoor and outdoor depth models are separately fine-tuned or trained on domain-specific datasets.Source: §3.1 · pp. 5–6, Figure 2 · p. 6
- The essential-matrix depth-scale method back-projects inlier matches to 3D and selects the translation scale with maximum correspondence consensus.It estimates the essential matrix with a 5-point solver and MAGSAC++ before decomposing rotation and unitary translation.Source: §3.1 · p. 6
- The PnP variant back-projects reference-image features to 3D and uses query-image 2D features to recover metric pose with RANSAC.The final estimate is refined using all inliers.Source: §3.1 · p. 6
- The 3D-3D Procrustes variant aligns depth-back-projected correspondences, optionally refines with ICP, and performs significantly worse than the preceding geometric variants.Its results are reported in the supplement.Source: §3.1 · p. 6
3.2 Relative Pose Regression
- Relative pose regression predicts metric pose in one forward pass, using a Siamese image network and alternative rotation and translation parameterizations.The tested outputs include quaternions, 6D rotations, discrete Euler angles, several translation parameterizations, and predicted 3D-3D correspondences.Source: §3.2 · p. 7, Figure 3 · p. 7
4 Map-free Relocalization Datasets
Section 4 reviews benchmark gaps, introduces the 655-scene dataset, and defines its pose, reprojection, and confidence evaluation protocol.
4.1 Existing Relocalization Datasets
- Existing relocalization datasets fit map-free evaluation poorly: scenes are not well captured by one image, and each dataset has only 1–12 distinct scenes.ScanNet and MegaDepth lack distinct mapping and query sequences, and MegaDepth poses lack metric scale.Source: §4.1 · p. 8
4.2 Niantic Map-free Relocalization Dataset
- The dataset covers 655 outdoor places, with two training scans per scene and reference-image/query-sequence pairs for validation and test scenes.Training, validation, and test splits contain 460, 65, and 130 scenes; capture conditions and scan overlap vary across places.Source: §4.2 · pp. 8–10
4.3 Evaluation Protocol
- Rotation error is angular, translation error is camera-center distance, and VCRE measures reprojection disagreement for virtual points as a proxy for AR-content misalignment.The evaluation uses VCRE thresholds of 5% and 10% of the image diagonal; the authors regard even a noticeable 10% offset as potentially acceptable for AR.Source: §4.3 · pp. 10–11
- Confidence evaluation trades the share of non-rejected query images against the precision of accepted poses at each confidence threshold.Confidence can be estimated from feature-matching inlier counts; models unable to estimate confidence have a flat precision curve.Source: §4.3 · p. 11
5 Experiments
The 7Scenes results show that relative-pose methods remain competitive with many mapping frames and degrade more gracefully as reference frames become scarce. On the more challenging Niantic dataset, feature-matching methods and relative-pose regression have different strengths, with practical accuracy limited by visual overlap, depth estimates, and confidence.
5.1 7Scenes
- For K = 1 on 7Scenes, SuperGlue with essential-matrix estimation and depth scale recovers over 50% of queries below 40px reprojection error, outperforming competing methods.Pose triangulation works well at K = 10 but cannot provide estimates at K = 1; feature matching plus depth scale outperforms the competition in that map-free setting.Source: §5.1 · p. 12
- On 7Scenes, feature matching plus depth scale outperforms relative-pose regression with K = 10 and K = 1, while regression performs relatively well at broader accuracy thresholds.Regression's lack of estimated confidence hurts its precision curves; DSAC* remains competitive in sparse regimes but requires per-scene training.Source: §5.1 · p. 13
5.2 Niantic map-free relocalization dataset
- Niantic is harder than 7Scenes across methods, with low query-reference overlap, changing conditions, and depth networks trained on KITTI for non-driving outdoor scenes contributing to the challenge.Feature-matching quality is affected by lighting and seasonal variation.Source: §5.2 · p. 13
- On Niantic, SuperGlue with depth scale achieves median angular rotation below 10° on many scenes, while translation error varies partly with depth-estimate variance.The qualitative results include good angular pose with inaccurate absolute scale, pointing to outdoor depth prediction as an improvement target.Source: §5.2 · p. 13
- On Niantic, relative-pose regression can outperform feature matching at broad VCRE thresholds, but feature matching has higher precision at practical thresholds.When geometric optimization fails from poor matches, feature-based estimates can be arbitrarily far from ground truth; relative-pose regression fails more gracefully.Source: §5.2 · p. 13
- Relative-pose regression does not predict confidence to flag spurious poses, and feature-matching confidence based on inliers is not always reliable.The authors identify confidence modeling for both method families as a direction for combining their advantages.Source: §5.2 · p. 13
6 Conclusion and Future Work
The paper introduces map-free relocalization and presents a diverse benchmark with an evaluation protocol intended to match augmented-reality use cases. It identifies future work in depth estimation, metric relative-pose accuracy, confidence estimation, and evaluation of stricter thresholds and burst-query variants.
6 Conclusion and Future Work
- The authors propose map-free relocalization and show that single-image depth prediction and relative-pose regression address the task with some success.They suggest improving outdoor depth-based scale estimates, metric relative-pose accuracy, and confidence estimates.Source: §6 · p. 14
- The Niantic dataset and benchmark cover diverse places of interest and provide an evaluation protocol intended to match augmented-reality use cases.The dataset and evaluation service are publicly available.Source: §6 · p. 14
- The authors leave stricter pose-error thresholds and burst-query evaluation as future directions; the dataset is already suitable for the burst-query variant.Stricter thresholds are intended to correspond to visually more pleasing results.Source: §6 · p. 14