Source-linked AI summary
Camera Relocalization by Computing Pairwise Relative Poses Using Convolutional Neural Network
Zakaria Laskar, Iaroslav Melekhov, Surya Kalia, Juho Kannala
TL;DR
Camera relocalization must remain robust and scalable across scenes despite limitations of point matching and scene-specific absolute-pose models. The paper trains a Siamese CNN on image pairs to retrieve database images, estimate relative poses, and recover absolute pose. It reports generalization to scenes unseen during CNN training and competitive results on challenging datasets.
Problem
Existing localization methods can be fragile in difficult visual conditions, dependent on scene coordinate frames, or restricted by scene-specific training and depth-map requirements.
Method
A Siamese CNN learns relative pose from image pairs; retrieval identifies similar database images, while triangulation and RANSAC-based fusion recover the query's absolute pose.
Results
The approach generalizes to scenes unseen during CNN training and achieves competitive results on challenging localization datasets.
Takeaways & Limitations
Decoupling relative-pose learning from scene coordinates supports localization across multiple scenes without requiring scene-specific network training.
Takeaways & Limitations
The retrieval stage uses a simple approach and is not the paper's primary focus; deeper retrieval analysis is deferred to future work.
Abstract
from arXiv · showhide
We propose a new deep learning based approach for camera relocalization. Our approach localizes a given query image by using a convolutional neural network (CNN) for first retrieving similar database images and then predicting the relative pose between the query and the database images, whose poses are known. The camera location for the query image is obtained via triangulation from two relative translation estimates using a RANSAC based approach. Each relative pose estimate provides a hypothesis for the camera orientation and they are fused in a second RANSAC scheme. The neural network is trained for relative pose estimation in an end-to-end manner using training image pairs. In contrast to previous work, our approach does not require scene-specific training of the network, which improves scalability, and it can also be applied to scenes which are not available during the training of the network. As another main contribution, we release a challenging indoor localisation dataset covering 5 different scenes registered to a common coordinate frame. We evaluate our approach using both our own dataset and the standard 7 Scenes benchmark. The results show that the proposed approach generalizes well to previously unseen scenes and compares favourably to other recent CNN-based methods.
1. Introduction
Camera relocalization is important but existing approaches face robustness, scalability, and scene-dependence limitations. The paper proposes scene-independent relative-pose learning and introduces a challenging indoor dataset for evaluation.
- Camera relocalization determines camera pose from visual scene representations and supports navigation, structure from motion, augmented reality, and SLAM.
- Point-based methods recover 6-DoF pose from 2D-3D matches but can fail under textureless scenes, illumination changes, occlusions, and repetitive structures.
- Depth-dependent learning methods restrict applicability because they require depth maps associated with training images.
- Absolute-pose CNNs depend on a particular scene coordinate frame, complicating multi-scene localization and limiting scalability to large environments.
- The proposed Siamese CNN learns relative pose from image pairs and computes absolute pose from several query-to-database relative-pose estimates.
- The approach is designed for multiple disjoint scenes, including scenes unavailable or sparsely represented during network training.
- The paper contributes a deep-learning relocalization method and a challenging indoor dataset with accurate ground-truth poses.
2. Related work
Prior localization methods use visual retrieval, 3D correspondences, regression forests, or absolute-pose CNNs, each with practical limitations. The paper instead uses Siamese relative-pose estimation with triangulation for scalable, scene-independent localization.
- Visual place recognition retrieves geo-tagged database images using descriptors, spatial matching, or visual words.
- Structure-based localization establishes 2D-3D matches and recovers 6-DoF pose with RANSAC and Perspective-n-Point, but descriptor matching is costly for large scenes.
- Combining visual place recognition with local SfM improves localization performance over 3D-based methods, although localization remains time-consuming.
- Regression-forest methods predict 3D locations for image pixels and use RANSAC to recover camera pose, while uncertainty-aware variants exploit predicted point uncertainty.
- CNN methods such as PoseNet regress 6-DoF camera pose directly from RGB images, while later architectures and losses improve localization accuracy.
- This paper predicts relative orientation and translation with a Siamese network, triangulates translations for absolute location, and targets scenes with different coordinate frames without training depth maps.
3. Proposed approach
The proposed system uses a Siamese CNN to retrieve nearby database images and regress relative poses, then fuses triangulated translation and orientation hypotheses into a full 6-DoF query pose.
- 3.1. Pairwise Pose Estimation: Training image pairs supervise a Siamese CNN with shared ResNet34 branches to regress relative orientation and translation.The orientation is represented by a 4-dimensional quaternion and translation by a 3-dimensional vector; the loss balances their errors with β > 0.
- 3.2. Localization Pipeline: The trained branch encodes query and database images as 512-dimensional global descriptors, ranked by dot-product similarity.The top N ranked database images are selected as nearest neighbours for the query.
- 3.2. Localization Pipeline: The retrieval stage is intentionally simple, while deeper retrieval analysis is deferred to future work and perfect retrieval is evaluated separately.The paper identifies camera localization, rather than image retrieval, as its primary focus.
- 3.2. Localization Pipeline: Relative pose is predicted for each query–neighbour pair, with normalized translation vectors providing directions from database cameras toward the query location.Although the network predicts scale information, the paper reports that recovering scale through the localization pipeline is more accurate and reliable.
- 3.2. Localization Pipeline: Pairs of shortlisted database images are triangulated into query-translation hypotheses, which are filtered by angular-consensus inliers using the remaining images.The hypothesis with the highest inlier count is selected, with tied estimates averaged.
- 3.2. Localization Pipeline: Each relative orientation yields a query-orientation hypothesis, and consensus filtering selects or robustly averages the highest-supported hypotheses.The final fused translation and rotation, combined with known database poses, produce the query camera's full 6-DoF pose.
4. Datasets
The evaluation uses the standard 7-Scenes benchmark and introduces University, a challenging five-scene indoor dataset with all locations registered to one global coordinate frame.
- 7Scenes: 7-Scenes provides RGB-D images from 7 indoor locations with variation in camera pose, motion blur, and perceptual aliasing.The experiments use the original paper's train and test split for each scene.
- University: University contains five indoor scenes—Office, Meeting, Kitchen, Conference, and Coffee Room—registered to a common global coordinate frame.The dataset includes multiple traversals per scene, with provided training and test splits totaling 9694 training and 5068 test images.
5. Experiments
The experiments evaluate the proposed relocalization system on 7Scenes and the University dataset, comparing it with CNN-based methods and a baseline while testing generalization and retrieval robustness. Results show favorable performance, scalability across scenes, transfer to unseen datasets, and robustness to nearest-neighbor viewpoint variation, although proposed improvement factors remain unvalidated.
- 7Scenes results: The proposed method outperforms similar-loss CNN methods on 7Scenes and surpasses PoseNet2 in four scenes, with direct comparability limited by PoseNet2’s different architecture and loss.The table caption reports median translation and orientation errors.
- Baseline comparison: The proposed system consistently improves rotation and translation over the ResNet34-Pose baseline across all scenes while using one jointly trained network instead of scene-specific models.The authors attribute the improvement plausibly to shared representations, multiple pose hypotheses with robust filtering, and a larger training set, but do not experimentally validate these factors.
- University dataset: On the University dataset, the translation-error improvement over the baseline increases from 2 cm on combined 7Scenes to 130 cm, supporting scalability across a 2500 m^2 multi-scene area.The University dataset contains five scenes registered to a common coordinate system.
- Generalization: When scenes are held out, performance drops gracefully; on University’s unseen Coffee Room scene, median position and orientation errors are 1.44 m and 19.22 degrees.The model is trained on the remaining four University scenes for this evaluation.
- Retrieval robustness: Using true nearest neighbors across wide viewpoint variation produces consistent performance, indicating robustness to nearest-neighbor quality, although N = 5 may not be optimal for 7Scenes.Queries often have more than 300 true nearest neighbors because training images are closely spaced.
6. Conclusion
The proposed system addresses limitations of camera localization setups by leveraging training images during both training and testing. Its results support extension to scenes without prior training, while improved generic relative pose estimation remains future work.
- The system leverages training images at training and test time to mitigate limitations of existing camera localization setups.
- Results demonstrate that the proposed system is extendable to scenes without prior training.
- Improving generalization through joint relative-pose and image-similarity training is identified as future work.
- Supplementary Material -
The supplementary material presents challenging cases and localization-performance visualizations, alongside accuracy results across viewpoint changes. These materials compare the proposed approach with baseline CNN methods on 7Scenes.
- Figure 4 presents challenging cases from the University dataset.
- Average errors are reported as paired translational and angular values for the evaluated methods.
- Table 6 reports camera relocalization accuracy for different viewpoint changes between query and database images.
- Figure 5 compares the proposed approach, ResNet34-Pose, and PoseNet using normalized cumulative error histograms across all 7Scenes.