Source-linked AI summary
Back to the Feature: Learning Robust Camera Localization from Pixels to Pose
Paul-Edouard Sarlin, Ajaykumar Unagar, Måns Larsson, Hugo Germain, Carl Toft, Viktor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, Torsten Sattler
TL;DR
Existing learning-based camera localization often regresses geometry from images, limiting scene generalization or tying models to particular environments. PixLoc learns robust multiscale features and aligns them with an explicit 3D model through differentiable geometric optimization. It reports competitive pose accuracy and generalization to new scenes without retraining or fine-tuning, while also improving some existing localization poses.
Problem
Existing end-to-end methods regress poses or 3D geometry from images, limiting generalization to new viewpoints and scenes or tying parameters to specific scenes.
Method
PixLoc trains a CNN and differentiable Levenberg-Marquardt alignment end-to-end to estimate pose by matching multiscale deep features against an explicit 3D model.
Results
PixLoc achieves competitive pose accuracy and generalizes to new scenes widely differing from training data without retraining or fine-tuning.
Takeaways & Limitations
Robust generic features combined with classical image alignment can support accurate localization while keeping scene geometry outside the network.
Takeaways & Limitations
PixLoc can converge to incorrect local minima under excessively large initial reprojection errors from large viewpoint changes and can be sensitive to outliers and camera miscalibration.
Abstract
from arXiv · showhide
Camera pose estimation in known scenes is a 3D geometry task recently tackled by multiple learning algorithms. Many regress precise geometric quantities, like poses or 3D points, from an input image. This either fails to generalize to new viewpoints or ties the model parameters to a specific scene. In this paper, we go Back to the Feature: we argue that deep networks should focus on learning robust and invariant visual features, while the geometric estimation should be left to principled algorithms. We introduce PixLoc, a scene-agnostic neural network that estimates an accurate 6-DoF pose from an image and a 3D model. Our approach is based on the direct alignment of multiscale deep features, casting camera localization as metric learning. PixLoc learns strong data priors by end-to-end training from pixels to pose and exhibits exceptional generalization to new scenes by separating model parameters and scene geometry. The system can localize in large environments given coarse pose priors but also improve the accuracy of sparse feature matching by jointly refining keypoints and poses with little overhead. The code will be publicly available at https://github.com/cvg/pixloc.
1. Introduction
Visual localization estimates camera position and orientation in a known scene, but existing end-to-end regressors struggle with scene dependence and changing viewpoints or appearances. PixLoc instead learns robust features and uses geometric alignment with explicit scene structure, achieving broad generalization.
- Motivation: Visual localization estimates a camera’s position and orientation from an image in a known scene, supporting autonomous robotics and AR/VR.It is framed as a key step toward truly autonomous robots and a prerequisite for augmented and virtual reality systems.
- Limitations of prior methods: End-to-end pose and coordinate regression can be scene-specific and struggle with novel viewpoints, appearances, and larger complex scenes.These methods require training for or adaptation to new scenes, while changing conditions such as nighttime imagery remain challenging.
- Motivation: Existing regression methods hinder generalization by predicting poses or 3D geometry solely from images despite available pose priors and reconstructed scene geometry.Pose priors can come from retrieval or sensors, while Structure-from-Motion and SLAM provide 3D geometry.
- PixLoc: PixLoc learns robust features while leaving geometric estimation to classical principles, rather than encoding 3D maps or relearning basic geometric relations.The approach is motivated by direct image alignment and learned representations for outlier rejection.
- PixLoc: PixLoc aligns dense CNN features with an explicit 3D scene model and is trained end-to-end from pixels to pose, making it scene-agnostic.Given an initial pose from image retrieval, the pipeline uses classical geometric optimization and can also refine poses from other localization methods.
- Results: PixLoc is reported as the first end-to-end visual localization approach with exceptional generalization to new scenes.The claim concerns deployment in scenes widely differing from training data without retraining or fine-tuning.
2. Related work
Prior localization systems combine learned feature processing, explicit 3D representations, retrieval, and geometric solvers, but end-to-end learning often remains fragmented or scene-bound. PixLoc instead uses multilevel feature alignment with a learned optimization pipeline for wide-baseline, cross-condition localization.
- Feature-based localization: Feature-based localization commonly matches 2D pixels to 3D points using explicit scene models and image retrieval to scale to large environments.Recent systems learn detection, description, matching, and outlier filtering, but these components are often trained independently.
- Regression-based methods: Pose regression lacks geometric constraints and generalizes poorly, while coordinate regression is more robust but limited by network capacity and scene requirements.Both approaches are described as scaling poorly and requiring adaptation for each scene.
- Regression-based methods: Relative pose regression avoids memorizing scene geometry but remains less accurate than feature-based methods.SANet removes scene representation from the network by regressing 3D coordinates from an input point cloud.
- Learned optimization: Learned optimization methods may predict robust costs, damping factors, or pose updates, adding complexity and potentially tying optimization to training distributions.The related-work discussion covers unrolled optimizers, implicit derivatives, and losses designed to mimic optimization steps.
- PixLoc: PixLoc uses direct multilevel feature alignment trained for wide-baseline, cross-condition pose estimation from sparse measurements.It is presented as addressing the small convergence basin and limited scalability associated with prior direct alignment approaches.
3. PixLoc: from pixels to pose
PixLoc separates visual representation learning from geometric pose estimation by aligning multiscale CNN features against an explicit 3D model. It learns uncertainty and optimization parameters end-to-end while remaining scene-agnostic and compatible with varied 3D representations.
- Localization pipeline: PixLoc estimates camera pose by aligning query and reference images using deep features and known 3D scene structure.The CNN and optimization parameters are trained end-to-end from ground-truth poses.
- Image representation: Multiscale, L2-normalized CNN features provide robustness to illumination and viewpoint changes while preserving gradients for pose alignment.Coarse levels improve robustness, whereas finer levels enhance accuracy.
- Direct alignment: PixLoc minimizes a weighted nonlinear least-squares feature-alignment cost with Levenberg–Marquardt updates on the SE(3) pose manifold.The alignment begins from an initial pose and iteratively updates the six pose parameters.
- Infusing visual priors: Learned uncertainty maps downweight locations that are unreliable or that create misleading local minima, including dynamic objects and repeated patterns.The weights approach zero when either query or reference locations are uncertain and are learned to maximize pose accuracy.
- Fitting the optimizer to the data: PixLoc learns per-level, per-parameter damping factors instead of semantic optimizer heuristics, allowing motion priors to be learned from pose or residual distributions.The damping is distinct for each of the six pose parameters and each feature level.
- Learning from poses: Because the CNN never processes 3D points, PixLoc can use sparse or dense point clouds, depth maps, meshes, LiDAR scans, lines, and other primitives.Training can also tolerate imperfect or noisy 3D representations such as sparse SfM models.
5. Experiments
PixLoc is evaluated across standard and large-scale localization benchmarks, where it generalizes across scenes, competes with complex pipelines, and can refine existing poses. Its performance depends on learned features, optimization choices, and sufficiently accurate initialization.
- 5.1. Comparison to learned approaches: PixLoc generalizes from outdoor training data to unseen outdoor and indoor scenes, outperforming SANet and matching or exceeding scene-specific approaches.It is also comparable to hloc, a complex feature-matching pipeline.
- 5.2. Large-scale localization: PixLoc is competitive with complex feature-matching pipelines and consistently outperforms ESAC on the large-scale Aachen, RobotCar, and CMU benchmark.With oracle initialization, it becomes competitive on driving datasets but remains weaker on Aachen’s sparse-reference setting.
- 5.2. Large-scale localization: PixLoc is less robust when retrieval provides an incorrect location, because direct alignment cannot converge from a naive or excessively coarse pose prior.This issue is especially apparent on Aachen, where reference images are sparser and initial priors are coarser.
- 5.4. Additional insights: Unrolling the optimizer, learning features, damping, and confidences all contribute to performance over classical photometric alignment.Compact learned features perform worse than high-dimensional representations, while a Gauss-Newton loss failed to converge on the challenging training data.
- 5.3. Pose post-processing with PixLoc: PixLoc can refine existing localization outputs, yielding up to +2.4% recall on CMU and improved accuracy on RobotCar Night.Refinement does not improve RobotCar Day and is detrimental on Aachen at 0.25m.
6. Conclusion
The paper concludes that robust, generic features combined with classical image alignment can support accurate, scene-agnostic camera localization. PixLoc generalizes to new scenes without retraining, while still requiring good initialization and remaining vulnerable to difficult geometric and imaging conditions.
- 6. Conclusion: PixLoc avoids learning basic geometry or encoding 3D maps, instead learning robust generic features for alignment with existing 3D models.The system combines end-to-end training with uncertainty modeling to learn complex yet interpretable priors.
- 6. Conclusion: PixLoc can be deployed in new scenes widely differing from training data without retraining or fine-tuning, while achieving accuracy competitive with more complex pipelines.The conclusion presents this as the paper’s central generalization result.
- 6. Conclusion: PixLoc requires a good initialization and is presented as a first step toward handling long-term, extreme changes in appearance and 3D structure.The conclusion frames robust, reliable localization under such changes as future progress rather than a completed capability.
A. Convergence and initial pose
PixLoc’s convergence depends strongly on the quality of its initial pose, while dense reference coverage or pose averaging can improve initialization. On Aachen, convergence is substantially reduced when the initial reprojection error is large.
- Convergence: PixLoc can converge to spurious local minima when the initial pose is too coarse.This issue is pronounced on Aachen, where reference images are sparse.
- Initial pose: The required reference-image density for high success depends on the camera’s distance to the scene.The initial 2D reprojection error reflects rotation, translation, and distance-to-structure errors.
- Convergence: 80% of cases converge within 1 meter only when the initial reprojection error is smaller than 200 pixels.Success is significantly reduced for larger initial errors.
- Initial pose: On datasets with dense reference poses or constrained motion, initializing with the first retrieved image’s pose is sufficient.This applies to 7Scenes, Cambridge, RobotCar, and CMU under the stated motion conditions.
- Initial pose: Averaging the poses of top retrieved images improves PixLoc’s convergence compared with selecting the first image’s pose.On Aachen, additional filtering clusters the top-three images by covisibility before weighted pose averaging.
B. Benefits of training on different datasets
PixLoc learns environment-specific semantic and motion priors from different training datasets, and these priors affect cross-dataset localization performance. Accurate 3D models also improve localization, especially for indoor scenes.
- Benefits of training on different datasets: Training separately on CMU and MegaDepth enables PixLoc to learn task-specific priors adapted to driving and landmark-photography environments.The datasets differ in camera motion, scene semantics, and appearance changes.
- Benefits of training on different datasets: PixLoc learns to ignore different unreliable visual elements depending on the training dataset.Tree silhouettes are reliable on CMU but ignored by the MegaDepth-trained model.
- Benefits of training on different datasets: CMU and MegaDepth induce different motion priors through the damping factors.CMU emphasizes translation along camera x and z, whereas MegaDepth distributes motion more uniformly across the 6 DoF.
- Benefits of training on different datasets: CMU-trained PixLoc outperforms MegaDepth-trained PixLoc on the driving-oriented RobotCar dataset.On Aachen, it still outperforms the scene-specific ESAC approach, indicating generalization across scenarios alongside environment adaptation.
- Benefits of training on different datasets: In 7Scenes, depth-fused 3D points yield more accurate localization than SfM-triangulated points.Replacing SfM points with back-projected interpolated depth produces a more accurate model with the same sparsity.
D. Inaccuracy of the ground truth poses
Pose inaccuracies in RobotCar v2 and possibly Extended CMU Seasons complicate evaluation and training. RobotCar ground-truth artifacts may explain both the leaderboard ceiling and limited daytime refinement gains.
- D. Inaccuracy of the ground truth poses: RobotCar v2 contains large reprojection errors in some publicly evaluable ground-truth poses despite qualitatively more accurate hloc poses.The issue may explain why no method localizes more than 58% of daytime images at the finest public threshold.
- D. Inaccuracy of the ground truth poses: RobotCar ground-truth inaccuracies may explain why PixLoc pose refinement does not improve daytime queries.The passage links the evaluation artifacts to the lack of observed daytime gains.
- D. Inaccuracy of the ground truth poses: Similar pose artifacts make training supervision noisier in Extended CMU Seasons.Whether test-sequence poses have the same issue is unknown because they are not publicly available.
E. Qualitative examples
The paper presents qualitative successes and failures for Extended CMU Seasons and Aachen Day-Night, alongside released visualizations of uncertainty and optimization.
- E. Qualitative examples: Qualitative examples include successful and failed localization cases for Extended CMU Seasons and Aachen Day-Night.Videos and animations of uncertainties and optimization are provided with the code and trained weights.
F. Attraction basin
PixLoc characterizes attraction basins by tracing feature-gradient directions across image pixels, scales, and network levels. The resulting basin scores visualize where optimization can converge toward selected points.
- Computation: Attraction basins are computed by backtracking feature gradients across levels and scales for each selected point.For each pixel, the method examines two opposite-gradient neighbors in an 8-connected neighborhood.
- Computation: Pixels join a point’s basin when at least one opposite-gradient neighbor is already in the basin, with soft voting based on gradient angle.The algorithm initializes the point of interest and iterates from the finest to the coarsest level until basin scores stabilize.
- Visualization: The visualization colors basin pixels by total-gradient angle, with additional examples displaying the gradient field as arrows.The main-paper example appears in Figure 5, while Figure 14 provides further examples.
G. Experimental details
The experiments specify PixLoc’s implementation, training, inference, ablation setup, and qualitative behavior across CMU, Aachen, and RobotCar examples. The reported figures also identify successful cases, inaccurate reference poses, and failure modes such as repeated structures, occlusion, and appearance changes.
- Implementation: PixLoc uses PyTorch, Cholesky decomposition for the Levenberg-Marquardt system, bilinear interpolation, and a Cauchy robust cost.Residuals and Jacobians ignore projections outside the image or within 2 pixels of image borders; λmin=−6 and λmax=5.
- Training: Training samples 512 visible 3D points per query-reference pair using SfM covisibility information.Training runs for 50k iterations with Adam at a constant learning rate of 10−5; pair losses are clamped to 50 pixels and gradients clipped to [−1, 1].
- Training: CMU training uses slices 8-12 and 22-25, validation uses slices 6, 13, and 21, and batches contain 3 image pairs.Images are resized to a smallest dimension of 720 pixels, cropped to 720-pixel squares, and initialized with the reference-image pose.
- Training: MegaDepth training follows the Dusmanu et al. scene split, samples overlap scores in [0.3, 1], and uses 512-pixel square crops with batches of 6 pairs.Non-upright images are rotated using scene gravity; the supplied passage begins describing the initial-pose sampling range but does not complete it.
- Inference: Inference uses 5 or 3 reference images, allows up to 100 iterations per level, and stops when gradients or steps become sufficiently small.When refining hloc poses, optimization uses only medium and fine levels; images are resized so their longest dimension is 1024 pixels.
- Qualitative evaluation: CMU examples show successful localization despite large initial errors and cross-season appearance changes, while failures involve repeated elements or insufficient spatial context.Aachen failures additionally include occlusion, large viewpoint changes, and local incorrect minima; RobotCar figures indicate inaccurate ground-truth query poses relative to hloc.