Source-linked AI summary
Geometric Loss Functions for Camera Pose Regression with Deep Learning
Alex Kendall, Roberto Cipolla
TL;DR
PoseNet offers scalable monocular metric localisation but its naive loss produces insufficient accuracy and requires costly balancing of position and orientation. This paper introduces geometry-based losses, including reprojection error and data-learned weighting, and reports large improvements across indoor and outdoor datasets while narrowing the gap to traditional feature-based methods.
Problem
PoseNet is scalable and robust but lacks sufficient metric accuracy, while its position–orientation loss requires expensive hyper-parameter tuning.
Method
The paper trains PoseNet with geometrically formed losses, including scene reprojection error that combines rotation and translation, and explores learning their weighting from data.
Results
The geometric approach substantially improves PoseNet across datasets ranging from indoor rooms to Dubrovnik, narrowing the performance gap to traditional point-feature methods.
Takeaways & Limitations
Geometry-based training enables PoseNet to retain scalable monocular relocalisation while achieving more accurate camera-pose estimates across varied scene scales.
Takeaways & Limitations
On Dubrovnik, the method still does not reach the fine-grained accuracy of geometric SIFT-registration methods, possibly because only 6k town images were available for training.
Abstract
from arXiv · showhide
Deep learning has shown to be effective for robust and real-time monocular image relocalisation. In particular, PoseNet is a deep convolutional neural network which learns to regress the 6-DOF camera pose from a single image. It learns to localize using high level features and is robust to difficult lighting, motion blur and unknown camera intrinsics, where point based SIFT registration fails. However, it was trained using a naive loss function, with hyper-parameters which require expensive tuning. In this paper, we give the problem a more fundamental theoretical treatment. We explore a number of novel loss functions for learning camera pose which are based on geometry and scene reprojection error. Additionally we show how to automatically learn an optimal weighting to simultaneously regress position and orientation. By leveraging geometry, we demonstrate that our technique significantly improves PoseNet's performance across datasets ranging from indoor rooms to a small city.
1. Introduction
Large-scale visual localisation remains difficult under viewpoint and appearance changes. The paper improves PoseNet by replacing its naive pose loss with geometry-based losses that avoid expensive hyper-parameter tuning and narrow the gap to traditional methods.
- Large-scale localisation is important for applications including autonomous vehicles, unmanned aerial vehicles, and augmented reality, but remains challenging in uncontrolled environments.
- Point features such as SIFT and ORB struggle across weather, lighting, and environmental changes because they lack robust global context.
- PoseNet uses end-to-end deep learning to regress camera pose from a single image, leveraging entire-image appearance and shape features for robust and fast localisation.
- PoseNet’s main weakness was insufficient metric accuracy compared with geometric methods, despite its scalability and robustness.
- The paper introduces geometrically formed loss functions, including scene-geometry reprojection error, to remove or learn the position–orientation weighting factor.
- Outdoor scenes spanning 50,000m^2 to 2km^2 achieved relocalisation accuracies of a few meters and degrees, while small rooms reached 0.2–0.4m accuracy.
2. Related work
Related localisation methods differ in whether they recognise places or estimate metric pose. PoseNet targets metric localisation with a scalable, globally relocalising neural representation, while addressing its accuracy deficit through geometry.
- Place recognition identifies which landmark is visible, whereas metric localisation estimates the camera’s metric position and orientation.
- Traditional metric localisation uses 2D–3D feature correspondences in a structure-from-motion model to estimate six-degree-of-freedom pose precisely.
- Traditional feature-based methods require large feature databases, can be expensive and poorly scalable, and may be vulnerable to changing environmental conditions.
- PoseNet combines global relocalisation without a good initial pose estimate with continuous metric pose prediction, without memory proportional to scene size.
- The paper focuses on PoseNet’s insufficient accuracy relative to traditional methods by incorporating multi-view scene geometry into its learning objective.
3. Model for camera pose regression
The model estimates a six-degree-of-freedom camera pose directly from a monocular image. It represents position and orientation jointly, using a quaternion for orientation and supervised pose labels.
- The network maps a monocular image I to an estimated pose containing 3-D camera position and orientation.
- Orientation is represented with a quaternion, while pose labels are obtained through structure-from-motion.
- The pose is defined relative to an arbitrary global reference frame centered in practice at the mean location of all camera poses.
3.1. Architecture
PoseNet adapts a classification network for camera-pose regression by replacing its classification head with a pose-output layer and normalizing quaternion orientation.
- The architecture adapts GoogLeNet for pose regression, allowing use of pretrained classification weights.
- The classification regression and softmax layers are replaced by a fully connected layer producing a seven-dimensional pose vector.
- The pose vector contains three position dimensions and four quaternion orientation dimensions.
- A normalization layer constrains the four-dimensional quaternion orientation vector to unit length.
3.2. Pose representation
The paper compares several representations for camera orientation because the output space strongly affects learning. It selects quaternions as a smooth, continuous representation whose unit-length constraint is easy to enforce during back-propagation.
- The study evaluates Euler angles, axis-angle, SO(3) rotation matrices, and quaternions for representing camera orientation.The comparison focuses on their suitability for deep learning.
- Euler angles are difficult to learn because they are non-injective and suffer from gimbal lock.Wrapping at 2π creates multiple values for the same angle, while gimbal lock prevents a unique parametrisation.
- SO(3) rotation matrices are over-parametrised, and back-propagation must enforce their orthogonality constraint.For 3-D rotations, they are 3×3 matrices belonging to the special orthogonal group SO(3).
- Quaternions map arbitrary four-dimensional values to valid rotations by normalization and provide a continuous, smooth representation.Their unit-manifold constraint is simpler to enforce than rotation-matrix orthonormalization, although each rotation has two quaternion mappings.
3.3. Loss function
The paper develops pose-regression losses that address the challenge of jointly learning translation and rotation with different units and scales. It replaces fixed balancing with geometry-based reprojection and learned task uncertainty, while examining regression norms and joint supervision.
- Pose regression must learn position and orientation together despite their different units and scales.
- Simultaneously learning position and orientation: The original PoseNet loss combines position and orientation errors through a weighted sum requiring scale factor β.β balances the expected position and orientation errors.
- Simultaneously learning position and orientation: Jointly regressing position and orientation produces a better pose representation than training separate models or branching into separate features.Separate features deny each task information needed to factor out the other quantity.
- Simultaneously learning position and orientation: β requires expensive tuning, with reported indoor values of 120 to 750 and outdoor values of 250 to 2000.Each grid-search experiment can take days.
- Learning an optimal weighting: The proposed uncertainty-based loss learns the weighting between position and orientation without manually specified hyperparameters.It models homoscedastic task uncertainty and learns free scalar uncertainty values through backpropagation.
- Geometry-based loss: Reprojection error combines rotation and translation in one scalar by comparing ground-truth and predicted projections of scene points onto the image plane.The projection function maps 3-D scene points to 2-D image coordinates using camera position, orientation, intrinsic calibration K, and rotation matrix R.
- Geometry-based loss: The reprojection loss can adapt position–orientation weighting to scene geometry, balancing them differently for near and distant geometry.
- Regression norm: Huber and Tukey robust norms reduced performance on the evaluated datasets by over-attenuating difficult examples.The authors suggest robust norms might help on noisier datasets.
4. Experiments
Experiments evaluate geometrically grounded losses across indoor and outdoor datasets, showing improved PoseNet localisation and a practical two-step training strategy. The approach narrows but does not eliminate the gap to traditional feature-based methods.
- Loss comparison: Homoscedastic uncertainty learns an effective position-orientation weighting, whereas reprojection loss fails from random initialization but works for fine-tuning.The recommended scheme first trains with learned weighting, then fine-tunes with reprojection error when scene geometry is available.
- Datasets and setup: The experiments benchmark loss functions and PoseNet variants on Cambridge Landmarks, 7 Scenes, and Dubrovnik 6K.The datasets span outdoor urban imagery, indoor rooms, and tourist photographs from Dubrovnik.
- Benchmarking localisation accuracy: The geometry-based loss consistently improves over PoseNet’s naive loss across Cambridge Landmarks and 7 Scenes.The improvement is reported as especially pronounced on 7 Scenes, which provides more training data per scene.
- Benchmarking localisation accuracy: Reprojection optimisation improves rotation accuracy overall, sometimes at the expense of positional precision.This reveals a different trade-off between position and orientation compared with homoscedastic-uncertainty optimisation.
- Comparison to SIFT-feature approaches: On Dubrovnik, the proposed method improves substantially over original PoseNet but does not reach the fine-grained accuracy of geometric SIFT-registration methods.The authors hypothesise that the gap reflects limited training data, with only 6k images across the town; the method is nevertheless faster.
5. Conclusions
The paper concludes that scene geometry supports more effective simultaneous regression of position and orientation without hyper-parameter tuning. It reports improved PoseNet results that narrow the gap to traditional point-feature approaches, while identifying video input as future work.
- The proposed training algorithm regresses position and orientation with scene geometry without requiring hyper-parameter tuning.
- The method produces large improvements over PoseNet’s original loss and narrows the performance gap to traditional point-feature approaches.
- The authors identify extending the architecture to video input with multi-view stereo as future work.They note that video data is readily available for many localisation applications, including mobile robotics.