Source-linked AI summary
Modelling Uncertainty in Deep Learning for Camera Relocalization
Alex Kendall, Roberto Cipolla
TL;DR
Visual relocalization must remain accurate when viewpoint and appearance changes make conventional SLAM unreliable. The paper addresses this with a Bayesian convolutional network that regresses 6-DOF pose from one RGB image, improving relocalization while producing usable uncertainty estimates for error prediction and scene detection.
Problem
Large viewpoint and appearance changes limit conventional SLAM, motivating relocalization methods that can assess whether pose predictions are trustworthy.
Method
A Bayesian convolutional neural network uses Monte Carlo dropout pose samples to regress camera pose and estimate uncertainty from their covariance.
Results
Monte Carlo dropout considerably improves localization accuracy over PoseNet, while uncertainty strongly correlates with metric relocalization error.
Takeaways & Limitations
The uncertainty estimate can predict relocalization error and detect whether a previously observed landmark or scene is present.
Abstract
from arXiv · showhide
We present a robust and real-time monocular six degree of freedom visual relocalization system. We use a Bayesian convolutional neural network to regress the 6-DOF camera pose from a single RGB image. It is trained in an end-to-end manner with no need of additional engineering or graph optimisation. The algorithm can operate indoors and outdoors in real time, taking under 6ms to compute. It obtains approximately 2m and 6 degrees accuracy for very large scale outdoor scenes and 0.5m and 10 degrees accuracy indoors. Using a Bayesian convolutional neural network implementation we obtain an estimate of the model's relocalization uncertainty and improve state of the art localization accuracy on a large scale outdoor dataset. We leverage the uncertainty measure to estimate metric relocalization error and to detect the presence or absence of the scene in the input image. We show that the model's uncertainty is caused by images being dissimilar to the training dataset in either pose or appearance.
I. MOTIVATION
The paper targets visual relocalization under large viewpoint and appearance changes, where conventional SLAM and coarse scene-classification approaches are limited. It extends direct image-to-pose regression with Bayesian uncertainty to improve localization and assess prediction confidence.
- Large viewpoint or appearance changes undermine point-based and dense SLAM because their representations and small-frame-change assumptions are fragile.
- Appearance-based SLAM provides only coarse poses by classifying images among a limited set of discrete scene locations.
- PoseNet directly regresses the full 6-DOF camera pose from a monocular image without tracking, landmark matching, or frame-to-frame correspondence.
- The Bayesian extension estimates localization uncertainty while preserving real-time operation and requiring no additional memory.The system operates in under 6ms per frame on a GPU.
- The uncertainty measure is used to estimate metric position and orientation error and to detect whether the scene is present in the input image.
II. RELATED WORK
Related work shows that feature-based relocalization struggles with real-world viewpoint and appearance variation, while existing probabilistic methods typically model sensor noise rather than model uncertainty. The proposed framework instead measures uncertainty in the model’s ability to represent the input.
- Point-based SLAM and relocalization features commonly lose tracking under rapid motion, large viewpoint changes, or significant appearance changes.
- CNN representations offer hierarchical shape and contour cues, motivating their use for direct full 3D camera-pose regression.
- Random-forest scene-coordinate methods were demonstrated only at a few m3 scale and require RGB-D data, constraining them in practice to indoor scenes.
- Earlier Bayesian localization approaches estimate uncertainty from sensor-noise models rather than uncertainty about whether the model represents the data.
- This framework assumes no input noise and measures model uncertainty for localization.
- Bayesian neural networks represent uncertainty through distributions over network weights, but inference is often computationally expensive and requires posterior approximations.
III. MODEL FOR DEEP REGRESSION OF CAMERA POSE
The model builds on PoseNet’s end-to-end deep regression of camera pose and introduces Bayesian inference through dropout-based weight sampling. Pose estimates are averaged across stochastic evaluations, while uncertainty is derived from their variance.
- PoseNet replaces classifiers with regressors for 3D position x and quaternion orientation q, training a 7-dimensional pose vector with Euclidean loss.
- The loss parameter β balances learning position and orientation simultaneously during end-to-end stochastic-gradient training.
- Dropout sampling is interpreted as approximate Bayesian inference over network weights, forming a Bayesian PoseNet.
- Variational inference approximates the intractable posterior p(W | X, Y) by learning q(W) through KL-divergence minimization.
- At test time, stochastic dropout evaluations produce pose samples whose average is the final pose and whose variance yields uncertainty.
- Dense sliding-window pose samples also correlate with localization error, but less strongly than samples generated by Bernoulli weight sampling.
A. Estimating Uncertainty
The Bayesian pose network samples a posterior pose distribution and summarizes it with a unimodal Gaussian. Its covariance trace provides a scalar uncertainty measure, while alternative multimodal summaries performed worse.
- A. Estimating Uncertainty: Monte Carlo dropout samples reveal posterior pose distributions, with the final network producing an approximately single-modal estimate.Shallower features can instead produce multiple pose hypotheses for visually ambiguous landmarks.
- A. Estimating Uncertainty: The final pose samples are fit with a unimodal Gaussian whose mean gives the pose estimate and covariance trace gives scalar uncertainty.The trace is rotationally invariant and empirically correlates strongly with metric relocalization error.
- A. Estimating Uncertainty: The covariance determinant was numerically poorer because elongated Gaussians can combine large and small eigenvalues into a small product.The determinant measures Gaussian volume, whereas the trace sums eigenvalues.
- A. Estimating Uncertainty: Geometric medians, Dirichlet-process Gaussian mixtures, and largest-cluster means all produced poorer localization uncertainty than a single unimodal Gaussian.These alternatives were considered to account for multimodality.
B. Creating a Comparable Uncertainty Statistic
The paper normalizes scene-specific uncertainty values into comparable Z-scores. Gamma-distribution percentiles for translation and rotation are averaged into one confidence score.
- B. Creating a Comparable Uncertainty Statistic: The proposed Z-score normalizes model uncertainty so values can be directly compared between models.It is designed as a comparable uncertainty statistic rather than a raw scene-specific value.
- B. Creating a Comparable Uncertainty Statistic: A Gamma distribution models scene-test uncertainty because it uses two parameters, stays positive, and is right skewed.The Street-scene uncertainty histogram is described as a reasonable Gamma fit.
- B. Creating a Comparable Uncertainty Statistic: Gamma cumulative distribution functions convert translational and rotational uncertainties into separate percentiles treated as Z-scores.The percentile measures each new image’s position relative to the scene’s uncertainty population.
- B. Creating a Comparable Uncertainty Statistic: Because rotational and translational uncertainties are highly correlated, their Z-scores are averaged into one overall pose-confidence percentile.This produces a single uncertainty value representing the model’s overall localization confidence.
C. Architecture
The architecture uses selective dropout and Monte Carlo sampling to obtain probabilistic poses without fully Bayesian convolutional layers. Sampling improves localization, with convergence at approximately 40 samples.
- C. Architecture: Rotational and translational uncertainty values are strongly correlated, supporting a single overall uncertainty value.The relationship is shown for individual scenes and all scenes.
- C. Architecture: Dropout after every convolutional layer degraded localization by approximately 10% because it acted as too strong a regularizer.The paper therefore used dropout selectively rather than throughout the fully Bayesian network.
- C. Architecture: The authors attribute the difficulty of transferring pretrained filters with dropout to fragile feature co-adaptations formed without dropout.They suggest these co-adaptations are not preserved when optimizing the dropout-equipped network.
- C. Architecture: The dropout probability was neither changed nor optimized, and this hyperparameter choice was left for future work.This is an explicit architectural limitation.
- C. Architecture: Less than five Monte Carlo samples typically surpass a single pose-regression network, while approximately 40 samples reach convergence.Additional samples produce no further localization-accuracy increase.
V. RESULTS
Experiments on Cambridge Landmarks and 7 Scenes show that Bayesian PoseNet improves localization over PoseNet. Its probabilistic model performs consistently better across the reported error thresholds.
- V. RESULTS: Monte Carlo dropout produces a considerable localization-accuracy improvement over PoseNet on Cambridge Landmarks and 7 Scenes.The evaluation averages 100 Monte Carlo dropout samples from probabilistic PoseNet.
- V. RESULTS: The mean of probabilistically sampled poses yields a more accurate solution than the non-probabilistic PoseNet estimate.The improvement is attributed to accounting for uncertainty in model selection.
- V. RESULTS: Probabilistic PoseNet performs consistently better than non-probabilistic PoseNet for all error thresholds shown in the cumulative error histograms.The comparison is reported for two scenes.
A. Uncertainty as an Estimate of Error
The probabilistic PoseNet’s uncertainty is strongly correlated with translational and rotational relocalization error, while its predictions improve over the non-probabilistic model.
- Uncertainty is very strongly correlated with metric relocalization error for both translational and rotational predictions.The relationship is linear, although its proportionality gradient varies significantly between scenes.
- Up to 10% relocalization-accuracy improvement is observed over Dense PoseNet on the Cambridge Landmarks and 7 Scenes datasets.The comparison includes PoseNet, a nearest-neighbour baseline, and SCORE Forests, with the latter requiring depth input.
- The probabilistic PoseNet performs consistently better than the non-probabilistic PoseNet across all error thresholds.
- An overall uncertainty estimate can be formed by averaging normalized rotational and translational uncertainty values.Metric error and uncertainty are correlated across both position and orientation.
B. Uncertainty as a Landmark Detector
The uncertainty metric can identify whether an input image belongs to a trained landmark, even though the network was not trained for landmark classification.
- The model assigned the lowest uncertainty to images evaluated by the model trained on their source scene.This procedure used normalized uncertainty and compared each image across all scene models.
- A confusion matrix evaluated landmark recognition across Cambridge Landmarks and 7 Scenes, excluding Street because it overlaps other landmarks.
- Combining normalized rotation and translation uncertainties often produced a superior and more robust error metric.
- The probabilistic architecture correctly identifies when the trained landmark is absent, despite no landmark-classification training.The uncertainty measure estimates both prediction accuracy and whether the presented landmark is outside the model’s training scope.
C. What Makes the Model Uncertain About a Pose?
Model uncertainty increases when test images are dissimilar to training examples in the localization feature space, with difficult visual conditions producing additional uncertainty.
- The nearest-training-image distance is a feature-space similarity measure rather than a comprehensive explanation of uncertainty.An initial plot using this distance showed only a slight increasing trend and was insufficient for conclusions.
- PoseNet’s 2048-dimensional feature vector represents image conditions such as weather, lighting, and object pose.Euclidean distance between feature vectors measures image similarity for nearest-neighbour analysis.
- Translational and rotational errors are plotted against their respective estimated uncertainties for King’s College and all scenes.The uncertainty scale varies significantly between models, motivating per-model normalization.
- A strong relationship links model uncertainty to distance from the nearest training image in PoseNet’s localization feature space.The model is more uncertain for images less similar to the training dataset.
- Images with unusually large uncertainty are typically difficult to localize because of heavy occlusion or strong silhouette lighting.
D. System Efficiency
The Bayesian PoseNet retains real-time, scalable operation while adding uncertainty estimation through Monte Carlo dropout, with modest sampling overhead and no extra parametrisation.
- The probabilistic approach retains PoseNet’s scalability because memory and computational requirements do not scale with map size or training data.The weight file remains constant at 50 MB, and dropout uncertainty adds no parametrisation.
- Uncertainty values decrease when test images have similar training examples, while difficult images show larger uncertainty and localization errors.
- 5.4ms is sufficient to compute pose from 40 Monte Carlo dropout samples on an NVIDIA Titan X.By comparison, 128 samples take 6ms and a single PoseNet evaluation takes 5ms per image.
- Averaging Monte Carlo dropout samples improves real-time relocalization accuracy for indoor and outdoor scenes without extra parametrisation.
- The trace of the posterior sample covariance provides a scalar model-uncertainty estimate for error prediction and landmark-presence detection.