PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Camera Re-Localization from RGB and RGB-D Images Using DSAC

Eric Brachmann, Carsten Rother

arXiv:2002.12324v4cs.CVcs.LG

TL;DR

Visual re-localization needs precise camera poses from limited sensing and environment information, especially where GPS and existing pipelines have practical constraints. DSAC* predicts dense scene coordinates and uses differentiable RANSAC for flexible RGB/RGB-D re-localization with optional 3D models. It achieves state-of-the-art accuracy across varied settings while reducing training, memory, and runtime costs.

  • Problem

    Existing re-localization methods impose different requirements or limitations: RGB-D needs depth-capable devices, feature matching needs SfM reconstruction, and RGB-only retrieval or pose regression has lower precision.

  • Method

    DSAC* uses a neural network to predict dense scene coordinates and differentiable robust pose optimization, supporting RGB or RGB-D inputs with optional 3D scene models.

  • Results

    DSAC* achieves state-of-the-art accuracy across indoor and outdoor datasets and settings, while training faster, using less memory, and running with low runtime.

  • Takeaways & Limitations

    The framework provides a flexible single-image re-localization baseline that can operate from RGB images and ground-truth poses alone while exploiting depth or scene models when available.

  • Takeaways & Limitations

    Hand-crafted sparse features fail in some motion-blurred or textureless scenarios, and SfM reconstructions tend to fail indoors with ambiguous repeating structures.

Abstract

from arXiv · show

We describe a learning-based system that estimates the camera position and orientation from a single input image relative to a known environment. The system is flexible w.r.t. the amount of information available at test and at training time, catering to different applications. Input images can be RGB-D or RGB, and a 3D model of the environment can be utilized for training but is not necessary. In the minimal case, our system requires only RGB images and ground truth poses at training time, and it requires only a single RGB image at test time. The framework consists of a deep neural network and fully differentiable pose optimization. The neural network predicts so called scene coordinates, i.e. dense correspondences between the input image and 3D scene space of the environment. The pose optimization implements robust fitting of pose parameters using differentiable RANSAC (DSAC) to facilitate end-to-end training. The system, an extension of DSAC++ and referred to as DSAC*, achieves state-of-the-art accuracy an various public datasets for RGB-based re-localization, and competitive accuracy for RGB-D-based re-localization.

1 INTRODUCTION

Visual camera re-localization seeks precise pose estimates where GPS is unreliable, while existing methods trade off input requirements, precision, and reconstruction effort. DSAC* addresses these trade-offs with a flexible scene-coordinate and differentiable-pose framework supporting RGB, RGB-D, and optional scene models.

  • GPS becomes unreliable indoors and between skyscrapers, motivating high-precision camera position and orientation estimates for autonomous driving and extended reality.
  • Existing systems trade off precision and requirements: RGB-D offers highest precision but needs depth-capable devices, while feature matching requires an SfM reconstruction.
  • Image retrieval and pose regression require only RGB images and poses but provide low precision comparable to GPS.
  • DSAC* can train from RGB images and ground-truth poses alone, use a single RGB image at test time, and automatically discover scene geometry.
  • The framework predicts dense scene coordinates with a neural network and fits camera pose through differentiable RANSAC, enabling end-to-end training.
  • RGB-D support achieves comparable accuracy to state-of-the-art indoor methods, while a unified initialization reduces training time from 6 days to 2.5 days on identical hardware.
  • 80.7% (DSAC*) versus 53.1% (DSAC++) for indoor re-localization when training without a 3D model demonstrates improved accuracy.
  • The improved ResNet-based architecture reduces memory usage by 75% and total inference time from 200ms to 75ms compared with DSAC++.

2 RELATED WORK

Related work spans retrieval, pose regression, sparse feature matching, and scene-coordinate regression, with differing accuracy, scalability, and sensor requirements. DSAC* extends differentiable robust estimation to unify RGB- and RGB-D-based re-localization while improving efficiency and precision.

  • Image retrieval compares global descriptors against posed database images, scales to large environments, but its sampling density limits accuracy.
  • Absolute pose regression learns image-to-pose mappings but does not consistently outperform image retrieval in practice.
  • Relative pose regression predicts transformations to retrieved images, with recent work reporting accuracy comparable to structure-based methods.
  • Sparse feature matching yields precise poses from 2D-3D correspondences, but SfM reconstruction can fail indoors with textureless or repeating structures.
  • Hand-crafted features struggle with motion blur and textureless areas, while learned sparse features have not yet consistently exceeded hand-crafted predecessors.
  • Feature-based systems and SfM tools lack direct depth-sensor integration at test time or during scene reconstruction.
  • Scene coordinate regression predicts dense pixel-to-3D correspondences that feed RANSAC-based pose optimization, replacing discrete feature detection with direct correspondence prediction.
  • DSAC makes RANSAC differentiable by sampling model parameters according to consensus, allowing optimization of expected task loss for end-to-end training.

3 FRAMEWORK

The framework predicts dense scene-coordinate correspondences from a single RGB or RGB-D image, then estimates camera pose robustly with RANSAC-based optimization. RGB inputs use 2D-3D PnP correspondences, while RGB-D inputs use depth-derived 3D-3D correspondences and Kabsch.

  • Scene Coordinate Regression: Scene coordinates provide dense correspondences between image pixels and points in the environment’s reference frame.They can be generated from a 3D model and ground-truth poses, depth maps, or ground-truth poses alone through multi-frame reprojection optimization.
  • Robust Pose Optimization: RANSAC-based pose optimization samples correspondence subsets, scores pose hypotheses, and refines the best hypothesis using its inliers.Refinement and inlier-set recalculation are iterated until convergence to obtain the final camera pose estimate.
  • Robust Pose Optimization: The system fixes the number of sampled hypotheses M and trains the network to adapt to this setting, making M a hyper-parameter controlling tolerance for inaccurate predictions.The standard RANSAC procedure can instead adapt M using an online estimate of the correspondence outlier ratio.
  • 3.1 Case RGB: For RGB images, pose estimation uses 2D-3D correspondences and PnP, with reprojection error as the residual for scoring hypotheses.PnP recovers pose from at least four correspondences; refinement uses nonlinear reprojection-error optimization with more than four inliers.
  • System Overview: The full pipeline combines a fully convolutional scene-coordinate regressor with differentiable pose estimation, including PnP or Kabsch inside robust RANSAC.The final pose is refined, and differentiability of all components supports end-to-end training.
  • 3.2 Case RGB-D: For RGB-D images, depth supplies camera-frame 3D points, creating dense 3D-3D correspondences solved with Kabsch.Three correspondences are used to sample hypotheses, while refinement uses more than three inliers and scores hypotheses by 3D Euclidean distance.

4 DEEP SCENE COORDINATE REGRESSION

The section develops scene-coordinate regression for RGB and RGB-D inputs, with training objectives adapted to the available scene information. DSAC* combines robust coordinate targets and reprojection-based supervision while supporting training with or without an explicit 3D model.

  • Initialization: End-to-end training from scratch is unstable, so auxiliary scene-coordinate losses initialize the network before full pipeline optimization.Without initialization, training converges to a local minimum that does not provide reasonable pose estimates.
  • RGB-D training: RGB-D initialization minimizes Euclidean distance to scene coordinates obtained by reprojecting depth and transforming points with the ground-truth pose.The plain Euclidean distance is preferred for robustness to outliers.
  • RGB + 3D Model: RGB initialization uses reprojection error because the PnP solver optimizes pose using reprojection rather than 3D Euclidean residuals.This aligns network initialization with the objective used during RGB pose estimation.
  • RGB + 3D Model: The RGB-plus-model objective can fail when optimized from scratch because reprojection error is ambiguous with respect to camera viewing direction.With a 3D model, the method first trains using the RGB-D objective, then switches to reprojection error after several iterations.
  • RGB + 3D Model: DSAC* replaces DSAC++’s two initialization stages with one per-pixel objective that switches from 3D distance to reprojection error for valid predictions.The formulation also accommodates missing ground-truth scene coordinates and supports dense scans, sparse SfM models, and incomplete depth.
  • RGB: When no 3D model is available, heuristic targets use ground-truth poses and 10m assumed depth before switching to reprojection supervision.This setting avoids requiring SfM preprocessing, which can fail indoors on textureless or repetitive surfaces.

5 DIFFERENTIABLE POSE OPTIMIZATION

The section makes pose optimization differentiable across RGB-D and RGB pipelines, allowing scene-coordinate predictions to be trained directly through robust pose estimation. It combines differentiable solvers, iterative refinement, and soft inlier selection within DSAC.

  • End-to-end pose training: Differentiable pose optimization lets the scene-coordinate network adjust predictions end-to-end according to their effect on estimated pose accuracy.This addresses the non-trivial relationship between coordinate quality and pose quality under RANSAC, where outliers may be discarded.
  • End-to-end pose training: The pose loss combines translation and rotation errors, with γ controlling their trade-off and set to 100 when rotation is measured in degrees and translation in centimeters.The loss is robustified using soft clamping.
  • Differentiating Kabsch: For RGB-D inputs, Kabsch estimates pose from 3D-3D correspondences by minimizing squared residuals and is differentiable through covariance computation and SVD.The resulting rotation and translation are recovered in closed form.
  • Differentiable PnP: For RGB inputs, PnP estimates pose from 2D-3D correspondences by minimizing squared reprojection errors.Because PnP lacks a closed-form solution here, the method uses analytical approximate gradients based on the objective and Gauss-Newton optimization.
  • Pose refinement: Iterative pose refinement repeatedly resolves the pose using current inliers, updates the inlier set, and approximates gradients by fixing the final refinement iteration.The solver may begin from an arbitrary non-differentiable PnP initialization.
  • Differentiable RANSAC: DSAC replaces hard inlier counting with a Sigmoid-based differentiable approximation, while probabilistically selecting hypotheses according to a softmax over scores.The computation graph represents deterministic operations with square frames and sampling operations with circular frames.

6 EXPERIMENTS

The experiments evaluate DSAC* across RGB, RGB + 3D model, and RGB-D settings on two indoor and one outdoor dataset, using different pose solvers and training information.

  • Experimental setup: The evaluation covers 7Scenes, 12Scenes, and Cambridge, with comparisons against pose-regression, relative-pose, retrieval-based, and feature-based methods.The Cambridge street scene is omitted because its reconstruction contains outlier poses and points, duplicated geometry, and diverging geometry.
  • Task variants: RGB-D uses depth during training and testing, renders initialization coordinates from scene scans, and applies a Kabsch solver for pose estimation.RGB + 3D model instead uses PnP hypothesis sampling and Levenberg–Marquardt refinement.
  • Task variants: RGB training uses only RGB images and ground-truth poses, while RGB + 3D model training additionally renders scene coordinates from sparse or dense geometry.The RGB setting initializes scene-coordinate regression with the heuristic objective of Eq. 13.
  • Training and evaluation: The system applies grayscale conversion, image resizing, geometric and photometric augmentation, scene-specific networks, and differentiable RANSAC-based pose optimization.RGB and RGB-D optimization use 10px and 10cm inlier thresholds, respectively, with 64 sampled hypotheses.

6.2 Results for Indoor Localization (7Scenes)

On 7Scenes, DSAC* improves RGB-only re-localization over DSAC++ and performs strongly when a 3D model is available, while also reducing training time in the latter setting.

  • RGB: +27.6%: DSAC* increases RGB-only accuracy compared with DSAC++ when training uses RGB images and ground-truth poses only.The authors attribute some, but not all, of the gain to training data augmentation.
  • RGB + 3D model: 85.2%: DSAC* achieves its highest reported 7Scenes accuracy when RGB images are trained with a 3D model.SCoCR achieves similar accuracy with a more complicated network architecture.
  • RGB + 3D model: DSAC* trains in 2.5 days versus 6 days for DSAC++ on identical hardware in the RGB + 3D model setting.Both methods benefit from having a 3D model available for rendering ground-truth scene coordinates.

6.3 Results for Indoor Localization (12Scenes)

On 12Scenes, DSAC* consistently outperforms DSAC++ and reaches state-of-the-art accuracy across settings, with average performance near 99% for several methods.

  • 12Scenes results: DSAC* achieves state-of-the-art accuracy in all 12Scenes settings and consistently outperforms DSAC++.The results are reported for indoor localization in Fig. 5.
  • Cambridge comparison: For Cambridge, DSAC* and DSAC++ achieve similar accuracy with a 3D model, while DSAC* trains significantly faster.NG-DSAC++ achieves the best results for many scenes, and the authors expect neural guidance to add little because results are already strong.
  • Cambridge comparison: Without a 3D model, DSAC* outperforms DSAC++ across all Cambridge scenes and exceeds every model-trained method on Great Court.The Great Court SfM reconstruction has a high outlier ratio that might hinder training.
  • Cambridge visualization: High localization error correlates with camera distance from the scene, particularly for Old Hospital and King’s College.The visualizations otherwise show high-quality re-localization suitable for augmented-reality applications.

6.5 Network Architecture and Runtime

Ablations separate the effects of DSAC*’s architecture, training schedule, and augmentation, while the figures visualize accuracy, estimated positions, and position errors across indoor settings.

  • Network architecture and runtime: The DSAC* architecture combined with DSAC* training achieves the best ablation accuracy, with faster runtime and a smaller memory footprint.Both the DSAC++ and DSAC* architectures benefit from the updated training schedule.
  • Visualizations and metrics: Fig. 5 reports the average percentage of frames below 5cm and 5° across 7Scenes and 12Scenes, grouped by available RGB, 3D-model, and depth information.Fig. 6 complements this with training and test camera positions, estimated positions colored by error, and per-setup percentages below the same threshold.
  • Impact of data augmentation: +9.1%, +7.7% and 4.1%: data augmentation improves DSAC* on 7Scenes depending on the task setup.The largest gain is +51.5% on the Stairs sequence in RGB-only mode.
  • Impact of data augmentation: +8.6%: augmentation improves RGB-only re-localization on 12Scenes, while other 12Scenes settings are saturated without it.No significant augmentation advantage is found on Cambridge, and DSAC* remains state of the art across settings without augmentation.

6.7 Impact of the Receptive Field

DSAC* uses an 81px receptive field, balancing image context against generalization to viewpoint changes. Smaller receptive fields slightly reduce accuracy, while larger fields are more sensitive to viewpoint changes.

  • Receptive Field Design: The DSAC* architecture uses an 81px receptive field for predicting each scene coordinate.The receptive field determines the image area used for each prediction and is analogous to an image patch in sparse feature matching.
  • Receptive Field Design: Reducing the receptive field to 49px slightly decreases accuracy.The study changes 3x3 and 1x1 convolutions to alter receptive-field size, also changing the number of learnable parameters.
  • Viewpoint Generalization: Larger receptive fields are more affected by viewpoint changes between training and test images.The accuracy cost of less image context from smaller receptive fields is counteracted by better viewpoint generalization.

6.8 Impact of End-to-End Training

End-to-end training primarily improves the precision of successful pose estimates at fine error thresholds. Its effect is small for coarse thresholds and does not necessarily reduce failure rates.

  • Accuracy by Threshold: End-to-end training increases precision at finer thresholds of 2cm/2° and 1cm/1°, while its effect is small at 5cm/5°.The comparison reports accuracy before and after end-to-end training on 7Scenes and 12Scenes.
  • Accuracy by Threshold: End-to-end training does not necessarily decrease the failure rate.The optimization improves successful pose estimates rather than uniformly increasing the fraction of successful estimates.
  • Learned Structure: The system learns to focus on reliable structures and generally improves scene-coordinate quality for close objects.Close objects are described as more helpful than distant structures for precise camera-pose estimation.

6.9 Learned 3D Geometry

DSAC* implicitly encodes dense scene geometry in its scene-coordinate network, including when trained only with RGB images and poses. The encoded representation also supports scene compression at different memory budgets.

  • Geometry Recovery: The network generates an explicit scene point cloud by predicting scene coordinates for training images and associating each prediction with image color.Poisson surface reconstruction is then used to obtain a mesh because the point cloud contains many outliers.
  • Geometry Recovery: DSAC* learns complex 3D scene geometry from only RGB images and ground truth poses.The method also recovers dense scene representations when trained with sparse 3D models on Cambridge scenes.
  • Scene Compression: With a 28MB memory footprint, DSAC* achieves the highest average re-localization accuracy in the reported compression comparison.The comparison uses Cambridge Landmarks scenes with varying sizes.
  • Scene Compression: DSAC* Tiny reduces the per-scene memory footprint to 4MB by clamping each layer to 128 channels.It is introduced to analyze how a leaner network affects scene-compression properties.

7 CONCLUSION

DSAC* is a versatile single-image re-localization pipeline based on scene-coordinate regression and differentiable RANSAC. It supports RGB-D and RGB inputs, optional scene models during training, and achieves strong accuracy across indoor and outdoor settings.

  • Conclusion: DSAC* derives gradients for all steps of robust pose estimation, including PnP solvers, enabling end-to-end training.The system is based on scene-coordinate regression and differentiable RANSAC.
  • Conclusion: The pipeline supports RGB-D and RGB re-localization and training with or without a 3D scene model.These configurations cover the system's different information settings.
  • Conclusion: DSAC* trains faster, uses less memory, and has low runtime while achieving state-of-the-art accuracy across various datasets and settings.The conclusion covers indoor and outdoor datasets and multiple training configurations.
  • Conclusion: The paper provides extensive visualizations of pose estimates and learned 3D geometry.The visual analyses include end-to-end training effects and geometry learned under different training setups.
Loading 2002.12324v4…