PaperlayerView exact revision on arXiv

Source-linked AI summary

Accelerated Coordinate Encoding: Learning to Relocalize in Minutes using RGB and Poses

Eric Brachmann, Tommaso Cavallari, Victor Adrian Prisacariu

arXiv:2305.14059v1cs.CVcs.LG

TL;DR

Learning-based visual relocalizers offer strong pose accuracy but require hours or days of scene-specific training, making rapid deployment difficult. ACE combines a scene-agnostic backbone with an MLP scene-specific head and a reprojection-loss curriculum to accelerate optimization. It maps new environments in 5 minutes, with compact 4MB maps and only posed RGB images, while reporting comparable accuracy and substantially lower mapping costs.

  • Problem

    Scene-specific training for learning-based visual relocalizers can take hours or days, despite their strong pose accuracy.

  • Method

    ACE uses a scene-agnostic feature backbone, an MLP scene-specific prediction head, simultaneous optimization over many views, and a curriculum over reprojection loss.

  • Results

    ACE maps a new scene in 5 minutes, compiles it into 4MB of weights, and requires only posed RGB images for mapping.

  • Takeaways & Limitations

    ACE reduces mapping cost and energy consumption by two orders of magnitude compared with previous RGB-based scene coordinate regression approaches.

  • Takeaways & Limitations

    Reported mapping-time and map-size comparisons depend on hardware, implementations, and hyperparameters, although the authors expect the order of magnitude to be correct.

Abstract

from arXiv · show

Learning-based visual relocalizers exhibit leading pose accuracy, but require hours or days of training. Since training needs to happen on each new scene again, long training times make learning-based relocalization impractical for most applications, despite its promise of high accuracy. In this paper we show how such a system can actually achieve the same accuracy in less than 5 minutes. We start from the obvious: a relocalization network can be split in a scene-agnostic feature backbone, and a scene-specific prediction head. Less obvious: using an MLP prediction head allows us to optimize across thousands of view points simultaneously in each single training iteration. This leads to stable and extremely fast convergence. Furthermore, we substitute effective but slow end-to-end training using a robust pose solver with a curriculum over a reprojection loss. Our approach does not require privileged knowledge, such a depth maps or a 3D model, for speedy training. Overall, our approach is up to 300x faster in mapping than state-of-the-art scene coordinate regression, while keeping accuracy on par.

1. Introduction

ACE addresses the long mapping time of learning-based visual relocalizers by restructuring scene coordinate regression for rapid, stable training. It maps new scenes in minutes using only posed RGB images while retaining state-of-the-art accuracy and compact scene representations.

  • Motivation: Learning-based relocalizers require repeated, hours-long scene training despite strong accuracy, limiting their practicality for new environments.DSAC* requires 15 hours to reach top accuracy on a premium GPU.
  • Implications: Fast mapping reduces delay, computation cost, energy consumption, and reproducibility barriers for learning-based relocalization.The paper reports that all dataset scenes can be mapped almost five times over on a budget GPU in the time DSAC* needs for one scene on a premium GPU.
  • Method: ACE splits scene coordinate regression into a scene-agnostic feature backbone and a scene-specific prediction head.The approach treats regression as mapping high-dimensional features to 3D scene points.
  • Method: An MLP head optimizes thousands of mapping views simultaneously, producing decorrelated gradients and stable convergence at high learning rates.A reprojection-loss curriculum replaces slower end-to-end training through robust pose estimation.
  • Results: 5 minutes is the reported mapping time for ACE, compared with hours for previous state-of-the-art scene coordinate regression systems.The system targets state-of-the-art relocalization accuracy while reducing mapping time substantially.
  • Results: ACE compiles a scene into 4MB of network weights and requires only posed RGB images for mapping.Previous systems either used substantially more storage or relied on privileged information such as depth maps or scene meshes.

2. Related Work

Prior relocalization methods trade mapping speed, accuracy, storage, privacy, or input requirements in different ways. ACE targets the long training time of deep scene coordinate regression while retaining RGB-only operation and compact implicit maps.

  • Background: Visual relocalization maps represent the environment, and mapping is the process of creating those representations.The paper focuses primarily on mapping time and secondarily on map storage.
  • Relocalization families: Feature-matching relocalizers build explicit 3D reconstructions, while scene coordinate regression stores implicit scene representations in neural-network weights.Feature matching uses 3D points with local feature descriptors; coordinate regression predicts 3D scene coordinates for image pixels.
  • Scene coordinate regression: 15 hours is the reported DSAC* training time for a single scene, while DSAC++ previously required 6 days.These methods train a neural network separately for each scene.
  • Scene coordinate regression: ACE reports a two-orders-of-magnitude speedup over previous RGB-based scene coordinate regression approaches.The paper presents this speedup as making deep scene coordinate regression practical for applications.
  • Fast mapping with depth: Fast mapping methods can reach minutes when depth is available, but the cited approaches require depth for mapping, adaptation, relocalization, or mesh construction.The paper positions ACE as applicable when mapping images are RGB-only.

3. Method

ACE separates scene-agnostic feature extraction from scene-specific coordinate prediction, enabling batches that mix many mapping views. It combines gradient decorrelation with a reprojection-loss curriculum to accelerate stable training.

  • The relocalization pipeline estimates camera pose from image-to-scene correspondences using a robust pose solver.The network predicts 3D scene coordinates for image locations, which are combined with pixels to estimate the pose.
  • Standard training predicts dense scene coordinates from one mapping image at a time and supervises them with ground-truth pose and reprojection loss.
  • 3.1. Efficient Training by Gradient Decorrelation: ACE randomizes features across mapping views, decorrelating gradients and producing stable training signals that tolerate high learning rates.Standard batches contain spatially correlated predictions, whereas ACE mixes features from the entire training set.
  • 3.1. Efficient Training by Gradient Decorrelation: ACE splits the regression network into a scene-agnostic convolutional backbone and a scene-specific MLP head that maps features to 3D scene coordinates.The MLP head needs no spatial context, allowing random samples from all mapping images to form training batches.
  • 3.1. Efficient Training by Gradient Decorrelation: A feature buffer is generated once, then shuffled so each training step can update the MLP using several thousand features from potentially thousands of mapping views.The buffer stores features with pixel positions, camera intrinsics, and ground-truth mapping poses.
  • 3.2. Curriculum Training: ACE replaces costly end-to-end pose-solver training with a curriculum over reprojection loss that becomes more restrictive as training progresses.The curriculum uses a moving inlier threshold, while tanh-clamped reprojection error dynamically rescales with threshold τ.

4. Experiments

Experiments evaluate ACE across indoor and outdoor relocalization datasets, mapping hardware, map sizes, and qualitative comparisons. ACE achieves high accuracy with fast, compact mapping, while comparisons involving reported mapping times require caution because hardware and estimation choices differ.

  • Indoor Relocalization: ACE achieves top accuracy in less than 10 minutes per scene, uses less than 10MB per scene, and does not require depth for mapping.The comparison includes feature matching and scene coordinate regression methods across 7Scenes and 12Scenes.
  • Experimental Caveat: Mapping-time and map-size comparisons depend on hardware, implementation details, and hyperparameters, although the authors report confidence in the order of magnitude.DSAC* timings were re-run on the authors’ hardware, while other values came from papers or estimates based on public statistics.
  • Outdoor Relocalization: Poker outperforms DSAC* on average on Cambridge Landmarks while remaining comparatively lean in mapping time and storage.Poker splits a scene into four ACE models and selects the pose estimate with the largest inlier count.
  • Outdoor Relocalization: ACE outperforms DSAC* on average on Wayspots while being two orders of magnitude faster in mapping.Wayspots uses phone-scan trajectories for mapping and SfM poses only for evaluation.
  • Mapping Time: ACE experiences only a 10% slowdown on a T4 GPU, whereas DSAC* mapping takes twice as long on the cheaper GPU.The comparison uses an NVIDIA V100 and an NVIDIA T4; the T4 has less compute capability.
  • Map Size: 2.5MB ACE maps achieve relocalization rates above 95% on 7Scenes, while 5.5MB maps do not pay off because they complete fewer training-buffer passes.The standard ACE configuration uses 4MB maps, and ACE reaches approximately 80% accuracy after one epoch of 75 seconds.

5. Conclusion and Future Work

ACE maps new environments in 5 minutes and reduces mapping cost and energy consumption by two orders of magnitude. The paper identifies further speedups through engineering improvements such as parallel buffer creation and training.

  • ACE maps new environments in 5 minutes, reducing mapping cost and energy consumption by two orders of magnitude compared with previous RGB-based scene coordinate regression.
  • The proposed changes are mainly conceptual and leverage gradient decorrelation from patch-level training.
  • Further speedups may come from interleaving buffer creation with training or stopping early on easy scenes.

A.1. Backbone Training

The backbone uses the first 10 layers of the DSAC* design, processes grayscale images, and reduces spatial resolution while increasing channel capacity. Backbone training uses strong augmentation and parallel scene-specific regression heads.

  • The backbone takes grayscale images as input and reduces spatial resolution to 1/8 while increasing channel dimensions to 512.
  • The backbone is trained on 100 scenes in parallel with 100 six-layer MLP regression heads of width 512.
  • Training applies brightness, contrast, saturation, hue, scale, and homography augmentations to the mapping images.
  • Gradient computation processes 10 heads at a time, while gradients from all 100 heads are accumulated before each parameter update.

A.2. ACE Head Training

The ACE head predicts scene coordinates with a homogeneous representation whose scale is constrained before de-homogenization. Predictions are centered relative to the mean camera translation and shifted back before PnP-RANSAC.

  • The scene-coordinate head uses eight 1x1 convolutional layers, skip connections after layers 3 and 6, and a final layer producing scene coordinates.
  • A biased, clipped Softplus transforms the predicted scale to keep homogeneous coordinates valid for the subsequent RANSAC step.
  • The scale factor is clipped between 0.01 and 4.0, while the parameter β makes a zero predicted scale map to a neutral homogeneous parameter of 1.
  • The network learns coordinates relative to the mean camera translation, then adds that mean back before passing predictions to PnP-RANSAC.

A.2.2 Training Details

ACE accelerates head training by prefilling a large patch-feature buffer and optimizing the regression head over repeated epochs with a scheduled reprojection loss. An ensemble variant divides large Cambridge scenes among multiple independently trained heads.

  • A.2.2 Training Details: A buffer of patch features is prefetched in approximately 1 minute, leaving only regression-head optimization for ACE map training.
  • A.2.2 Training Details: The buffer stores 8 million 512-channel descriptors with source-image locations, camera poses, and camera intrinsics.
  • A.2.2 Training Details: A 5-minute mapping run, including buffer filling, performs 16 training epochs over the shuffled buffer.
  • A.2.2 Training Details: The training loss uses tanh-based reprojection errors with a circular schedule that decreases its threshold over training.
  • A.2.3 Choice of Loss Function: The decreasing threshold improves reliable-coordinate prediction over DSAC*’s piecewise loss, while homogeneous representations particularly help difficult scenes.
  • A.3. Ensemble Variant: The Poker ensemble uses four independently trained heads for larger Cambridge environments, motivated by splitting landmarks into spatial sub-regions.

A.3.1 Ensemble Training

ACE trains multiple scene-specific regression heads on spatially clustered subsets of posed RGB frames. The ensemble independently maps each subset and selects the pose with the strongest geometric support during localization.

  • Ensemble construction: ACE clusters posed RGB mapping images by camera translation into N disjoint training sets.Hierarchical clustering repeatedly splits the largest cluster with k-means until N clusters remain.
  • Ensemble construction: Each cluster receives an independently trained ACE regression head, so mapping time and map size scale linearly with ensemble size.Training can be parallelized across multiple GPUs to reduce elapsed mapping time.
  • Ensemble localization: During localization, ACE runs the standard pipeline for every map and selects the pose with the most RANSAC inliers after LM refinement.The ensemble therefore evaluates competing scene-specific predictions before choosing the final camera pose.
  • Ensemble localization: Because the backbone is scene-agnostic, each query image is feature-extracted once before passing through all ACE heads and pose solvers.This avoids repeating backbone computation for every ensemble member.

A.3.3 Ensemble Results

Increasing the ACE ensemble size initially improves Cambridge Landmarks accuracy, but gains saturate beyond four regressors. The four-map Poker variant is presented because it outperforms DSAC* within a fraction of the mapping time, while the simple clustering design remains an extension rather than the paper’s focus.

  • Ensemble-size evaluation: Accuracy initially improves as more ACE heads are added, then saturates beyond an ensemble of 4 regressors.The trend suggests that limiting each head’s spatial training extent benefits aggressive training, although the Shop scene gains little from further clustering.
  • Ensemble-size evaluation: The four-map Poker ensemble outperforms DSAC* within a fraction of the mapping time.This comparison motivates using four maps in the main text.
  • Scope of the extension: The clustering and localization design is intentionally simple and could be improved by tailoring ensemble size to each scene’s covered area.The authors describe this ensemble treatment as an extension rather than the work’s central focus.

B.1.1 Active Search [51,52]

This appendix documents mapping-time and map-size estimates for feature-matching and related baselines, alongside assumptions about reconstruction, descriptor storage, and training costs. It also notes dataset and preprocessing caveats affecting comparisons.

  • Mapping time: 7Scenes reconstruction time ranges from 6 minutes for Heads to 10 hours for RedKitchen, averaging 3.3 hours per scene.The reported average over 7Scenes and 12Scenes is 1.5 hours per scene.
  • Dataset assumptions: Cambridge Landmarks reconstruction time is assumed to be 35 minutes because public reconstruction-time records were unavailable.The assumption is based on comparable mapping-frame counts to 12Scenes, and training-only SfM quality remains uncertain.
  • Comparison caveat: Several reported mapping times exclude scene-specific preprocessing performed after SfM reconstruction.Examples include building acceleration data structures for faster matching.
  • Map size: Active Search’s average descriptor storage demand is estimated at 200MB per scene for 7Scenes and 12Scenes.The estimate uses observed feature statistics and 128-byte SIFT descriptors.
  • Comparison estimates: The appendix estimates 1GB average maps for D.VLAD+R2D2 and 2GB average maps for hLoc across 7Scenes and 12Scenes.Both estimates reuse reconstruction statistics because public descriptor or reconstruction statistics were unavailable.
  • Dataset caveat: MapFree contains only scenes where COLMAP triangulation and registration succeeded, potentially favoring sparse feature-based methods.The paper therefore compares scene coordinate regression approaches on this dataset, where such bias is not expected.

B.2.3 Phone-to-SfM Pose Alignment

Phone trajectories are aligned to bundle-adjusted SfM poses for Wayspots evaluation, but their mismatch introduces measurable uncertainty. ACE results improve only moderately when SfM poses replace phone poses, while gradient decorrelation remains important for stable training.

  • Pose registration: Phone-to-SfM alignment uses Kabsch registration within RANSAC, sampling 1000 triplets and accepting hypotheses with a 10cm inlier threshold.The winning hypothesis is refit using all inlier correspondences.
  • Alignment quality: Median alignment error stays below 10cm for every scene, making at least 50% relocalization achievable under the 10cm/5° threshold.Some scenes nevertheless show noticeable drift between phone-tracked and SfM poses.
  • Impact on evaluation: Using SfM poses for ACE mapping improves results only moderately on average, and difficult scenes remain difficult.The paper specifically identifies Winter Sign and Statue as persistent challenging scenes.
  • Training stability: Image-wise gradient correlation deteriorates ACE accuracy across datasets, with some Cambridge scenes diverging completely.The correlated-gradient variant still exceeds the 5-minute DSAC* version, partly because ACE uses strong backbone features and more iterations.
  • Backbone choice: A ScanNet-trained backbone achieves the best results among the evaluated ACE backbone choices.The authors note that scene-coordinate regression can benefit from higher-dimensional features with little computation or memory impact.
Loading 2305.14059v1…