PaperlayerView exact revision on arXiv

Source-linked AI summary

Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization

Siyan Dong, Shuzhe Wang, Shaohui Liu, Lulu Cai, Qingnan Fan, Juho Kannala, Yanchao Yang

arXiv:2412.08376v2cs.CV

TL;DR

Visual localization must estimate a query image’s pose relative to posed database images, while existing methods struggle to combine accuracy and generalization. Reloc3r pairs a symmetric relative-pose network with motion averaging and large-scale training; it reports strong results across datasets, with a collinear-retrieval degeneracy limiting metric-scale recovery.

  • Problem

    Existing pose-regression methods struggle to generalize to new scenes or provide accurate pose estimates, while scene-coordinate approaches can require intensive supervision and face generalization limits.

    Source: §1 · p. 1

  • Method

    Reloc3r uses a symmetric relative-pose regression network and motion averaging, trained on around eight million diverse image pairs.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3

  • Results

    Reloc3r consistently outperforms RPR methods on unseen-dataset localization, with an average median error of 0.04m/1.02°; in multi-view settings it leads on RRA@15 (95.8%), RTA@15 (93.7%), and mAA@30 (82.9%).

    Source: §4.1 · p. 7, §4.2 · p. 8

  • Takeaways & Limitations

    Reloc3r provides accurate localization across diverse scenes without requiring scene-specific training, while retaining real-time inference speed.

    Source: §4.1 · p. 7, §4.2 · p. 8

    When the query and all retrieved database images are perfectly collinear, motion averaging cannot solve the metric scale.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Visual localization aims to determine the camera pose of a query image relative to a database of posed images. In recent years, deep neural networks that directly regress camera poses have gained popularity due to their fast inference capabilities. However, existing methods struggle to either generalize well to new scenes or provide accurate camera pose estimates. To address these issues, we present Reloc3r, a simple yet effective visual localization framework. It consists of an elegantly designed relative pose regression network, and a minimalist motion averaging module for absolute pose estimation. Trained on approximately eight million posed image pairs, Reloc3r achieves surprisingly good performance and generalization ability. We conduct extensive experiments on six public datasets, consistently demonstrating the effectiveness and efficiency of the proposed method. It provides high-quality camera pose estimates in real time and generalizes to novel scenes. Code: https://github.com/ffrivera0/reloc3r.

1. Introduction

Visual localization registers a query image to a database-defined world coordinate system, but existing approaches struggle to combine generalization, efficiency, and accuracy. Reloc3r addresses this with a symmetric relative-pose network and motion averaging, trained on around eight million image pairs.

1. Introduction

  • Visual localization estimates a query image’s 6-DoF camera pose in the world coordinate system defined by posed database images.The task supports applications including augmented reality and robot navigation.Source: §1 · p. 1
  • Existing approaches struggle to achieve novel-scene generalization, test-time efficiency, and camera-pose accuracy together.Structure-based methods can be accurate but inefficient; scene-coordinate methods often generalize poorly and require intensive supervision, while pose-regression methods face accuracy or scene-specificity limits.Source: §1 · pp. 1–2
  • Reloc3r combines a fully symmetric relative-pose regression network with minimalist motion averaging, trained on around eight million image pairs from diverse sources.The training data span object-centric, indoor, and outdoor scenes.Source: §1 · p. 2

2. Related Work

Prior localization approaches trade off accuracy, speed, scalability, or generalization. Reloc3r builds on pose regression and DUSt3R’s Transformer backbone, emphasizing large-scale diverse training and a fully symmetric design.

Structure-based visual localization

  • Structure-based methods solve poses from image or 3D-model correspondences, but inference, robust estimation, and supervision can limit real-time use and large-scale training.The paper chooses camera-pose regression as a simpler route to efficient large-scale training.Source: §2 · p. 2

Camera pose regression

  • APR methods regress world-coordinate poses quickly, but often underperform structure-based localization and rely on scene-specific training or dense viewpoints.View synthesis can improve APR accuracy but adds substantial training cost.Source: §2 · p. 2
  • RPR aims to generalize across scenes, yet leading methods lag APR accuracy and often lose accuracy on unseen scenes or require separate indoor and outdoor models.Reloc3r adopts DUSt3R’s Transformer backbone with a fully symmetric design and trains on a diverse mixture of object-centric, indoor, and outdoor datasets.Source: §2 · pp. 2–3

3. Method

Reloc3r retrieves database images for a query, predicts paired relative poses, and combines them into an absolute metric pose. Its symmetric Transformer network learns relative rotation and translation direction, while a parameter-free motion averaging module resolves rotation and scale.

3. Method

  • Top-K retrieved database images are paired with the query, and the network estimates relative poses that motion averaging combines into the query’s absolute metric pose.Database poses support absolute rotation estimates, while the relative translation scale remains uncertain until camera-center triangulation.Source: §3 · p. 3

3.1. Relative Camera Pose Regression

  • A shared-weight, symmetric ViT encodes image patches, exchanges token information through cross-attention, and regresses relative poses in both image orders.The regression head uses feed-forward layers and average pooling; rotation is represented in 9D and converted to a rotation matrix by SVD orthogonalization.Source: §3.1 · pp. 3–4, Figure 2 · p. 4
  • Training minimizes rotation and translation angular errors, learning translation direction rather than metric scale to avoid balancing dataset-specific scales.The design also removes image-order bias and reduces computation and storage through shared branch weights.Source: §3.1 · p. 4

3.2. Motion Averaging

  • Motion averaging uses median rotation estimates and a least-squares camera-center solution from translation directions to produce the absolute metric pose.The module avoids robust estimation and has no trainable parameters.Source: §3.2 · p. 4

4. Experiments

Reloc3r is evaluated on relative-pose and visual-localization benchmarks using a single model trained on about eight million image pairs. The experiments report strong results across unseen settings, while analyses examine design choices and a degenerate retrieval configuration.

4. Experiments

  • Reloc3r is trained once on about eight million image pairs from seven public datasets spanning object-centric, indoor, and outdoor scenes.The default model is used across evaluations, and localization retrieves the top 10 image pairs without spatial filtering or clustering.Source: §4 · p. 5, Table 1 · p. 5

4.1. Relative Camera Pose Estimation

  • Reloc3r outperforms pose-regression competitors across ScanNet1500, RealEstate10K, and ACID, achieves state-of-the-art results against non-regression methods, and runs at 42 ms per image pair.It exceeds DUSt3R by around 13% on ScanNet1500 AUC@20, scores 66.70 / 80.20 / 88.39% on RealEstate10K AUC@5/10/20, and reaches 70.34% on ACID AUC@20, which was unseen in training.Source: §4.1 · pp. 5, 7, Table 3 · p. 6
  • 95.8% RRA@15, 93.7% RTA@15, and 82.9% mAA@30 give Reloc3r the best reported Co3dv2 multi-view scores among competitors.The evaluation uses pairwise poses, although other methods generally use all 10 frames simultaneously.Source: Table 2 · p. 5, §4.1 · p. 7

4.2. Visual Localization

  • 0.04m/1.02° average median error on 7 Scenes lets Reloc3r outperform all RPR methods, including same-dataset-trained methods, while remaining comparable to APR without scene-specific training.On Cambridge Landmarks, it improves across evaluated scenes without retraining; in unseen conditions, average errors are 0.38m/0.52° across the last four scenes, about half those of ImageNet+NCM.Source: Table 4 · p. 6, Table 5 · p. 7, §4.2 · p. 8

4.3. Analyses

  • Higher image resolution improves accuracy but increases runtime, while asymmetric decoders and metric-pose prediction both underperform the default design.Several decoder layers develop patch-correspondence matching behavior, and Reloc3r’s cross-attention regions are reported as superior to Efficient LoFTR’s despite pose-only supervision.Source: §4.3 · p. 8, Figure 4 · p. 8
  • Perfect collinearity of the query and all retrieved database images makes metric scale unsolvable by motion averaging.Source: §4.3 · p. 8

5. Conclusion

Reloc3r combines relative pose regression with minimalist motion averaging, trained at scale to estimate accurate poses efficiently and generalize across datasets.

5. Conclusion

  • Reloc3r pairs a relative pose regression network with a minimalist motion averaging module for absolute pose estimation.Source: §5 · p. 8
  • Training on around eight million image pairs supports Reloc3r’s reported generalization across multiple datasets.Source: §5 · p. 8
  • Reloc3r is reported to combine high efficiency with accurate pose estimation while remaining parsimonious.Source: §5 · p. 8

A. Training Details

Training uses sampled image pairs, color augmentation, and resolution-specific settings; the shared symmetric design reduces training overhead.

A. Training Details

  • Each epoch samples 50K image pairs per dataset, with random color jittering applied during training.Source: A. Training Details · p. 9
  • Reloc3r-512 trains at up to 512-pixel resolution with randomly selected aspect ratios, while Reloc3r-224 uses fixed 224×224 images.Source: A. Training Details · p. 9
  • The symmetric ViT-Large encoder and ViT-Base decoder freeze the encoder and train only the decoder and pose head.Source: A. Training Details · p. 9
  • The symmetric design uses one image order, reducing training time, memory, and storage compared with DUSt3R’s two-order training.Source: A. Training Details · p. 9

B. Detailed Ablation Studies

The ablations support Reloc3r’s shared symmetric architecture, non-metric translation design, continuous rotation representation, and pretrained initialization.

B. Detailed Ablation Studies

  • The shared decoder and prediction head exploit relative-pose symmetry, and the asymmetric variant performs worse on ScanNet1500.The symmetric model has 0.43B parameters, approximately 28% fewer than the asymmetric variant, while achieving superior accuracy.Source: B. Detailed Ablation Studies · p. 9
  • Learning metric translation scales decreases relative-pose accuracy on ScanNet1500 and Cambridge Landmarks compared with Reloc3r’s non-metric design.The authors state that the non-metric design lets the network focus on camera orientation and movement direction.Source: B. Detailed Ablation Studies · p. 9
  • The 9D rotation representation uses a continuous mapping to SO(3) to avoid discontinuities and is reported effective in the ablation.The experiments train on ScanNet++ and test on ScanNet1500.Source: B. Detailed Ablation Studies · p. 9, Table 7 · p. 9
  • Alternative or absent pretrained initialization degrades pose accuracy, although random initialization still yields meaningful cross-attention interactions.The tested alternatives include MASt3R, CroCo, and DUSt3R encoder-only initialization.Source: B. Detailed Ablation Studies · pp. 9–10, Table 8 · p. 10

C. More Analyses

Further analyses examine focal-length failures, decoder attention, model scale and data diversity, and limitations in metric translation estimation.

C. More Analyses

  • When focal length changes by 3× to 4×, rotation estimates remain largely unaffected but translation becomes noticeably inaccurate.The authors relate this failure to scale-distance ambiguity in two-view geometry.Source: Figure 5 · p. 10
  • Decoder cross-attention builds patch correspondences even from random initialization, while DUSt3R initialization makes responses more accurate and concentrated.The visualizations compare initialization conditions on ScanNet1500 and Cambridge Landmarks.Source: C. More Analyses · p. 10, Figure 6 · p. 10, Figure 7 · p. 10
  • Reloc3r’s 0.43B-parameter network achieves real-time inference on consumer-grade NVIDIA 3090/4090 GPUs.The authors report that larger training sets consistently improve pose estimation accuracy; diverse data helps generalization, while domain-specific data improves accuracy within its domain.Source: C. More Analyses · p. 11, Table 10 · p. 11
  • Motion averaging cannot solve metric translation scale when all images are perfectly collinear, leaving this a primary limitation.Source: C. More Analyses · p. 11
  • The regression network does not use camera intrinsic information, which contributes to failures in predicting camera-center movement.The paper suggests embedding intrinsics or regressing the essential matrix as future directions.Source: C. More Analyses · p. 11

D. Additional Comparisons

Additional evaluations show Reloc3r trailing matching-based methods on MegaDepth1500, while outperforming FAR on popular benchmarks. Motion averaging improves metric pose results, though increasing the number of image pairs trades lower error for longer inference.

Relative pose estimation on MegaDepth1500

Visualizations

  • Figures 8 and 9 visualize relative and absolute pose estimates, respectively, with translation scales set to 1 meter.Figure 9 uses two database images per video, with their poses estimated by the pose regression network.Source: Figure 8 · p. 12, Figure 9 · p. 12

Comparison with FAR

  • FAR achieves AUC@20 scores of 28.19 on ScanNet1500, 37.67 on RE10K, and 44.98% on ACID, but underperforms Reloc3r on these popular datasets.FAR performs well on images with few overlaps.Source: D. Additional Comparisons · p. 12

Metric pose estimation

  • Top-2 motion averaging yields significantly better metric pose results than single pairs; using more top-K pairs reduces error but increases inference time.LazyLoc's rotation and translation averaging modules provide limited improvements across most scenes, except GreatCourt, which has extensive repetitive patterns and similar regions.Source: D. Additional Comparisons · p. 12
  • Like other pose-regression methods, Reloc3r remains less accurate than state-of-the-art feature-matching methods on large-scale scenes, while scene-coordinate methods are more accurate but require per-scene training and can take longer to infer.Reloc3r cannot use LazyLoc's post-optimization step because it does not produce matches.Source: D. Additional Comparisons · pp. 12–13

E. Details for the Compared Methods

The compared-method evaluations use dataset-specific resolutions and checkpoint choices, with some results taken from original publications. Multi-view evaluation samples image pairs from CO3Dv2 sequences and uses an eight-view setup for RayReg and RayDiffusion.

Relative pose estimation comparisons

  • For relative-pose benchmarks, input resolutions vary by method and dataset; Reloc3r preserves aspect ratio and caps maximum resolution at 512px.NoPoSplat resizes and center-crops to 256×256 before upscaling; DUSt3R and MASt3R use 512×512, while MegaDepth1500 resolutions follow each method's original setting.Source: E. Details for the Compared Methods · p. 13

Multi-view pose estimation comparisons

  • CO3Dv2 evaluation samples 10 images per test sequence to form 45 pairs, totaling 76,905 pairs; RayReg and RayDiffusion results use the paper's 8-view setup because 10 views did not yield reasonable results.Source: E. Details for the Compared Methods · p. 13

Absolute metric pose estimation comparisons

  • For absolute metric pose estimation, most results come from original publications; Map-free and ExReNet are evaluated in two versions, with Map-free checkpoints selected by dataset setting.ScanNet-trained Map-free checkpoints are used for 7 Scenes, while Map-free-dataset checkpoints are used for Cambridge.Source: E. Details for the Compared Methods · p. 13

Other methods

F. In-The-Wild Camera Pose Estimations

In-the-wild tests show Reloc3r estimating relative poses across diverse viewpoint changes and image types, while video localization uses two posed database images. Metric scale cannot be reliably recovered when database and query images are collinear.

Relative pose estimation

  • Reloc3r is tested on internet-collected and self-captured in-the-wild images and videos; Figure 8 reports relative-pose results, showing generalization across viewpoint changes and pose inference between paintings, sketches, and real images.Source: F. In-The-Wild Camera Pose Estimations · p. 13
  • Reloc3r generalizes across diverse viewpoint changes, including between paintings, sketches, and real images, and produces reasonable estimates for different people's faces.The authors attribute this generalization to large-scale training.Source: F. In-The-Wild Camera Pose Estimations · p. 13

Visual localization

  • Video localization uses two database images per video, with database poses estimated by the regression network; collinear database and query images make metric scale unreliable.Figure 9 visualizes the absolute pose estimates, with the translation between database images scaled to 1 meter.Source: F. In-The-Wild Camera Pose Estimations · p. 13
Loading 2412.08376v2…