PaperlayerView exact revision on arXiv

Source-linked AI summary

Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction

Chin-Yang Lin, Yang-Che Sun, Cheng Sun, Fu-En Yang, Min-Hung Chen, Yen-Yu Lin, Wei-Chen Chiu, Yu-Lun Liu

arXiv:2609.04201v1cs.CV

TL;DR

Long-video online 3D reconstruction is limited by first-frame global pose regression and its out-of-distribution extrapolation, despite stable local geometry. Scal3R uses lightweight token-based multi-reference relative pose querying in a frozen backbone with online pose-graph optimization. It reports geometrically consistent reconstructions on sequences spanning hundreds of meters after 8 hours of single-GPU finetuning.

  • Problem

    First-frame global pose regression faces stability and scalability bottlenecks on long sequences because models must extrapolate beyond limited scene-scale training coverage.

  • Method

    Scal3R injects lightweight learnable pose tokens into a frozen backbone to query relative transformations from multiple reference frames, then aggregates them with online pose-graph optimization.

  • Results

    Scal3R produces geometrically consistent reconstructions on sequences spanning hundreds of meters and converges in 8 hours on a single GPU.

  • Takeaways & Limitations

    The framework enables accurate online reconstruction of long video streams using relative pose querying on a frozen backbone.

  • Takeaways & Limitations

    Performance remains bounded by the frozen backbone, while loop closure, keyframe selection, and loop detection depend on failure-prone or hand-set mechanisms.

Abstract

from arXiv · show

Online 3D reconstruction models perform poorly on long videos. This happens because regressing poses relative to a fixed first-frame anchor forces extrapolation far beyond the training distribution. Small drifts accumulate and amplify into significant geometric collapse. However, we observe that per-frame depth remains stable throughout this failure. The backbone's local geometry remains intact; only the global pose head breaks down. Motivated by this decoupling, we introduce Scal3R. This approach reformulates online reconstruction as multi-reference relative pose querying. We use lightweight learnable tokens, which make up about ~1% of the parameters, and inject them into a completely frozen backbone via asymmetric attention. This setup queries poses relative to multiple past keyframes. An online pose-graph optimization system with loop closure suppresses long-range drift. Scal3R reaches convergence in 8 hours on a single GPU. It reduces the average ATE by over 60% on KITTI compared to the online baseline. It also achieves state-of-the-art performance across Virtual KITTI, Sintel, TUM-Dynamic, ScanNet, and 7-Scenes. Project page: https://linjohnss.github.io/scal3r/

1 Introduction

Long-sequence online reconstruction fails because first-frame global pose regression extrapolates beyond training coverage, even though local geometry remains reliable. Scal3R addresses this by querying multiple relative poses on a frozen backbone and aggregating them online for global consistency.

  • Motivation: First-frame global pose regression works for short sequences but faces severe stability and scalability bottlenecks in long-range environments.Limited scene-scale coverage forces extrapolation to global coordinates outside the training distribution.
  • Motivation: Global pose errors diverge catastrophically out of distribution, while per-frame depth remains stable, isolating the failure to global pose regression.The finding indicates that the backbone’s local geometric representations remain intact.
  • Method: Scal3R replaces global regression with local relative transformations conditioned on reference viewpoints while keeping the backbone entirely frozen.This exploits local geometry in the viewpoint domain where the model is most reliable.
  • Method: Multi-reference querying and online pose-graph optimization aggregate local constraints into a globally consistent trajectory, with loop closure supporting long-range consistency.Relative poses are queried against multiple dynamically selected reference frames simultaneously.
  • Results: Scal3R produces geometrically consistent reconstructions on sequences spanning hundreds of meters.The reported result concerns long-range online reconstruction after integrating the proposed querying and optimization design.
  • Results: 8 hours on a single GPU is sufficient for convergence, using only 4-view training samples.The framework is presented as efficient for scalable online reconstruction.

2 Related Work

Prior work spans offline optimization, feed-forward and streaming reconstruction, and parameter-efficient adaptation of frozen models. Scal3R combines these directions for scalable relative pose estimation on unbounded video streams while preserving the reconstruction backbone.

  • Prior reconstruction systems: Offline Structure-from-Motion, Multi-View Stereo, and optimization-based methods jointly refine poses and scene representations but remain per-scene and offline.Feed-forward methods later improved efficiency by predicting geometry in a single forward pass.
  • Online reconstruction: Online methods process video incrementally through recurrent fusion, differentiable bundle adjustment, radiance-field optimization, persistent states, or causal Transformers.CUT3R and STream3R extend feed-forward reconstruction to streaming systems and SLAM integrations.
  • Online reconstruction: Streaming systems commonly regress poses relative to the first frame, while suppressing drift over kilometer-scale sequences remains an open challenge.Existing remedies trade off online capability against global consistency.
  • Efficient prompt tuning: Parameter-efficient adaptation uses adapters, prompt tokens, low-rank perturbations, and related modules to transfer frozen models with under 2% of parameters.This paradigm has also been applied to frozen 3D vision and reconstruction backbones.
  • Efficient prompt tuning: Scal3R applies frozen-backbone prompt tuning to relative pose estimation through asymmetric attention injection and multi-reference prompt queries.The design targets scalable pose estimation on unbounded video streams while preserving pointmap quality.

3 Method

Scal3R reformulates online 3D reconstruction as multi-reference relative pose querying over frozen backbones, then aggregates pairwise constraints with online pose-graph optimization to maintain global consistency.

  • Scal3R replaces unstable global pose regression with relative pose queries against maintained reference frames, avoiding long-horizon extrapolation in a unified world coordinate system.
  • The method preserves frozen backbone geometry while adding lightweight learnable pose tokens; the new parameters account for approximately 1% of the backbone.Historical camera tokens condition queries for selected past keyframes.
  • Each reference token is assembled by adding a projected reference camera token to a shared base query, allowing the active reference count to change at inference without retraining.The relative transformation is decoded from each pose token with a lightweight MLP head.
  • Pose tokens are injected asymmetrically as queries that attend to image tokens, while image tokens retain their original self-attention and key/value representations.This one-directional flow preserves the pretrained image representation space and pointmap reconstruction fidelity.
  • Online pose-graph optimization aggregates multi-reference relative poses as between-factors, using keyframe selection, robust weighting, and gap-dependent uncertainty to correct trajectory drift.The graph models camera poses as SE(3) variable nodes and uses incremental optimization for incoming frames.

4 Experiments

The experiments evaluate Scal3R with frozen online reconstruction backbones, randomized multi-reference training, and comparisons across offline, streaming, and SLAM-style systems. Results span outdoor, indoor, synthetic, and unseen-domain benchmarks.

  • Setup: Scal3R uses frozen 24-layer CUT3R and STream3R backbones with lightweight learnable relative pose query tokens.The backbones combine a DINOv2 encoder with a Transformer decoder and remain entirely frozen.
  • Setup: Training uses TartanAir with four views per sample, including one current frame and three reference frames, while randomly perturbing temporal intervals.This sampling strategy targets varying motion velocities and baseline lengths.
  • Baselines: Scal3R is compared with offline transformers, streaming models, and the incremental SLAM system MASt3R-SLAM using RGB-only, intrinsic-free inputs.Evaluation follows official default settings for the compared methods.
  • Results: Scal3R consistently outperforms online and offline baselines across KITTI, Virtual KITTI, Sintel, TUM-Dynamic, and ScanNet.On KITTI, average ATE is 69.7 versus 182.2 for TTT3R; on vKITTI, Scal3R (CUT3R) reaches 5.63, while other reported best ATE values are 0.168 on Sintel, 0.018 on TUM-Dynamic, and 0.049 on ScanNet.

3D Reconstruction.

Scal3R improves reconstruction and trajectory quality over streaming baselines on long sequences. Ablations show that multi-reference supervision, keyframe selection, PGO, more references, and loop closure all contribute to performance and efficiency.

  • 3D Reconstruction: Scal3R improves geometric consistency on 7-Scenes, with the STream3R variant achieving the best Normal Consistency across all reported sequences.Its NC mean is 0.579 and median is 0.622.
  • 3D Reconstruction: On Virtual KITTI sequences of 332 m and 113 m, Scal3R recovers scene geometry more closely than CUT3R and STream3R.The baselines produce distorted, collapsed, or substantially deviating point clouds.
  • Ablations: Removing reference supervision sharply degrades RPEtrans to 3.336, while full multi-reference training achieves the best ATE of 5.632.A single reference reduces ATE to 15.764, supporting denser reference supervision.
  • Ablations: Increasing inference-time references from 4 to 12 reduces ATE from 15.748 to 5.632, while removing keyframe selection leaves ATE at 38.258.The ablation also identifies PGO as an essential component of the full system.
  • Runtime: The frozen forward pass dominates runtime, while keyframe selection, PGO, and loop detection add only a small fraction.At K=12, the full pipeline runs at 14.4 FPS on CUT3R and 7.95 FPS on STream3R.
  • Ablations: Loop closure reduces average KITTI ATE from 143.45 to 75.01, a 48% improvement concentrated on loop-heavy sequences.Qualitative trajectories show recovery of globally consistent loop structure.
  • Robustness: Scal3R maintains the lowest per-frame ATE across the full KITTI evaluation range, unlike methods that fail at moderate sequence lengths.Trajectory visualizations likewise show faithful recovery of long-sequence loop structures.

5 Conclusion

Scal3R addresses long-sequence pose instability by replacing global pose regression with multi-reference relative pose querying on a frozen backbone. Lightweight tokens and an online pose-graph backend support accurate, globally consistent online reconstruction.

  • Conclusion: Scal3R reformulates camera localization as multi-reference relative pose querying on a frozen backbone.Lightweight tokens query relative poses, which an online pose-graph backend aggregates into a globally consistent trajectory.
  • Conclusion: The frozen forward pass dominates latency, while auxiliary online components add only a small runtime fraction.This supports the efficiency profile reported for the complete pipeline.
  • Conclusion: Scal3R maintains low error across the KITTI evaluation range and avoids the catastrophic divergence shown by competing methods.Loop closure qualitatively restores globally consistent trajectory structure after accumulated drift.
  • Conclusion: The method reaches convergence after 8 hours of training on a single GPU.The conclusion presents this as enabling accurate online reconstruction of long video streams.
  • Limitations: Performance is bounded by frozen-backbone failures under occlusion or textureless regions, while loop closure and keyframe selection depend on hand-set thresholds.The authors identify improving both limitations as future work.

Overview

The supplementary material adds implementation details, evaluation and optimization settings, qualitative and ablation results, profiling, scalability analysis, and additional experiments.

  • Supplementary Material: The supplement documents the inference pipeline, evaluation protocol, and pose-graph optimization parameters.These details appear in Section A.
  • Supplementary Material: Additional sections cover qualitative results, ablation visualizations, runtime and memory profiling, reference-count scalability, metric-scale pose estimation, attention variants, test-time training, and dynamic-scene robustness.These experiments span vKITTI, KITTI, and TUM-Dynamic.

A Implementation Details

Scal3R combines frozen-backbone relative pose querying with online keyframe management, loop closure, and pose-graph optimization. Its implementation uses scale-normalized training and Sim(3)-aligned evaluation, with an optional metric-scale setting for CUT3R.

  • Backbone and optimization: Scal3R builds on frozen CUT3R and STream3R backbones, training only lightweight pose-query components without modifying pretrained weights.The implementation uses PyTorch and GTSAM’s iSAM2 incremental solver for pose-graph optimization.
  • Online inference: At each timestep, the system selects references from a pose-token buffer and can reinject an archived loop candidate before frozen-backbone inference.Loop candidates are retrieved from DINOv2-SALAD descriptors indexed online with FAISS.
  • Relative pose querying: Pose tokens query image features through asymmetric attention and predict relative transformations to multiple references, which are added as factors to the pose graph.iSAM2 optimizes the graph and writes optimized poses back to the active buffer.
  • Keyframe management: Keyframes are selected using depth-normalized 3D overlap and confidence thresholds, while the buffer retains recent keyframes and non-keyframe entries.The first 5 frames initialize the system as keyframes, and the active keyframe count varies by dataset.
  • Scale handling: Training uses scale-normalized predicted and ground-truth point clouds, while evaluation applies Sim(3) alignment; CUT3R is additionally evaluated with fixed-scale SE(3) alignment.The scale-normalized setup addresses monocular scale ambiguity, whereas CUT3R retains absolute scale information from metric-scale supervision.

B.1 More Qualitative Results

Qualitative comparisons show that Scal3R preserves globally coherent reconstructions on long and dynamic sequences. Ablations indicate that keyframe selection, pose-graph optimization, and loop closure each contribute distinct aspects of trajectory and structural consistency.

  • More Qualitative Results: Scal3R produces compact, globally consistent vKITTI point clouds with well-aligned trajectories, unlike baselines showing drift, elongation, skew, or structural distortion.The comparison covers vKITTI sequences 18 and 20.
  • More Qualitative Results: On TUM-Dynamic, Scal3R yields cleaner reconstructions with fewer ghosting artifacts from moving pedestrians on both CUT3R and STream3R backbones.Baselines produce fragmented point clouds and erratic trajectories under dynamic occlusion.
  • Ablation Visualizations: Removing keyframe selection causes severe trajectory drift and distorted global structure because the reference pool lacks geometric diversity.The degradation appears on both vKITTI and KITTI.
  • Ablation Visualizations: Removing PGO preserves local smoothness but permits cumulative error to bend the overall trajectory, especially on curved vKITTI segments.The ablation isolates PGO’s role in controlling accumulated global error.
  • Ablation Visualizations: Disabling loop closure leaves visible gaps at revisited KITTI regions, whereas the full system closes loops and produces a globally consistent reconstruction.This comparison directly visualizes the effect of loop edges on revisited areas.

C.1 Runtime Analysis

Runtime and reference-count analyses show that Scal3R’s frozen-backbone pipeline remains efficient. The forward pass dominates latency, system components add limited overhead, and K=12 provides the reported accuracy-throughput balance on vKITTI.

  • Runtime Analysis: 89.8% (57.8 ms) without loop closure and 86.3% (59.9 ms) with it are consumed by the CUT3R model forward pass.Keyframe selection and PGO together add fewer than 7 ms.
  • Runtime Analysis: On STream3R, the forward pass remains the dominant cost and all system-level components together add less than 10% overhead.The latency profile closely mirrors the CUT3R configuration.
  • Reference-count scalability: Increasing the reference count K from 4 to 12 reduces vKITTI ATE from 15.75 to 5.63 by providing broader geometric coverage.The reported ATE minimum occurs at K=12.
  • Reference-count scalability: Beyond K=12, ATE rises to 12.16 at K=24, while latency grows from 63.23 ms to 70.00 ms and memory remains below 3.3 GB.The authors adopt K=12 as the default, balancing accuracy and throughput at approximately 15 FPS.

D.1 Metric-Scale Pose Estimation

Metric-scale evaluation, SLAM comparisons, and design ablations characterize Scal3R’s accuracy and cost trade-offs. The method performs competitively without camera intrinsics, while scale alignment and asymmetric attention are especially important outdoors.

  • Metric-Scale Pose Estimation: Without Sim(3) scale alignment, Scal3R ATE rises from 5.63 to 9.25 on vKITTI and from 69.73 to 88.66 on KITTI.Indoor degradation is more moderate, such as TUM increasing from 0.033 to 0.046.
  • Comparison with Classic SLAM Systems: Without camera intrinsics, Scal3R achieves KITTI average ATE of 69.73 with CUT3R and 70.83 with STream3R, outperforming calibration-free baselines.ORB-SLAM2 reaches 54.82 with calibration but fails catastrophically on sequence 01.
  • Comparison with LongStream: LongStream reaches lower KITTI ATE of 51.9 versus Scal3R’s 69.7 by retraining a 1.3B backbone, whereas Scal3R adapts a frozen backbone with approximately 1% trainable parameters.LongStream uses 32 A100 GPUs for more than three days; Scal3R converges in 8 hours on one GPU from 4-view TartanAir samples.
  • Attention-injection ablation: Asymmetric attention reduces outdoor ATE from 57.78 to 5.63 on vKITTI and from 197.45 to 69.73 on KITTI compared with symmetric injection.Indoor benchmark performance is comparatively similar between the two variants.

D.5 Compatibility with Zero-Shot Methods

Scal3R is compatible with zero-shot test-time training, which consistently improves its globally optimized pose estimates. On KITTI Seq. 09, this refinement lowers ATE from 29.6 to 20.6.

  • Compatibility with Zero-Shot Methods: 29.6 → 20.6 is the ATE reduction shown for KITTI Seq. 09 after adding TTT3R.The accompanying qualitative description says the estimated trajectory tightens against ground truth.
  • Compatibility with Zero-Shot Methods: TTT3R consistently improves ATE across all five benchmarks when applied on top of Scal3R.The combination is described as complementary to Scal3R’s globally optimized poses.
  • Compatibility with Zero-Shot Methods: 0.092 → 0.064 is the reported ScanNet ATE improvement from adding TTT3R to Scal3R.
  • Compatibility with Zero-Shot Methods: 69.73 → 62.05 is the reported KITTI ATE improvement from adding TTT3R to Scal3R.
  • Dynamic-Scene Robustness: On TUM-Dynamic, pose-query attention concentrates on static structures and down-weights moving pedestrians without explicit motion segmentation.This behavior is associated with accurate pose estimation despite large occlusions.
Loading 2609.04201v1…