PaperlayerView exact revision on arXiv

Source-linked AI summary

Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction

Tao Xie, Peishan Yang, Yudong Jin, Yingfeng Cai, Wei Yin, Weiqiang Ren, Qian Zhang, Wei Hua, Sida Peng, Xiaoyang Guo, Xiaowei Zhou

arXiv:2604.08542v1cs.CV

TL;DR

Long RGB sequences challenge 3D reconstruction because limited memory and weak global context undermine accuracy and consistency. Scal3R addresses this with neural global context stored in online-adapted lightweight sub-networks and shared through context aggregation. It achieves state-of-the-art pose and 3D reconstruction accuracy across large-scale evaluations while remaining scalable and efficient.

  • Problem

    Long-sequence 3D reconstruction methods struggle to retain global contextual cues and maintain accuracy and consistency across large-scale scenes.

  • Method

    Scal3R uses neural global context representation and aggregation with lightweight sub-networks adapted online through self-supervised objectives.

  • Results

    Scal3R achieves state-of-the-art pose estimation and 3D reconstruction performance across diverse large-scale datasets.

  • Takeaways & Limitations

    The framework supports globally consistent, scalable reconstruction of kilometer-scale scenes from RGB-only sequences.

Abstract

from arXiv · show

This paper addresses the task of large-scale 3D scene reconstruction from long video sequences. Recent feed-forward reconstruction models have shown promising results by directly regressing 3D geometry from RGB images without explicit 3D priors or geometric constraints. However, these methods often struggle to maintain reconstruction accuracy and consistency over long sequences due to limited memory capacity and the inability to effectively capture global contextual cues. In contrast, humans can naturally exploit the global understanding of the scene to inform local perception. Motivated by this, we propose a novel neural global context representation that efficiently compresses and retains long-range scene information, enabling the model to leverage extensive contextual cues for enhanced reconstruction accuracy and consistency. The context representation is realized through a set of lightweight neural sub-networks that are rapidly adapted during test time via self-supervised objectives, which substantially increases memory capacity without incurring significant computational overhead. The experiments on multiple large-scale benchmarks, including the KITTI Odometry~\cite{Geiger2012CVPR} and Oxford Spires~\cite{tao2025spires} datasets, demonstrate the effectiveness of our approach in handling ultra-large scenes, achieving leading pose accuracy and state-of-the-art 3D reconstruction accuracy while maintaining efficiency. Code is available at https://zju3dv.github.io/scal3r.

1. Introduction

Scal3R targets kilometer-scale 3D reconstruction from long RGB sequences, where existing approaches struggle with global consistency, long-range dependencies, and scalability. It introduces neural global context representation and aggregation to retain and share scene information across chunks, achieving strong large-scale reconstruction results.

  • Large-scale reconstruction must align thousands of viewpoints while handling varying depth and lighting and preserving global consistency with local detail.
  • Feed-forward models reduce computation, but quadratic attention limits ultra-long sequences and token compression can discard spatial cues and weaken global structure.
  • Scal3R compresses and stores long-term scene context through lightweight sub-networks adapted online with self-supervised objectives.
  • Its context aggregation mechanism shares global cues across the sequence, improving local accuracy and consistency while preserving scalability and efficiency.
  • Scal3R achieves state-of-the-art pose estimation and 3D reconstruction performance across diverse large-scale datasets, including Virtual KITTI, KITTI, and Oxford Spires.

2. Related Work

Related work spans classical geometric reconstruction, feed-forward neural models, and memory mechanisms. These approaches improve robustness or efficiency but retain limitations in scalability, optimization cost, or long-range context capacity.

  • Classical SfM and SLAM estimate poses and structure through matching, triangulation, and optimization, but struggle with weak correspondences, long trajectories, or complex scenes.
  • Feed-forward models directly regress geometry from RGB images, removing reliance on explicit priors or known cameras, but early two-view designs limit scalability to larger scenes.
  • Linear-attention memory models avoid quadratic attention but compress history into finite hidden states, limiting complex long-range dependency modeling.
  • Memory mechanisms also face practical challenges in controlling memory growth and computational overhead.

3. Preliminary

The preliminary section describes VGGT’s unified geometric prediction pipeline and Test-Time Training as a dynamic memory mechanism. TTT adapts fast weights during inference to enlarge context capacity beyond fixed-size hidden states.

  • VGGT maps an RGB sequence to camera parameters, depth maps, point clouds, and feature grids using a unified Transformer.
  • VGGT alternates frame-wise and global self-attention before dedicated heads predict geometric outputs and tracking features.
  • Standard recurrent memory updates a fixed-size hidden state from the current token and previous state, but its capacity limits information retention in long sequences.
  • Test-Time Training uses rapidly adaptable fast weights updated by self-supervised objectives during inference to store contextual dependencies.
  • Scal3R processes overlapping chunks in parallel, shares global context through its backbone, and aligns and fuses chunk-level poses and depth maps into one reconstruction.
  • By treating context as an unlabeled dataset, TTT enlarges context capacity beyond fixed-size vectors while retaining scalability.

4. Method

Scal3R combines overlapping-chunk processing with neural global context memory and synchronization to reconstruct kilometer-scale scenes while retaining sequence-wide information. Its test-time adaptation and chunk-level updates improve scalability, global consistency, and local reconstruction.

  • Model Overview: Scal3R processes overlapping image chunks in parallel across GPUs, while Global Context Synchronization communicates sequence-wide context across devices.Each chunk produces camera parameters, depth maps, and point clouds locally before synchronization enriches reconstruction with global observations.
  • Model Overview: The neural Global Context Memory module adds adaptive memory parameters to VGGT’s alternating-attention architecture for sequence-level context aggregation.A learnable gate balances the GCM output with the original tokens, and the resulting tokens feed dedicated 3D prediction heads.
  • Test-Time Training as Memory: GCM updates its adaptive memory units using all tokens in a chunk as one update unit, improving parallelism and GPU utilization for long-context test-time training.The self-supervised objective uses a token-wise learning rate and dot-product loss to store contextual information in the adaptive memory units.
  • Test-Time Training as Memory: Global Context Synchronization aggregates gradients across chunks and applies them to adaptive memory units on all GPUs, enriching local chunks with sequence-wide observations.The method uses all-reduce primitives to limit communication overhead during training and inference.
  • Evaluation: Table 1 evaluates camera pose and resources on Virtual KITTI, KITTI Odometry, and Oxford Spires using RRE, RTE, ATE, peak GPU memory, and inference time.Failed scenes receive the worst valid score, and methods requiring known camera intrinsics are marked separately.
  • Training: The method is trained end-to-end with VGGT on diverse indoor, outdoor, synthetic, and real-world datasets using camera, depth, and point-cloud supervision.Training uses 32 A800 GPUs for 60k iterations, with grouped synchronization producing effective sequence lengths from 1 to 32 chunks.

5. Experiment

Scal3R is evaluated on long-sequence pose and 3D reconstruction benchmarks, with ablations examining state capacity and global-context components. The experiments report strong accuracy, robustness, and practical efficiency across challenging scenes.

  • Pose Accuracy: Scal3R is evaluated for pose accuracy on Virtual KITTI, KITTI Odometry, and Oxford Spires using ATE, RRE, and RTE after Sim(3) alignment.Additional pose comparisons cover ScanNet++, TUM-RGBD, and Waymo.
  • Pose Accuracy: Scal3R consistently outperforms feed-forward and streaming baselines, while several alternatives suffer tracking failures or out-of-memory errors on long sequences.VGGT-Long is strong on KITTI Odometry but degrades on other datasets.
  • Resource Comparison: Scal3R remains practical on a single GPU with moderate memory consumption, unlike FastVGGT’s A800 requirement and long-context methods’ substantial memory growth.DPVO++ and CUT3R achieve higher throughput, while COLMAP is over 20× slower than Scal3R.
  • Geometry Accuracy: Scal3R produces more reliable large-scale outdoor reconstructions and more consistent local geometry indoors, with evaluation using Chamfer distance and F1 score.The geometry evaluation covers ETH3D, Virtual KITTI, and Oxford Spires.
  • Ablation Study: Increasing lightweight sub-network state size from 1M to 4M improves ATE, RTE, and RRE, indicating that larger state capacity helps preserve long-range context.The state-size comparison is reported in the left block of Table 3.
  • Ablation Study: Removing either cross-chunk synchronization or global context memory worsens ATE, with the larger drop without GCM indicating distinct roles for memory and propagation.The right ablation block is a complementary long-sequence setting and is not directly comparable with the state-size block.

6. Conclusion

Scal3R combines neural global context, online-adapted lightweight sub-networks, and context aggregation for long RGB sequences. Experiments demonstrate state-of-the-art pose estimation and 3D geometry accuracy.

  • Conclusion: Scal3R combines neural global context with online-adapted lightweight sub-networks and context aggregation to preserve long-range dependencies efficiently.The framework targets scalable 3D reconstruction from long RGB sequences.

Supplementary Material

The model uses a Global Context Memory module attached to selected transformer layers to support depth and point-cloud prediction. Its state size depends on the number of heads, head dimension, and scaling factor.

  • Overall Architecture: The Global Context Memory module is attached after the 4th, 11th, 17th, and 24th global attention layers, whose outputs feed depth and point-cloud decoders.The overall architecture is based on a large transformer.
  • Global Context Memory: The GCM forward pass updates adaptive memory units with keys and values, then uses the updated units to compute the output from queries.The module projects upstream features into multi-head queries, keys, and values.
  • State Size: The total GCM state size is nh × hd × hd × k = d^2/nh × k.Here nh is the number of heads, hd is the head dimension, d is the feature dimension, and k is the scaling factor.
  • Implementation Choices: The implementation sets nh to 1 to maximize memory capacity and k to 4 to balance capacity with computational efficiency.These choices determine the lightweight sub-network state size.

B.1. Dataset Details

The benchmarks cover long, large-scale sequences across synthetic and real-world outdoor, indoor, and driving environments, with varied weather, lighting, loop closures, and view sparsity.

  • The evaluation uses Virtual KITTI, KITTI Odometry, Oxford Spires, and ETH3D, spanning synthetic, urban driving, indoor, and outdoor scenes.The datasets include diverse weather and lighting conditions and long sequences.
  • KITTI Odometry contains 11 real-world urban-driving sequences spanning 0.39–5.07 km and 271–4,661 frames.
  • Oxford Spires contains six sequences spanning 280–773 meters, with challenging loop closures and extreme view sparsity across indoor and outdoor scenes.
  • Virtual KITTI comprises 50 synthetic outdoor street-scene sequences spanning 52–711 meters and diverse weather and lighting conditions.

B.2. Evaluation Details

Evaluation measures camera pose and 3D reconstruction after Sim(3) alignment, using established trajectory and point-cloud metrics across full-length sequences with fixed chunking settings.

  • Pose accuracy is evaluated with ATE, RRE, and RTE after Sim(3) alignment of predicted trajectories with ground truth.RRE is reported in degrees per 100 meters and RTE in meters per 100 meters.
  • 3D reconstruction is evaluated with Chamfer Distance and F1-score using aligned predicted and ground-truth point clouds.Chamfer Distance averages accuracy and completeness, while F1-score is computed from precision and recall at a distance threshold.
  • Camera trajectory comparisons assess whether Scal3R preserves global structure with lower drift than baselines that may lose tracking or diverge.
  • Evaluations use all frames per sequence, with chunk size 60 and overlap 30 for Scal3R and VGGT-Long.Pose metrics are directly evaluated on Virtual KITTI, KITTI Odometry, and Oxford Spires, while reconstruction metrics use dataset-specific thresholds.
  • Point-cloud comparisons assess large-scale reconstruction accuracy and local geometric consistency across outdoor and indoor scenes.

C. Additional Results

Additional long-sequence results show that Scal3R maintains camera trajectories with small drift and improves reconstruction accuracy and local geometric consistency over baselines.

  • Scal3R reconstructs extremely large-scale long sequences with small drift, while baselines frequently lose tracking or diverge significantly.
  • Additional reconstruction results show improvements over baselines in both large-scale accuracy and local geometric consistency.

C.1. Additional Benchmark Comparisons

Supplementary evaluations examine pose accuracy on denser video benchmarks, runtime scaling with sequence length, and representative failure cases involving appearance inconsistency and extreme view sparsity.

  • Additional Benchmark Comparisons: Additional pose evaluations cover ScanNet++, TUM-RGBD, and Waymo, whose denser video regimes provide stronger short-range overlap.These settings are used to test whether gains persist when streaming and video-based baselines are better matched to the evaluation regime.
  • Runtime Scaling: Runtime grows approximately linearly from 150 to 990 frames while throughput remains around 2.6–2.9 FPS and relative pose error stays within 0.07–0.08 m.
  • Failure Cases: Severe appearance inconsistency weakens cross-chunk correspondences and can produce inaccurate global alignment.Abrupt illumination or color shifts are representative examples.
  • Failure Cases: Extreme view sparsity can cause local predictions to fail when only tens of images cover scenes spanning hundreds of meters or kilometers.
Loading 2604.08542v1…