PaperlayerView exact revision on arXiv

Source-linked AI summary

TTT3R: 3D Reconstruction as Test-Time Training

Xingyu Chen, Yue Chen, Yuliang Xiu, Andreas Geiger, Anpei Chen

arXiv:2509.26645v4cs.CV

TL;DR

3D reconstruction models have limited length generalization because recurrent states forget information beyond their training context, while attention-based alternatives impose high long-context costs. TTT3R reframes state updates as test-time online learning and uses alignment confidence to derive adaptive memory updates. It improves long-sequence pose estimation while preserving the baseline’s inference efficiency, although it does not match strong offline methods in reconstruction accuracy.

  • Problem

    Existing 3D reconstruction models face quadratic attention costs or recurrent-state forgetting, and methods trained on short contexts still struggle with sequences comprising hundreds of images.

  • Method

    TTT3R treats the recurrent state as a test-time-trained fast weight and uses memory-observation alignment confidence to derive a closed-form, per-token learning-rate state update.

  • Results

    TTT3R achieves robust long-sequence reconstruction and accurate pose estimation with a 2× improvement while preserving CUT3R’s inference speed and memory efficiency.

  • Takeaways & Limitations

    The training-free forward-pass modification provides a lightweight, plug-and-play way to enhance recurrent 3D reconstruction length generalization without fine-tuning or additional parameters.

  • Takeaways & Limitations

    TTT3R mitigates but does not resolve state forgetting and has not matched strong offline methods such as VGGT in reconstruction accuracy.

Abstract

from arXiv · show

Modern Recurrent Neural Networks have become a competitive architecture for 3D reconstruction due to their linear-time complexity. However, their performance degrades significantly when applied beyond the training context length, revealing limited length generalization. In this work, we revisit the 3D reconstruction foundation models from a Test-Time Training perspective, framing their designs as an online learning problem. Building on this perspective, we leverage the alignment confidence between the memory state and incoming observations to derive a closed-form learning rate for memory updates, to balance between retaining historical information and adapting to new observations. This training-free intervention, termed TTT3R, substantially improves length generalization, achieving a $2\times$ improvement in global pose estimation over baselines, while operating at 20 FPS with just 6 GB of GPU memory to process thousands of images. Code is available in https://rover-xingyu.github.io/TTT3R

1 INTRODUCTION

3D reconstruction models must handle long image sequences efficiently, but existing approaches face quadratic resource costs or forgetting and poor length generalization. TTT3R reframes recurrent state updates as test-time training and improves long-sequence reconstruction without additional inference cost.

  • Motivation: Quadratic Transformer attention limits the scalability of 3D reconstruction models as sequence length grows.Engineering optimizations do not change the underlying softmax-attention limitation for long contexts.
  • Motivation: CUT3R provides constant memory usage through an RNN-based design but fails to generalize from mostly 64-frame training sequences to much longer inputs.Existing methods still struggle with sequences containing hundreds of images.
  • Approach: The paper interprets recurrent state updating as online learning, with historical information compressed into a state viewed as a test-time-trained fast weight.This perspective connects length-generalization failure to state overfitting and forgetting.
  • Approach: TTT3R derives a closed-form inference-time state transition that uses internal confidence signals to suppress low-quality updates and mitigate catastrophic forgetting.The intervention requires neither fine-tuning nor additional parameters.
  • Results: TTT3R remains competitive on short-sequence benchmarks and shows significant long-sequence improvements without additional computational cost over the baseline.The proposed update rule is training-free and specifically targets length generalization.

2 RELATED WORK

3D reconstruction research spans offline foundation models, online memory-based systems, and efficient recurrent sequence models. These approaches trade computational efficiency and memory usage against long-context retention, motivating test-time-training interpretations of recurrent updates.

  • 3D Reconstruction: SfM and SLAM reconstruct 3D structure and camera poses through correspondence or photometric-error optimization followed by bundle adjustment.These classical systems can be highly effective when assembled comprehensively.
  • Offline Reconstruction Foundation Model: DUSt3R introduced an end-to-end foundation-model formulation that predicts pixel-aligned pointmaps from image pairs using Transformer-based architecture and direct point supervision.The formulation integrates image matching and pose estimation into reconstruction.
  • Online Reconstruction Foundation Model: Online methods maintain historical information incrementally, but full-attention systems grow in computation and memory while recurrent systems risk forgetting earlier frames.CUT3R uses a constant-sized state, whereas StreamVGGT caches historical keys and values.
  • Modern RNN: Modern recurrent layers offer linear-time processing by compressing context into finite-sized states, but equal compression can degrade performance as sequence length increases.Forgetting gates and related mechanisms attenuate previous values before storing new memory.
  • Modern RNN: Test-time-training formulations view recurrent state updates as online learning that balances retaining historical memory with adapting to new information.The resulting in-context states are fast weights functioning as associative memory, while slow weights act as meta-learners during training.

3 METHOD

The method reformulates online 3D reconstruction as sequence modeling with a fixed-length recurrent state, then interprets state updates through test-time training. TTT3R uses alignment confidence as adaptive per-token learning rates to balance historical retention and new observations without fine-tuning.

  • Sequence Modeling: Online reconstruction tokenizes each incoming image, updates a scene state, reads output tokens, and detokenizes them into pixel-aligned 3D pointmaps.The state encodes scene information with constant length, while the output tokens are converted into dense pointmaps.
  • Sequence Modeling: Full-attention methods grow state by appending key-value pairs, incurring O(t^2) computation or O(t) memory as sequence length increases.Causal attention reduces computation to O(t) for streaming inputs but retains a redundantly growing key-value state.
  • Sequence Modeling: RNN-based methods use one-to-one cross-attention with a fixed-length state, reducing computation and inference memory to O(1) but suffering forgetting as views accumulate.The recurrent state interacts with each incoming frame and remains constant in length.
  • Test-Time Training Perspective: Test-time training represents the recurrent state as a fast weight updated by gradient descent, while frozen slow weights predict the gradient and learning rate.This online learning process encodes current-observation key-value information into fixed-length memory.
  • Test-Time Training Perspective: CUT3R’s normalized softmax update prioritizes new observations over historical state, causing catastrophic forgetting and exposing a mismatch with standard test-time training.Because softmax weights sum to 1 across observation tokens, the update lacks a mechanism to balance retention and adaptation.
  • Confidence-Guided State Update: TTT3R uses memory-observation alignment confidence as an adaptive per-token learning rate, yielding a training-free, plug-and-play state-update intervention for CUT3R.The intervention is designed to selectively control memory plasticity without additional fine-tuning.

4 EXPERIMENTS

TTT3R is evaluated on camera pose estimation, video depth estimation, and 3D reconstruction against online and offline baselines. It improves long-sequence performance while retaining online inference efficiency and low memory usage.

  • Evaluation setup: TTT3R is evaluated on camera pose estimation, video depth estimation, and 3D reconstruction using established datasets and metrics.Camera pose uses ATE on TUM dynamics and ScanNet; video depth uses Abs Rel and δ < 1.25 on KITTI and Bonn; reconstruction uses Chamfer Distance and Normal Consistency on 7-scene.
  • Camera pose estimation: 2× accuracy improvement over CUT3R is achieved for camera pose estimation while retaining real-time efficiency and CUT3R's memory efficiency.Full-attention baselines are slower and memory-intensive, while Point3R runs out of memory beyond 700 frames.
  • Video depth estimation: Best overall video depth performance is achieved without fine-tuning, while Point3R degrades on longer sequences and metric-scale prediction.Full-attention methods run out of memory after about 150 frames; Point3R is strong on short scale-invariant sequences but degrades at longer lengths.
  • 3D reconstruction: TTT3R significantly outperforms online reconstruction methods and achieves accuracy comparable to VGGT while operating online with only 6GB GPU memory.CUT3R exhibits catastrophic forgetting with drifted poses, broken geometry, distortions, and ghosting artifacts.
  • Qualitative analysis: Qualitative results show improved sequence-length generalization, reduced forgetting, and online loop closure compared with CUT3R.VGGT and Point3R are omitted from the qualitative long-sequence comparison because they run out of memory.

5 DISCUSSION

The discussion presents TTT3R as a lightweight forward-pass modification that improves long-sequence reconstruction without fine-tuning. It also identifies remaining limitations in state forgetting, offline accuracy, and the design space of recurrent architectures.

  • Discussion: TTT3R improves length generalization through a lightweight, plug-and-play update performed during the forward pass without model fine-tuning.The method is described as a modification to CUT3R that supports robust long-sequence reconstruction.
  • Limitations: TTT3R mitigates but does not resolve state forgetting and has not matched strong offline methods such as VGGT in reconstruction accuracy.Full attention preserves the entire history context but is slower and more memory-demanding.
  • Limitations: Periodic state resets are explored as an optional variant to prevent state overfitting while retaining CUT3R's inference speed and memory efficiency.Reset chunks are aligned using global metric poses without additional optimization.
  • Future work: The design space for more effective, stable, and parallelizable recurrent architectures remains largely unexplored.The authors identify this as an opportunity for future work on reconstruction accuracy and length generalization.

A MORE EXPERIMENTAL ANALYSIS

The experimental analysis examines TTT3R against learnable gating mechanisms and studies the effects of fine-tuning and confidence-guided state updates.

  • A More Experimental Analysis: The analysis compares TTT3R with learnable gating mechanisms and investigates TTT3R fine-tuning and confidence-guided learning-rate updates.These studies are designed to validate TTT3R's state-update choices.

A.1 COMPARISON WITH STANDARD LEARNABLE GATING MECHANISMS

TTT3R uses a confidence-guided, conditioned per-token learning rate derived in closed form rather than learned through additional gating parameters. Experiments show it outperforms standard learnable gating mechanisms, while fine-tuning provides limited or mixed benefits on short training sequences.

  • Gating mechanisms: Standard gating mechanisms model learning rates as scalar, input-conditioned scalar, or input-conditioned per-token functions.ScalarLR uses one learnable scalar, ConditionLR conditions a scalar on the observation, and TokenLR assigns rates per state token.
  • TTT3R learning rate: TTT3R derives a conditioned per-token learning rate directly from state-observation alignment confidence without additional parameters or training overhead.The formulation follows the TokenLR paradigm but uses a training-free closed-form rule.
  • Comparison results: Longer training sequences improve learnable gating performance, but training beyond 64 frames is prohibitively expensive.CUT3R + TokenLR is the strongest learnable gating variant yet still significantly underperforms TTT3R.
  • Comparison results: TTT3R achieves state-of-the-art performance across the compared learnable gating mechanisms for camera pose and video depth estimation.The comparison evaluates camera pose on 1000-frame TUM-dynamic sequences and video depth on 500-frame KITTI sequences.
  • Fine-tuning analysis: Fine-tuning improves camera pose estimation but degrades video depth estimation when training uses short 4–64-view sequences.The authors associate this trade-off with prioritizing global alignment over per-view prediction accuracy.
  • Fine-tuning analysis: TTT3R provides only minimal gains over CUT3R on short sequences, motivating longer training sequences as a possible route to further improvement.The update rule helps pose estimation during training but offers no significant benefit when training sequences remain short.

A.3 TTT-DERIVED UPDATE RULE VS. NON-TTT BASELINES

TTT3R is compared with periodic reset, EMA shrinkage, and burn-in state-update baselines. It outperforms these non-TTT methods, while TTT3R + Reset performs best overall.

  • Baseline mechanisms: CUT3R + Reset periodically restores the state to its initial value every n frames and globally aligns the resulting chunks.The ablation tests n ∈ {50, 100, 150}; n = 100 is selected for later experiments.
  • Baseline mechanisms: CUT3R + EMA shrinks the recurrent state toward the initial state S0 during inference.The update is St = (1 − α)St−1 + αS0, with α = 0.001 selected after ablation.
  • Baseline mechanisms: CUT3R + BurnIn updates the state only at keyframes and leaves it unchanged between them.An interval of n = 100 frames is selected after comparing 50, 100, and 150 frames.
  • Comparison with non-TTT baselines: TTT3R significantly outperforms Reset, EMA, and BurnIn in camera pose and video depth estimation.The comparison validates the TTT-derived state update rule across both tasks.
  • Integrated variant: TTT3R + Reset achieves the best performance among the compared methods.The integrated variant combines TTT3R with the Reset mechanism using n = 100.

B STATE RESET

TTT3R mitigates but does not fully resolve state forgetting, failing beyond 1000 frames. State Reset addresses this boundary by restarting the state every 100 frames and aligning the resulting chunks.

  • Limitation: TTT3R still fails beyond 1000 frames because state forgetting is only mitigated, not fully resolved.This limitation is attributed to recurrence producing state distributions not encountered during training.
  • State Reset mechanism: State Reset resets the state every 100 frames to prevent state overfitting in extended sequences.The resulting chunks are aligned using global metric camera poses without optimization.
  • Evaluation scope: State Reset is used only for visualization demonstrations exceeding 1000 frames, not for the main-paper experiments except Figure 1.Quantitative State Reset results are provided separately in Figure 13.
  • Results: TTT3R + State Reset enables robust long-sequence reconstruction beyond 1000 frames while preserving CUT3R’s inference speed and memory footprint.The reset is performed during the forward pass as a plug-and-play augmentation.

C.1 EXPERIMENTAL SETTINGS

The evaluation separates long-sequence testing from short-sequence benchmarking because many baselines cannot process long inputs due to memory limits. TTT3R is compared with pairwise and all-pointmap reconstruction models across these settings.

  • Long-sequence evaluation: Long-sequence evaluation measures whether online states can memorize entire sequences containing hundreds of images.This setting compares TTT3R with online methods capable of handling long inputs.
  • Short-sequence evaluation: Short-sequence evaluation compares TTT3R with a wider range of baselines that are infeasible on long sequences because of out-of-memory constraints.The short-sequence results are reported separately from the long-sequence evaluation.
  • Datasets: Short-sequence camera-pose evaluation uses Sintel, TUM-dynamics, and ScanNet datasets.The section’s table reports these datasets for camera pose estimation.
  • Compared methods: The comparison includes pairwise methods requiring global alignment and models such as AETHER and VGGT that predict all pointmaps simultaneously.The evaluated methods cover multiple 3D reconstruction foundation-model designs.

C.3 CAMERA POSE ESTIMATION

TTT3R improves camera-pose estimation over CUT3R in long-sequence qualitative comparisons and achieves the best overall performance among online methods. Its accuracy remains below strong offline full-attention methods in the reported short-sequence comparison.

  • Quantitative comparison: TTT3R achieves the best overall camera-pose performance among online methods, particularly on TUM-dynamics and ScanNet.A performance gap remains between online and offline methods.
  • Qualitative comparison: TTT3R provides more accurate and robust camera-pose estimation than CUT3R in the qualitative comparison.The comparison focuses on estimated camera trajectories in Figure 16.
  • Qualitative comparison: TTT3R estimates camera trajectories that deviate less from ground truth than CUT3R.The trajectories are plotted along the two axes with the highest variance.
  • Method connection: TTT3R applies a plug-and-play TTT perspective to derive a novel CUT3R state-transition rule.The approach is presented as requiring only a few lines of code while leveraging the foundation model’s existing knowledge.

C.4 VIDEO DEPTH ESTIMATION

TTT3R provides online video depth estimation for short sequences with strong performance across Sintel, Bonn, and KITTI, while supporting both video and sparse photo inputs. It also extends online reconstruction to long sequences without sacrificing the efficiency of the underlying recurrent design.

  • Evaluation protocol: Video depth estimation measures per-frame depth quality and inter-frame consistency using per-sequence scale alignment to ground truth.The reported metrics are absolute relative error and δ < 1.25.
  • Short-sequence evaluation: TTT3R achieves the best overall performance among online baselines without any fine-tuning.It leads or remains competitive across the evaluated video depth estimation datasets.
  • Length generalization: TTT3R improves length generalization over CUT3R while preserving speed and memory efficiency on long sequences.Offline methods such as VGGT can reconstruct 150-frame sequences but fail on 400-frame sequences because of memory constraints.
  • Short-sequence evaluation: TTT3R leads KITTI on both metric and scale-invariant evaluations and ranks first or second on Sintel and Bonn.The evaluation covers scale-invariant relative depth and metric-scale absolute depth accuracy.
  • Online reconstruction: TTT3R performs online reconstruction for each incoming image by estimating camera parameters and dense geometry across static and dynamic scenes.It supports both video streams and sparse photo collections with varying-length inputs.
Loading 2509.26645v4…