Source-linked AI summary
tttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction
Chen Wang, Hao Tan, Wang Yifan, Zhiqin Chen, Yuheng Liu, Kalyan Sunkavalli, Sai Bi, Lingjie Liu, Yiwei Hu
TL;DR
Existing reconstruction models struggle to scale to long and streamed visual contexts while producing explicit 3D outputs. tttLRM uses TTT fast weights as implicit 3D memory, decodes them into explicit representations, and supports autoregressive reconstruction with linear complexity. Across object- and scene-level datasets, it reports superior quality and scalability, with efficient rendering and benefits from pretrained novel-view-synthesis knowledge.
Problem
Existing feedforward reconstruction models have limited input-view scalability, while long-context methods hinder efficient processing of longer and streamed contexts.
Method
tttLRM updates TTT fast weights from image observations, treats them as implicit 3D representations, and decodes them into explicit formats such as Gaussian splats and triplanes.
Results
Experiments on object- and scene-level datasets show superior reconstruction quality and efficiency compared with baseline and prior feedforward methods.
Takeaways & Limitations
The framework supports feedforward long-context and autoregressive 3D modeling while narrowing the gap between neural rendering and real-time explicit 3D systems.
Takeaways & Limitations
The fixed-size fast-weight memory may limit handling of highly complex scenarios with extremely large numbers of input views, and quality slightly degrades relative to pretrained LVSM.
Abstract
from arXiv · showhide
We propose tttLRM, a novel large 3D reconstruction model that leverages a Test-Time Training (TTT) layer to enable long-context, autoregressive 3D reconstruction with linear computational complexity, further scaling the model's capability. Our framework efficiently compresses multiple image observations into the fast weights of the TTT layer, forming an implicit 3D representation in the latent space that can be decoded into various explicit formats, such as Gaussian Splats (GS) for downstream applications. The online learning variant of our model supports progressive 3D reconstruction and refinement from streaming observations. We demonstrate that pretraining on novel view synthesis tasks effectively transfers to explicit 3D modeling, resulting in improved reconstruction quality and faster convergence. Extensive experiments show that our method achieves superior performance in feedforward 3D Gaussian reconstruction compared to state-of-the-art approaches on both objects and scenes.
1. Introduction
tttLRM targets long-context, autoregressive explicit 3D reconstruction from streaming visual input, addressing scalability limits in existing methods. It uses TTT fast weights as implicit 3D representations decoded into explicit formats, achieving strong quality and efficiency across objects and scenes.
- Existing reconstruction methods either require slow scene-specific optimization or scale poorly with input views, limiting long-context reconstruction.
- The framework supports high-quality novel view synthesis while retaining explicit 3D outputs for real-time rendering and downstream use.
- tttLRM uses TTT fast weights as implicit latent-space 3D representations that can be decoded into explicit formats such as 3DGS or NeRFs.
- LaCT blocks provide linear computational complexity, enabling feedforward long-context and autoregressive 3D modeling.
- Experiments on object- and scene-level datasets report superior reconstruction quality and efficiency compared with baseline methods.
2. Related Work
Related work spans classical geometry recovery, neural scene representations, feedforward large reconstruction models, and sequence models. These approaches improve reconstruction or rendering, but existing feedforward models remain constrained in input-view scalability.
- Structure-from-motion and multi-view stereo primarily recover 3D geometry, while deep learning enables feedforward reconstruction from images.
- NeRF represents scenes as continuous fields and supports differentiable volumetric rendering, but original NeRF requires hours to optimize a single scene.
- Large Reconstruction Models use transformer architectures for feedforward 3D object reconstruction, with later work predicting pixel-aligned 3D Gaussian primitives.
- State Space Models represent historical information with state variables and use state updates to retain recent sequence information.
3. Method
The method tokenizes posed images, updates LaCT fast weights, and queries them with virtual tokens to decode explicit 3D representations. It supports causal streaming reconstruction and distributed training with rendering-based supervision.
- TTT and LaCT: TTT treats input key-value pairs as training data for fast-weight updates, then applies the learned weights to queries with fixed-size memory.
- TTT and LaCT: LaCT updates fast weights over large chunks, improving utilization while preserving linear update and application complexity in sequence length.
- Model Architecture: Posed images are concatenated with ray embeddings, patchified, and tokenized into the input sequence.
- Model Architecture: Visual tokens update LaCT fast weights, while virtual tokens query those weights without updating them for 3D representation prediction.
- Model Architecture: A linear decoder converts query tokens into explicit representations, directly predicting Gaussian color, scale, rotation, opacity, and positions.
- Autoregressive Reconstruction: Causal updates allow each incoming view batch to update fast weights and immediately produce Gaussian splats, yielding online autoregressive reconstruction.
- Distributed Training: Sequence parallelism shards input tokens across GPUs, synchronizes fast weights, gathers predicted Gaussians, and reduces rendering gradients across devices.
- Training Objective: Training renders reconstructed Gaussians on target views and minimizes a combination of MSE and VGG-19 perceptual loss without explicit 3D supervision.
4. Experiments
Experiments evaluate tttLRM on object- and scene-level reconstruction across resolutions, input-view counts, representations, and training strategies. The results show improved reconstruction quality, speed, scalability, autoregressive refinement, format flexibility, and pretrained convergence, with a fixed-memory limitation.
- Experimental Setup: The model uses 24 LaCT blocks with hidden dimension 768 and is trained on 730K Objaverse objects rendered from 32 views.The object-level setup uses 512 × 512 images and compares against GS-LRM under 8 input views.
- Object-level Results: Across object-level resolutions and input-view counts, tttLRM consistently outperforms GS-LRM in reconstruction quality and inference speed.At lower resolutions and shorter sequences, its inference speed is comparable to full-attention models.
- Object-level Results: At 512 × 512, tttLRM runs twice as fast as attention-based models while achieving over a 1 dB PSNR improvement.When trained with 8 views, it directly generalizes to 16 or 24 views; longer sequences further improve speed and rendering quality through test-time training.
- High-resolution and Image-to-3D Results: The model scales to 1024 × 1024, where GS-LRM encounters out-of-memory issues, and supports detailed image-to-3D reconstruction when combined with a multi-view diffusion model.The demonstrated outputs include humans, animals, and text.
- Scene-level and Autoregressive Results: On scene-level datasets, tttLRM is competitive with or better than optimization-based methods, substantially outperforms Long-LRM by approximately 1 dB PSNR, and is hundreds times faster with more views.A single model generalizes across sequence lengths and to Tanks & Temples, while autoregressive reconstruction progressively improves rendering quality and scene coverage as views increase.
- Ablations, Formats, and Limitations: Pretrained initialization accelerates convergence and improves final rendering quality, while the framework can decode latent 3D memory into triplane-based NeRFs as well as 3DGS.The experiments also report optimizer and regularization effects, and note a fixed fast-weight memory that may limit extremely large input-view counts.
5. Conclusion
tttLRM supports feedforward long-context and autoregressive 3D modeling by converting implicit fast-weight representations into explicit 3D formats. Experiments show improved quality and scalability while approaching the speed of explicit representations.
- tttLRM supports both feedforward long-context and autoregressive 3D modeling.
- The framework converts Test-Time Training fast-weight representations into explicit formats including Gaussian splats and triplanes.
- Experiments on object- and scene-level datasets show higher quality and scalability than prior feedforward methods.
- The model approaches the speed of explicit representations for efficient, high-quality novel view synthesis.
A. Further Discussions
The further discussions examine fast-weight capacity, selective updates, view scaling, and computational trade-offs. Fast weights have bounded memory, while the linear-complexity design supports scaling to hundreds of views.
- Effect of Scene Complexity on Fast Weights: Fast-weight memory has fixed, bounded capacity, particularly in autoregressive reconstruction.Higher scene complexity and longer sequences degrade performance, with earlier inputs potentially forgotten.
- Selective Update of Fast Weights in AR Setting: Training-free selective updates use history states, Fisher-based parameter importance, and an exponential-moving-average anchor to mitigate weight drift.
- Scaling to More Input Views: With distributed training and sufficient compute, tttLRM can scale to hundreds of views and achieved 26.80 PSNR on 128 input views.
- Possible Usage of Attention Layers: Attention becomes a computational bottleneck because its O(N^2d) complexity grows faster than LaCT blocks' linear O(Nd^2) complexity.At 2M tokens, even three attention layers are slower than 24 LaCT layers.
B.1. Scene-level Training
Scene-level training uses curriculum learning from low to high resolution, randomized multi-view sampling, staged optimization, and autoregressive finetuning. Several engineering techniques improve training speed, memory use, and stability.
- Resolution Curriculum: Curriculum training progresses through 144×256, 288×512, and 540×960 resolutions.Low-resolution pretraining enables larger batches and faster iterations, while avoiding excessive memory use from early high-resolution Gaussian rendering.
- Scene Sampling: Each scene sample randomly selects a frame range of 128–512 and then samples 124 frames with overlapping input and target views.
- Optimization Schedule: Training uses three resolution stages with progressively refined objectives, including depth and opacity losses in the final stage.
- Autoregressive Finetuning: Autoregressive training finetunes final-stage checkpoints for about 3K iterations using 8–64 input views on 64 Nvidia A100 80GB GPUs.
- Systems Optimization: The implementation combines gsplat, torch.compile, gradient checkpointing, BFloat16 mixed precision, and deferred backpropagation.torch.compile provides roughly a 30% per-iteration speedup, while the other techniques target memory and stability.
B.2. Object-level training
Object-level training uses separate Gaussian-splat and triplane configurations with different input and supervision view counts. Both configurations train at 256×256, with the triplane model additionally finetuned at 512×512.
- Gaussian-Splat Model: The Gaussian-splat model uses 8 input views and 8 supervision views with 16×16 patches.
- Gaussian-Splat Model: Gaussian-splat training samples 15 images from 32 renderings and trains at 256×256 for 80K iterations.
- Triplane Model: The triplane model uses 4 input views and 4 supervision views with 16×16 patches.
- Triplane Model: Triplane training runs at 256×256 for 60K iterations and is finetuned at 512×512 for another 20K iterations.
C. More results and Comparison
A few post-optimization steps further improve tttLRM and enable it to surpass optimization-based methods and Long-LRM under matched post-optimization settings. These comparisons cover DL3DV-140 and Tanks&Temples with 32 or 64 input views.
- A few additional optimization steps further improve the reconstructed model with minimal optimization cost.
- The enhanced model surpasses purely optimization-based methods and Long-LRM under the same post-optimization setup.
- Long-LRM with 3-step post-optimization remains lower quality than tttLRM without post-optimization despite requiring more optimization time.
- Table 7 compares both methods on DL3DV-140 and Tanks&Temples using 32 or 64 input views.