Source-linked AI summary
Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training
Fangfu Liu, Diankun Wu, Jiawei Chi, Yimo Cai, Yi-Hsin Hung, Xumin Yu, Hao Li, Han Hu, Yongming Rao, Yueqi Duan
TL;DR
Long-horizon video streams require spatial systems to maintain and organize 3D evidence over time, beyond simply extending context windows. Spatial-TTT uses adaptive fast weights, a hybrid streaming architecture, spatial-predictive 3D convolutions, and dense spatial supervision; it improves long-horizon spatial understanding and achieves state-of-the-art performance on video spatial benchmarks.
Problem
Streaming spatial understanding must maintain, update, and reason over spatial evidence from potentially unbounded visual streams involving changing viewpoints and occlusions.
Method
Spatial-TTT uses online-updated fast weights with hybrid TTT and self-attention layers, large-chunk updates, sliding-window attention, spatial-predictive 3D convolutions, and dense scene-description supervision.
Results
Spatial-TTT improves long-horizon spatial understanding and achieves state-of-the-art performance on video spatial benchmarks and a wide range of visual-based spatial tasks.
Takeaways & Limitations
The framework provides a streaming approach for accumulating and organizing structured 3D spatial evidence over long-horizon scene videos.
Abstract
from arXiv · showhide
Humans perceive and understand real-world spaces through a stream of visual observations. Therefore, the ability to streamingly maintain and update spatial evidence from potentially unbounded video streams is essential for spatial intelligence. The core challenge is not simply longer context windows but how spatial information is selected, organized, and retained over time. In this paper, we propose Spatial-TTT towards streaming visual-based spatial intelligence with test-time training (TTT), which adapts a subset of parameters (fast weights) to capture and organize spatial evidence over long-horizon scene videos. Specifically, we design a hybrid architecture and adopt large-chunk updates parallel with sliding-window attention for efficient spatial video processing. To further promote spatial awareness, we introduce a spatial-predictive mechanism applied to TTT layers with 3D spatiotemporal convolution, which encourages the model to capture geometric correspondence and temporal continuity across frames. Beyond architecture design, we construct a dataset with dense 3D spatial descriptions, which guides the model to update its fast weights to memorize and organize global 3D spatial signals in a structured manner. Extensive experiments demonstrate that Spatial-TTT improves long-horizon spatial understanding and achieves state-of-the-art performance on video spatial benchmarks. Project page: https://liuff19.github.io/Spatial-TTT.
Spatial TTT
Spatial-TTT updates spatial state from streaming chunks before answering a visual-based spatial task.
- Spatial-TTT updates spatial state with streaming chunks, then answers the visual-based spatial question.
1 Introduction
Spatial-TTT addresses streaming spatial understanding by maintaining spatial evidence from long-horizon videos through adaptive fast weights, spatially aware updates, and dense supervision. The framework combines efficient long-context processing with mechanisms designed to preserve geometric and temporal structure, and it achieves state-of-the-art performance on video spatial benchmarks.
- Streaming spatial understanding requires selectively maintaining, progressively updating, and reasoning over spatial memory from long-horizon video inputs.Continuous visual streams involve camera motion, changing viewpoints, and occlusion.
- Spatial-TTT uses online-updated fast weights as compact nonlinear memory to accumulate 3D evidence from long-horizon spatial video streams.
- A hybrid architecture interleaves TTT and self-attention anchor layers, with large-chunk updates and sliding-window attention for efficient long-context processing.The anchor layers preserve pretrained visual-semantic knowledge while the other mechanisms improve parallelism and hardware efficiency.
- The spatial-predictive mechanism uses 3D spatiotemporal convolutions to capture geometric correspondence and temporal continuity across visual tokens.It replaces point-wise projections with local-neighborhood aggregation for fast-weight updates.
- A dense scene-description dataset provides rich supervision for fast-weight updates to preserve structured scene-level spatial information along the video stream.Descriptions cover global context, objects and counts, and spatial relations.
- Spatial-TTT significantly improves long-horizon spatial understanding and achieves state-of-the-art performance across a wide range of visual-based spatial tasks and video spatial benchmarks.
2 Related Work
Prior work applies test-time training to continual parameter adaptation and efficient memorization, while visual spatial intelligence remains challenging because MLLMs often lack 3D geometric priors.
- MLLMs perform strongly on 2D perception but struggle with precise 3D spatial alignment in tasks such as robotic manipulation and 3D question answering.
- Test-time training updates model parameters during inference using unlabeled test data, unlike test-time scaling with frozen parameters.
- Prior TTT work shows that adapting fast weights over large chunks can support efficient memorization across language, view-synthesis, and video-generation tasks.
3 Spatial-TTT
Spatial-TTT combines adaptive fast-weight memory with hybrid attention, spatial-predictive convolutions, and dense scene-description supervision for streaming spatial understanding.
- 3.1 Preliminary: TTT updates designated fast weights during inference by associating key-value pairs through gradient descent and applying the updated network to queries.
- 3.2 Overall Framework: The hybrid architecture interleaves TTT and self-attention anchor layers, preserving pretrained semantic reasoning while compressing long-range dependencies into adaptive fast weights.
- 3.2 Overall Framework: Large chunks improve parallelism and hardware efficiency while keeping spatially coherent visual content within update boundaries.
- 3.2 Overall Framework: Each TTT layer combines sliding-window attention with a fast-weight branch, whose output adds local-window context to memory-based query processing.
- 3.2 Overall Framework: The spatial-predictive mechanism replaces point-wise projections with depthwise 3D spatiotemporal convolutions, enabling fast weights to capture geometric correspondence and temporal continuity.
- 3.3 Spatial-Aware Progressive Training: Dense scene-description supervision trains fast-weight updates using comprehensive 3D scene walkthroughs rather than sparse, local spatial answers.
- 3.4 Spatial-Aware Progressive Training: Dual KV caches provide constant-memory streaming by retaining a fixed sliding window and batching pending key-value pairs into fast-weight updates.
4 Experiments
Spatial-TTT is evaluated across general spatial understanding, viewpoint-change reasoning, streaming spatial sensing, ablations, and computational efficiency. It achieves strong benchmark results while its components and resource scaling support long-horizon processing.
- Results on VSI-Bench: Spatial-TTT-2B achieves the best overall VSI-Bench performance, with an Avg. of 64.4 despite its compact 2B scale.It shows particular advantages on Relative Direction and Route Plan tasks.
- Results on MindCube Benchmark: 76.2 ACC on MindCube-Tiny surpasses all baselines, improving by 12.3 and 24.5 percentage points over the strongest proprietary and open-source spatial baselines.The model performs best on ACROSS and AMONG subsets.
- Streaming Spatial Sensing Results: Spatial-TTT is competitive on VSI-SUPER-Recall across video lengths and significantly outperforms all baselines on VSI-SUPER-Count.VSR tests temporal-order recall, whereas VSC tests object counting across extended sequences.
- Ablation Study and Analysis: Removing the spatial-predictive mechanism lowers Avg. from 64.4 to 62.1, with Numerical performance falling from 64.0 to 60.7.The ablation supports the contribution of local spatiotemporal inductive bias to fast-weight updates and metric-level spatial reasoning.
- Ablation Study and Analysis: Removing dense scene-description data reduces Avg. from 64.4 to 61.3, while Numerical and Multiple-Choice scores drop by 3.0 and 3.3 points.The reported interpretation is that dense scene-level supervision strengthens online update dynamics and retention of globally useful 3D evidence.
- Ablation Study and Analysis: Removing self-attention anchor layers causes the largest ablation degradation, reducing Avg. from 64.4 to 53.9 and Multiple-Choice from 64.8 to 52.4.The authors associate hybrid interleaving with retaining cross-modal alignment and global-context reasoning.
- Memory Usage and Theoretical TFLOPs: Spatial-MLLM exhibits super-linear computational scaling and runs out of memory beyond 256 frames, limiting its practicality for streaming video scenarios.The comparison analyzes decoding memory usage and theoretical TFLOPs across input lengths.
5 Conclusion
Spatial-TTT presents a streaming spatial-intelligence framework that uses adaptive fast weights to accumulate 3D evidence from long-horizon videos. Its hybrid architecture, spatial-predictive convolutions, and dense scene-description supervision support reported state-of-the-art performance across spatial benchmarks.
- 5 Conclusion: Spatial-TTT uses test-time training to maintain adaptive fast weights as compact memory for accumulating 3D evidence from long-horizon video streams.The framework combines hybrid TTT and self-attention layers, large-chunk updates, sliding-window attention, spatial-predictive 3D convolutions, and dense scene-description supervision.
A.1 Spatial Dataset Curation
The spatial dataset curation uses two training stages: dense scene-description supervision followed by large-scale spatial question answering. The datasets combine scene-graph-derived descriptions, open-sourced spatial data, and self-collected data.
- A.1 Spatial Dataset Curation: The first-stage dense scene-description dataset is built from SceneVerse object-centric 3D scene graphs and trains fast weights to retain scene information through chunk-by-chunk updates.Descriptions cover global context, object categories and counts, and spatial relations.
- A.1 Spatial Dataset Curation: The second training stage uses a large-scale spatial question-answering dataset combining approximately 2.5M open-sourced and 0.5M self-collected examples.The open-sourced data includes VSI-590K, VLM-3R, InternSpatial, and ViCA.
Evaluation and Inference Prompts
The appendix documents evaluation prompts and the hybrid TTT-layer procedure. The algorithm combines shared projections, spatial-predictive 3D convolution, fast-weight updates, and sliding-window attention, while anchor layers retain standard causal self-attention.
- Evaluation and Inference Prompts: Table 6 provides prompt templates for the evaluation benchmarks in a unified Python f-string format.The prompts are presented to support reproducibility.
- A.3 Algorithms of Spatial-TTT: Hybrid layers interleave TTT layers with standard causal self-attention anchor layers at a 3:1 ratio.Anchor layers are omitted from the single-layer pseudocode but preserve the stated standard attention structure.
- A.3 Algorithms of Spatial-TTT: Algorithm 1 specifies prefilling for a single hybrid TTT layer using fast weights, a Conv3D kernel, chunk size, learning rate, and sliding-window size.These inputs define the layer’s update and attention configuration.
- A.3 Algorithms of Spatial-TTT: The layer projects hidden states into Q, K, and V, then applies learnable Q/K scale-shift operations for the TTT branch.The algorithm labels this as shared QKV projection followed by a learnable scale/shift.
- A.3 Algorithms of Spatial-TTT: The spatial-predictive branch packs visual tokens into a spatiotemporal volume and applies depth-wise 3D convolution.This operation aggregates local spatiotemporal context before the fast-weight computation.
- A.3 Algorithms of Spatial-TTT: Fast weights are updated with normalized gradient-based optimization, after which the fast-weight network is applied to the tail tokens.The pseudocode then concatenates the TTT outputs.
- A.3 Algorithms of Spatial-TTT: The sliding-window-attention branch computes attention over Q, K, and V within window B and is combined with the TTT output before projection.This is the algorithm’s final output-combination step.
B.1 More Qualitative Comparisons
The section provides qualitative QA visualizations across four evaluation benchmarks and reports complete VSI-Bench ablations. Spatial-TTT achieves the strongest overall ablation performance, while removing the SP-Mechanism reduces performance.
- Spatial-TTT achieves the best overall VSI-Bench ablation performance, with an average score of 64.4.
- It ranks first across all numerical-question categories: Obj. Count, Abs. Dist, Obj. Size, and Room Size.
- Spatial-TTT also attains top results on the Rel. Dir and Appr. Order multiple-choice questions.
- Removing the SP-Mechanism causes a clear performance drop, especially on numerical questions and Route Plan.
- QA example visualizations are provided for VSI-Bench, VSI-SUPER-RECALL, VSI-SUPER-COUNT, and MindCube.