Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training

1 Introduction

• Streaming spatial understanding is needed because real-world 3D evidence emerges across changing viewpoints, while current spatial-aware MLLMs are limited to single images or short clips.Extending inputs naively incurs quadratic attention costs, while temporal subsampling can discard fine-grained spatial details.Source: §1 · p. 2
• Spatial-TTT uses online-updated fast weights as compact nonlinear memory, with interleaved TTT and self-attention layers, large-chunk updates, and sliding-window attention.The hybrid design balances long-context compression with full-context reasoning and preserves intra-chunk spatiotemporal continuity.Source: §1 · pp. 2–3
• Spatial-predictive TTT layers use lightweight depth-wise 3D spatiotemporal convolutions to capture local context, geometric correspondence, and temporal continuity.The mechanism replaces point-wise projections, encouraging predictive mappings across spatiotemporal contexts and more stable, effective online updates.Source: §1 · pp. 2–3
• A dense scene-description dataset supervises fast-weight updates to preserve structured global 3D information, including scene context, objects and counts, and spatial relations.The paper motivates this dataset by noting that existing spatial datasets are sparse and local, providing weak training signals for update dynamics.Source: §1 · pp. 2–3
• Spatial-TTT significantly improves long-horizon spatial understanding and achieves state-of-the-art performance on video spatial benchmarks.The introduction reports state-of-the-art performance across a wide range of visual-based spatial understanding tasks.Source: §1 · p. 3

2.1 Visual-based Spatial Intelligence

• MLLMs remain weaker on precise 3D tasks than on 2D perception and reasoning, motivating benchmarks for video spatial intelligence, spatial-temporal understanding, and spatial recall [Liu et al., 2023; Bai et al., 2025a; Li et al., 2024c; Hong et al., 2023; Xu et al., 2024; Li et al., 2024b; Yang et al., 2025a; Yang et al., 2025c].Examples include VSI-Bench, STI-Bench, and VSI-Super.Source: §2.1 · p. 3
• Prior methods enhance spatial intelligence through depth and multi-view inputs, geometry models, SFT or RL, output feature alignment, and large datasets or staged training, but mostly focus on pre- or post-training [Daxberger et al., 2025; Wu et al., 2025a; Fan et al., 2025; Ouyang et al., 2025; Chen et al., 2025; Yang et al., 2025b; Li et al., 2025a; Yang et al., 2025c].Examples include VST’s 4.1M SFT and 135K RL datasets, SpatialLadder’s 26K dataset, and Cambrian-S’s four-stage framework with VSI-590K.Source: §2.1 · p. 3

2.2 Test-Time Training

• Unlike test-time scaling with frozen parameters, test-time training updates parameters on unlabeled test data during inference; large-chunk fast-weight adaptation has enabled efficient memorization [Ba et al., 2016; Wang et al., 2021; Gandelsman et al., 2022; OpenAI, 2024; Schlag et al., 2021; Liang et al., 2024; Zhang et al., 2025].Prior applications include novel view synthesis, language modeling, and autoregressive video diffusion.Source: §2.2 · p. 3
• End-to-end TTT and few-shot learning studies show continual inference-time weight adaptation can improve reasoning beyond in-context learning, while broader work explores optimizers, losses, and memory representations [Tandon et al., 2025; Aky¨urek et al., 2025; Wang et al., 2025; Behrouz et al., 2024; Karami and Mirrokni, 2025].Despite TTT’s success in language modeling, its use to enhance MLLM visual capabilities remains comparatively limited [Shu et al., 2022; Sun et al., 2024a].Source: §2.2 · p. 4

3.1 Preliminary

• Test-Time Training updates fast weights from each key-value pair and applies the updated memory to queries, allowing it to encode context from preceding tokens.The fast weights parameterize a small neural network and act as adaptive memory while processing the input stream.Source: §3.1 · pp. 4–5

3.2 Overall Framework

• Spatial-TTT interleaves TTT and anchor self-attention layers at a 3:1 ratio to preserve pretrained semantics while compressing long-range dependencies into fast weights.Large chunks improve parallelism and preserve frame structure; sliding-window attention runs alongside TTT to restore within-chunk causal interactions, with window size w at least chunk size b.Source: §3.2 · p. 5
• Depthwise spatiotemporal convolutions enrich TTT queries, keys, and values with local visual context, helping fast weights capture geometric correspondence and temporal continuity.The convolutions are initialized as identity mappings; the method also uses Muon updates and L2-normalized fast weights to promote stable updates.Source: §3.2 · p. 6

3.3 Bridging Sparse Spatial QA with Dense Supervision

• Dense scene descriptions complement sparse spatial QA by supervising global context, object categories and counts, and spatial relations, encouraging persistent global 3D representations.The walkthrough targets are derived from object-centric 3D scene graphs and provide higher-coverage gradient signals for learning fast-weight update dynamics.Source: §3.3 · p. 6

3.4 Spatial-Aware Progressive Training

• Spatial-TTT first trains on dense scene descriptions with an annealed sliding window, then fine-tunes on 2M spatial VQA samples with w = b for streaming reasoning.The second-stage tasks include relative direction and distance estimation, spatial counting, route planning, and room size estimation.Source: §3.4 · p. 7
• At inference, a fixed-length sliding-window KV cache supports local context, while a pending KV cache accumulates tokens for fast-weight updates every chunk.After each update, the pending cache is cleared, supporting constant-memory streaming.Source: §3.4 · p. 7

4.1 Experiment Setup

• Spatial-TTT starts from Qwen3-VL-2B-Instruct and is trained on dense ScanNet and ARKitScenes descriptions before large-scale spatial-VQA data.The dense-description stage contains approximately 16k samples; the second stage uses 3M spatial question-answering examples. TTT layers are interleaved with attention layers, use shared QKV projections, and train with chunk size 2648; evaluation compares proprietary, general-purpose, and spatial-intelligence baselines.Source: §4.1 · pp. 7–8

4.2 General Spatial Understanding Results

• Spatial-TTT-2B achieves the best VSI-Bench overall Avg. of 64.4, with advantages in multiple-choice spatial reasoning and numerical distance estimation.It is particularly strong on Relative Direction and Route Plan, achieves the best Absolute Distance score, and remains competitive on Room Size and Object Count.Source: Table 1 · p. 8, §4.2 · p. 9
• Spatial-TTT achieves 76.2 ACC on MindCube-Tiny, outperforming all baselines and leading the ACROSS and AMONG subsets.It improves by 12.3 percentage points over Gemini-3-pro (63.9% average) and 24.5 points over MindCube-3B (51.7% average), supporting stronger reasoning under viewpoint changes and occlusions.Source: §4.2 · p. 9, Table 2 · p. 9

4.3 Streaming Spatial Sensing Results

• On VSI-SUPER, Spatial-TTT significantly outperforms all baselines on counting and remains competitive on recall across video lengths.Unlike models whose performance collapses on longer sequences, it maintains stable performance by incrementally integrating observations and retaining long-term spatiotemporal evidence; results cover the 10–120 minute subsets sampled at 1 fps.Source: §4.3 · pp. 9–10, Table 3 · p. 10

4.4 Ablation Study and Analysis

• The full model leads VSI-Bench ablations at 64.4 Avg.; removing the spatial-predictive mechanism, dense data, or hybrid architecture lowers Avg. to 62.1, 61.3, and 53.9, respectively.Removing the spatial-predictive mechanism lowers Numerical scores from 64.0 to 60.7; removing dense data drops Numerical and Multiple-Choice scores by 3.0 and 3.3 points, while removing hybrid attention layers lowers Multiple-Choice from 64.8 to 52.4.Source: §4.4 · pp. 10–11, Table 4 · p. 11
• At 1024 frames, Spatial-TTT uses over 40% fewer TFLOPs and less memory than Qwen3-VL-2B, with computation scaling approximately linearly as input length doubles.Spatial-MLLM-4B has super-linear computational scaling and runs out of memory beyond 256 frames; the efficiency gap over Qwen3-VL-2B widens with context length.Source: §4.4 · p. 11, Table 5 · p. 11

• Spatial-TTT achieves state-of-the-art performance across spatial benchmarks.Source: §5 · p. 11
• Its hybrid architecture interleaves TTT layers with self-attention anchor layers, using large-chunk updates and sliding-window attention for efficient long-context processing.The anchor layers preserve pretrained knowledge while TTT layers enable adaptive fast weights as compact memory for accumulating 3D evidence.Source: §5 · p. 11
• A spatial-predictive mechanism with 3D spatiotemporal convolutions injects local-neighborhood inductive bias to promote spatial awareness.Source: §5 · p. 11
• A dense scene-description dataset provides rich supervision for learning effective fast-weight update dynamics.Source: §5 · p. 11
• The paper presents Spatial-TTT as a direction for MLLMs with persistent spatial memory for robust, scalable spatial intelligence in real-world applications.Source: §5 · p. 11

A.1 Spatial Dataset Curation

• The first-stage dataset contains approximately 16K SceneVerse-derived samples: 3.6K from ScanNet and 12.5K from ARKitScenes.Each pairs a spatial video stream with a coherent scene walkthrough covering scene context, object categories and counts, layouts, and pairwise relations.Source: §A.1 · p. 16
• The second stage uses ∼2.5M open-sourced and ∼0.5M self-collected spatial question-answering examples.Open-sourced sources include VSI-590K, VLM-3R, InternSpatial, and ViCA.Source: §A.1 · p. 16
• For self-collected data, ScanNet reconstructions are sampled into indoor-scene videos at 24 fps and 640× 480 resolution.Scene meshes and semantic annotations supply the spatial and semantic metadata.Source: §A.1 · p. 16
• The curation pipeline aligns each mesh with its axis-alignment matrix, converts it to a point cloud, and estimates room extent and centroid using alpha shapes.At object level, it fits oriented bounding boxes for valid instances and assigns semantic labels, excluding structural elements and ambiguous categories.Source: §A.1 · p. 16
• ScanNet semantic IDs are remapped to a consolidated 40-class indoor label set, and 2D projected semantic annotations are stored for appearance-order reasoning.Source: §A.1 · p. 16
• Each self-collected sample stores room size and center, 2D scene-video semantic projections, object OBB rotations, extents and centers, and object labels.Source: §A.1 · p. 16

• Table 6 lists the detailed prompt templates used for each evaluation benchmark in a unified Python f-string format.Source: Evaluation and Inference Prompts · p. 16, Table 6 · p. 16

A.3 Algorithms of Spatial-TTT

• The layer projects hidden states to Q, K, and V, shifts Q and K, then applies depth-wise 3D convolution to video-packed features before scattering them back to tokens.The convolution operates on the video mask and spatial-temporal grid.Source: §A.3 · p. 17
• For each chunk, the layer applies fast weights before a Muon update with orthogonalized gradients and momentum, then L2-normalizes the updated weights; the tail receives only fast-weight application.The update uses learning rate η and chunk size C.Source: §A.3 · p. 17
• The layer concatenates TTT outputs, adds sliding-window attention over Q, K, and V with window B, and projects the combined output.The attention branch uses the original projected Q, K, and V.Source: §A.3 · p. 17
• TTT layers are interleaved with standard causal self-attention anchor layers at a 3:1 ratio.The pseudocode covers one hybrid TTT layer; anchor-layer details are omitted.Source: §A.3 · pp. 16–17

B.1 More Qualitative Comparisons

• Spatial-TTT achieves the best overall VSI-Bench performance (Avg. 64.4), ranks first across all numerical-question categories, and leads on key multiple-choice categories Rel. Dir and Appr. Order.Source: §B.1 · p. 17
• Removing the SP-Mechanism causes clear drops, especially on numerical questions and Route Plan, indicating its importance for metric-aware spatial perception and planning.Source: §B.1 · p. 17
• Removing Dense Data degrades Room Size, Rel. Dir, and Appr. Order, suggesting dense scene supervision supports long-horizon spatial memory and temporally dependent reasoning.Source: §B.1 · p. 17

B.2 QA Examples Visualization

• QA-example visualizations cover VSI-Bench, VSI-SUPER-RECALL, VSI-SUPER-COUNT, and MindCube in Figs. 3–6, respectively.Source: §B.2 · p. 17, Figure 3 · p. 18, Figure 4 · p. 18
• The VSI-Bench examples ask viewers to identify the order in which Pikachu and a panda appear in video.Source: §B.2 · p. 18, Figure 3 · p. 18
• Other examples ask viewpoint-conditioned spatial-relation questions, including what lies right of a sneaker or left of a bag from specified viewpoints.Source: §B.2 · p. 19