Source-linked AI summary
Video4Spatial: Towards Visuospatial Intelligence with Context-Guided Video Generation
Zeqi Xiao, Yiwei Zhao, Lingxiao Li, Yushi Lan, Ning Yu, Rahul Garg, Roshni Cooper, Mohammad H. Taghavi, Xingang Pan
TL;DR
Video4Spatial investigates whether video generative models can perform spatial reasoning from video context alone, without auxiliary 3D signals. It uses video diffusion for scene navigation and object grounding, reporting geometry-consistent navigation, target grounding, longer-context extrapolation, and generalization beyond indoor scenes.
Problem
Whether video generative models can perform visuospatial tasks using only video-based scene context, rather than auxiliary modalities such as depth or poses.
Source: §1 · pp. 2–3Method
Video4Spatial conditions a video diffusion model on scene context and instructions, using context frames as noise-free inputs and design choices including joint guidance and auxiliary bounding boxes.
Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 4Results
Takeaways & Limitations
The results advance video-only spatial reasoning by showing that video generative models can execute navigation and grounding tasks from scene context.
Source: §1 · p. 3, §5 · p. 9The approach operates at 416 × 256 resolution without context compression, and the paper identifies higher-resolution visual fidelity as a key improvement area.
Source: §5 · p. 9
Abstract
from arXiv · showhide
We investigate whether video generative models can exhibit visuospatial intelligence, a capability central to human cognition, using only visual data. To this end, we present Video4Spatial, a framework showing that video diffusion models conditioned solely on video-based scene context can perform complex spatial tasks. We validate on two tasks: scene navigation - following camera-pose instructions while remaining consistent with 3D geometry of the scene, and object grounding - which requires semantic localization, instruction following, and planning. Both tasks use video-only inputs, without auxiliary modalities such as depth or poses. With simple yet effective design choices in the framework and data curation, Video4Spatial demonstrates strong spatial understanding from video context: it plans navigation and grounds target objects end-to-end, follows camera-pose instructions while maintaining spatial consistency, and generalizes to long contexts and out-of-domain environments. Taken together, these results advance video generative models toward general visuospatial reasoning.
1. Introduction
VIDEO4SPATIAL asks whether video diffusion models can perform spatial tasks from video context alone, without auxiliary 3D signals. It evaluates navigation and object grounding, reporting strong performance, longer-context extrapolation, and transfer beyond indoor training environments.
1. Introduction
- The framework uses a standard video diffusion architecture trained only with the diffusion objective to complete instructed spatial tasks while preserving scene geometry and temporal coherence.Its inputs are several context frames from the same environment and instructions.Source: §1 · p. 2
- Scene navigation follows camera-pose instructions consistently with scene geometry, while object grounding moves toward and visibly localizes an instructed target.Together, the tasks probe whether models can infer and act on scene structure using video alone.Source: §1 · p. 2
- VIDEO4SPATIAL performs scene navigation and object grounding from video context, extrapolates to longer inference windows, and generalizes to outdoor environments and unseen object categories.The model is trained on indoor scenes; longer-context inference further improves performance.Source: §1 · p. 2
- Joint guidance over context and instruction, auxiliary bounding boxes, and non-contiguous context sampling are key design choices for coherence, grounding, and context efficiency.The framework also uses data curation to construct task examples from indoor scanning datasets.Source: §1 · p. 2
- The two tasks test geometry and semantic layout without auxiliary modalities such as depth or point clouds.The paper positions these results as progress toward video-only spatial reasoning.Source: §1 · p. 3
2. Related Work
Related work spans video models as controllable renderers and as emerging reasoners, as well as video-to-video generation and visuospatial intelligence. The paper positions scene-level spatial reasoning from raw video context as underexplored relative to methods using explicit 3D signals or camera-pose annotations.
Generative Video Models: from Renderer to Reasoner
Video-to-Video Generation
- Video-to-video methods often transform frames in constrained ways, while free-form generation from video context demands stronger long-range modeling and remains underexplored for scene-level spatial tasks.Related directions include multi-shot movie generation and autoregressive video generation [9] [26] [77] [31].Source: §2 · p. 3
Visuospatial Intelligence
- Unlike approaches using reconstructed 3D memories or camera-pose annotations [40] [76] [14] [78] [96] [100], this work uses raw video sequences alone for spatial understanding.The paper presents this as an end-to-end, scalable approach without explicit 3D signals such as depth or pose.Source: §2 · p. 3
3. Methods
The methods frame video generation as spatial reasoning: a model conditions on scene context and instructions to generate goal-directed video. The paper evaluates object grounding and scene navigation, then introduces conditioning, sampling, and auxiliary-output choices for these tasks.
3.1. Preliminaries
- Video diffusion models learn to denoise conditioned samples, and classifier-free guidance combines conditional and unconditional predictions using a guidance scale.Rotary positional embeddings encode relative spatial-temporal position through query and key rotations.Source: §3.1 · p. 3
3.2. Video Generation Models as Spatial Reasoners
- The framework treats context and target as part of one generation sequence, keeping context frames noise-free while denoising target frames.Instructions can enter through cross-attention, learned embedding addition, or token-wise concatenation.Source: §3.2 · p. 4, Figure 2 · p. 4
3.3. Solving Spatial Tasks with Videos
- Object grounding generates a sequence ending with the instructed object visibly localized, while scene navigation generates viewpoints along instructed egocentric 6-DoF pose waypoints.The tasks assess geometric consistency, contextual consistency, and coherent motion; navigation uses no explicit context pose, depth, or reconstructed geometry.Source: §3.3 · p. 4
3.4. Key Design Choices
- Joint classifier-free guidance applies guidance to instruction and video context together, and an auxiliary target-centered bounding box strengthens object grounding.The box is added to the final training frames as a visual reasoning pattern [20].Source: §3.4 · p. 4, Figure 3 · p. 4
- Non-contiguous context sampling reduces redundancy, while applying RoPE at original source-frame indices supports temporal coherence and longer-context extrapolation.Generated-frame RoPE indices begin after the last context index, with a fixed interval of 50 to separate context from targets and mitigate information leakage.Source: §3.4 · p. 4
4. Experiments
Across object grounding and scene navigation, Video4Spatial generates instruction-aligned videos while maintaining scene consistency. Experiments also report benefits from context and design choices, plus generalization to longer contexts and out-of-domain settings.
4.1. Video-based Object Grounding
- Video4Spatial grounds targets faithfully with Spatial Distance near ground truth, competitive Imaging Quality, and high Dynamic Degree, unlike hallucinatory Wan2.2, Veo3, and FramePack outputs.The comparison is reported in Table 1 and illustrated in Figure 5.Source: Table 1 · p. 5, §4.1 · p. 7
- Grounding evaluation combines point-cloud Spatial Distance with Instruction Following and reports IF(SD<δ) to count target hits only when consistency is below a threshold.The default thresholds are δ = 0.2 and δ = 0.1.Source: §4.1 · pp. 5–6
4.2. Ablations on Object Grounding
- Longer inference context lowers SD, while IF first drops and then rises as context becomes sufficient for accurate grounding; excessive training context can reduce IF.Training with 169 context frames balances these effects and generalizes to longer inference contexts such as 337 frames.Source: §4.2 · p. 7
- Auxiliary boxes raise IF(SD<0.2) from 0.5401 to 0.6486; removing context CFG, using continuous RoPE, or skipping pretraining worsens reported spatial metrics.Without a predicted box, the monitor example drifts to irrelevant objects; with one, it grounds the monitor correctly.Source: §4.2 · pp. 7–8
4.3. Video-based Scene Navigation
- Video4Spatial delivers competitive PSNR and LPIPS with substantially better IQ than AnySplat, and outperforms Gen3C and TrajectoryCrafter across all reported metrics.Unlike those video-based methods, Video4Spatial does not depend on external estimators for extra 3D information.Source: Table 2 · p. 8, §4.3 · p. 8
4.4. Out-of-domain generalization.
- Despite training only on indoor scenes, Video4Spatial grounds rarely encountered categories such as trees outdoors and supports free-form navigation, including 360° rotations.Figure 11 presents the out-of-domain results.Source: §4.4 · p. 8, Figure 11 · p. 8
5. Conclusions and Future Work
Video4Spatial uses scene video alone to probe visuospatial intelligence, performing geometry-consistent navigation and object grounding while generalizing to longer contexts and out-of-domain environments. The authors identify resolution, temporal artifacts, long-tail grounding, and dynamic-scene coverage as limitations or future-work areas.
5. Conclusions and Future Work
- Video4Spatial conditions on scene context alone and performs scene navigation and object grounding with geometry-consistent trajectories and target grounding.It also generalizes to long contexts and out-of-domain environments.Source: §5 · p. 9
- The current approach operates at 416 × 256 resolution because it lacks context compression, which the authors identify as a lever for higher visual fidelity.They also point to temporal modeling, targeted augmentation, and improved grounding objectives as ways to address temporal discontinuities and incorrect long-tail-category grounding.Source: §5 · p. 9
- Extending the framework beyond static scenes to dynamic environments and a broader range of tasks remains future work.Source: §5 · p. 9
6. Suppplementary Materials
The supplementary materials present qualitative examples, implementation and benchmark details, experimental ablations, and computational and failure-case observations.
6.1. More Qualitative Results
- The qualitative-results section directs readers to the project webpage for visualizations.Source: §6.1 · p. 10
6.2. Implementation Details.
- Grounding data curation samples frames every three frames, uses Qwen3-VL for object grounding, and selects a centered target instance for each clip.The target clip ends on a frame where that object is centered; Qwen3-VL also captions the clip.Source: §6.2 · p. 10
- Training uses 161-frame target clips and contiguous groups of four context frames, with overlap chosen to support geometric consistency.The materials also describe registering generated and ground-truth point clouds using 40 ground-truth frames as anchors before measuring distance.Source: §6.2 · p. 10
6.3. Benchmark setting.
- The benchmark section lists distinct input, output, resolution, and control settings for comparison methods and notes that perfectly fair settings are infeasible.TrajectoryCrafter’s 121-frame output is assembled across three inference runs because the model generates 49 frames at a time.Source: §6.3 · pp. 10–11
6.4. Experiments.
- Repeated sampling improves grounding success, while CFG scales of 3–7 remain robust and non-contiguous context sampling improves geometric consistency and grounding fidelity.Randomly choosing the zero-pose reference frame alleviates restricted spatial exploration and improves generalization.Source: §6.4 · p. 11
- The supplementary experiments include a table on context-sampling ablations and figures on grounding success over repeats and CFG weights.Source: Figure 13 · p. 11, Figure 14 · p. 11, Table 3 · p. 12
- Inference uses about 34GB of VRAM and takes two minutes on one A100 GPU, with a 416 × 256 resolution and 50 denoising steps.The authors report temporal discontinuities and incorrect grounding in some cases.Source: §6.4 · p. 12