Source-linked AI summary
SpatialVID: A Large-Scale Video Dataset with Spatial Annotations
Jiahao Wang, Yufeng Yuan, Rujie Zheng, Youtian Lin, Jian Gao, Lin-Zhuo Chen, Yajie Bao, Yi Zhang, Chang Zeng, Yanxi Zhou, Xiao-Xiao Long, Hao Zhu, Zhaoxiang Zhang, Xun Cao, Yao Yao
TL;DR
Spatial intelligence lacks large, diverse real-world video datasets that combine semantic richness with explicit geometric supervision. SpatialVID addresses this gap by curating and annotating in-the-wild videos, and experiments report advances across controllable video generation, world simulation, and geometric reconstruction.
Problem
Existing large-scale video datasets lack explicit 3D information, while spatial datasets are limited in scale, diversity, or real-world dynamic-scene coverage.
Method
SpatialVID manually screens internet videos, applies hierarchical filtering, and adds camera poses, depth maps, dynamic masks, structured captions, and motion instructions.
Results
SpatialVID advances controllable video generation, world simulation, and geometric reconstruction across extensive experiments.
Takeaways & Limitations
SpatialVID provides a large-scale resource unifying explicit 3D motion control with rich textual semantics for spatial intelligence research.
Takeaways & Limitations
The annotation pipeline can degrade under extreme scenarios, and predicted camera poses may be non-metric while motion-derived masks can be suboptimal in complex scenes.
Abstract
from arXiv · showhide
Significant progress has been made in spatial intelligence, spanning both spatial reconstruction and world exploration. However, the scalability and real-world fidelity of current models remain severely constrained by the scarcity of large-scale, high-quality training data. While several datasets provide camera pose information, they are typically limited in scale, diversity, and annotation richness, particularly for real-world dynamic scenes with ground-truth camera motion. To this end, we collect SpatialVID, a dataset consists of a large corpus of in-the-wild videos with diverse scenes, camera movements and dense 3D annotations such as per-frame camera poses, depth, and motion instructions. Specifically, we collect more than 21,000 hours of raw videos, and process them into 2.7 million clips through a hierarchical filtering pipeline, totaling 7,089 hours of dynamic content. A subsequent annotation pipeline enriches these clips with detailed spatial and semantic information, including camera poses, depth maps, dynamic masks, structured captions, and serialized motion instructions. Analysis of SpatialVID's data statistics reveals a richness and diversity that directly fosters improved model generalization and performance, establishing it as a key asset for the video and 3D vision research community.
1. Introduction
SpatialVID addresses the divide between semantically rich video datasets without geometric ground truth and spatial datasets with geometry but limited scale, diversity, or realism. It combines large-scale dynamic videos with explicit geometric, semantic, and motion annotations for spatial intelligence research.
- Current video datasets offer rich semantics but lack explicit 3D information, while spatial datasets provide geometry but are often small, object-centric, or synthetic.
- SpatialVID processes over 21,000 hours of manually screened internet video into 7,089 hours of high-quality 720P clips with reliable camera motion.
- SpatialVID-HQ is a 1,111-hour balanced subset optimized for robust training and evaluation.
- SpatialVID is presented as the largest dataset of dynamic videos with explicit geometric annotations.
- The dataset provides camera poses, depth maps, structured captions, and motion instructions derived from camera trajectories.
2. Related Work
Related work spans classical and learnable methods for reconstruction, video generation, and spatially annotated datasets. SpatialVID is positioned as a large-scale real-world resource combining dynamic video, geometry, and semantic information.
- Scene Reconstruction: Classical SLAM and SfM methods provide accurate geometry and tracking but depend heavily on handcrafted features, motivating learnable reconstruction methods.
- Datasets with Spatial Annotations: SpatialVID is described as a million-level, dynamic, open-scenario dataset with annotated geometric and semantic information.
- Scene Reconstruction: Enhanced MegaSaM is used to generate initial geometric annotations because of its demonstrated robustness in unconstrained, in-the-wild videos.
- World Simulator: Video-generation research has progressed from UNet-based diffusion models to DiT-based architectures and other approaches for fidelity, scalability, and temporal consistency.
- Datasets with Spatial Annotations: Existing spatial datasets include synthetic resources with precise geometry and real-world datasets that often have sparse trajectories or struggle with dynamic motion.
3. SpatialVID Curation
SpatialVID uses filtering, annotation, and sampling to convert manually collected web videos into motion-rich clips with geometric, semantic, and motion annotations. The pipeline also produces structured captions and a balanced high-quality subset.
- Curation Pipeline: The curation pipeline comprises filtering, annotation, and sampling, with sampling balancing clips by motion and category into SpatialVID-HQ.
- Data Collection and Preprocessing: The dataset collects 33,443 YouTube videos totaling 21,789 hours and segments them into 3–15 second clips.
- Filtering: Filtering evaluates aesthetic quality, motion intensity, text interference, and luminance to retain clear clips with diverse motion.
- Geometry Information Annotation: MegaSaM provides primary camera estimates, while the annotation pipeline can degrade under extreme scenarios and produce non-metric poses in some cases.
- Geometry Information Annotation: Depth modules are replaced with UniDepth v2 and Depth Anything v2, and dynamic masks are derived through thresholding, contour detection, and SAM2 prompts.
- Motion Instruction Decomposition: Motion instructions are derived from smoothed camera pose sequences and mapped to cinematographic terms such as dolly in, pan left, and truck right.
- Semantic Information Annotation: Structured captions combine VLM scene and motion descriptions with LLM refinement using camera pose priors to improve directional correctness and spatial coherence.
- Dataset Quality: SpatialVID-HQ shows consistently superior quality distributions compared with SpatialVID and Panda70M-test under the same processing pipeline.
4. Dataset Analysis
SpatialVID is curated to improve quality and motion diversity beyond large generic video datasets. Compared with Panda-70M, it provides more consistent visual quality, richer camera motion, and better trajectory coverage.
- SpatialVID-HQ comprises over one thousand hours of high-quality spatial videos after tightening quality thresholds and balancing semantic tags with trajectory statistics.
- SpatialVID and SpatialVID-HQ show more compact Aesthetics, Luminance, and Motion distributions, indicating greater consistency and higher average quality than Panda-70M.
- Over 80% of Panda-70M videos cannot be reconstructed by MegaSaM because of insufficient motion, whereas SpatialVID-HQ increases clips with curved or turning trajectories.
5. SpatialVID Validation Tasks
SpatialVID is validated on camera-controlled generation, scene reconstruction, and camera pose estimation. Across these tasks, the reported results indicate strong camera controllability, visual quality, and generalization to realistic dynamic scenes.
- Camera-Controlled Video Generation: SpatialVID-HQ models achieve the highest camera-control accuracy across Sekai-Real, RealEstate10K, and SpatialVID benchmarks.Models are evaluated with camera pose and text as inputs under identical training configurations.
- Camera-Controlled Video Generation: SpatialVID-HQ models attain the highest CLIP-T and stable improvements across VBench metrics, particularly Imaging Quality.
- Camera-Controlled Video Generation: Qualitative comparisons show SpatialVID-HQ-trained generators produce more realistic and temporally consistent videos with improved camera controllability.
- Scene Reconstruction: GS-LRM is trained separately on RealEstate10K and a clip-matched SpatialVID-HQ subset for novel-view synthesis evaluation on DL3DV and SpatialVID sequences.
- Camera Pose Estimation: Fine-tuning CUT3R and VGGT on SpatialVID yields notable gains on TUM-dynamics, improvements for both models on Sintel except VGGT’s slight ATE regression, and limited changes on Dycheck.Dycheck remains challenging because of high dynamics and complex handheld-jitter camera trajectories.
6. Discussion and Conclusion
SpatialVID provides large-scale real-world videos with aligned geometric and semantic annotations, while its annotation pipeline has defined failure modes in extreme visual and motion conditions.
- Limitations: The annotation pipeline can degrade on object-dominated frames, varying focal lengths, and severe radial distortion because it inherits MegaSaM failure modes.
- Limitations: Predicted camera poses may be non-metric in specific scenarios, while motion-probability-derived masks can perform suboptimally in complex scenes.
- Conclusion: SpatialVID unifies camera motion, depth, and structured motion-aware descriptions across diverse real-world scenes for spatially grounded video and 3D research.
Overview
The SpatialVID pipeline progressively filters in-the-wild videos, reconstructs geometry, and adds structured semantic and motion annotations. Its filtering and annotation stages are designed to retain visually suitable clips with diverse real-world contexts and interpretable camera dynamics.
- Video Curation: Aesthetic filtering discards clips whose average score across sampled frames is below 4.0 on a 0–10 scale.
- Video Curation: Luminance filtering removes clips with average luminance outside [20, 140], excluding overly dark or bright footage.
- Video Curation: OCR filtering removes clips where detected text occupies more than 30% of the frame area.
- Geometric Annotation: MegaSaM generates initial geometric annotations, while MoveDist, RotAngle, and TrajTurns quantify camera travel, viewpoint change, and trajectory turns.
- Semantic Annotation: Gemini-2.0-flash performs visual parsing, and Qwen3-30B-A3B refines calibrated poses into scene abstracts, narratives, semantic tags, and Motion Trends labels.
- Video Curation: Raw videos are filtered for quality defects, dimensions, irrelevant titles, aesthetics, luminance, OCR content, and motion before clip ranking and annotation.
- Data Diversity: The collected data covers diverse indoor and outdoor contexts, including house tours, walking, train rides, and drone flights.
B. Details of Dataset Analysis
This section analyzes SpatialVID’s semantic composition, caption statistics, and qualitative examples to characterize its coverage and annotation richness.
- The analysis examines camera-motion and scene-attribute distributions, caption diversity, and representative dataset samples.
B.1. Semantic Analysis
SpatialVID combines balanced motion coverage, enriched multi-level captions, hierarchical scene tags, and synchronized spatial annotations to support spatially aware video analysis.
- Camera Motion Distribution: SpatialVID-HQ has a more balanced distribution of camera-motion directions than the original dataset, reducing overrepresentation of any single direction.The original dataset includes forward, lateral, and rotational movements but is not well-balanced.
- Caption Length and Enrichment: Both motion and scene captions become substantially longer after LLM refinement, adding richer context and more detailed spatial reasoning.
- Hierarchical Scene Tags: The dataset organizes semantic attributes across weather, time of day, crowd density, lighting, and hierarchical scene type categories.Scene-type subcategories include street, park, interior, and vehicle, while the word cloud emphasizes spatial and motion vocabulary.
- Multi-Level Caption Design: OptCamMotion averages 50.3 words, SceneSummary 28.6 words, and ShotImmersion 89.7 words for progressively richer annotation formats.ShotImmersion combines scene semantics and camera motion for reasoning-intensive tasks such as video understanding and story grounding.
- Structured Annotations: Structured annotations include camera-motion captions, scene descriptions, normalized camera-position data, and serialized motion guidance.The design supports downstream applications including camera control and multimodal spatial reasoning.
C.1. Implementation Details
The validation setup trains camera-controlled generation and spatial reconstruction models on established datasets and SpatialVID-HQ using explicit camera conditioning and structured captions.
- Camera-Controlled Video Generation: Camera-controlled video generation is evaluated on Sekai-Real, RealEstate10K, and SpatialVID-HQ under dataset-specific sampling and caption protocols.SpatialVID-HQ uses all high-quality clips and Immersive Shot Summary captions, while RealEstate10K follows its original split.
- Camera-Controlled Video Generation: The generation models use 382 × 480 inputs, 81-frame sequences, 20K training steps, batch size 32, and an initial learning rate of 1 × 10^-5.Training uses AdamW, cosine decay, and a 2K-step warmup.
- Camera-Controlled Video Generation: Each frame’s 12-parameter camera extrinsic matrix is projected into feature dimension d and combined with visual tokens through a per-block projector.The projector is initialized as an identity mapping to preserve pretrained feature scale.
- CUT3R Fine-tuning: CUT3R is fine-tuned for 6,500 iterations on 4-to-64-frame sequences with the encoder frozen while the decoder and output heads are updated.Training uses batch size 64, learning rate 1 × 10^-6, weight decay 0.05, and frames capped at 512 pixels on the longer side.
C.2. Qualitative Results
Qualitative evaluations indicate that SpatialVID-HQ supports accurate camera-controlled generation and consistent novel-view synthesis across diverse trajectories and scenes.
- Camera-Controlled Video Generation: Models trained on SpatialVID-HQ follow complex camera trajectories while maintaining realistic spatial continuity and dynamic visual coherence.
- Camera-Controlled Video Generation: SpatialVID-HQ training improves prompt understanding and produces more accurate environmental details, including trees and decorations.
- Novel View Synthesis: Novel-view synthesis maintains consistent spatial layouts and detailed texture synthesis across diverse motion trajectories.
- Across the evaluated tasks, the authors report that SpatialVID’s explicit 3D motion control and rich textual semantics provide robust 3D inductive biases.