Source-linked AI summary
INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modeling
InSpatio Team, Donghui Shen, Guofeng Zhang, Haomin Liu, Haoyu Ji, Hujun Bao, Hongjia Zhai, Jialin Liu, Jing Guo, Nan Wang, Siji Pan, Weihong Pan, Weijian Xie, Xianbin Liu, Xiaojun Xiang, Xiaoyu Zhang, Xinyu Chen, Yifu Wang, Yipeng Chen, Zhenzhou Fan, Zhewen Le, Zhichao Ye, Ziqiang Zhao
TL;DR
Existing video diffusion models struggle with long-horizon roaming in complex dynamic environments because spatial persistence and visual fidelity remain limited. INSPATIO-WORLD addresses this with STAR and JDMD, achieving high-consistency, high-precision real-time camera control while mitigating synthetic-data appearance degradation.
Problem
Existing video diffusion models can synthesize visually striking short clips but face fundamental challenges during long-horizon roaming in complex dynamic environments.
Method
INSPATIO-WORLD combines a SpatioTemporal Autoregressive architecture with implicit spatiotemporal caching, explicit spatial constraints, and Joint Distribution Matching Distillation.
Results
The STAR architecture achieves high-consistency, high-precision camera control in real time, while JDMD enhances the fidelity of generated regions.
Takeaways & Limitations
INSPATIO-WORLD provides a real-time interactive roaming framework that transforms monocular videos into dynamic, immersive navigation experiences.
Takeaways & Limitations
The system does not persistently encode fine-grained textures in autonomously generated regions and remains challenged by multi-view consistency of dynamic elements during wide-angle omnidirectional transitions.
Abstract
from arXiv · showhide
Building world models with spatial consistency and real-time interactivity remains a fundamental challenge in computer vision. Current video generation paradigms often struggle with a lack of spatial persistence and insufficient visual realism, making it difficult to support seamless navigation in complex environments. To address these challenges, we propose INSPATIO-WORLD, a novel real-time framework capable of recovering and generating high-fidelity, dynamic interactive scenes from a single reference video. At the core of our approach is a Spatiotemporal Autoregressive (STAR) architecture, which enables consistent and controllable scene evolution through two tightly coupled components: Implicit Spatiotemporal Cache aggregates reference and historical observations into a latent world representation, ensuring global consistency during long-horizon navigation; Explicit Spatial Constraint Module enforces geometric structure and translates user interactions into precise and physically plausible camera trajectories. Furthermore, we introduce Joint Distribution Matching Distillation (JDMD). By using real-world data distributions as a regularizing guide, JDMD effectively overcomes the fidelity degradation typically caused by over-reliance on synthetic data. Extensive experiments demonstrate that INSPATIO-WORLD significantly outperforms existing state-of-the-art (SOTA) models in spatial consistency and interaction precision, ranking first among real-time interactive methods on the WorldScore-Dynamic benchmark, and establishing a practical pipeline for navigating 4D environments reconstructed from monocular videos.
1 Introduction
INSPATIO-WORLD addresses long-horizon interactive roaming by combining a spatiotemporal autoregressive architecture with joint distribution matching distillation. The framework targets spatial persistence, synthetic-to-real fidelity, and precise camera control while operating in real time.
- Motivation: Existing video diffusion models struggle with long-horizon roaming because spatial persistence, synthetic-to-real fidelity, and control precision remain limited.These bottlenecks involve scene drift, distribution shifts in visual statistics, and inaccurate execution of user-defined trajectories.
- Overview: INSPATIO-WORLD transforms a monocular reference video into a real-time, interactive 4D world model.The system supports real-time interaction rather than being limited to text and image inputs.
- STAR Architecture: STAR combines an implicit spatiotemporal cache with explicit spatial constraints for consistent generation and high-precision camera control.The cache aggregates reference frames and historical generations within a fixed sliding window, while spatial constraints integrate user interactions into camera-controlled generation.
- JDMD: JDMD uses unified weight sharing across controllable video rerendering and text-to-video distillation to align features with real-world distributions.The method uses real-world distribution guidance to improve generated-region fidelity while preserving motion control.
- Results: 24 FPS is achieved while maintaining exceptional spatiotemporal consistency, and evaluations report superiority over existing generative world models in motion robustness and visual quality.The reported evaluation combines quantitative and qualitative evidence.
2 Related Work
Related work spans video diffusion, camera-controllable and novel-view generation, autoregressive video diffusion, and distribution-matching distillation. Figure 2 summarizes a framework that combines cached spatiotemporal information, geometric constraints, and multi-task distillation.
- Video Diffusion Models: Video diffusion models have evolved from U-Nets toward transformer-based architectures to improve realism and dynamic fidelity.This progress provides a generative backbone for interactive spatiotemporal simulation.
- Novel View Synthesis and Camera Control: Classical novel-view synthesis uses explicit 3D representations such as neural radiance fields or 3D Gaussian splatting, typically requiring multi-view input and per-scene optimization.Diffusion-based approaches instead explore camera-controllable video generation through parameter injection and related mechanisms.
- Novel View Synthesis and Camera Control: Recent open-ended generation methods target unbounded horizons and dynamic scene exploration, including Infinite-World, CameraCtrl II, LingBot-World, and related systems.The passage identifies these methods as part of the broader camera-controllable generation literature.
- Autoregressive Video Diffusion: Autoregressive video diffusion models generate sequences through step-wise conditionals, including hybrids that combine autoregressive and diffusion formulations.These approaches aim to support unbounded-length generation of videos and other continuous sequences.
- Distribution Matching Distillation: Distribution-matching distillation addresses diffusion inference efficiency while retaining alignment with high-fidelity target distributions.The passage frames distribution alignment as a continuing challenge for accelerated video generation.
- Framework Overview: Figure 2 depicts a spatiotemporal cache, depth-based geometric constraints, and shared-weight multi-task distillation supervised by perceptual and motion teachers.The pipeline links reference and historical observations to consistent autoregressive video generation.
3 Method
INSPATIO-WORLD formulates long-horizon interactive roaming as chunk-wise conditional autoregressive generation, combining cached temporal-spatial context with explicit geometric constraints and multi-condition initialization. Its training further combines controllability-focused synthetic supervision with real-world distribution guidance to improve consistency and visual fidelity.
- Problem Formulation: Chunk-wise conditional autoregression generates K-frame blocks from reference context, historical latents, and real-time interaction instructions.The i-th block is conditioned on earlier generated blocks, reference guidance, and interaction terms.
- Spatiotemporal Autoregressive Framework: The framework injects implicit memory states and explicit geometric constraints into a Diffusion Transformer for real-time interactive environment generation.Historical and reference frames provide memory, while geometric information translates user commands into camera control.
- Spatiotemporal Autoregressive Framework: Historical latents preserve local temporal continuity, reference latents anchor long-range scene appearance, and reprojection features with valid masks prevent structural distortion.The cache combines a sliding window of generated latents with globally stable reference anchors; geometric guidance supplies deterministic spatial structure.
- Geometry-Aware Explicit Constraints: User rotation, translation, and perspective instructions accumulate into 6-DoF poses that drive geometric alignment of reference features to the current viewpoint.The pose is recursively updated from the previous camera state, after which reference features are projected using geometric priors.
- Joint Distribution Matching Distillation: JDMD alternates controllable V2V and T2V distillation, using synthetic data for motion control and real-world distributions to calibrate visual fidelity.The combined mechanism emphasizes condition adherence for camera movement and spatiotemporal consistency while improving generated appearance.
4 Experiments
INSPATIO-WORLD is evaluated across WorldScore, long-term image-to-video generation, and camera-controlled video rerendering, using quantitative and qualitative comparisons against state-of-the-art methods. Across these settings, it combines strong control, visual quality, spatial consistency, and real-time efficiency.
- Experimental Setup: The evaluation covers WorldScore next-scene generation, long-term RE10K image-to-video generation, and camera-controlled video rerendering on real-world and synthetic data.WorldScore measures instruction control, spatial stability, and physical dynamics; the other tasks assess long-range camera control, distribution consistency, visual quality, and adherence to reference conditions.
- WorldScore Benchmark: INSPATIO-WORLD (1.3B) achieves a 68.72 overall dynamic score and 81.51 camera control accuracy on WorldScore while reaching 24 FPS real-time generation.The authors report state-of-the-art performance and leading computational efficiency among real-time or interactive methods.
- Long-term Image-to-Video Generation: On 100 RE10K sequences exceeding 150 frames, INSPATIO-WORLD obtains FID 42.68 and FVD 100.55 while substantially improving long-horizon camera motion accuracy.The evaluation uses identical input conditions and the 14B version for consistency with LingBot-World.
- Long-term Image-to-Video Generation: Qualitative RE10K-Long comparisons show baseline failures including structural distortion, static-frame degeneration, and inaccurate camera poses, whereas INSPATIO-WORLD preserves geometry and trajectory control.The comparison presents intermediate and final frames for two scenes as sequence length increases.
- Camera Controlled Video Rerendering: Figure 5 shows higher structural fidelity, better textural detail, and camera trajectories nearly identical to rendered ground truth compared with TrajectoryCrafter, ReCamMaster, and NeoVerse.Each row represents a distinct scene, with reference, warped, and generated final frames arranged from left to right.
- Camera Controlled Video Rerendering: INSPATIO-WORLD achieves state-of-the-art results on both Blender and OpenVid rerendering datasets, outperforming existing methods in FID, FVD, and comprehensive video quality metrics.Its camera control accuracy is comparable to current state-of-the-art models, and it is reported as the only open-source rerendering solution capable of real-time execution.
5 Discussion and Conclusions
INSPATIO-WORLD combines efficient spatiotemporal autoregression, persistent anchoring, explicit spatial constraints, and JDMD for real-time interactive roaming. The system achieves high-efficiency navigation while retaining limitations in long-term generated-region memory and omnidirectional dynamic consistency.
- Contributions: INSPATIO-WORLD integrates an implicit ST-Cache, explicit spatial constraints, and JDMD to address spatial persistence, control precision, and synthetic-to-real fidelity.The framework combines long-term spatiotemporal anchoring with geometric constraints and dual-teacher distillation for motion fidelity and perceptual realism.
- Results: 24 FPS enables high-degree-of-freedom navigation in synthesized virtual worlds while establishing state-of-the-art spatial continuity and visual precision.
- Limitations: Long-term memory remains limited because external anchors and explicit point-cloud memory do not persistently encode fine-grained textures in autonomously generated regions.
- Limitations: Seamless 360-degree dynamic roaming remains unresolved because wide-angle transitions can disrupt multi-view consistency and spatiotemporal coherence of dynamic elements.
- Future Work: Future work targets deeper semantic memory, tighter geometric-textural coupling, and long-range dynamic constraints with stronger physical priors.