Source-linked AI summary
Advancing Open-source World Models
Robbyant Team, Zelin Gao, Qiuyu Wang, Yanhong Zeng, Jiapeng Zhu, Ka Leong Cheng, Yixuan Li, Hanlin Wang, Yinghao Xu, Shuailei Ma, Yihang Chen, Jie Liu, Yansong Cheng, Yao Yao, Jiayi Zhu, Yihao Meng, Kecheng Zheng, Qingyan Bai, Jingye Chen, Zehong Shen, Yue Yu, Xing Zhu, Yujun Shen, Hao Ouyang
TL;DR
Existing video generators often lack grounded interaction, long-horizon coherence, and real-time control. LingBot-World addresses these gaps with an open-source, staged world-model framework and reports strong dynamic degree, real-time throughput, and broad controllable generation, while acknowledging long-horizon drift.
Problem
Video generation models often lack grounded causality and interaction, while interactive world models face scarce action data, limited minute-long coherence, and costly inference.
Method
LingBot-World combines hybrid data, hierarchical captions, multi-stage training, causal architecture adaptation, and few-step distillation to create an interactive world simulator.
Results
LingBot-World achieves a dynamic degree score of 0.8857 versus 0.7612 for Yume-1.5 and 0.7217 for HY-World 1.5, while maintaining the best overall consistency.
Takeaways & Limitations
The open-source framework supports promptable global events, action-agent training, and consistent 3D reconstruction from generated videos.
Takeaways & Limitations
Coherent generation length remains insufficient for extended gameplay, with increasing video length causing environmental drifting.
Abstract
from arXiv · showhide
We present LingBot-World, an open-sourced world simulator stemming from video generation. Positioned as a top-tier world model, LingBot-World offers the following features. (1) It maintains high fidelity and robust dynamics in a broad spectrum of environments, including realism, scientific contexts, cartoon styles, and beyond. (2) It enables a minute-level horizon while preserving contextual consistency over time, which is also known as "long-term memory". (3) It supports real-time interactivity, achieving a latency of under 1 second when producing 16 frames per second. We provide public access to the code and model in an effort to narrow the divide between open-source and closed-source technologies. We believe our release will empower the community with practical applications across areas like content creation, gaming, and robot learning.
1 Introduction
LingBot-World addresses the gap between short, passive video generation and persistent, interactive world simulation through an open-source framework. It combines scalable data, staged training, broad embodied-AI applications, and high dynamic degree with long horizons.
- Video generators can render short coherent clips but often lack causality, object permanence, and interaction consequences.
- Interactive world modeling is constrained by scarce action-contingent data, weak minute-long coherence, and costly diffusion sampling.
- LingBot-World uses hybrid real, game, and synthetic data with hierarchical narrative, static-scene, and temporal captions to learn action-contingent dynamics.
- Its three-stage training strategy establishes a video prior, injects world knowledge and action controllability, then adapts the model for real-time interaction.
- The framework supports promptable world events, action-agent training, and consistent 3D reconstruction from generated videos.
- LingBot-World combines general-domain capability, long generation horizons, and high dynamic degree while releasing its code and model weights openly.
2 Data Engine
The data engine builds an interactive training corpus from diverse real, game, and synthetic sources, then expands behavioral and environmental coverage through systematic trajectory collection.
- The data engine unifies acquisition, profiling, and captioning to support novel viewpoints, complex dynamics, and long-horizon planning.
- It combines diverse first- and third-person videos, game recordings paired with controls and camera parameters, and synthetic Unreal Engine data.
- Game data collection synchronizes RGB frames, user controls, and camera trajectories with high-precision timestamps while excluding interface overlays.
- The standardized collection strategy covers navigation, sightseeing, long-tail scenarios, and causal world interaction.
- Unreal Engine rendering supplies precise camera poses and customizable trajectories, expanding motion diversity beyond real-world dataset biases.
- Trajectory generation balances stochastic exploration with behavioral authenticity through procedural paths and imported real-world motions.
2.2 Data Profiling
Data profiling standardizes heterogeneous videos into training-ready assets by combining physical filtering, semantic analysis, geometric annotation, and hierarchical textual conditioning.
- The profiling engine extracts multi-dimensional metadata at three levels of granularity.
- Basic attributes such as duration, resolution, and file size support quality filtering and semantically coherent clip segmentation.
- A vision-language model evaluates brightness, sharpness, motion magnitude, scene type, and camera perspective for precise data selection.
- MegaSAM supplies camera-pose annotations for videos lacking geometric information, providing 3D structural priors for training.
- The profiling layers bridge raw collections and training-ready assets by combining physical, semantic, and geometric metadata.
- Hierarchical captioning provides global narrative, action-decoupled static-scene, and time-aligned dense-temporal descriptions.
3.1 Formulation
LingBot-World formulates simulation as action-conditioned visual-state prediction and progressively evolves a video generator into a long-horizon, causal, real-time interactive system.
- 3.1 Formulation: The model represents video frames as visual states V and aligned control signals A, learning environment transitions driven by agent actions.
- 3.1 Formulation: Its objective predicts future states from history frames and current controls over a horizon L ≥1 rather than only one step.
- 3.1.1 Stage I: Pre-Training — Establishing the General Video Prior: Stage I initializes a general video prior from large-scale open-domain video, providing spatiotemporal coherence, semantic understanding, and high-fidelity scene structure.
- 3.1.2 Stage II: Middle-Training — Injecting World Knowledge & Long-Term Dynamics: Stage II converts the generator into a bidirectional world model and trains it on extended sequences for long-term consistency.
- 3.1.2 Stage II: Middle-Training — Injecting World Knowledge & Long-Term Dynamics: Action signals are incorporated through adaptive normalization so generated trajectories follow user-specified controls.
- 3.1.3 Stage III: Post-Training — Causal Architecture Adaptation & Few-Step Distillation: Stage III shifts to causal autoregressive generation by conditioning on past context, enabling step-by-step inference for interaction.
- 3.1.3 Stage III: Post-Training — Causal Architecture Adaptation & Few-Step Distillation: Block causal attention and KV caching improve inference efficiency while preserving temporal coherence.
- 3.1.3 Stage III: Post-Training — Causal Architecture Adaptation & Few-Step Distillation: Few-step distillation with self-rollout and adversarial optimization preserves action-conditioned dynamics and visual fidelity across extended rollouts.
3.2 Pre-Training
Pre-training establishes a general video prior from large-scale open-domain video data, providing LingBot-World with visual dynamics, semantic understanding, and coherent scene generation for later interactive modeling.
- 3.2 Pre-Training: Large-scale open-domain video pre-training establishes a general prior over visual dynamics for subsequent interactive stages.The prior provides spatiotemporal coherence and open-domain semantic understanding.
- 3.2 Pre-Training: The pretrained model supplies high-fidelity object textures and coherent scene structures as a general visual canvas rather than task-specific physical rules.
- 3.2 Pre-Training: Pre-training enables LingBot-World to generate diverse, coherent, and high-fidelity videos for later world-model development.
3.3 Middle-Training
Middle-training transforms a pretrained video diffusion model into a coherent, interactive world model by extending temporal modeling, incorporating spatial memory, and injecting controllable actions while addressing substantial training-memory demands.
- 3.3 Middle-Training: Action-conditioned, temporally extended sequences transform the pretrained short-clip model into a bidirectional world model.The model generates controllable visual worlds from an image or video, noisy latents, and user-defined actions.
- 3.3 Middle-Training: Long-term video consistency and spatial memory are learned from arbitrary initial states through fundamental world-model training.
- 3.3 Middle-Training: A two-expert MoE assigns global structure and coarse layout to the high-noise expert while the low-noise expert polishes fine-grained details.
- 3.3 Middle-Training: 5-second training sequences are progressively extended to 60 seconds to improve long-term consistency and spatial memory.
- 3.3 Middle-Training: Image-to-video and video-to-video objectives support future-state prediction from single frames and extrapolation beyond observed video.
- 3.3 Middle-Training: Continuous camera rotation and discrete keyboard inputs are fused and injected through AdaLN to guide action-consistent denoising.
- 3.3 Middle-Training: Freezing the main DiT blocks and tuning action adapters preserves the fundamental model’s video synthesis abilities while learning control signals.
- 3.3 Middle-Training: 28B-parameter training on one-minute sequences requires multi-GPU parallelism because model size, token length, and memory-intensive operations exceed single-device capacity.
3.4 Post-Training
Post-training converts the bidirectional world model into a real-time autoregressive generator through causal attention, then uses self-rollout, distribution matching, and adversarial training to reduce long-horizon drift and improve distilled quality.
- 3.4 Post-Training: The bidirectional architecture is converted into an efficient autoregressive model for real-time interactive generation.
- 3.4 Post-Training: The high-noise expert initializes the causal student because it provides stronger dynamics modeling and supports variable-length rollouts.
- 3.4 Post-Training: Block causal attention combines local bidirectional attention with global causality constraints to support autoregressive generation.
- 3.4 Post-Training: Key-value caching reuses representations from previous chunks and computes attention only for newly generated tokens, reducing per-step overhead.
- 3.4 Post-Training: Training partitions noisy video sequences into chunks with independent noise timesteps and distills from selected target timesteps.
- 3.4 Post-Training: Autoregressive inference accumulates drift beyond the training horizon because training and inference distributions mismatch.
- 3.4 Post-Training: Self-rollout training conditions the model on its own generated frames, teaching recovery from generation artifacts and accumulated errors.
- 3.4 Post-Training: Distribution matching distillation and adversarial optimization improve sample quality and temporal consistency using the middle-trained MoE as the real score function.
4 Evaluation
LingBot-World produces high-fidelity, dynamically consistent environments across diverse scenarios, while its fast variant supports real-time generation. Evaluations further demonstrate long-term memory, unobserved-state reasoning, and coherent videos lasting up to ten minutes.
- Diverse Results: LingBot-World-Base generates smooth, logically consistent transitions while handling varied object properties and complex spatial configurations.The results highlight fine-grained environmental dynamics.
- Diverse Results: 16 fps throughput at 480p on one GPU node enables LingBot-World-Fast to preserve structural integrity and physical logic with marginal perceptual degradation.The acceleration trades off theoretical upper-bound quality but avoids significant visual artifacts and mode collapse.
- Emergent Memory Capability: Up to 60 seconds out of view, landmarks such as Stonehenge retain structural integrity without explicit 3D representations.The video-based approach also models non-rigid dynamics such as flowing water and moving pedestrians.
- Emergent Memory Capability: Unobserved-state reasoning preserves spatio-temporal consistency: bridges reflect camera movement and vehicles continue physically plausible trajectories while off-screen.These behaviors indicate simulation of underlying world dynamics rather than pixel memorization.
- Ultra-long Video Generation: Up to ten minutes of video remain temporally coherent without significant degradation in visual quality or narrative consistency.This demonstrates robustness to long-term temporal dependencies.
- Quantitative Analysis: 0.8857 dynamic degree exceeds Yume-1.5’s 0.7612 and HY-World 1.5’s 0.7217, while visual quality and temporal smoothness remain competitive.The comparison uses VBench on 100 generated videos exceeding 30 seconds.
5 Applications
LingBot-World turns video generation into an interactive simulator conditioned on language and actions. Its applications include prompt-driven world events, action-agent exploration, and geometrically consistent 3D reconstruction.
- Interactive Simulation: Conditioning synthesis on natural-language prompts and discrete actions enables interactive simulation and supports downstream embodied-AI applications.The framework demonstrates promptable events, action-agent training, and 3D reconstruction.
- Promptable World Events: Promptable world events branch one initial context into diverse future trajectories steered by semantic text prompts.This replaces passive navigation with reactive simulation.
- Global Events: Global-event prompts modify weather, lighting, or style while preserving scene geometry, motion dynamics, and temporal consistency.Examples include winter, night, pixel art, and steampunk transformations.
- Local Events: Local-event prompts inject objects or agents such as fireworks, birds, and fish with physically consistent and temporally stable behavior.This supports interactive training environments for reasoning about fine-grained physical interactions.
- Action Agent: An action agent predicts future controls from visual observations, converts them into motion trajectories, and drives subsequent world-model video rollouts.The agent is trained on image-action pairs and outputs action chunks for exploration.
- 3D Reconstruction: Generated videos can be converted into high-quality point clouds with strong spatial coherence across frames, providing data for embodied intelligence.Results span indoor, sci-fi, and outdoor scenarios.
6 Conclusion and Discussion
LingBot-World establishes an open-source framework for actionable simulation, combining a data engine, causal transformer, and real-time distillation with applications in agentic actions, world editing, and 3D reconstruction. The report also identifies limits in memory stability, computation, action diversity, interaction precision, generation length, and multi-agent support.
- Summary: A New Open-Source Frontier: The framework bridges video generation and actionable simulation through a scalable data engine, causal transformer, and real-time distillation.Its applications include agentic actions, consistent world editing, and 3D environment reconstruction.
- Limitations: Memory stability is limited because memory emerges from the context window rather than an explicit storage module, causing long-term inconsistencies.The roadmap proposes a dedicated explicit memory module.
- Limitations: Inference remains computationally expensive, requiring enterprise-grade GPUs and limiting accessibility on consumer hardware.The report identifies computational cost as a remaining barrier.
- Limitations: The controllable action space mainly covers navigation and basic movements, lacking diverse complex interactions.The authors aim to expand the action space and improve the physics engine.
- Limitations: Fine-grained object interaction remains difficult because the model lacks precise object-level grounding.Picking up a specific object in a cluttered scene is given as an example.
- Limitations: Extended gameplay suffers from insufficient coherent generation length and scene drifting as environments gradually lose their original structure.The roadmap targets longer generation and reduced drifting.
- Limitations: The current framework supports only single-agent perspectives and does not model multi-agent interactions.Multi-agent simulation remains outside the present scope.
7 Contributors
The report identifies contributors across the base model, post-training, games data, rendering data, data pipeline, applications, and project sponsorship. An asterisk denotes the leaders of each sub-module.
- Contributors: Contributors are organized by sub-module, including the base model, post-training, games data, rendering data, data pipeline, applications, and project sponsorship.The listed roles associate named contributors with each area.
- Contributors: An asterisk denotes the leaders of each sub-module.The notation is defined in the contributor list.