Source-linked AI summary
Infinite Worlds with Versatile Interactions
Zelin Gao, Qiuyu Wang, Jiapeng Zhu, Jingye Chen, Zichen Liu, Qingyan Bai, Jiahao Wang, Yufeng Yuan, Hanlin Wang, Yichong Lu, Ka Leong Cheng, Haojie Zhang, Jian Gao, Tianrui Feng, Yuzheng Liu, Yao Yao, Yinghao Xu, Xing Zhu, Yujun Shen, Hao Ouyang
TL;DR
Interactive world models must remain stable over long horizons while responding to live, high-fidelity controls, but existing systems face drift and compute constraints. LingBot-World-Infinity combines causal world-model training, long-rollout distillation, and a pilot–director harness to address these requirements. The resulting system sustains an unbounded, drift-free world at 720p and 60 fps, with rich actions and environmental events, while remaining open-sourced.
Problem
Existing interactive world models face long-horizon error accumulation and computationally expensive high-fidelity interaction, with prior systems often sacrificing resolution, smoothness, or control.
Method
The paper trains a causal, action-conditioned world model, distills it using long self-rollout trajectories, and wraps it in a pilot–director agentic harness.
Results
The system sustains an unbounded, drift-free interactive world at 720p and 60 fps, with diverse actions, environmental changes, and no perceptible decay during an uninterrupted session exceeding one hour.
Takeaways & Limitations
The released model and harness provide an open interactive world-modeling system combining long-horizon generation, real-time operation, and rich controllability.
Takeaways & Limitations
The model lacks genuine long-term memory, so revisited regions may be regenerated rather than recalled, and real-time high-fidelity operation still requires substantial computational resources.
Abstract
from arXiv · showhide
We present LingBot-World 2.0 (also known as LingBot-World-Infinity), an advanced iteration of LingBot-World featuring four distinct upgrades. (1) Our model achieves an unbounded interaction horizon while maintaining consistent output quality, benefiting from a carefully crafted causal pretraining paradigm. (2) Through distilling a real-time variant from the base model, our system guarantees rapid response time, sufficient to drive 720p video streams at 60 fps. (3) Compared to the previous version, this update introduces highly diverse interactive elements, comprising a broader spectrum of actions (e.g., attacking, archery, spell-casting, and shooting) alongside a richer variety of text-driven events. (4) We pioneer the integration of an agentic harness within the domain of world modeling, wherein a pilot agent is tasked with planning and executing character behaviors, while a director agent is responsible for synthesizing novel environmental elements as the scene progresses. Additionally, to facilitate a shared experience, we develop an interface that permits multiple players to simultaneously immerse themselves in this vivid world simulator. We pair our primary 14B model with a lightweight 1.3B counterpart, which supports effortless deployment on a single GPU.
1 Introduction
LingBot-World-Infinity addresses long-horizon drift and the computational cost of high-fidelity interaction through causal training, distillation, and an agentic harness. It supports unbounded generation, real-time 720p interaction, diverse actions, and self-sustaining world progression.
- Motivation: Existing interactive world models degrade over long horizons as autoregressive errors accumulate, while high-fidelity live interaction strains compute budgets.Observed failures include smeared textures, warped geometry, scene drift, and trade-offs among resolution, smoothness, and control.
- Core approach: A causal backbone is trained to resist error accumulation, then distilled into a practical model that sustains an explorable world without a fixed endpoint.The authors report an hour-long uninterrupted session with no visible quality decay.
- Core results: The real-time model renders an unbounded interactive world at 720p and 60 fps while maintaining visual quality over more than one hour of continuous generation.This combines the reported deployment target with the long-horizon stability verification.
- Interaction: The system supports combat, archery, spell-casting, shooting, and on-the-fly environmental changes such as snow and rain.These interactions extend beyond navigation to character actions and environmental events.
- Agentic harness: A pilot agent selects controllable-character behaviors while a director agent adds fresh content, props, and events as exploration proceeds.Together, the agents make the experience self-sustaining, goal-directed, and open-ended without author scripting each moment.
2 Data pipeline
The data pipeline converts heterogeneous videos into temporally localized, visually grounded training annotations. It combines mixed-source curation, filtering, profiling, hierarchical event annotation, and chunk-wise caption optimization.
- Data sources: The corpus combines egocentric videos, synthetic game and Unreal Engine data, and large-scale web videos to provide visual diversity and controllable interaction signals.Synthetic data supplies scene geometry and aligned signals such as jumping, attacking, driving, and flying.
- Curation: Raw videos are normalized, segmented into temporally coherent clips, filtered for technical quality, and profiled by a vision-language model.Technical filters provide low-cost quality control, while VLM profiling organizes clips by quality and semantic content.
- Annotation motivation: Global captions alone create a train–inference mismatch because interactive conditioning varies over time and requires localized instructions.The pipeline therefore combines video-level context with temporally localized chunk-wise descriptions.
- Event annotation: The annotation process independently tracks subject visibility, motion, interaction, environmental dynamics, and static scene state across temporal chunks.Decoupling event tracks reduces ambiguity by preventing unrelated changes from being merged.
- Caption optimization: Active track states are composed into chunk-wise captions and refined for terminology consistency, temporal smoothness, and visual grounding.Future-revealing, speculative, and redundant expressions are removed before training.
3 LingBot-World-Infinity
LingBot-World-Infinity formulates interactive world simulation as a causal, action-conditioned video process and trains it in two stages: causal pre-training followed by few-step distillation. Its design combines camera and textual controls, hybrid attention, and rollout-aware distillation to support long-horizon generation with high visual fidelity and reduced drift.
- 3.1 Formulation: The simulator generates each future world state autoregressively from historical visual context and current user inputs.The causal formulation conditions each state only on prior frames and inputs up to the current step.
- 3.2 Pre-Training: Causal World Model: The pre-training stage supports camera poses and prompts as actions for controlling boundless, high-fidelity video generation.Camera poses are represented with Plücker embeddings, while textual control uses chunk-wise prompts.
- 3.2 Pre-Training: Causal World Model: MoBA attention appends a bidirectional component to teacher forcing, enabling autoregressive generation while preserving visual fidelity and reducing overfitting.Its corresponding cross-attention design uses causal prompt access to prevent future information leakage.
- 3.2 Pre-Training: Causal World Model: The training objective constructs noisy latents and predicts flow velocity conditioned on clean historical frames, camera poses, and prompts.The target velocity follows rectified-flow interpolation.
- 3.3 Post-Training: Few-Step Distillation: Post-training combines consistency distillation and distribution matching distillation to produce a few-step real-time generator while suppressing long-horizon rollout drift.Consistency distillation reduces denoising steps, while DMD trains on long self-rollout trajectories and the student-induced state distribution.
4 Deployment
The deployment stack combines inference and streaming optimizations with a Director-Pilot harness for responsive, physically coherent interactive world simulation. It supports semantic, object-centric, and text-driven interventions, while refining frames for sharper and smoother output.
- System overview: The inference stack targets low latency, rich interactions, high visual quality, and a user-friendly interface through four coordinated components.These components are systematic optimization, an agentic interaction harness, visual quality enhancement, and user interface design.
- Systematic Optimization: System-level optimizations improve throughput and responsiveness through compiler and attention-kernel acceleration, hybrid parallelism, asynchronous decoding, and incremental streaming.Dynamic KV-cache scheduling is also identified as a key component of the real-time system.
- Agentic Interaction Harness: The Director performs causal reasoning and proposes event updates, while the Pilot grounds them in physically consistent video rollouts.Users can interact through semantic or object-centric actions or introduce high-level textual events.
- Interaction modes: Users can interact directly with scene context or manipulate tracked objects through SAM-assisted proposals and maintained spatial consistency.The two modes support holistic semantic interactions and precise object-centric manipulation.
- World intervention: Text prompts let users alter global conditions or inject local entities, with the VLM integrating changes into the ongoing narrative and physical space.Examples include changing time or weather and spawning birds or other creatures.
- Visual quality: The spatio-temporal refiner sharpens decoded frames and synthesizes intermediate frames for smoother motion and higher frame rate with limited added latency.Its spatial and temporal stages run as TensorRT engines with multi-GPU parallelism and asynchronous streaming.
5 Results
The evaluation examines both the real-time distilled model and its causal pretrained teacher across interactive capabilities, visual quality, and long-horizon stability. The distilled model remains open and real-time at high resolution, while the backbone sustains coherent generation substantially longer than competing causal models.
- Evaluation setup: The evaluation covers the distilled deployment model and the causal pretrained backbone, with emphasis on interactive performance and long-horizon capabilities.The distilled model is compared with proprietary and open-source systems, while the backbone is evaluated against prior causal world models.
- Causal Distilled Model: The distilled model matches or exceeds the strongest closed-source baselines in visual quality while uniquely sustaining high-resolution real-time generation without visible degradation.The comparison also evaluates temporal stability, action responsiveness, and real-time throughput.
- Long-horizon stability: Over one hour of uninterrupted generation shows no perceptible visual-quality decay, while the world remains coherent and explorable throughout the rollout.The experiment uses a single continuous session on a 60:00 timeline covering 20 distinct scenarios.
- Interactive capabilities: The model supports a broad action space and diverse scenes and interaction patterns beyond navigation.Qualitative results are presented in Figures 7 and 8.
- Causal Pretrained Model: The causal pretrained backbone consistently outperforms competing models and retains sharp textures, stable geometry, and coherent scenes over substantially longer rollouts.The authors attribute this advantage to anti-drift training that discourages compounding errors.
6 Conclusion and Discussion
LingBot-World-Infinity combines durable, real-time interactive world generation with broad actions and agentic orchestration. The authors identify long-term memory, identity and style consistency, physical understanding, and computational cost as remaining limitations.
- The model sustains an unbounded, drift-free world at 720p and 60 fps while supporting combat, archery, spell-casting, shooting, and environmental changes.
- A pilot and director agentic harness orchestrates the model into a self-sustaining, goal-directed, open-ended experience.
- Limitations: Long-term memory remains limited: revisited regions are regenerated rather than recalled, so persistence concerns appearance rather than identity.
- Limitations: Over very long rollouts, character appearance and overall art style may drift even when the scene remains coherent.
- Limitations: Pixel-based learning lacks explicit geometry and collision, causing occasional physically implausible intersections or pass-throughs.
- Limitations: Although the distilled model runs in real time, real-time high-fidelity world modeling on commodity hardware requires further efficiency gains.
7 Authors
The paper lists its authors and identifies their contribution areas through superscripts. These areas include pre-training, post-training, deployment, data, project sponsorship, and project leadership.
- The author list includes contributors marked with superscripts that identify their contribution areas.
- The authors are Zelin Gao, Qiuyu Wang, Jiapeng Zhu, Jingye Chen, Zichen Liu, Qingyan Bai, Jiahao Wang, Yufeng Yuan, Hanlin Wang, Yichong Lu, Yinghao Xu, Haojie Zhang, Tianrui Feng, Yuzheng Liu, Ka Leong Cheng, Jian Gao, Yao Yao, Xing Zhu, Yujun Shen, Hao Ouyang, Yipengjing Sun, Liangxiao Hu, Yue Yu, Yihang Chen, Zikun Dai, Leyi Xu, Jiayi Zhu, Yihao Meng, Yanhong Zeng, and Yangyan Li.
- The listed contribution areas are pre-training, post-training, deployment, data, project sponsors, and project lead.