PaperlayerView exact revision on arXiv

Source-linked AI summary

ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU

Fan Jiang, Zhaoxu Sun, Mengchao Wang, Ziyu Zhu, Chiyu Wang, Yunpeng Zhang, Wenlin Liu, Yun Wang, Xue Zheng, Rui Sun, Junfeng Ni, Hongyu Pan, Zhongxu Sun, Fei Yu, Zengye Ge, Mengmeng Du, Nianfei Fan, Mingchao Sun, Yu Liu, Yongchang, Yanqing Zhu, Jiahang Wang, Ning Ying, Yuze Xuan, Di Yang, Zhicheng Liu, Zhe Gao, Tingbing Xu, Jiacheng Sui, Wenjin Yang, Junnan Lai, Shufeng Liu, Yuan Liu, Zheng Zhou, Yingliang Peng, Dawei Cao, Kaifeng Sheng, Yuxiang Cai, Fei Lu, Mu Xu, Ning Guo

arXiv:2607.19191v1cs.CVcs.AIcs.LG

TL;DR

Interactive video world models must remain controllable and coherent over long-horizon closed-loop interaction, while practical deployment remains challenging. ABot-World-0 combines multi-source action-video data, bidirectional-to-causal distillation, LongForcing, and a streaming inference stack; it achieves competitive controllability and coherent long-horizon evolution while streaming 720P at up to 16 FPS on one RTX 5090.

  • Problem

    Interactive world models must preserve controllability, state persistence, and coherence as actions continuously change the world during long-horizon interaction.

  • Method

    ABot-World-0 combines multi-source interactive data, unified keyboard control, bidirectional-to-causal distillation, LongForcing, and deployment-oriented streaming inference.

  • Results

    720P streaming reaches up to 16 FPS with 1.2 s action-to-first-frame latency on one RTX 5090, while evaluations show competitive controllability and improved long-horizon visual stability.

  • Takeaways & Limitations

    A single video world model supports unified interactive control, stable long-horizon inference, and real-time desktop deployment across environments and controllable subjects.

  • Takeaways & Limitations

    The deployment stack does not yet use quantization-aware training, which the authors leave for future work to improve the speed–memory–quality trade-off.

Abstract

from arXiv · show

We present ABot-World-0, an action-conditioned video world model for real-time, long-horizon closed-loop interaction, supported by a multi-source data infrastructure spanning AAA games, simulation engines, and internet videos to learn controllable world dynamics. WorldExplorer performs agent-driven collection guided by training feedback, while a unified pipeline applies 14 deterministic quality checks, VLM-based assessment, and synchronized action and text annotation. We progressively distill a bidirectional action-conditioned teacher into a causal student through teacher forcing and ODE distillation, and introduce LongForcing to align long student self-rollouts with an extended-horizon teacher, mitigating accumulated distribution shift and autoregressive drift. Raw keyboard actions provide a unified control interface for scene roaming and third-person character interaction, while reference-character memory provides persistent appearance cues for identity consistency during third-person rollouts. For deployment, we co-design a streaming inference stack with a lightweight VAE decoder, efficient attention, memory-aware scheduling, and low-bit DiT inference. Across optimized low-bit configurations, ABot-World-0 streams 720P video at up to 16 FPS on a single NVIDIA RTX 5090 desktop GPU, with 1.2s action-to-first-frame latency and approximately 19GiB peak VRAM. Experiments on WorldRoamBench and extended interactive rollouts demonstrate competitive controllability and coherent long-horizon world evolution.

1 Introduction

ABot-World-0 treats interactive world modeling as a full-stack problem spanning synchronized multi-source data, action-conditioned causal learning, long-horizon consistency, and efficient real-time deployment. Its system supports continuously controllable world evolution and streams 720P video at up to 16 FPS on a single desktop GPU.

  • Motivation: Interactive world models must maintain coherent state as actions change the world and subsequent observations guide continued interaction.The introduction distinguishes persistent, controllable environments from sequences of visually plausible clips.
  • Data infrastructure: ABot-World-0 combines AAA game data, simulation-engine data, and real-world internet video, with WorldExplorer collecting agent-driven trajectories guided by training feedback.The sources are treated as complementary because passive video, game recordings, and simulations expose different control and diversity properties.
  • Action-conditioned model: The model uses a unified framesynchronous raw-keyboard action interface for scene roaming and character control, with reference-character memory supporting third-person appearance consistency.Source-native controls and pose-derived pseudo-actions are mapped into the same action space during data construction.
  • Long-horizon learning: LongForcing supervises long student self-rollouts with an extended-horizon bidirectional teacher, correcting rollout contexts that short-horizon objectives cover weakly.The learning pipeline first distills a bidirectional teacher into a causal student through teacher forcing and ODE distillation.
  • Real-time deployment: 16 FPS is achieved for 720P streaming with 1.2 s action-to-first-frame latency on a single NVIDIA RTX 5090, within approximately 19GiB peak VRAM.Deployment co-design combines a lightweight VAE decoder, memory-aware scheduling, low-bit DiT inference, efficient low-precision attention, and bounded local-context KV caching.

2 Related Work

Video generation has progressed from bidirectional diffusion models toward causal autoregressive methods that support streaming but face long-horizon drift. Interactive video generation further conditions dynamics on user controls, with discrete keyboard actions offering incremental character and camera control instead of calibrated global trajectories.

  • Video generation: Diffusion models, increasingly using spatiotemporal diffusion transformers, achieve strong video quality by modeling dependencies across entire clips.Earlier video diffusion models commonly used U-Net architectures, whereas recent methods employ diffusion transformers with spatiotemporal attention.
  • Causal generation: Bidirectional full-clip attention improves temporal consistency but prevents incremental low-latency or streaming frame generation.This limitation motivates causal and autoregressive methods that convert or distill bidirectional models.
  • Causal generation: Autoregressive methods generate frames sequentially from prior content, enabling streaming and reducing long-video memory overhead with bounded context or caching.Recent autoregressive work extends beyond early discrete token-based transformers toward other generation paradigms.
  • Long-horizon stability: Autoregressive rollouts suffer exposure bias and accumulated error, causing quality degradation, identity inconsistency, motion stagnation, or collapse to static frames.Existing mitigation strategies include heterogeneous noise scheduling, sliding latent windows, rollout-aligned objectives, and memory-centric mechanisms, but long-range drift can persist.
  • Interactive video generation: Interactive video generation conditions dynamics on user actions, while discrete keyboard actions provide fixed-space local controls for character movement and camera motion relative to the current state.This contrasts with approaches based on calibrated global camera trajectories, whose long-rollout poses may leave the training distribution or create re-anchoring inconsistencies.

3 Data Infrastructure

ABot-World builds training data as an active, multi-source infrastructure spanning AAA games, simulation engines, and internet videos. WorldExplorer, progressive quality filtering, and unified multimodal annotation connect data collection to training feedback and continual refinement.

  • Data Sources: The corpus combines AAA game, simulation-engine, and real-world internet data to cover complementary environments, viewpoints, motion patterns, and domain-generalization signals.Game data provides synchronized controls and observations; simulation data provides geometric precision and environmental control; internet data contributes natural camera dynamics and diverse lighting but requires pose estimation.
  • Quality Filtering: The pipeline applies 14 deterministic quality checks across six dimensions together with VLM-based semantic assessment to remove technically flawed and semantically invalid samples.This quality-control stage is applied systematically to maximize the effective data ratio.
  • Annotation: Surviving clips receive synchronized action labels, structured scene descriptions, and semantic tags in a canonical representation for action- and text-conditioned training.Source-native raw control signals are preserved when available, while source-specific actions are converted into a unified format; game controls retain ground-truth precision through runtime APIs.
  • WorldExplorer: WorldExplorer autonomously collects synchronized multimodal game and simulation data through closed-loop, training-feedback-driven exploration.Its navigation strategy progressively relaxes exploration goals, while adaptive rebalancing targets under-performing scene-action combinations and preserves minimum coverage floors.
  • Data–Model Loop: The curated multi-source corpus supports identity-conditioned third-person generation, with model evaluation feeding subsequent WorldExplorer collection to close the data–model loop.Identity references provide persistent cues for character-consistent video synthesis.

4 ABot-World-0

This section presents the complete ABot-World-0 pipeline, covering interactive world modeling, bidirectional teacher training, and causal distillation. Its core design combines unified raw-keyboard control with reference-character memory and progressive bidirectional-to-causal training for efficient, stable rollout.

  • Pipeline overview: The complete pipeline defines interactive world modeling through action-conditioned inputs and training objectives, then introduces bidirectional teacher training and causal distillation.These components are presented in Sections 4.1–4.3.
  • Unified interactive generation: Raw keyboard inputs serve as the sole action interface for both scene roaming and third-person character interaction.This unifies control across the two interaction modes.
  • Unified interactive generation: Reference-character memory provides complementary appearance conditioning to support character identity consistency during third-person interaction.The memory mechanism supplies persistent appearance cues for the character.
  • Progressive causal training: Teacher forcing and ODE distillation form a progressive bidirectional-to-causal training pipeline for efficient and stable rollout.The pipeline progressively transfers capabilities from a bidirectional teacher to a causal model.

4.1 Formulation

ABot-World-0 formulates interactive world modeling as predicting future video from visual history, actions, and multimodal conditions, with iterative chunk prediction enabling long-horizon rollouts. Its training progressively distills a bidirectional action-conditioned teacher into a causal autoregressive student for interactive deployment.

  • Formulation: The model predicts future video conditioned on past visual observations, future actions, text prompts, and reference images.The formulation treats video as the environment state and incorporates multimodal conditions alongside observations and agent interactions.
  • Formulation: Iteratively predicting and appending future video chunks enables autoregressive rollout of long-horizon trajectories.Each predicted chunk contains the next L video frames.
  • Bidirectional Teacher Learning: A bidirectional teacher jointly generates a full-horizon video from an initial frame, complete action sequence, and multimodal conditions.Its non-causal architecture enables information exchange across the temporal horizon, supporting visual consistency, motion quality, and action alignment.
  • Progressive Distillation: The training pipeline progressively converts the bidirectional teacher into a causal autoregressive student through Teacher Forcing, Causal ODE Distillation, and LongForcing.This conversion addresses the teacher’s incompatibility with interactive autoregressive rollout while enabling few-step, low-latency inference and stable long-horizon interaction.

4.2 Bidirectional Teacher Training with Action Control Injection

This section trains a bidirectional video model by injecting raw keyboard actions and reference images into a pretrained Wan2.2 backbone. Actions are temporally and spatially aligned with DiT latent tokens, while reference-character memory supports identity consistency during long-horizon third-person rollouts.

  • Bidirectional teacher and action control: Full-parameter fine-tuning injects action controls and reference images into the pretrained Wan2.2 backbone to learn action-conditioned visual dynamics.The approach targets the missing mapping from discrete controls to visual state transitions while retaining pretrained world priors.
  • Bidirectional teacher and action control: Raw keyboard input is the sole control signal, represented per frame as an 8-dimensional multi-hot vector over W/A/S/D and I/J/K/L.The keys control character or camera movement and camera rotation, and align directly with game recordings and user intent.
  • Bidirectional teacher and action control: Packing every 4 frame-level actions produces 32-dimensional temporal action tokens aligned with one latent frame after VAE compression.The packing matches the VAE temporal patch size of 4.
  • Bidirectional teacher and action control: An Action Control Adapter aligns action features with DiT patchified latents through PixelUnshuffle, convolution, and residual blocks, enabling token-wise additive injection.PixelUnshuffle uses the VAE spatial compression ratio, while the convolution kernel size and stride match the DiT spatial patch size.
  • Reference-character memory: Reference-character memory uses canonical images encoded into identity-memory tokens, prepended to the video sequence to mitigate identity drift in long-horizon third-person rollouts.Fixed negative temporal RoPE indices distinguish memory from video tokens, and video tokens attend to memory tokens.

4.3 Causal Progressive Student Distillation

The paper progressively distills a bidirectional teacher into an efficient causal generator through teacher forcing, causal ODE distillation, and LongForcing. These stages adapt full-context video modeling to autoregressive deployment, reduce denoising steps, and improve long-horizon rollout stability.

  • Progressive Distillation: The three-stage pipeline uses teacher forcing, ODE distillation, and LongForcing to transform a high-quality bidirectional teacher into an efficient causal student.The bidirectional teacher is unsuitable for real-time interaction because it requires full-sequence generation and multi-step denoising.
  • Teacher Forcing: Teacher forcing initializes the causal student from the trained bidirectional teacher and trains it to generate future video chunks using only previously observed visual context.Clean ground-truth history latents and causal attention masking prevent access to future visual contexts while retaining the teacher’s visual priors and long-range dynamics.
  • ODE Distillation: Causal ODE distillation trains a few-step model to approximate the Stage 1 causal diffusion model’s probability-flow ODE endpoint under deployment-available causal conditioning.The reference and distilled models share the same causal factorization, enabling substantially fewer denoising steps without future visual observations.
  • LongForcing: LongForcing improves rollout stability by matching the student and an extended-horizon teacher on long-horizon conditional video distributions.It exposes the student to self-generated contexts where small prediction errors accumulate and applies distribution-level corrective supervision during final DMD.

4.4 Full-Stack Co-Design for Real-Time World Rollout

ABot-World-0 treats real-time deployment as a full-stack co-design problem balancing throughput, action-to-first-frame latency, and peak GPU memory. The resulting stack combines optimized decoding, transformer execution, attention, context management, and runtime scheduling to support sustained high-resolution rollout on one GPU.

  • Deployment co-design: Full-stack co-design jointly optimizes temporal generation, VAE decoding, low-precision DiT execution, attention and RoPE kernels, bounded context memory, and runtime scheduling.The objective balances sustained generation throughput, action-to-first-frame latency, and peak GPU memory usage.
  • Decoding and scheduling: LightVAE reduces decoding time and peak memory, while memory-aware scheduling stages modules by execution order and memory requirements to lower peak VRAM without changing the architecture.Chunk-wise causal generation decodes a complete latent chunk before its first response frame becomes available, so faster decoding directly reduces action-to-first-frame latency.
  • Transformer and context optimization: Fast-RoPE, efficient attention, low-bit mixed precision, and bounded or quantized KV caches reduce transformer computation, positional-encoding overhead, memory traffic, and long-rollout cache growth.The bounded local-context cache uses rolling eviction, keeping its footprint independent of total rollout duration, while numerically sensitive components remain in higher precision.
  • Deployment results: 16 FPS is reached across optimized low-bit configurations while peak VRAM remains below 19.3 GiB on a single NVIDIA RTX 5090 GPU.The reported 19.281 GiB configuration underscores that peak VRAM depends on the complete runtime configuration rather than an individual operator.

5 Evaluation

ABot-World-0 is evaluated quantitatively on WorldRoamBench, through a 60-second LongForcing ablation, and with qualitative stress tests covering extended rollouts, out-of-domain control, and physical interaction. The results show competitive benchmark performance, sustained long-horizon coherence and controllability, generalized action following, plausible physical responses, and reduced visual error accumulation with LongForcing.

  • WorldRoamBench: WorldRoamBench evaluates action controllability, visual plausibility, physical consistency, and temporal memory against Genie 3, HappyOyster, LingBot-World, and HY-World 1.5.The comparison follows the benchmark protocol and reports selected sub-dimensions across Action, Visual, Physics, and Memory.
  • Long-horizon stress tests: Hour- and day-scale rollouts preserve recognizable environments, viewpoints, and controllable actors while continuing to respond to input actions under accumulated-error conditions.The hour-scale evaluation includes five independent rollouts, while day-scale checkpoints retain scene structure and active motion.
  • OOD control: OOD tests show action-consistent motion and environmental coherence when both scenes and controllable characters fall outside the training distribution.The model applies a unified action representation across diverse novel scenes and characters.
  • Physical interaction: Physical-interaction tests produce plausible collisions, contact effects, persistent environmental changes, and geometric constraints without interpenetration.Examples include pushing a box, water disturbances, snow footprints, wall blocking, and railing collisions.
  • LongForcing ablation: LongForcing retains higher HPSv3 and lower artifact-related metrics than the Causal-Forcing-style baseline later in 60-second rollouts, indicating less visual error accumulation.Both variants use student self-rollouts and final-stage DMD, while LongForcing adds extended-horizon teacher supervision.

6 Discussion and Future Work

The discussion identifies distribution shift, explicit-action conditioning, and whole-stack systems optimization as central to reliable interactive world modeling, while outlining extensions for richer controls, persistent memory, and broader hardware support.

  • Discussion: Explicit keyboard actions provide discrete, temporally aligned control signals, so additive injection at patch embedding supplies reliable conditioning while preserving the pretrained visual prior.More elaborate conditioning may remain useful for latent actions, continuous camera trajectories, or semantic instructions.
  • Discussion: Long-horizon drift is framed as distribution shift: accumulated visual-context errors move autoregressive generation beyond states covered during short-horizon training.Sink-based stabilization and fixed reference frames can delay drift, but excessive anchoring may restrict motion and scene evolution.
  • Discussion: Interactive performance requires joint optimization of VAE decoding, attention, memory transfer, and KV-cache memory and bandwidth costs beyond reducing DiT denoising steps.The passage attributes single-desktop-GPU real-time inference to optimizing these components together.
  • Future work: Future work targets structured conditioning for richer actions and semantic events, multi-scale LongForcing, persistent scene memory, and more efficient decoding and context management.These extensions aim to improve long-horizon consistency and broaden high-resolution real-time generation across consumer hardware.

7 Conclusion

ABot-World-0 unifies action-conditioned world modeling, stable long-horizon generation, and real-time desktop deployment in a single video world model. Its multi-source training, LongForcing-based rollout alignment, and optimized inference stack support interactive control and extended coherent world evolution.

  • Unified interactive modeling: A single model supports scene navigation and character control across environments and controllable subjects using multi-source action-video data.The data spans AAA games, simulation engines, and internet videos; reference-character conditioning preserves appearance cues during long-horizon third-person generation.
  • Long-horizon training: LongForcing aligns causal student self-rollouts with an extended-horizon teacher, extending supervision to long-horizon contexts encountered during repeated autoregressive generation.The student is progressively obtained from a bidirectional action-conditioned teacher through teacher forcing and ODE distillation.
  • Real-time deployment: 720P streaming reaches up to 16 FPS with 1.2 s action-to-first-frame latency on a single RTX 5090, while peak VRAM remains below 19.3 GiB.The deployment stack combines a lightweight VAE decoder, low-bit inference, efficient attention, and memory-aware scheduling.
  • Long-duration evaluation: Day-scale rollouts preserve recognizable visual quality, active dynamics, and scene coherence without observable collapse at evaluated timestamps.WorldRoamBench results also show competitive performance across action following, trajectory following, visual quality, physical mechanics, and memory retention.
  • Applications: The model provides an open foundation for interactive content creation, game simulation, agent learning, and embodied-AI research by extending generated worlds beyond predefined scene boundaries.This conclusion follows from combining unified interactive control, stable long-horizon inference, and consumer-grade real-time deployment.

8 Contributors

The project credits sponsors and contributors across foundation-model, data, AI-infrastructure, benchmark, and engineering teams. The listed contributors span the research, data, systems, evaluation, and implementation efforts.

  • Project sponsors are Mu Xu and Ning Guo.
  • The Foundation Model Team comprises Fan Jiang†, Zhaoxu Sun, Mengchao Wang, Ziyu Zhu, Chiyu Wang, Yunpeng Zhang, Wenlin Liu, Yun Wang, Xue Zheng, Rui Sun, and Junfeng Ni.
  • The Data Team includes Hongyu Pan†, Zhongxu Sun, Fei Yu, Zengye Ge, Mengmeng Du, Nianfei Fan, Mingchao Sun, Yu Liu, and Yongchang.
  • AI infrastructure, benchmarking, and engineering are credited to their respective teams and listed contributors.The AI Infra Team is led by Yanqing Zhu†; the Benchmark Team by Zhicheng Liu†; and the Engineering Team by Junnan Lai†, with each passage providing the full membership list.
Loading 2607.19191v1…