Source-linked AI summary
Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control
Mingyang Chen, Shengdong Chen, Xiaoxiao Fu, Bosheng Gong, Haoyuan Guo, Bowen Li, Jiawen Li, Kejun Li, Tianpeng Li, Yin Liu, Haoze Sun, Zeyang Tian, Meng Wang, Xinmiao Wu, Jiangqiao Yan, Zining Zhao
TL;DR
Playable generated worlds need controls that let users both navigate and alter unfolding events while responding to visual feedback. Zing-0.5 combines joint keyboard-text conditioning with event-scale distillation and context-preserving streaming; it reports 81.0 overall and 88.5 consistency across 158 WBench Navigation cases. The paper also identifies persistent world consequences and explicit state tracking as unresolved boundaries.
Problem
Generated-world interaction needs to support exploration, event intervention, and feedback-guided action rather than movement and observation alone.
Method
Zing-0.5 combines magnitude-aware keyboard inputs and online text conditioning with a segment-level teacher, block-level causal student, and context-preserving streaming.
Results
81.0 overall and 88.5 consistency across 158 WBench Navigation cases are reported for Zing-0.5 across 158 image-conditioned cases.
Takeaways & Limitations
The joint-control demonstration shows a text-directed event change during continued navigation without restarting generation, and the released artifacts support further work on playable generated worlds.
Takeaways & Limitations
The examples do not establish whether event consequences persist through later interactions or viewpoint changes, and the model does not separately track entity states or transition rules.
Abstract
from arXiv · showhide
We introduce Zing-0.5, a 5B autoregressive world model designed for playability: users can explore generated worlds, influence unfolding events, and respond to the resulting feedback through joint keyboard and online text control. Our approach brings together three technical contributions: (1) Unified action and text conditioning, combining magnitude-aware keyboard inputs with temporally aligned text instructions and jointly annotated videos to learn navigation and event control within the same sequence; (2) Event-scale supervision for incremental generation, using a segment-level teacher trained on connected multi-prompt videos to supervise a block-level causal student through distribution-matching distillation; and (3) Low-cost real-time interaction, combining four-step generation with context-preserving streaming to support 832 x 480 inference at 24 FPS at an estimated server rental cost of approximately USD 0.009 per stream-minute. Zing-0.5 achieves an overall score of 81.0 and a consistency score of 88.5 across 158 WBench Navigation cases. A joint-control demonstration shows a text-directed event change during continued navigation without restarting generation. We release the model weights, inference code, and Zing-SGLang serving implementation to support further work on playable generated worlds.
1 Introduction
Zing-0.5 targets playable generated worlds by combining continuous keyboard navigation with online text control, enabling users to influence events while continuing exploration. Its architecture aligns event-scale supervision with incremental generation and supports real-time interaction, achieving strong WBench Navigation scores.
- Motivation: Playability requires users to explore worlds, intervene in events, and use resulting feedback to decide subsequent actions.The motivation distinguishes navigation and observation from richer intervention in events and behavior.
- Results: 81.0 overall and 88.5 consistency across 158 WBench Navigation cases establish Zing-0.5’s reported navigation results.The Figure 2 session separately demonstrates online text control during continued navigation.
- Unified action and text conditioning: Zing-0.5 combines magnitude-aware keyboard inputs, temporally aligned text instructions, and jointly annotated videos for navigation and event control in one sequence.The model continues from existing visual context while users navigate and issue event-directed instructions.
- Event-scale supervision: A segment-level teacher trained on connected multi-prompt videos supervises a block-level causal student through distribution-matching distillation across multiple generation blocks.Rollout and replay, long-sequence adaptation, and history perturbation prepare the student for extended interaction.
- Real-time interaction: Four-step generation and context-preserving streaming support 480p inference at 24 FPS at an estimated server rental cost of approximately $0.009 per stream-minute.The runtime retains visual context across prompt updates using cache reuse, lightweight local decoding, and bounded media queues.
2 Related Work
Related work develops interactive world models, language-guided control, temporal prompt conditioning, continuous autoregressive generation, and efficient real-time serving. Zing-0.5 combines these directions by aligning action and text inputs during ongoing generation while training across different temporal partitions.
- Interactive world models: Interactive world models establish navigation and action response as central capabilities for generated worlds across games and broader scenes.Prior systems generate visual continuations from user actions so users can observe outcomes and choose subsequent inputs.
- Language-guided world interaction: Language-guided systems extend interaction beyond directional navigation to scene changes, events, and character behavior.Prior work places semantic instructions within interactive video interfaces, including time-aligned language actions.
- Multi-prompt video generation: Unlike offline multi-prompt video generation, online sessions accept future instructions as interaction proceeds and continue from existing visual context.Zing aligns action and text conditions on separate timelines and uses jointly annotated videos for navigation and text-directed changes.
- Continuous autoregressive generation: Continuous autoregressive systems address accumulated history errors through approaches including history corruption, self-rollouts, and rollout-replay training.Zing combines long-sequence adaptation, history perturbation, and generated-history training while using different temporal partitions for teacher and student.
- Efficient generation for real-time interaction: Efficient interactive generation combines few-step prediction with deployment techniques such as KV caching, lightweight decoding, cache reuse, and bounded media queues.Zing uses four-step generation and an optimized streaming pipeline for real-time delivery.
3 Zing-0.5
Zing-0.5 combines diverse visual and control data with unified magnitude-aware action and temporally aligned text conditioning for interactive generation. Its architecture and training address joint control, incremental generation, and visual stability, though prolonged training can introduce noise and reduced color saturation.
- 3.1 Data Construction: Zing-0.5 combines image–text, video–text, gameplay, real-world, and synthetic videos to preserve visual capabilities and learn interactive control.Joint-control sequences align captions and continuous action strengths with video frames, while 30-second gameplay clips extend training history.
- 3.1 Data Construction: Magnitude-aware directional actions represent movement and view changes with continuous intensities, keeping control consistent between training and inference.Recorded keyboard, mouse, controller, and camera-derived signals are mapped to common directional channels; direct keyboard conditioning avoids an intermediate camera-pose representation.
- 3.2 Model Architecture: The lightweight action encoder averages controls within latent-frame windows, adds causal control features to visual tokens, and contributes 3.68M parameters.It uses magnitude embeddings, a residual MLP, causal temporal convolutions, and a zero-initialized projection before the Transformer stack.
- 3.2 Model Architecture: Temporally aligned prompt intervals let text control subject, scene, and subject–scene interactions while keyboard actions continue within the same sequence.Visual tokens attend to the prompt assigned to their interval, and the model supports prompt changes during ongoing generation.
- 3.3 Progressive Training and Distillation: Joint training reduces motion-magnitude fluctuations and severe visual degradation, but prolonged training can produce high-frequency noise and lower color saturation.The reported stability improvements are accompanied by these quality artifacts under extended training.
4 System Infrastructure
Zing-0.5’s infrastructure packs variable-length multimodal data efficiently, uses memory-aware attention and parallelism, and streams bounded-context video in real time.
- Data pipeline: Token-budgeted variable-length batches pack images, clips, and long sequences without padding to the longest sample, while coordinated workers perform decoding, alignment, and cache-backed encoding.Packed tensors carry sequence lengths, block identifiers, positions, masks, prompt spans, and token-aligned metadata.
- Training systems: Variable-length FlashAttention, sparse FlexAttention masks, sequence parallelism, all-to-all attention exchange, FSDP, and activation checkpointing support long-sequence training.The 5B backbone consumes the packed tensor contract directly.
- Serving: 832×480 streams run at a client-visible 24 FPS, with an unpaced measurement of 24.63 FPS on eight RTX 5090 GPUs.Each GPU hosts a self-contained model, cache, and decoder replica, avoiding cross-GPU latent transfer.
- Generation: Four denoising steps generate four latent frames that decode into 16 video frames, while resident model components and reused conditioning metadata reduce serving overhead.The native VAE is used only for initial-image encoding; recurrent blocks use the lighter TAEHV decoder.
- Streaming: The media pipeline uses direct local decoding, bounded non-blocking fragment queues, worker-based browser decoding, and completion barriers to preserve live playback and session order.The ring favors the live edge under backpressure.
- Context management: A bounded visual KV cache combines a fixed prefix sink with a sliding recent-context window, retaining initial context while discarding older unprotected entries.Prompt updates replace text K/V values while retaining visual K/V context.
5 Results
Zing-0.5 is evaluated quantitatively on WBench Navigation and qualitatively through sessions combining navigation, scene changes, subject behavior, and cross-scene exploration.
- 5.1 Quantitative Results: 81.0 overall and 88.5 consistency are reported across 158 WBench Navigation cases.The benchmark uses official aggregation across quality, setting, interaction, consistency, and physical plausibility; scores use a 0–100 scale with higher better.
- 5.2 Qualitative Results: Text instructions trigger scene events such as a tree releasing golden sparks and a candy house melting while surrounding visual context remains coherent.The examples extend control beyond navigation to scene dynamics and subject–environment interaction.
- 5.2 Qualitative Results: Exploration examples include navigating a boat through a sea of clouds and following a character through a mirror into a garden and outdoor path.The sequences are recorded as single sessions with timestamps.
- 5.2 Qualitative Results: Joint sessions combine keyboard navigation with online instructions that introduce environmental changes including rainbows and fiery projectiles.These demonstrations pair ongoing exploration with text-directed event changes.
- 5.2 Qualitative Results: Subject-directed instructions produce behaviors including a character rising into flight and a flying mount breathing fire during keyboard-controlled sessions.Both changes unfold within continued navigation rather than as isolated demonstrations.
6 Discussion and Limitations
The discussion frames joint control as a basis for playable participation but identifies persistent world consequences and explicit state tracking as unresolved requirements.
- Joint Control for Playable Worlds: Joint keyboard and text control lets users alter events while continuing exploration, responding to visual feedback with movement and new instructions.The intended interaction is a continuous exchange in which users help shape generation as it unfolds.
- Meaningful Interaction Requires Lasting Consequences: The reported demonstrations show visible responses only within selected intervals and do not establish whether consequences persist across later interactions or viewpoint changes.Persistent consequences are needed for users to learn environmental behavior and build later decisions on earlier outcomes.
- Visual History Does Not Fully Specify World State: Zing-0.5 does not separately track entity states or enforce transition rules, so facts outside the current visual view may remain implicit or unavailable.A bounded KV cache and autoregressive errors can cause later continuations to conflict with earlier events.
- Visual History Does Not Fully Specify World State: Long-sequence training and history perturbation improve exposure to extended and imperfect context but do not explicitly maintain persistent facts or enforce interaction rules.A continuation may remain visually plausible while conflicting with an earlier event.
- Persistent Worlds Need Architectural Support: Future architectures should retain and consistently update relevant facts across viewpoint and instruction changes, with evaluation testing whether users can later continue from prior consequences.Capacity and longer context should be assessed alongside interaction reliability rather than video quality alone.
- Persistent Worlds Need Architectural Support: Preserving the consequences of user influence is identified as the next step toward sustained, meaningful participation in generated worlds.Reliable consequences support expectations, planning, and building on earlier choices.
7 Conclusion
Zing-0.5 targets playable worlds through continuous joint keyboard and text control, with event-scale distillation for incremental generation. It supports real-time interaction and achieves strong WBench Navigation scores, while persistence of consequences remains an open challenge.
- Magnitude-aware keyboard inputs and online text instructions let users explore worlds, influence events, and respond to feedback within one continuous stream.
- A segment-level teacher trained on connected multi-prompt videos supervises a block-level causal student, spanning event learning across multiple incremental generation blocks.
- 81.0 overall and 88.5 consistency across 158 WBench Navigation cases demonstrate Zing-0.5's reported navigation performance.
- Four-step generation and context-preserving streaming support 480p inference at 24 FPS for approximately $0.009 per stream-minute.
- State and rule persistence remain open challenges because later interactions must preserve consistency with earlier actions and generated outcomes.
Contributions and Acknowledgments
The acknowledgments credit collaborators and supporting teams for research, engineering, product, operations, and launch contributions.
- The authors thank the Wan, training and distillation, and WBench teams for making relevant research and implementations available.
- Loopit's engineering, product, and operations teams supported preparation and launch, including user experience improvements, capability showcases, and demo selection.