PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling

Wenqiang Sun, Haiyu Zhang, Haoyuan Wang, Junta Wu, Zehan Wang, Zhenwei Wang, Yunhong Wang, Jun Zhang, Tengfei Wang, Chunchao Guo

arXiv:2512.14614v2cs.CVcs.GR

TL;DR

Real-time interactive world modeling has struggled to combine low latency with long-term geometric consistency. WorldPlay addresses this gap with dual action control, reconstructed context memory, and memory-aligned distillation, achieving 24 FPS 720p streaming video with long-term consistency and broad scene generalization. The paper demonstrates a practical framework for interactive world modeling while identifying longer-duration generation and broader interaction as open challenges.

  • Problem

    Existing interactive world models have not simultaneously achieved low-latency generation and high long-term geometric consistency.

  • Method

    WorldPlay combines Dual Action Representation, Reconstituted Context Memory, and Context Forcing, which aligns teacher and student memory contexts during distillation.

  • Results

    24 FPS at 720p is achieved with long-term geometric consistency under streaming user control, with strong generalization across diverse scenes.

  • Takeaways & Limitations

    WorldPlay provides a systematic framework for control, memory, and distillation in consistent, interactive virtual worlds.

  • Takeaways & Limitations

    The model generates videos of approximately 30 seconds, while scaling to minutes or hours remains a significant challenge.

Abstract

from arXiv · show

This paper presents WorldPlay, a streaming video diffusion model that enables real-time, interactive world modeling with long-term geometric consistency, resolving the trade-off between speed and memory that limits current methods. WorldPlay draws power from three key ingredients. 1) We use a Dual Action Representation to enable robust action control in response to the user's keyboard and mouse inputs. 2) To enforce long-term consistency, our Reconstituted Context Memory dynamically rebuilds context from past frames and uses temporal reframing to keep geometrically important but long-past frames accessible, effectively alleviating memory attenuation. 3) We also propose Context Forcing, a novel distillation method designed for memory-aware model. Aligning memory context between the teacher and student preserves the student's capacity to use long-range information, enabling real-time speeds while preventing error drift. Taken together, WorldPlay generates long-horizon streaming 720p video at 24 FPS with superior consistency, comparing favorably with existing techniques and showing strong generalization across diverse scenes. Project page and online demo can be found: https://3d-models.hunyuan.tencent.com/world/ and https://3d.hunyuan.tencent.com/sceneTo3D.

1. Introduction

Interactive world modeling seeks both real-time responsiveness and long-term geometric consistency, a combination that existing speed- or memory-focused methods do not simultaneously achieve. WorldPlay addresses this challenge with action control, reconstructed memory, and memory-aligned distillation, producing real-time consistent video across diverse scenes.

  • Interactive world models must predict future video in response to user keyboard commands while preserving low latency and long-term geometric consistency.
  • WorldPlay formulates interactive modeling as next-chunk prediction of 16 frames conditioned on user actions for streaming video generation.
  • Dual Action Representation combines keyboard inputs with continuous camera poses to support robust control and accurate location caching.Keyboard actions provide plausible scale-adaptive movement, while camera poses provide spatial locations but can destabilize training under scene-scale variance.
  • Reconstituted Context Memory rebuilds context from past frames using spatial and temporal proximity, then temporally reframes retrieved frames to reduce long-range decay.Temporal reframing rewrites positional embeddings so geometrically important distant frames remain accessible.
  • Context Forcing aligns teacher and student memory contexts during distillation, enabling real-time generation without eroding long-term memory.The alignment also alleviates error accumulation over long sequences.
  • 24 FPS at 720p is achieved with long-term geometric consistency under streaming user control, alongside generalization across diverse real and stylized scenes.The model uses 320K real and synthetic videos and supports applications including 3D reconstruction and promptable events.

2. Related Work

Related work develops video generation and interactive world models through diffusion, autoregression, action conditioning, memory, and distillation. However, long-horizon geometric consistency with real-time interaction remains unresolved.

  • Diffusion and autoregressive video-generation methods support efficient or few-step generation, while large-scale training enables modeling and manipulation of visual worlds.
  • Table 1 frames WorldPlay as combining long-horizon video generation, flexible action control, real-time interactivity, and long-term geometric consistency in a general-domain model.The table uses “Con.” for continuous action and “Dis.” for discrete action.
  • Interactive world models predict future states from current and past states using discrete, continuous, or textual action signals.
  • Explicit 3D approaches enforce spatial consistency through reconstruction and rendering, but their long-term consistency depends heavily on reconstruction quality.
  • Implicit-memory methods retrieve historical context using field of view, while context compression provides fixed-length consistency for interactive generation.

3. Method

WorldPlay combines action representations, reconstituted memory, and memory-aligned distillation in a chunk-wise autoregressive diffusion model for real-time interactive video generation with long-term geometric consistency.

  • 3. Method: WorldPlay predicts 16-frame video chunks autoregressively, conditioning each chunk on past observations, action sequences, current action, and a world-describing image or text prompt.The model is designed for streaming generation rather than full-sequence prediction.
  • Dual Action Representation for Control: Dual Action Representation combines discrete keyboard actions with continuous camera poses to support plausible movement and precise spatial memory retrieval.Keyboard signals adapt across scene scales but are ambiguous for revisiting locations, whereas camera poses provide locations but introduce scale-related training instability.
  • Reconstituted Context Memory for Consistency: Reconstituted Context Memory rebuilds temporal and spatial context from past chunks, using recent frames for motion smoothness and geometrically relevant non-adjacent frames to limit drift.Spatial sampling uses field-of-view overlap and camera distance.
  • Reconstituted Context Memory for Consistency: Temporal Reframing reassigns positional encodings to retrieved context frames, keeping their relative distance to the current chunk small despite long temporal gaps.This addresses RoPE extrapolation artifacts and weakened influence from distant spatial memories.
  • Context Forcing: Context Forcing aligns teacher and student memory contexts during distillation, preserving long-term consistency while mitigating error accumulation with 4-denoising-step generation.The method uses memory-augmented student self-rollouts, a memory-augmented bidirectional teacher, and progressive increases in rollout length.

4. Experiments

Experiments evaluate WorldPlay on short- and long-term video generation, action control, geometric consistency, efficiency, and downstream reconstruction. Across these settings, the model maintains strong visual quality, control, and long-term consistency while supporting real-time inference and diverse interactions.

  • Evaluation Protocol: 600 test cases from DL3DV, game videos, and AI-generated images evaluate visual quality, action precision, and long-term consistency.Short-term evaluation compares generated frames with ground truth; long-term evaluation uses cycle trajectories and return-path comparisons.
  • Main Results: WorldPlay achieves superior short-term visual fidelity and competitive control accuracy, while remaining more stable in challenging long-term scenarios.The evaluation compares LPIPS, PSNR, SSIM, Rdist, and Tdist against action-controlled diffusion baselines with and without memory.
  • Main Results: WorldPlay achieves improved long-term geometric consistency through Reconstituted Context Memory and further reduces error accumulation through Context Forcing.Methods without memory perform poorly on long-term consistency, while explicit 3D-cache methods are constrained by depth accuracy and alignment.
  • Efficiency: Tailored parallelization and quantization for DiT and VAE significantly increase inference throughput while preserving real-time interactivity.The acceleration strategies are evaluated quantitatively in Table 4.
  • Qualitative Results: WorldPlay generalizes across first- and third-person real and stylized scenes, supports promptable events, and enables point-cloud reconstruction from generated videos.The qualitative comparisons report stronger visual fidelity and long-term geometric consistency than the compared methods.
  • Ablations: The dual-action representation improves overall control by combining discrete keys for plausible movement with continuous camera poses for fine-grained location information.Discrete-only inputs struggle with movement distance and rotation degree, while pose-only training is harder to converge because of scale variance.

5. Conclusion

WorldPlay combines real-time interaction with long-term geometric consistency in a systematic framework for control, memory, and distillation. Its architecture also shows potential for richer text-triggered interactions, while longer-duration generation and broader dynamics remain open challenges.

  • Conclusion: WorldPlay provides real-time interaction and long-term geometric consistency for customizable worlds from a single image or text prompt.The conclusion frames control, memory, and distillation as a systematic framework for consistent interactive virtual worlds.
  • Limitations: Generating videos beyond approximately 30 seconds, preventing autoregressive error accumulation during training, and supporting broader multi-agent physical interactions remain open directions.The paper also identifies failures of FOV-based memory retrieval under significant occlusions.

A. Training and Inference Details

Training proceeds through staged action-control, memory, and Context Forcing phases, followed by streaming inference with KV caching. The implementation uses chunk-wise autoregression, reconstituted memory, and a curated multi-category dataset.

  • Stage One: Action Control: Stage One trains dual-action control for 30K iterations bidirectionally and 30K additional iterations with block-causal attention for autoregressive action modeling.The staged design trains on 61-frame sequences organized into four chunks.
  • Stage Two: Memory: Stage Two trains bidirectional and autoregressive action models with variable-length context memory under a Flow Matching framework.The bidirectional model generates four chunks while using the remaining context as memory.
  • Stage Three: Context Forcing: Stage Three distills the bidirectional teacher into the autoregressive student using progressive increases in generated latent length.The student and teacher use learning rates of 1e-6 and 2e-7, respectively.
  • Context Forcing: The Context Forcing algorithm self-rolls out chunks with reconstituted memory, then aligns teacher context before score and flow-matching updates.The algorithm progressively increases maximum chunk length during training.
  • Inference: During inference, WorldPlay reconstitutes context memory for each chunk, denoises through multiple steps, appends outputs, and uses a KV cache for streaming generation.Camera poses can be converted into discrete actions, while discrete actions can be converted into predefined relative translations and rotations.
  • Data Organization: The dataset organization records four data categories, their sources, action-annotation availability, clip counts, and final-dataset ratios.These categories support the training and evaluation setup.

B. Dataset

The dataset combines real-world, synthetic, and game-video sources with diverse camera trajectories and revisit patterns. Filtering, reconstruction, rendering, and annotation procedures are used to improve quality and support generalization.

  • Dataset Composition: The dataset deliberately combines simulation-engine and real-world data with static and dynamic environments to support strong model generalization.The collection is curated for diversity and quality.
  • Real-World 3D Scenes: DL3DV processing adds 60K high-quality real video clips with balanced movement speeds through reconstruction, trajectory rendering, quality filtering, and video repair.The workflow addresses limited camera-speed and trajectory diversity in the original videos.
  • Synthetic and Simulation Data: Synthetic and game-video sources contribute 50K rendered clips and 170K player-recorded clips from first- and third-person AAA games.Synthetic clips use complex customized trajectories, while game recordings come from a dedicated platform.
  • Processing Pipeline: Long videos are segmented into 30-to-40-second clips, annotated with descriptive text, assigned camera poses when possible, and filtered for pose collapse.Filtering removes videos with erratic adjacent-frame camera positions or rotation angles.
  • Camera Trajectories: Complex and diverse camera trajectories, including many revisit trajectories, support learning precise action control and long-term geometric consistency.Figure 11 illustrates the collected camera trajectories.

C.1. More Qualitative Results

WorldPlay supports precise control across complex actions and diverse agents while preserving long-term consistency and constant-time streaming generation. Context forcing further improves the student model’s performance and real-time interaction.

  • C.1. More Qualitative Results: WorldPlay follows composite actions and intricate trajectories, including rotations and alternating movement sequences, across virtual environments.The paper attributes this enhanced control to the dual action representation.
  • C.1. More Qualitative Results: WorldPlay controls different agent types, including humans and animals, enabling them to roam within scenes.
  • C.1. More Qualitative Results: 637 frames of generated results retain high visual quality and long-term consistency, including consistency between frame 1 and frame 252.
  • C.1. More Qualitative Results: Context memory keeps generation time per chunk constant as video length grows, enabling real-time interactivity.
  • C.1. More Qualitative Results: Context forcing mitigates error accumulation, maintains or surpasses the student model’s long-term consistency, and reduces inference steps.The comparison reports improved overall performance and real-time interaction.

C.4. Ablation for Memory Size

The memory-size ablation examines how spatial and temporal memory affect image quality, temporal continuity, overall performance, and teacher memory requirements.

  • C.4. Ablation for Memory Size: Larger spatial memory slightly improves PSNR, while larger temporal memory better preserves pretrained temporal continuity and improves overall performance.
  • C.4. Ablation for Memory Size: Larger spatial memory can significantly increase teacher memory because adjacent chunks may have different spatial memories while sharing temporal memory.

C.5. Evaluation on VBench

WorldPlay achieves superior VBench performance across diverse metrics, particularly consistency, motion smoothness, and scene generalizability.

  • C.5. Evaluation on VBench: WorldPlay achieves superior performance on VBench, with outstanding results in consistency, motion smoothness, and scene generalizability.Baselines receive the same image and action inputs for long-horizon video generation.

C.6. Evaluation on WorldScore

WorldPlay achieves the highest average WorldScore among compared models and is evaluated through controllability, generation quality, user preference, promptable events, and video continuation. These results connect benchmark performance with diverse interactive and generative capabilities.

  • C.6. Evaluation on WorldScore: WorldPlay achieves the highest average WorldScore among compared models on 2,000 diverse static test cases.The benchmark evaluates controllability and generation quality across novel views using an input image, text prompt, and camera trajectory.
  • C.6. Evaluation on WorldScore: The user study compares visual quality, control accuracy, and long-term consistency using paired videos generated from identical initial images and actions.The study selects 300 cases from diverse benchmarks and 300 customized trajectories.
  • C.6. Evaluation on WorldScore: KV-recache refreshes cached key–value states after text-prompt changes while preserving motion and visual cues for temporal continuity.
  • C.6. Evaluation on WorldScore: WorldPlay generates follow-up content consistent with an initial video clip in motion, appearance, and lighting, supporting stable video continuation.
Loading 2512.14614v2…