PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamX-World 1.0: A General-Purpose Interactive World Model

DreamX Team, Yancheng Bai, Rui Chen, Xiangxiang Chu, Rujing Dang, Hao Dou, Bingjie Gao, Qiwen Gu, Siyu Hong, Jiachen Lei, Geng Li, Jifan Li, Ruimin Lin, Qingfeng Shi, Bingze Song, Lei Sun, Jing Tang, Ruitian Tian, Jun Wang, Jiahong Wu, Pengfei Zhang, Shen Zhang, Jiashu Zhu

arXiv:2606.16993v1cs.CV

TL;DR

Interactive world models must follow camera controls, preserve scenes across long horizons, and support events across diverse visual domains. DreamX-World 1.0 combines geometry-aware conditioning, scene memory, long-rollout training, and alignment, achieving the highest camera-control and overall scores in its basic evaluation while maintaining competitive visual quality.

  • Problem

    Interactive world models must preserve scene consistency during camera changes, revisits, and prompted events across photorealistic, game-style, and stylized domains.

  • Method

    DreamX-World 1.0 combines multi-source geometry-aware data, E-PRoPE camera conditioning, scene memory, autoregressive long-horizon training, and post-distillation alignment.

  • Results

    DreamX-World 1.0 achieves the highest camera-control and overall scores while maintaining competitive visual quality in the basic evaluation.

  • Takeaways & Limitations

    DreamX-World 1.0 establishes a promising foundation for next-generation interactive world models through coordinated data, training, evaluation, and inference improvements.

  • Takeaways & Limitations

    Long-horizon visual and geometric consistency remains difficult, and control signals may conflict when events are incompatible with future caption-specified observations.

Abstract

from arXiv · show

DreamX-World 1.0 is a general-purpose interactive text/image-to-video world model for controllable long-horizon generation. It supports camera navigation, revisits to previously observed regions, and promptable events across photorealistic, game-style, and stylized domains. Our data engine combines camera-accurate Unreal Engine rendering, action-rich gameplay recordings, and real-world videos with recovered camera geometry. For camera control, we introduce E-PRoPE, a lightweight variant of projective positional encoding that retains PRoPE's projective camera geometry while applying camera-aware attention to spatially reduced tokens. We convert a bidirectional video generator into a few-step autoregressive world model using causal forcing, DMD-style distillation, and long-rollout training. Training on self-generated long-horizon contexts exposes the model to its own generated history and reduces the style and color drift that accumulates across autoregressive chunks. Memory-Conditioned Scene Persistence retrieves earlier views through camera-geometry-based retrieval, while residual recycling makes the conditioning path less sensitive to imperfect memory latents. Event Instruction Tuning adds composable event control, and reinforcement learning alignment recovers camera control and visual quality after distillation. With mixed-precision DiT execution, residual reuse, 75\%-pruned VAE decoding, and asynchronous pipeline parallelism, DreamX-World 1.0 reaches up to 16\,FPS on eight RTX\,5090 GPUs. On our 5-second basic evaluation, DreamX-World 1.0 achieves a camera-control score of 73.75 and an overall score of 84.76, outperforming HY-WorldPlay 1.5 and LingBot-World in overall score, which achieve 80.79 and 80.45, respectively.

1 Introduction

DreamX-World 1.0 is presented as a general-purpose interactive world model designed to combine controllable camera motion, persistent scene state, event interaction, and long-horizon generation. Its pipeline integrates multi-source data, efficient geometric conditioning, memory retrieval, autoregressive training, reinforcement-learning alignment, and optimized serving.

  • Motivation: DreamX-World 1.0 addresses interactive-world-model requirements for user control and scene-state preservation over long horizons.Interactive systems must respond to controls, maintain consistent viewpoints under camera changes, and preserve previously observed content during revisits.
  • Data engine: The data engine combines Unreal Engine trajectories, action-rich gameplay recordings, and real-world videos with recovered camera poses across multiple visual domains.Geometric filtering and normalization provide a common representation spanning photorealistic, game-style, and stylized domains.
  • Scene persistence: Geometry-based memory retrieval and residual recycling support scene persistence when earlier views leave the local context or memory latents are imperfect.The memory-conditioned stage supplies non-local visual evidence for revisiting previously observed regions.
  • Camera control: E-PRoPE applies projective attention to spatially reduced tokens, retaining comparable trajectory-following performance to PRoPE while reducing inference latency by approximately 30%.It is introduced as an efficient camera-conditioning mechanism that limits the computational cost of the video backbone.
  • Interaction and long-horizon generation: Event Instruction Tuning adds structured multi-entity event control, while causal forcing, DMD, and long student rollouts convert the bidirectional model into a few-step autoregressive generator.Training on generated history reduces style and color drift across chunks; reinforcement learning then restores camera control and video quality after distillation.
  • Efficient serving: Up to 16 FPS is achieved on eight RTX 5090 GPUs through mixed-precision DiT execution, residual reuse, pruned VAE decoding, and asynchronous pipeline parallelism.The serving optimizations enable streaming generation after the model’s alignment stage.

2 Data

DreamX-World’s training data unifies UE-generated, real-world, and game videos through shared annotation, geometric processing, and quality-control pipelines. The dataset supports camera/action supervision, diverse observation modes, and composable event descriptions.

  • Data sources: The unified data pipeline combines UE-generated, real-world, and game videos to balance precise camera and action annotations with visual and domain diversity.Real-world data provides visual diversity but often lacks reliable annotations, while synthetic and game data offer precise control with limited domain coverage.
  • UE data generation: UE data provides first-person, third-person, and event observations under a unified schema with controllable motion and per-frame ground-truth annotations.First-person exploration, character-following views, and object interactions capture camera motion, agent behavior, and visible state changes.
  • UE data generation: Validated UE trajectories are rendered offline with poses, actions, and metadata, while distributed GPU execution supports checkpoint resumption and automatic failure recovery.The two-stage design discovers high-quality trajectories before rendering, avoiding wasted computation on invalid or low-motion clips.
  • Quality control: The three-stage quality-control pipeline filters clips, cleans camera geometry, and adds captions and attribute tags for downstream filtering and training.Camera cleaning densifies sparse real-world poses and checks trajectory consistency, while tags cover aesthetics, motion, scene category, style, subjects, and motion category.
  • Event annotations: Event Instruction Tuning uses cleaned clips with visible state changes, pairing global scene descriptions with time-aligned entity-level records for single-object and composable events.Event records specify entity references, predicates, spatial anchors, and temporal intervals, while interactions such as collision and handoff are described explicitly.

3 Progressive Training Pipeline · 3.1 Camera-Aware Training

DreamX-World 1.0 is progressively adapted from Wan2.2-TI2V to support camera control, memory conditioning, event interaction, and autoregressive generation. Camera-aware training introduces E-PRoPE, which preserves projective camera geometry while reducing attention computation through spatial token downsampling and lower-dimensional projections.

  • 3 Progressive Training Pipeline: DreamX-World 1.0 initializes from Wan2.2-TI2V and progressively adds camera control, memory conditioning, event interaction, and autoregressive generation.
  • 3.1 Camera-Aware Training: Pose-annotated videos train the bidirectional model for explicit 6-DoF camera control absent from the Wan2.2-TI2V inputs.PRoPE conditions attention on inter-camera frustum relationships and camera-agnostic token positions.
  • 3.1 Camera-Aware Training: PRoPE encodes full projective camera geometry in one submatrix and standard rotary position embeddings in a second submatrix applied to attention queries and keys.
  • 3.1 Camera-Aware Training: Direct PRoPE nearly doubles DiT computational cost because its extra attention modules operate layer-wise over long-video tokens.The method targets view-dependent high-level semantics rather than full-resolution fine-grained semantic modeling.
  • 3.1 Camera-Aware Training: E-PRoPE downsamples tokens spatially and projects them into a lower-dimensional query/key/value space, making camera control more efficient during training and inference.It uses only the projective submatrix, upsamples the output, adds it to the original DiT attention output, and trains the PRoPE modules with the DiT backbone frozen.
  • 3.1 Camera-Aware Training: During inference, downstream models can use a pretrained PRoPE component plug-and-play even when trained without it, indicating robust geometric bias.
  • 3.1 Camera-Aware Training: Table 1 evaluates PRoPE and E-PRoPE on Omni-WorldBench, with E-PRoPE reported to achieve comparable camera-control performance while improving computational efficiency.Latency is averaged for generating a 5-second, 1280 × 720 video on 8 NVIDIA H20 GPUs.

3.2 Memory-Conditioned Scene Persistence

Memory-Conditioned Scene Persistence conditions each target prediction on recent history and geometry-retrieved earlier views, packing them into a shared DiT self-attention stream. Geometry-aware temporal placement and residual recycling address non-local consistency and imperfect generated memories.

  • Conditioning and supervision: The DiT jointly uses recent denoised history frames and clean latent memory frames retrieved from earlier history to predict target frames.Training uses a rectified-flow objective with loss computed only on target latent frames.
  • Alternative injection mechanisms: Cross-attention and VACE-style conditioning consistently produce worse generation performance than the memory-conditioning approach.These mechanisms are evaluated as alternative memory-injection strategies.
  • Conditioning and supervision: Memory frames, recent history frames, and target frames share the same DiT self-attention stream, while supervision applies only to target prediction.The residual-recycling path perturbs conditioning tokens without changing the supervised target.
  • Geometry-based retrieval: Geometry-based retrieval selects history frames using camera pose and view overlap, rather than temporal distance alone, to match the target view.Retrieved frames retain RoPE embeddings corresponding to their original temporal locations, preventing distant memories from appearing immediately adjacent to the target.
  • Residual recycling and robustness: Conditioning-token error injection narrows the training–inference gap by perturbing generated-memory inputs while keeping the target latent clean.This teaches the model to use sampled memories when helpful and fall back to its learned prior when memories contain explicit errors.

3.3 Event Instruction Tuning for Composable Events · 3.4 Autoregressive Long Video Generation and Distillation

DreamX-World 1.0 enables composable multi-object event control through structured natural-language instructions and converts a bidirectional generator into a few-step autoregressive model for stable, camera-controllable long-video generation. Long-rollout training, DMD distillation, and image-conditioned supervision address drift, control degradation, and temporal coherence over sequences up to one minute.

  • 3.3 Event Instruction Tuning for Composable Events: Event Instruction Tuning lets users specify regions, object actions, and interactions, producing all requested events in one forward pass.The tuning uses unchanged DiT architecture and introduces event semantics through natural-language text conditioning.
  • 3.3 Event Instruction Tuning for Composable Events: Composable events handle multiple objects with distinct actions and mutual interactions in one generation.This capability targets coordinated changes such as pedestrians, vehicles, signals, and interacting characters.
  • 3.3 Event Instruction Tuning for Composable Events: Structured event prompts cover both the global scene and per-entity dynamics while mixing event and non-event clips preserves general world-generation capability.Event semantics enter exclusively through the text-conditioning interface, without architectural changes.
  • 3.4 Autoregressive Long Video Generation and Distillation: A bidirectional model is distilled into a few-step autoregressive generator that streams from generated history while preserving visual quality and camera controllability.Causal forcing trains the autoregressive model on large-scale high-quality video data near the bidirectional model’s visual distribution.
  • 3.4 Autoregressive Long Video Generation and Distillation: Long-rollout adaptation with Infinity-RoPE supports extended autoregressive context and reduces identity drift, background mutation, and weakened prompt or motion control.Training uses long sequences and local temporal windows.
  • 3.4 Autoregressive Long Video Generation and Distillation: E-PRoPE camera-controlled distillation and repeated long-video DMD training improve camera control and motion smoothness across autoregressive chunks.Student rollouts are matched to a bidirectional E-PRoPE teacher over local temporal windows sampled from long videos.
  • 3.4 Autoregressive Long Video Generation and Distillation: Up to one minute of stable long-duration inference is achieved while maintaining camera controllability and temporal coherence.I2V DMD uses the decoded first latent frame as the teacher’s image condition for supervising the camera-controlled autoregressive student.

3.5 Reinforcement Learning

Reinforcement learning is applied after DMD distillation to recover video quality and camera following while preserving stable long-horizon, few-step generation. The method uses gradual DiffusionNFT updates driven by rewards from short clips sampled from long-horizon rollouts.

  • Rollouts and rewards: The model generates several long-horizon rollout candidates, while short sampled clips provide rewards and DiffusionNFT training.Backpropagation through short temporal windows keeps GPU memory practical while preserving autoregressive context in the full rollout.
  • Motivation: RL post-training enhances video quality and strengthens camera following after DMD distillation.DMD distillation can degrade both properties when denoising steps are reduced.
  • Stable optimization: Gradual model updates stabilize RL after DMD distillation and avoid early collapse.Very few denoising steps make strong reward updates highly unstable, so the model changes step by step.
  • Results: RL improves camera-command following and generated-video visual quality without compromising stable long-horizon generation or few-step inference efficiency.The post-training stage retains the key strengths of DMD distillation while recovering quality and controllability.

4 Inference Acceleration

DreamX-World enables interactive streaming through chunk-wise autoregressive generation with a rolling KV cache and chunk-relative camera controls. Its deployment combines DiT and VAE optimizations with asynchronous pipeline parallelism, reaching up to 16 FPS on eight RTX 5090 GPUs.

  • Streaming autoregressive inference: Each video chunk starts from noise, uses the distilled few-step sampler and rolling KV cache, then writes generated tokens back for subsequent chunks.The model carries autoregressive history instead of regenerating previous video content.
  • Streaming autoregressive inference: For camera-controlled I2V, only the first chunk differs from T2V: its first frame is replaced by the input image, while later chunks use the same rolling-cache procedure.The input image anchors generation to the reference frame.
  • Streaming autoregressive inference: Camera controls use chunk-relative poses, referenced to the first frame initially and the previous chunk’s last frame thereafter.This keeps camera conditioning aligned with the current autoregressive context over long sequences.
  • DiT and VAE acceleration: DiT acceleration combines INT8 SageAttention, FP8 AngelSlim FFN quantization, sequence parallelism, fused Triton kernels, and denoising-residual reuse.These optimizations reduce activation memory, intermediate allocations, kernel-launch overhead, and selected denoising work.
  • DiT and VAE acceleration: 75% pruning reduces single-chunk VAE decoding to approximately 0.25 seconds, while asynchronous pipeline parallelism overlaps decoding with control reception, cache updates, and next-chunk DiT denoising.Eight RTX 5090 GPUs jointly execute DiT denoising and VAE decoding, reaching up to 16 FPS.

5 Evaluation

The evaluation suite measures DreamX-World-1.0-5B across camera control, perceptual quality, long-horizon behavior, memory consistency, and human preference. It shows strong qualitative coherence, leading overall performance in long-horizon rollouts, stronger revisit memory across most abstraction levels, and favorable human preferences over both baselines.

  • Evaluation setup: The evaluation jointly probes camera controllability, perceptual quality, long-horizon behavior, memory consistency, and human preference through qualitative, quantitative, revisit-based, and blind-study protocols.Basic evaluation uses 5-second clips, long-horizon evaluation uses approximately 30-second rollouts, and memory is tested through revisit protocols.
  • Qualitative evaluation: Qualitative results show smooth, temporally coherent transitions across diverse scene types, camera trajectories, and visual styles.The evaluation also reports maintained visual quality as the camera moves through generated scenes.
  • Long-horizon evaluation: DreamX-World-1.0-5B achieves the highest overall score of 70.41 on 30-second rollouts, exceeding HY-WorldPlay 1.5 at 68.85 and LingBot-World at 67.43.It also obtains the best imaging quality and artifact detection scores, indicating stronger long-horizon visual fidelity than the larger competing models.
  • Memory consistency: DreamX-World-1.0-5B achieves the highest gains on pixel-level, perceptual, semantic, and place-recognition revisit metrics, while HY-WorldPlay 1.5 leads on SP-Match and CLIP-Video.The revisit suite uses trajectories that return to previously visited regions and evaluates consistency at multiple abstraction levels.
  • Human preference: DreamX-World-1.0-5B wins or ties 57.5/14.4/28.1 against HY-WorldPlay 1.5 and 61.9/10.6/27.5 against LingBot-World in overall preference.It also records visual-quality win rates of 57.5% and 61.3% and artifact-detection win rates of 59.4% and 56.2% against the two baselines, respectively.

6 Related Work

Related work spans interactive world models, camera-controlled video generation, long-horizon memory, efficient sampling, reinforcement learning, and evaluation beyond frame-level quality. DreamX-World builds on these directions by combining their core ideas for controllable, efficient, and persistent video-world generation.

  • Video Generation and Interactive World Models: Interactive world models condition future video observations on user inputs or agent actions, extending diffusion-based video generation beyond frame synthesis.Diffusion models have progressed from image synthesis to complex cinematic video generation.
  • Camera-controlled Video Generation: Camera-controlled video generation uses explicit motion features, camera embeddings, or geometric representations to control camera and object motion.MotionCtrl separates camera and object motion, CameraCtrl injects camera trajectories, and AC3D studies 3D camera control in video DiTs.
  • Long-horizon Generation and Memory: Long-horizon autoregressive generation faces exposure bias and accumulated errors, motivating training on model outputs and robustness to imperfect context.Self-Forcing trains directly on model outputs, while Stable Video Infinity injects prediction errors into model inputs.
  • Efficient Sampling: Efficient interactive video systems combine few-step distillation, causal generation, and KV caching, alongside component-level system optimizations.ParaVAE distributes VAE decoding across devices, while DreamX-World combines autoregressive generation with quantized DiT execution, residual reuse, parallel VAE decoding, and asynchronous serving.
  • Reinforcement Learning: Reinforcement learning helps diffusion and flow-matching models optimize non-differentiable objectives including perceptual quality and prompt alignment.DDPO formulates denoising as a sequential decision process, Flow-GRPO extends online policy optimization to flow-matching models, and DiffusionNFT fine-tunes the forward process.
  • World Model Evaluation: World-model evaluation is shifting from frame-level video quality toward controllability, consistency, dynamics, and interactive response.WorldScore evaluates prescribed camera trajectories, while Omni-WorldBench evaluates intended interaction outcomes and intermediate state transitions.

7 Limitations

DreamX-World 1.0 improves interaction, controllability, and efficiency, but long-horizon consistency and compatibility among control signals remain challenging. Generated worlds can drift in object appearance or layout, while captions, camera controls, and events may conflict in some settings.

  • DreamX-World 1.0 improves interaction, controllability, and efficiency, but several challenges remain.
  • Long-horizon generation can cause drastic drift in object appearance or layout, making visual and geometric consistency difficult.
  • Captions, camera controls, and event instructions may conflict when an event creates content incompatible with future observations in some caption-defined settings.

8 Conclusion

DreamX-World 1.0 presents a general-purpose interactive world model and argues that advancing world modeling requires a full-stack perspective spanning data curation, training, evaluation, and inference acceleration. Future work targets character-centric models and native audio-visual world models.

  • DreamX-World 1.0 is presented as a general-purpose interactive world model.
  • World modeling is framed as a full-stack problem requiring coordinated improvements in data curation, training, evaluation, and inference acceleration.
  • Future work: Future work includes character-centric world models that preserve character identity, coordinate actions with freely moving cameras, and support richer multi-character interactions over long horizons.
  • Future work: Native audio-visual world models are proposed to generate synchronized speech, ambient sound, and action-dependent audio while using sound as an interactive signal.

Authors

DreamX-World 1.0 is credited to a 22-member team listed alphabetically, with ordering not indicating relative contributions. The authors acknowledge the AMAP-ML team for infrastructure, engineering support, and discussions.

  • Team Members: The 22 team members are listed alphabetically by last name, then first name for identical last names; ordering does not indicate relative contributions.The listed members are Yancheng Bai, Rui Chen, Xiangxiang Chu, Rujing Dang, Hao Dou, Bingjie Gao, Qiwen Gu, Siyu Hong, Jiachen Lei, Geng Li, Jifan Li, Ruimin Lin, Qingfeng Shi, Bingze Song, Lei Sun, Jing Tang, Ruitian Tian, Jun Wang, Jiahong Wu, Pengfei Zhang, Shen Zhang, and Jiashu Zhu.
  • Acknowledgements: The authors thank the AMAP-ML team for computational infrastructure, engineering support, and discussions.These contributions are credited as helping make the work possible.
Loading 2606.16993v1…