PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamForge-World 0.1 Preview: A Low-Compute Real-Time Controllable World Model

Daniyel Ayupov, Artur Markov-Tsoy

arXiv:2606.30292v1cs.LGcs.CV

TL;DR

Real-time interactive world models must sustain controllable visual rollouts under streaming actions and self-generated context. DreamForge-World 0.1 Preview adapts an open causal video prior with residual action conditioning and multimodal history initialization, demonstrating a consumer-GPU preview supporting live control, reprompting, and dual-view operation.

  • Problem

    Interactive world models must maintain controllable frame-by-frame rollouts conditioned on visual history, user actions, and optional semantic inputs.

  • Method

    DreamForge adapts an open causal video prior with a residual action pathway, multimodal latent-history initialization, and a closed-loop autoregressive runtime.

  • Results

    The preview supports text, image, video, and mixed initialization, live keyboard/mouse control, first- and third-person operation, mid-stream reprompting, and minute-scale continuation.

  • Takeaways & Limitations

    DreamForge demonstrates a constrained-compute path toward real-time controllable world models with broad interactive capability coverage on consumer GPUs.

  • Takeaways & Limitations

    Persistent spatial memory is missing, so revisiting a region may re-synthesize scene content rather than preserve it.

Abstract

from arXiv · show

We present DreamForge-World 0.1 Preview, a preview foundational world model for real-time interactive world simulation. The system adapts the LongLive 1 autoregressive video stack, itself derived from Wan2.1-T2V-1.3B, with a residual action pathway inspired by the Matrix-Game family. DreamForge-World 0.1 Preview focuses on a complementary axis to frontier-scale world simulators: low-compute adaptation, consumer-GPU runtime, and broad interactive capability coverage. It supports live keyboard and mouse control, multimodal initialization, mid-stream reprompting, dual-view operation, and minute-scale interactive rollouts at native 480p resolution, reaching up to 14 to 15 FPS FPS on a single RTX 4090 with a low memory footprint. By leveraging open video backbones and applying targeted adaptation runs, we build the preview system with high cost-efficiency. DF-World 0.1 Preview is not yet a memory-complete or frontier-quality world simulator, but demonstrates a practical low-compute route toward real-time controllable world-model previews on consumer GPUs.

1 Introduction

Interactive world models must maintain controllable frame-by-frame rollouts under streaming user control and self-generated context, creating a latency-versus-state-retention tradeoff. DF-World 0.1 Preview addresses this tradeoff through broad interactive capability coverage and a compact, low-compute system recipe optimized for single-GPU preview interaction.

  • Problem setting: Interactive world models generate future visual observations in a closed control loop conditioned on visual history, user actions, and optional semantic inputs.Unlike offline video generation, they update scenes frame by frame under streaming control while generated outputs become subsequent conditioning.
  • Design tradeoff: Real-time interaction favors short context windows, cacheable causal structure, few-step inference, quantization, and lightweight decoding, whereas long-horizon consistency favors memory and state-aware conditioning.The field is organized around a latency-versus-state-retention tradeoff.
  • DF-World operating point: DF-World 0.1 Preview targets broad interactive capability coverage under tight adaptation compute by adapting the LongLive/Wan video-prior lineage into a streaming action-conditioned rollout system.It supports text, image, video, and mixed initialization, mid-stream reprompting, and separate first- and third-person control paths.
  • System recipe: The system recipe combines an open causal video prior, a residual action pathway, multimodal autoregressive history seeding, and single-GPU preview-loop optimization.This positions DF-World around capability coverage per unit direct adaptation compute rather than frontier visual fidelity, persistent spatial memory, or engine-level control.

2 Related Work

Related work has moved world models toward controllable visual environments, using action-conditioned generation and causal video-model rollouts. Recent systems also emphasize persistent memory and multimodal entry, while DF-World 0.1 Preview targets broad preview-scale interaction with persistent spatial memory remaining unresolved.

  • Interactive and action-conditioned generation: Action-conditioned systems shift world models toward controllable visual environments, including real-time game-like generation and frame-level keyboard/mouse conditioning.GameNGen uses diffusion models, while Matrix-Game 2.0 uses few-step autoregressive diffusion for streaming generation.
  • Open video priors and causal rollout: Open video priors support causal rollout: Wan underlies the LongLive lineage, which extends short clips into frame-level autoregressive long-video generation.LongLive uses KV recache, streaming long tuning, short-window attention, and frame sinks.
  • Open video priors and causal rollout: Causal or chunked autoregressive video generators are increasingly treated as substrates for streaming continuation.MAGI-1 and related autoregressive video systems reinforce this direction.
  • Memory, multimodal entry, and promptability: Persistent memory is a defining frontier for interactive world simulation, emphasizing revisit consistency, off-screen persistence, and long-horizon context retention.WorldPlay, Infinite-World, DreamX-World, Genie 2, and Genie 3 exemplify these memory objectives.
  • Memory, multimodal entry, and promptability: Image-conditioned and promptable entry points span Genie-style environments, Wan-family I2V, MAGI-style autoregressive video, DreamX-World, WorldPlay, LongLive, and BiWM.These systems provide multimodal or promptable ways to enter world-generation environments.
  • Memory, multimodal entry, and promptability: DF-World 0.1 Preview provides broad preview-scale interaction, while persistent spatial memory remains its main missing capability.This positions the system within prior work on memory, multimodal entry, and promptability.

3 System Overview · 4 Qualitative Capabilities

DF-World 0.1 Preview is a streaming autoregressive world model that advances rollouts from visual history, text conditioning, and live actions. Its shared runtime supports multimodal initialization, dual-view control, mid-stream reprompting, and minute-scale continuation.

  • 3 System Overview: DF-World 0.1 Preview predicts future latent frames from visual history, text conditioning, and live keyboard/mouse actions in a closed-loop rollout.Decoded observations are recycled as conditioning for the next prediction step.
  • 4 Qualitative Capabilities: The runtime exposes six capabilities: prompt-only rollout, multimodal entry, live keyboard/mouse control, first- and third-person views, mid-stream reprompting, and minute-scale continuation.These capabilities operate through the same autoregressive loop rather than independent modules.
  • 4 Qualitative Capabilities: Prompt-only rollout initializes a trajectory from text, while image, video, or mixed inputs can initialize continuation from a provided visual state.Multimodal inputs are inserted into the latent history before continuation.
  • 4 Qualitative Capabilities: Live keyboard and mouse inputs condition the next generated frames after initialization, keeping the rollout interactive.The interaction remains within the same autoregressive continuation loop.
  • 4 Qualitative Capabilities: First-person mode emphasizes egocentric camera motion and navigation-like response, whereas third-person mode coordinates character motion, camera behavior, parallax, and background continuation.The preview uses separate view-specific action checkpoints for these observation mappings.
  • 4 Qualitative Capabilities: Mid-stream reprompting changes semantic conditioning while retaining visual history, enabling promptable continuation and event injection without restarting.A coastal rollout is redirected by a text update that summons a tsunami.
  • 4 Qualitative Capabilities: The preview can continue generation autoregressively beyond a fixed rollout horizon, with the first minute providing the strongest qualitative window for interactive use.The stated condition applies to the current preview.

5 Backbone, Control, and Adaptation

DF-World 0.1 Preview builds on LongLive 1 and Wan2.1-T2V-1.3B, then adds staged LoRA adaptation and a Matrix-Game-style residual action pathway for interactive control. Dual-view operation uses separate high-rank LoRA adaptations for first-person and third-person control.

  • Backbone and rollout formulation: DF-World 0.1 Preview uses LongLive 1, built on Wan2.1-T2V-1.3B, as its causal autoregressive video substrate.The retained runtime includes frame-level rollout, prompt switching through KV recache, cacheable causal structure, short-window attention, frame-sink context, and efficient streaming inference.
  • Backbone LoRA: A rank-64 LoRA first adapts the LongLive/Wan backbone toward interactive game-like domains before explicit user control is introduced.Training combines NitroGen gameplay videos, GameGen-X/Open-World Video Game Dataset material, and a smaller manually collected high-quality set.
  • Action dataset and labels: The control subset contains 5-second clips at 24 FPS with continuous mouse motion and discrete movement/control annotations.Labels include interaction, jump, sprint, crouch, act_1, and act_2, with view-type metadata across data sources.
  • Residual action pathway: The system transplants Matrix-Game 2.0-style action-module weights onto the LongLive-based DiT, then briefly fine-tunes the transferred control path.Matrix-Game 2.0 reports frame-level keyboard/mouse action injection inside a few-step autoregressive diffusion loop, trained on roughly 1200 hours of interaction-annotated data.
  • View-specific control checkpoints: Dual-view control uses two high-rank LoRA adaptations: one trained on first-person samples and another on third-person samples.Separate controllers avoid forcing one controller to cover incompatible action-to-observation mappings; training uses an x_0 MSE objective with trajectory/pose losses.

6 Multimodal Initialization via Latent History Conditioning

DF-World 0.1 Preview adds multimodal initialization by encoding images and videos into the autoregressive latent history used for rollout. Clean latent prefixes enable image-to-world and video-to-world entry, while subsequent persistence depends on autoregressive history.

  • Multimodal initialization: DF-World adds multimodal initialization inside the DreamForge runtime by inserting image and video encodings as clean initial history for continuation.This extends LongLive 1’s native prompt-switching interface without using a separate image- or video-initialization architecture.
  • Image initialization: An input image is encoded by the video VAE into a latent observation prefix, after which rollout follows the current text prompt and live action stream.This provides an image-to-world entry point.
  • Video initialization: A provided video segment is encoded as multiple latent frames or chunks, enabling autoregressive prediction of future observations.This provides a video-to-world entry mode alongside LongLive-native prompt switching.
  • Trajectory persistence: Prefix conditioning anchors the rollout’s beginning, but persistence after divergence from the initial context depends on autoregressive history.The initial prefix does not by itself determine long-term trajectory persistence.

7 Runtime

DreamForge-World’s runtime adapts LongLive 1 for interactive streaming by adding asynchronous generation and decoding, low-compute execution paths, KV-cache quantization, and training-free cache management. Throughput is measured end-to-end at native 480×832 resolution, with LightTAEW 2.1 improving throughput and memory footprint while the default VAE remains the quality-reference path.

  • Runtime architecture: The runtime builds on LongLive 1’s causal autoregressive stack, including prompt switching, KV caching, short-window attention, and frame-sink mechanisms.DreamForge further adds asynchronous streaming around generation and VAE decoding, low-compute execution paths, KV-cache quantization, and Deep Forcing-style training-free cache management.
  • Measurement protocol: Runtime measurements report observed end-to-end preview throughput at native 480×832 resolution rather than isolated transformer-only speed.The measured path includes diffusion transformer execution, action conditioning, VAE decoding, and streaming overhead.
  • VAE paths: Preview measurements compare the default Wan2.1 VAE path with the same runtime using LightTAEW 2.1 decoding.The comparison is reported at 480×832 resolution.
  • Runtime trade-offs: LightTAEW 2.1 improves measured preview throughput and memory footprint, while the default VAE remains the reference path for quality-preserving demonstrations.Although related systems use quantized caches, asynchronous decoding, and low-precision infrastructure, DreamForge uses preview-specific implementation choices.

8 Limitations

DF-World 0.1 Preview remains limited by missing persistent spatial memory and degradation over long self-conditioned rollouts. Its controls are less precise and responsive than explicit simulation, with narrower sensory and interaction scope.

  • Memory and revisit consistency: Persistent spatial memory is missing: revisiting a region can re-synthesize scene content, introducing previously unseen trees and altered scene structure.The system can continue a trajectory but does not maintain a reliable external map of generated space.
  • Long-horizon drift: During minute-scale rollouts, quality, object identity, and layout consistency degrade as the model conditions on its own imperfect history.Color oversaturation and texture degradation are visible; later frames partially recover color balance but not lost texture fidelity.
  • Control, latency, and sensory scope: Keyboard and mouse actions influence trajectories, but action diversity and precision remain below explicit simulation, while aggressive camera motion can destabilize scenes.Third-person control is more fragile than first-person navigation.
  • Control, latency, and sensory scope: Latency is real-time but not conventionally game-feel responsive, and the preview lacks sound generation, multi-agent interaction, and reliable physical interaction with persistent objects.These omissions narrow the system’s sensory and interactive scope.

9 Adaptation Scale

DF-World 0.1 Preview was adapted using 64 hours of curated gameplay and control video across its backbone and action-module LoRA stages. The paper compares this reported training-video scale with systems whose primary sources provide direct hour-level figures.

  • Adaptation scale: 64 hours of curated gameplay/control video were used across the backbone LoRA and two action-module LoRA stages.This is the filtered adaptation set for the preview system, not the raw size of upstream data-sourcing corpora.
  • Adaptation scale: Reported training-video scale is compared only where primary sources provide direct hour-level figures, because interactive world-model cost metrics are not uniform.The comparison concerns training-video scale rather than a standardized cost metric.
  • Adaptation scale: Matrix-Game 2.0 reports about 1200 hours of interaction-annotated data, while Genie reports a 30,000-hour filtered platformer training set.Figure 7 annotates these raw hours relative to DF-World’s 64-hour curated adaptation set, with visual compression above 1,000 hours preserving ordering and separation.

10 Release Status and Next Steps

DF-World 0.1 Preview is not a full model release and its checkpoint is not currently open-sourced. Next steps prioritize persistent spatial memory, broader generation and control capabilities, and improvements to robustness, latency, and evaluation.

  • Release status: DF-World 0.1 Preview is a preview rather than a full model release, and DreamForge does not currently open-source its checkpoint.DF-World 0.5 is planned as a more complete model generation and may be released publicly if licensing, safety, and engineering constraints allow.
  • Next technical targets: Persistent spatial memory is the next technical target, with candidate approaches including latent retrieval, camera-aware memory, loop-closure training, self-generated-history training, and hybrid external scene-state mechanisms.The project is also exploring larger video backbones, stronger action-module architectures, improved dual-view control, and continuously updated streaming multimodal generation.
  • Next technical targets: Further work covers audiovisual real-time world generation with synchronized audio, alongside latency, action diversity, third-person stability, aggressive-camera robustness, and systematic evaluation.Streaming multimodal generation is intended to support continuously updated image and video inputs rather than using them only to seed real-time video output.

11 Conclusion

DF-World 0.1 Preview demonstrates a constrained-compute route to real-time controllable world models, combining an adapted video prior, residual action pathway, multimodal conditioning, and consumer-GPU optimization. It supports broad interactive capabilities while leaving persistent memory, frontier-scale fidelity, and precise physical control as open challenges.

  • Core approach: DF-World adapts a LongLive/Wan video prior with a Matrix-Game-style residual action pathway and latent history conditioning.The system also optimizes inference for consumer GPUs.
  • Interactive capabilities: The live generated-world loop supports text, image, and video initialization, mid-stream reprompting, and dual-view control.These capabilities provide broad interactive coverage in the preview system.
  • Limitations: Persistent spatial memory, frontier-scale fidelity, and precise physical control remain open limitations.These limitations define areas where the preview does not yet provide complete world-model behavior.
  • Future work: Future work targets persistent memory, lower latency, larger backbones, stronger action conditioning, streaming multimodal and audiovisual control, and more stable dual-view interaction.The stated operating point emphasizes broad interactive capability coverage, single-GPU runtime, and targeted adaptation cost.
Loading 2606.30292v1…