Source-linked AI summary
Yume: An Interactive World Generation Model
Xiaofeng Mao, Shaoheng Lin, Zhen Li, Chuanhao Li, Wenshuo Peng, Tong He, Jiangmiao Pang, Mingmin Chi, Yu Qiao, Kaipeng Zhang
TL;DR
Yume addresses challenges in generating realistic, continuously controllable video worlds, especially under camera-motion control in complex urban scenes. It presents a preview interactive world-generation model using keyboard-controlled exploration from an input image, organized around four system components and demonstrated through autoregressive world generation.
Problem
Existing video diffusion methods struggle with realistic interactive video generation because they often rely on synthetic or static scenarios and precisely annotated absolute camera motions.
Method
Yume combines discrete camera-motion quantization, an autoregressive video-generation architecture, advanced sampling, and model acceleration for keyboard-controlled world exploration from an input image.
Results
Yume provides a preview model that lets users explore a dynamic world created from an input image using keyboard inputs and supports infinite autoregressive video generation.
Takeaways & Limitations
The preview establishes a foundation for interactive world exploration with intuitive keyboard control while targeting improved visual quality and realism in complex scenes.
Takeaways & Limitations
Yume still faces challenges in visual quality, runtime efficiency, and control accuracy, and does not yet support interaction with objects.
Abstract
from arXiv · showhide
Yume aims to use images, text, or videos to create an interactive, realistic, and dynamic world, which allows exploration and control using peripheral devices or neural signals. In this report, we present a preview version of \method, which creates a dynamic world from an input image and allows exploration of the world using keyboard actions. To achieve this high-fidelity and interactive video world generation, we introduce a well-designed framework, which consists of four main components, including camera motion quantization, video generation architecture, advanced sampler, and model acceleration. First, we quantize camera motions for stable training and user-friendly interaction using keyboard inputs. Then, we introduce the Masked Video Diffusion Transformer~(MVDT) with a memory module for infinite video generation in an autoregressive manner. After that, training-free Anti-Artifact Mechanism (AAM) and Time Travel Sampling based on Stochastic Differential Equations (TTS-SDE) are introduced to the sampler for better visual quality and more precise control. Moreover, we investigate model acceleration by synergistic optimization of adversarial distillation and caching mechanisms. We use the high-quality world exploration dataset \sekai to train \method, and it achieves remarkable results in diverse scenes and applications. All data, codebase, and model weights are available on https://github.com/stdstu12/YUME. Yume will update monthly to achieve its original goal. Project page: https://stdstu12.github.io/YUME-Project/.
1 INTRODUCTION
Yume presents an interactive world-generation model that creates dynamic video worlds from input images and supports keyboard-controlled exploration. Its framework combines quantized camera control, autoregressive video generation, training-free sampling improvements, and acceleration techniques.
- Yume creates a dynamic world from an input image and enables exploration through keyboard actions.
- Existing methods struggle with continuous camera control, real-world urban complexity, and artifacts such as flickering, unnatural textures, and geometric distortions.
- Quantized Camera Motion represents camera trajectories as discrete translational and rotational actions that can be combined and transferred through keyboard input.
- Masked Video Diffusion Transformers and a memory module support chunk-based autoregressive generation with theoretically infinite duration.
- AAM refines latent high-frequency components without additional training, while joint distillation and caching improve sampling efficiency without compromising visual fidelity or temporal coherence.
- TTS-SDE uses later denoising stages to guide earlier ones while adding stochasticity to improve textual controllability.
2 RELATED WORKS
Related work spans learned world models, controllable long-duration video generation, explicit camera conditioning, artifact reduction, and diffusion acceleration. These efforts motivate systems that combine realistic generation with navigation, control, and efficient sampling.
- Video diffusion models advanced dynamic-scene synthesis through larger scale, new architectures, and training strategies for long, coherent video.
- Camera-control methods progressed from implicit motion to explicit camera parameters, including pose sequences, decoupled pan and zoom, and training-free latent guidance.
- World-model research predicts future states from actions, while newer systems directly generate interactive environments and controllable long-duration video sequences.
- Generated videos can contain unnatural textures, flickering, and semantic inconsistencies, motivating both training-based and training-free artifact-reduction methods.
- Diffusion acceleration research uses distillation, adversarial training, and timestep feature similarity to reduce video-generation computation.
3 PRELIMINARIES
The preliminaries formulate video generation with Rectified Flow in latent space. Training learns a velocity field between endpoint distributions, and inference integrates or discretizes that field to recover videos.
- Rectified Flow: Rectified Flow minimizes transport cost between marginal distributions π0 and π1 using an ODE-based training formulation.
- Rectified Flow: The interpolation xt = tx1 + (1 − t)x0 trains vθ(xt, t) to match the target velocity x1 − x0.
- Rectified Flow: Inference approximates the continuous integral with Euler updates across predefined time steps.
- Latent Video Formulation: For video, conditioning can include text and images, while VAEs compress videos into latent representations to reduce computational costs.
- Latent Video Formulation: The foundational architecture uses a spatio-temporal VAE encoder and a denoising DiT model.
4 DATA PROCESSING
Yume processes real-world walking and drone videos into a curated, annotated training corpus, then quantizes continuous camera trajectories into discrete navigational actions. This representation supports stable, intuitive control without extra learnable modules.
- Dataset: Sekai-Real-HQ contains large-scale walking clips with camera-trajectory and semantic annotations.
- Dataset: The source collection includes 10471 hours of walking videos and 628 hours of drone videos, recorded at least in 1080P and 30–60 FPS.
- Dataset: Preprocessing applies shot detection, clip extraction, luminance and quality filtering, and camera-trajectory filtering to obtain 6620 hours of clips.
- Dataset: Annotations include location, categories, captions, and camera trajectories, followed by diversity-based sampling to produce 400 hours of Sekai-Real-HQ.
- Camera Motion Quantization: Trajectory quantization addresses imprecise, high-variance camera estimates by converting continuous poses into discrete actions that filter jitter and ease training.
- Camera Motion Quantization: Each predefined motion corresponds to a canonical relative transformation, and the closest motion is selected using translational and rotational dissimilarity.
- Camera Motion Quantization: Action descriptions are injected as text conditions, enabling camera-pose-controlled generation without additional learnable parameters.
5 METHOD
Yume combines discrete camera control, masked video diffusion, compressed autoregressive history, artifact refinement, and cache-aware acceleration for interactive long-form world generation.
- 5 METHOD: Yume’s framework combines camera motion quantization, model architecture, long-video training, generation, and an advanced sampler.The overview identifies these components as the method’s main dimensions.
- 5.2.1 MASKED VIDEO DIFFUSION TRANSFORMERS: MVDT selectively processes visible tokens, reconstructs masked content with a side-interpolator, and uses remaining DiT blocks for decoding.This design reduces memory and computational overhead while preserving temporal coherence.
- 5.2.3 LONG VIDEO GENERATETION: The long-video method compresses historical frames with Patchify and increases compression as video length grows to address inter-frame discontinuity and weak temporal coherence.The baseline Patchify configuration uses 2× temporal and 4× spatial compression in height and width.
- 5.3.1 TRAINING-FREE ANTI-ARTIFACT MECHANISM: AAM uses a two-stage denoising process that removes motion control during refinement, preserves low-frequency structure, and enhances high-frequency details without additional training.The refinement reduces artifacts and improves sharpness and perceptual quality in generated urban scenes.
- 5.4 CAMERA MOTION CONTROL: Yume replaces continuous pose matrices with discrete camera-motion text conditions, enabling keyboard-based control without additional learnable parameters.The representation includes translational and rotational actions that can be combined through keyboard input.
- 5 METHOD: Yume generalizes beyond real-world scenes to animation, video games, and AI-generated images, and supports world editing through image-editing methods.The method also supports V2V adaptation to live images captured by an iPhone.
- 5.6 ACCELERATION: The acceleration framework co-optimizes step distillation with caching, reducing sampling to 14 steps while selecting 10 low-impact DiT blocks as cacheable layers.Importance analysis finds central blocks least influential and initial and terminal blocks most influential.
6 EXPERIMENT
Yume is evaluated for interactive control, visual quality, long-video stability, sampler effectiveness, acceleration, and artifact reduction across complex scenes. It follows keyboard-controlled camera motions well, maintains reasonable temporal stability, and preserves visual quality after acceleration.
- Evaluation details: 70 videos or images form Yume-Bench, covering complex quantized camera-motion combinations and rare actions such as backward walking and camera tilting.The benchmark evaluates visual quality and instruction following with six fine-grained metrics.
- Image-to-video generation: 0.657 instruction-following score: Yume significantly outperforms Wan-2.1 and MatrixGame while achieving optimal or near-optimal performance on other metrics.The comparison uses the same random seed across models, with Wan-2.1 using text control and MatrixGame using native keyboard/mouse control.
- Long-video generation: 0.5% subject-consistency decay and 0.6% background-consistency decay occur across the evaluated long-video segments, indicating reasonable stability over time.The 18-second sequence is generated incrementally in 2-second segments, with motion changing after the first 8 seconds.
- Long-video generation: 22.3% instruction-following recovery occurs after 12 seconds as motion inertia diminishes, following an 8.6% drop during the transition phase.The reported recovery is from 0.636 to 0.819; the transition changes from the initial motion pattern to continuous forward movement.
- Sampler ablation: TTS-SDE improves instruction following relative to ODE and produces clearer, more detailed videos, although other indicators decline slightly.The ablation replaces ODE sampling with SDE and TTS-SDE for comparison.
- Acceleration and artifact reduction: Reducing sampling steps from 50 to 14 causes minimal differences in metrics except instruction following, whose text-control capability may weaken.The result compares the distilled model with the original model.
- Acceleration and artifact reduction: Yume follows predefined quantized motion paths, while AAM produces clearer videos and avoids illogical scenes such as aberrant snowman artifacts.These qualitative results are illustrated in Figures 6 and 7.
7 CONCLUSION
Yume is presented as an interactive world-generation model using keyboard inputs to explore an input-image-created dynamic world, with autoregressive infinite video generation. The preview establishes four main components while acknowledging unresolved challenges and future interaction capabilities.
- Yume enables keyboard-based exploration of a dynamic world created from an input image.
- Its framework comprises camera motion quantization, video generation architecture, advanced sampler, and model acceleration.
- Yume still faces challenges in visual quality, runtime efficiency, and control accuracy, while object interaction remains a future function.
A ABLATION STUDY ON IMAGE TO VIDEO
The image-to-video ablation study evaluates controlled condition injection and MVDT using comparative experiments, with MVDT improving structural frame relationships and reducing generated-video artifacts.
- Ablation setup: The study validates MVDT and AAM in an image-to-video pipeline using 20 randomly selected video sequences.All models were trained exclusively for 1,000 iterations.
- Controlled condition injection: Controlled condition injection comparisons evaluate adaLN-zero, cross-attention, and text injection under identical training parameters.The methods show complementary advantages in V-bench metrics, and text injection is adopted for superior controllability.
- Effect of MVDT: MVDT consistently enhances Yume’s Generate capabilities compared with configurations without MVDT.The comparison excludes AAM from all configurations.
D CORE ALGORITHM IMPLEMENTATION OF GAUSSIAN BLUR KERNEL
The implementation uses separable height- and width-direction blur operators with SVD-based projections to separate low- and high-frequency latent components, alongside vocabularies for motion controls.
- Gaussian blur operator: The separable 2D linear operator applies height/width blur kernels with SVD decomposition.
- Frequency projections: The projection B(z) = APinvAz extracts low-frequency components, while z − B(z) = (I − APinvA)z preserves high-frequency details.
- Motion vocabulary: Keyboard vocabulary combines translational actions such as forward, backward, left, and right, including diagonal combinations.
- Motion vocabulary: Camera vocabulary represents right and left turns, upward and downward tilts, combined motions, and stillness.
- Projection implementation: The null-space projection implementation applies the operator, its pseudoinverse, and subtraction from the input before restoring the original shape.
OPERATOR APPLICATION METHODS
The operator applies SVD-based transformations separately along the height and width dimensions, producing the composed blur operation.
- The forward operation processes height and width dimensions sequentially through SVD factors and returns the composed result.The height transformation precedes the width transformation.
OPERATOR INITIALIZATION EXAMPLE
The operator initialization example constructs two blur kernels and instantiates a 2D linear operator for 544×960 inputs.
- The height blur kernel is [0.1, 0.8, 0.1].
- The width blur kernel is [0.2, 0.6, 0.2].
- The LinearOperator2D instance uses H=544 and W=960.
- The operator selects CUDA when available and otherwise uses CPU.
E THE JOINT OPTIMIZATION OF ADVERSARIAL DISTILLATION AND CACHING
The acceleration design jointly trains adversarial distillation with feature caching to preserve denoising fidelity when cached features are used during inference.
- Intermediate features are cached for selected DiT layers during transitions between diffusion states.Caching applies to layers l ∈Lcache during the transition from x_tn+k to x_tn−1, where t_n+k < 1.
- Stop Grad truncates gradients to simulate cached-feature encounters during inference.
- The optimization combines Diffusion Loss for denoising fidelity with adversarial loss.