Source-linked AI summary
Yume-1.5: A Text-Controlled Interactive World Generation Model
Xiaofeng Mao, Zhen Li, Chuanhao Li, Xiaojie Xu, Kaining Ying, Tong He, Jiangmiao Pang, Yu Qiao, Kaipeng Zhang
TL;DR
Existing interactive-world methods face limited generalizability, high latency, growing context, and weak text control. Yume1.5 addresses these issues with compressed long-video modeling, accelerated inference, and text-controlled events, while retaining keyboard-based exploration and acknowledging artifacts and crowd-density degradation.
Problem
Existing methods have limited cross-domain generalizability, high generation latency, rapidly growing historical context, and insufficient text control for interactive world generation.
Method
Yume1.5 autoregressively generates infinite video worlds from a single image using joint temporal-spatial-channel compression, linear attention, accelerated inference, and text-controlled event generation.
Results
Yume1.5 supports infinite video generation with keyboard-based camera control, efficient long-video generation, mitigated inference error accumulation, and text-controlled world events.
Takeaways & Limitations
The framework provides a unified approach to interactive, realistic, temporally coherent world generation with text-based event control.
Takeaways & Limitations
Generation artifacts persist, especially backward-moving vehicles and reverse-walking characters, with degradation in extremely high-crowd-density scenes.
Abstract
from arXiv · showhide
Recent approaches have demonstrated the promise of using diffusion models to generate interactive and explorable worlds. However, most of these methods face critical challenges such as excessively large parameter sizes, reliance on lengthy inference steps, and rapidly growing historical context, which severely limit real-time performance and lack text-controlled generation capabilities. To address these challenges, we propose \method, a novel framework designed to generate realistic, interactive, and continuous worlds from a single image or text prompt. \method achieves this through a carefully designed framework that supports keyboard-based exploration of the generated worlds. The framework comprises three core components: (1) a long-video generation framework integrating unified context compression with linear attention; (2) a real-time streaming acceleration strategy powered by bidirectional attention distillation and an enhanced text embedding scheme; (3) a text-controlled method for generating world events. We have provided the codebase in the supplementary material.
1. Introduction
Yume1.5 targets interactive world generation by addressing limited domain generalizability, high latency, and insufficient text control. It combines compressed long-video modeling, accelerated inference, and text-controlled event generation.
- Existing methods struggle with realistic dynamic urban scenes, real-time continuous generation, and text-controlled random event generation.
- Yume1.5 generates interactive infinite video worlds autoregressively from a single image with stable keyboard-based camera control.
- TSCM compresses historical frames temporally and spatially while processing channel-compressed features with a parallel linear DiT.This reduces memory consumption and improves inference speed as video duration increases.
- The acceleration method combines score distillation with Self-Forcing-like training to improve sampling efficiency and mitigate inference error accumulation.It replaces the KV cache with TSCM in the training paradigm.
- Text-controlled world events are enabled through architectural design and mixed-dataset training with minimal data requirements.
- Yume1.5 combines TSCM, Self-Forcing integration, and dataset and architecture design for infinite-context generation, accelerated inference, and world generation and editing.
2. Related Works
Related work spans diffusion-based video synthesis, explicit camera control, and autoregressive long-video generation. Yume1.5 instead uses keyboard-discretized camera control and combines efficient attention with compressed historical context.
- Diffusion models evolved from image synthesis into video generation through latent diffusion, temporal modeling, and large-scale architectures.
- Recent video-generation methods control cameras with explicit pose sequences, including unified motion controllers and decoupled pan-zoom mechanisms.
- Dataset re-annotation distinguishes detailed scene-context captions for T2V training from dynamic-event captions for I2V training.
- Yume1.5 discretizes camera pose space for intuitive keyboard-based control instead of requiring fine-grained absolute camera trajectories.
- Long-video methods use sliding windows, KV caches, or self-generated historical frames to support autoregressive continuation and reduce error accumulation.
3. Data Processing
Yume1.5 combines real-world, synthetic, and event data for T2V and I2V training. Processing adds discrete camera controls and event-focused captions to support motion control, general video quality, and event generation.
- Dataset construction: The training dataset integrates real-world, synthetic, and specialized event sources to balance motion control, general video quality, and event generation.
- Real-world Dataset: Sekai-Real-HQ supplies walking clips with camera trajectories and semantic labels for real-world training.
- Real-world Dataset: Trajectory data is converted into discrete keyboard and mouse action descriptions that condition generation.
- Annotation processing: Original scene-context annotations are retained for T2V, while InternVL3-78B generates event-focused captions for I2V.
- Synthetic Dataset: A synthetic dataset is added to reduce catastrophic forgetting and domain overfitting while maintaining general video-generation ability.It is produced from 80,000 sampled captions and filtered to 50,000 videos synthesized at 720p.
- Event Dataset: The Event Dataset covers urban, sci-fi, fantasy, and weather descriptions, producing curated image-to-video sequences for complex semantic alignment.From 10,000 synthesized sequences, manual screening retained 4,000 videos.
4. Method
Yume1.5 combines unified context compression, linear attention, alternating training, and self-generated conditioning to support text/image-to-video generation and long interactive worlds.
- Text and interaction control: Event and Action Descriptions are encoded separately and concatenated, specifying the target event and keyboard or mouse controls.The finite action-description set can be precomputed and cached.
- Context compression: Linear attention replaces the exponential kernel with a ReLU-feature dot product for efficient fusion of compressed historical and predicted-frame tokens.The method applies normalization to query, key, and attention-output representations to prevent gradient instability.
- Context compression: TSCM compresses historical frames temporally and spatially for standard attention, while channel-compressed features are fused through linear attention.This joint design reduces memory and preserves generation quality as context grows.
- Architecture and training: Yume1.5 establishes joint text-to-video and image-to-video generation through alternating training on text-to-video and image-to-video datasets.The resulting foundation model supports world generation, editing, and exploration.
- Real-time acceleration: The generator autoregressively uses its own predicted frames as historical context, while distillation matches the student trajectory to the teacher for few-step inference.Replacing ground-truth conditioning alleviates train-inference discrepancy and associated error accumulation; TSCM enables substantially longer context than a KV cache.
5.1. Experimental Settings
The experiments use Wan2.2-5B2 with standardized video-generation training settings and evaluate visual quality alongside instruction following and camera-motion tracking.
- Training Configuration: Wan2.2-5B2 is pretrained at 704×1280 resolution and 16 FPS using batch size 40, Adam, and a 1e-5 learning rate.Foundation training uses 10,000 iterations on NVIDIA A100 GPUs; Self-Forcing with TSCM uses 600 iterations under otherwise identical hyperparameters.
- Evaluation Metrics: Yume-Bench evaluates visual quality and instruction following through six fine-grained metrics, including camera-motion tracking.The visual metrics include subject consistency, background consistency, motion smoothness, aesthetic quality, and image quality.
- Comparison Setup: Table 1 compares models using Instruction Following, Subject Consistency, Background Consistency, Motion Smoothness, Aesthetic Quality, and Imaging Quality.Wan-2.1 uses text-based control, whereas MatrixGame uses native keyboard/mouse control.
5.2. Quantitative Results
Yume1.5 shows stronger instruction following than compared models and maintains long-video quality more consistently across later video segments.
- Image-to-Video Comparison: 0.836 instruction-following score: Yume1.5 significantly outperforms Wan-2.1, MatrixGame, and Yume on the real-world test set.The model also achieves an average generation speed of 12 fps at 540p using a single A100 GPU.
- Metric Dynamics: Figure 5 plots Aesthetic Score against the number of chronological video blocks in long-video generation.The x-axis represents video blocks, while the y-axis represents Aesthetic Score.
- Long-Video Quality: 0.523 final aesthetic score: Self-Forcing with TSCM exceeds the 0.442 score of the model without Self-Forcing and TSCM in the sixth video segment.Self-Forcing with TSCM maintains more stable aesthetic scores from the fourth through sixth segments.
- Long-Video Quality: Image Quality scores remain more consistent between the fifth and sixth segments for the Self-Forcing with TSCM model.The comparison tracks aesthetic and image quality across six consecutive five-second clips from each 30-second video.
5.3. Ablation study
Ablations indicate that TSCM improves instruction following and stabilizes inference time as context grows, while Self-Forcing with TSCM addresses error accumulation during accelerated sampling.
- TSCM Verification: TSCM significantly improves Instruction Following over a baseline replacing it with spatial compression under identical training configurations.The authors attribute this improvement to reduced influence from motion directions in historical frames on predicted frames.
- Inference Speed: After eight video blocks, TSCM keeps inference time per step constant as context length increases, unlike full-context input.TSCM also produces the most stable change in autoregressive inference time and reduces fluctuations from expanding context.
- Inference Speed: Full-context input is the slowest method and shows the largest performance gap at the third inference step.Figure 7 compares TSCM, spatial compression, and full-context input at 704×1280 resolution.
5.4. Qualitative Results
Qualitative results show that Yume1.5 provides effective camera control together with high generation quality.
- Visual Results: Yume1.5 demonstrates effective camera control while achieving superior generation quality in qualitative results.Additional visual results are provided in the supplementary materials.
6. Conclusion
Yume1.5 enables autoregressive infinite video generation from a single image with intuitive keyboard-based camera control. Its key innovations combine efficient long-video modeling, inference acceleration, and text-controlled world-event generation, while future work targets broader interactions and applications.
- Yume1.5 generates infinite videos autoregressively from a single input image while supporting intuitive keyboard-based camera control.
- Its joint temporal-spatial-channel modeling approach supports efficient long-video generation while maintaining temporal coherence.
- An acceleration method mitigates error accumulation during inference, improving sampling efficiency while preserving visual quality.
- Text-controlled world-event generation uses architectural design and mixed-dataset training to provide text-based control with minimal data requirements.
- The authors envision extending Yume1.5 to support more sophisticated world interactions and broader applications in virtual environments and simulation systems.
7. Limitations
Yume1.5 exhibits motion artifacts and degraded performance in extremely crowded scenes. Higher resolution provides some mitigation, but the issues persist, reflecting a capacity–latency trade-off in the 5B-parameter model.
- Yume1.5 produces artifacts including backwards-moving vehicles and characters walking in reverse.
- Performance degrades in scenarios with extremely high crowd density, and increasing resolution from 540p to 720p only partly mitigates the issues.
- The authors attribute these limitations to the constrained capacity of the 5B parameter model, while larger models would impose prohibitively high generation latency.
- Mixture-of-Experts architectures are proposed as a direction for increasing parameter counts while reducing inference latency.