PaperlayerView exact revision on arXiv

Source-linked AI summary

MiniWorld: Democratizing the Training of Video World Models from Scratch

Yian Zhao, Ruochong Zheng, Hongcan Guo, Yu Yan, Jian Zhang, Jie Chen

arXiv:2608.01127v2cs.CV

TL;DR

Researchers lack a lightweight, reproducible way to train streaming video world models from scratch without complex, resource-intensive pipelines. MiniWorld provides such a framework, and experiments show broad improvements across robot-action- and camera-pose-conditioned benchmarks, including a 249% gain in DROID Trajectory Accuracy.

  • Problem

    Existing video world-model training relies on complex, resource-intensive adaptation of bidirectional models, while causal streaming deployment requires training from scratch under modest budgets.

  • Method

    MiniWorld trains a block-causal Video Diffusion Transformer with latent-space Flow Matching, chunk-wise noise scheduling, and two-stage continued training.

  • Results

    Across DROID and RE10K, MiniWorld improves broadly across geometry, fidelity, consistency, smoothness, and perceptual metrics, including a 249% gain in DROID Trajectory Accuracy.

  • Takeaways & Limitations

    MiniWorld offers a reproducible baseline for stable, competitive long-horizon streaming predictions across robot-action-conditioned and camera-pose-conditioned benchmarks.

  • Takeaways & Limitations

    Experiments use modest model and data scales, cover limited world dynamics, and still exhibit accumulating errors in extended complex interactive rollouts.

Abstract

from arXiv · show

Video world models predict future observations conditioned on historical observations and control signals, enabling long-horizon generation through autoregressive state transitions. Unlike conventional video generation models that primarily capture visual appearance and motion, video world models learn the underlying dynamics governing environment evolution under agent actions, providing a foundation for embodied AI and interactive simulation. Recent progress has largely relied on adapting pretrained video generation models through post-training or distillation. Although effective, these approaches often require complex training pipelines, substantial computational resources, and suffer from the mismatch between bidirectional pretraining and causal streaming inference. Recent studies have shown that training autoregressive video world models from scratch is feasible and scalable. However, the community still lacks a lightweight, transparent, and fully reproducible baseline trainable end-to-end with modest computational resources. We present MiniWorld, a reproducible framework for training streaming video world models from scratch. MiniWorld employs a block-causal Video Diffusion Transformer trained with Flow Matching in the latent space of a pretrained Video VAE. Building on Diffusion Forcing, it adopts a chunk-wise non-decreasing noise schedule and two-stage continued training to improve temporal modeling and stability. During inference, MiniWorld combines a rolling KV cache with pipelined asynchronous denoising for efficient streaming generation under bounded computation. The entire model can be trained within several days on a single 8-GPU server. By releasing the training and inference codebase and pretrained checkpoints, we hope MiniWorld will facilitate future research on video world modeling.

1 Introduction

MiniWorld presents a lightweight, reproducible framework for training streaming video world models from scratch under modest computational budgets. It combines causal latent-space training with efficient streaming inference and releases a complete open-source pipeline.

  • Motivation: Video world models autoregressively predict future observations from histories and control signals while modeling environment dynamics for persistent long-horizon state prediction.They target interactive visual simulation and embodied AI rather than appearance and motion synthesis alone.
  • Motivation: Adapting pretrained bidirectional video diffusion models remains effective but requires complex multi-stage pipelines and substantial resources, limiting from-scratch training for many researchers.Prevailing methods include fine-tuning, distillation, and reinforcement learning.
  • Motivation: Prior work shows chunk-wise autoregressive pretraining and Diffusion Forcing make end-to-end streaming video world-model training from scratch increasingly practical.MAGI-1 demonstrates scalability, while SkyReels-V2 applies Diffusion Forcing to long-horizon video generation and world modeling.
  • MiniWorld: MiniWorld trains a block-causal Video Diffusion Transformer with Flow Matching in latent space, using non-decreasing chunk-wise noise scheduling and two-stage continued training.These components aim to improve efficient and stable world modeling while preserving a simple training procedure.
  • MiniWorld: Rolling-KV caching and pipelined asynchronous denoising support bounded-computation long-horizon generation, while the released pipeline covers data processing, training, inference, and evaluation.MiniWorld avoids proprietary datasets and sophisticated post-training, providing a transparent and extensible open-source baseline.

2 Preliminaries: Rectified Flow

MiniWorld uses Rectified Flow in the latent space of a pretrained video VAE, learning a velocity field that is integrated from Gaussian noise to generate latent video sequences.

  • 2 Preliminaries: Rectified Flow: Rectified Flow formulates Flow Matching by linearly interpolating between a latent video sample x and Gaussian noise ϵ.MiniWorld performs all modeling in the latent space of a pretrained video VAE.
  • 2 Preliminaries: Rectified Flow: The model is trained to predict the velocity field v∗(x, ϵ) = x −ϵ.
  • 2 Preliminaries: Rectified Flow: During inference, generation starts from Gaussian noise and progressively integrates the learned velocity field to recover the target latent sequence.MiniWorld uses the pretrained Wan2.2 VAE to encode videos, with training and streaming inference conducted in this latent space.

3 Method

MiniWorld formulates streaming video world modeling as action-conditioned next-state prediction and implements it with a block-causal Video DiT. Its method combines chunk-wise monotonic diffusion schedules, flexible action conditioning, and asynchronous chunk processing to support causal streaming generation.

  • Problem formulation: MiniWorld predicts future observations from historical frames and future control signals, directly targeting the training–inference discrepancy in fixed-window video diffusion.The formulation generates x_h+1:T conditioned on x_1:h and c_h+1:T.
  • Architecture: The block-causal Video DiT partitions tokens into temporal chunks, allowing bidirectional attention within chunks and strictly causal attention across chunks.This preserves spatio-temporal interactions within each chunk while enforcing dependencies on preceding chunks for streaming generation.
  • Action conditioning: Action conditioning combines timestep-aware embeddings with residual action modulation through AdaLN, while AdaLN-LoRA adds lightweight layer-specific adaptations to a shared modulation branch.The low-rank residual is zero-initialized, preserving shared modulation at initialization while increasing conditioning capacity with marginal parameter overhead.
  • Model scaling: MiniWorld supports models up to 3B parameters and reports MiniWorld-0.5B and MiniWorld-1B configurations sharing the same block-causal architecture and conditioning interface.The reported variants differ in depth, width, and attention-head count; MiniWorld-0.5B uses 28 layers, hidden size 1152, and 16 heads, while MiniWorld-1B uses 28 layers, hidden size 1536, and 12 heads.
  • Training strategy: Independent chunk timesteps enable asynchronous denoising, and a non-decreasing noise schedule makes earlier chunks cleaner than later chunks to stabilize causal denoising.The monotonic constraint also reduces the search space of asynchronous diffusion trajectories.

4 Experiments

MiniWorld is evaluated on DROID and RE10K with domain-specific action or camera conditioning, and it consistently improves prediction quality across both benchmarks. Ablations and scaling studies show that longer contexts and larger models help, while KV-cache streaming substantially improves efficiency with little quality loss.

  • Datasets: MiniWorld evaluates embodied robot-action modeling on DROID and camera-controlled scene prediction on RE10K.Videos are resized to 240 × 320 and encoded with the pretrained Wan2.2 VAE.
  • Condition Processing: DROID uses 28-dimensional latent-frame action tokens formed by concatenating four normalized 7-dimensional robot-action vectors.Each action contains 6-DoF Cartesian end-effector displacement and gripper position, with values clipped to [−1, 1].
  • Condition Processing: RE10K uses 720-channel spatial conditions formed from four pose maps, each encoding per-pixel ray origins and directions with 15 sinusoidal frequency bands.The pose representation has 180 channels per RGB frame.
  • Benchmark Results: On DROID, MiniWorld improves Trajectory Accuracy by 249%, Depth Accuracy by 238%, LPIPS by 216%, and SSIM by 125% over the bidirectional short-video baseline.Appearance metrics improve by 26% to 82%, while VLM judge scores improve by 63% to 78%.
  • Benchmark Results: On RE10K, MiniWorld improves Photometric Smoothness by 89%, Depth Accuracy by 55%, Subject Consistency by 50%, Background Consistency by 46%, and Perspectivity by 46%.PSNR, SSIM, and LPIPS gain 34%, 19%, and 27%, respectively, while the architecture transfers from robot actions to camera-conditioned geometry.
  • Efficiency and Ablations: KV-cache streaming increases throughput from 3.31 FPS to 7.29 FPS, a 2.20× speedup, while first-chunk latency falls from 74.0 seconds to 4.86 seconds, a 15.2× reduction.The 8 in-flight window preserves nearly the same quality as the 32-chunk full window; DiT throughput rises from 0.41 to 0.91 chunks/s while VAE throughput remains around 15.2 chunks/s.

5 Discussion

MiniWorld presents a simple, reproducible streaming video world-model baseline trained from scratch under modest computational budgets, emphasizing deployment consistency over state-of-the-art perceptual quality. The discussion highlights remaining limitations and directions for improving data coverage, train–test alignment, and deployment efficiency.

  • Positioning: MiniWorld trains a streaming video world model from scratch using an accessible, reproducible recipe under modest computational budgets.It directly optimizes streaming autoregressive prediction with block-causal attention, chunk-wise diffusion forcing, and train-test-aligned inference.
  • Positioning: MiniWorld is intended as a transparent baseline for studying streaming world models rather than maximizing state-of-the-art generation quality.Large bidirectional video generation models remain preferable when perceptual quality is the primary objective, whereas MiniWorld emphasizes reproducibility and deployment consistency.
  • Positioning: MiniWorld supports research on temporal memory, causal representation learning, long-horizon error accumulation, and train-test-aligned optimization while lowering engineering barriers.The framework is positioned as a practical foundation for investigating problems specific to streaming world models.
  • Limitations: MiniWorld uses modest model and data scales, covers limited world dynamics, and still accumulates prediction errors during extended rollouts, especially in complex interactive scenarios.Long-horizon drift is substantially reduced but not eliminated.
  • Future Directions: Future work should expand and curate video-action data, improve train–test alignment, and reduce deployment latency and computational cost.Proposed approaches include broader game, simulation, and real-world corpora; self-forcing or reinforcement-learning objectives; memory compression, few-step distillation, and quantization.

6 Related Work

World-model research spans predictive environment dynamics, visually realistic controllable simulation, and diffusion-based streaming video generation. MiniWorld complements increasingly capable systems by offering a lightweight, fully reproducible recipe for training streaming video world models from scratch with modest computational resources.

  • World Models: World models learn predictive environment dynamics conditioned on historical observations and control signals, extending from compact latent dynamics to controllable interactive simulation.Representative systems include Genie 3, Matrix-Game 2.0, Yume-1.5, HY-World 1.5, DreamDojo, Mirage 2, and LingBot-World 2.0.
  • MiniWorld: MiniWorld provides a lightweight, fully reproducible recipe for training streaming video world models from scratch under modest computational resources.The framework is positioned as a complementary direction to increasingly capable and often large-scale world model systems.
  • Diffusion and Autoregressive Generation: Diffusion Forcing, DFoT, and Geometry Forcing combine diffusion models with autoregressive generation for streaming or action- and camera-conditioned video generation.Diffusion Forcing independently corrupts tokens, DFoT extends the formulation to video with arbitrary historical conditioning, and Geometry Forcing aligns diffusion representations with a pretrained geometric foundation model.
  • Causal Video Generation: Ca2-VDM, Rolling Forcing, Causal Forcing, Self Forcing, and CausVid adapt pretrained bidirectional video diffusion models into causal generators using attention, caching, rolling-window denoising, or distillation.MiniWorld instead prioritizes simplicity and reproducibility through end-to-end training from scratch with modest computational resources.

7 Conclusion

MiniWorld presents a reproducible, modest-budget framework for training streaming video world models from scratch. It aligns training with autoregressive deployment through causal modeling, continued training, and efficient streaming inference, offering a transparent baseline for future research.

  • MiniWorld enables training streaming video world models from scratch under modest computational budgets.The framework is presented as reproducible and designed for streaming generation.
  • A block-causal Video DiT, chunk-oriented probability propagation scheduler, and two-stage continued training align training with autoregressive deployment.These components retain the scalability and stability of latent diffusion models.
  • Streaming inference with a structured rolling KV cache supports deployment-time autoregressive generation.The rolling KV cache is part of MiniWorld’s streaming inference design.
  • MiniWorld proposes a simple, transparent, and reproducible recipe instead of increasingly complex training pipelines.The authors position it as a practical baseline for the community.
  • The framework aims to facilitate advances in memory mechanisms, long-horizon generation stability, action-conditioned datasets, and reproducible training pipelines.These directions target more robust and scalable world models.
Loading 2608.01127v2…