Source-linked AI summary
MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft
Junliang Guo, Yang Ye, Tianyu He, Haoyu Wu, Yushu Jiang, Tim Pearce, Jiang Bian
TL;DR
MineWorld addresses the efficiency and controllability challenges of video-based world modeling through a real-time, open-source interactive model for Minecraft. It combines visual-action autoregressive Transformers with parallel decoding and evaluates both visual quality and action adherence, reporting strong performance and real-time generation.
Problem
Video world models face costly inference from large token sequences, while standardized metrics for measuring adherence to control signals remain lacking.
Method
MineWorld tokenizes Minecraft game states and actions into interleaved discrete sequences for autoregressive Transformer prediction, using parallel decoding and inverse-dynamics-based controllability metrics.
Results
MineWorld achieves strong video quality and controllability while generating 2 ∼6 FPS, and its models outperform the open-sourced Oasis baseline on both evaluation aspects.
Takeaways & Limitations
MineWorld provides an open-source framework for real-time simulation and interaction in Minecraft, with released code and model weights supporting further research.
Takeaways & Limitations
Training exclusively on fixed, downsampled Minecraft data limits generalization to other video domains and higher resolutions, while temporal consistency is not guaranteed beyond 16 state-action pairs.
Abstract
from arXiv · showhide
World modeling is a crucial task for enabling intelligent agents to effectively interact with humans and operate in dynamic environments. In this work, we propose MineWorld, a real-time interactive world model on Minecraft, an open-ended sandbox game which has been utilized as a common testbed for world modeling. MineWorld is driven by a visual-action autoregressive Transformer, which takes paired game scenes and corresponding actions as input, and generates consequent new scenes following the actions. Specifically, by transforming visual game scenes and actions into discrete token ids with an image tokenizer and an action tokenizer correspondingly, we consist the model input with the concatenation of the two kinds of ids interleaved. The model is then trained with next token prediction to learn rich representations of game states as well as the conditions between states and actions simultaneously. In inference, we develop a novel parallel decoding algorithm that predicts the spatial redundant tokens in each frame at the same time, letting models in different scales generate $4$ to $7$ frames per second and enabling real-time interactions with game players. In evaluation, we propose new metrics to assess not only visual quality but also the action following capacity when generating new scenes, which is crucial for a world model. Our comprehensive evaluation shows the efficacy of MineWorld, outperforming SoTA open-sourced diffusion based world models significantly. The code and model have been released.
1 INTRODUCTION
MineWorld targets the efficiency and controllability challenges of video-based world models with a real-time, open-source interactive model for Minecraft. It combines visual-action autoregressive modeling, parallel decoding, and action-following evaluation metrics.
- Video world models learn environmental and interaction knowledge, but efficiency and controllability remain crucial unresolved challenges.Large latent video representations make inference costly, while standardized evaluation of adherence to control signals is lacking.
- MineWorld tokenizes game scenes and actions into discrete representations, interleaves them, and trains an autoregressive Transformer to model their relationships.The architecture uses visual and action tokenizers followed by a Transformer decoder trained with an autoregressive objective.
- Parallel decoding exploits dependencies between spatially adjacent tokens, achieving more than 3× speedup over standard autoregressive generation without sacrificing result quality.The method predicts certain token groups simultaneously rather than sequentially.
- 4 to 7 frames per second enables MineWorld to support real-time interaction between humans and the world model.The reported generation rate is presented as making real-time interaction feasible.
- New controllability metrics compare input actions with actions inferred from consecutive generated frames using an inverse dynamic model.This action agreement supplements conventional visual-quality assessment and measures whether generated sequences follow control signals.
- MineWorld is presented as an open-sourced real-time interactive world model that establishes a new benchmark for world modeling.The authors release the code and model weights to support further research.
2 FRAMEWORK
MineWorld models Minecraft state transitions with interleaved visual and action tokens in an autoregressive Transformer. Its diagonal parallel decoding accelerates generation by exploiting spatial dependencies, while fine-tuning helps preserve quality during faster inference.
- 2.1 OVERVIEW: MineWorld predicts future Minecraft states from past observations and current actions using an autoregressive Transformer.The model represents game states and actions as discrete tokens and trains by predicting each token conditioned on preceding tokens.
- 2.2 ARCHITECTURES: Visual states and mouse-keyboard actions are converted into discrete tokens and concatenated in an interleaved sequence.Separate tokenizers process game scenes and actions, allowing the model to learn relationships between the two modalities.
- 2.3 PARALLEL DECODING: Diagonal decoding generates adjacent-row and adjacent-column tokens simultaneously instead of following a sequential raster-scan order.For a token at row i and column j, tokens at positions i,j+1 and i+1,j are generated together in the next step.
- 2.3 PARALLEL DECODING: Larger image resolutions yield faster theoretical parallel-decoding speedups because the method exploits spatial redundancy among adjacent tokens.The paper states that decoding speed increases with image resolution.
- 2.3 PARALLEL DECODING: Fine-tuning with a parallel attention mask addresses inference-time degradation while retaining real-time interaction frequency and generation quality.The mask aligns training with the parallel decoding procedure.
3 EVALUATION
The evaluation measures both visual quality and action controllability on paired Minecraft videos and actions. It uses standard video metrics alongside IDM-based action evaluation, with experiments covering multiple MineWorld scales and the Oasis baseline.
- 3.1 DATASET: The evaluation uses paired Minecraft gameplay videos and corresponding keyboard and mouse actions from the VPT dataset.Videos are filtered, divided into 16-frame clips, and resized before tokenization.
- 3.2 EVALUATION METRICS: Visual quality is assessed with FVD, PSNR, LPIPS, and SSIM.These metrics cover video distributional quality, reconstruction fidelity, perceptual similarity, and structural similarity.
- 3.2 EVALUATION METRICS: Controllability is evaluated by comparing provided actions with actions inferred from generated state transitions by a bidirectional IDM.The IDM achieves 90.6% keypress accuracy, while action imbalance and mixed discrete-continuous controls complicate metric design.
- 3.2 EVALUATION METRICS: Table 1 reports FPS, discrete-action precision, recall, and F1, plus camera-control L1 loss for Oasis and MineWorld model scales.The table caption defines FPS as frames generated per second and identifies P, R, F1, and L1 as controllability measures.
- 3.3 IMPLEMENTATION: MineWorld experiments compare 300M, 700M, and 1.2B-parameter Transformer configurations.The models use a LLaMA architecture with an 8,262-token vocabulary.
4 EXPERIMENTS
MineWorld is evaluated against Oasis and through analyses of efficiency, controllability, video quality, and autonomous gameplay. Results show strong performance across these dimensions, including faster parallel decoding and action-responsive generation.
- Comparative evaluation: All MineWorld models outperform the Oasis baseline on both video quality and controllability accuracy.The comparison uses Oasis’s open-sourced 500M model and inference code.
- Scaling and efficiency: Larger MineWorld models achieve better controllability and video quality, while parallel decoding preserves real-time responsiveness across model scales.The largest 1.2B model reaches 3 FPS, corresponding to 180 Actions Per Minute for amateur players.
- Parallel decoding: 3× speedup is achieved for the 1.2B model with parallel decoding while maintaining comparable controllability and video quality to autoregressive decoding.Parallel decoding provides this speedup without fine-tuning in the reported comparison.
- Parallel decoding: Fine-tuning with the parallel attention mask improves parallel decoding, allowing the 300M model to match its autoregressive baseline while retaining a 3× latency speedup.The reported benefit is strongest for smaller models in the comparison.
- Controllability metrics: The proposed controllability metric has a significant positive correlation with human action-following evaluations.The correlation is measured on 20 clips scored by five experienced game players using the 700M model.
- General capability: MineWorld generates action-consistent, high-fidelity game states in door opening, wood chopping, camera movement, and same-state regeneration cases.The case studies report coherent responses to diverse actions and detailed rendering of resulting scenes.
- Controllability: Starting from the same game state, different actions produce different corresponding generation results, demonstrating controllability in the case study.The reported examples show accurate responses with respect to each supplied action.
- Serve as an Agent: MineWorld can iteratively generate future states and actions from initial gameplay inputs, simulating long-horizon play as a self-contained game agent.The case study reports diverse, contextually appropriate actions alongside accurate, high-fidelity states.
5 CONCLUSION AND LIMITATIONS
MineWorld is presented as an open-source, real-time interactive Minecraft world model using interleaved visual-action tokens and parallel decoding. Its reported 2 ∼6 FPS inference supports real-time interaction, while its scope remains limited by Minecraft-only, downsampled training and bounded context length.
- Conclusion: MineWorld tokenizes game states and actions separately, trains an autoregressive Transformer with next-token prediction, and uses parallel decoding for faster inference.The model exploits redundancy between adjacent image tokens while preserving generation quality.
- Limitations: Training exclusively on fixed-resolution, downsampled Minecraft data limits generalization to other video domains and higher-resolution output.Downsampling may also remove fine-grained details from game states.
- Limitations: Temporal consistency is not guaranteed when game-state distances exceed the 5.5k-token context, which covers 16 state-action pairs.The reported strong consistency applies within that input-length range.
A.1 DETAILS OF ACTION FOLLOWING METRIC
The action-following evaluation decomposes Minecraft controls into multiple classification tasks to accommodate action exclusions and the complexity of the action space.
- Action-following metric: The evaluation focuses on 10 common actions and camera movements rather than Minecraft’s full complex action space.This simplification is used to validate action execution quality.
- Action-following metric: The 10 actions are divided into 3 triple-classification tasks and 4 binary-classification tasks based on exclusion relationships.The decomposition reflects mutually exclusive action combinations.
A.2 TASK-SPECIFIC ACTION ACCURACY
Task-specific evaluation reports precision, recall, and F1 scores for the 700M MineWorld model across the defined action-prediction tasks. The drop action is substantially harder for the model than the other evaluated actions.
- Task-specific accuracy: The 700M MineWorld model is evaluated with precision, recall, and F1 scores across the defined classification tasks.These task-level results are reported in Table 5.
- Task-specific accuracy: The drop action performs significantly worse than the other actions, indicating greater learning difficulty for this action.The comparison is based on the reported task-specific classification results.
B MODEL CONFIGURATIONS
This section specifies the configurations, optimization settings, and reconstruction evaluation used across different model sizes.
- Table 6 lists the configurations for the different model sizes.
- Three LLaMA-based Transformer sizes—300M, 700M, and 1.2B—are trained to validate decoder scaling behavior.Hidden dimension, intermediate dimension, and layer count are tuned to obtain the target sizes.
- Table 7 lists the optimizer hyperparameters used during training.
- Table 8 reports visual-tokenizer reconstruction performance on the validation set.
C.1 RECONSTRUCTION RESULTS OF VISUAL TOKENIZER
Fine-tuning the pre-trained Amused VQ-VAE on pre-processed VPT data significantly improves reconstruction performance, demonstrating that this step is necessary.
- Fine-tuning the pre-trained Amused VQ-VAE on pre-processed VPT data significantly improves reconstruction performance.The comparison evaluates the pre-trained tokenizer against its fine-tuned version.