Source-linked AI summary
Genie: Generative Interactive Environments
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, Tim Rocktäschel
TL;DR
Genie addresses the lack of interactive experiences in video generation by learning controllable environments from unlabelled Internet videos. It combines a video tokenizer, autoregressive dynamics model, and unsupervised latent action model, producing frame-by-frame control and supporting policy inference from unseen action-free videos. The approach scales to an 11B-parameter foundation world model, while remaining limited by unrealistic futures, 16-frame memory, slow interaction, and unreleased training resources.
Problem
Video generation has less interaction and engagement than language tools, motivating models that generate entire interactive experiences from Internet video corpora.
Method
Genie combines a spatiotemporal video tokenizer, an autoregressive dynamics model, and an unsupervised latent action model trained from video-only data.
Results
11B parameters: Genie forms a foundation world model trained on 30,000 filtered hours of Internet gameplay videos.
Takeaways & Limitations
Learned latent actions support inferring policies from unseen action-free videos of simulated reinforcement-learning environments.
Takeaways & Limitations
Genie can hallucinate unrealistic futures, retains only 16 frames of memory, operates around 1FPS, and does not release checkpoints or training data.
Abstract
from arXiv · showhide
We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.
1. Introduction
Genie addresses the gap between video generation and interactive experiences by learning frame-by-frame controllable virtual worlds from video-only data. Its 11B-parameter system scales from Internet gameplay videos and extends latent-action learning toward agent training.
- Motivation and contribution: Genie generates interactive experiences from a single text or image prompt using video-only training data without action or text annotations.The model is trained on over 200,000 hours of publicly available Internet gaming videos and can use unseen images as prompts.
- Core approach: Genie combines a novel video tokenizer, a causal latent action model, and an autoregressive dynamics model built with spatiotemporal transformers.Video tokens and latent actions are passed to the dynamics model, which predicts the next frame using MaskGIT.
- Scaling and training: 11B parameters: Genie scales to a foundation world model for 2D platformer gameplay trained on 30,000 filtered hours of Internet videos.The architecture is scaled from 40M to 2.7B parameters during analysis before producing the final 11B-parameter model.
- Generality and agents: A separate model trained on action-free RT1 robot videos learns a generative environment with consistent latent actions.The paper also reports using Internet-video latent actions to infer policies from unseen action-free simulated RL videos.
- Positioning: Genie is presented as a novel video and world model controllable frame by frame while requiring only video data at training time.The comparison is summarized in Table 1 as a new class of generative model.
2. Methodology
Genie uses spatiotemporal representations to learn discrete video tokens and unsupervised latent actions, then autoregressively predicts future frames conditioned on user actions. At inference, prompts are tokenized and iteratively decoded into action-controllable video.
- Spatiotemporal architecture: Spatiotemporal transformers alternate spatial attention within each frame with causal temporal attention across frames to reduce video computation.Each block contains spatial, temporal, and feed-forward layers; the design balances capacity with computational constraints.
- Model components: Genie has three components: a latent action model, a video tokenizer, and a dynamics model that predicts the next frame from actions and past frame tokens.The tokenizer is trained first, followed by co-training of the latent action and dynamics models.
- Latent action model: The latent action model learns actions without labels by encoding video histories and the next frame, then decoding the next frame from history and latent actions.A VQ-VAE objective restricts actions to a small discrete vocabulary, with |A| = 8 in the experiments.
- Inference control: The latent action decoder supplies a training signal and is discarded at inference, where user-selected actions replace its outputs.The VQ codebook remains available to map discrete user inputs into latent action embeddings.
- Video tokenizer: The video tokenizer uses a VQ-VAE with an ST-transformer so discrete encodings incorporate temporal dynamics and each encoding reflects previously seen frames.This temporal-aware design is described as improving video generation quality while avoiding the compute profile of more expensive temporal tokenizers.
- Dynamics and inference: The dynamics model receives video tokens and latent actions, predicts masked future tokens autoregressively, and is trained against ground-truth tokens.At inference, a prompt image is tokenized, combined with each user action, iteratively predicted, and decoded into frames.
3. Experimental Results
Genie is evaluated as a scalable, controllable generative environment across platformer, robotics, out-of-distribution image-prompt, and transfer settings. Results show coherent motion, consistent latent actions, emergent scene understanding, and benefits from the proposed architecture and raw-pixel action inputs.
- Controllability: Consistent latent-action meanings emerge across inputs despite initial action semantics being unknown.The paper compares discovering these meanings to learning the buttons on a new controller.
- Qualitative results: Genie generates controllable videos from text-generated images, sketches, and real-world photos, including visually distinct out-of-distribution prompts.The examples show clear character movement after repeatedly applying latent actions.
- Qualitative results: Genie emulates parallax by moving foreground, middle-ground, and background elements at different rates.This capability is demonstrated from an image generated by Imagen2.
- Robotics and transfer: Latent actions transfer to unseen reinforcement-learning environments, matching an oracle score with as few as 200 expert samples for adaptation.The mapping from latent actions to real actions contains no information about the current observation, supporting consistency and meaningfulness for transfer.
- Ablation studies: Raw-pixel inputs improve controllability over tokenized-image inputs, while ST-ViViT improves both FVD and Δ_tPSNR with a reasonable memory trade-off.The token-input model’s slight FVD advantage on Platformers does not persist on Robotics, and its controllability is worse in both environments; C-ViViT uses more memory without better performance.
4. Related Work
Genie is positioned at the intersection of world models, video generation, playable video generation, procedural content generation, and latent-action agent training. Its distinction is scaling latent-action control to prompted interactive environments without ground-truth action labels.
- World models: World models predict action-conditioned next frames and can support policy learning without direct environment experience.
- Video models: Genie resembles transformer-based video models that operate over tokenized images, while adding latent actions for interactive control.
- Playable video generation: Unlike playable video generation, Genie generates entirely new environments from prompts rather than modeling domain-specific static examples.
- Environment generation: Procedural content generation commonly produces game levels or game code, whereas Genie generates visually interactive environments from prompts.
- Latent-action agent training: Prior latent-action methods addressed imitation, planning, or reinforcement-learning pre-training, but had not been applied at Genie’s scale.
- Latent-action agent training: Earlier work used human-provided action labels to annotate videos, whereas Genie infers policies from latent actions learned from Internet videos.
5. Conclusion and Future Work
Genie demonstrates that video-only training can produce diverse, controllable interactive environments. The authors identify longer-horizon consistency, realistic futures, and interaction speed as key remaining challenges while highlighting future agent-training applications.
- Genie enables diverse interactive environments to be generated and controlled despite training exclusively from video data.
- Genie can hallucinate unrealistic futures, limiting the realism of generated environments.
- A 16-frame memory makes consistent environments over long horizons challenging, while the current interaction rate is around 1FPS.
- Genie may support future agent training by providing more diverse environments and using latent actions to learn from unseen videos.
Broader Impact
Genie could broaden access to creating game-like experiences, while raising responsibilities around future deployment and access to training resources. The paper discusses both creative potential and practical reproducibility constraints.
- Genie could let more people, including children, create and enter their own game-like imagined worlds.
- Future advances could help amplify human game-generation creativity and support industries developing playable worlds.
- The authors withheld model checkpoints, the training dataset, and accompanying data examples to pursue respectful, safe, and responsible future releases.
- Reproducing the main results may be difficult for researchers with fewer computational resources, so the authors provide a smaller reproducible example.
Core Contributors
The project combined leadership, model research, scaling, infrastructure, dataset curation, agent training, metrics, and technical and strategic guidance across the contributor team.
- Jake Bruce led the project and contributed to the video tokenizer, action model, dynamics model, scaling, infrastructure, and demonstration.
- Michael Dennis contributed to dynamics research, scaling, metrics, demonstrations, and infrastructure, while Ashley Edwards contributed to the concept, leadership, action research, agent training, and demonstrations.
- Other contributors covered dynamics research, dataset curation, infrastructure, project management, technical advice, strategic advice, and scaling.
Additional Example Trajectories
Genie generates interactive trajectories from sketches, text-to-image outputs, and photographs, with human input supplying the actions. In Platformers, repeated latent actions remain consistent across different starting frames and acquire semantic meanings.
- Genie accepts hand-drawn sketches, text-to-image model outputs, and realistic photographs as prompts for generated trajectories.
- Human input provides the actions that drive the dynamics of these generated trajectories.
- Repeated latent actions produce consistent outcomes across four different Platformers starting frames.
- Platformers latent actions acquire semantic meanings corresponding to left, right, jump, and no-op despite no action-label training.
B.1. Platformers Dataset
The Platformers dataset is built by filtering publicly available gameplay videos with keyword rules, quality checks, and learned classification. Curation reduces the dataset to over 30,000 hours while improving model performance relative to the much larger original collection.
- The initial collection targets 2D platformer videos using gameplay-related keywords and excludes titles containing negating terms such as “movie” or “unboxing”.
- Quality filtering removes videos lacking clear gameplay or containing distractors such as menus or streamer faces.
- The filtering pipeline labels 10k videos, trains an 11M-parameter ResNet18 classifier, and uses prediction confidence to decide which videos to retain.
- The curated dataset contains 6.8M videos totaling over 30k hours, and outperforms the original dataset on FVD despite being just over 10% its size.
C.1. Latent Action Model Training
The latent action model is trained within Genie’s scalable video-generation architecture, learning discrete action codes without labels. More codes improve performance but reduce playability, while scaling experiments produce the final large model.
- Increasing the number of action codes provides a benefit but reduces playability for both human and AI agents.
- The Platformers action model uses the latent-action and dynamics components specified by the corresponding model and optimizer configurations.
- A frozen Internet-video LAM labels unseen expert sequences with latent actions, enabling a policy to predict those actions before mapping them to real actions using a small labeled set.
- The resulting agent adapts with as few as 200 expert labels when mapping latent actions to real actions.
E.2. Architecture
The behavioral-cloning architecture trains transformer policies for both oracle and latent-action agents, using frame histories and action embeddings. A compact CoinRun case study demonstrates consistent playable latent actions under single-device training constraints.
- Both oracle and Genie LAM agents use transformer policies trained for behavioral cloning.
- Policies encode four-frame histories with ST-ViViT and combine frame encodings with one-hot embeddings of previous actions.
- Oracle and Genie LAM policies use cross-entropy losses with real-action and latent-action targets, respectively.
- The reproducible CoinRun case study is designed to train on a single mid-range TPU or GPU in under a week.
- CoinRun training collects 10M transitions from random-policy trajectories across 10,000 level seeds.
- The CoinRun tokenizer fits on one 16G-memory TPU, while latent action and dynamics training uses a 36-sequence batch of 16-frame sequences.
- After 200k parallel training steps, the CoinRun model generates consistent playable latent actions resembling the original environment.