Source-linked AI summary
GAIA-1: A Generative World Model for Autonomous Driving
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, Gianluca Corrado
TL;DR
Autonomous driving requires models that can predict diverse future outcomes in complex real-world environments, but existing approaches face limitations in data requirements, realism, or dynamic understanding. GAIA-1 addresses this by combining a multimodal token-based world model with a video diffusion decoder to generate realistic, controllable driving scenarios. It demonstrates high-level scene understanding, generalization, geometry, and future-event prediction that support synthetic data generation and autonomous-driving training and validation.
Problem
Autonomous driving needs accurate prediction of possible future events, while existing world models may rely on difficult-to-scale labels, miss real-world complexity, or generate low-fidelity samples.
Method
GAIA-1 combines a multimodal world model using discrete tokens and next-token prediction with a video diffusion decoder for high-resolution, temporally smooth video generation.
Results
GAIA-1 demonstrates high-level structure learning, contextual awareness, generalization, 3D geometry understanding, reactive behavior, and extrapolation beyond its training data.
Takeaways & Limitations
GAIA-1 provides controllable future-scenario generation and learned representations that can support synthetic data generation, training, and validation for autonomous driving.
Takeaways & Limitations
The autoregressive generation process does not yet run in real time, although it can be parallelized to generate multiple samples concurrently.
Abstract
from arXiv · showhide
Autonomous driving promises transformative improvements to transportation, but building systems capable of safely navigating the unstructured complexity of real-world scenarios remains challenging. A critical problem lies in effectively predicting the various potential outcomes that may emerge in response to the vehicle's actions as the world evolves. To address this challenge, we introduce GAIA-1 ('Generative AI for Autonomy'), a generative world model that leverages video, text, and action inputs to generate realistic driving scenarios while offering fine-grained control over ego-vehicle behavior and scene features. Our approach casts world modeling as an unsupervised sequence modeling problem by mapping the inputs to discrete tokens, and predicting the next token in the sequence. Emerging properties from our model include learning high-level structures and scene dynamics, contextual awareness, generalization, and understanding of geometry. The power of GAIA-1's learned representation that captures expectations of future events, combined with its ability to generate realistic samples, provides new possibilities for innovation in the field of autonomy, enabling enhanced and accelerated training of autonomous driving technology.
1 Introduction
GAIA-1 combines a world model that learns future dynamics with generative video modeling to predict and render diverse, controllable driving scenarios. Its multimodal, self-supervised design supports realistic generation, high-level scene understanding, and fine-grained control over vehicle behavior and scene features.
- Accurate future prediction supports autonomous vehicles in anticipating and planning actions for safer, more efficient driving.
- Existing world models often require difficult-to-scale labeled data, may miss real-world complexity, and can generate low-fidelity future samples.
- Video, text, and action inputs condition diverse future rollouts, including ego-vehicle behavior, speed, curvature, and scene attributes.
- GAIA-1 combines a high-level world model for scene dynamics with a diffusion decoder that renders latent representations as realistic videos.
- The model uses discrete video representations and next-token prediction, while its architecture encodes multimodal inputs into tokens before decoding predicted image tokens.
- GAIA-1 exhibits high-level structure learning, contextual awareness, generalization, geometric understanding, and reactive behavior from other road users.
- Its learned future representations and controllable generation support synthetic data creation and accelerated training and validation of autonomous-driving systems.
2 Model
GAIA-1 combines a token-based world model with a video diffusion decoder to model future scene dynamics and render realistic, temporally consistent driving video. Video, text, and action inputs share a representation, enabling conditioning and control across generation tasks.
- Encoding Video, Text and Action: GAIA-1 encodes video, text, and action into a shared d-dimensional space, interleaving text, image, and action tokens at each time step.Image frames use discrete tokens, text uses T5-large representations, and actions represent speed and curvature.
- Image Tokenizer: The image tokenizer downsamples frames into discrete tokens and guides compression toward semantic features using DINO representations.This reduces sequence length while discouraging representations dominated by high-frequency signals.
- World Model: The world model is an autoregressive transformer that predicts each next image token from preceding text, image, and action tokens using causal masking.Conditioning-token dropout supports unconditional, action-conditioned, and text-conditioned generation.
- Video Decoder: Temporal subsampling lets the world model reason over longer periods, while the video decoder restores full-frame-rate predictions through temporal super-resolution.The decoder uses denoising diffusion over frame sequences to improve temporal consistency.
- Video Decoder: A jointly trained image-and-video diffusion decoder supports high-resolution rendering, temporal upsampling, and multiple generation tasks.Image training improves individual-frame quality, while video training teaches temporal consistency; tasks include image, video, autoregressive, and interpolation generation.
3 Data
GAIA-1 is trained on a large London driving corpus with feature-balanced sampling and validation splits designed to examine generalization across seen and unseen roads. The video decoder uses masked-frame tasks to support several generation settings.
- Dataset: The dataset contains 4,700 hours of proprietary London driving data collected between 2019 and 2023, representing approximately 420M unique images.Sampling weights are computed from binned empirical feature distributions to control the training-data distribution.
- Decoder Tasks: Video-decoder training tasks mask ground-truth images and context tokens, providing noise for frames to predict and tokens except during video interpolation.The task design supports generation conditioned on different combinations of image context and tokens.
- Data Sampling: Training data for the tokenizer is balanced over latitude, longitude, and weather category to represent geographically and visually diverse scenes.The stated goal is to ensure the tokenizer adequately represents diverse scene conditions.
- Data Sampling: World-model and video-diffusion training additionally balance steering and speed behavior categories so different dynamics are captured.These behavior categories supplement geographic and weather balancing.
- Validation: The validation dataset contains 400 hours from runs excluded from training and includes strict geofences for evaluating roads never seen during training.Additional validation areas around the main collection routes monitor overfitting and generalization on seen roads.
4 Training Procedure
GAIA-1 trains separate image-tokenizer, world-model, and video-decoder components with large-scale optimization configurations. World-model sampling compares strategies for balancing diversity against unreliable token choices.
- Image Tokenizer: The image tokenizer has 0.3B parameters and encodes 288 × 512 images into 576 discrete tokens with vocabulary size K = 8192.Its encoder uses spatial downsampling D = 16 in both image dimensions.
- World Model: The 6.5B-parameter world model processes T = 26 frames at 6.25 Hz, with 32 text tokens and 2 action tokens per time step.The resulting total sequence length is 15,860 tokens.
- Sampling: Top-k=50 sampling produces a token-perplexity distribution similar to real tokens, unlike argmax repetition or full-distribution sampling from unreliable tails.The comparison uses n = 576 tokens from a single generated image frame.
- Video Decoder: The 2.6B-parameter video decoder trains on seven images at 288 × 512 resolution sampled at 6.25, 12.5, or 25 Hz, with tasks sampled equally.It is trained for 300k steps using a cosine β-noise schedule.
5 Inference
GAIA-1 combines autoregressive token sampling with classifier-free text guidance and a multi-task video decoder to generate diverse, realistic, temporally coherent driving videos.
- World Model: The world model autoregressively predicts image tokens conditioned on previous text, image, and action tokens, generating each new frame through multiple forward steps.Top-k sampling avoids repetitive loops by selecting among the most likely next-token choices; k depends on image-token count and codebook size.
- World Model: Top-k sampling balances diversity and realism by drawing each next image token from the k most likely choices.The chosen k depends on the number of tokens per image frame and the pre-learnt codebook vocabulary size.
- Video Decoder: GAIA-1 supports rollouts from an initial context or from scratch, and uses a sliding window when generated videos exceed the world model’s context length.The decoder uses DDIM sampling with 50 diffusion steps and balances token information against temporal consistency through a weighted average.
- World Model: Text conditioning directs video prediction, while classifier-free guidance increases text-image alignment by reducing the diversity of possible samples.Negative prompting uses a separate text condition to encourage positive-prompt features while removing negative-prompt features.
- World Model: Guidance is scheduled across tokens and frames: token scheduling trades prompt adherence against diversity, while frame scheduling controls transitions and mitigates compounding guidance.Figure 7 illustrates a guidance schedule over twelve frames, typically with linearly decreasing guidance across tokens.
6 Scaling
GAIA-1 exhibits scaling behavior analogous to large language models: cross-entropy can be predicted from smaller training runs, and extrapolation indicates further gains from more data and compute.
- Scaling Laws: GAIA-1’s world-model formulation predicts the next token, and scaling laws analogous to those in large language models also apply to the model.This shared next-token formulation motivates examining performance as model size, data, and compute increase.
- Scaling Experiments: A power law fitted to models trained with less than 20× the compute predicted GAIA-1’s final held-out validation cross-entropy with high accuracy.The validation set was geofenced, and the fitted data covered models from 0.65M to 650M parameters.
- Scaling Experiments: The fitted models ranged from 10,000× to 10× smaller than the full model in parameter count, spanning 0.65M to 650M parameters.Compute was estimated from parameter count using C = 6N floating-point operations per token before multiplying by the number of training tokens.
- Implications: Extrapolation indicates substantial potential for further improvement through expanding both data and computational resources.
7 Capabilities and Emerging Properties
GAIA-1 generates diverse, plausible driving futures from video, text, and action prompts while modeling scene structure, context, geometry, and agent interactions.
- GAIA-1 generates coherent scenes with plausible object locations and realistic interactions governed by traffic lights and road rules.
- It generates novel and diverse videos containing object, movement, and scene combinations not explicitly present in training data.
- The model captures contextual responses and 3D geometry, including pitch and roll caused by road irregularities such as speed bumps.
- GAIA-1 can generate stable driving videos lasting minutes entirely from imagination, with complex roads, buildings, cars, and pedestrians.
- From one video context, repeated sampling produces multiple plausible futures that remain consistent with initial conditions and may differ in road interactions or ego behavior.
- Text prompts can fully specify imagined driving scenes, including weather and illumination conditions.
- Action conditioning enables fine-grained, out-of-distribution control of ego-vehicle dynamics while preserving surrounding-scene responses and geometry.Examples include steering left or right outside the lane and observing realistic reactions from other agents.
- Combining text and action allows the model to imagine a bus ahead and generate an overtaking scenario.
8 Related Work
Related work spans video generation and predictive world models, while GAIA-1 frames world modeling as scalable unsupervised sequence modeling and reports analogous scaling trends.
- Video generative models include VAE-, GAN-, diffusion-, and autoregressive-based approaches for producing realistic video samples.
- VAE-based models may produce blurry outputs, GANs can suffer instability and limited diversity, diffusion models improve realism and control, and autoregressive models optimize exact likelihood.
- World models learn general predictive representations of the world for understanding action consequences, supporting representation learning, planning, and policy learning.
- Prior world-model research uses self-supervised pretraining, look-ahead search, and imagined outcomes, especially in games and board-game environments.
- A single sequence-model perspective treats states, actions, and rewards as a data stream, enabling large-scale unsupervised training and motivating GAIA-1 for real-world driving.
- Scaling research studies relationships between model or data size, loss, and compute allocation, including power laws for transformer language models.
- GAIA-1 reports that scaling trends observed in language models also apply to world models when world modeling is cast as unsupervised sequence modeling.
9 Conclusions and Future Work
GAIA-1 combines discrete-token world modeling with video diffusion to generate controllable driving scenarios and learn environmental concepts through self-supervision. Its current autoregressive generation is not yet real-time, while the model is positioned as a neural simulator for autonomy research.
- GAIA-1 converts vector-quantized video representations into a next-token prediction task and uses video diffusion to generate realistic, controllable driving scenarios.
- The model distinguishes cars, trucks, buses, pedestrians, cyclists, road layouts, buildings, and traffic lights through self-supervision.
- Autoregressive generation is not yet real-time, although its parallelizable structure permits concurrent generation of multiple samples.
- GAIA-1 is presented as a neural simulator that can support understanding, prediction, adaptation, and generation of additional driving scenarios.