PaperlayerView exact revision on arXiv

Source-linked AI summary

GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving

Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, Gianluca Corrado

arXiv:2503.20523v1cs.CVcs.AIcs.RO

TL;DR

Autonomous-driving simulation needs controllable, realistic generation of interacting agents, domain-specific conditions, and consistent multi-camera video. GAIA-2 addresses this with a latent diffusion world model combining structured and external latent conditioning, and reports scalable simulation across diverse and rare scenarios. Its stated limitation is occasional temporal or semantic inconsistency, especially in long-horizon or complex scenes.

  • Problem

    Autonomous-driving generative models need fine-grained control over agents, ego motion, environmental factors, road semantics, and consistent multi-camera streams.

  • Method

    GAIA-2 combines a surround-view video tokenizer and latent diffusion world model conditioned on actions, agent states, metadata, embeddings, camera geometry, and environmental controls.

  • Results

    GAIA-2 supports coherent generation across up to five camera views and diverse typical and rare driving scenarios, including generation from scratch, rollouts, inpainting, and editing.

  • Takeaways & Limitations

    GAIA-2 serves as a tool for scalable training, broader scenario coverage, and stress-testing autonomous-driving systems across complex long-tail conditions.

  • Takeaways & Limitations

    GAIA-2 occasionally produces temporal or semantic inconsistencies, particularly in long-horizon or complex scenarios.

Abstract

from arXiv · show

Generative models offer a scalable and flexible paradigm for simulating complex environments, yet current approaches fall short in addressing the domain-specific requirements of autonomous driving - such as multi-agent interactions, fine-grained control, and multi-camera consistency. We introduce GAIA-2, Generative AI for Autonomy, a latent diffusion world model that unifies these capabilities within a single generative framework. GAIA-2 supports controllable video generation conditioned on a rich set of structured inputs: ego-vehicle dynamics, agent configurations, environmental factors, and road semantics. It generates high-resolution, spatiotemporally consistent multi-camera videos across geographically diverse driving environments (UK, US, Germany). The model integrates both structured conditioning and external latent embeddings (e.g., from a proprietary driving model) to facilitate flexible and semantically grounded scene synthesis. Through this integration, GAIA-2 enables scalable simulation of both common and rare driving scenarios, advancing the use of generative world models as a core tool in the development of autonomous systems. Videos are available at https://wayve.ai/thinking/gaia-2.

1 Introduction

Autonomous-driving simulation requires controllable, realistic, multi-agent and multi-camera generation beyond the capabilities typically offered by general-purpose video models. GAIA-2 addresses this gap with a domain-specialized latent diffusion model supporting fine-grained conditioning across diverse driving scenarios.

  • Realistic driving simulation supports the development, training, and evaluation of autonomous driving systems while reducing reliance on expensive real-world data collection.
  • Autonomous-driving models must control ego actions, other agents, environmental context, road configuration, edge cases, and consistent multi-camera streams.
  • GAIA-2 generates up to five spatially and temporally consistent camera streams at 448 × 960 resolution.
  • GAIA-2 conditions generation on ego kinematics, geography, weather, time of day, road layouts, and dynamic-agent geometry to support typical and edge-case scenarios.
  • External latent spaces, including CLIP and driving-specific embeddings, provide semantic control alongside generation from scratch, context prediction, and inpainting.
  • GAIA-2 unifies controllability, multi-camera coherence, and diverse simulation within one framework for scalable training and robust evaluation.

2 Model

GAIA-2 combines a video tokenizer with a latent diffusion world model to generate semantically coherent surround-view video under structured control. Its continuous latent representation and flexible conditioning interface support multiple generation modes while preserving cross-view and temporal consistency.

  • GAIA-2 consists of a video tokenizer and latent world model for realistic, semantically coherent multi-view video generation with rich conditional control.
  • The world model predicts future latent states from past latents, actions, and domain-specific controls, while also supporting scratch generation and inpainting.
  • The tokenizer decodes predicted latents back into pixel-space video, with Figure 3 describing its temporal-consistency mechanism.
  • A higher spatial compression rate produces fewer, semantically richer latent tokens, improving inference speed, memory efficiency, and capture of temporal dynamics.
  • Compared with GAIA-1’s discrete latents, GAIA-2 uses continuous latents and supports conditioning on actions, agent states, metadata, embeddings, and camera geometry.

2.1 Video Tokenizer

The video tokenizer compresses high-resolution video into a compact latent space and reconstructs frames using temporally aware decoding. Its architecture combines asymmetric encoder-decoder transformers, aggressive downsampling, and reconstruction, semantic, regularization, and adversarial losses.

  • The tokenizer maps video into continuous, semantically structured latents and reconstructs full-frame video using temporal context for consistency.
  • The encoder downsamples video spatially and temporally into latent tokens, with the supplied configuration using TL = 8 and L = 64.
  • The encoder uses convolutional downsampling, 24 spatial transformer blocks, and a projection modeling a Gaussian distribution over latents.
  • The decoder combines linear projection, upsampling convolutional blocks, and space-time factorized transformer blocks to reconstruct RGB frames.
  • The decoder jointly reconstructs 24 frames from 3 temporal latents and uses sliding-window inference for temporal consistency.
  • Training combines L1, L2, perceptual, DINO cosine-similarity, and KL-divergence losses, followed by decoder fine-tuning with a GAN loss.

2.2 World Model

GAIA-2 is an 8.4B-parameter latent world model that predicts future video latents from past context, actions, and diverse structured or external conditioning. Its architecture combines space-time factorized attention with specialized encodings for actions, cameras, timestamps, agents, and scene metadata.

  • Architecture: GAIA-2 uses an 8.4B-parameter space-time factorized transformer trained with flow matching to predict future latent states.The model predicts future latents conditioned on past latents, actions, and domain-specific control signals.
  • Architecture: The world model independently encodes each camera view and supplies action and conditioning vectors at every timestep.Inputs comprise multi-camera latent sequences, action vectors, and conditioning vectors.
  • Architecture: Actions and flow-matching time enter transformer blocks through adaptive layer normalization, while other conditioning variables use cross-attention.Adaptive layer normalization provides an explicit pathway for action information affecting every spatial token.
  • Conditioning: Camera geometry, timestamps, and spatial positions receive separate positional encodings that are added to the input latents.Camera geometry uses learnable linear layers, while spatial position and timestamps use sinusoidal encodings.
  • Training objective: Training interpolates future latents with Gaussian noise, and the model learns target velocities using an L2 loss.Context latents remain unchanged while future latents are perturbed; the loss averages over context-frame and flow-matching-time distributions.
  • Conditioning: Structured controls cover ego actions, dynamic-agent boxes, metadata, camera configurations, timestamps, and external latent representations.External representations include CLIP and proprietary scenario embeddings.

3 Data

GAIA-2 is trained on approximately 25 million short driving videos collected across three countries, multiple vehicle platforms, and varied camera systems. The dataset balances joint scenario distributions and evaluates geographic generalization on unseen locations.

  • Dataset composition: Approximately 25 million two-second video sequences were collected from 2019 to 2024 across the UK, US, and Germany.The geographic span is intended to cover diverse driving and environmental conditions.
  • Dataset composition: The recordings use three car models and two van types, with five or six cameras providing 360-degree surround-view coverage.Capture frequencies vary among 20 Hz, 25 Hz, and 30 Hz.
  • Camera diversity: Camera positions and calibrations varied across platforms and over time, providing training diversity across spatial configurations.This variability supports generalization across camera rigs.
  • Scenario diversity: The dataset spans diverse weather, times of day, road types, and traffic environments while balancing their joint probability distribution.The balancing strategy models realistic co-occurrences among geographic, lighting, weather, and behavioral factors.
  • Evaluation: Geographically held-out validation excludes defined regions from training to assess performance on unseen locations.The strategy targets generalization across different environments.
  • Evaluation: The dataset combines broad temporal, spatial, vehicle, camera, and scenario coverage with a geographically held-out validation setup.These properties form the stated foundation for controllable driving-video generation across varied conditions.

4 Training Procedure

GAIA-2 trains its video tokenizer and latent world model independently with large-scale compute and tailored objectives. Training uses aggressive compression, mixed generation tasks, conditioning dropout, and optimization choices designed for stability and generalization.

  • Training setup: The video tokenizer and world model are trained independently with separate objectives and procedures.Both components use large-scale compute infrastructure.
  • Video tokenizer: The tokenizer trains for 300,000 steps on 128 H100 GPUs using 24-frame, 448 × 960 crops from randomly selected camera views.Each camera stream is encoded independently.
  • Video tokenizer: 8× temporal and 32× spatial downsampling produces a compact latent representation with dimension L = 64.The resulting representation has an effective compression rate of approximately 400.
  • Video tokenizer: Tokenizer training combines reconstruction, perceptual, semantic-alignment, and low-weight KL losses, followed by decoder fine-tuning with a GAN loss.The loss includes DINO v2 distillation, L1, L2, LPIPS, and KL terms.
  • World model: The world model trains for 460,000 steps on 256 H100 GPUs with five-camera inputs represented by 12,600 latent tokens.Training inputs contain 48 video frames at 448 × 960 resolution.
  • World model: Training samples 70% from-scratch generation, 20% contextual prediction, and 10% spatial inpainting while randomly dropping conditioning variables.Each conditioning variable is independently dropped with 80% probability, and all are jointly dropped with 10% probability.
  • Regularization: Latent tokens are normalized with µx = 0.0 and σx = 0.32, while input camera views are randomly dropped with 10% probability.These choices target robustness to partial observability and latent-scale variation.

5 Inference

GAIA-2 provides a shared latent-space denoising process for generation from scratch, prediction from context, and content editing. Its inference procedure combines a hybrid noise schedule with optional and spatially selective classifier-free guidance for difficult or targeted scenarios.

  • Inference modes: GAIA-2 unifies its inference tasks through latent-space denoising followed by pixel-space decoding with the video tokenizer.The supported modes cover generation, prediction, and editing scenarios.
  • Inference modes: Generation from scratch denoises Gaussian noise, while autoregressive prediction forecasts future latents from past context.Rolling-window decoding produces temporally consistent output for generated sequences.
  • Inference modes: The inference modes support scene manipulation starting from noise, context, or existing video.This positions GAIA-2 as a general-purpose simulator for varied scene operations.
  • Noise schedule: A linear-quadratic schedule uses early linear noise levels for coarse layouts and motion, then quadratic steps for high-frequency refinement.Experiments use 50 denoising steps, with the schedule intended to improve quality and computational efficiency.
  • Multi-rig generation: Camera-parameter conditioning preserves spatial and temporal consistency across sports-car, SUV, and large-van camera rigs.The model supports multiple vehicle embodiments and camera setups.
  • Guidance: Classifier-free guidance is normally disabled but activated at scales from 2 to 20 for rare edge cases or unusual agent configurations.The guidance scale depends on scene complexity.
  • Guidance: Spatially selective guidance applies classifier-free guidance only to regions influenced by known agent conditioning, such as 3D bounding boxes.This targets controlled regions while preserving global scene coherence.

6 Results

GAIA-2 is evaluated qualitatively and quantitatively as a controllable generator for diverse driving scenarios, real-data augmentation, safety-critical cases, and selective editing. Its metrics assess visual fidelity, temporal consistency, and conditioning accuracy, with validation loss correlating strongly with human perceptual preference.

  • Diverse Scenario Generation: GAIA-2 generates diverse, contextually plausible scenarios across countries, weather conditions, times of day, road layouts, and camera configurations.CLIP and scenario embeddings provide semantic control, while camera conditioning supports consistent viewpoints across sports cars, SUVs, and vans.
  • Augmenting Real-World Data: GAIA-2 diversifies real-world sequences by altering weather, lighting, and time of day while preserving scene semantics and ego motion.Partial noising and denoising creates multiple visual variants from a single real-world example.
  • Generating Safety-Critical Scenarios: GAIA-2 generates rare safety-critical scenarios caused by unsafe ego actions or hazardous behaviors from other agents.Examples include steering into oncoming traffic, aggressive driving, emergency braking, and hazardous crossings.
  • Inpainting: GAIA-2 inserts dynamically conditioned agents into masked video regions without disrupting unrelated areas.The inpainting capability supports selective content editing while preserving existing context.
  • Quantitative Evaluation: Validation loss and other metrics increased positively during training, with validation loss showing the strongest correlation with human perceptual preference on 1,024 from-scratch samples.The evaluation suite measures visual fidelity, temporal consistency, and dynamic-agent conditioning using FDD, FID, FVMD, and class-based IoU.

7 Related Work

Related work has advanced latent video generation and driving-specific control, but existing systems generally cover only subsets of autonomous-driving simulation requirements. GAIA-2 is presented as a unified latent diffusion framework combining multi-camera generation, structured semantic conditioning, flexible inference modes, and external latent spaces.

  • Limitations of Existing Models: General-purpose video models prioritize visual quality, while autonomous-driving systems require structured control over scene elements and agent behavior.The gap is especially relevant for precise, semantic, multimodal control across space and time.
  • Generative World Models in Autonomous Driving: Existing driving world models variously support ego actions, 3D bounding boxes, HD maps, environmental conditions, or multi-camera generation, but not the full requirement set.Examples include DriveDreamer, Drive-WM, UniMLVG, and MaskGWM.
  • Limitations of Existing Models: Many prior systems remain limited by single-camera settings, missing agent-level control, incomplete multi-view semantics, or insufficient support for inpainting and targeted scene modification.These limitations constrain fine-grained editing for data augmentation and scenario variation.
  • GAIA-2: GAIA-2 unifies up to five-view multi-camera generation with structured and semantic conditioning over ego actions, dynamic agents, and scene metadata.Its contribution is framed as addressing domain-specific needs for realistic, controllable, multi-view driving scenes.
  • GAIA-2: GAIA-2 also supports generation from scratch, inpainting, scene editing, long-horizon rollouts, and external CLIP or driving-scenario embeddings.These capabilities are combined with continuous latent representations and geographically and environmentally diverse training data.

8 Conclusions and Future Work

GAIA-2 unifies multi-camera coherence, structured semantic conditioning, and fine-grained control in a scalable latent diffusion world model for autonomous-driving simulation. The authors identify temporal or semantic inconsistencies, computational cost, and the need for richer behavior, environmental conditions, language control, and rare-event data as future challenges.

  • Conclusions: GAIA-2 unifies multi-camera coherence, structured semantic conditioning, and fine-grained control within a scalable latent diffusion architecture.
  • Conclusions: The model conditions generation on ego-vehicle dynamics, environmental features, road layout, and dynamic agents while maintaining spatial and temporal consistency across up to five camera views.
  • Conclusions: GAIA-2 supports generation from scratch, autoregressive rollouts, spatial inpainting, and real-scene editing for systematic exploration of driving scenes.
  • Conclusions: Scaling controllability and scene diversity supports training, evaluation, iteration, out-of-distribution generalization, and stress-testing across complex long-tail scenarios.
  • Future Work: Future work targets long-horizon inconsistencies, computationally intensive synthesis, richer agent and environmental modeling, natural-language control, and expanded rare-event training data.
Loading 2503.20523v1…