PaperlayerView exact revision on arXiv

Source-linked AI summary

DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers

Yuntao Chen, Yuqi Wang, Zhaoxiang Zhang

arXiv:2412.18607v1cs.CV

TL;DR

Driving world models primarily rely on video diffusion, which has limited flexibility for incorporating modalities such as actions, while integrating planning and world modeling remains unresolved. DrivingGPT interleaves image and action tokens in a unified autoregressive sequence model. It demonstrates effectiveness in action-conditioned video generation and end-to-end planning, supporting unified learning of both tasks with one model.

  • Problem

    Current driving world models primarily use video diffusion and lack flexibility for multimodal generation, while seamless differentiable integration of world modeling and planning remains unresolved.

    Source: §1 · p. 2

  • Method

    DrivingGPT interleaves discrete image tokens and quantized frame-to-frame action tokens in a multimodal driving language trained with next-token prediction.

    Source: §1 · p. 2

  • Results

    DrivingGPT demonstrates effectiveness in action-conditioned video generation and end-to-end planning, outperforming SVD on FID and FVD and a visual-encoder/MLP planner on driving scores.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results show the viability of unified learning of world modeling and planning with a single model.

    Source: §5 · p. 9

Abstract

from arXiv · show

World model-based searching and planning are widely recognized as a promising path toward human-level physical intelligence. However, current driving world models primarily rely on video diffusion models, which specialize in visual generation but lack the flexibility to incorporate other modalities like action. In contrast, autoregressive transformers have demonstrated exceptional capability in modeling multimodal data. Our work aims to unify both driving model simulation and trajectory planning into a single sequence modeling problem. We introduce a multimodal driving language based on interleaved image and action tokens, and develop DrivingGPT to learn joint world modeling and planning through standard next-token prediction. Our DrivingGPT demonstrates strong performance in both action-conditioned video generation and end-to-end planning, outperforming strong baselines on large-scale nuPlan and NAVSIM benchmarks.

1. Introduction

Driving world models rely largely on video diffusion, while integrating world modeling and planning in a differentiable framework remains unresolved. DrivingGPT unifies them through autoregressive modeling and reports strong results in video generation and planning.

1. Introduction

  • The unresolved challenge is seamlessly integrating world modeling and planning in a differentiable framework, while diffusion-based world models have limited multimodal generation flexibility.The paper identifies generation of multiple modalities, including text and action sequences, as a challenge for video-diffusion approaches.Source: §1 · pp. 1–2
  • Autoregressive transformers are presented as promising for integrated driving planners because they model sequential data and multiple modalities.The introduction situates this capability across language, visual question answering, image and video generation, sequential decision making, and robot manipulation.Source: §1 · p. 2
  • DrivingGPT performs strongly in both action-conditioned video generation and end-to-end planning, with reported gains over strong baselines.The introduction reports better FID and FVD than SVD and higher driving scores than a visual-encoder/MLP trajectory-decoder planner.Source: §1 · p. 2
  • The paper interleaves discrete image and action tokens into a driving language and trains a Llama-like model to jointly predict them.Frames are tokenized with pretrained VQ-VAEs, trajectories become quantized frame-to-frame relative actions, and training uses standard next-token prediction.Source: §1 · p. 2

2. Related Works

Prior driving world models emphasize visual prediction, often with diffusion, while autoregressive models offer a route to generating visual states and actions together. DrivingGPT adopts that multimodal autoregressive direction and evaluates planning on NAVSIM.

2.1. World Models for Autonomous Driving

  • Earlier driving world models primarily predict future visual states, and most use diffusion; this paper instead explores autoregressive video prediction that also outputs actions.The related work notes challenges in diffusion models with temporal consistency and longer video generation.Source: §2.1 · p. 2

2.2. Generation with Autoregressive Models

  • Building on discrete-token autoregressive generation, the paper adopts a multimodal autoregressive paradigm that outputs video generation and planning together.Prior approaches cited include autoregressive image and video generation and multimodal generation with discrete tokens.Source: §2.2 · pp. 2–3

2.3. End-to-end Autonomous Driving

  • Unlike previous BEV-based planning frameworks, the paper evaluates its autoregressive, world-model-trained approach in an open-loop setting on NAVSIM.Source: §2.3 · p. 3

3. Driving as Next Token Prediction

DrivingGPT formulates driving policy prediction and world dynamics modeling as one sequence-modeling problem. It tokenizes front-camera observations and frame-wise relative actions, interleaves them, and uses an autoregressive transformer to predict both modalities.

3.1. Problem Formulation

  • DrivingGPT unifies trajectory policy prediction and observation-space driving dynamics modeling as a single sequence-modeling task.The paper frames driving as an MDP and identifies both the policy π(at|ot) and transition function P(ot+1|ot, at) as important.Source: §3.1 · p. 3, §3 · p. 3

3.2. Multimodal Driving Language

  • The method tokenizes front-camera images with VQ-VAE, balancing image compression against transformer context length for longer driving sequences.The observation space excludes surrounding cameras, LiDAR, and IMUs; image tokens represent downsampled frames.Source: §3.2 · p. 3
  • Driving actions are frame-wise relative pose changes rather than long-horizon absolute trajectories, limiting privileged future-action information in the causal sequence.Each action component is clamped to the 1st–99th percentile range and quantized; x, y, and θ use different vocabularies because their units and magnitudes vary.Source: §3.2 · pp. 3–4
  • The model interleaves image and action tokens into one causal sequence and predicts them with a transformer using a unified vocabulary and frame-wise 1D rotary embeddings.The vocabulary combines VQ-VAE codes with three action-component vocabularies; the model represents both world modeling and end-to-end driving as subtasks.Source: §3.2 · p. 4, Figure 2 · p. 4
  • Predicted frame-to-frame actions are converted into transformation matrices and composed to recover absolute poses and trajectories.Source: §3.2 · p. 4

4. Experiments

DrivingGPT is evaluated for video generation and end-to-end planning, with ablations examining tokenizers, action prediction, data curation, and positional embeddings. The reported results show favorable video-generation and planning comparisons, alongside findings on long-video consistency and planning design choices.

4.1. Implementation Details; 4.2. Datasets and Metrics

  • Training uses resized front-camera images, discrete image and action tokens, and next-token prediction; datasets and evaluation protocols include nuPlan and NAVSIM.NAVSIM planning uses PDMS and its five subscores after a four-second non-reactive simulation; video quality is assessed with FVD and FID.Source: §4.1 · p. 5, §4.2 · pp. 5–6

4.3. Video Generation

  • Trained from scratch, DrivingGPT surpasses previous methods in video quality on navtest.The comparison includes publicly available models and SVD fine-tuned on navtrain.Source: §4.3 · p. 6, Table 1 · p. 6
  • On long navtest sequences, DrivingGPT maintains video quality and lower FVD, while SVD struggles as sequence length increases.The evaluation uses 512 clips with more than 64 frames; SVD’s fixed-frame training limitation is associated with a quality drop.Source: §4.3 · p. 6, Table 2 · p. 6
  • DrivingGPT generates more diverse long-video scenes and maintains object consistency, while SVD fine-tuning can repeat content and diffusion methods can hallucinate objects.Figure 3 illustrates repeated content such as remaining at a red light; Figure 4 depicts object appearance and disappearance.Source: Figure 3 · p. 5, §4.3 · p. 6, Figure 4 · p. 7

4.4. End-to-end Planning

  • DrivingGPT compares favorably with constant-velocity baselines and a ResNet-50/MLP planner on NAVSIM end-to-end planning.The evaluation conditions on two seconds of observations and actions to predict four-second trajectories.Source: §4.4 · p. 7, Table 3 · p. 7

4.5. Ablation Study

  • Among evaluated visual tokenizers, the authors select LlamaGen for the world model.The comparison uses navtest with 12,146 video samples and a shared spatial downsampling rate of 16.Source: §4.5 · p. 7, Table 4 · p. 7
  • DrivingGPT outperforms variants that copy history actions, and 100k NAVSIM sequences outperform 650k nuPlan sequences in the reported planning comparison.The authors conclude that data quality matters more than quantity for driving language modeling.Source: §4.5 · p. 8, Table 5 · p. 8, Table 6 · p. 8
  • Applying 1D rotary embeddings to both image and action tokens is reported as crucial for strong planning performance.The study contrasts this choice with methods that use zero position embeddings for actions.Source: §4.5 · p. 8, Table 7 · p. 8

5. Conclusion

DrivingGPT unifies visual world modeling and trajectory planning as sequence modeling, jointly learning to generate image and action tokens. Experiments and ablations demonstrate effectiveness on action-conditioned video generation and end-to-end planning.

5. Conclusion

  • DrivingGPT unifies visual world modeling and trajectory planning in a sequence-modeling task.The model jointly learns to generate image and action tokens for both tasks.Source: §5 · p. 8
  • Experiments and ablations on large-scale nuPlan and NAVSIM benchmarks demonstrate effectiveness in action-conditioned video generation and end-to-end planning.The conclusion describes this as evidence for the viability of unified world-modeling and planning with a single model.Source: §5 · pp. 8–9
  • The authors frame unified learning of world modeling and planning as a stepping stone for future exploration of differentiable model-based planning.Source: §5 · p. 9

A. Refining Video Generation with SVD Decoder.

The SVD decoder refines predicted frame tokens into high-resolution, temporally consistent video. Long clips are generated autoregressively before their tokens are decoded and refined in consecutive frame chunks.

A. Refining Video Generation with SVD Decoder.

  • A video diffusion decoder conditioned on frame tokens addresses temporal inconsistency from decoding each frame token independently.During training it receives tokens from discretized input images; at inference it receives image tokens predicted by DrivingGPT.Source: A. Refining Video Generation with SVD… · p. 13
  • DrivingGPT generates 64 frames by repeatedly predicting the next 8 frames from the last 8 generated frames, producing 32-second clips.Source: A. Refining Video Generation with SVD… · p. 13
  • The SVD refinement stage decodes 16 consecutive frames at a time, repeating four times to produce 64 frames.VQ-VAE features from the transformer’s 288 × 512 resolution are upsampled to match SVD’s 576 × 1024 resolution before conditioning its denoising UNet.Source: A. Refining Video Generation with SVD… · p. 13

B. The Effect of Sampling Parameters on Video Generation.

Sampling parameter k trades image smoothness against detail in generated videos: smaller values produce smoother appearances, while larger values retain more detail but can distort car shapes.

B. The Effect of Sampling Parameters on Video Generation.

Related papers

Loading 2412.18607v1…