ADriver-I: A General World Model for Autonomous Driving

1. Introduction

• The system targets direct control from visual information and near-future scene prediction, in contrast to the perception-prediction-planning-control sequence of modular driving systems.The modular design is described as interpretable, while human drivers are characterized as acting from visual information and anticipating the near future.Source: §1 · p. 1
• ADriver-I introduces interleaved vision-action pairs to unify visual features and control signals, allowing historical pairs and current visual tokens to inform current-frame control prediction.The control signals include steer angles and ego speed, converted into text-like expressions.Source: §1 · p. 1
• The predicted current action conditions video diffusion to generate future frames, which can then support further control prediction.The abstracted process is described as repeatable in a world generated by ADriver-I itself.Source: §1 · p. 2
• ADriver-I reports current-frame L1 errors of 0.072 m/s for speed and 0.091 rad for steer angle, with FID 5.52 and FVD 97 for generated future four frames.These reported metrics use three historical vision-action pairs for control prediction.Source: §1 · p. 2

2.1. Multimodal Large Language Models

• MLLMs support cross-modal comprehension and have inspired embodied-intelligence work, including systems that predict motor actions or low-level control signals.The related work also identifies DriveGPT4 as a concurrent MLLM-based system that outputs control signals and explanations.Source: §2.1 · p. 2

2.2. End-to-end Autonomous Driving

• End-to-end driving methods range from direct waypoint prediction to approaches using scene representations such as cost maps, segmentation, motion flow, occupancy, or vectorized agents and maps.Transfuser and TCP predict waypoints without constructing scene representations.Source: §2.2 · p. 2

2.3. Generative Models for Autonomous Driving

• Driving-scene generation includes GAN, VAE, and diffusion approaches, with examples spanning action-correlated videos, BEV-conditioned images, and controllable street-view generation.Video latent diffusion is cited as an approach to high-resolution video synthesis.Source: §2.3 · pp. 2–3

2.4. World Models for Autonomous Driving

• ADriver-I unifies control-signal prediction with future-scene generation, whereas GAIA-1 focuses on scenario generation and DriveDreamer relies on HD maps and 3D bounding boxes for future generation.The authors introduce “infinite driving” and state that ADriver-I achieves it in the world it creates.Source: §2.4 · p. 3

3. Method

• The method uses an MLLM and video diffusion model, trained separately and merged during inference.The method section introduces architecture, prompt construction, and training as its three parts.Source: §3 · p. 3, §3.3 · p. 5

3.1. Architecture

• The MLLM predicts current-frame control from the current frame and three historical vision-action pairs, then its action conditions the VDM to predict four future frames.At inference, the predicted next frame becomes the current frame for the next cycle, enabling repeated control and generation.Source: §3.1 · p. 3

3.2. Prompt Construction

• Visual tokens paired with text-tokenized low-level controls form interleaved vision-action pairs, which support variable conversation lengths and unify prediction and generation in the word-embedding space.For VDM prompting, GPT3.5 converts consecutive-frame controls into motion descriptions such as accelerating, decelerating, or turning.Source: §3.2 · pp. 4–5

3.3. Model Training

• MLLM pretraining uses nearly 1.4M private-dataset highway pairs, with nuScenes and private data used for supervised fine-tuning; VDM is pretrained on the private data and fine-tuned on about 23K nuScenes video samples.The models are trained separately before being merged for inference.Source: §3.3 · p. 5

4.1. Implementation Details

• The MLLM and video diffusion model are trained separately and merged for inference; the MLLM uses three historical vision-action pairs and the current image as input.The VDM is pretrained on private data and fine-tuned on nuScenes.Source: §4.1 · pp. 5–6

4.2. Evaluation Metrics

• Control prediction is evaluated with L1 error and threshold accuracy, while future-frame quality uses frame-wise FID and FVD.Threshold accuracy uses θ values of 0.01, 0.03, 0.05, and 0.07.Source: §4.2 · p. 6

4.3. Control Signal Prediction

• ADriver-I outperforms three baselines on nuScenes; private-dataset speed and steer-angle L1 errors are 0.035 m/s and 0.015 rad.The private dataset focuses on highway scenes and has more supervised-finetuning data than nuScenes (1.4M vs. 23K).Source: §4.3 · pp. 6–7
• Direct absolute-number prediction performs better than English-description and relative-difference encodings.The caption also identifies special-token encoding as an evaluated method, but the text does not state its comparative outcome.Source: Table 3 · p. 6, §4.3 · p. 7
• Two decimal places perform almost as well as three and outperform fewer decimal places.The paper attributes accuracy error to integer or one-decimal-place representations.Source: §4.3 · p. 7, Table 4 · p. 7
• Multi-round conversation outperforms single-round supervision by a large margin on speed prediction.Multi-round training adds supervision on intermediate action predictions.Source: §4.3 · p. 7, Table 5 · p. 7

4.4. Future Scene Generation

• 5.5 FID and 97.0 FVD are reported for four generated future frames conditioned on four historical frames, without 3D bounding boxes or HD maps.The diffusion model uses historical frames and MLLM-predicted control signals to generate future scenes.Source: §4.4 · p. 8

4.5. Joint Control&Generation

• With three historical vision-action pairs, recurrent inference uses MLLM-predicted controls and diffusion-generated frames, with each affecting the next stage.The reported sequence shows predicted controls affecting future scenes and generated scenes pushing the model to take corresponding actions.Source: §4.5 · p. 8

• ADriver-I integrates an MLLM with a video diffusion model and outputs low-level control without the mainstream sequential pipeline.Source: §5 · p. 8
• ADriver-I reports impressive performance on nuScenes and private data and achieves infinite driving in its generated world from initial vision-action pairs.Source: §5 · p. 8
• ADriver-I predicts near-future frames from historical information.Source: §5 · p. 8
• Quick control changes may lead the video diffusion model to generate low-quality frames that disturb the next control prediction.Source: §5 · p. 8
• The authors consider performance far from satisfactory for deployment and plan to train on more vision-control pairs.Source: §5 · p. 8
• The MLLM and video diffusion model are trained separately, so they do not benefit from end-to-end optimization; the authors call for a unified framework.Source: §5 · p. 8
• The system lacks routing information for long-distance driving, and the authors suggest introducing a navigation map.Source: §5 · p. 8
• The authors hope future work will observe a scaling law from the generative perspective.Source: §5 · p. 9