PaperlayerView exact revision on arXiv

Source-linked AI summary

GenAD: Generalized Predictive Model for Autonomous Driving

Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, Jun Zhang, Andreas Geiger, Yu Qiao, Hongyang Li

arXiv:2403.09630v2cs.CV

TL;DR

Autonomous-driving models have limited generalization across environments and dynamic futures, motivating a scalable video-prediction approach. GenAD trains on diverse web and licensed driving data with text conditions, then adapts the learned representation to prediction, simulation, and planning. It generalizes zero-shot to unseen datasets and outperforms prior video-prediction methods, while increased model capacity challenges training efficiency and real-time deployment.

  • Problem

    Autonomous-driving models struggle to generalize across geographic locations, sensor configurations, weather, open-set objects, rare scenarios, and different driving intentions.

  • Method

    GenAD combines a large web-and-publicly licensed driving dataset with diverse text annotations and temporal reasoning for video prediction, then adapts the pretrained model to downstream driving tasks.

  • Results

    GenAD generalizes zero-shot to unseen driving datasets, surpasses prior methods in image fidelity and video coherence, and supports action-conditioned prediction and planning.

  • Takeaways & Limitations

    Video prediction can serve as a shared learned representation for driving-world understanding, simulation, and motion planning.

  • Takeaways & Limitations

    The increased model capacity creates challenges for training efficiency and real-time deployment.

Abstract

from arXiv · show

In this paper, we introduce the first large-scale video prediction model in the autonomous driving discipline. To eliminate the restriction of high-cost data collection and empower the generalization ability of our model, we acquire massive data from the web and pair it with diverse and high-quality text descriptions. The resultant dataset accumulates over 2000 hours of driving videos, spanning areas all over the world with diverse weather conditions and traffic scenarios. Inheriting the merits from recent latent diffusion models, our model, dubbed GenAD, handles the challenging dynamics in driving scenes with novel temporal reasoning blocks. We showcase that it can generalize to various unseen driving datasets in a zero-shot manner, surpassing general or driving-specific video prediction counterparts. Furthermore, GenAD can be adapted into an action-conditioned prediction model or a motion planner, holding great potential for real-world driving applications.

1. Introduction

GenAD targets the limited generalization of autonomous-driving models by using driving videos as a universal interface for dynamic environments and future behavior. The paper builds a large, diverse multimodal dataset and adapts video prediction to simulation and planning.

  • Motivation: Autonomous-driving models struggle to generalize across locations, sensors, weather, open-set objects, rare scenarios, and driving intentions.These limitations restrict deployment across diverse areas and scenarios.
  • Motivation: Driving videos are proposed as a universal interface for representing diverse environments, dynamic futures, world operation, and safe maneuvering.The motivation is to learn both how the world evolves and how to drive from future-frame prediction.
  • Approach: The proposed model targets a scalable predictive model that generalizes to new conditions and environments while supporting downstream tasks.The paper frames scalable data collection, dynamic-scenario modeling, and downstream application as its central questions.
  • Approach: OpenDV-2K combines web and publicly licensed driving data with diverse text descriptions, containing more than 2000 hours and 374 times the scale of nuScenes.The web data spans geographic locations, terrains, weather, safety-critical scenarios, sensor settings, and traffic elements, with human verification of collected recordings.
  • Applications: GenAD extends pretrained video prediction to simulation and planning, including trajectory-conditioned future-frame prediction and motion planning.The model is presented as adapting learned video-prediction knowledge to real-world driving problems.

2. OpenDV-2K Dataset

OpenDV-2K combines globally sourced YouTube driving videos with seven public datasets to create a large, diverse multimodal corpus. Its curation and language annotation pipeline filters low-quality footage and adds commands and frame contexts for predictive modeling.

  • OpenDV-2K contains 2059 hours of text-paired driving videos: 1747 hours from YouTube and 312 hours from public datasets.
  • Diversity over Prior Datasets: The dataset spans more than 40 countries and 244 cities, extending geographic coverage beyond typical restricted-area public datasets.
  • Diversity over Prior Datasets: Its open-world footage includes rare environments, heavy snow, and interactive traffic behaviors, while varied camera setups broaden sensor diversity.
  • Towards High-quality Multimodal Dataset: Curators selected 2139 front-view videos from 43 YouTubers, removed introductory, ending, black, and transition segments, and used BLIP-2 to describe frames.
  • Towards High-quality Multimodal Dataset: The pipeline pairs frame descriptions with ego-vehicle commands, using a classifier for 14 action types and free-form command expressions.
  • Towards High-quality Multimodal Dataset: Public-dataset metadata is enhanced with GPT-generated contexts and trajectory-based commands before integration with OpenDV-YouTube.

3. GenAD Framework

GenAD adapts an image diffusion model to driving and then trains it to predict future video frames. Temporal reasoning blocks model causal time dependencies and large spatial changes, while the learned representation supports action-conditioned prediction and planning.

  • Two-stage training: GenAD trains in two stages: image-domain transfer to driving, followed by video-prediction pre-training.The first stage adapts a text-to-image model; the second lifts it into a video prediction model.
  • Image domain transfer: The first stage fine-tunes an SDXL denoising UNet on driving image-text pairs while retaining frozen text encoders and autoencoder components.The model predicts noise from corrupted latents conditioned on text and commands.
  • Video prediction: The second stage predicts future frames from past observations and text by denoising corrupted future latents.Only outputs from corrupted frames contribute to the training loss, while condition-frame outputs are ignored.
  • Temporal reasoning: Temporal Reasoning Blocks combine causal temporal attention with two decoupled spatial attention layers to model temporal dependencies and fast perspective changes.The spatial layers propagate features horizontally and vertically with linear computational complexity rather than quadratic full attention.
  • Extensions: The inherited stage-one model is frozen while newly inserted temporal reasoning blocks are trained to learn video dependencies without perturbing image-generation ability.The learned model can also be adapted for action-conditioned prediction and planning using trajectory conditions or a lightweight MLP.

4. Experiments

Experiments evaluate GenAD on unseen datasets, conditional generation, ablations, action-conditioned prediction, and planning. GenAD shows strong zero-shot generalization and improves conditional simulation and planning efficiency across the reported tasks.

  • Video prediction: GenAD produces robust, realistic, and reasonable zero-shot predictions on unseen driving datasets from the same starting frames.The evaluation includes OpenDV-YouTube, Waymo, KITTI, and Cityscapes under geofenced unseen-set testing.
  • Video prediction: GenAD surpasses prior nuScenes-trained methods in both FID and FVD, reducing FVD by 44.5% versus DrivingDiffusion without 3D future layouts.A nuScenes-only variant performs comparably on nuScenes but degrades on unseen Waymo data, unlike the OpenDV-2K-trained model.
  • Conditional generation: Language-conditioned prediction generates different reasonable futures from the same rainy intersection frames under three high-level text conditions.The reported samples show environmental detail and natural ego-motion transitions.
  • Ablation study: Interleaved temporal blocks improve FVD by 17%, while temporal causality and decoupled spatial attention improve CLIPSIM.The ablation attributes the FVD improvement to more sufficient spatiotemporal interactions.
  • Extensions: GenAD-act reduces Action Prediction Error by 20.4% versus text-conditioned GenAD when predicting futures from six-waypoint trajectories.The action-conditioned model generates six future frames following the specified trajectory sequence.
  • Extensions: A frozen GenAD encoder with an additional MLP achieves competitive open-loop planning with 73× fewer trainable parameters and front-view images alone.Pre-extracting encoder features makes planning adaptation take 10 minutes on one NVIDIA Tesla V100, reported as 3400× more efficient than UniAD training.

5. Limitations and Discussion

The discussion identifies increased model capacity as a constraint on training efficiency and real-time deployment despite improved open-domain generalization.

  • Limitations: Increased model capacity challenges both training efficiency and real-time deployment, constraining GenAD’s practical use despite improved open-domain generalization.The authors propose distilling encoded knowledge for a wider range of downstream tasks.

A. Discussions

The discussions explain why video is a scalable interface for generalized driving models, while identifying unresolved evaluation, captioning, computation, and deployment limitations.

  • Why videos: Video prediction offers a pose-agnostic interface that scales across diverse data sources without requiring camera extrinsics or fixed LiDAR configurations.This supports broader generalization than representations tied to particular sensor setups.
  • Prediction design: Joint denoising avoids the multiplied computational cost of autoregressive multi-frame prediction and better supports conditions requiring substantial temporal changes.The discussion uses turning as an example where a short autoregressive horizon may not realize the commanded maneuver.
  • Evaluation: No specific criterion currently establishes generalized driving-video quality because judgments are subjective and aligned comparison methods are unavailable.The authors report that larger datasets improve zero-shot generation, but they do not claim a guaranteed data threshold.
  • Evaluation: Existing metrics inadequately evaluate multimodal driving prediction because SSIM and LPIPS favor recorded-frame matching, while FVD and CLIPSIM have alignment and realism limitations.The authors call for metrics capturing both multiple possible futures and temporal consistency.
  • Broader impact: The dataset and model are intended to support applications beyond video prediction, including planning, perception, simulation, and future model-predictive control.The authors state that action-conditioned futures may support trajectory planning, while broader downstream uses remain future work.
  • Limitations: OpenDV-2K captions generated by BLIP-2 can be short, weakly aligned, object-centric, and inconsistent with ego intentions.The authors suggest richer, temporally aware vision-language models as a preliminary direction.
  • Limitations: GenAD’s SDXL foundation provides visual quality and capacity but incurs slow sampling and high computational expense.Faster sampling and more efficient diffusion models are identified as future directions.

B.4. Video Datasets from the Internet

Internet video is attractive for autonomous-driving datasets because laboratory and crowdsourced collection is costly and difficult to scale, while online videos span diverse topics and environments.

  • Internet-scale data: YouTube-like websites provide scalable access to diverse video environments when laboratory or crowdsourced collection is costly and hard to scale.This motivates using Internet videos as a source for large-scale driving data.

C. OpenDV-2K Dataset

OpenDV-2K is presented as a large public driving dataset combining 2059 hours of video with diverse textual contexts and commands.

  • Dataset overview: OpenDV-2K contains 2059 hours of driving video paired with diverse text conditions, including contexts and commands.The dataset includes curated YouTube footage and publicly licensed data, with collection and annotation details described in later subsections.

C.1.1 Data Collection

OpenDV-YouTube is constructed from globally sourced, quality-filtered driving videos and annotated with frame contexts plus ego-driver commands for multimodal training and control.

  • Data collection: 2139 high-quality front-view videos were selected from 43 worldwide YouTubers, with three uploaders reserved entirely for validation.The split is organized by YouTuber to support zero-shot evaluation across sources.
  • Data processing: Videos are converted into consecutive image frames and sampled at 10Hz, while non-driving introductions, reminders, black frames, and transitions are removed.The preprocessing targets consistent training inputs and reduced unwanted noise.
  • Language annotation: Contexts describe frame-level objects and scenarios, supporting text-to-image learning and recognition of open-world driving content.BLIP-2 generates these frame descriptions using a dedicated prompt.
  • Language annotation: Commands represent ego actions and enable language-controlled future prediction, with 13 action categories inferred using optical-flow-based annotation.The categories include driving, turning, lane changes, merging, U-turns, stopping or decelerating, and deviation.

C.1.3 Analyses Methods

OpenDV-YouTube is analyzed by inferring geographic, weather, and filming-period information from video titles, with rule-based post-processing for geographic labels.

  • GPT-3.5-turbo infers each video's geographic information from its title, followed by handmade rules that normalize city and country aliases.
  • The analysis assumes all clips from one video share a location, while videos with multiple inferred locations are divided across sites.
  • The geographic prompt requests concise city, state, and country names in English.
  • Weather and filming period are inferred from titles because terms such as “foggy” and “night” often appear there.

C.1.4 Diversity Highlights

OpenDV-2K combines geographically broad, visually varied, and scenario-rich driving data with annotations drawn from YouTube and public datasets, while acknowledging estimation and annotation-pattern limits.

  • OpenDV-YouTube spans over 244 cities in more than 40 countries, with validation locations unseen during training.It includes 36.4M clips from 40 US states and 12.9M clips from 26 Chinese provinces, while validation adds Bosnia and Herzegovina, Denmark, Hungary, and Maine.
  • Online videos provide diverse camera equipment, color settings, intrinsic parameters, and camera poses, including a bus-mounted front-view case absent from training.
  • A word cloud summarizes the 500 most frequent objects, agents, or scenarios in frame descriptions.
  • OpenDV-YouTube contains varied driver actions, weather conditions, filming periods, scenes, and safety-critical corner cases.Examples include dark tunnels, crowded nighttime intersections, beaches at sunset, rooftops, and rain-obscured views.
  • Annotations from YouTube follow limited syntax patterns, so seven public datasets are merged after converting labels into grammatical sentences and varied expressions.
  • Driver commands are generated or inherited from trajectories, camera poses, intentions, classifiers, and cleaned labeled datasets, then expressed through paraphrasing templates.
  • GenAD uses a 5.9B-parameter architecture built on SDXL with interleaved temporal reasoning blocks for video prediction.
  • Trajectory waypoints are embedded and projected as an additional condition, while frozen GenAD features support downstream policy learning.

D.2. Training Details

GenAD is trained in two stages: image-domain transfer followed by video-level denoising, with separate adaptations for action-conditioned prediction and planning.

  • The first stage fine-tunes SDXL on 65.1M image-text pairs using 2.7B trainable denoising-UNet parameters.
  • The second stage freezes the fine-tuned image model and trains 2.5B temporal-reasoning parameters for iterative future-frame prediction.
  • Both stages resize frames to 256 × 448 and drop text conditions with probability p = 0.1 for classifier-free guidance.
  • Action-conditioned prediction fine-tunes GenAD and its trajectory projection layer on nuScenes, while planning trains a lightweight MLP on frozen GenAD features.
  • At inference, GenAD uses two past frames and text to iteratively denoise six future frames with DDIM at 2Hz.Sampling uses 100 steps and classifier-free guidance scale 7.5.

E.1. Data Preparation

The evaluation uses unseen validation data, multiple video-quality and trajectory metrics, and qualitative tests covering image generation, zero-shot transfer, action conditioning, and failure cases.

  • All reported results use validation data completely unseen during GenAD training across OpenDV-YouTube, Waymo, KITTI, Cityscapes, and nuScenes.
  • Evaluation metrics include FID, FVD, CLIPSIM, Action Prediction Error, ADE, and FDE.
  • FID measures feature-distribution distance between generated and original frames, while FVD measures semantic similarity between real and synthesized videos.
  • CLIPSIM averages CLIP-feature similarity between six generated frames and the first conditional frame.
  • Action Prediction Error compares input and inferred trajectories, while ADE and FDE compare planned and ground-truth trajectories.
  • The fine-tuned image model generates realistic driving views with abundant visual detail, supporting subsequent futuristic-scene simulation.
  • GenAD generalizes zero-shot across four unseen datasets and produces trajectory-controlled futures when fine-tuned with an additional trajectory condition.
  • Failure cases include misleading contexts, poor human details, unsmooth motion, and inability to maintain performance under out-of-distribution camera height for 3 seconds.
Loading 2403.09630v2…