PaperlayerView exact revision on arXiv

Source-linked AI summary

Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models

Jiahao Wang, Zhenpei Yang, Yijing Bai, Yingwei Li, Yuliang Zou, Bo Sun, Abhijit Kundu, Jose Lezama, Luna Yue Huang, Zehao Zhu, Jyh-Jing Hwang, Dragomir Anguelov, Mingxing Tan, Chiyu Max Jiang

arXiv:2510.06209v1cs.CV

TL;DR

The paper asks whether generated driving videos elicit planner responses comparable to real videos and how E2E planner performance varies across domains. Drive&Gen compares planner behavior across matched real and synthetic scenes and uses controllable video generation for evaluation. It reports improved E2E planner generalization in out-of-distribution scenarios, while noting that its realism test does not assess physical fidelity or road safety.

  • Problem

    It remains unclear whether visual quality in generated driving videos corresponds to planner responses, and evaluating E2E planners on out-of-distribution domains is a challenge.

    Source: §I · pp. 1–2

  • Method

    Drive&Gen compares planner responses to real and generated videos with matched scene layouts, using controllable video generation and the Behavior Permutation Test to assess realism.

    Source: §I · p. 2, A. Evaluation of Controllable Video… · p. 5

  • Results

    The paper reports that synthetic data generated by its model can improve E2E planner generalization in out-of-distribution scenarios.

    Source: §VI · p. 7

  • Takeaways & Limitations

    The framework provides a methodology to systematically assess components of driving-video generation and E2E planning.

    Source: §V · p. 7

    BPT compares planner-output distributions under an identical-ground-truth-trajectory assumption across environmental contexts, but does not assess physical realism or road-safety implications.

    Source: §V · p. 7

Abstract

from arXiv · show

Recent advances in generative models have sparked exciting new possibilities in the field of autonomous vehicles. Specifically, video generation models are now being explored as controllable virtual testing environments. Simultaneously, end-to-end (E2E) driving models have emerged as a streamlined alternative to conventional modular autonomous driving systems, gaining popularity for their simplicity and scalability. However, the application of these techniques to simulation and planning raises important questions. First, while video generation models can generate increasingly realistic videos, can these videos faithfully adhere to the specified conditions and be realistic enough for E2E autonomous planner evaluation? Second, given that data is crucial for understanding and controlling E2E planners, how can we gain deeper insights into their biases and improve their ability to generalize to out-of-distribution scenarios? In this work, we bridge the gap between the driving models and generative world models (Drive&Gen) to address these questions. We propose novel statistical measures leveraging E2E drivers to evaluate the realism of generated videos. By exploiting the controllability of the video generation model, we conduct targeted experiments to investigate distribution gaps affecting E2E planner performance. Finally, we show that synthetic data produced by the video generation model offers a cost-effective alternative to real-world data collection. This synthetic data effectively improves E2E model generalization beyond existing Operational Design Domains, facilitating the expansion of autonomous vehicle services into new operational contexts.

I. INTRODUCTION

Drive&Gen addresses whether generated driving videos are realistic from a planner’s perspective and how E2E planners perform beyond their training domains. It uses controllable video generation and planner behavior to evaluate realism, compare performance across conditions, and test synthetic data for improving generalization.

I. INTRODUCTION

  • The motivating gaps are whether visual quality in generated videos corresponds to planner responses and how to evaluate simplified E2E models, especially on out-of-distribution domains.Real-world AV testing is described as expensive and time-consuming.Source: §I · pp. 1–2
  • The approach assumes expected driving behavior depends largely on traffic-scene layout and mostly independently of visual features such as weather, lighting, and agent appearance.The model can generate the same underlying traffic scenario under different visual conditions.Source: §I · p. 2
  • Drive&Gen evaluates video realism through planner behavior, comparing responses to real and synthetic videos with matched scene layouts and visual conditions.The framework introduces statistical measures from the E2E driving-model perspective.Source: §I · p. 2
  • Synthetic data generated by the video model can improve E2E planner generalization to out-of-distribution scenarios.The paper presents this as an effective mechanism for generalization beyond existing operational domains.Source: §I · p. 2
  • Controllable videos enable experiments on planner performance across operational domains, including in-distribution versus out-of-distribution settings.The experiments also support model diagnostics and assessment of readiness for new domains; Figure 2 shows rain and nighttime condition variants.Source: §I · p. 2, Figure 2 · p. 2

II. RELATED WORK

Related work covers controllable video world models, E2E planning approaches, and planner evaluation. The paper distinguishes its focus on directly measuring synthetic-video realism from work that uses generated data mainly for training.

World Models

  • Driving world models use video generation conditioned on controls, with prior work spanning future-world generation and controllable multi-view videos.Examples include GAIA-1 and Vista, as well as DriveDreamer, DrivingDiffusion, MagicDrive, and Panacea.Source: §II · pp. 2–3

End-to-end Planning Models

  • E2E planning research includes methods that learn from driving-video sequences, connect modules through gradient back-propagation, and use pretrained vision-language models.DriveVLM uses chain-of-thought prompting for hierarchical planning signals; Atlas incorporates 3D signals into a VLM.Source: §II · p. 3

Planner and World Model Evaluation

  • Planner evaluation uses open-loop comparisons with ground-truth labels and closed-loop operation in simulators or real environments, while this paper directly measures simulation realism.The passage notes that existing studies generally use synthetic data as additional fine-tuning material and report performance gains rather than directly evaluating its realism.Source: §II · p. 3

III. METHOD

Drive&Gen evaluates generated videos through both video-generation metrics and the responses of an E2E planner to real and synthetic scenes. Its Behavior Permutation Test compares planner trajectory sets to assess whether those responses differ.

A. Co-evaluation

  • Drive&Gen compares planner responses to real and synthetic videos of equivalent scenes, while varying generation conditions to examine planner behavior across scenarios.The framework also supports controlled experiments on planner performance under varied operational design domains.Source: A. Co-evaluation · p. 3, §III · p. 3
  • The Behavior Permutation Test (BPT) assesses whether the planner responds behaviorally similarly to generated and real scenes.For each scene, it compares sampled trajectory sets using a generalized Chamfer-distance statistic and permutation testing.Source: A. Co-evaluation · p. 3
  • ADE measures mean L2 distance between predicted and ground-truth trajectories, while FVD measures real-versus-generated video distribution distance in latent feature space.The authors note that ADE may not distinguish trajectories that deviate in opposite directions, and FVD alone does not measure adherence to conditions.Source: A. Co-evaluation · p. 3
  • BPT tests whether real-data and generated-data trajectory sets come from the same distribution; p < 0.05 indicates significantly different planner behavior.The reported p-value estimates the probability that the observed difference is due solely to random chance.Source: A. Co-evaluation · pp. 3–4

B. Video Generation Model

The video generator extends pretrained W.A.L.T with controls for driving-scene layout and operating conditions. It represents those controls as embeddings that condition the diffusion model.

B. Video Generation Model

  • The model extends pretrained W.A.L.T with controls for bounding boxes, road maps, ego-car pose, time of day, and weather.The model is fine-tuned on a large corpus of driving videos.Source: B. Video Generation Model · p. 4
  • Bounding boxes and road-map segments are encoded as features in ego-vehicle coordinates, with latent-query attention reducing road-map token count.Bounding-box and road-map inputs have maximums of 256 boxes per frame and 4,096 line segments, respectively.Source: B. Video Generation Model · p. 4
  • Time-of-day is represented through solar azimuth and elevation derived from local time, date, and location, while weather uses one-hot encoding.The solar angles support lighting control across different seasons and geographic locations.Source: B. Video Generation Model · p. 4
  • Condition embeddings are concatenated, encoded by a transformer, and injected into the diffusion model through cross-attention and adaptive feature scaling and shifting.A pooled condition representation modulates self-attention and feed-forward layers.Source: B. Video Generation Model · p. 4

C. End-to-end Driving Model

The E2E planner fine-tunes a pretrained vision-language model to generate waypoint trajectories from collaged temporal images, vehicle state, and routing instructions.

C. End-to-end Driving Model

  • The E2E planner is based on pretrained PaLI [42] and follows EMMA [1], with a 3×3 image collage encoding temporal frames as 1,536 tokens.The earliest frame occupies the collage’s top-left position.Source: C. End-to-end Driving Model · p. 4
  • The planner combines image tokens with text inputs for the vehicle’s past states and routing instructions, then generates trajectories as waypoint values.This frames planning as a visual question answering task.Source: C. End-to-end Driving Model · p. 4
  • The model fine-tunes the language decoder while keeping the vision encoder fixed, using cross-entropy loss.The reported video-generation evaluation uses ADE@5s and BPT alongside FVD; removing bounding-box conditioning notably worsens ADE and BPT.Source: C. End-to-end Driving Model · p. 4, Figure 4 · p. 5

IV. EXPERIMENTS

The experiments evaluate controllable video generation, planner behavior on generated scenarios, and synthetic-data effects on planner performance. The paper also states scope boundaries for its evaluation and describes the training data and steps.

IV. EXPERIMENTS

  • The evaluation covers condition alignment and real-versus-generated video similarity, varied driving scenarios for planner testing, and synthetic-data effects on planner performance.The authors state that improving E2E planner performance with generated data is demonstrated in Sec. IV-C.Source: §IV · p. 5
  • The study focuses on co-evaluation rather than state-of-the-art video generation, so detailed FVD and resolution comparisons are outside its scope.The authors also note that current methods lack the fine-grained controls needed for comprehensive planner evaluation, and UniAD’s deterministic prediction is incompatible with BPT.Source: §IV · p. 5
  • The video model is trained on about ten million driving segments, with 1% held out for testing, for 700k steps; the E2E planner is fine-tuned for 120k steps.Each segment has 17 frames at 10 Hz and 128 × 128 resolution, with driving and environmental features; conditions are randomly dropped during training with probability 0.1.Source: §IV · p. 5

A. Evaluation of Controllable Video Generation

This section evaluates generated-video realism through planner responses, since FVD and ADE do not by themselves capture whether synthetic inputs elicit similar plans. BPT compares predicted-trajectory sets across real and generated videos, alongside controlled comparisons of generation conditions.

A. Evaluation of Controllable Video Generation

  • Night-time generated videos have disproportionately worse FVD than videos generated under logged-data conditions despite comparable visual realism.FVD measures distributional differences, not necessarily visual quality.Source: A. Evaluation of Controllable Video… · p. 5
  • ADE cannot readily distinguish video-generation quality from planner performance, and trajectories with opposite deviations can have similar ADE.Higher ADE under conditions such as rain or night may reflect planner performance rather than unrealistic video inputs.Source: A. Evaluation of Controllable Video… · p. 5
  • BPT compares sets of predicted trajectories; its fail-to-reject rate has a 95% ceiling because the test rejects cases with p < 0.05.Similar sets fall within the permuted distributions, whereas significantly different sets lead to rejection.Source: A. Evaluation of Controllable Video… · p. 5, Figure 5 · p. 5
  • 69.62% BPT failure-to-reject rate (out of the 95% expected ceiling) indicates broadly similar planner responses to real and synthetic videos under matched conditions.Removing scene-layout constraints sharply lowers the rate, while rain and night conditions produce smaller but non-negligible behavior changes.Source: A. Evaluation of Controllable Video… · p. 6
  • Sun-angle time-of-day encoding outperforms local-time conditioning on FVD, ADE, and BPT.The comparison is reported as an ablation of the time-of-day encoding.Source: A. Evaluation of Controllable Video… · p. 6, Table I · p. 6

B. Evaluation of End-to-End Planner

This section uses controlled conditions to examine planner performance across scene layouts and operational settings. Removing scene-layout inputs has a larger reported impact on ADE than removing weather or time-of-day inputs, while performance also varies across weather and time.

B. Evaluation of End-to-End Planner

C. Improving Planner with Generated Videos

Fine-tuning with generated and real videos improves planner performance on real-world data, including rainy and nighttime scenarios. Nighttime traffic and illumination complicate interpretation of the results.

C. Improving Planner with Generated Videos

V. LIMITATIONS AND DISCUSSIONS

The Behavior Permutation Test evaluates realism through planner-output distributions under identical-ground-truth-trajectory assumptions, but does not assess physical fidelity or road safety.

V. LIMITATIONS AND DISCUSSIONS

  • BPT focuses on planner-output distributions under the assumption of identical ground-truth trajectories across environmental contexts.Its conclusions are therefore bounded by that assumption.Source: §V · p. 7
  • BPT does not inherently assess the physical realism of generated environments.The authors leave deeper investigation of physical-realism fidelity to future work.Source: §V · p. 7
  • BPT does not establish implications for road safety, and no single metric fully captures real-world driving complexity.Source: §V · p. 7

VI. CONCLUSION

Drive&Gen co-evaluates driving video generation and E2E planning using planner behavior to assess video realism. The authors report improved planner generalization in out-of-distribution scenarios with generated data.

VI. CONCLUSION

  • Drive&Gen introduces a framework for co-evaluating driving video generation and E2E planning.Source: §VI · p. 7
  • The Behavior Permutation Test assesses video realism by analyzing the distribution of a planner’s outputs.The paper describes this as the first attempt to evaluate driving video generation using a VLM-based driving model.Source: §VI · p. 7
  • The authors report that generated data improves E2E planner generalization in out-of-distribution scenarios.Their video generation model provides precise control over scene layout and operating conditions such as weather and time of day.Source: §VI · p. 7
Loading 2510.06209v1…