UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving

1 Introduction

• VLA methods cannot fully use unlabeled driving videos for visual causal learning, while world models lack large language models’ reasoning and interaction capabilities.The introduction also identifies open challenges in efficiently unifying pretrained VLMs with world models, repeatedly using diverse driving data, and evaluating complex-scenario understanding, reasoning, and planning.Source: §1 · pp. 1–2
• UniUGP jointly models scene reasoning, future-video generation, and trajectory planning through a hybrid expert architecture.It takes language instructions and image sequences as input, producing chain-of-thought reasoning, physically consistent trajectories, and coherent future videos.Source: §1 · pp. 2–3
• A four-stage training strategy progressively develops scene understanding, visual dynamics, text-based reasoning, and multi-capability fusion across more than 10 AD datasets.A multi-term loss promotes CoT logical consistency, trajectory temporal smoothness, and video visual coherence.Source: §1 · pp. 2–3
• The paper constructs specialized AD datasets with explanations, reasoning, and planning annotations for complex scenarios.These datasets support AD-oriented VLAs.Source: §1 · p. 2

2.1 VLA for Autonomous Driving

• VLA methods progressed from high-level meta-actions to direct trajectory prediction, but often cannot use unlabeled videos for visual causal reasoning.Approaches include text-based trajectory representations, discretized action tokens, and diffusion-based continuous trajectory generation.Source: §2.1 · p. 3

2.2 World Models for Autonomous Driving

• World models predict dynamic surroundings from historical observations, but often underuse pretrained VLMs and video generation models and rely on limited single-dataset scenarios.Prior work spans scenario generation, planning, and representation learning.Source: §2.2 · p. 3

2.3 Unified Models

• Existing unified AD models lack capabilities such as CoT reasoning, natural-language interaction, diffusion-based planning, and large-scale unlabeled-data use.Prior unified approaches aim to bridge understanding and generation, with some extending unified architectures to action.Source: §2.3 · p. 3

3 Method

• UniUGP unifies scene reasoning, future-video generation, and trajectory planning through a hybrid-expert architecture.The understanding expert predicts next tokens for causal reasoning, the planning expert jointly models with it to predict action velocities, and the generation expert produces future videos as a world model.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Challenging Long-tail Driving Dataset

• The authors collect challenging long-tail driving videos from Waymo-E2E, DADA2000, LaF, StHa, SOM, and AADV, then process them for perception, causal reasoning, planning, and instruction following.Comprehension tasks cover small objects, accident-subject relationships, and accident prediction; chain-of-thought annotations are manually calibrated.Source: §3.1 · p. 4

3.2 Unified Model With Hybrid Expert

• The understanding and planning experts form a Mixture-of-Transformers architecture, with the understanding expert predicting text and the planning expert learning action chunks through flow matching.Understanding and planning tokens pass through MoT layers, whose resulting hidden states produce text logits and a predicted planning denoising vector.Source: §3.2 · pp. 4–5
• The generation expert uses flow matching to produce future videos, conditioning denoising on history and noised future image tokens plus understanding states and action embeddings.At inference, action embeddings are projected from the planning expert's predicted future actions; the generation expert can be disabled on mobile devices without compromising the former experts' performance.Source: §3.2 · pp. 5–6

3.3 Training Recipe

• Training proceeds from scenario understanding to visual-dynamics and planning learning, then text-based causal reasoning, using long-tail data, ImpromptuVLA, public driving datasets, and an annotated chain-of-thought dataset.Stage 1 trains only the understanding expert; Stage 2 trains generation and planning experts with driving videos and trajectories; Stage 3 adds chain-of-thought reasoning to the understanding expert.Source: §3.3 · pp. 6–7
• In Stage 4, all three experts are jointly trained on a fixed-proportion mixture of earlier-stage datasets with a weighted sum of understanding, planning, and generation objectives.The objective weights are α = 0.3, β = 0.5, and γ = 0.2, respectively.Source: §3.3 · p. 7

4.1 Dataset

• Evaluation spans anomaly and accident datasets, 4,021 long-tail Waymo segments, DriveLM scene QA, and nuScenes planning and generation data.The Waymo segments capture rare, high-risk events; DriveLM provides full-stack driving QA annotations.Source: §4.1 · p. 7

4.2 Metrics

• The new long-tail benchmark covers understanding, CoT reasoning, planning, and instruction following, using accuracy, GPT-4o scores, L2 trajectory metrics, FID, and DriveLM language and QA metrics.Planning is also evaluated with collision rate; DriveLM uses BLEU, ROUGE_L, CIDEr, ChatGPT Score, and multiple-choice accuracy.Source: §4.2 · pp. 7–8

4.3 Evaluation of Understanding Ability

• UniUGP surpasses GPT-4o, Qwen-2.5-VL-72B, and both ablations across key metrics: Small 89.3%, Relationship 88.6%, Abnor. Pred. 95.8%, GPT 0.88, Blue 0.240, Planning L2 1.45, and Following L2 1.40.The benchmark compares understanding, CoT, planning, and following; GPT-4o and Qwen-2.5-VL-72B receive historical trajectories and trajectory explanations for trajectory evaluation.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 Evaluation of Planning Ability

• UniUGP records average L2 1.23m and collision rate 0.33%, improving on Doe-1's 1.26m and 0.53% with matching Camera*+QA input.The comparison table reports end-to-end motion planning on nuScenes; Camera* denotes front-camera-only input.Source: §4.4 · p. 8, Table 4 · p. 9

4.5 Evaluation of Generation Ability

• Under the Epona/FSDrive protocol, UniUGP significantly improves future-frame generation quality; trajectory-fed changes control its generated future frames.The trajectory-controllable visualization demonstrates controllability of the generation experts.Source: Table 5 · p. 9, §4.5 · p. 9, Figure 4 · p. 10

4.6 Results on DriveLM Dataset

• UniUGP leads DriveLM GVQA with 0.59 versus FSDrive 0.57, OmniDrive 0.56, and the DriveLM baseline 0.32.Its accuracy is 0.74, BLEU 0.78, ROUGE 0.76, and match 0.41, each reported as leading comparative methods.Source: §4.6 · pp. 9–10, Table 6 · p. 10

• UniUGP addresses challenges in temporal dynamics and long-tail generalization.Source: §5 · p. 10
• Its hybrid architecture combines pre-trained VLMs and video generation models to produce interpretable reasoning, physically consistent trajectories, and coherent future videos.Source: §5 · p. 10
• A four-stage training strategy progressively aligns these capabilities across diverse datasets, and experiments report state-of-the-art performance and robust generalization.Source: §5 · p. 10

• The authors are Hao Lu, Ziyang Liu, Guangfeng Jiang, Yuanfei Luo, Sheng Chen, Yangang Zhang, and Ying-Cong Chen.Source: Contributions · p. 10
• Affiliations listed are HKUST-GZ, ByteDance Seed, and Independent Researcher.Source: Contributions · p. 10
• Hao Lu and Ziyang Liu are co-first authors; Hao Lu and Yuanfei Luo are project leads; Ying-Cong Chen is corresponding author.Hao Lu's work was done at ByteDance Seed during an internship.Source: Contributions · p. 10

• The benchmark aims to measure understanding, reasoning, and action in diverse real-world conditions, beyond structured-scene or closed-loop simulation metrics.Source: A Dataset · p. 17
• The data combine road-abnormality and accident-anticipation datasets with Waymo scenes containing densely interactive, ambiguous safety-critical events.Source: A Dataset · p. 17
• The four task dimensions are perception and understanding, causal CoT reasoning, planning and decision-making, and instruction following.They cover visual semantics and risk awareness, motion-intention explanations, physically feasible trajectories, and alignment with high-level navigation commands.Source: A Dataset · p. 17

A.1 Perception and Understanding:

• Anomaly and hazard annotations are converted into true/false and multiple-choice questions, while accident footage and abnormal entities support questions about accident relationships.The accident-relationship questions are based on abnormal traffic footage annotated with the entities involved.Source: §A.1 · p. 18
• The questions cover semantic comprehension, driving commonsense, corner cases, and rare or unseen categories, including why objects matter for safety.The intended capability extends from identifying scene contents to understanding their safety relevance.Source: §A.1 · p. 18
• Small elongated objects are identified from dataset segmentation labels and used to construct judgment questions with varied wording.The question format uses randomized variants to encourage answer generalization.Source: §A.1 · p. 18
• Accident-prediction questions classify videos as abnormal using the dataset indication and the relevant timestamp.Source: §A.1 · p. 18

A.2 Causal CoT Reasoning:

• For observable future outcomes, QA answers use structured multi-step reasoning aligned with future images and the ground-truth ego trajectory.The reasoning describes scene context, interactive agents and intentions, then justifies the action leading to the observed behavior.Source: §A.2 · p. 20

A.3 Planning and Instruction Following

• Planning predicts K future trajectory points in ego coordinates, either directly or after CoT when complex interactions benefit from reasoning.The ego-local frame places x forward and y left.Source: §A.2 · p. 20, §A.3 · p. 22
• Navigation commands derived from the ground-truth future trajectory condition training so predicted motion aligns with the specified intent.The setup also evaluates whether motion adjusts correctly to the high-level instruction.Source: §A.3 · p. 22

B More Cases

• The examples compare UniUGP and GPT4o on understanding and reasoning, with UniUGP providing more specific planning suggestions.Source: B More Cases · p. 22
• Trajectory and weather-control visualizations are presented as examples of the method’s generation capability.Source: B More Cases · p. 22

C Limitation and Future Directions

• The authors state that UniUGP has remaining limitations and point to future research directions.Source: C Limitation and Future Directions · p. 22

C.1 Limitations

• Extreme rare-event generalization is constrained by training-data coverage, while the resource-intensive generation expert must be disabled on mobile platforms for real-time performance.The authors also note that linguistic reasoning may not tightly align with physically consistent trajectories in complex interactions such as ambiguous pedestrian–vehicle encounters.Source: §C.1 · p. 22
• Fixed dataset proportions in final-stage fusion cannot adapt to datasets’ complementary strengths, limiting task synergy.Source: §C.1 · p. 23

C.2 Future Directions

• Proposed directions include synthetic data and few-shot or zero-shot learning for rare events, lighter generation experts, improved multimodal alignment, self-supervision, and continual learning.The authors also propose real-time feedback, multi-agent reasoning, and closed-loop real-world testing.Source: §C.2 · p. 23
• The visualizations show controllable generation across different weather and trajectory conditions, while the CoT comparison describes UniUGP’s planning as more specific than GPT4o’s.Source: Figure 7 · p. 24, Figure 8 · p. 25, Figure 9 · p. 26