PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving

Xin Zhou, Zongchuang Zhao, Zhibo Yang, Mingsheng Li, Humen Zhong, Shuai Bai, Du Chu, Ruizhe Chen, Zhaohai Li, Jun Tang, Qiuyue Wang, Mingkun Yang, Jiazhao Zhang, Dayiheng Liu, Dingkang Liang, Xiang Bai

arXiv:2609.00111v1cs.CV

TL;DR

Qwen-Drive-1.0 targets the gap between broad VLM capability and driving-specific 3D understanding and planning. It adds external perception and planning modules to a pretrained VLM and trains them with staged, mixed-domain supervision. The resulting system is highly competitive across 3D perception, driving scene understanding, and motion planning while largely preserving general vision-language capability.

  • Problem

    VQA-centered driving adaptation does not directly constrain or evaluate 3D structure and can cause catastrophic forgetting of general pretrained knowledge.

  • Method

    Qwen-Drive-1.0 retains the pretrained VLM architecture, adds an external BEV perception head and Planning Expert, and uses staged training with driving and general vision-language supervision.

  • Results

    43.95 mAP and 60.99 map mIoU on nuScenes, alongside competitive driving scene understanding and motion planning across open-loop, pseudo-closed-loop, and closed-loop evaluations.

  • Takeaways & Limitations

    Explicit 3D perception and trajectory generation can be added to a pretrained VLM while retaining broader vision-language competence.

  • Takeaways & Limitations

    Planning reasoning can remain unstable when identifying a scene’s governing cause and temporal scope, and generated trajectories may not always follow the textual rationale.

Abstract

from arXiv · show

We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird's-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.

1 Introduction

Qwen-Drive-1.0 addresses limitations of VQA-centered driving adaptation by unifying explicit 3D perception, scene understanding, and motion planning while preserving general-purpose VLM capabilities. Its staged training and unified data pipeline support competitive performance across perception and planning tasks.

  • Motivation: VQA-based adaptation can produce fluent scene descriptions without directly constraining or evaluating 3D layout, depth, and occupancy.It also risks catastrophic forgetting of general knowledge acquired during pretraining.
  • Results: 43.95 mAP and 60.99 map mIoU on nuScenes, plus 43.45 mAP and 71.27 map mIoU on OpenScene, demonstrate competitive 3D perception.The model also significantly surpasses Qwen3.5-4B in driving scene understanding while preserving general capability.
  • Results: Motion planning reaches a Predictive Driver Model Score of 90.7 on NAVSIM and a Rater Feedback Score of 7.91 on WOD-E2E.Evaluations span open-loop, pseudoclosed-loop, and closed-loop settings, including promising closed-loop potential in AlpaSim.
  • Approach: Qwen-Drive-1.0 unifies 3D perception, driving VQA, and motion planning within a pretrained VLM without changing its architecture.The framework uses external modules for explicit 3D predictions and future ego-trajectory generation.
  • Approach: The external BEV perception head jointly learns 3D detection, semantic occupancy prediction, and BEV map segmentation as explicit, inspectable outputs.It probes the 3D scene information available in shared VLM representations.
  • Training: A staged training and data recipe combines driving supervision with general-purpose vision-language data while unifying labels, VQA responses, and trajectory representations.The recipe is designed to support domain adaptation while mitigating catastrophic forgetting.

2 Method

Qwen-Drive-1.0 keeps the pretrained VLM unchanged while adding external BEV perception and planning modules, supported by task-specific input serialization and staged optimization. Its method addresses explicit 3D prediction, trajectory generation, task-level reward optimization, and cross-dataset annotation constraints.

  • Unified architecture: The shared vision encoder and VLM process driving and general images, while external modules reuse their features without architectural changes.The pathway supports single-view, multi-view, temporal, and general-image inputs.
  • Input serialization: View and frame tags identify image direction and timestep, with frame-major serialization for question answering and view-major serialization for planning.View-major order keeps consecutive observations from each view adjacent, exposing temporal variation for control.
  • BEV perception: The BEV perception head constructs a shared ego-frame representation for 3D detection, semantic occupancy prediction, and BEV map segmentation.It combines low-level vision-encoder features with contextual VLM features and uses depth-based view transformation before BEV aggregation.
  • BEV perception: The perception branches use deformable-attention detection, height-aware 3D occupancy prediction, and UNet-style map segmentation with jointly optimized losses.Detection uses Hungarian matching and combines focal and ℓ1 losses; occupancy and map objectives include task-specific segmentation losses.
  • Training and data: Four-stage training first adapts the shared pathway for perception and VQA, then trains planning by flow matching and reward-based optimization.The reward stage uses group-relative advantages and a policy gradient over selected stochastic transitions, while label unification aligns heterogeneous perception sources.

3 Experiments

Experiments evaluate unified 3D perception, driving VQA, general vision-language retention, and causal reasoning. Qwen-Drive-1.0-SFT performs strongly across these evaluations, while cross-dataset occupancy remains constrained by source-specific label artifacts.

  • 3.1 3D Perception: Qwen-Drive-1.0-SFT establishes the best nuScenes detection and map-segmentation results, exceeding BEVFormerV2* by 2.01 mAP, BEVFormerV2 by 3.05 NDS, and PETRv2 by 3.37 map mIoU.On OpenScene, it also improves over Head-only (joint) in both detection metrics and map mIoU.
  • 3.1 3D Perception: Mixed-source training raises OpenScene NDS from 16.50 to 41.86, but occupancy mIoU decreases by 26.6% on nuScenes because source-specific voxel-label artifacts limit joint adaptation.The residual differences between label semantics and annotation pipelines introduce negative transfer, especially for occupancy.
  • 3.1 3D Perception: Joint adaptation increases nuScenes mAP and map mIoU by 10.46 and 9.84 points, while OpenScene map mIoU gains a further 4.93 points after perception objectives update the pretrained encoder and VLM.These gains cannot be explained by continued head optimization alone because the head-only model had already converged.
  • 3.2.1 Driving Visual Question Answering: Driving QA average rises from 63.52 for Qwen3.5-4B to 69.43, with gains distributed across complementary capabilities including LingoQA and SURDS.LingoQA rises by 7.40 points and SURDS by 13.18 points.
  • 3.2.1 Driving Visual Question Answering: Causal-reasoning average reaches 58.30 versus 22.05 for Gemma4-12B, while overall accuracy reaches 41.26, more than seven times Cosmos-Reason2-32B’s 5.73.Qwen-Drive-1.0-SFT improves key-object, decision, and overall accuracy over Qwen3.5-4B by 56.45, 46.42, and 38.68 points, respectively.
  • 3.2.2 General Vision-Language Capability: After driving adaptation, general vision-language performance averages 66.41 versus 67.40 for Qwen3.5-4B on knowledge, reasoning, and recognition benchmarks, while specialized-model comparisons remain favorable.Qwen-Drive-1.0-SFT matches or exceeds Cosmos-Reason2-32B on 10 of 15 settings and achieves a 5.48-point higher average.

4 Conclusion

Qwen-Drive-1.0 unifies explicit 3D perception and motion planning with a pretrained VLM while preserving its architecture. Experiments report competitive performance across perception, scene understanding, planning, and general vision-language capability.

  • The framework adds external modules for unified 3D perception and motion planning without changing the pretrained VLM architecture.The BEV perception head provides explicit detection, occupancy, and map predictions, while the Planning Expert generates trajectories through flow matching.
  • Experiments show highly competitive results in 3D perception, driving scene understanding, and motion planning while largely preserving general vision-language capability.Planning is evaluated in open-loop, pseudo-closed-loop, and closed-loop settings.

5 Limitations and Future Work

The paper identifies limitations in causal planning reasoning, rationale–trajectory consistency, and cross-task transfer. It proposes multi-timescale causal modeling, consistency supervision, and better alignment of task configurations as future directions.

  • Planning reasoning can misidentify the governing cause and its temporal scope when causes with different time scales coexist.A stated immediate cause may not match the decision executed within the next 1 to 2 s.
  • Generated trajectories do not always adhere to textual rationales, although reasoning improves downstream planning performance.Some of the gain may come from model-internal information contributed by the self-generated trace.
  • Different input formats, temporal contexts, and image resolutions across the three tasks may limit transfer of learned representations.The paper suggests aligning these configurations and optimizing the tasks more closely together.

Authors

The supplied passages list contributors, affiliations, reinforcement-learning reward definitions, and qualitative capability demonstrations. They cover multi-source reward construction, displacement-error terms, and examples of camera-based 3D grounding.

  • Authors: The paper lists Xin Zhou, Zongchuang Zhao, Zhibo Yang, and additional authors as core contributors, alongside alphabetical contributors and external advisors.The listed affiliations are Qwen Team and Huazhong University of Science and Technology.
  • A Reward Definitions for Reinforcement Learning: The displacement error uses positional channels over future waypoints, excludes heading, and is measured in meters.The recorded future trajectory is denoted τgt, and the error over the first n waypoints is abbreviated ADEn.
  • A Reward Definitions for Reinforcement Learning: For PAI-AV, horizon weights and scales make the first-second error carry the largest relative influence.The resulting coefficients from 1 s to 4 s are 2.5, 1, 0.5, and 0.25 against 1 for the full 5 s horizon.
  • A Reward Definitions for Reinforcement Learning: The reinforcement-learning reward design combines displacement terms with task-level scores for NAVSIM and WOD-E2E, while PAI-AV uses displacement terms alone.A shared displacement term supplies a comparable learning signal across sources, with multi-horizon refinement applied to PAI-AV.
  • Qualitative Capability Demonstrations: Qualitative examples cover camera-based 3D grounding, traffic-signal detection, roadwork detection, road-element recognition, and reasoning-based motion planning.In one urban turning scene, the model localizes three vehicles and reports their requested 3D-box parameters.

B.2 Traffic-Signal Detection

The traffic-signal detection task requires identifying each signal’s location, type, orientation, and relevance to the current intersection. The model response provides multiple structured detections in the requested format.

  • Traffic-Signal Detection: The task asks the model to detect all traffic-signal-related objects and return 2D boxes, signal labels, orientations, and intersection relevance.Outputs must use integer pixel coordinates and one JSON array without extra explanatory text.
  • Traffic-Signal Detection: Qwen-Drive-1.0-SFT localizes twelve signals and returns the requested attributes for each instance.The response includes structured dictionaries with bounding boxes, labels, directions, and relatedness fields.

B.3 Roadwork Detection

Qwen-Drive-1.0-SFT detects roadwork objects across an expanded temporary-traffic-control vocabulary, identifying thirteen instances in the scene.

  • B.3 Roadwork Detection: Qwen-Drive-1.0-SFT identifies thirteen roadwork instances, including a work vehicle, tubular markers, and cones.The response lists one work vehicle, four tubular markers, and eight cones with bounding-box coordinates.
  • B.3 Roadwork Detection: The roadwork category set includes temporary traffic-control devices, construction objects, and related road users.Examples include cones, fences, drums, barricades, work vehicles, tubular markers, workers, and temporary traffic-control signs.

B.4 Road Element Recognition

Qwen-Drive-1.0-SFT correctly interprets the highlighted single dashed yellow line in a Chinese expressway scene, whereas all eight comparison methods answer incorrectly.

  • B.4 Road Element Recognition: Qwen-Drive-1.0-SFT selects the correct interpretation of the highlighted single dashed yellow line from six candidate answers.The prompt asks for the meaning of the road marking in a Chinese expressway image.
  • B.4 Road Element Recognition: All eight comparison methods answer the road-element recognition question incorrectly.The comparison concerns the same six-choice interpretation task.
  • B.4 Road Element Recognition: The model’s reasoning associates the marking with a dedicated lane based on the visible road text and lane-divider context.The response translates “专用车道” as “Dedicated Lane” and evaluates the listed alternatives.

B.5 Reasoning-Based Motion Planning

Qwen-Drive-1.0-SFT combines multiview temporal observations, historical ego motion, and navigation commands to produce a rationale followed by future trajectory waypoints.

  • B.5 Reasoning-Based Motion Planning: The planning input includes camera views with four temporal frames, historical ego trajectories, and an active navigation command.The example uses frames at 0.5-second intervals and the command [GO STRAIGHT].
  • B.5 Reasoning-Based Motion Planning: The model outputs a one-sentence driving rationale followed by future waypoints serialized as text.In the example, the rationale instructs the ego vehicle to follow the lead vehicle through the intersection under a green light.
  • B.5 Reasoning-Based Motion Planning: The predicted trajectory closely tracks the ground-truth trajectory as the ego vehicle follows the lead vehicle through a signalized intersection.The example conditions the decision on the active navigation command and traffic-light context.
  • B.5 Reasoning-Based Motion Planning: The appendix provides complete prompts, ground-truth answers, and model responses for four cases summarized in Fig. 8.It states that all models receive the same images in the same order and that long reasoning chains are abridged.

C.1 (a) Temporal Understanding and Agent-State Estimation (LingoQA)

The temporal-understanding example asks for the final count of parked vehicles, but the reported responses disagree with the ground truth and with one another.

  • C.1 (a) Temporal Understanding and Agent-State Estimation (LingoQA): The benchmark ground truth reports zero parked vehicles, with “None” also accepted.A separate concise answer likewise states that no vehicles are parked on either side of the road.
  • C.1 (a) Temporal Understanding and Agent-State Estimation (LingoQA): Other responses report two or three parked vehicles, identifying vehicles that they claim are visible in the sequence.One response counts a red bus and white truck, while another counts three vehicles in the final frame.
  • C.1 (a) Temporal Understanding and Agent-State Estimation (LingoQA): An abridged reasoning trace ultimately changes its count to three after self-correction.The trace describes repeated checking before producing the final answer.

C.2 (b) Causal Reasoning for Planning (PAI-AV-CoC)

The evaluation examines whether the model can explain future ego driving decisions from temporal, multi-view inputs. Examples include stopping at a stop sign and continuing straight while avoiding construction barriers.

  • The task asks for a one-sentence explanation of the ego vehicle’s future driving decision.The input includes organized camera views and temporal frames, with historical trajectories expressed in the current ego coordinate system.
  • The stop-sign example requires stopping behind the lead vehicle at the intersection.The concise answer is “Stop for the stop sign at the intersection.”
  • The construction-barrier example requires continuing straight while maintaining the lane and navigating around barriers on the road’s right side.The associated brief reasoning describes steady-speed forward travel.

C.3 (c) Cross-View Spatial Distance Perception (Ego3D-Bench)

The cross-view distance task asks the model to estimate the separation between vehicles identified in different camera views. The reference distance is 22.93 m, while responses use visual spacing and approximate street geometry.

  • The task asks for the distance between a dark sedan in the back-right view and a beige sedan in the front view.The prompt requests a number-only final answer in boxed form.
  • 22.93 m is the ground-truth distance between the two cars.
  • The geometric reasoning separates transverse street width from longitudinal spacing along the street.It estimates a 6–8 m street width and at least 10–15 m longitudinal separation before combining them.
  • Another response estimates 20–25 meters and outputs 22, while acknowledging that explicit measurements are unavailable.The estimate combines perceived longitudinal spacing with approximate street dimensions.
  • An alternative estimate gives approximately 15 meters based on the vehicles’ relative positions across the views.The explanation places the dark sedan farther down the street from the beige sedan.

C.4 (d) Traffic-Road Recognition (VLADBench)

The traffic-road recognition task identifies the ego lane from a chronological image sequence and visible directional markings. The reference answer is left turn lane, although one response predicts straight lane.

  • The task asks which lane the ego occupies in the final frame, choosing among straight, turning, and U-turn lane categories.The sequence contains five frames from China in chronological order.
  • left turn lane is the ground-truth label for the ego lane.
  • The supporting reasoning tracks a persistent left-turn arrow in the ego vehicle’s lane from Frames 1–2 through Frame 5.It also uses the red traffic light and rejects combined-turn and U-turn alternatives.
  • A separate response instead predicts straight lane based on consistent lane position and the absence of an apparent turn or lane change.
Loading 2609.00111v1…