PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving

Xin Zhou, Zongchuang Zhao, Zhibo Yang, Mingsheng Li, Humen Zhong, Shuai Bai, Du Chu, Ruizhe Chen, Zhaohai Li, Jun Tang, Qiuyue Wang, Mingkun Yang, Jiazhao Zhang, Dayiheng Liu, Dingkang Liang, Xiang Bai

arXiv:2609.00111v1cs.CV

TL;DR

Textual VQA alone does not require explicit 3D predictions, and extensive driving adaptation can risk forgetting pretrained general knowledge. Qwen-Drive-1.0 adds 3D perception and trajectory planning to a pretrained VLM, achieving competitive results across tasks while largely preserving general vision-language capability.

  • Problem

    Textual VQA alone does not require explicit 3D predictions, while extensive driving adaptation can risk forgetting pretrained general knowledge.

    Source: §1 · p. 2

  • Method

    Qwen-Drive-1.0 retains the pretrained VLM architecture and adds a BEV perception head and Planning Expert, using staged driving and general-purpose vision-language supervision.

    Source: §2.1 · p. 3, §4 · p. 24

  • Results

    Qwen-Drive-1.0 achieves highly competitive results across 3D perception, driving scene understanding, and motion planning while largely preserving general vision-language capability; its NAVSIM Predictive Driver Model Score is 90.7.

    Source: §1 · p. 2, §4 · p. 24

  • Takeaways & Limitations

    The authors present Qwen-Drive-1.0 as an initial step toward a vision-language foundation model for autonomous driving.

    Source: §4 · p. 24

    Planning reasoning can misidentify a scene’s governing cause and time scale, and generated trajectories may not follow the textual rationale.

    Source: §5 · pp. 24–25

Abstract

from arXiv · show

We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird's-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.

1 Introduction

Qwen-Drive-1.0 combines 3D perception, driving VQA, and motion planning within a pretrained VLM, aiming to add driving competence while retaining general visual-language capability. Its external BEV head exposes explicit 3D predictions, and its staged training and planning components support the unified model.

Motivation

  • The design addresses the limits of language-only driving supervision for explicit 3D evaluation and the risk of forgetting pretrained knowledge needed for rare or unseen scenarios.Retaining general capability also supports cockpit-driving integration, where one model must serve both domains within a shared compute budget.Source: §1 · p. 2

Results

  • 43.95 mAP and 60.99 map mIoU on nuScenes, 43.45 mAP and 71.27 map mIoU on OpenScene, NAVSIM Predictive Driver Model Score 90.7, and WOD-E2E test Rater Feedback Score 7.91.The model significantly surpasses Qwen3.5-4B on driving scene understanding while preserving general capability, and shows promising potential for closed-loop driving in AlpaSim.Source: §1 · p. 2

Model contribution

  • Qwen-Drive-1.0 unifies 3D perception, driving VQA, and motion planning without changing the pretrained VLM architecture.It uses Qwen3.5-4B as the shared VLM, with a BEV perception head and Planning Expert as external modules.Source: §1 · pp. 2–3

3D perception

  • The external BEV head jointly predicts 3D objects, semantic occupancy, and BEV maps, providing explicit, inspectable 3D outputs from shared representations.The head serves as a probe of 3D information accessible from the shared representations.Source: §1 · pp. 2–3

Training and data

  • A staged training and data recipe unifies cross-dataset labels, rewrites and filters VQA responses for consistency, and combines driving data with general-purpose vision-language supervision.The recipe supports domain adaptation while mitigating catastrophic forgetting.Source: §1 · pp. 2–3

Motion planning

  • The Planning Expert uses flow matching on pretrained VLM representations to generate future ego trajectories, with unified waypoint annotations enabling joint training across public driving datasets.The paper reports highly competitive performance across open-loop, pseudo-closed-loop, and closed-loop evaluations.Source: §1 · pp. 2–3

2 Method

This section describes Qwen-Drive’s shared vision-language pathway with external BEV perception and trajectory-planning modules, along with its staged training recipe. It also covers the preparation of perception, driving vision-language, and planning data.

2.1 Model Architecture and Objectives

  • The shared vision encoder and VLM process driving and general images, while external BEV and planning modules use their features without changing the VLM architecture.The BEV head supports 3D detection, semantic occupancy, and map segmentation; the Planning Expert predicts future ego motion.Source: §2.1 · p. 3
  • View and frame tags identify each image’s direction and timestep; QA inputs use frame-major serialization, whereas planning inputs use view-major ordering.View-major ordering keeps consecutive observations from each camera adjacent, and ordinary vocabulary tokens encode both tag types without architectural changes.Source: §2.1 · pp. 3–4
  • The BEV head combines low-level vision features with contextual VLM features, lifting image features into a geometric volume before aggregating them into an ego-frame BEV representation.Detection, occupancy, and map-segmentation branches decode this representation; occupancy also fuses the height-preserving volume to recover vertical structure.Source: §2.1 · p. 4
  • The Planning Expert uses cached VLM attention keys and values to condition trajectory tokens and predicts 50 ego waypoints spanning 5 seconds at 10 Hz.A 32-layer diffusion transformer is trained by flow matching to estimate clean trajectories, with temporal regularization to discourage waypoint jitter and abrupt acceleration changes.Source: §2.1 · pp. 5–6

2.2 Training Recipe

  • Training proceeds from BEV-head-only pretraining to joint perception and VQA adaptation, then Planning Expert pretraining with fixed vision-language representations and optional textual reasoning.The joint stage mixes driving perception and general vision-language samples; the final planning stage trains on future trajectories without a text-generation objective.Source: §2.2 · p. 6
  • Reward-based refinement addresses limitations of single-recorded-future imitation by optimizing task-level driving criteria while keeping the vision encoder and VLM fixed.Rewards measure collision avoidance, drivable-area compliance, progress, and human preference; NAVSIM uses PDMS and WOD-E2E uses its Rater Feedback Score, with a shared displacement term.Source: §2.2 · pp. 7–8
  • Reward optimization makes the Planning Expert stochastic only during the final three of 10 Euler steps, using low-frequency cosine-mode perturbations for coherent trajectory variation.Rollout groups share initial trajectory noise, sample independent perturbations, and use group-relative advantages without a learned value function; the score correction is approximate for the restricted subspace.Source: §2.2 · pp. 7–8

2.3 Data Recipe

  • The data recipe organizes perception, vision-language, and planning sources into task groups and aligns heterogeneous task definitions before mixing sources.Perception uses nuScenes and OpenScene, whose differing taxonomies are aligned at the coarsest compatible granularity while source-specific classes remain restricted to their source unless auxiliary labels are available.Source: §2.3 · p. 8, §2.3.1 · p. 8

2.3.1 Perception Data

  • The perception pipeline unifies detection, occupancy, and map supervision across datasets while preserving native occupancy grids and aligning predicted features through differentiable sampling.Occupancy labels use a shared ten-class space with offline completion for missing driveable and generic_object labels; maps are rasterized online into six classes. The source occupancy annotations retain noise, including floating voxels; predicted features are aligned to each dataset's native grid, and maps use a shared ego-centered patch.Source: §2.3.1 · pp. 8–10

2.3.2 Vision-Language Data

  • Public driving vision-language data provide broad scene and question coverage, while constructed examples target causal planning reasoning and perception grounding.The 24 public datasets are rewritten into a common conversational schema and consistency-filtered, retaining 3.09M of 5.53M samples (55.9%). Constructed data include audited Chain-of-Causation planning traces, camera-ordering tasks, and 30K in-house perception examples.Source: §2.3.2 · pp. 10–11
  • Before repetition, the 1.54M-example Stage 2 mixture is 9.7% perception, 26.0% general-purpose vision-language, and 64.3% driving vision-language supervision.Public driving data are sampled at approximately 20% per source, stratified by task and question type; group-specific repetition raises the effective mixture to 12.7% perception, 31.0% general-purpose vision-language, and 56.3% driving vision-language supervision.Source: §2.3.2 · p. 11

2.3.3 Planning Data

  • Stage 3 combines four planning-data sources into approximately 2.83M samples, with accepted planning-reasoning traces conditioning 24.2% and r = ∅ for the remaining 75.8%.Trajectories are expressed in the current ego frame as 50 waypoints; sources are resampled to 10 Hz, with WOD-E2E spline-derived motion and acceleration and heading consistency filters.Source: §2.3.3 · p. 11
  • Planning examples use front, front-left, and front-right images at four timesteps, together with historical trajectory, current ego state, navigation instruction, and optional reasoning traces.The current images are resized to 720p and historical images to 320p, retaining more detail for the current observation while using fewer tokens for motion history.Source: §2.3.3 · pp. 11–12

3 Experiments

The experiments evaluate Qwen-Drive-1.0 across 3D perception, driving VQA, general vision-language understanding, and motion planning, including closed-loop behavior, ablations, and qualitative examples. Results describe strong driving performance with broad vision-language capability preserved, alongside perception limitations and a safer but more conservative shift in closed-loop driving.

3.1 3D Perception

  • 2.01 mAP over BEVFormerV2∗, 3.05 NDS over BEVFormerV2, and 3.37 map mIoU over PETRv2 mark Qwen-Drive-1.0-SFT's nuScenes detection and map-segmentation leads.Cross-dataset occupancy remains challenging: mixed-source training raises OpenScene NDS to 41.86, but source-specific voxel-label artifacts limit joint adaptation; Stage 2 raises nuScenes mAP and map mIoU by 10.46 and 9.84 points, respectively.Source: §3.1 · pp. 12–13, Table 1 · p. 13

3.2.1 Driving Visual Question Answering

  • 69.43 is Qwen-Drive-1.0-SFT's driving QA average, up from 63.52 for Qwen3.5-4B, and its causal-reasoning average reaches 58.30 versus 22.05 for Gemma4-12B.The evaluation uses public driving QA benchmarks and adds PAI-AV-CoC, judged on causal key objects and decisions, plus an expert-annotated Chinese urban driving-decision benchmark.Source: §3.2.1 · pp. 14–15, Table 2 · p. 15

3.2.2 General Vision-Language Understanding

  • 66.41 versus 67.40 on knowledge, reasoning, and recognition, and 53.96 versus 52.99 on spatial understanding and grounding, show broad capability retention after driving adaptation.Qwen-Drive-1.0-SFT matches or exceeds Cosmos-Reason2-32B on 10 of 15 settings, with a 5.48-point higher average; the retained abilities can support cockpit tasks alongside driving.Source: §3.2.2 · pp. 17–18, Table 3 · p. 17

3.3 Motion Planning

  • 90.7 NAVSIM PDMS and 7.91 test-split WOD-E2E RFS are the highest among compared methods for Qwen-Drive-1.0-RL.The held-out WOD-E2E RFS exceeds reinforced MindVLA-U1 by 0.04 points; NAVSIM uses a pseudo-closed-loop protocol with non-reactive surrounding agents.Source: §3.3 · pp. 18–20, Table 4 · p. 18, Table 6 · p. 19
  • 7.78 to 7.91 is the WOD-E2E test RFS gain with reinforcement learning, while reasoning SFT records 0.42 m 3 s average ADE on leakage-free PAI-AV.On PAI-AV, its minADE is 0.39 m versus 0.17 m for Alpamayo-1.5; reinforcement learning raises PAI-AV errors by only 3 to 5 cm.Source: Table 4 · p. 18, §3.3 · pp. 19–20, Table 5 · p. 19
  • 0.4 points from reasoning and a further 2.5 points from reinforcement learning raise NAVSIM PDMS to 90.7 for Qwen-Drive-1.0-RL.Selecting the highest-PDMS candidate from six samples raises RL PDMS to 91.4, though the non-reactive protocol cannot reveal interactive error accumulation.Source: Table 6 · p. 19, §3.3 · p. 20
  • Qwen-Drive-1.0-RL matches Alpamayo-1.5’s 11.0% at-fault close encounter rate, while off-road rate falls from 24.0% to 12.0% and at-fault score rises from 0.27 to 0.37, with progress falling from 54.0% to 48.0%.Its all-event and at-fault AlpaSim scores are 0.16 and 0.37, below Alpamayo-1.5’s 0.23 and 0.45. The authors suggest its sparser, broader visual history may limit responsiveness over AlpaSim’s short replanning horizon.Source: §3.3 · p. 20
  • Qwen-Drive-1.0-SFT grounds open-loop plans in scene evidence, while AlpaSim rollouts update reasoning and trajectories with traffic changes and remain consistent with navigation instructions and road geometry.Examples include yielding to a crossing animal, following a right-turn-only lane, stopping at a red light, and adjusting as another vehicle overtakes; on PAI-AV, a prediction remains close to the recorded future.Source: Figure 9 · p. 21, §3.3 · p. 21
  • Qwen-Drive-1.0-RL combines 11.0% at-fault close encounters and 12.0% off-road rate with 48.0% progress, while SimWAM records 22.0%, 19.0%, and 62.0%, respectively.DriveWAM’s at-fault score of 0.53 comes with only 35.0% progress, and its all-event close encounter rate and score are 56.0% and 0.10; these comparisons underscore that scores should be considered alongside progress and safety events.Source: §3.3 · pp. 20–21
  • In the same NAVSIM left-turn scene, Qwen-Drive-1.0-RL retains the high-level maneuver while reducing a small lateral deviation from the recorded future, consistent with the reported PDMS improvement.The passage attributes the finer correction to better alignment with NAVSIM scoring criteria.Source: §3.3 · p. 21, Figure 10 · p. 22

3.4 Ablation Study and Analysis

  • Stage 2 vision-language training improves Driving QA Avg. by 6.55 points and CoC reasoning, while adding 3D supervision keeps all three vision-language aggregates within one point of the vision-language-only variant.The complete mixture adds an inspectable 3D perception capability while largely preserving vision-language performance and yields the highest Stage 3 RFS by a small margin; this does not establish 3D supervision as the cause of that planning gain.Source: §3.4 · p. 22, Table 8 · p. 22
  • With NAVSIM-only training, shared ADE raises PDMS from 90.4 to 90.8; on WOD-E2E, it lowers RFS from 8.68 to 8.45 while reducing 5 s ADE from 2.24 m to 1.27 m.Joint NAVSIM, WOD-E2E, and PAI-AV training produces NAVSIM scores of 90.6 without shared ADE and 90.7 with it; the final recipe uses all three sources with shared ADE for each.Source: §3.4 · pp. 22–23, Figure 11 · p. 23
  • As PAI-AV planning data grows from 0.17M to 1.38M samples, standard-split 5 s Avg. ADE falls from 1.34 m to 1.05 m and Avg. FDE from 4.18 m to 3.24 m.Both metrics also decrease monotonically on the more challenging leakage-free subset, and performance continues improving at 1.38M samples with no clear sign of saturation.Source: Figure 12 · p. 23, §3.4 · p. 23
  • On unseen eight-camera WOD-E2E and six-camera PAI-AV rigs, Qwen-Drive-1.0-SFT produces visually plausible projected boxes and coherent occupancy and map outputs without dataset-specific adaptation.Because the datasets lack unified perception ground truth, these examples do not establish 3D accuracy; high-quality target-platform annotations are needed for assessment and improvement.Source: §3.4 · pp. 23–24, Figure 13 · p. 24

4 Conclusion

Qwen-Drive-1.0 extends a pretrained vision-language model with inspectable 3D perception and trajectory planning for autonomous driving. It achieves competitive driving results while largely preserving general vision-language capability.

4 Conclusion

  • The framework retains the pretrained VLM architecture and adds a BEV head for inspectable 3D detection, occupancy, and map predictions, plus a Planning Expert that generates future ego trajectories.The Planning Expert uses shared representations and flow matching, and supports joint training across multiple public driving datasets.Source: §4 · p. 24
  • A staged training and data recipe combines driving-specific supervision with general-purpose vision-language data.Source: §4 · p. 24
  • Qwen-Drive-1.0 achieves highly competitive 3D perception, driving scene understanding, and motion-planning results while largely preserving general vision-language capability.Performance across open-loop, pseudo-closed-loop, and closed-loop evaluations demonstrates the value of combining unified trajectory supervision with reward-based optimization.Source: §4 · p. 24

5 Limitations and Future Work · A Reward Definitions for Reinforcement Learning

Qwen-Drive-1.0’s planning has limitations in causal reasoning, rationale–trajectory consistency, and cross-task transfer. The reinforcement-learning appendix defines source-specific rewards that share displacement terms, with NAVSIM and WOD-E2E also using task-level scores and PAI-AV using multi-horizon displacement terms.

5 Limitations and Future Work

  • Planning reasoning can misidentify which coexisting cause governs at which timescale, and generated trajectories may not match textual rationales; some reasoning gains may reflect added self-generated-trace context.The proposed directions are multi-timescale causal modeling and explicit consistency supervision between rationale and trajectory.Source: §5 · pp. 24–25
  • Different input formats, temporal contexts, and image resolutions across the three tasks may limit representation transfer; better alignment and joint optimization could improve cross-task reinforcement.Source: §5 · p. 25

A Reward Definitions for Reinforcement Learning

  • NAVSIM’s reward combines PDMS with full-horizon displacement error.Its weights are wpdms = 1 and wade = 2, with δ = 2 and κ = 10; rollouts are subsampled to the scorer’s eight poses at 2 Hz over a 4 s scoring horizon.Source: A Reward Definitions for Reinforcement… · p. 33
  • WOD-E2E’s reward combines the Rater Feedback Score with the same displacement term.It uses wrfs = 1 and wade = 2, with δ = 2 and κ = 1; rollouts are resampled to 20 positions at 4 Hz over 5 s, and the smaller κ keeps displacement comparable in scale to the task reward.Source: A Reward Definitions for Reinforcement… · p. 33
  • PAI-AV uses displacement terms alone, weighting shorter horizons alongside the full 5 s horizon because near-term accuracy governs immediate control.The 1 s to 4 s horizon coefficients are 2.5, 1, 0.5, and 0.25, respectively, compared with 1 for the full horizon, so first-second error receives the largest weight.Source: A Reward Definitions for Reinforcement… · p. 33
  • All reward sources include displacement terms, which provide a comparable learning signal; NAVSIM and WOD-E2E also use task scores, while multi-horizon refinement applies only to PAI-AV.The displacement error uses positional channels only, excludes heading, and is measured in meters; reward offsets affect logged magnitude, while weight-to-scale ratios determine relative influence.Source: A Reward Definitions for Reinforcement… · p. 33

B Additional Visualizations · B.1 Camera-Based 3D Grounding

This appendix presents Qwen-Drive-1.0-SFT visual examples spanning perception, traffic-rule interpretation, and motion planning. Its camera-based 3D grounding example shows three vehicles localized in an urban turning scene, with box parameters returned in a requested format.

B Additional Visualizations

  • The visualizations cover camera-based 3D grounding, traffic-signal and roadwork detection, road-element recognition, and reasoning-based motion planning.The sequence moves from structured visual prediction to traffic-rule interpretation and planning.Source: B Additional Visualizations · p. 33
  • Where annotations are available, model predictions and ground truth are overlaid on the input image in orange and blue, respectively.The appendix includes prompts and raw model responses.Source: B Additional Visualizations · p. 33

B.1 Camera-Based 3D Grounding

  • Camera-based 3D grounding requires recovering metric 3D boxes from a single image.In the urban turning scene, Qwen-Drive-1.0-SFT localizes three vehicles and reports their box parameters in the requested format.Source: §B.1 · p. 34
  • The grounding prompt supplies camera intrinsics and requests a 3D box with center, size, orientation, and category fields.It specifies fx=1862.42, fy=1861.99, cx=1917.51, and cy=1078.37.Source: §B.1 · p. 34
  • The example includes a response from Qwen-Drive-1.0-SFT to the 3D-box grounding prompt.The passage identifies the response but provides no box values in the supplied text.Source: §B.1 · p. 34

B.2 Traffic-Signal Detection

Qwen-Drive-1.0-SFT detects traffic signals at intersections and reports each signal’s type, orientation, and relevance to the current intersection. In the example, it localizes twelve signals and marks their attributes.

B.2 Traffic-Signal Detection

  • The example response localizes twelve signals, reporting each instance’s type, orientation, and intersection relevance.The twelve detections are labeled car or black, front-facing, and related to the current intersection.Source: §B.2 · pp. 34–35
  • The annotation task requires tight integer-coordinate 2D boxes and one JSON array of dictionaries, with an empty array when no signals are present.Each dictionary includes bbox_2d, label, direction, and is_related; objects should be output once, without extra explanatory text.Source: §B.2 · p. 34
  • Signal labels distinguish vehicle, pedestrian, unlit, countdown, LED, and other devices; direction is front, back, or side, while relevance distinguishes current-intersection signals from unrelated ones.Related signals control traffic through the current intersection; unrelated signals may serve distant intersections, opposing traffic, or side roads.Source: §B.2 · p. 34

B.3 Roadwork Detection

The roadwork example shows Qwen-Drive-1.0-SFT detecting temporary traffic-control and construction-related objects. Its prompt specifies a broad category set and requests bounding boxes in JSON format.

B.3 Roadwork Detection

  • Qwen-Drive-1.0-SFT identifies 13 instances, including a work vehicle, tubular markers, and cones.The listed response contains one work vehicle, four tubular markers, and eight cones.Source: §B.3 · pp. 35–36
  • The example broadens the object vocabulary to temporary traffic-control devices and construction-related objects.Source: §B.3 · p. 35
  • The prompt covers 18 categories spanning police, roadwork objects, and temporary traffic-control devices.It requests each instance’s bounding-box coordinates and label in JSON format.Source: §B.3 · p. 35

B.4 Road Element Recognition

Qwen-Drive-1.0-SFT correctly interprets a Chinese expressway’s single dashed yellow line as indicating a dedicated lane, while all eight comparison methods answer incorrectly.

B.4 Road Element Recognition

  • Qwen-Drive-1.0-SFT selects the correct interpretation of the single dashed yellow line, while all eight comparison methods answer incorrectly.Its response connects the nearby “专用车道” marking, meaning “dedicated lane,” to the option “Located on both sides of the road, indicating a dedicated lane.”Source: §B.4 · p. 36

B.5 Reasoning-Based Motion Planning · C Detailed Driving VQA Cases

Qwen-Drive-1.0-SFT produces a one-sentence driving rationale followed by future waypoints, with the illustrated trajectory closely tracking ground truth through an intersection. The appendix documents the prompts, answers, and responses for four detailed VQA cases.

B.5 Reasoning-Based Motion Planning

  • The predicted trajectory closely tracks ground truth as the ego vehicle follows a lead vehicle through a signalized intersection.The response rationale says to follow the lead vehicle through the intersection under a green light.Source: §B.5 · p. 37
  • Given multiview temporal observations, historical ego trajectory, and the active navigation command, the model outputs a one-sentence rationale followed by future waypoints.Source: §B.5 · p. 37
  • Each camera view contains four temporal frames captured at 0.5-second intervals, with frame 3 representing the current time, t=0.Source: §B.5 · p. 37
  • Historical ego trajectories use (x, y, heading) in current-frame ego coordinates, where positive x is forward, positive y is left, and positive heading indicates a left turn.Source: §B.5 · p. 37
  • The GO STRAIGHT prompt requests one-sentence reasoning and a JSON trajectory with eight (x, y, heading) points at 2 Hz over 4 seconds, rounded to two decimals.Source: §B.5 · p. 37

C Detailed Driving VQA Cases

  • The appendix presents complete prompts, ground-truth answers, and model responses for four Fig. 8 cases, using identical image order and typesetting markdown emphasis while abridging long reasoning chains.Abridged spans are marked with [...] and described as restating surrounding reasoning without adding evidence.Source: C Detailed Driving VQA Cases · p. 38

C.1 (a) Temporal Understanding and Agent-State Estimation (LingoQA)

The LingoQA example asks for the final outcome of a video sequence: the ground truth is zero parked vehicles. Responses include both the correct answer and incorrect counts of two or three.

C.1 (a) Temporal Understanding and Agent-State Estimation (LingoQA)

  • The ground truth is zero parked vehicles, and one response correctly says there are none on either side of the road.The benchmark also accepts “None.”Source: §C.1 · p. 38
  • One response incorrectly counts two parked vehicles, naming a red bus on the left and a white truck on the right.It concludes that two vehicles are visible in the sequence.Source: §C.1 · p. 38
  • Another response incorrectly counts three parked vehicles on the right-hand side in the final frame.It identifies a white van, a car, and another vehicle.Source: §C.1 · p. 38

C.2 (b) Causal Reasoning for Planning (PAI-AV-CoC)

The example frames planning as a request for a brief causal explanation of the ego vehicle’s future driving decision, based on camera-view images and historical ego trajectories. The stated ground truth is to stop behind the lead vehicle at an intersection stop sign.

C.2 (b) Causal Reasoning for Planning (PAI-AV-CoC)

  • The prompt requests one sentence briefly reasoning about the ego’s future driving decision only.It supplies camera-view images with four frames at 0.5-second intervals and historical trajectories in the current ego coordinate system.Source: §C.2 · p. 38
  • The ground truth is to stop behind the lead vehicle at the stop sign at the intersection.The associated response says, “Stop for the stop sign at the intersection.”Source: §C.2 · p. 38
  • Two additional response passages describe continuing straight, one specifying steady speed and the other navigating around construction barriers.These descriptions differ from the stated ground truth to stop at the stop sign.Source: §C.2 · pp. 38–39

C.3 (c) Cross-View Spatial Distance Perception (Ego3D-Bench)

Ego3D-Bench tests cross-view spatial distance perception by asking for the distance between two parked cars shown across seven ego-mounted camera views. The supplied ground truth is 22.93 m, while example answers estimate either about 22 m or 15 m.

C.3 (c) Cross-View Spatial Distance Perception (Ego3D-Bench)

  • The task asks for the distance between a dark sedan in the back-right view and a beige sedan in the front view across seven ego-car camera views.The requested answer is in meters.Source: §C.3 · p. 39
  • The ground-truth distance between the cars is 22.93 m.Source: §C.3 · p. 39
  • Example answers give differing estimates: one concludes approximately 22 m, while another estimates 15 m.The latter answer is also described as approximately 15 m in a separate response.Source: §C.3 · pp. 39–40

C.4 (d) Traffic-Road Recognition (VLADBench)

This VLADBench example asks for the ego lane in the final frame of a five-frame China driving sequence. The provided ground truth is “left turn lane,” matching the answer supported by the accompanying reasoning.

C.4 (d) Traffic-Road Recognition (VLADBench)

  • The task asks which lane the ego vehicle occupies in the final frame of a five-frame China driving sequence.The prompt offers seven lane-type options and requests a boxed final answer.Source: §C.4 · p. 40
  • The provided ground truth labels the ego lane as “left turn lane.”Source: §C.4 · p. 40
  • The accompanying reasoning selects “left turn lane,” citing the left-turn arrow and the ego vehicle remaining in that lane across the sequence.It distinguishes the adjacent lane’s right-turn arrow and rejects straight, combined-turn, and U-turn options.Source: §C.4 · p. 40

Related papers

Loading 2609.00111v1…