PaperlayerView exact revision on arXiv

Source-linked AI summary

Roadside-Cooperative Autonomous Driving: From Data Platform to Vision-Language End-to-End Reasoning

Yitao Xu, Tong Wu, Yiyan Wu, Guoji Xu, Yanbo Jiang, Jiahao Wang, Zehong Ke, Junkai Jiang, Fang Zhang, Jianqiang Wang

arXiv:2608.21032v1cs.RO

TL;DR

Existing V2X systems and benchmarks provide limited language-grounded supervision and closed-loop evaluation for cooperative end-to-end driving. This paper introduces V2XBench, Chat-V2XBench, and AURORA, whose cross-view VLM framework achieves state-of-the-art cooperative perception and closed-loop driving, including a 76.02 Driving Score.

  • Problem

    Existing V2X systems rely largely on modular pipelines and benchmarks offer limited closed-loop evaluation and language-grounded supervision for cooperative end-to-end reasoning.

  • Method

    The paper introduces V2XBench and Chat-V2XBench, then uses AURORA’s CQAF module and LoRA-adapted VLM to align views, reason cooperatively, and generate trajectories.

  • Results

    AURORA achieves state-of-the-art cooperative perception and closed-loop driving, attaining a Driving Score of 76.02 while maintaining low roadside communication bandwidth.

  • Takeaways & Limitations

    The work supports a VLM-driven V2X paradigm for cooperative driving under occlusion-heavy scenarios with low query-level communication bandwidth.

Abstract

from arXiv · show

Vehicle-to-Everything (V2X) cooperation enables beyond-line-of-sight perception, mitigating occlusions in single-vehicle sensing. However, existing V2X benchmarks provide limited support for closed-loop evaluation and language-grounded supervision, hindering the development of vision-language models (VLMs) for end-to-end cooperative driving. To address these limitations, we introduce V2XBench, a simulation platform featuring synchronized ego--roadside sensing and closed-loop evaluation, together with Chat-V2XBench, a progressively structured VQA dataset for cooperative reasoning. Building upon this benchmark infrastructure, we propose AURORA, an end-to-end cooperative driving framework. Equipped with a dual-view perception architecture, AURORA mitigates spatial and semantic discrepancies across ego and roadside viewpoints through a query-level Cross-View Query Alignment and Fusion (CQAF) module. Leveraging the resulting unified tokens, a LoRA-adapted VLM bridges semantic reasoning and generative trajectory planning. Extensive closed-loop evaluations on V2XBench demonstrate that AURORA achieves state-of-the-art performance in heavily occluded scenarios, with a Route Completion rate of 98.21% and a Driving Score of 76.02, while requiring low roadside communication bandwidth. Ultimately, this work pioneers an extensible V2X--VLM paradigm, paving the way for next-generation cooperative autonomous driving.

1 Introduction

The introduction motivates cooperative V2X driving as a remedy for single-agent occlusions and limitations of modular, ego-only systems, then presents V2XBench, Chat-V2XBench, and AURORA as an integrated benchmark, reasoning, and end-to-end driving solution.

  • Motivation: V2X cooperation addresses limited visibility and severe occlusions in single-agent driving, especially at complex intersections (Prakash, Chitta, and Geiger 2021; Shao et al. 2023).The introduction positions complementary exchanged observations as the motivation for cooperative driving.
  • Limitations: Existing V2X research emphasizes module-level perception within modular pipelines, leaving perception and planning isolated and limiting semantic understanding and logical reasoning (Li et al. 2022; Lu et al. 2024; Hu et al. 2023).The introduction identifies this separation as a central limitation of current V2X systems (Cui et al. 2022).
  • Motivation: Ego-only VLMs offer reasoning and trajectory-planning capabilities but become unreliable under severe occlusions, motivating their extension to cooperative perspectives (Sima et al. 2024; Chen et al. 2025; Ma et al. 2025; Cui et al. 2024; Yang et al. 2024).The stated motivation is to combine VLM reasoning with complementary roadside observations.
  • Benchmark and dataset: V2XBench provides synchronized ego–roadside data generation, offline training, online closed-loop evaluation, and continuous RSU coverage, while Chat-V2XBench adds progressively structured VQA supervision for cooperative reasoning.Table 1 identifies V2XBench as the only compared V2X benchmark supporting closed-loop end-to-end simulation with VQA capabilities.
  • AURORA: AURORA is an end-to-end cooperative driving framework whose CQAF module aligns and fuses ego and roadside queries before connecting unified representations to a LoRA-adapted VLM and generative trajectory planner.CQAF transforms roadside queries into the ego coordinate frame, associates spatially corresponding queries, and fuses complementary semantic tokens through a dedicated waypoint token.

2 Related Work

Prior V2X research has progressed from synchronized real-world datasets and scalable simulation platforms toward closed-loop cooperative driving, but largely lacks language-grounded VLM evaluation. End-to-end driving and feature-level fusion address planning and multi-agent aggregation, while CQAF explicitly structures cross-view information for VLM-guided trajectory planning.

  • V2X datasets and simulation: V2X datasets and simulators evolved from synchronized cooperative perception to scalable and closed-loop driving benchmarks, yet generally lack language-grounded annotations and VLM evaluation tasks.DAIR-V2X and V2V4Real provide real-world synchronized observations; OPV2V, V2X-Sim (Li et al. 2022), and V2Xverse support simulation and closed-loop evaluation.
  • End-to-end autonomous driving: End-to-end autonomous driving reduces modular error propagation, but single-agent methods such as LAV (Chen and Krähenbühl 2022) and TCP remain constrained by onboard visibility.The related work frames VLM and LLM integration as a concurrent direction in autonomous driving.
  • Cooperative feature fusion: Intermediate feature-level fusion balances late fusion’s limited information exchange against early fusion’s communication cost and underpins mainstream cooperative aggregation.F-Cooper (Chen et al. 2019), V2X-ViT, and Where2comm (Hu et al. 2022) aggregate features through spatial attention or point-pillar representations.
  • Cross-view VLM reasoning: CQAF addresses a gap in prior cooperative methods by aligning cross-view spatial queries geometrically and injecting them into semantic tokens for unified VLM-guided trajectory planning.Earlier methods, including Coopernaut (Cui et al. 2022), UniV2X, and UniMM-V2X (Song et al. 2026), connect cooperative representations with perception and planning but do not explicitly structure them as a VLM reasoning interface.

3 V2XBench: A Closed-Loop Platform for End-to-End Cooperative Autonomous Driving

V2XBench is a CARLA-based closed-loop platform integrating synchronized ego–RSU sensing, progressive VQA supervision, automated multimodal data generation, and online cooperative-driving evaluation. It targets severe occlusions and cross-view reasoning through diverse safety-critical scenarios, large-scale synchronized data, and Chat-V2XBench’s four-level QA hierarchy.

  • Scenario Design: V2XBench emphasizes 38 safety-critical scenarios with complex multi-agent interactions, severe intersection occlusions, heavy occluders, and unpredictable pedestrian or cyclist jaywalking.Route-adaptive RSU deployment provides continuous infrastructure support along every driving route, while trucks and vans create realistic visual obstructions.
  • Data Generation: 500 scenario runs across 38 scenario types produce approximately 140K synchronized frames and over 2.5 million 3D bounding boxes.Data are collected with the privileged expert agent PDM-Lite from ego vehicles and dynamically selected nearest RSUs using multimodal sensing.
  • Closed-Loop Evaluation: Closed-loop evaluation requires real-time throttle, steering, and braking commands whose consequences compound within a reactive simulation, unlike open-loop trajectory comparison.This protocol assesses cooperative driving through actual vehicle interaction rather than only comparing predictions with recorded ground truth.
  • Chat-V2XBench: Chat-V2XBench contains approximately 88K QA pairs organized into scene understanding, critical perception, spatial alignment, and motion planning levels.The hierarchy teaches holistic scene description, cross-view object localization and safety reasoning, ego-centric coordinate inference from roadside views, and physically justified driving behavior.
  • Platform Overview: V2XBench unifies dual-link ego–RSU perception, Chat-V2XBench VQA, synchronized multi-agent data generation, and closed-loop evaluation in a CARLA-based simulation platform.The platform overview explicitly integrates route-adaptive roadside infrastructure, synchronized sensing, offline data generation, VQA construction, and online evaluation.

4 AURORA: Cooperative End-to-End Driving Framework

AURORA is a holistic V2X cooperative driving framework that encodes synchronized ego and roadside views, aligns their spatial queries through CQAF, and grounds fused representations in a VLM for trajectory planning. Its design combines bandwidth-efficient roadside query transmission, language-guided reasoning, waypoint-based planning, and five-stage end-to-end training.

  • Cross-View Query Alignment and Fusion: Roadside transmission sends spatial detection and map queries rather than dense semantic tokens, reducing bandwidth while supplementing ego representations with beyond-view information.Roadside queries are aligned into the ego frame, unmatched distant queries are retained, and roadside-only map structure is appended.
  • Cross-View Query Alignment and Fusion: CQAF separately aligns dynamic-agent queries, confidence-filters and appends roadside map queries, and refines ego semantic tokens through query-conditioned cross-attention.DET-CQAF uses cross-view extrinsic transformation, MAP-CQAF preserves topological continuity, and the resulting fused tokens feed the downstream VLM.
  • Framework Overview: AURORA integrates dual-view perception, CQAF cross-view fusion, VLM reasoning, and generative trajectory planning into a four-stage cooperative driving pipeline.Synchronized ego and roadside streams produce spatial queries and semantic tokens; CQAF fuses them before VLM-based planning.
  • VLM Reasoning and Planning: A LoRA-fine-tuned VLM uses fused tokens and a dedicated waypoint token to connect cooperative semantic reasoning with probabilistic VAE trajectory generation.The planner samples latent codes from Gaussian distributions and rolls out candidate trajectories with a GRU decoder.
  • Training Strategy: A five-stage curriculum progresses from independent ego and roadside perception pretraining to CQAF-VLM optimization, planner training, and full end-to-end refinement.The overall objective combines perception, cross-view fusion, language, trajectory, lane-boundary, collision-avoidance, and KL-divergence losses.

5 Experiments

Experiments on V2XBench show that AURORA achieves state-of-the-art cooperative perception and closed-loop driving performance. Ablations attribute these gains to CQAF fusion, probabilistic VAE planning, and Chat-V2XBench VQA supervision.

  • 3D Perception Evaluation: +0.155 Avg. mAP over CoDriving and +0.083 NDS over V2X-ViT establish AURORA as state-of-the-art in cooperative 3D perception.Despite being camera-based, AURORA surpasses heavily parameterized LiDAR-based methods in overall NDS, supporting CQAF’s high-dimensional cross-view alignment.
  • End-to-End Driving Evaluation: 76.02 Driving Score and 98.21% Route Completion make AURORA state-of-the-art in closed-loop cooperative driving, exceeding CoDriving by 19.27 DS points and 6.98% RC.Models are evaluated with closed-loop interactive metrics and open-loop Avg. L2 distance; UniV2X and UniMM-V2X use AURORA’s controller for fair comparison.
  • Effectiveness of Cross-View Fusion: 76.02 Driving Score versus 52.09 for ego-only AURORA shows that CQAF fusion substantially improves cooperative perception and downstream closed-loop planning.CQAF increases mAP by 0.070 and NDS by 0.071 when roadside sensing is added.
  • Planner Architecture: 76.02 DS, 98.21% RC, and 3.47% collision rate make the probabilistic VAE planner the best-performing decoder among evaluated generative planners.The VAE models multimodal future states through latent Gaussian sampling, while the Diffusion planner outperforms the deterministic MLP decoder.
  • Impact of VQA Supervision: 67.98 DS without VQA supervision versus 76.02 with it shows that language-grounded reasoning is critical for interactive driving, despite open-loop L2 improving from 1.60 m to 1.54 m.Removing the VQA loss reduces closed-loop Driving Score by over 8 points while slightly lowering open-loop trajectory error.

6 Conclusion

The paper introduces V2XBench and Chat-V2XBench to support closed-loop evaluation and language-grounded supervision, then proposes AURORA for end-to-end cooperative driving. AURORA aligns ego and roadside views for VLM-based reasoning and achieves state-of-the-art cooperative perception and closed-loop driving, surpassing LiDAR-based baselines in overall detection accuracy despite using only cameras.

  • 6 Conclusion: V2XBench and Chat-V2XBench address limited support for closed-loop evaluation and language-grounded supervision in cooperative autonomous driving.V2XBench is a simulation platform, while Chat-V2XBench is a VQA dataset introduced to support these capabilities.
  • 6 Conclusion: AURORA uses Cross-View Query Alignment and Fusion to bridge ego and roadside perspectives, connecting cooperative representations to VLM-based semantic reasoning and safe, executable trajectory generation.The framework builds on the introduced benchmark infrastructure to support end-to-end cooperative driving.
  • 6 Conclusion: AURORA achieves state-of-the-art performance in cooperative 3D perception and closed-loop driving, while surpassing LiDAR-based baselines in overall detection accuracy using only camera inputs.The evaluation establishes AURORA’s performance advantage across both perception and driving outcomes.

A V2XBench Dataset Details … Scenario Types.

V2XBench is a CARLA-based closed-loop dataset platform that synchronizes ego and route-adaptive roadside sensing across diverse towns and challenging driving scenarios. Its sensor suite and infrastructure deployment provide continuous beyond-line-of-sight coverage, with 38 safety-critical scenario types emphasizing occlusion-heavy V2X hazards.

  • A.1 Platform Construction: V2XBench is built on CARLA and CARLA Leaderboard 2.0, using offline closed-loop rollouts from PDM-Lite while recording synchronized ego and infrastructure observations.Scenarios span eight towns, including the large-scale Town12 and Town13 maps, which contribute most runs.
  • Sensor Configuration.: The ego vehicle uses six 360° pinhole cameras providing aligned RGB, depth, and semantic-segmentation images at 1024 × 512 resolution with 110° FOV.A roof-mounted LiDAR operates at 10 Hz, complemented by a rasterized BEV semantic stream.
  • Sensor Configuration.: Each roadside unit combines co-located RGB-depth cameras and a 64-beam LiDAR, elevated to 5 m and pitched downward to overlook roadway conflict zones.The cameras use 1024 × 512 resolution and 90° FOV; the LiDAR provides up to 1.0M points/s and 75 m range.
  • Sensor Configuration.: Route-adaptive deployment maintains an active RSU for the ego vehicle at every timestep, supplying effective beyond-line-of-sight information instead of relying on fixed infrastructure poses.RSUs are placed along routes to support continuous cross-view coverage.
  • Sensor Configuration.: RSUs are positioned about every 80 m on straight segments, at intersections, and near safety-critical event triggers, while redundant nearby units are pruned.At runtime, the nearest forward-facing RSU is selected for each synchronized frame.
  • Scenario Types.: V2XBench covers 38 safety-critical scenario types, grouped into two categories and emphasizing occlusion-heavy, beyond-line-of-sight V2X hazards.The dataset also retains diverse challenging autonomous-driving scenarios to broaden coverage.

Data Types. … Annotation Pipeline.

V2XBench provides synchronized ego–roadside multimodal data, calibrated annotations, and 38 safety-critical scenario types, while Chat-V2XBench adds progressively structured, V2X-aware language supervision. Its annotation pipeline combines simulator-derived exact labels with ground-truth-conditioned VLM labeling and human verification.

  • Data Types.: 140K synchronized frames and over 2.5 million annotated 3D bounding boxes support ego–roadside cooperative perception with RGB, depth, LiDAR, and calibration data.The platform records synchronized sensing from the ego vehicle and active RSU, with complete calibration and intrinsic/extrinsic matrices for cross-view transformation.
  • Data Types.: 38 safety-critical scenario types cover occlusion- and beyond-line-of-sight-dominated V2X cases alongside other challenging autonomous-driving scenarios.The scenario taxonomy highlights settings where roadside infrastructure is most beneficial.
  • Data Types.: Three 3D bounding-box views—ego-centric, roadside-centric, and cross-view fused—enable consistent cooperative and single-agent training and evaluation.Annotations include class, pose, dimensions, velocity, vehicles, VRUs, traffic-control elements, and other road objects.
  • A.2 Details on Chat-V2XBench: Chat-V2XBench annotates one keyframe every 10 frames, pairing synchronized ego–roadside visual inputs with V2X-aware QA while avoiding near-duplicate temporal labels.At the 10 Hz recording rate, this corresponds to one annotated keyframe per second while retaining temporal coverage.
  • Question–Answer Content.: The four-level question hierarchy progresses from scene understanding through spatial alignment to motion planning and other increasingly difficult cooperative-reasoning skills.Questions cover global scene context, cross-view coordinate transformation for jointly visible objects, and behavior decisions with physical justification.
  • Annotation Pipeline.: Objective queries receive exact simulator-derived answers from logged states or rigid transformations, avoiding annotation noise.This includes traffic-light, previous-behavior, speed-trend, and spatial-alignment questions.
  • Annotation Pipeline.: Open-ended questions are auto-labeled by Qwen3-VL-Plus using multi-view images and ground-truth object hints, then human-spot-checked for factual reliability.The process provides fluent natural-language supervision while preserving consistency with true scene geometry.

A.3 Closed-Loop Evaluation Details

The framework is evaluated in CARLA’s Town12 through 50 distinct closed-loop scenarios spanning complex environments from urban intersections to multi-lane highways. Assessment covers route progress, safety, kinematics, comfort, and communication cost.

  • A.3 Closed-Loop Evaluation Details: Closed-loop testing uses Town12’s large, realistic map and 50 distinct scenarios covering diverse road topologies and complex driving environments.The scenarios range from dense urban intersections to multi-lane highways.
  • A.3 Closed-Loop Evaluation Details: Five metrics evaluate driving quality across route progress, safety, kinematics, system efficiency, and communication cost.The metric suite comprises Route Completion, Driving Score, Efficiency, Comfortness, and Communication Bandwidth.
  • A.3 Closed-Loop Evaluation Details: Route Completion measures navigated route distance, while Driving Score multiplies Route Completion by an infraction penalty for collisions, red-light violations, and lane invasions.The infraction score is bounded between 0 and 1 and decays exponentially after safety infractions.
  • A.3 Closed-Loop Evaluation Details: Efficiency penalizes unnecessarily slow driving, Comfortness penalizes harsh dynamics and jerk, and Communication Bandwidth measures continuous RSU-to-ego payload in B/s.Efficiency follows Bench2Drive (Jia et al. 2024), while Comfortness evaluates acceleration, steering, jerk, and yaw-rate smoothness.

B AURORA Training Details

AURORA is trained through a five-stage curriculum that incrementally develops perception, cross-view vision-language understanding, trajectory planning, and joint reasoning.

  • B AURORA Training Details: AURORA uses a five-stage curriculum in which each stage unlocks a new capability.The curriculum progressively expands the model’s functionality rather than training all capabilities at once.
  • B AURORA Training Details: The curriculum progresses from single-view perception to cross-view fusion and vision-language alignment.This sequence establishes increasingly cooperative and semantically grounded representations.
  • B AURORA Training Details: The final stages add trajectory planning and joint reasoning while reusing weights learned in previous stages.Each stage builds on the preceding stage’s parameters.

Stage 1: Ego-view perception pre-training. … Stage 5: Joint reasoning and planning fine-tuning.

AURORA is trained through five progressively integrated stages, moving from separate ego and roadside perception to cross-view language alignment, trajectory planning, and joint reasoning–planning fine-tuning.

  • Stage 1: Ego-view perception pre-training.: Stage 1 pre-trains the ego image backbone, 3D detection head, and vectorized map head using ego-view detection and map losses.Detection predicts multi-class 3D boxes and traffic-light state, while mapping predicts lane and boundary polylines with focal, ℓ1, and auxiliary assignment terms.
  • Training schedule: The staged schedule progressively couples perception, cross-view fusion, language supervision, planning, and mixed reasoning objectives rather than training all components simultaneously.This progression follows separate branch pre-training, joint CQAF alignment, planner activation, and final mixed QA fine-tuning.
  • Stage 2: Roadside-view perception pre-training.: Stage 2 independently pre-trains a roadside backbone, detection head, and map head with viewpoint-specific range and roadside-coordinate supervision.The static forward-facing roadside branch does not estimate ego motion or traffic-light state.
  • Stage 3: Cross-view fusion and vision–language alignment.: Stage 3 jointly fine-tunes both perception branches and CQAF while LoRA-adapting the language model to align ego, roadside, and fused queries with language.The planner remains unsupervised, and alignment uses next-token prediction on question–answer pairs.
  • Stage 4: Trajectory planning.: Stage 4 activates trajectory planning with frozen image backbones, feeding fused semantic tokens to the language model and waypoint hidden state to interchangeable VAE, MLP, or diffusion decoders.The objective is L4 = Lvlm + Lplan.
  • Stage 5: Joint reasoning and planning fine-tuning.: Stage 5 jointly fine-tunes language reasoning and planning with mixed question-answering and planning samples, preserving language capability while refining trajectories.Waypoint supervision applies only to samples containing a waypoint token; QA-only samples contribute solely to the language loss.

Training Configuration.

Training uses deterministic single-seed execution and a shared AdamW, cosine-annealing, warmup, and mixed-precision optimization setup across Stages 1–5. Stage-specific batch sizes, epochs, base learning rates, and backbone-freezing policies are reported in Table 9 for the submitted AURORA checkpoint.

  • Random seed: The reported models use random seed 0 with deterministic execution across Python, NumPy, PyTorch, CUDA, and cuDNN, without averaging across multiple seeds.All training runs synchronize seeds through the MMCV random-seed utility.
  • Optimization: Stages 1–5 share AdamW optimization with β1 = 0.9, β2 = 0.999, weight decay 10−5, cosine annealing, 500-iteration linear warmup, and FP16 dynamic loss scaling.The warmup ratio is 1/3.
  • Optimization: Table 9 summarizes stage-specific batch sizes, epoch counts, base learning rates, and backbone-freezing policies for the submitted AURORA checkpoint.The table uses the shared optimizer and schedule settings described in the text.
Loading 2608.21032v1…