PaperlayerView exact revision on arXiv

Source-linked AI summary

CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planning

Kang Yang, Shuai Liu, Hang Li, Yance Fang, Deying Li, Yongcai Wang

arXiv:2609.35823v1cs.CV

TL;DR

Cooperative-driving systems use roadside observations through geometric representations, while VLMs offer a way to use paired views for scene understanding and planning. CoVLM-Bench evaluates CDQA and planning on shared vehicle–infrastructure scenes, and CoVLM-Drive improves QA accuracy and planning error metrics in the reported comparisons.

  • Problem

    Cooperative systems use roadside observations through geometric representations, motivating direct VLM use of paired views for scene understanding and planning.

    Source: §1 · pp. 1–2

  • Method

    CoVLM-Bench aligns scene-grounded CDQA and cooperative-planning targets on paired scenes, and CoVLM-Drive is a unified VLM baseline for both tasks.

    Source: §1 · pp. 2–3

  • Results

    CDQA adaptation improves answer accuracy, while QA initialization and rationale supervision each reduce planning error, and standard CoVLM-Drive achieves lower FDE than compared V2X planners.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The benchmark and baseline support learning and evaluating cooperative scene understanding and planning.

    Source: §6 · p. 10

    Underlying images and cooperative labels follow upstream dataset access terms, so users must obtain restricted source data separately where required.

    Source: I RELEASE, ETHICS, AND LIMITATIONS · p. 33

Abstract

from arXiv · show

Vision-language models (VLMs) have made substantial progress in autonomous driving, but their success has primarily been studied in ego-centric scenes. Infrastructure-side observations provide views beyond the ego vehicle's field of view, yet conventional cooperative-driving systems typically transform them into geometric representations for downstream perception and planning. Directly incorporating these views into VLMs offers an opportunity to improve cooperative scene understanding and trajectory planning. However, question answering and trajectory planning have not been jointly evaluated on the same real-world vehicle-infrastructure scenes. We present CoVLM-Bench, a benchmark for cooperative driving question answering (CDQA) and cooperative planning (CP) on vehicle-infrastructure paired scenes. CoVLM-Bench provides scene-grounded CDQA annotations, three-part rationales as auxiliary supervision, and future trajectory targets derived from recorded ego motion. It contains 2,196 paired frames with 35,136 CDQA annotations, while CP predicts six waypoints over a three-second horizon. The annotations combine model-assisted drafting, record-based computation, and human verification. Built upon CoVLM-Bench, we introduce CoVLM-Drive, a unified VLM baseline that directly uses paired views for both CDQA and CP. Experiments show that CDQA adaptation improves answer accuracy and that CoVLM-Drive reaches a lower FDE than the compared V2X planners; QA initialization and rationale supervision each reduce planning error. Together, CoVLM-Bench and CoVLM-Drive support the training and comparison of VLMs for cooperative scene understanding and planning.

1 INTRODUCTION

Cooperative VLM research needs to connect scene understanding and planning on the same real-world paired scenes. CoVLM-Bench supplies those tasks and CoVLM-Drive provides a unified baseline, with reported gains across answering and planning.

1 INTRODUCTION

  • Conventional cooperative pipelines pass geometric representations of roadside observations to planners, whereas this work explores direct paired-image use by VLMs.Infrastructure cameras can capture participants outside the ego vehicle’s field of view or hidden by other road users.Source: §1 · pp. 1–2
  • Question answering and trajectory planning had not been jointly evaluated on the same real-world paired scenes, despite complementary existing resources for each task.The introduction frames joint evaluation as a way to assess both abilities on corresponding scenes and examine whether understanding-oriented supervision improves planning.Source: §1 · p. 2
  • CDQA adaptation improves answer accuracy, while QA initialization and rationale supervision each reduce planning error and standard CP achieves lower FDE than compared V2X planners.These results are reported for the unified CoVLM-Drive baseline.Source: §1 · p. 3
  • CoVLM-Bench aligns scene-grounded CDQA, three-part rationales, and motion-derived planning targets on the same real-world vehicle–infrastructure scenes.It contains 2,196 paired frames, 35,136 question–answer records, and 2,129 complete-horizon planning examples.Source: §1 · pp. 2–3
  • The paper evaluates cooperative VLMs across task adaptation, supervision sources, backbone choices, and roadside-input conditions.CoVLM-Drive is the unified VLM baseline for cooperative understanding and planning.Source: §1 · p. 3

2 RELATED WORK

Related work spans driving QA and visual evidence, driving VLMs for trajectory prediction, and cooperative perception and planning. CoVLM-Bench is positioned around evaluating understanding and planning together on the same paired real-world scenes.

2.1 DRIVING QA AND VISUAL EVIDENCE

  • Driving QA work tests whether answers depend on visual evidence, and COVLM-BENCH scores answers recomputed from source records and reports conditions with roadside input withheld or replaced.The cited work also examines object removal, corrupted or missing vision, and hallucination diagnostics.Source: §2.1 · p. 3

2.3 V2X PERCEPTION, PLANNING, AND COOPERATIVE EVIDENCE

  • Prior cooperative QA and reasoning benchmarks cover view availability or grounded outputs, while other works emphasize cross-view geometry, simulated V2X, or closed-loop behavior.COVLM-BENCH’s positioning combines CDQA and planning targets on real-world paired scenes.Source: §2.1 · p. 3, §2.3 · p. 4

2.2 VLMS FOR DRIVING AND TRAJECTORY PLANNING

  • Driving VLMs connect scene understanding to commands or trajectories through varied output designs; COVLM-DRIVE uses auxiliary rationale text and structured command and trajectory heads.The cited literature notes that generated rationales need not fully describe a model’s internal decision process.Source: §2.2 · p. 3

2.3 V2X PERCEPTION, PLANNING, AND COOPERATIVE EVIDENCE

  • Cooperative perception research measures detection, bandwidth, or robustness, while MDrive reports that stronger cooperative perception need not improve planning.This motivates measuring understanding and decisions on the same frames rather than inferring one from the other.Source: §2.3 · p. 4
  • CoVLM-Bench places CDQA annotations and three-part rationales on real-world vehicle-to-infrastructure scenes that also carry cooperative-planning targets.Table 1 compares benchmark support for paired V2X input, source records, QA, reasoning annotations, and planning targets.Source: §2.3 · p. 4, Table 1 · p. 4

3 COVLM-BENCH: BENCHMARK DESIGN

COVLM-Bench organizes synchronized vehicle and infrastructure observations, object records, and ego-motion into supervision for cooperative QA and planning. Its annotation pipeline combines record-based evidence, model-assisted drafts, rationale generation, and human verification.

3 COVLM-BENCH: BENCHMARK DESIGN

  • The benchmark uses 2,196 paired frames for CDQA and 2,129 complete three-second motion records for CP targets and L4 rationales.The annotation layers organize supervision, with L3 providing CDQA data and evaluation.Source: §3 · p. 4

3.1 L1: OBJECT EVIDENCE

  • L1 matches cooperative boxes to ego-side and roadside labels within 3 m, assigning both, ego-only, roadside-only, or unmatched source views.Of 54,171 objects, 23,792 (43.9%) matched only roadside labels, and these objects appeared in 98.3% of frames; track context distinguishes parking from temporary stops and captures other motion states.Source: §3.1 · p. 5

3.2 L2–L3: SCENE DESCRIPTIONS AND QUESTION–ANSWER ANNOTATIONS

  • Qwen3-VL-32B drafts scene descriptions and CDQA items, while fixed rules compute answers for eight templates from records, motion, views, and geometry.The 16 templates per frame yield 35,136 records; five record-computed templates support reported CDQA accuracy.Source: §3.2 · pp. 5–6

3.3 L4: RATIONALES

  • Each three-part rationale covers a scene overview, V2X-aware critical objects, and decision reasoning that distinguishes roadside contributions from ego observations.Rationales are auxiliary supervision and generated outputs, not a scored task.Source: Figure 2 · p. 5, §3.3 · p. 6
  • CP samples six ego-relative waypoints at 0.5-second intervals through 3.0 seconds and derives a six-class command from recorded motion.Every scene description, question–answer record, and rationale is human-verified against paired images and source records, with rejected items corrected or regenerated.Source: §3.3 · p. 6

4 COVLM-DRIVE: A UNIFIED VLM BASELINE

CoVLM-DRIVE uses one VLM to process paired ego and roadside images for cooperative question answering and planning. Its decoder generates answers or rationales, while structured heads predict commands and waypoints under joint task supervision.

4 COVLM-DRIVE: A UNIFIED VLM BASELINE

  • The same VLM backbone supports CDQA responses and cooperative planning from ego and roadside images plus a task prompt.The decoder generates CDQA answers or three-part rationales; two structured heads predict the command and six ego-relative waypoints.Source: §4 · p. 6, Figure 3 · p. 6

4.1 PAIRED-VIEW ENCODING AND TASK OUTPUTS

  • The ego image, roadside image, and prompt are placed in one Qwen3-VL input sequence, with LoRA adapting the backbone.The structured heads use the masked mean of prompt-token hidden states, including image tokens; the waypoint MLP outputs six planar waypoints and the command MLP outputs a command distribution.Source: §4.1 · p. 7

4.2 TASK SUPERVISION AND INFERENCE

  • CDQA uses autoregressive cross-entropy on question–answer annotations, while CP jointly trains adapters and structured heads with rationale, command, and trajectory supervision.The loss combines rationale token-level cross-entropy, command cross-entropy, Smooth L1 waypoint regression, and an endpoint term; CP may initialize adapters from CDQA fine-tuning.Source: §4.2 · p. 7
  • At inference, the decoder generates CDQA answers and can generate rationales, while CP heads directly output a command and trajectory.Source: §4.2 · p. 7

5 EXPERIMENTS

Experiments evaluate CDQA and CP on a scene-disjoint split, with matched planning examples and shared input conditions for comparisons. CDQA adaptation improves answer accuracy, and the standard CP configuration has lower FDE than the compared V2X planners; supervision choices and roadside-input controls further test these results.

5.1 EXPERIMENTAL SETUP

  • The scene-disjoint evaluation uses 654 planning examples, while CDQA accuracy is measured on 2,861 eligible items per input condition across five objective tasks.FDE is waypoint L2 at 3.0 seconds; CP rows share the same examples and ego-relative waypoint convention.Source: §5.1 · p. 7

5.3 CP RESULTS AND BACKBONE COMPARISON

  • 4.911 m FDE for standard COVLM-DRIVE is lower than MAP’s 5.240 m and UniMM-V2X’s 5.357 m; its inference runs at 223.1 ms per frame and 4.48 FPS.Alternative backbones achieve 5.076–5.538 m FDE, while the best zero-shot result is 11.227 m.Source: Table 4 · p. 8, §5.3 · pp. 8–9

5.2 CDQA RESULTS

  • 14.9 percentage points: CDQA adaptation raises Qwen3-VL-8B accuracy from 24.7% to 39.6%.Zero-shot accuracy with matched paired views ranges from 16.9% to 51.1%; Qwen3-VL-32B has the highest accuracy.Source: §5.2 · pp. 7–8, Table 3 · p. 8

5.4 EFFECT OF QA AND RATIONALE SUPERVISION

  • Relative to planning targets alone at 5.197 m FDE, QA initialization reaches 5.083 m and rationale supervision reaches 4.911 m.QA initialization also raises CDQA accuracy after CP training; the standard CP configuration uses rationale supervision alone.Source: §5.4 · p. 9, Table 5 · p. 9

5.5 EFFECT OF ROADSIDE OBSERVATIONS

  • With matched roadside input, CP reaches 4.911 m FDE versus 5.776 m ego-only, 5.085 m blank, and 4.976 m shuffled; matched input also beats absent-view controls for seven of eight zero-shot CDQA models.CDQA adaptation increases the gap over the best absent-view control from 5.2 to 25.3 percentage points; shuffling roadside input moves predicted waypoints by a mean of 0.73 m, exceeding 0.1 m on 95.3% of examples.Source: §5.5 · p. 9, Table 6 · p. 9

6 CONCLUSION

The paper introduces CoVLM-Bench and CoVLM-Drive for cooperative scene understanding and planning. Experiments report improved CDQA accuracy, lower planning error with QA initialization and rationale supervision, and lower FDE than compared V2X planners.

6 CONCLUSION

  • CoVLM-Bench connects scene-grounded CDQA annotations and ego-trajectory targets on real-world vehicle–infrastructure observations, while CoVLM-Drive provides a unified VLM baseline for both tasks.The benchmark and baseline support learning and evaluating cooperative scene understanding and planning.Source: §6 · pp. 9–10
  • CDQA adaptation improves answer accuracy, and the standard CP configuration achieves a lower FDE than the compared V2X planners.QA initialization and rationale supervision each reduce planning error.Source: §6 · p. 10

A BENCHMARK CONSTRUCTION AND VALIDATION

CoVLM-Bench builds CDQA and planning supervision from paired vehicle–infrastructure scenes, using model-assisted drafts, recorded data, and human review. Its templates distinguish released annotation tasks from evaluation prompts and outputs.

A BENCHMARK CONSTRUCTION AND VALIDATION

  • Paired images and structured records support scene descriptions and CDQA and rationale drafts, while computable answers and supporting facts derive from object visibility, geometry, and motion records.Rationale prompts include observed behavior references, but checks remove explicit future-target references; recorded motion supplies separate command and waypoint targets.Source: A BENCHMARK CONSTRUCTION AND VALIDATION · p. 18
  • Planning targets transform subsequent ego positions into the current ego frame and sample six waypoints at 0.5-second intervals through 3.0 seconds; the complete-horizon set has 2,129 examples.The scene-disjoint split contains 1,475 training and 654 validation examples, with a six-class command derived by fixed rules.Source: A BENCHMARK CONSTRUCTION AND VALIDATION · p. 18
  • Every scene description, CDQA record, and rationale was checked against paired images; rejected items were corrected or regenerated, and about 10% of a 100-item audit failed first-pass inspection.The pass criterion excludes errors contradicting images or structured records, rather than grading annotation quality; the checks are independent of model training.Source: A BENCHMARK CONSTRUCTION AND VALIDATION · p. 18, Table 7 · p. 19

B ANNOTATION TEMPLATES AND MODEL PROMPTS

B.1 CDQA EVALUATION PROMPTS

  • Evaluation gives each model the same seven-question frame–condition bundle, but Overall accuracy covers only five objective tasks; two roadside-evidence response fields are excluded.Decoding is deterministic, and no evaluator model or judge prompt is used.Source: §B.1 · p. 21, Figure 5 · p. 22, Table 10 · p. 22

B.2 CP PROMPTS AND RATIONALE FORMAT

  • The CP rationale output has three parts—scene overview, V2X-aware critical objects, and decision reasoning—while command classes and numerical waypoints remain separate structured outputs.The zero-shot prompt supplies ego and matched roadside images and requests a command plus six waypoints in JSON.Source: §B.2 · p. 23, Figure 6 · p. 23

C TRAINING AND EVALUATION PROTOCOLS

The protocols describe task-specific adaptations, scoring rules for planning and CDQA, and coordinate alignment for external planners. Planning comparisons use a shared validation split and aligned output conventions.

C TRAINING AND EVALUATION PROTOCOLS

  • CDQA adaptation and CP adaptation use task-specific configurations of CoVLM-Drive; standard CP adds three-part rationale supervision without QA initialization.The supervision study also initializes CP from the CDQA-adapted checkpoint; the standard CP setup uses Qwen3-VL-8B and ten-epoch training.Source: C TRAINING AND EVALUATION PROTOCOLS · pp. 23–24

C.1 METRICS AND SCORING

  • Planning rows use ego-relative waypoints on the same 654-frame validation split, reporting L2 at the first five horizons and FDE at 3.0 seconds, with lower error better.Command accuracy measures label agreement, while balanced accuracy weights command classes equally, a distinction relevant to unequal class frequencies.Source: §C.1 · p. 24
  • CDQA exact-answer accuracy covers five objective tasks over 2,861 eligible items per condition; Overall is their micro-average, and invalid frame–task pairs are excluded consistently.Zero-shot parse failures remain in every metric denominator, with unparseable outputs assigned an unknown command and zero-displacement waypoints.Source: §C.1 · pp. 24–25

C.2 EXTERNAL BASELINE ALIGNMENT

  • External planner FDE is computed after converting coordinates and aligning the six-step horizon; UniV2X and MAP are evaluated on 654 aligned examples.UniV2X’s native protocol differs in frame coverage, trajectory format, and reported task set, so the table uses the shared evaluation protocol.Source: §C.2 · p. 25

D SUPPLEMENTARY CDQA RESULTS

Roadside-input comparisons show that V2X-evidence answers depend on a scene-matched roadside image, while object-related CDQA gains over absent-view controls are larger than the overall gains that include maneuver QA.

D SUPPLEMENTARY CDQA RESULTS

  • 62.4% to 1.8% for Qwen3-VL-32B and 39.5% to 2.7% for CDQA-adapted CoVLM-Drive: mismatched roadside input reduces V2X accuracy to near ego-only levels of 0.9% and 1.3%.The V2X questions therefore require the roadside image of the same scene.Source: D SUPPLEMENTARY CDQA RESULTS · p. 25, Table 11 · p. 26
  • +33.2 to +46.8 points for Qwen3-VL-32B and +25.3 to +32.3 points for CDQA-adapted CoVLM-Drive: removing maneuver QA increases the advantage over the best absent-view control.The four remaining object-related tasks pool 2,207 items; LLaVA-OneVision-7B has a positive gap of +0.6.Source: D SUPPLEMENTARY CDQA RESULTS · pp. 25–26, Table 11 · p. 26

E DETAILED SUPERVISION ABLATIONS

The ablation separates the effects of QA initialization and three-part rationale supervision on planning and post-training CDQA accuracy. Their planning gains do not combine additively.

E DETAILED SUPERVISION ABLATIONS

  • 5.197 m FDE with planning targets alone falls to 5.083 m with QA initialization or 4.911 m with rationale supervision; using both gives 5.095 m.The comparison holds paired images, structured heads, planning targets, and CP training budget fixed.Source: E DETAILED SUPERVISION ABLATIONS · p. 27
  • The combined 5.095 m FDE indicates that QA initialization and rationale supervision act as substitutes rather than complements, so standard CP uses rationale supervision alone.The table caption identifies the standard CP configuration as highlighted.Source: E DETAILED SUPERVISION ABLATIONS · p. 27, Table 12 · p. 27
  • QA initialization raises post-CP CDQA accuracy from 24.8% and 25.8% without it to 37.4% and 34.9% with it.Source: E DETAILED SUPERVISION ABLATIONS · p. 27

F ROADSIDE-INPUT ANALYSIS

The roadside-input analysis compares matched, missing, blank, and scene-mismatched views for planning and CDQA, then illustrates model outputs and estimates roadside-image transmission cost. The release materials include evaluation artifacts alongside upstream-data access constraints.

F ROADSIDE-INPUT ANALYSIS

  • 4.922 m FDE with valid roadside input is lower than ego-only, blank, and shuffled controls by 0.956, 0.269, and 0.051 m on 513 examples with a roadside-only critical object.The critical-object list comes from a fixed distance-and-motion rule, not a measured effect on the decision.Source: F ROADSIDE-INPUT ANALYSIS · p. 28
  • Valid input has lower FDE on 63.5% of examples versus ego-only, 52.6% versus blank, and 51.4% versus shuffled; valid and shuffled predictions differ by a mean 0.73 m.The mean valid-versus-shuffled waypoint displacement exceeds 0.1 m on 95.3% of examples; this measures sensitivity, not prediction quality.Source: F ROADSIDE-INPUT ANALYSIS · p. 28

G.1 CDQA RESPONSES

  • CDQA examples show matched roadside views supporting reference-consistent answers, while withholding, blanking, or shuffling the view can remove, misstate, or misidentify queried objects.Examples include future-distance queries, recorded-maneuver comparison, roadside-only object lists, and catalog distance.Source: Figure 8 · p. 29, Figure 9 · p. 30

G.2 GENERATED RATIONALES

  • The generated rationale examples use three response parts, with object identifiers [E1] and [E2] distinct from camera-neutral CDQA tokens.Commands and trajectories come from separate structured heads.Source: Figure 10 · p. 31, §G.2 · p. 31

G.3 COOPERATIVE PLANNING

  • In two command-correct examples, CoVLM-Drive stays closest to the reference among shown planners; another predicts a left turn for a straight-driving reference.The figure visualizes structured commands and waypoints evaluated in Table 4.Source: Figure 11 · p. 32

H ROADSIDE IMAGE TRANSMISSION COST

  • At 10 Mbit/s, source-resolution roadside images take 252.3 ms to transmit, versus 41.9–98.9 ms for resized images across tested JPEG qualities.The estimates assume one roadside image per planning step and exclude protocol overhead, queueing, retransmission, packet loss, and contention.Source: H ROADSIDE IMAGE TRANSMISSION COST · p. 32, Table 13 · p. 33

I RELEASE, ETHICS, AND LIMITATIONS

  • The release package includes splits, annotations, prompts, evaluation and reproduction artifacts, while underlying images and cooperative labels remain subject to upstream access terms and privacy constraints.The dataset adds no personal identity labels and follows upstream privacy-filtering and redistribution constraints.Source: I RELEASE, ETHICS, AND LIMITATIONS · p. 33

I.1 EVALUATION SCOPE

The evaluation is limited to CDQA and open-loop trajectory prediction from paired camera observations. It does not establish driving safety or cover several deployment-relevant conditions.

I.1 EVALUATION SCOPE

  • The benchmark measures CDQA and open-loop trajectory prediction from paired camera observations; driving safety would require interactive or closed-loop evaluation.Input controls assess dependence on roadside evidence.Source: §I.1 · p. 33
  • Maneuver QA compares answers to the ego vehicle's recorded maneuver, not to a safety-optimal action.The question asks for the safest action, but its reference is discretized from the recorded trajectory.Source: §I.1 · p. 33
  • The tests use one real-world V2I source and omit temporal accumulation, calibration and pose errors, communication impairments, interactive responses, and cross-domain generalization.The reported evaluation is camera-only and offline.Source: §I.1 · p. 33

Related papers

Loading 2609.35823v1…