V2V-GoT: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Models and Graph-of-Thoughts

• Local cameras and LiDAR can be limited when nearby large objects occlude vehicles or other potential obstacles, motivating V2V sharing of perception information.Cooperative perception is described as improving overall detection and tracking accuracy.Source: §I · p. 1
• The introduction identifies a gap in prior work: V2V-LLM [12] fuses CAV perception features but does not explore chain-of-thought or graph-of-thoughts reasoning.Source: §I · p. 1
• V2V-GoT uses perception features from both current and previous timesteps, unlike V2V-LLM [12] and DriveLM, which use current-timestep features.The stated aim is to better capture temporal dynamics of surrounding objects.Source: §I · p. 2
• V2V-GoT achieves the lowest planning collision rates and L2 errors among the compared baselines, while ablations support the usefulness of its new QA types across perception, prediction, and planning.The QA types are occlusion-aware perception and planning-aware prediction.Source: §I · p. 2
• The framework adds graph-of-thoughts reasoning designed for MLLM-based cooperative autonomous driving, including occlusion-aware perception and planning-aware prediction.Answers from parent QA nodes provide input context to child QA nodes.Source: §I · pp. 1–2
• V2V-GoT-QA is curated as a dataset, and V2V-GoT is developed as a benchmark and baseline for future comparative research on MLLM-based cooperative driving with graph-of-thoughts reasoning.Source: §I · p. 2

A. Vehicle-to-Vehicle Cooperative Perception

• V2V cooperative perception shares perception information between CAVs to improve overall detection and tracking accuracy.Source: A. Vehicle-to-Vehicle Cooperative… · p. 2

B. LLM-based Autonomous Driving

• The described LLM-based autonomous-driving research focuses on individual vehicles, whereas this work develops MLLM-based models for V2V cooperative driving.DriveLM considers graph-of-thoughts for a single vehicle without V2V cooperation.Source: B. LLM-based Autonomous Driving · p. 2

C. LLM-based Cooperative Autonomous Driving

• Prior cooperative-driving approaches either used language-only coordination or had not explored graph-of-thoughts reasoning for MLLM-based cooperation.V2V-LLM [12] uses MLLMs for cooperative perception and planning questions but leaves graph-of-thoughts unexplored.Source: C. LLM-based Cooperative Autonomous… · p. 2

A. Problem Setting

• The MLLM fuses perception features from all CAVs to answer questions, with directed QA edges passing a parent answer as context to a child.The dataset organizes nine QA types across perception (Q1–Q4), prediction (Q5–Q7), and planning (Q8–Q9).Source: A. Problem Setting · p. 3, Figure 2 · p. 3

B. Dataset Details

• V2V-GoT-QA contains 110,610 training and 31,014 testing QA pairs, curated using V2V4Real as its base dataset.V2V4Real has 7,105 training frames and 1,993 testing frames.Source: B. Dataset Details · p. 3

C. Dataset Curation

• The dataset uses ground-truth boxes, future trajectories, and geometric relationships; child-node context comes from ground-truth parent answers during training and model outputs during inference.Source: C. Dataset Curation · p. 3

D. Perception QAs

• The perception-QA design identifies notable objects near the ego CAV’s planned trajectory by separately handling visible and invisible objects, then merging them.The passage presents this divide-and-conquer design as potentially making it easier for the model to learn where to focus in perception features.Source: D. Perception QAs · p. 4
• Q1 labels up to three ego-detected ground-truth objects within 10 meters of a six-waypoint, three-second reference trajectory and uses F1 score.Source: D. Perception QAs · p. 4
• Q2 supplies up to three closest ground-truth objects not occluded by other objects as context for identifying invisible notable objects.Q2 is evaluated with F1 score.Source: D. Perception QAs · p. 4
• Q3 identifies up to three ground-truth objects within 10 meters of the reference trajectory that the ego detector misses, using Q2’s occluding-object answer as context.Q4 merges the Q1 visible-object and Q3 invisible-object answers into the final notable-object answer used by prediction questions; both use F1 score.Source: D. Perception QAs · p. 4

E. Prediction QAs

• Planning-aware prediction merges perception-based and other CAVs’ planned-trajectory predictions into an overall prediction for subsequent planning.Q5 uses current and previous perception features; Q6 includes other CAVs’ planned trajectories and identifies whether those CAVs are notable objects.Source: E. Prediction QAs · p. 4
• Q5 predicts notable objects’ future trajectories and classifies their motion; Q6 uses other CAVs’ planned future trajectories as prediction context.Q5 uses Q4’s notable-object answer as context; Q6 also uses Q4’s answer and evaluates identification of other CAVs as notable objects.Source: E. Prediction QAs · p. 4

F. Planning QAs

• Q8 classifies suggested speed and steering settings from Q7’s prediction, and Q9 uses those settings to generate six future waypoints over three seconds.Q8 uses five speed categories and five steering categories; Q9’s trajectory is evaluated by L2 error and collision rate.Source: F. Planning QAs · pp. 4–5

A. Architecture

• PointPillars extracts LiDAR perception features for each CAV, and V2V-GoT uses current and previous timesteps to form visual tokens for LLaVA-v1.5-7b.The MLLM combines those visual tokens with language tokens from the question and context to generate a natural-language answer.Source: A. Architecture · p. 5

B. Training and Inference

• Training updates the projector layers and LoRA parts while freezing the rest of the model, using Adam for 10 epochs at a starting learning rate of 2e−5.Training uses NVIDIA H100-80GB GPUs and batch size 32.Source: B. Training and Inference · p. 5
• At inference, each directed edge passes a parent QA’s answer as input context to its child QA.Inference follows the proposed graph-of-thoughts structure.Source: B. Training and Inference · p. 5

A. Baseline Methods

• The baseline set includes no fusion, early fusion, intermediate fusion, and V2V-LLM, which fuses scene-level and object-level features from multiple CAVs.No fusion uses one CAV’s features, while early fusion uses the union of both CAVs’ point clouds; intermediate fusion uses cooperative 3D detectors.Source: A. Baseline Methods · p. 5
• The graph structures cover perception Q1–Q4, prediction Q5–Q7, and planning Q8–Q9; Table II reports testing performance across QA tasks and inference graphs.Table II identifies F1, L2, L1, collision rate, and communication cost as reported metrics.Source: Figure 4 · p. 6, Table II · p. 6
• The final planning comparison uses the same V2V4Real base dataset and task, and modifies V2V-LLM baselines to use current and previous timestep features.This adjustment is made for a fair comparison with V2V-GoT.Source: A. Baseline Methods · pp. 5–6

B. Quantitative Results

• V2V-GoT achieves the lowest L2 errors and collision rates among the baselines with different fusion approaches in the final planning task.The authors report that graph-of-thoughts reasoning boosts planning performance.Source: B. Quantitative Results · p. 6

C. Ablation Study

• The ablation study compares the proposed graph with simplified perception and prediction graphs to assess the proposed graph designs.The simplified perception graph is shown in Figure 4b, and the simplified prediction graph in Figure 4c.Source: C. Ablation Study · p. 6

1) Simplified Perception Graph:

• The simplified perception graph performs worse than V2V-GoT on Q4 perception and subsequent Q7 prediction and Q9 planning.The ablation creates additional Q4 training samples without context before testing with the simplified perception graph.Source: 1) Simplified Perception Graph · pp. 6–7

2) Simplified Prediction Graph:

• The simplified prediction graph produces poorer planning than V2V-GoT because its prediction result is poorer; either simplified graph still improves CR over V2V-LLM.The authors attribute the full graph's advantage to planning-aware prediction.Source: 2) Simplified Prediction Graph · p. 7

VI. COMMUNICATION COST

• V2V-GoT has the same overall communication cost as V2V-LLM because shared perception features can be reused across graph questions and timesteps.Intermediate question-and-answer texts need not be transferred unless requested.Source: §VI · p. 7

VII. QUALITATIVE RESULTS

• Qualitative testing shows perception, prediction, and planning outputs generally close to their ground-truth answers.The examples are Q4 overall notable objects, Q7 overall prediction, and Q9 suggested trajectory.Source: §VII · p. 7

VIII. CONCLUSION

• V2V-GoT outperforms all baselines, with ablations indicating its graph designs improve cooperative perception, prediction, and planning.The V2V-GoT-QA dataset and model code have been publicly released.Source: §VIII · p. 7
• Fig. 5 compares model-output yellow crosses and ground-truth green circles against the reference trajectory for Q4 overall notable objects.Its context comes from the Q3 invisible-object and Q1 visible-object inference outputs.Source: Figure 5 · p. 7

Q9: Suggested Trajectory

• The Q9 example asks for a collision-avoiding future trajectory given moderate speed and straight steering settings.Fig. 7 marks the model's suggested trajectory and endpoint in blue, with the ground-truth trajectory and endpoint in green.Source: Figure 7 · p. 8