PaperlayerView exact revision on arXiv

Source-linked AI summary

CMP: Cooperative Motion Prediction with Multi-Agent Communication

Zehao Wang, Yuping Wang, Zhuoyuan Wu, Hengbo Ma, Zhaowei Li, Hang Qiu, Jiachen Li

arXiv:2403.17916v3cs.ROcs.AIcs.CVcs.LGcs.MA

TL;DR

Prior V2V work largely treats cooperative perception and motion prediction separately, despite uncertainty in trajectories derived from upstream detection and tracking. CMP shares information across both stages and aggregates CAV predictions; it lowers prediction errors on OPV2V and V2V4Real, including a 12.3% minADE6 reduction versus V2VNet on OPV2V at 5s.

  • Problem

    V2V research largely studies cooperative perception or motion prediction separately, and prediction methods using ground-truth trajectories do not address uncertainty from upstream detection and tracking.

    Source: §I · p. 1

  • Method

    CMP uses LiDAR-based cooperative perception and prediction, sharing intermediate features and individual predictions across CAVs for final prediction aggregation.

    Source: §I · pp. 1–2

  • Results

    Across OPV2V and V2V4Real, CMP lowers prediction errors; on OPV2V at 5s, minADE6 falls 12.3% versus V2VNet.

    Source: D. Quantitative and Qualitative… · pp. 5–6

  • Takeaways & Limitations

    The framework integrates cooperative perception with trajectory prediction and supports CAVs sharing and fusing LiDAR data to improve detection, tracking, and motion prediction.

    Source: §VI · p. 7

Abstract

from arXiv · show

The confluence of the advancement of Autonomous Vehicles (AVs) and the maturity of Vehicle-to-Everything (V2X) communication has enabled the capability of cooperative connected and automated vehicles (CAVs). Building on top of cooperative perception, this paper explores the feasibility and effectiveness of cooperative motion prediction. Our method, CMP, takes LiDAR signals as model input to enhance tracking and prediction capabilities. Unlike previous work that focuses separately on either cooperative perception or motion prediction, our framework, to the best of our knowledge, is the first to address the unified problem where CAVs share information in both perception and prediction modules. Incorporated into our design is the unique capability to tolerate realistic V2X transmission delays, while dealing with bulky perception representations. We also propose a prediction aggregation module, which unifies the predictions obtained by different CAVs and generates the final prediction. Through extensive experiments and ablation studies on the OPV2V and V2V4Real datasets, we demonstrate the effectiveness of our method in cooperative perception, tracking, and motion prediction. In particular, CMP reduces the average prediction error by 12.3% compared with the strongest baseline. Our work marks a significant step forward in the cooperative capabilities of CAVs, showcasing enhanced performance in complex scenarios. More details can be found on the project website: https://cmp-cooperative-prediction.github.io.

I. INTRODUCTION

The introduction identifies a gap in jointly applying cooperative perception and motion prediction, especially when prediction depends on uncertain upstream detections and tracking. CMP shares information across both stages and aggregates CAV predictions, with reported state-of-the-art performance under practical settings.

I. INTRODUCTION

  • Current V2V research largely treats cooperative perception and motion prediction separately, while ground-truth trajectory inputs in prediction methods omit uncertainties from detection and tracking.The introduction describes their joint application as unexplored.Source: §I · p. 1
  • CMP jointly addresses perception and prediction using raw sensor data, with CAVs sharing information in both components.The pipeline shares intermediate features and historical trajectories, then broadcasts individual predictions for aggregation.Source: §I · pp. 1–2
  • An attention-based prediction aggregation module uses predictions shared by other CAVs to improve prediction accuracy.The introduction also identifies latency robustness as a framework contribution.Source: §I · p. 2
  • CMP achieves state-of-the-art cooperative prediction performance under practical settings on OPV2V and V2V4Real.Source: §I · p. 2

II. RELATED WORK

Related work covers cooperative-perception fusion strategies and advances in single-vehicle motion prediction. The problem formulation defines cooperative prediction as forecasting movable agents detectable by multiple CAVs from LiDAR-derived trajectories.

A. Cooperative Perception

  • Intermediate fusion shares encoded features, balancing early fusion’s bandwidth demands against late fusion’s loss of context and individual detection accuracy.Examples include graph aggregation, attention-based fusion, and adaptive feature selection.Source: A. Cooperative Perception · p. 2

B. Motion Prediction

  • Single-vehicle prediction has progressed from graph-based models to transformer approaches, including MTR’s motion-query pairs for motion-mode prediction.The passage characterizes MTR as more efficient than goal-based strategies and faster-converging than direct regression.Source: B. Motion Prediction · p. 2

III. PROBLEM FORMULATION

  • Cooperative prediction aims to infer future trajectories of movable agents detectable by multiple CAVs, using LiDAR for perception and agents’ historical trajectories for forecasting.The formulation denotes the observed history and future prediction horizons as Th and Tf.Source: §III · p. 2

IV. METHOD

CMP comprises cooperative perception, trajectory prediction, and prediction aggregation. Its perception stage fuses multiple CAVs’ LiDAR-derived features to detect and track agents, under explicit communication-delay assumptions.

A. Method Overview

  • CMP processes cooperative perception, per-CAV trajectory prediction, and prediction aggregation in sequence, with aggregation producing the final prediction hypotheses.Source: A. Method Overview · p. 2

B. Cooperative Perception

  • The cooperative-perception stage modifies CoBEVT for 3D LiDAR object detection and uses AB3DMOT to generate agents’ historical trajectories.AB3DMOT follows the detector in the stated pipeline.Source: B. Cooperative Perception · p. 2, A. Method Overview · p. 2
  • CAVs compress and share BEV features, which are aligned to the ego vehicle and fused before 3D object detection.The method assumes synchronized 100 ms processing; frames taking longer to transmit are dropped, and random delays within that window are treated as one-frame delays.Source: B. Cooperative Perception · pp. 2–3
  • A 3D Kalman filter predicts track states, data association matches them to detections, and birth/death memory manages new and disappeared objects.The tracker outputs historical trajectories for the prediction module.Source: B. Cooperative Perception · p. 3

C. Motion Prediction

The motion-prediction module builds on MTR, encoding trajectory and map context before decoding multimodal future trajectories. Its decoder combines intention queries with dynamic scene searches and represents future positions with Gaussian mixtures.

C. Motion Prediction

  • CMP builds its trajectory-prediction module on MTR’s scene-context encoder and motion decoder.Source: C. Motion Prediction · p. 3
  • The context encoder processes agent trajectories and map information, captures local scene context with a Transformer, and re-encodes initial movement predictions alongside historical context.Trajectory vectors use a PointNet-like encoder, while a Vision Transformer encodes map information.Source: C. Motion Prediction · p. 3
  • A Transformer motion decoder generates multimodal hypotheses through joint global-intention localization and local-movement refinement.It uses 64 representative intention points obtained by k-means clustering ground-truth trajectory endpoints.Source: C. Motion Prediction · p. 3
  • The decoder’s static intention queries share information across motion intentions, while dynamic searching queries gather trajectory-specific scene context.Each decoder layer adds a prediction head; trajectory distributions use Gaussian mixtures.Source: C. Motion Prediction · p. 3
  • Future-position distributions are expressed through Gaussian components and their likelihoods, whose centers provide trajectory predictions.Source: C. Motion Prediction · p. 4

D. Prediction Aggregation

Prediction aggregation combines CAV-specific trajectory predictions with map and BEV context, then uses attention to produce final prediction hypotheses while accommodating delayed or missing inputs.

D. Prediction Aggregation

  • CAVs combine predictions of differing reliability; nearby vehicles may predict better, and the aggregation mechanism scales to varying numbers of CAVs.The paper attributes potential complementarity to predictions from different CAVs.Source: D. Prediction Aggregation · p. 4
  • Each CAV concatenates GMM predictions with map and BEV features, and multi-head self-attention fuses those contextualized inputs across CAVs.Received predictions are combined again with the ego CAV’s map and BEV features before attention.Source: D. Prediction Aggregation · p. 4
  • Two MLPs derive Gaussian parameters from the aggregated feature, which are used to sample the final prediction hypotheses.The aggregated feature represents agent j from CAV i’s perspective.Source: D. Prediction Aggregation · p. 4
  • Attention dynamically weights CAV predictions by consistency with BEV and map context, adapting when predictions are missing due to delays or occlusions.Other CAVs’ GMM components are delayed by one frame.Source: D. Prediction Aggregation · p. 4

E. Loss Functions

The loss functions supervise detection, trajectory prediction, and prediction aggregation, with hard assignment selecting a positive Gaussian component for optimization.

E. Loss Functions

  • Detection combines smooth L1 bounding-box localization and focal classification losses, weighted by βloc = 2/3 and βcls = 1/3 and normalized by positive instances.The detection head uses two convolutional layers.Source: E. Loss Functions · p. 4
  • Motion prediction combines L1 regression for output refinement and negative log-likelihood for actual trajectories using a weighted-average loss.The negative log-likelihood is based on Eq. (2).Source: E. Loss Functions · p. 4
  • Hard assignment selects the motion-query pair nearest the ground-truth endpoint as the positive Gaussian component; auxiliary and Gaussian regression losses receive equal weights.Gaussian regression loss is applied at every decoder layer.Source: E. Loss Functions · p. 4
  • Prediction aggregation uses the same loss function as motion prediction to supervise its final outputs.Source: E. Loss Functions · p. 4

A. Dataset

Evaluation uses OPV2V and V2V4Real, compares against V2VNet, and applies task-specific metrics to detection, tracking, and motion prediction.

A. Dataset

  • OPV2V has 73 roughly 25-second scenarios with 2–7 CAVs; V2V4Real has 67 U.S. scenarios with two vehicles lasting 10–20 seconds.OPV2V evaluation uses a 100 m × 100 m surrounding area at 39 cm map resolution.Source: A. Dataset · p. 4

B. Baselines and Evaluation Metrics

C. Implementation Details

The implementation uses compressed BEV features and a 100 ms communication delay. Results report cooperative-prediction gains alongside detection, tracking, and coverage-dependent findings.

C. Implementation Details

  • Training treats each CAV in a scene as the ego vehicle and uses a 100 ms communication delay with 256× BEV-feature compression.Tracking uses Fmin = 3, Agemin = 2, IoUmin = 0.01 for vehicles, and Distmax = 10.Source: C. Implementation Details · p. 5
  • The prediction model uses six encoder and decoder layers, while aggregation encodes GMM, map, and BEV features before transformer-based decoding.The prediction decoder uses 64 motion-query pairs; aggregation uses an 8-head, 5-layer transformer encoder.Source: C. Implementation Details · p. 5

D. Quantitative and Qualitative Results with Ablation

  • At 5s in OPV2V, CMP reduces minADE6 by 12.3% vs V2VNet and 16.4% vs No Cooperation, and minFDE6 by 15.1% and 19.7%, respectively.The comparisons are for CMP with cooperation in both perception and prediction stages.Source: D. Quantitative and Qualitative… · p. 5
  • In V2V4Real at the same prediction horizon, CMP achieves reductions of 12.0% and 22.2% in minADE6, and 7.3% and 19.0% in minFDE6.Source: D. Quantitative and Qualitative… · p. 6
  • When CAV coverage exceeds 200 m2, the performance improvement reaches 17.5%/28.4% compared with the other settings.Coverage is approximated by the convex hull formed by the CAVs; the comparison is shown at the 5s prediction horizon in OPV2V.Source: D. Quantitative and Qualitative… · p. 6, Figure 4 · p. 6
  • In two scenarios, cooperative prediction detects more surrounding vehicles and its predicted trajectories align more closely with ground truth.Without cooperation, some vehicles are missed and predicted trajectories deviate from ground truth.Source: D. Quantitative and Qualitative… · p. 6, Figure 3 · p. 6
  • With 256× BEV compression, OPV2V bandwidth falls from 82.5 MB/s to 0.32 MB/s with a 4% decrease in detection accuracy.The implementation adopts 256× compression and a 100 ms communication latency.Source: D. Quantitative and Qualitative… · pp. 6–7, Table II · p. 7
  • Cooperative perception increases true positives and reduces false positives and false negatives; tracking shows no detrimental effect from substantial BEV feature compression.The authors report higher MOTA and MOTP in OPV2V than V2V4Real.Source: D. Quantitative and Qualitative… · p. 7, Table III · p. 7

E. Real-time Performance Analysis

The pipeline’s measured average runtime is below the 100 ms communication-delay constraint. Figure 5 breaks latency distributions down by module.

E. Real-time Performance Analysis

  • The pipeline averages 67.3 ms on a single NVIDIA 6000 Ada GPU, below the 100 ms communication-delay constraint.The paper describes this runtime as feasible for real-time onboard vehicle deployment.Source: E. Real-time Performance Analysis · p. 7
  • On a single NVIDIA 6000 Ada GPU, the pipeline averaged 67.3 ms—below the 100 ms communication-delay constraint—supporting feasibility for real-time onboard vehicle deployment.Source: E. Real-time Performance Analysis · p. 7
  • Figure 5 reports cumulative latency distributions for individual pipeline modules, with each module’s average shown alongside.Colored rectangles mark the 25th–75th percentile, whiskers the minimum and maximum, and red dots the median.Source: Figure 5 · p. 7

VI. CONCLUSION

CMP combines cooperative perception and trajectory prediction, using shared LiDAR data to improve object detection, tracking, and motion prediction. The framework includes latency-robust communication and prediction aggregation, but is not fully end-to-end because its tracker is non-differentiable.

VI. CONCLUSION

  • CMP integrates cooperative perception with trajectory prediction, sharing and fusing LiDAR point-cloud data to improve object detection, tracking, and motion prediction.The framework is presented as a cooperative pipeline for connected and automated vehicles.Source: §VI · p. 7
  • The framework includes a latency-robust cooperative prediction pipeline, communication bandwidth analysis, and an aggregation mechanism for motion prediction.The aggregation mechanism unifies predictions from connected vehicles.Source: §VI · p. 7
  • CMP is not fully end-to-end because its tracker is non-differentiable; future work will jointly optimize perception, prediction, and aggregation.The authors aim to minimize accumulative errors and cross-module inconsistencies, and plan to investigate multi-modal sensor fusion with heterogeneous CAVs.Source: §VI · p. 8
Loading 2403.17916v3…