Source-linked AI summary
Co-MTP: A Cooperative Trajectory Prediction Framework with Multi-Temporal Fusion for Autonomous Driving
Xinyu Zhang, Zewei Zhou, Zhaoyi Wang, Yangjie Ji, Yanjun Huang, Hong Chen
TL;DR
Incomplete multi-agent histories and the need to account for surrounding vehicles’ reactions to ego plans leave temporal V2X fusion for prediction and planning underexplored. Co-MTP fuses V2X information across history and future domains using a heterogeneous graph framework. It achieves state-of-the-art performance on V2X-Seq, and both forms of fusion are reported to benefit prediction and planning.
Problem
Different sensors and occlusion can leave multi-agent histories incomplete, while prediction must account for surrounding vehicles’ reactions to ego planning.
Source: §I · p. 1Method
Co-MTP uses a heterogeneous graph to fuse multi-agent history features and infrastructure predictions with ego planning across history and future domains.
Source: §I · pp. 1–2Results
Co-MTP achieves state-of-the-art performance on V2X-Seq, and the authors report that history and future fusion benefit prediction and planning.
Source: §I · p. 2Takeaways & Limitations
The paper concludes that planning-oriented prediction should capture interactions in both history and future domains, including other vehicles’ actions.
Source: §V · p. 6Communication noise reduces Co-MTP’s accuracy, and the authors identify improving noise robustness as future work.
Source: D. Robustness Assessment · p. 6
Abstract
from arXiv · showhide
Vehicle-to-everything technologies (V2X) have become an ideal paradigm to extend the perception range and see through the occlusion. Exiting efforts focus on single-frame cooperative perception, however, how to capture the temporal cue between frames with V2X to facilitate the prediction task even the planning task is still underexplored. In this paper, we introduce the Co-MTP, a general cooperative trajectory prediction framework with multi-temporal fusion for autonomous driving, which leverages the V2X system to fully capture the interaction among agents in both history and future domains to benefit the planning. In the history domain, V2X can complement the incomplete history trajectory in single-vehicle perception, and we design a heterogeneous graph transformer to learn the fusion of the history feature from multiple agents and capture the history interaction. Moreover, the goal of prediction is to support future planning. Thus, in the future domain, V2X can provide the prediction results of surrounding objects, and we further extend the graph transformer to capture the future interaction among the ego planning and the other vehicles' intentions and obtain the final future scenario state under a certain planning action. We evaluate the Co-MTP framework on the real-world dataset V2X-Seq, and the results show that Co-MTP achieves state-of-the-art performance and that both history and future fusion can greatly benefit prediction.
I. INTRODUCTION
V2X can address incomplete histories and occlusion, but using its temporal information for prediction and planning remains underexplored. Co-MTP fuses history and future information through a heterogeneous graph framework, and is reported to achieve state-of-the-art performance on V2X-Seq.
I. INTRODUCTION
- Incomplete histories arise when different sensors have different capabilities and coverage, and simple association-and-stitching can miss gaps shared across agents.The paper identifies integration of incomplete multi-agent histories as a challenge.Source: §I · p. 1
- Co-MTP uses V2X information across history and future domains, extending cooperative prediction beyond single-frame perception.Infrastructure shares history and prediction results with the ego vehicle.Source: §I · p. 1, Figure 1 · p. 2
- A heterogeneous graph and Transformer fuse history features from multiple agents to address incomplete trajectories.The framework also uses a heterogeneous graph to capture history interactions.Source: §I · p. 1
- The future-domain graph captures interactions between ego planning and other vehicles’ actions derived from infrastructure predictions.The authors motivate this with the need to reason about surrounding vehicles’ reactions to ego actions.Source: §I · pp. 1–2
- Co-MTP achieves state-of-the-art performance on V2X-Seq, with both history and future fusion reported to benefit prediction and planning.The paper evaluates the framework on a real-world dataset.Source: §I · p. 2
II. RELATED WORK
Prior cooperative-prediction work uses V2X but leaves comprehensive temporal fusion across domains underexplored; planning-informed methods also need to account for other vehicles’ future intentions. Co-MTP’s vehicle-side model uses infrastructure prediction results alongside history information.
II. RELATED WORK
- Cooperative prediction work includes end-to-end models and graph-based representations, but fully exploiting V2X temporal information across domains remains underexplored.The cited approaches include V2VNet, V2XPnP, and V2X-Graph.Source: §II · p. 2
- Using only ego planning in the future domain can overestimate its road priority, so prediction should also integrate other vehicles’ future intentions.The related-work discussion connects this concern to interaction features across history and future domains.Source: §II · p. 2
III. METHODOLOGY
- The vehicle-side prediction model takes infrastructure-side prediction results as future-domain input in addition to history information.The paper states that AVs and infrastructure can also operate prediction individually.Source: §III · pp. 2–3
A. Problem Definition
The task uses observations from an AV and infrastructure that may be incomplete, together with map data and a simulated ego planning trajectory, to predict surrounding objects’ future trajectories. Predictions may contain multiple modes to represent uncertainty.
A. Problem Definition
- A scenario contains objects observed by the AV, infrastructure, or both, with object features including bounding boxes and object types.The task describes observations over a time interval from multiple viewpoints.Source: A. Problem Definition · p. 3
- The inputs include history trajectories, AV future ground truth, and map data; the future ground truth is interpolated with a quintic curve to simulate planning.Map data includes roads and signals represented as polylines or polygons.Source: A. Problem Definition · p. 3
- Occlusion and other factors can leave some trajectory frames unobserved, represented by a validity mask for each object.The mask indicates whether data exists at a given time step.Source: A. Problem Definition · p. 3
- The prediction target is the future trajectories of surrounding objects, and outputs may include multiple modes with corresponding probabilities.The paper frames the task as learning a model that outputs those trajectories.Source: A. Problem Definition · p. 3
B. Scene Representing with Graph
Co-MTP represents objects and map elements in a heterogeneous graph, with edges encoding their relationships. It preprocesses multi-view trajectories and encodes nodes and edges in local coordinate systems before graph-based fusion.
B. Scene Representing with Graph
- Objects and map elements become typed graph nodes, with edges representing relationships; the graph limits each node’s edge count based on distance.Object-node distances use last-observed positions, while map-node distances use distance to the polyline or polygon.Source: B. Scene Representing with Graph · p. 3
- Before fusion, Co-MTP stitches AV and infrastructure history trajectories using high-detection-score data and reorganizes their masks.The original infrastructure histories remain encoded as graph nodes, and infrastructure predictions of other objects are included in the future domain alongside AV planning.Source: B. Scene Representing with Graph · p. 3
C. Relative Spatial-Temporal Encoding
- Each node uses a local coordinate system based on its final observed position and heading, or the map element’s center and direction.This normalized representation is used for subsequent node computations.Source: C. Relative Spatial-Temporal Encoding · p. 3
- Object positions and attributes are encoded with MLPs, temporal features with self-attention, and map and signal features with PointNet and GRUs.The history mask is also input to temporal encoding.Source: C. Relative Spatial-Temporal Encoding · p. 3
- Edges encode source-to-node coordinate transformations with a shared MLP, followed by a type-specific MLP to generate initial edge features.Source: C. Relative Spatial-Temporal Encoding · p. 3
D. Cross-Temporal and Cross-Agent (CTCA) Fusion
Co-MTP uses a cross-temporal graph and STFA to fuse multi-view information across history and future domains. A Transformer updates node and edge features to capture interactions among agents and planning-related nodes.
D. Cross-Temporal and Cross-Agent (CTCA) Fusion
- STFA integrates information from different views and temporal domains through a cross-temporal graph with a K-layer Transformer that updates node and edge features.AV-view objects and infrastructure-view objects form independent nodes; AV planning and infrastructure predictions are linked to historical nodes to model future interaction.Source: D. Cross-Temporal and Cross-Agent… · pp. 3–4, Figure 2 · p. 4
- Infrastructure nodes influence interpolated vehicle-side nodes, reusing infrastructure information and learning relationships between the two views.Future nodes also contribute information when agent nodes aggregate features.Source: D. Cross-Temporal and Cross-Agent… · p. 4
- Node features are updated by a feed-forward network whose parameters depend on node type, while edge updates combine encoded context with previous edge features through an MLP.The edge update retrieves outputs from the edge encoding stage before combining them with the previous layer’s edge features.Source: D. Cross-Temporal and Cross-Agent… · p. 4
E. Multimodal Decoder
The multimodal decoder produces multiple trajectory possibilities and probabilities for each target agent. It refines mode features, models relationships among modes, and applies regression and classification with dynamic constraints.
E. Multimodal Decoder
- The decoder generates K trajectory modes by replicating final Transformer embeddings and refining node information with cross-attention before an FFN outputs the modes.A self-attention layer then captures competitive relationships among the K modes for each object.Source: E. Multimodal Decoder · p. 4
- The decoder classifies K modalities and regresses trajectories, reintroducing original node features and applying dynamic constraints to reduce trajectory drift.Regression output and refined features are concatenated and passed through an MLP, with temporal processing by refined CNNs.Source: E. Multimodal Decoder · p. 4
- The classification and regression outputs have shapes N × K and N × K × T × 2, respectively, for N target agents and 2-D coordinates.The training loss combines regression and classification heads; the selected mode has the smallest average deviation from ground truth, using smooth L1 regression loss.Source: E. Multimodal Decoder · p. 4
- The Co-HTTP baseline first stitches AV-perceived and infrastructure history trajectories, then fuses them through learning.Source: E. Multimodal Decoder · p. 5
A. Experimental Setup
Experiments use V2X-Seq trajectory fragments and standard displacement and missing-rate metrics. The setup slices fragments into samples and downsamples trajectories for training.
A. Experimental Setup
- V2X-Seq contains over 50,000 vehicle-infrastructure cooperation trajectory fragments, each lasting 10 seconds at 10 Hz.Source: A. Experimental Setup · p. 5
- Each fragment is sliced into 20 samples, each with 3 seconds of history and 5 seconds of future trajectories.Source: A. Experimental Setup · p. 5
- Evaluation uses minADE, minFDE, and MR; training trajectories are downsampled to 5 Hz, with six prediction modes and a six-layer Transformer encoder.Training uses 45 epochs with batch size 24 on four NVIDIA 4090 GPUs.Source: A. Experimental Setup · p. 5
B. Compared Methods
On V2X-Seq, Co-MTP ranks first across the reported prediction metrics, and ablations indicate benefits from more complete history fusion and future fusion. The compared methods include trajectory stitching and history-only graph fusion.
B. Compared Methods
- PP-VIC stitches AV and infrastructure trajectories before applying TNT or HiVT, while V2X-Graph fuses history trajectories only.PP-VIC uses CBMOT for stitching; V2X-Graph focuses on history-domain fusion.Source: B. Compared Methods · p. 5
- 35% reduction in minADE versus V2X-Graph; Co-MTP ranks first across minADE/minFDE/MR on V2X-Seq.The comparison reports the benchmark ranking and minADE reduction for Co-MTP.Source: B. Compared Methods · p. 5
- The authors report that heterogeneous graph networks and cross-temporal fusion strategies preliminarily benefit cooperative prediction.Source: B. Compared Methods · p. 5
C. Ablation Study
- More comprehensive multi-view histories reduce prediction error; full-infrastructure learned fusion outperforms variant 3, while adding ego intent and others’ futures further improves Co-HTTP and Co-MTP.The ablation compares history variants by vehicle-side and infrastructure data availability and fusion method; future-fusion models outperform single-history-dimension models.Source: C. Ablation Study · p. 5
D. Robustness Assessment
Robustness tests introduce communication noise and delay, and report strong robustness across the tested model variants. Co-MTP accuracy declines under noise, while temporal information is credited with mitigating delay effects.
D. Robustness Assessment
- Under 0.2-meter positional deviation and 0.5-second delay, all tested models exhibit strong robustness to communication noise and delay.The tests include models using history or future communication and both stitching and learning methods.Source: D. Robustness Assessment · p. 5, Table III · p. 6, Table IV · p. 6
- Co-MTP’s delay robustness is attributed to leveraging temporal information to mitigate delay impact.Source: D. Robustness Assessment · p. 6
- Communication noise lowers Co-MTP accuracy through compounded errors in history and future information, though it remains more accurate than other models.The authors identify improving noise robustness as a direction for future research.Source: D. Robustness Assessment · p. 6
E. Qualitative Result
Co-MTP combines history and future fusion for planning-oriented prediction; its V2X-Seq qualitative examples show road-constrained trajectories and a turn delayed by another vehicle’s straight-ahead intention.
E. Qualitative Result
- In the illustrated turn scenario, another vehicle’s straight-ahead intention leads the target vehicle to wait, while predicted trajectories adhere to road structure and traffic safety constraints.The examples also include following, highway driving, and accelerating through an intersection.Source: E. Qualitative Result · p. 6
- Figure 3 marks AVs in red, targets in orange, objects in blue, predicted trajectories in green, history ground truth in blue, and future ground truth in brown.Source: Figure 3 · p. 6
V. CONCLUSION
- Co-MTP achieves state-of-the-art performance on V2X-Seq, and the authors report that history and future fusion benefit prediction.The framework uses history fusion to address incomplete trajectories and future interaction to capture ego planning alongside other vehicles’ actions.Source: §V · p. 6
- Co-MTP is presented as a cooperative trajectory prediction framework with multi-temporal fusion across history and future domains.Its history-domain heterogeneous graph uses a multi-layer Transformer to fuse information from multiple agents; the future domain captures interaction between ego planning and other vehicles’ actions.Source: §V · p. 6