Learning Cooperative Trajectory Representations for Motion Forecasting

• Prior work largely fuses single-frame cooperative perception, which may trade off states across views and underuse agents’ historical motion and interaction context.The paper motivates considering observations from each view holistically instead.Source: §1 · pp. 1–2
• V2X-Graph independently represents cooperative trajectories’ motion and interaction features, then uses customized relative spatial-temporal encodings and graph-guided association for interpretable, end-to-end fusion.The framework aims to enhance agents’ historical representations for cooperative motion forecasting.Source: §1 · p. 2, Figure 1 · p. 2
• V2X-Traj is the first public, real-world V2X motion-forecasting dataset to include multiple autonomous vehicles and infrastructure in every scenario.The dataset broadens the evaluation beyond V2I cooperation.Source: §1 · p. 2
• V2X-Graph achieves state-of-the-art results on both V2X-Seq and V2X-Traj.The paper presents V2X-Traj as a public, real-world dataset for V2X motion forecasting.Source: §1 · p. 2

• Cooperative autonomous-driving datasets and methods have mostly targeted perception, and cooperative motion forecasting has not been well studied.A recent effort supplies historical infrastructure perception to improve forecasting [51].Source: §2 · p. 3
• V2X-Graph replaces perception supplementation followed by forecasting with end-to-end fusion of cooperative trajectories’ motion and interaction contexts.The paper also presents V2X-Traj, a public real-world dataset including both V2I and V2V cooperation in every scenario.Source: §2 · p. 3
• Motion-forecasting methods have moved from rasterized scenes [34] [21] to vectorized agent and map representations [10], using pooling [14] [10] [35], convolution [25] [52] [5], and attention [26] [33] [19] [40].Source: §2 · p. 3
• Earlier homogeneous GNNs represent scenes uniformly [30] [10] [23] [25] [13] [11], while later heterogeneous GNNs distinguish feature types [16] [32] [19]; V2X-Graph uses heterogeneous edge encodings and interpretable graph link prediction.Source: §2 · p. 3

• The scenario is represented by multi-source historical trajectories and vector maps, with other trajectories potentially received from infrastructure and cooperative vehicles.Trajectory attributes include tracking identity, location, heading, bounding box, and agent type; lane segments encode map geometry and attributes.Source: §3 · pp. 3–4
• Each vectorized lane segment connects neighboring centerline sample points and carries attributes such as location and road type.Source: §3 · p. 4
• The task outputs K=6 future trajectories for the target agent, evaluates the best one using minADE, minFDE, and MR, and treats lower values as better.Source: §3 · p. 4
• The stated challenges are representing the cooperative scenario effectively and efficiently using valuable information from redundant cooperative trajectories.Source: §3 · p. 4

4.1 Scene Representation with Graph

• Trajectory and lane features become graph nodes, while spatial-temporal and relative-spatial relationships form edges; differential motion, temporal dependencies, and masked missing frames shape trajectory representations.Lane segments are represented by agent-centric relative coordinates; trajectory spatial-temporal features support identification and motion correlation measurement.Source: §4.1 · p. 4

4.2 Feature Fusion with Interpretable Graph

• Agent-Lane fusion queries relevant lane interactions with agent motion features, while Cooperative Interaction represents dense cross-view agent interactions using associations and relative spatial correlations.Cooperative Interaction considers interactions among trajectories observed in the current frame.Source: §4.2 · pp. 5–6
• Interpretable Association links same-agent trajectories across views, while Motion Fusion aggregates associated motion using temporal-spatial correlations and pseudo-label matching supervision.The pseudo-label generator collects trajectory matches across the historical horizon for training.Source: §4.2 · pp. 5–6

4.3 Multimodal Future Decoder

• A prediction head aggregates intermediate features to predict multimodal future trajectories represented by Laplacian mixture distributions.The decoder predicts distinct agents from the ego view at inference, guided by interpretable associations.Source: §4.3 · p. 6

4.4 Training Losses

• End-to-end training combines cross-entropy for graph link prediction with regression and classification losses for motion forecasting.Source: §4.4 · p. 7

5.1 Experimental Setup

• Evaluations cover V2X-Seq’s 51,146 V2I scenarios and V2X-Traj’s 10,102 challenging-intersection scenarios, each including two intelligent vehicles and infrastructure.V2X-Seq uses five seconds of observations to predict five seconds; V2X-Traj uses four seconds to predict four seconds.Source: §5.1 · p. 7, §5 · p. 7

5.2 Main Results

• V2X-Graph improves V2X-Seq minADE by −0.07, minFDE by −0.19, and MR by −5% against the reported cooperative comparison.Perception completion improves HiVT and V2X-Graph but degrades DenseTNT, whereas trajectory feature fusion improves V2X-Graph.Source: §5.2 · p. 7, Table 1 · p. 8
• On V2X-Traj V2V&I, V2X-Graph achieves the best performance, with −0.22 minADE, −0.43 minFDE, and −6% MR versus compared methods.It outperforms compared methods in all cooperative settings and remains competitive in the vehicle-only task.Source: Table 2 · p. 8, §5.2 · p. 8

5.3 Ablation Study

• Component ablations show that historical-agent motion features and interactions with surrounding agents each contribute to forecasting performance.The study alternately removes components of the interpretable graph to assess their contributions.Source: §5.3 · p. 8, Table 3 · p. 8
• Masking MFG cooperative motion features yields −0.08 minADE; masking ALG/CIG interaction features affects long-term prediction (−0.59 minFDE, −7% MR).ALG and CIG fuse interaction features from lane segments and cooperative trajectories.Source: §5.3 · p. 8, Table 4 · p. 8
• Association-based interpretable fusion improves performance at low computational cost; fully connected feature fusion shows no obvious positive effect over the ego setting.Table 5 defines interpretable fusion as aggregating motion and interaction features through associations.Source: §5.3 · pp. 8–9, Table 5 · p. 9
• As disturbed pseudo-label proportions rise, forecasting performance falls; the results indicate contributions from pseudo-label supervision and interpretable fusion.Figure 5 shows target-agent predictions in cooperative intersection scenarios, with ground truth in red and multimodal predictions in green.Source: Figure 4 · p. 9, §5.3 · p. 9, Figure 5 · p. 9

6 Conclusion and Limitation

• V2X-Graph provides interpretable, end-to-end trajectory feature fusion for cooperative forecasting, and experiments on V2X-Seq and V2X-Traj demonstrate its effectiveness.Source: §6 · p. 9
• Compared with single-frame methods, trajectory feature fusion mitigates errors from perception completion and improves agent motion and interaction representations.Source: §6 · p. 9
• Performance still relies on tracking quality from each view; jointly optimizing perception through forecasting is identified as future work.Source: §6 · p. 9

Appendix

• The appendix provides training-loss details, a comprehensive V2X-Traj dataset description, additional experiments, implementation details, and qualitative V2X-Seq results.Source: Appendix · p. 14

A Details of Training Loss

• The framework is trained end to end with two optimization components.Source: A Details of Training Loss · p. 14
• Cross-entropy distillation trains cross-view trajectory associations in adjacency matrix A using pre-pruned pseudo-labels to mitigate class imbalance and exclude non-overlapping boxes or differing agent types.Source: A Details of Training Loss · p. 14
• For forecasting, the method selects the mode with minimum average L2 distance to ground truth among K candidates, using cross-entropy for selection and negative log-likelihood for regression.Source: A Details of Training Loss · p. 14
• The regression likelihood uses a Laplace distribution for the best-predicted trajectory's coordinates and corresponding uncertainties.Source: A Details of Training Loss · p. 14
• The final training loss sums the distillation, regression, and classification losses.Source: A Details of Training Loss · p. 14

B Details of V2X-Traj Dataset

• V2X-Traj is presented as the first real-world V2X cooperative motion forecasting dataset.Source: B Details of V2X-Traj Dataset · p. 14
• The section presents a comparison with public motion forecasting datasets in Table 6.Source: B Details of V2X-Traj Dataset · p. 14
• Further dataset details cover composition, collection and annotation, and additional visualizations in appendices B.1–B.3.Source: B Details of V2X-Traj Dataset · p. 14

B.1 Dataset Composition

• V2X-Traj contains 10,102 scenarios, split into 6,062 training, 2,020 validation, and 2,020 test scenarios.Source: §B.1 · p. 14
• Each scenario includes three independent trajectory sets from two autonomous vehicles and an infrastructure-side perception device.Source: §B.1 · p. 14
• Each trajectory records timestamp, unique ID, agent type, location, a 7-dimensional detection bounding box, heading, and velocity.Source: §B.1 · p. 14
• Vector maps include lanes, crosswalks, stoplines, and junctions, with lane centerlines, boundaries, and semantic attributes.Source: §B.1 · p. 14
• V2X-Traj supports V2V and broader V2V&I forecasting with real-world cooperative trajectories, vector maps, and real-time traffic lights.Source: Table 6 · p. 15
• Traffic-light data are recorded at 10 Hz and include timestamp, location, direction, lane ID, color status, and remaining time.Source: §B.1 · p. 15
• The dataset provides trajectories from eight agent classes, with statistics covering total trajectory counts and lengths per class.Source: §B.1 · p. 15, Table 7 · p. 15

B.2 Data Collection and Annotation

• At 28 Beijing intersections, infrastructure uses 4–6 pairs of 300-beam LiDARs and high-resolution cameras, while each autonomous vehicle uses one 40-beam LiDAR and six cameras.The collected views are stored independently; overlap in the perception ranges of two autonomous vehicles and infrastructure defines cooperative scenarios.Source: §B.2 · p. 15
• Mining 8-second segments by ego-trajectory behaviors produced 10,102 scenarios, each with one target trajectory and the remaining trajectories assigned as others.Trajectory scoring considers turning, speeding up, slowing down, and lane changing.Source: §B.2 · p. 15

B.3 Additional Dataset Visualization

• Each V2X-Traj scenario combines trajectories from the ego vehicle, cooperative infrastructure, and a cooperative autonomous vehicle.The authors state that these cooperative trajectories add information available to the ego vehicle.Source: §B.3 · p. 16

C.2 Effectiveness of Cooperative Representation

• On V2X-Seq, the MFG reduces minADE by 0.15, while ALG and CIG yield maximum reductions of 0.20 in minFDE and 3% in MR when infrastructure trajectories are masked out.MFG integrates associated-trajectory motion features; ALG and CIG fuse lane-segment and cooperative-trajectory interaction features through compact spatial encoding.Source: §C.2 · p. 17, Table 9 · p. 17

C.3 Scalability Study

• As training data increases, both vehicle-only and cooperative settings improve, with V2X-Graph showing greater advantages in cooperation.The evaluation trains on subsets of V2X-Seq and tests on its full validation set.Source: §C.3 · p. 17, Figure 8 · p. 17

C.4 Robustness of Latency and Data Loss

• With 50% infrastructure data loss, V2X-Graph still outperforms the compared methods; dropping the latest one or two frames causes little degradation after interpolation.Forecasting performance decreases as the data-loss ratio rises.Source: §C.4 · pp. 17–18, Table 11 · p. 18, Table 10 · p. 18

C.5 Parameter Size and Inference Cost Comparison

• V2X-Graph has parameter size comparable to other forecasting methods and is faster at inference than the compared vanilla methods on one NVIDIA GTX 4090.The authors attribute the inference speed to synchronous temporal state modeling and integration.Source: Table 12 · p. 18, §C.5 · p. 18, Table 13 · p. 18

D Additional Implementation Details

• The authors reimplemented HiVT, DenseTNT [13], and HDGT [19] for comparisons, using larger models or dataset-specific training settings.The HiVT model uses hidden size 128; HDGT uses hidden size 256 and 64 attention heads.Source: D Additional Implementation Details · pp. 18–19

E Additional Qualitative Results

• Appendix E presents V2X-Seq visualizations of interpretable associations and comparisons between V2X-Graph and another method.The association visualizations are described in E.1, and method comparisons in E.2.Source: E Additional Qualitative Results · p. 19

E.1 Visualizations of Interpretable Association

• The association visualizations show trajectories linked across ego and infrastructure views, with infrastructure trajectories supplying additional information for end-to-end representation learning.The figure highlights a subset of the extra infrastructure-side information that can be used.Source: §E.1 · p. 19

E.2 Qualitative Results on V2X-Seq

• On three V2X-Seq challenge scenarios, V2X-Graph compares favorably with HiVT using PP-VIC [51], particularly for long-range intentions.Figure 10 marks historical target trajectories in yellow, ground truth in red, and predictions in green.Source: §E.2 · p. 19, Figure 10 · p. 21

NeurIPS Paper Checklist

• The authors report that code is in the supplemental material and that code and dataset will be released with instructions once accepted.They also state that the new dataset is documented and that local government granted permission for its release.Source: NeurIPS Paper Checklist · pp. 24, 27
• Formal error bars are omitted because they are computationally expensive, although the authors report multiple runs and representative results.Source: NeurIPS Paper Checklist · p. 24
• The authors hope V2X-Graph and V2X-Traj will facilitate further development of cooperative motion forecasting.Source: NeurIPS Paper Checklist · p. 25