PaperlayerView exact revision on arXiv

Source-linked AI summary

UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction

Lan Feng, Mohammadhossein Bahari, Kaouther Messaoud Ben Amor, Éloi Zablocki, Matthieu Cord, Alexandre Alahi

arXiv:2403.15098v3cs.CV

TL;DR

Vehicle trajectory prediction research has not sufficiently examined cross-domain generalization or how larger datasets affect performance, and combining datasets is complicated by their differences. UniTraj unifies datasets, models, and evaluation, then uses the framework to study these questions. Experiments show substantial transfer drops and improved performance with larger, more diverse data, including a leading nuScenes result.

  • Problem

    Cross-domain generalization and the effect of increasing dataset size on vehicle trajectory prediction remain underexplored, while dataset incompatibilities complicate multi-dataset study.

    Source: §1 · pp. 1–2

  • Method

    UniTraj unifies multiple vehicle trajectory datasets, prediction models, and evaluation approaches in one framework.

    Source: §1 · p. 3

  • Results

    Models show substantial performance drops across data sources, while larger, more diverse training data significantly improves performance and brings the combined-data MTR model to nuScenes leaderboard rank 1.

    Source: §1 · p. 3, §4.2 · p. 11

  • Takeaways & Limitations

    The study finds that larger, more diverse datasets boost model performance and generalization, and releases UniTraj as a tool for further exploration.

    Source: §5 · p. 14

    Fundamental discrepancies between datasets may persist despite standardizing data formats and aligning features.

    Source: Cross-city evaluation · p. 9

Abstract

from arXiv · show

Vehicle trajectory prediction has increasingly relied on data-driven solutions, but their ability to scale to different data domains and the impact of larger dataset sizes on their generalization remain under-explored. While these questions can be studied by employing multiple datasets, it is challenging due to several discrepancies, e.g., in data formats, map resolution, and semantic annotation types. To address these challenges, we introduce UniTraj, a comprehensive framework that unifies various datasets, models, and evaluation criteria, presenting new opportunities for the vehicle trajectory prediction field. In particular, using UniTraj, we conduct extensive experiments and find that model performance significantly drops when transferred to other datasets. However, enlarging data size and diversity can substantially improve performance, leading to a new state-of-the-art result for the nuScenes dataset. We provide insights into dataset characteristics to explain these findings. The code can be found here: https://github.com/vita-epfl/UniTraj

1 Introduction

UniTraj addresses the limited study of cross-domain generalization and data scaling by unifying datasets, prediction models, and evaluation. Experiments reveal substantial transfer performance drops, while larger and more diverse training data improves performance.

1 Introduction

  • Cross-domain generalization and the effect of increasing dataset size on performance remain underexplored research questions.Studying them across datasets is complicated by differing formats, resolutions, annotations, settings, and evaluation metrics.Source: §1 · p. 2
  • UniTraj integrates datasets, prediction models, and evaluation approaches in a unified framework for vehicle trajectory prediction research.It supports unified data structures, model integration, and evaluations including long-tail instances and data clusters.Source: §1 · p. 3
  • Experiments show a large performance drop when transitioning between data sources, while scaling dataset size and diversity significantly enhances performance without architectural modifications, and training on all existing datasets ranks 1st on the nuScenes public leaderboard.The unified dataset contains more than 2M samples, 1337 hours of data, and data from 15 different cities.Source: §1 · p. 3
  • The paper examines dataset characteristics to explain experimental findings and investigates model generalization across datasets and cities.Its contributions include exploring data scaling and establishing a new state-of-the-art nuScenes result.Source: §1 · p. 3

2 Previous work

Prior work provides real-world driving datasets and benchmarks, but datasets have often been developed independently. Existing vehicle trajectory research includes related tools and generalization studies, while UniTraj combines datasets, models, and evaluation methods in an open-source framework.

2 Previous work

  • Driving datasets developed independently have incompatibilities that complicate combining them, despite their distinct contributions to the field.Examples include lane graphs in Argoverse, scene variety in nuScenes, fine-grained information in Waymo, and unique roadways in Argoverse 2.Source: §2 · p. 4
  • Prior multi-dataset trajectory benchmarks include human-trajectory tools and ScenarioNet for vehicle planning, while UniTraj targets vehicle prediction with datasets, models, and evaluations.The authors describe UniTraj as the first open-source framework for vehicle trajectory prediction.Source: §2 · p. 4
  • Earlier generalization studies cover domains, datasets, cities, agent types, perception outputs, and adversarial situations, but some offer limited insight into generalization gaps.UniTraj reports broader cross-dataset and cross-city analyses, multi-dataset training, and dataset-characteristic analysis.Source: §2 · p. 4

3 UniTraj framework

UniTraj unifies dataset formats and features, adapts prediction models to the unified representation, and evaluates them through shared metrics and analyses. This structure supports cross-dataset training, evaluation, and dataset analysis.

3 UniTraj framework

  • The framework combines dataset unification, model adaptation, and shared evaluation to support cross-dataset training, evaluation, and dataset analysis.Source: §3 · p. 5

3.1 Unified data

  • Trajectory datasets differ in data organization and in features such as spatiotemporal resolution, range, and agent or map annotation taxonomies.The framework addresses both format and feature discrepancies.Source: §3.1 · p. 5

Unified data format:

  • ScenarioNet supplies a unified scenario format with HD maps and object annotations, and UniTraj extends its dataset conversion support to Argoverse 2.The format simplifies decoding different raw dataset formats and preparing batched training data.Source: Unified data format · p. 5

Unified data features:

  • Feature harmonization truncates trajectories to 8 seconds, standardizes agent inputs, and normalizes lane-point spacing to 0.5 meters.The unified agent inputs include 2D coordinates, velocity, and heading; map points can also be downsampled.Source: Unified data features · p. 6
  • Users can configure processing features, while multiprocessing and caching support efficient processing across four real-world datasets and 15 cities.The included datasets contain over 1337 hours of driving data.Source: Unified data features · p. 6

3.2 Unified models

  • UniTraj integrates recent prediction models, including MTR's intention priors and local movement refinement and Wayformer's latent-query approach to multidimensional inputs.The framework standardizes model outputs for shared evaluation and logging.Source: §3.2 · p. 7

3.3 Unified evaluation

General evaluations:

  • Fine-grained evaluation stratifies results by trajectory type and by Kalman difficulty, defined as ground-truth FDE against a linear Kalman filter prediction.The trajectory taxonomy does not distinguish motion dynamics such as acceleration versus deceleration within straight trajectories.Source: General evaluations · p. 8

4 Experiments

Experiments assess cross-dataset and cross-city transfer, data scaling, and dataset characteristics. Performance drops under domain transfer, while combining datasets improves results, especially on nuScenes.

Cross-dataset evaluation:

  • All three model architectures show significant performance declines when evaluated on datasets different from their training data.Across datasets, WOMD-trained models generalize best, followed by Argoverse 2- and then nuScenes-trained models.Source: Cross-dataset evaluation · p. 9, Table 2 · p. 10

Cross-city evaluation:

  • AutoBot performance drops when evaluated in cities other than its training city, with Pittsburgh-trained brier-minFDE rising from 2.4 in Pittsburgh to 2.7 in Boston and 3.5 in Singapore.The experiment uses 10K samples per city in nuPlan.Source: Cross-city evaluation · p. 10, Table 3

4.2 Scaling data to 2M trajectories.

  • Combined-data MTR improves over single-dataset training and ranks 1st on the nuScenes leaderboard at submission.The combined training set includes all datasets in UniTraj.Source: §4.2 · p. 11, Table 4 · p. 12
  • MTR and Wayformer benefit more than AutoBot from increased data on Argoverse 2 and nuScenes, a difference attributed to model capacity.MTR has 60.1 M parameters versus AutoBot’s 1.5 M.Source: §4.2 · p. 11

4.3 Analyzing the results

  • AutoBot’s average brier-minFDE across three datasets consistently decreases as training data increases from 20% to 100% of the combined set.The experiment progressively increases the number of training samples.Source: §4.2 · p. 11, Figure 2 · p. 12

Dataset analysis.

  • Straight trajectories account for 54–68% of samples, while WOMD has more turning trajectories and a more balanced difficulty distribution than nuScenes and Argoverse 2.The easiest difficulty range contains about 24% of WOMD samples, versus about 42% for nuScenes and 46% for Argoverse 2.Source: Dataset analysis · p. 12, Figure 3 · p. 13
  • With training sets controlled to 30K samples, WOMD still generalizes best; combined-data training improves all datasets, with diversity explaining transfer and scaling gains.The combined-data MTR also outperforms the nuScenes-trained model across trajectory types and improves on medium-difficulty samples.Source: Dataset analysis · pp. 13–14

5 Conclusions

The study finds substantial challenges in generalizing vehicle trajectory models across datasets and cities, while larger, more diverse datasets significantly boost performance and generalization. It also releases UniTraj as a tool for further research.

5 Conclusions

  • Models face considerable performance drops when encountering new datasets or cities.The conclusion identifies this as a central generalization challenge.Source: §5 · p. 14
  • Larger, more diverse datasets significantly boost model performance and generalization capabilities.The conclusion underscores data richness as important within the study's scope.Source: §5 · p. 14

Appendix

  • The appendix adds experiments, cross-checks model performance with other metrics, and provides further framework details.These materials are organized in Appendices A, B, and C, respectively.Source: Appendix · p. 20

A Further experimentation with UniTraj

UniTraj is presented as a flexible framework that opens research opportunities in trajectory prediction.

A Further experimentation with UniTraj

A.1 Continual learning on multiple datasets

Continual fine-tuning between WOMD and Argoverse 2 improves performance on the new dataset but reduces performance on the original one. The authors attribute this trade-off to the considerable domain gap.

A.1 Continual learning on multiple datasets

  • +0.92 Argoverse 2 improvement paired with -0.52 WOMD forgetting; WOMD fine-tuning yielded +2.06 WOMD improvement and -0.98 Argoverse 2 forgetting.Table 8 reports brier-minFDE on validation sets, with separate improvement and forgetting columns.Source: §A.1 · p. 20, Table 8 · p. 20
  • The authors attribute the adaptation trade-off to the considerable domain gap between WOMD and Argoverse 2.They present this outcome as an example of catastrophic forgetting during adaptation.Source: §A.1 · p. 20

A.2 Synthetic data

AutoBot pre-trained on procedural-generation synthetic data showed no substantial nuScenes improvement after fine-tuning, which the authors relate to synthetic-real domain differences and limited scenario diversity.

A.2 Synthetic data

  • The synthetic training set comprised 30,000 unique scenarios generated with MetaDrive procedural generation.Scenarios used traffic density of 15 vehicles per 100 meters and two roadblocks each.Source: §A.2 · p. 21
  • The procedural-generation pre-training did not produce substantial nuScenes performance improvement after fine-tuning.The authors attribute the limited effectiveness to the synthetic-real domain gap and the synthetic data's simplicity and limited diversity.Source: §A.2 · p. 21

A.3 Comparing different models’ performances using fine-grained evaluations:

Fine-grained evaluations expose model-specific weaknesses, while UniTraj also supports future directions including foundation-model training and dataset compression.

A.3 Comparing different models’ performances using fine-grained evaluations:

  • MTR generally outperforms AutoBot, but loses on straight-right, straight-left, and left-U-turn trajectories; AutoBot performs better in medium-difficulty scenarios.The fine-grained comparisons reveal weaknesses obscured by the coarser overall comparison.Source: §A.3 · p. 22

A.4 Other potential future directions

  • UniTraj can combine synthetic datasets with its real-data collection for foundation-model pretraining followed by real-world fine-tuning.The framework also supports integrating synthetic datasets alongside the field’s largest real-data collection, as characterized in the passage.Source: §A.4 · p. 22
  • UniTraj also supports coreset selection and dataset distillation to create compact trajectory-data subsets that retain most of the combined dataset’s information.Source: §A.4 · p. 22

B Complementary results

The complementary results check baseline integrations against reported performance and add comparisons for models on WOMD.

B Complementary results

  • The complementary-results section also reports integrated-model comparisons with official results and additional metrics; AutoBot results are referenced in the main-paper nuScenes leaderboard table.Source: B Complementary results · p. 23, §B.1 · p. 23

B.1 Cross-checking the performance of baselines with original papers

  • MTR integrated in UniTraj achieves performance similar to its original implementation on WOMD under the official setting and validation evaluation.The comparison uses one second of past trajectories, eight seconds of future trajectories, and the official WOMD validation set and evaluation tool.Source: §B.1 · p. 23, Table 12 · p. 23
  • Wayformer outperforms MTR on WOMD in minADE and minFDE, but its mAP comparison remains discrepant because details of the non-public implementation are unavailable.Source: §B.1 · p. 23

B.2 Reporting other metrics

Additional metrics confirm reduced performance on unfamiliar datasets and improvements from combined-dataset training, especially for nuScenes.

B.2 Reporting other metrics

  • Reduced performance on unfamiliar datasets persists across the considered model architectures and datasets, with WOMD-trained models ahead of Argoverse 2- and nuScenes-trained models.The expanded cross-dataset evaluation reports minFDE, minADE, and miss rate for models trained on each dataset and evaluated across the others.Source: §B.2 · pp. 23–24, Table 14 · p. 24, Table 15 · p. 24
  • Combined-dataset training yields notable performance improvements, especially for nuScenes.Source: §B.2 · p. 24

B.3 Experiments using pedestrian trajectories

The framework supports pedestrian trajectory prediction as well as vehicle prediction, and the appendix describes its training pipeline and settings.

B.3 Experiments using pedestrian trajectories

  • AutoBot pedestrian experiments show findings similar to those for vehicles; nuScenes is excluded because it does not officially support pedestrian trajectory prediction.Source: §B.3 · p. 25, Table 17 · p. 25

Training pipeline

Training settings

  • Experiments use eight A100 GPUs, with batch sizes of 256 for MTR and Wayformer and 128 for AutoBot; checkpoints are selected by validation min-brierFDE.Per-epoch training times are approximately 1 hour for MTR, 15 minutes for Wayformer, and 5 minutes for AutoBot.Source: Training settings · p. 25
Loading 2403.15098v3…