PaperlayerView exact revision on arXiv

Source-linked AI summary

Leapfrog Diffusion Model for Stochastic Trajectory Prediction

Weibo Mao, Chenxin Xu, Qi Zhu, Siheng Chen, Yanfeng Wang

arXiv:2303.10895v1cs.CV

TL;DR

Stochastic trajectory prediction needs expressive, diverse future-trajectory distributions, but standard diffusion models are too slow for real-time use because they require many denoising steps. LED learns a trainable leapfrog initializer that skips most denoising while allocating correlated samples for diversity, achieving state-of-the-art results across four datasets and real-time inference.

  • Problem

    Stochastic trajectory prediction must model multiple possible future trajectories, while diffusion-based predictors require many denoising steps that hinder real-time inference.

  • Method

    LED uses a trainable leapfrog initializer to directly model an expressive denoised distribution, generate correlated diverse samples, and refine it with only a few denoising steps.

  • Results

    LED achieves state-of-the-art performance across four real-world datasets; on NBA, it improves ADE/FDE by 15.6%/13.4% and reduces prediction time from ∼886ms to ∼46ms versus MID.

  • Takeaways & Limitations

    LED provides precise and diverse trajectory predictions with substantially faster inference, satisfying real-time prediction needs within the evaluated trajectory datasets.

  • Takeaways & Limitations

    The reported acceleration partly depends on trajectory data having relatively low dimensionality and distributions that are easier to learn than those of image or video data.

Abstract

from arXiv · show

To model the indeterminacy of human behaviors, stochastic trajectory prediction requires a sophisticated multi-modal distribution of future trajectories. Emerging diffusion models have revealed their tremendous representation capacities in numerous generation tasks, showing potential for stochastic trajectory prediction. However, expensive time consumption prevents diffusion models from real-time prediction, since a large number of denoising steps are required to assure sufficient representation ability. To resolve the dilemma, we present LEapfrog Diffusion model (LED), a novel diffusion-based trajectory prediction model, which provides real-time, precise, and diverse predictions. The core of the proposed LED is to leverage a trainable leapfrog initializer to directly learn an expressive multi-modal distribution of future trajectories, which skips a large number of denoising steps, significantly accelerating inference speed. Moreover, the leapfrog initializer is trained to appropriately allocate correlated samples to provide a diversity of predicted future trajectories, significantly improving prediction performances. Extensive experiments on four real-world datasets, including NBA/NFL/SDD/ETH-UCY, show that LED consistently improves performance and achieves 23.7%/21.9% ADE/FDE improvement on NFL. The proposed LED also speeds up the inference 19.3/30.8/24.3/25.1 times compared to the standard diffusion model on NBA/NFL/SDD/ETH-UCY, satisfying real-time inference needs. Code is available at https://github.com/MediaBrain-SJTU/LED.

1. Introduction

Stochastic trajectory prediction must represent multiple possible futures, but diffusion models face a tension between expressive distributions, sample diversity, and real-time inference. LED addresses this with a trainable leapfrog initializer and reports strong performance across four datasets.

  • Motivation: Stochastic trajectory prediction models distributions of multiple possible future trajectories rather than a single outcome.This supports applications where future agent behavior is indeterminate.
  • LED: LED uses a trainable leapfrog initializer to estimate an expressive denoised distribution, replacing many traditional denoising steps with a few refinement steps.The forward diffusion process remains standard, while the reverse process initializes correlated diverse samples before refinement.
  • LED: LED generates correlated sample positions from shared social-temporal features, enabling adaptive sample allocation and prediction diversity.The initializer estimates mean trajectory, variance, and normalized sample positions through trainable modules.
  • Experiments: 15.6%/13.4% ADE/FDE improvement and inference reduction from ∼886ms to ∼46ms are reported on NBA compared with MID.Experiments cover NBA, NFL, SDD, and ETH-UCY, where the method is reported to achieve state-of-the-art performance and around 20× acceleration over standard diffusion.

2. Related Work

Prior stochastic trajectory predictors use GANs, CVAEs, normalizing flows, and diffusion models to represent future-trajectory distributions. LED targets the remaining inference bottleneck by replacing many denoising steps with a trainable initializer.

  • Generative trajectory prediction: Stochastic trajectory prediction research has used GANs, CVAEs, and conditional normalizing flows to model future-trajectory distributions.These approaches differ in how they represent or relax the prior distribution.
  • Diffusion models: Diffusion models reconstruct data distributions through denoising and have been applied to probabilistic time-series forecasting and trajectory prediction.MID is identified as the first diffusion model built for trajectory prediction.
  • Fast sampling: Standard diffusion models use hundreds of denoising steps, limiting their suitability for real-time applications.DDIM and knowledge-distillation approaches accelerate sampling by reducing or compressing denoising operations.
  • LED: LED uses a trainable leapfrog initializer to initialize an expressive distribution and replace many former denoising steps.The approach is designed to obtain much faster inference speed while retaining distributional representation ability.

3. Background

The trajectory-prediction formulation conditions future-trajectory distributions on observed ego and neighboring-agent movements. Standard diffusion learns these distributions through noisy forward diffusion and iterative reverse denoising, creating an efficiency–representation trade-off.

  • Problem formulation: The observed ego trajectory contains Tp timestamps of 2D coordinates, while neighboring trajectories are represented for L surrounding agents.The future ego trajectory contains Tf future timestamps of 2D coordinates.
  • Problem formulation: Stochastic trajectory prediction trains a model to generate a future-trajectory distribution conditioned on the ego and neighboring agents’ past trajectories.Multiple predictions are used to capture the indeterminacy of future trajectories.
  • Problem formulation: The learning objective selects predicted trajectories from the model distribution to minimize their distance from the ground-truth future trajectory.The displayed objective combines distributional generation with best-match prediction error.
  • Standard diffusion: Standard diffusion adds noise to ground-truth futures and learns conditional denoising operations that recover future trajectories through successive steps.Training uses the noisy trajectory at each diffusion step as supervision for the corresponding denoised trajectory.
  • Standard diffusion: At the final stage, standard diffusion produces K denoised trajectories initialized as independent samples from a normal distribution and refined iteratively.The forward-process steps are not used during inference.
  • Efficiency trade-off: LED introduces a trainable initializer that replaces many small denoising steps with one powerful leapfrog step to accelerate inference without losing representation ability.The leapfrog step is denoted by τ.

4. Leapfrog Diffusion Model

LED replaces many early diffusion denoising steps with a trainable leapfrog initializer that directly models an expressive denoised trajectory distribution, then applies only the remaining steps. Its reparameterized, jointly generated correlated samples support diverse predictions while preserving the standard diffusion process and conditioning on past trajectories.

  • System Architecture: LED leapfrogs the initial Γ−τ denoising steps by directly modeling the τth denoised distribution, then refines sampled trajectories with the remaining τ steps.The initializer is intended to replace a large number of small denoising steps without losing representation ability.
  • System Architecture: The inference pipeline generates K initialized trajectories from a trainable leapfrog initializer and iteratively denoises each trajectory to obtain final predictions.The initializer learns statistics and generates correlated samples through reparameterization.
  • Leapfrog Initializer: The initializer reparameterizes the τth denoised distribution into a mean trajectory, global variance, and normalized sample positions.Three trainable modules estimate these components from past trajectories and neighboring-agent information.
  • Leapfrog Initializer: K normalized predictions are generated simultaneously from shared features, with variance estimation helping allocate trajectories to capture multiple modes.The sample prediction module also uses the estimated standard deviation as input.
  • Network Design: Each initializer module combines social encoding, temporal encoding, and feature fusion, while denoising uses a context encoder and noise estimator conditioned on past trajectories.The denoising module applies the standard denoising step after estimating the noise to reduce.
  • Training Objective: Explicit supervision of the initializer is avoided because denoising M≫K samples for statistics would require approximately Γ−τ≈Γ steps and incur unacceptable training time and storage costs.The passage gives approximately 6 days per epoch on NBA as an example.

5. Experiments

Experiments on four trajectory datasets show that LED achieves strong prediction performance while reducing diffusion inference cost. Ablations and visualizations support the roles of the leapfrog initializer, variance estimation, correlated sampling, and the selected leapfrog step.

  • Datasets and metrics: Experiments cover NBA, NFL, SDD, and ETH-UCY using minADEK and minFDEK to evaluate trajectory prediction.Sports datasets are also evaluated at different timestamps.
  • Comparison with SOTA methods: 15.6%/13.4% ADE/FDE improvement is achieved over MID on NBA at 4.0s, reducing ADE/FDE from 0.96/1.27 to 0.81/1.10.Performance improvement over previous methods increases with prediction timestamps.
  • Comparison with SOTA methods: 23.7%/21.9% ADE/FDE improvement is achieved over MID on NFL at 3.2s, reducing ADE/FDE from 1.14/1.92 to 0.87/1.50.The model outperforms the compared baselines at all reported timestamps.
  • Comparison with SOTA methods: LED reduces FDE from 11.85 to 11.66 on SDD and from 0.35 to 0.33 on ETH-UCY compared with the reported strongest baselines.The ETH-UCY result is a 5.7% improvement, with best or second-best performance on most subsets.
  • Ablation studies: τ = 5 provides the best performance on NBA, balancing initializer representation ability against inference time.Too-small τ targets overly sophisticated distributions, while too-large τ wastes inference time after capturing the denoised distribution.
  • Ablation studies: Mean estimation, variance estimation, and sample prediction each contribute to prediction accuracy, while the initializer remains effective when K is small.The initializer produces stable results across prediction numbers.
  • Ablation studies: LED outperforms PD and DDIM under similar inference time, which the experiments attribute to stronger correlation between predictions.This comparison evaluates LED against two fast sampling methods on NBA.
  • Qualitative results: Correlated sampling allocates diversity appropriately and captures more modalities than I.I.D. sampling when K is small.The comparison is shown on NFL with K = 4.

6. Conclusion

LED accelerates diffusion-based trajectory prediction while allocating correlated predictions appropriately, achieving state-of-the-art performance and real-time inference on four real-world datasets. Its acceleration is partly enabled by the relatively low dimensionality of trajectory data, leaving higher-dimensional tasks for future work.

  • 6. Conclusion: LED directly models and samples from the denoised distribution through a leapfrog initializer with reparameterization.The initializer enables appropriate allocation of multiple correlated predictions during inference.
  • 6. Conclusion: LED achieves state-of-the-art performance on four real-world datasets and satisfies real-time inference needs.
  • 6. Conclusion: Trajectory-data dimensionality partly enables LED's inference acceleration because its distribution is easier to learn than image or video distributions.
  • 6. Conclusion: Higher-dimensional tasks remain a future direction for diffusion models and fast sampling methods.
Loading 2303.10895v1…