PaperlayerView exact revision on arXiv

Source-linked AI summary

BiTraP: Bi-directional Pedestrian Trajectory Prediction with Multi-modal Goal Estimation

Yu Yao, Ella Atkins, Matthew Johnson-Roberson, Ram Vasudevan, Xiaoxiao Du

arXiv:2007.14558v2cs.CVcs.RO

TL;DR

Long-horizon pedestrian prediction can accumulate recurrent decoding errors while needing to represent multiple plausible futures. BiTraP estimates future goals and uses a CVAE-based bi-directional decoder, achieving state-of-the-art results across FPV and BEV datasets and showing that target-model choices affect predicted distribution diversity.

  • Problem

    Recurrent trajectory predictors can accumulate errors over long horizons, while pedestrian motion is stochastic and has multiple plausible futures.

  • Method

    BiTraP uses a CVAE to estimate multimodal trajectory goals and generate future trajectories with a goal-conditioned bi-directional decoder.

  • Results

    12%–51%: BiTraP-NP outperforms PECNet on best-of-20 ADE/FDE results across ETH-UCY scenes, while BiTraP models achieve state-of-the-art results on FPV and BEV datasets.

  • Takeaways & Limitations

    Different Gaussian-latent non-parametric and categorical-latent GMM choices affect the diversity of predicted future-trajectory distributions.

Abstract

from arXiv · show

Pedestrian trajectory prediction is an essential task in robotic applications such as autonomous driving and robot navigation. State-of-the-art trajectory predictors use a conditional variational autoencoder (CVAE) with recurrent neural networks (RNNs) to encode observed trajectories and decode multi-modal future trajectories. This process can suffer from accumulated errors over long prediction horizons (>=2 seconds). This paper presents BiTraP, a goal-conditioned bi-directional multi-modal trajectory prediction method based on the CVAE. BiTraP estimates the goal (end-point) of trajectories and introduces a novel bi-directional decoder to improve longer-term trajectory prediction accuracy. Extensive experiments show that BiTraP generalizes to both first-person view (FPV) and bird's-eye view (BEV) scenarios and outperforms state-of-the-art results by ~10-50%. We also show that different choices of non-parametric versus parametric target models in the CVAE directly influence the predicted multi-modal trajectory distributions. These results provide guidance on trajectory predictor design for robotic applications such as collision avoidance and navigation systems.

I. INTRODUCTION

Pedestrian trajectory prediction must represent multiple plausible futures while avoiding long-horizon error accumulation. BiTraP addresses this with goal estimation, bi-directional decoding, and comparisons of latent-variable target models.

  • Motivation: Pedestrian trajectories are stochastic, so prediction should represent multiple plausible and distinct future behaviors.Accurate forecasts support safe, socially aware robot navigation and anomaly alarms.
  • Motivation: Existing recurrent predictors can deteriorate rapidly over time because they repeatedly condition future predictions on previous outputs.The cited deterioration occurs beyond 560 ms.
  • BiTraP: BiTraP estimates pedestrian endpoints and combines forward propagation from the current position with backward propagation from estimated goals.This differs from prior goal-conditioned methods that use goals only as inputs to a forward decoder.
  • Modeling choices: BiTraP compares non-parametric Gaussian-latent and parametric GMM categorical-latent variants to study how latent distributions affect multimodal trajectory predictions.The comparison uses BiTraP-NP and BiTraP-GMM with best-of-many L2 and negative log-likelihood losses.
  • Evaluation: Experiments cover first-person and bird’s-eye-view datasets and report improvements especially for prediction horizons of ≥2 seconds.The paper presents the approach as applicable across both viewing domains.
  • BiTraP: Bi-directional temporal propagation extends goal information backward through the decoder, while current trajectory information is passed forward toward endpoints.The authors distinguish this design from prior endpoint-encoding approaches.

III. BITRAP: BI-DIRECTIONAL TRAJECTORY PREDICTION WITH GOAL ESTIMATION

BiTraP uses a CVAE to estimate multimodal future goals and then generate trajectories conditioned on those goals. Its architecture combines observation encoding, latent-variable inference, goal generation, and bi-directional trajectory decoding.

  • Problem formulation: BiTraP maps an observed trajectory Xt to a future endpoint Gt and future trajectory Yt in either FPV or BEV coordinates.Gt is defined as Yt+δ, with the endpoint available during training but unknown during testing.
  • CVAE architecture: The CVAE comprises prior, recognition, goal-generation, and trajectory-generation networks parameterized by θ, φ, ω, and ψ.The prior models latent variables from observations, while the recognition network also uses the target trajectory during training.
  • Non-parametric variant: BiTraP-NP uses a Gaussian latent variable and a non-parametric target distribution within an RNN encoder-decoder framework.It predicts the goal before using that goal to predict trajectories.
  • Training and inference: During training, the recognition network captures observation-target dependencies, whereas testing samples latent variables from the observation-conditioned prior to generate goals.The sampled latent variable is concatenated with the encoded observation feature for goal generation.
  • Bi-directional decoder: The trajectory decoder uses forward and backward RNNs, initializing backward prediction with the estimated goal and combining both directions for waypoint outputs.The forward RNN does not directly transform its output into trajectory space before the bidirectional combination.

B. BiTraP with GMM Distribution

BiTraP-GMM represents each trajectory modality parametrically through Gaussian-mixture components, while BiTraP-NP predicts residual coordinates and uses best-of-many training to encourage diverse trajectories.

  • GMM representation: BiTraP-GMM models the goal and each waypoint with a K-component Gaussian mixture, treating each component as one trajectory modality.Its generation networks estimate Gaussian component means rather than directly predicting trajectory coordinates.
  • GMM representation: Mixture weights πi sum to one and define categorical latent-variable probabilities for sampling trajectory modalities.Higher-confidence modalities are more likely to be sampled during testing.
  • Training and inference: BiTraP-GMM trains one latent sample from each category, then samples from Cat(K, π1:K) during testing.This training procedure ensures all trajectory modalities receive training.
  • Residual prediction and loss: BiTraP-NP predicts residuals relative to the current location rather than locations or integrated velocities, using best-of-many L2 loss for multimodal prediction.The final loss combines goal L2, trajectory L2, and KL-divergence terms.

D. Bi-directional NLL Loss for BiTraP-GMM

BiTraP-GMM uses forward and reverse integration to compute a bi-directional NLL loss, addressing the temporal emphasis introduced by standard forward integration. Its final objective combines goal estimation, forward and backward waypoint likelihoods, and KL divergence.

  • BiTraP-GMM models pedestrian velocity as a GMM at each timestep and integrates it forward to obtain waypoint distributions.The model assumes linear pedestrian dynamics and uses a single integrator.
  • Standard forward NLL emphasizes earlier waypoints because each early waypoint contributes to later integration results.A waypoint at t + 1 affects integration results over t+2, t+3, ..., whereas later waypoints do not affect it.
  • BiTraP-GMM adds reverse integration from the goal to compute a backward NLL consistent with its bi-directional temporal model.The backward formulation complements the forward integration direction rather than relying only on early-waypoint likelihoods.
  • The final loss combines goal-estimation NLL, forward NLL, backward NLL, and KL divergence.The forward and backward terms are computed from forward and backward integration, respectively.

IV. EXPERIMENTS AND RESULTS

Experiments on FPV datasets evaluate BiTraP across JAAD and PIE using deterministic and multi-modal variants, trajectory errors, likelihood, and qualitative predictions. The results show complementary strengths of BiTraP-NP and BiTraP-GMM for displacement accuracy and distributional fit.

  • Datasets and setup: JAAD and PIE provide FPV pedestrian trajectories, with 0.5-second observations and 1.5-second prediction horizons used for evaluation.JAAD contains 2,800 trajectories and PIE contains 1,800, both annotated at 30Hz.
  • Evaluation: The evaluation compares BiTraP with Kalman, LSTM, Bayesian-LSTM, PIEtraj, PIEfull, and FOL-X baselines, plus deterministic BiTraP-D.Metrics include bounding-box ADE, center ADE, center FDE, and KDE-NLL for multi-modal models.
  • Quantitative results: BiTraP-D consistently achieves lower displacement errors than baseline methods across prediction horizons.It also outperforms PIEfull despite using only past trajectory information, whereas PIEfull requires ego-motion and semantic intention annotations.
  • Quantitative results: BiTraP-NP performs better on displacement metrics, while BiTraP-GMM performs better on NLL, reflecting diverse versus more ground-truth-similar predicted distributions.BiTraP-GMM represents endpoint uncertainty with GMM components, while BiTraP-NP samples a broader set of possible trajectories.
  • Qualitative results: Qualitative examples show multi-modal BiTraP can represent alternatives such as crossing or changing direction when deterministic BiTraP-D misses the intended endpoint.BiTraP-NP heatmaps visualize likelihood fitted by KDE over samples, with orange denoting higher probability.

B. Experiments on ETH-UCY Datasets

Experiments on ETH-UCY assess trajectory accuracy, likelihood quality, ablations, computation, and long-horizon behavior. BiTraP-NP leads displacement metrics, while BiTraP-GMM yields stronger likelihood results and improves at longer horizons.

  • Trajectory accuracy: BiTraP-NP outperforms PECNet by approximately 12%-51% and improves over Trajectron++ by approximately 12%-24% on most ETH-UCY scenes.These comparisons use best-of-20 ADE/FDE in meters.
  • Distributional accuracy: BiTraP-GMM outperforms Trajectron++ on ANLL and FNLL for ETH, Univ, Zara1, and Zara2, but not Hotel.On Hotel, combining BiTraP-GMM with the interaction encoder improves ANLL/FNLL by -1.88/0.27.
  • Distributional behavior: BiTraP-NP produces wider, more diverse trajectory distributions, whereas BiTraP-GMM produces more compact distributions associated with lower NLL.Figure 4 visualizes twenty sampled trajectories and GMM endpoint ellipses whose size indicates Σk and transparency indicates πk.
  • Long-horizon prediction: After 1.2 seconds on ETH, Univ, Zara1, and Zara2, BiTraP-GMM outperforms Trajectron++ in per-timestep KDE-NLL.The authors attribute the longer-horizon improvement to backward passing from the estimated goal.

C. Additional Experiments

Additional experiments examine computational time and robot-navigation behavior. The two model variants show different operational trade-offs: BiTraP-NP is more collision-sensitive, while BiTraP-GMM produces fewer false alarms.

  • Computational time: BiTraP inference time changes by approximately 3ms when increasing samples from 20 to 2000.The comparison includes Social GAN and Trajectron++, with Trajectron++ timing including scene-graph generation.
  • Robot navigation: A robot using BiTraP-NP is more sensitive to potential collisions because it predicts diverse surrounding-pedestrian distributions.The navigation simulation uses the ETH-UCY dataset.
  • Robot navigation: A robot using BiTraP-GMM reports fewer false alarms while navigating among pedestrians and is therefore more efficient in the simulation.This contrasts with the greater collision sensitivity observed for BiTraP-NP.

V. CONCLUSION

The paper concludes that BiTraP provides goal-conditioned bi-directional multi-modal trajectory prediction across FPV and BEV datasets. Its non-parametric and parametric variants produce different target-distribution diversity, with implications for predictor selection in robotic planning.

  • Conclusion: BiTraP achieves state-of-the-art pedestrian trajectory prediction results on both first-person-view and bird’s-eye-view datasets.The models use only observed trajectories as inputs while surpassing methods requiring additional information.
  • Conclusion: BiTraP-NP and BiTraP-GMM differ in the diversity of their predicted future-trajectory target distributions because of their latent-variable choices.The comparison links non-parametric and parametric target modeling to different distributional behavior.
  • Implications and future work: Collision metrics can guide selection between predictor variants for real-world applications such as robot path planning.The paper proposes adding scene semantics and social components in future work.

I. CVAE PRELIMINARIES

A CVAE generates targets conditioned on observations and stochastic latent variables through prior, recognition, and generation networks. BiTraP modifies this framework by jointly optimizing two generation networks.

  • A CVAE generates target data Y from observation X and latent variable Z.
  • Its three modules are conditional prior, recognition, and generation networks, which model latent variables, dependencies, and target generation.
  • BiTraP uses two generation networks and optimizes both networks end-to-end within a modified CVAE.

II. ROBOT NAVIGATION SIMULATION EXPERIMENT USING BITRAP

The simulation evaluates BiTraP for robot path selection and collision detection using predicted pedestrian distributions and Monte Carlo robot trajectories. Results support safer-path selection and characterize differences between target models.

  • Experimental Setup: The experiment uses ETH-UCY bird’s-eye-view scenes, treating the tested pedestrian as a robot navigating among neighboring pedestrians.
  • Experimental Setup: Monte Carlo robot trajectories are generated with Bezier curves, and predicted pedestrian distributions are represented as heat maps around walking directions.
  • Task 1: Predict the Safest Path: BiTraP-selected safest paths reduce collision rates by a large margin compared with randomly selected paths.The safest path is the minimum-collision-score path among 100 Monte Carlo paths, with collisions defined by a distance below 0.2 meters.
  • Task 2: Predict Collision for Any Path: Collision prediction is evaluated by thresholding path collision scores and reporting ROC and precision-recall curves with AUC and AP.The evaluation varies threshold γ to compute TPR, FPR, and precision.
  • Task 2: Predict Collision for Any Path: Both models achieve AUC values above 90 on ETH, while BiTraP-GMM reaches 95.5 AUC and 26.0 AP versus 92.3 AUC and 24.2 AP for BiTraP-NP.
  • Task 2: Predict Collision for Any Path: BiTraP-NP predicts more diverse distributions, producing more false alarms and lower precision than the more compact BiTraP-GMM distributions at matched recall or TPR.

III. EXPERIMENT AND RESULT ON NUSCENES DATASET

The nuScenes experiment evaluates BiTraP in bird’s-eye-view autonomous-driving scenes using FDE and KDE-NLL. BiTraP-GMM improves distribution accuracy and longer-horizon displacement accuracy relative to Trajectron++-based methods.

  • Experimental Setup: The nuScenes dataset provides bird’s-eye-view trajectories from 850 scenes, with 700 for training and 150 for testing.
  • Evaluation Metrics: Evaluation uses most-likely FDE and kernel density estimation negative log-likelihood to assess prediction accuracy.
  • Results: Adding dynamic integration and map encoding to Trajectron++ improves distribution accuracy substantially without changing most-likely FDE.
  • Results: BiTraP-GMM improves KDE-NLL at every evaluated time step and improves FDE after 2 seconds while using only the target pedestrian’s past trajectory.
  • Results: BiTraP-GMM outperforms Trajectron++-based methods, suggesting map information may be less important for pedestrian prediction on nuScenes.
Loading 2007.14558v2…