PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory Prediction with Latent Belief Energy-Based Model

Bo Pang, Tianyang Zhao, Xu Xie, Ying Nian Wu

arXiv:2104.03086v1cs.LG

TL;DR

Pedestrian trajectories are multimodal and depend on movement history and surrounding agents, making forecasting important for autonomous platforms. LB-EBM learns an expressive energy-based cost in latent space and uses a social-aware belief to plan before trajectory prediction. It reports improvements over prior state-of-the-art performance on SDD and ETH-UCY.

  • Problem

    Pedestrian forecasting must account for movement history and social context, while the same observed history can lead to multiple possible future paths.

    Source: §1 · p. 1

  • Method

    LB-EBM learns an energy-based cost over latent beliefs projected from human trajectories, then uses a sampled or optimized belief to make a plan before predicting the trajectory.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    10.9% on SDD and 27.6% on ETH-UCY improvement over prior state-of-the-art performance are reported for the proposed model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The model is reported to produce accurate, multimodal, socially compliant trajectory predictions using a two-step approach.

    Source: §5 · p. 8

    The model assumes agents’ latent beliefs are independent given joint trajectory history because an agent cannot observe other agents’ beliefs.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Human trajectory prediction is critical for autonomous platforms like self-driving cars or social robots. We present a latent belief energy-based model (LB-EBM) for diverse human trajectory forecast. LB-EBM is a probabilistic model with cost function defined in the latent space to account for the movement history and social context. The low-dimensionality of the latent space and the high expressivity of the EBM make it easy for the model to capture the multimodality of pedestrian trajectory distributions. LB-EBM is learned from expert demonstrations (i.e., human trajectories) projected into the latent space. Sampling from or optimizing the learned LB-EBM yields a belief vector which is used to make a path plan, which then in turn helps to predict a long-range trajectory. The effectiveness of LB-EBM and the two-step approach are supported by strong empirical results. Our model is able to make accurate, multi-modal, and social compliant trajectory predictions and improves over prior state-of-the-arts performance on the Stanford Drone trajectory prediction benchmark by 10.9% and on the ETH-UCY benchmark by 27.6%.

1. Introduction

LB-EBM models multimodal pedestrian behavior with an energy-based cost in a low-dimensional latent space, learned from human trajectories. It uses a social-aware latent belief to form a path plan before predicting a long-range trajectory, and reports gains on two benchmarks.

1. Introduction

  • The introduction frames pedestrian forecasting as important for autonomous platforms and difficult because future paths depend on movement history, goals, and surrounding pedestrians.Given the same past trajectories, multiple future paths are possible.Source: §1 · p. 1
  • LB-EBM combines a low-dimensional learned cost function with a policy generator, explicitly modeling multimodality rather than relying only on the generator.This middle ground contrasts with traditional IRL’s computationally expensive outer cost-learning and inner policy-learning loops.Source: §1 · p. 1
  • The latent energy model is learned from human trajectories projected into the latent space, where its expressive form can capture multimodal trajectory distributions.An encoder is jointly learned to project trajectories into that space.Source: §1 · p. 1
  • Sampling or optimizing the cost yields a social-aware belief about future motion, which is used to make a rough path plan before trajectory prediction.The plan consists of positions at well-separated future steps; the subsequent prediction also uses individual motion history and social cues.Source: §1 · pp. 1–2
  • 10.9% on SDD and 27.6% on ETH-UCY improvement over prior state-of-the-art performance support the model’s empirical effectiveness.The model and its modules are learned end-to-end.Source: §1 · p. 2
  • The paper presents LB-EBM as an inverse-reinforcement-learning approach that captures multimodal human trajectory distributions.Its contributions also include multimodal, socially compliant future predictions.Source: §1 · pp. 1–2

2. Related Work

Related work covers social-interaction modeling, goal and multimodality prediction, value-function approaches, and energy-based models. This paper combines these themes through latent-space energy-based modeling and a two-step prediction scheme.

Social interaction modeling

  • Prior social-interaction methods aggregate RNN states, convolve spatial feature maps, or use attention and relational reasoning on agent graphs; this work adapts such approaches.Some recent studies also explore transformers for temporal dependencies and social information.Source: §2 · p. 2

Modeling Goals

  • Goal-based forecasting predicts goals before trajectories, while coarse-to-fine planning predicts through grid-map stages; this work follows a two-step scheme with well-separated future positions as plans.Those plan positions can be extracted from trajectory data.Source: §2 · p. 2

Multimodality

  • Prior work models multimodality with maneuvers, variational autoencoders, generative adversarial networks, datasets, latent inference, and new loss functions; this paper uses variational inference with a latent-space EBM.Trajectories are projected into latent space to leverage the energy-based model.Source: §2 · p. 2

Value Function

  • Earlier trajectory methods model hidden costs through Q-learning, inverse optimal control, GAIL, game theory, or maximum-entropy IRL; this work learns human cost functions explicitly in latent space.Source: §2 · p. 2

Energy-Based Models

  • Energy functions can act as objectives, costs, or critics that capture regularities, but sampling or optimization requires iterative computation such as MCMC.Prior work on low-dimensional latent EBMs makes those operations more efficient and convenient, an approach this paper follows.Source: §2 · p. 2

3. Model and Learning

The model factorizes joint trajectory prediction through latent beliefs and plans, using history and social context to represent multimodal futures. It learns the latent energy model from projected expert plans and trains the full network end to end.

3.1. Problem Definition

  • Directly modeling p(Y|X) can fail to capture multimodality, so the model introduces latent beliefs and plans to factorize joint future prediction.A plan is represented by a few well-separated future positions, making it observable and available for supervision.Source: §3.1 · p. 3

3.2. LB-EBM

  • The LB-EBM learns a conditional cost over latent beliefs from projected expert plans, given trajectory history and social context.Expert latent beliefs are treated as samples from the learned distribution and therefore have low cost.Source: §3.2 · p. 3

3.3. Plan

  • The plan module models each agent’s plan as a Gaussian conditioned on its latent belief and pooled social context.Plans are conditionally independent across agents given their latent beliefs and pooled context.Source: §3.3 · p. 4

3.4. Prediction

  • The prediction module models each future trajectory as a Gaussian conditioned on the agent’s plan and pooled social context.Its mean function takes the encoded plan and pooled context as input.Source: §3.4 · p. 4

3.5. Pooling

  • Self-attention pools scene histories using a spatial-temporal binary mask, whose adjacency can be varied through hyperparameter d.The mask design follows prior work [32].Source: §3.5 · p. 4

3.6. Joint learning

  • The full network is learned end to end by optimizing a variational lower bound, with the energy-model gradient contrasting expert beliefs and samples from LB-EBM.Langevin dynamics supplies samples for the energy-model gradient; the paper describes this sampling as affordable because the latent space and model are small.Source: §3.6 · p. 5

4. Experiments

Experiments compare LB-EBM with varied trajectory-forecasting baselines on Stanford Drone and ETH-UCY, using ADE and FDE and qualitative and ablation analyses. LB-EBM improves average ADE over prior state of the art on both benchmarks, while qualitative results show diverse predictions and interaction modeling.

4.1. Implementation Details and Design Choices

  • The non-autoregressive MLP generator is selected for efficiency and to avoid exposure bias; the authors argue the plan over the full horizon reduces the need for autoregression.The paper notes a possible limitation: a non-autoregressive model may fail to capture dependencies across time steps.Source: §4.1 · p. 5

4.2. Datasets

  • Stanford Drone contains 20 top-down campus scenes, while ETH-UCY comprises five scenes evaluated with leave-one-out training and testing.The ETH-UCY trajectories are segmented into 8-second clips with 3.2 seconds of history and a 4.8-second prediction horizon.Source: §4.2 · pp. 5–6

4.3. Baseline Models

  • The evaluation compares LB-EBM with baselines spanning different learning regimes for modeling trajectories, multimodality, and social interaction.The listed baselines include deterministic social pooling, stochastic GAN and VAE models, conditional latent-space learning, and inverse reinforcement learning.Source: §4.3 · p. 6

4.5. Qualitative Results

  • Figure 2 shows close-to-ground-truth best-of-20 predictions, diverse sampled trajectories, and social interactions such as yielding and collision avoidance.The qualitative examples cover crossing, sidewalk, and roundabout scenarios.Source: §4.5 · pp. 7–8, Figure 2 · p. 7

4.4. Quantitative Results

  • LB-EBM improves ADE over prior state of the art by 10.9% on Stanford Drone and 27.6% on ETH-UCY; Table 2 reports the best average ADE and FDE.The improvement is most pronounced on ETH (44.4%) and least on ZARA1 (9.1%); the FDE improvement over PECNet on Stanford Drone is not significant.Source: §4.4 · p. 6, Table 1 · p. 6, Table 2 · p. 8

4.6. Ablation Study

  • Ablations indicate the latent belief EBM and two-step approach are effective; removing social pooling decreases ADE and FDE when social cues are available.Table 3 reports ADE and FDE across ablation conditions.Source: §4.6 · p. 8, Table 3 · p. 8

5. Conclusion

LB-EBM forecasts diverse human trajectories through a social-aware latent belief and a two-step plan-to-trajectory process, with reported gains on two benchmarks.

5. Conclusion

  • The model improves prior state-of-the-art performance by 10.9% on Stanford Drone and 27.6% on ETH-UCY.The paper reports accurate, multimodal, and socially compliant trajectory predictions.Source: §5 · p. 8
  • LB-EBM is a probabilistic cost function in latent space that accounts for movement history and social context.Its low-dimensional latent space and expressive energy-based model are intended to capture multimodal pedestrian trajectory distributions.Source: §5 · p. 8
  • Sampling from or optimizing LB-EBM yields a social-aware belief vector for path planning, which then helps predict a long-range trajectory.The model is learned from expert human trajectories projected into latent space.Source: §5 · p. 8

Model Formulation

The formulation represents scene histories, future trajectories, latent beliefs, and plans in a generative model with separate planning and prediction stages.

Model Formulation

  • X denotes all agents’ past trajectories, Y their future trajectories, Z their latent beliefs, and P their plans.Source: Model Formulation · p. 12
  • The plan module models P conditioned on latent belief Z and past trajectories X as pβ(P | Z,X).Source: Model Formulation · p. 12
  • The prediction module models future trajectories Y conditioned on plan P and past trajectories X as pγ(Y | P,X).Source: Model Formulation · p. 12

Maximum Likelihood Learning

Maximum-likelihood learning fits the generative model to the data distribution over plans and future trajectories given past trajectories, but its gradient involves an intractable posterior.

Maximum Likelihood Learning

  • Learning minimizes the KL divergence from the data distribution over plans and futures given past trajectories to the model distribution.With sampled training examples, this minimization is approximated by maximizing log-likelihood, yielding the maximum-likelihood estimate.Source: Maximum Likelihood Learning · p. 12
  • The training examples are sampled from the data distribution over plans, future trajectories, and past trajectories.Source: Maximum Likelihood Learning · p. 12
  • The gradient of a single-scene log-likelihood involves the posterior pψ(Z | P,X), which is intractable.Source: Maximum Likelihood Learning · p. 12

Variational Learning

Variational learning makes the intractable likelihood objective tractable and provides objectives for the LB-EBM, plan, and prediction modules, which are trained end-to-end.

Variational Learning

  • The variational objective upper-bounds the KL divergence between data and model distributions and equivalently lower-bounds log-likelihood.The variational distribution qφ(Z | P,X) is Gaussian with diagonal covariance in this work.Source: Variational Learning · p. 12
  • The module objectives are major but not exclusive because end-to-end training lets gradients from one module flow to others.Source: Variational Learning · p. 12
  • The LB-EBM objective compares expert latent beliefs with the model distribution, using data samples, reparameterized variational samples, and Langevin samples.The objective includes the latent cost Cα(Z,X) and log partition function Zα(X).Source: Variational Learning · pp. 12–13
  • The plan-module objective is equivalent to maximizing the likelihood of pβ(P | Z,X).Expression 33 drives learning of the plan module.Source: Variational Learning · p. 13
  • Expression 34 drives learning of the prediction module, and sampling plans from pβ(P | Z,X) instead of the data distribution facilitates learning of other modules and improves performance.Source: Variational Learning · p. 13

Appendix B: Negative Log-Likelihood Evaluation

Table 4 compares LB-EBM and baselines on ETH-UCY using NLL; LB-EBM outperforms S-GAN and Trajectron but underperforms Trajectron++.

Appendix B: Negative Log-Likelihood Evaluation

  • KDE NLL scores ground-truth trajectory likelihood at each timestep using kernel density estimates, then averages across timesteps.Source: Appendix B · p. 13
  • Because few papers report NLL on these benchmarks, the authors note that fair comparison with most baselines may be difficult.Source: Appendix B · p. 13
  • The authors suggest Trajectron++'s bivariate Gaussian mixture versus LB-EBM's unimodal Gaussian may explain the gap, and leave mixture outputs as future work.Source: Appendix B · p. 13
  • On ETH-UCY NLL, LB-EBM outperforms S-GAN and Trajectron but underperforms Trajectron++.Table 4 reports the ETH-UCY NLL evaluation; lower is better.Source: Table 4 · p. 13, Appendix B · p. 13
  • Trajectron++ was concurrent work discovered during the reviewing process.Source: Appendix B · p. 13

Related papers

Loading 2104.03086v1…