PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data

Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, Marco Pavone

arXiv:2001.03093v5cs.ROcs.HCcs.LG

TL;DR

Many trajectory-forecasting methods omit agent dynamics and environmental information, despite the importance of behavior prediction for safe interactive autonomy. Trajectron++ combines dynamics-aware forecasting with heterogeneous inputs and optional conditioning on future ego motion. It achieves state-of-the-art performance across standard and new real-world datasets, outperforming a range of deterministic and generative methods.

  • Problem

    Many forecasting methods do not enforce agent dynamics or use environmental information such as maps, although human-motion prediction is important for safe human-interactive autonomy.

    Source: §1 · pp. 1–2

  • Method

    Trajectron++ uses a graph-structured approach to forecast dynamically feasible trajectories for diverse interacting agents from heterogeneous data, with optional conditioning on future ego motion plans.

    Source: §1 · p. 3, §4 · pp. 7–8

  • Results

    Trajectron++ achieves state-of-the-art prediction performance across standard and new real-world datasets, outperforming a wide range of deterministic and generative methods.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    Its full probability distributions are suited to downstream robotic tasks including motion planning, decision making, and control.

    Source: §6 · p. 14

    In the ego-conditioned setting, the future ego-agent motion plan is assumed known for the next T timesteps.

    Source: §3 · p. 4

Abstract

from arXiv · show

Reasoning about human motion is an important prerequisite to safe and socially-aware robotic navigation. As a result, multi-agent behavior prediction has become a core component of modern human-robot interactive systems, such as self-driving cars. While there exist many methods for trajectory forecasting, most do not enforce dynamic constraints and do not account for environmental information (e.g., maps). Towards this end, we present Trajectron++, a modular, graph-structured recurrent model that forecasts the trajectories of a general number of diverse agents while incorporating agent dynamics and heterogeneous data (e.g., semantic maps). Trajectron++ is designed to be tightly integrated with robotic planning and control frameworks; for example, it can produce predictions that are optionally conditioned on ego-agent motion plans. We demonstrate its performance on several challenging real-world trajectory forecasting datasets, outperforming a wide array of state-of-the-art deterministic and generative methods.

1 Introduction

Predicting human behavior matters for safe human-interactive autonomy, but many forecasting methods omit agent dynamics, environmental information, or the ego-agent’s motion. Trajectron++ addresses these gaps with forecasts for interacting agents that can use heterogeneous data and be conditioned on future ego motion.

1 Introduction

  • Human behavior prediction is presented as a prerequisite for safe human-interactive autonomy and as support for more informed decisions and proactive actions around other agents.The introduction points to self-driving vehicles being tested in the real world and targeting widespread deployment.Source: §1 · pp. 1–2
  • Many existing methods omit agent dynamics, the ego-agent’s future motion, and environmental information such as camera images, lidar, and maps.Table 1 summarizes recent methods’ coverage of these desiderata for pedestrian and vehicle forecasting.Source: §1 · p. 2, Table 1 · p. 3
  • The proposed model targets dynamics-aware, optionally ego-conditioned forecasts and an open, extensible way to use heterogeneous environmental data.Its motivation includes adapting predictions to scene structure, such as the difference between an urban intersection and an open sports field.Source: §1 · p. 2
  • Trajectron++ forecasts dynamically feasible trajectories for multiple interacting agents of distinct semantic types using heterogeneous inputs, and can condition predictions on future ego-agent plans.Across several datasets, it outperforms many state-of-the-art deterministic and generative methods, with up to 60% lower average prediction error in some cases.Source: §1 · p. 3

2 Related Work

Prior trajectory forecasting includes deterministic regression and generative probabilistic approaches. The section highlights that position-based probabilistic outputs make dynamics constraints difficult to enforce, while use of dynamics and non-trajectory data remains limited.

Deterministic Regressors

  • Earlier deterministic forecasting methods include Social Forces, Gaussian Process Regression, Inverse Reinforcement Learning, and recurrent neural networks.The section describes these as time-series regression approaches and points to for a review.Source: §2 · p. 3

Generative, Probabilistic Approaches

  • Generative methods predict distributions of possible futures rather than a single best trajectory, which can support downstream planning and decision making through information such as variance.CVAE-based methods explicitly encode multimodality, while GAN-based approaches do so implicitly.Source: §2 · pp. 3–4
  • CVAE and GAN approaches typically output position distributions, making it difficult to enforce dynamics constraints such as no-side-slip conditions.The section identifies Trajectron and MATF as the strongest-performing CVAE- and GAN-based methods, respectively, on standard pedestrian benchmarks.Source: §2 · p. 4

Accounting for Dynamics and Heterogeneous Data

  • Few works use dynamics or data beyond past trajectories; positional reasoning may produce predictions unrealizable by control variables, such as a car moving sideways.The section attributes limited use of other modalities partly to standard benchmarks seldom including them, while noting newer autonomous-vehicle datasets with rich sensor data.Source: §2 · p. 4

3 Problem Formulation

The formulation forecasts future states for a time-varying set of interacting agents from their histories and surrounding semantic maps. It also considers a setting where the ego-agent’s future motion plan is known.

3 Problem Formulation

  • The target is a distribution over future states for a time-varying number of interacting agents, conditioned on their histories and available semantic maps.Figure 2 depicts a directed spatiotemporal graph whose nodes and edges represent agents and their interactions.Source: §3 · p. 4, Figure 2 · p. 5
  • Maps are centered around each agent and can range from obstacle-occupancy grids to layered semantic information such as sidewalks, road boundaries, and crosswalks.The formulation specifies map context size, spatial resolution, and semantic channels.Source: §3 · p. 4
  • For ego-conditioned forecasting, the formulation assumes the ego-agent’s motion plan is known for the next T timesteps.This setting supports predictions conditioned on a specified future plan.Source: §3 · p. 4

4 Trajectron++

Trajectron++ builds a graph-structured recurrent model that forecasts interacting agents from their histories, interactions, and heterogeneous inputs. It predicts controls and integrates them with agent dynamics, while also supporting future ego-motion conditioning and multimodal outputs.

4 Trajectron++

  • A spatiotemporal graph represents agents as nodes and their influences as directed edges, allowing asymmetric interactions and different semantic-class perception ranges.Edges are determined by an agent-pair distance threshold keyed to the influenced agent’s semantic class.Source: §4 · pp. 5–6
  • The model encodes agent histories with an LSTM and neighboring-agent influences with edge-type LSTMs, sum aggregation, and additive attention.Element-wise sums handle variable neighbor counts while preserving count information; the resulting influence representation is combined with node history.Source: §4 · p. 6
  • For heterogeneous inputs, Trajectron++ encodes each agent’s local map with a CNN and can incorporate additional modalities as vectors.Maps are rotated to match agent heading; examples of further inputs include lidar, camera images, skeletons, and gaze estimates.Source: §4 · pp. 6–7
  • A bidirectional LSTM encodes future ego-agent motion plans, so predictions can account for possible robot motion when evaluating responses from other agents.The encoded plan is concatenated into the model’s backbone representation.Source: §4 · p. 7
  • A discrete Categorical latent variable represents high-level behavior modes, supporting multimodal predictions and an analytic output distribution.Training uses a modified InfoVAE objective for discrete latent states; its 25-element latent term is computed directly rather than backpropagated through Gumbel-Softmax.Source: §4 · pp. 7–8
  • Integrating predicted controls with agent dynamics guarantees dynamically feasible trajectory samples, using single integrators for pedestrians and dynamically-extended unicycles for vehicles.The approach can generalize to other dynamics models when their parameters are assumed or quickly estimated online.Source: §4 · pp. 6–8

5 Experiments

Experiments evaluate Trajectron++ on pedestrian and autonomous-driving datasets, comparing deterministic and probabilistic forecasts and testing the effects of dynamics, maps, and ego-motion conditioning. The reported results include benchmark improvements and fewer obstacle- or road-boundary-violating predictions with map encoding.

5 Experiments

  • Evaluation spans ETH and UCY pedestrian trajectories and nuScenes vehicle and pedestrian trajectories, using leave-one-out evaluation for ETH and UCY and the validation set for nuScenes.ETH and UCY use 8 observed and 12 predicted timesteps; nuScenes uses a 15% training-set split for hyperparameter tuning.Source: §5 · pp. 8–10

5.1 ETH and UCY Datasets

  • On ETH and UCY, the method is competitive with state-of-the-art deterministic regressors and outperforms existing approaches by 33% on mean FDE.This comparison uses node-history and edge encoders without map or future-motion-plan encoding.Source: Table 2 · p. 10, §5.1 · p. 11

5.2 nuScenes Dataset

  • Trajectron++ outperforms existing approaches on nuScenes vehicle forecasting without sharp performance degradation after 3 s, despite training only to predict 3 s ahead.Other methods do not report values at 2 s and 4 s; their detection and tracking errors were subtracted for comparison.Source: Table 4 · p. 12, §5.2 · p. 13

5.1 ETH and UCY Datasets

  • 55–60% lower average errors than state-of-the-art on Best-of-N comparisons.The paper reports that dynamics integration also makes its advantage clear on KDE-based NLL, where Trajectron++ significantly outperforms other methods.Source: Table 2 · p. 10, §5.1 · pp. 11–12, Table 3 · p. 11
  • Map encoding reduces colliding predictions from 4.6% to 1.0% overall and from 21.5% to 4.9% for pedestrians close to obstacles.These rates come from 2,000 trajectory samples on the ETH-University scene.Source: §5.1 · p. 12

5.2 nuScenes Dataset

  • Dynamics integration sharply reduces nuScenes vehicle NLL and FDE at all prediction horizons, but slightly increases road-boundary violations; adding maps maintains accuracy while reducing violations.Dynamics integration is identified as the dominant performance-improving module.Source: §5.2 · pp. 13–14
  • Conditioning on the ego-vehicle’s future motion plan significantly reduces prediction error and road-boundary violations.The paper situates this use case in autonomous driving, where the ego-vehicle evaluates motion primitives repeatedly.Source: §5.2 · p. 13
  • In the nuScenes example, dynamics integration captures both turning and straight-ahead possibilities, while map encoding places nearly all probability mass on the correct side of the road.The base model undershoots the turn and may predict movement into oncoming traffic.Source: §5.2 · p. 14, Figure 3 · p. 14

6 Conclusion

Trajectron++ produces full-distribution forecasts for diverse agents using heterogeneous data and dynamics constraints, and achieves state-of-the-art performance across several metrics on real-world datasets. For single integrators, it models uncertain velocity controls and integrates them to predict positions.

6 Conclusion

  • Trajectron++ achieves state-of-the-art prediction performance across several metrics on standard and new real-world multi-agent behavior datasets.Source: §6 · p. 14
  • The framework supports future-conditional, dynamics-respecting forecasts from heterogeneous data and full probability distributions for downstream robotic tasks.The conclusion names motion planning, decision-making, and control as examples.Source: §6 · p. 14

A Single Integrator Distribution Integration

  • For single integrators, the state is position and the model predicts a Gaussian distribution over velocity controls, whose uncertainty is integrated into position predictions.The control covariance is the prediction model’s only uncertainty source.Source: A Single Integrator Distribution… · p. 18

A.1 Mean Derivation2

At test time, Trajectron++ uses the predicted mean velocity and current mean position to obtain the next mean position.

A.1 Mean Derivation2

  • At test time, the predicted mean velocity and current mean position are passed through Equation (5) to produce the next mean position.Source: §A.1 · p. 18
  • The output mean positions are obtained using the sum of Gaussian random variables.Source: §A.1 · p. 18
  • The current agent position is combined with the predicted mean velocity in the position update.Source: §A.1 · p. 18

A.2 Covariance Derivation2

The covariance derivation uses Gaussian-sum covariance rules, while the dynamically extended unicycle model integrates uncertain controls and computes mean positions from mean control actions.

A.2 Covariance Derivation2

  • Position covariance is obtained using the covariance of a sum of Gaussian random variables, as in the Kalman filter prediction step.Source: §A.2 · p. 18

B Dynamically-Extended Unicycle Distribution Integration

  • For wheeled-agent dynamics, the model uses acceleration and heading rate as controls, assumes piecewise-constant controls per step, and uses a limit form when |ω| ≤ 10^-3.At each timestep and latent value, it predicts a Gaussian distribution over control actions and integrates the controls and uncertainties through the dynamics.Source: B Dynamically-Extended Unicycle… · pp. 19–20

B.1 Mean Derivation3

  • The output mean positions are computed by applying the mean control actions to the discrete dynamics.Source: §B.1 · p. 20

B.2 Covariance Derivation3

The covariance propagation linearizes the dynamics around a state and control; the figure caption describes how ADE and FDE distributions are compared across datasets.

B.2 Covariance Derivation3

  • The covariance update uses the system Jacobians to propagate state uncertainty and incorporate control uncertainty.The system is made linear Gaussian by linearizing the dynamics about a specific state and control.Source: §B.2 · pp. 20–21
  • The covariance equations are also used in the Extended Kalman Filter prediction step.Source: §B.2 · p. 20
  • Figure 4 compares ADE and FDE distributions per dataset and average performance; compare the generative-model error distributions with deterministic-baseline lines.The figure caption identifies mean ADE with X markers and shows 2,000 sampled trajectories per model at each prediction timestep.Source: Figure 4 · p. 21

C Average and Final Displacement Error Evaluation

The section compares full ADE and FDE error distributions, preserving probabilistic information from generative methods. Trajectron++ has lower, more concentrated errors than other generative approaches and outperforms state-of-the-art deterministic methods.

C Average and Final Displacement Error Evaluation

D Additional Training Information

Training choices tune the latent-variable objective and map encoder, while scene rotation augmentation reduces reliance on environment-specific movement directions. Map encoding also reduces obstacle-violating predictions.

D.1 Choosing α, β in Equation (4)

  • Encoding a local map significantly reduces obstacle-violating predictions compared with using trajectory data alone.Source: Figure 5 · p. 22
  • β weights the KL penalty and varies with encoder-output size, increasing when map information enters the condition; it is sigmoid-annealed, while α remains 1.0.The schedule uses low β early so the model can encode information in z, then gradually increases β during training; β helps disentangle the latent space and encourage multimodality.Source: §D.1 · pp. 21–22

D.2 Separate Map Encoder Learning Rate

  • The map CNN uses a smaller learning rate and leaky ReLU with α = 0.2; regular ReLU, sigmoid, and tanh saturate early and fail to recover.The smaller learning rate is used to prevent large gradients early in training, when map encodings are not yet useful to the rest of the model.Source: §D.2 · p. 22

D.3 Data Augmentation

  • Scene trajectories are rotated in 15° increments from 0° to 360° (exclusive) to reduce overfitting to environment-specific movement directions.The same augmentation is applied to pedestrian and autonomous-driving datasets.Source: §D.3 · p. 22

E ETH Pedestrians Map Encoding

Map encoding reduces obstacle-violating predictions for nearby ETH pedestrians, and the model’s online updates support low-latency forecasting across tested datasets.

E ETH Pedestrians Map Encoding

  • Map encoding reduces obstacle-violating predictions for ETH University pedestrians especially close to obstacles.The cited example concerns pedestrians with at least one obstacle-violating trajectory in their Full output.Source: E ETH Pedestrians Map Encoding · p. 22

F Online Runtime

  • Runtime stays below 1.2s on the tested laptop and datasets, allowing 3–5 predictions per horizon.Most runtimes are at or below 1s; measurements used a 2.7 GHz Intel Core i5 Broadwell CPU with 8 GB RAM.Source: Figure 6 · p. 23
  • Runtime varies with agent count and interaction volume; insufficient heatmap values are imputed using an optimization-based scheme.Source: F Online Runtime · p. 23
  • Stateful spatiotemporal graphs enable online updates: new observations feed the final encoder LSTM cells, after which the rest of the model runs on the updated representation.Source: F Online Runtime · p. 23

Related papers

Loading 2001.03093v5…