PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Deep Generative Models for Trajectory Prediction: A Conditional Variational Autoencoder Approach

Boris Ivanovic, Karen Leung, Edward Schmerling, Marco Pavone

arXiv:2008.03880v2cs.ROcs.HCcs.LG

TL;DR

Human behavior prediction must account for uncertain, multimodal responses in interactive settings. The paper presents a CVAE trajectory-prediction tutorial conditioned on interaction history and future robot actions. It reports strong benchmark comparisons for Trajectron++ and describes the model as attractive for multi-agent settings with large, heterogeneous data and non-Markovian behavior.

  • Problem

    Human behavior varies across situations, and interactive robots need to account for multiple possible human responses to their actions.

    Source: §I · p. 1

  • Method

    The paper presents a CVAE trajectory predictor conditioned on interaction history and future robot actions, using discrete latent modes to represent multimodal futures.

    Source: B. Interaction-aware Human Behavior… · p. 4

  • Results

    Trajectron++ significantly outperforms Social GAN and Social BiGAT on BoN ADE, BoN FDE, and KDE NLL on ETH and UCY, and shows strong vehicle-modeling performance against varied approaches on nuScenes.

    Source: A. Quantitative Performance · p. 7

  • Takeaways & Limitations

    The authors describe the CVAE approach as attractive for multi-agent prediction with large, heterogeneous datasets and non-Markovian interaction histories.

    Source: §VII · p. 7

Abstract

from arXiv · show

Human behavior prediction models enable robots to anticipate how humans may react to their actions, and hence are instrumental to devising safe and proactive robot planning algorithms. However, modeling complex interaction dynamics and capturing the possibility of many possible outcomes in such interactive settings is very challenging, which has recently prompted the study of several different approaches. In this work, we provide a self-contained tutorial on a conditional variational autoencoder (CVAE) approach to human behavior prediction which, at its core, can produce a multimodal probability distribution over future human trajectories conditioned on past interactions and candidate robot future actions. Specifically, the goals of this tutorial paper are to review and build a taxonomy of state-of-the-art methods in human behavior prediction, from physics-based to purely data-driven methods, provide a rigorous yet easily accessible description of a data-driven, CVAE-based approach, highlight important design characteristics that make this an attractive model to use in the context of model-based planning for human-robot interactions, and provide important design considerations when using this class of models.

I. INTRODUCTION

Human behavior is inherently uncertain, and interactive robots must account for multiple possible responses to their actions. This paper tutorials a CVAE-based, model-based approach to trajectory prediction and situates it among methods for interactive human behavior prediction.

I. INTRODUCTION

  • Human behavior varies across people, settings, and time, so robots need to reason about multiple possible responses rather than an average future.The introduction illustrates this with pedestrians who may pass on either side of one another.Source: §I · p. 1
  • A model-based approach learns interaction dynamics separately from policy construction, which can make planner decisions more transparent than model-free policies.The introduction also notes that conditional models trained on offline data may confuse correlation with causation; proposed mitigations include data strategies and safety-preserving fallback controllers.Source: §I · pp. 1–2
  • The paper focuses on a CVAE model that predicts multimodal human-action distributions conditioned on interaction history and candidate future robot actions,.History can represent factors such as experience, mood, or engagement, while future robot actions account for response dynamics.Source: §I · p. 2
  • The tutorial builds a taxonomy of state-of-the-art methods and explains the CVAE architecture, its design characteristics, performance comparisons, and implementation considerations,,,.Its stated focus includes scalability to multi-agent settings, heterogeneous data, and an analytic representation rooted in a dynamics model.Source: §I · p. 2

II. RELATED WORK

Prior approaches range from models built on assumptions about physics or decision-making to data-driven predictors. The paper motivates phenomenological generative models for settings with abundant data, multimodal futures, and interaction history.

II. RELATED WORK

  • Physics-based ontological models such as Social Forces and IDM can simulate large-scale interactions, but they are unimodal and do not use interaction history.These models encode assumptions about the system’s underlying physics.Source: §II · p. 2
  • Reward-based approaches can be interpretable and data-efficient, but typical IRL features may limit expressivity or omit interaction history, while learned multimodality may be difficult to use in planning,,.Gradient-based local optimization in one cited MaxEnt IRL application yields a unimodal assumption on interaction outcomes.Source: §II · pp. 2–3
  • Deterministic trajectory regressors produce a single future and therefore do not represent the uncertainty that motivates forecasting multiple outcomes in safety-critical settings,.The cited discussion describes growing interest in methods that forecast multiple futures or distributions over outcomes.Source: §II · p. 3
  • GANs and CVAEs are deep generative alternatives; GANs can suffer mode collapse and unstable training, while CVAEs optimize likelihood across training examples and can provide analytic distributions as well as samples,, [28],,,.The paper notes that GANs’ empirical-sample output may limit compatibility with planners that rely on parameterized distributions.Source: §II · p. 3
  • The paper selects CVAEs for settings with large datasets and a need for expressive, multimodal predictions conditioned on interaction history.It describes phenomenological approaches as relying on observations rather than inherent assumptions about interaction structure or agent decision-making.Source: §II · p. 3

III. THE CONDITIONAL VARIATIONAL AUTOENCODER FOR INTERACTION-AWARE BEHAVIOR PREDICTION

The section presents a CVAE as a conditional generative model that represents latent alternatives and decodes them into predicted outputs. It explains how a proposal distribution supports training through an evidence lower bound on conditional likelihood.

III. THE CONDITIONAL VARIATIONAL AUTOENCODER FOR INTERACTION-AWARE BEHAVIOR PREDICTION

  • Conditional generative modeling fits p(y | x) to observed input-output pairs, with neural networks used to represent complex, high-dimensional distributions.The model parameters are fit by maximizing the likelihood of observed data.Source: A. Conditional Variational Autoencoder… · p. 3
  • A CVAE encodes input x as a latent distribution pθ(z | x), then decodes x and a latent sample into pφ(y | x, z) before marginalizing z.The tutorial focuses on a discrete latent space; the equations also apply to continuous latents by replacing summation with integration.Source: A. Conditional Variational Autoencoder… · p. 3
  • The proposal qϕ(z | x, y) is used to sample latent values likely to have produced y, supporting efficient marginalization of the latent variable.The proposal is parameterized, often as a neural network.Source: A. Conditional Variational Autoencoder… · p. 4
  • The evidence lower bound combines expected reconstruction log-likelihood with a divergence term and serves as a proxy for the intractable log p(y | x).The section says the bound is tractable with the reparameterization trick and can be optimized by stochastic gradient descent.Source: A. Conditional Variational Autoencoder… · p. 4
  • Training minimizes a Monte Carlo estimate of the expected loss over the training set.The per-example loss is the negative expected decoder log-likelihood plus a KL divergence term.Source: A. Conditional Variational Autoencoder… · p. 4

B. Interaction-aware Human Behavior Prediction

The model predicts future human trajectories from interaction history and candidate robot actions, using a discrete latent space for multimodal outcomes and flexible outputs for planning.

B. Interaction-aware Human Behavior Prediction

  • Conditioning on interaction history and a future robot trajectory lets the model represent coupled interactions and predict future human states or actions.The inputs can include features such as agent positions, velocities, and actions; maps or camera images can also be added.Source: B. Interaction-aware Human Behavior… · p. 4
  • A discrete latent space supplies trajectory modes, while autoregressive Gaussian mixture outputs model within-mode variation across successive steps.The latent space, not the Gaussian mixtures, is the main mechanism for multimodality; latent values are not guaranteed to have semantic meanings.Source: B. Interaction-aware Human Behavior… · p. 4
  • The model can provide empirical samples or analytic distribution parameters, with recurrent networks and extensions for multiple agents and heterogeneous inputs.Examples of heterogeneous inputs include state trajectories, images, and maps.Source: B. Interaction-aware Human Behavior… · pp. 4–5
  • Figure 3 depicts human action predictions conditioned on a robot action sequence, with colors indicating discrete latent modes.The figure is adapted from [3].Source: Figure 3 · p. 5

C. Traffic-Weaving Case Study

The traffic-weaving case study applies a CVAE to lane-swapping cars, illustrating how predicted human responses can inform robot action selection.

C. Traffic-Weaving Case Study

  • The case study reproduces a scenario where two side-by-side cars must swap lanes quickly, negotiating who passes whom; it uses an LSTM and predicts actions.Future robot actions are inputs, while other applications may use states instead of actions.Source: C. Traffic-Weaving Case Study · p. 5
  • The CVAE conditions on both agents’ interaction history and the future robot action sequence, then autoregressively samples human actions from decoder GMMs.Each sampled action is fed back into the LSTM decoder to generate the next action.Source: C. Traffic-Weaving Case Study · p. 5
  • The planner searches candidate robot action sequences for the highest expected reward, and the model-based planner was tested and validated in simulation and on a full-scale test vehicle.Figure 3 depicts different predicted human responses for candidate robot action sequences.Source: C. Traffic-Weaving Case Study · p. 5

IV. SCALING UP TO MULTI-AGENT INTERACTIONS

Real-world interactions can involve many agents, motivating extensions that handle both a general number of agents and their spatio-temporal relationships.

IV. SCALING UP TO MULTI-AGENT INTERACTIONS

  • Real-world settings such as crowds, intersections, and highway merges involve agents interacting with many others.Source: §IV · p. 5
  • The model from the previous section needs to extend beyond interactions among a limited set of agents.Source: §IV · p. 5
  • The extension must account for spatio-temporal relationships among a general number of agents.Source: §IV · p. 5

A. Modeling a General Number of Agents

The section represents agent interactions as a time-evolving graph and describes an edge-based architecture for variable numbers of neighbors, while noting that agents and interactions also change over time.

A. Modeling a General Number of Agents

  • A spatio-temporal graph represents agents as nodes and interactions as edges, with spatial proximity often used as an interaction proxy,,,.The cited formulation marks an edge when the distance between agents is at most a threshold d.Source: A. Modeling a General Number of Agents · p. 5
  • Graph abstraction supports applying similarly structured approaches across environments and domains, including human-object interaction modeling.Source: A. Modeling a General Number of Agents · p. 5
  • Because neighbor counts vary, the approach in adds an LSTM for each connecting edge and aggregates influences from neighboring nodes of the same type.The cited structure demonstrated that it can model neighboring agents’ influence.Source: A. Modeling a General Number of Agents · p. 5
  • Agent sets and interactions change over time as agents enter or leave sensor range and their spatial proximity shifts, so edge encoding must capture time-varying structure.Source: A. Modeling a General Number of Agents · pp. 5–6

B. Modeling Time-varying Interactions

Time-varying interactions make naive per-timestep graph rebuilding expensive. The section presents an edge-weighting approach and introduces dynamics and heterogeneous inputs as additional output-design considerations.

B. Modeling Time-varying Interactions

  • Rebuilding a scene graph at every timestep is expensive and fails to reuse information about edges that persist or change over time.Edges may be new, established, or recently removed.Source: B. Modeling Time-varying Interactions · p. 6
  • A recency-dependent scalar reweights edge-encoding LSTM outputs, filtering abrupt changes and supporting fast online updates from streaming observations.The filter can reject high-frequency noise from upstream perception systems.Source: B. Modeling Time-varying Interactions · p. 6

V. INCORPORATING AGENT DYNAMICS AND HETEROGENEOUS INPUT DATA

  • The following section considers trajectory feasibility through agent dynamics and incorporating additional inputs such as HD maps.Source: §V · p. 6

A. Producing Dynamically-feasible Outputs

Directly predicting positions can yield trajectories that violate agent dynamics. Predicting actions and integrating them through a dynamics model instead produces dynamically feasible trajectories while retaining position-based training.

A. Producing Dynamically-feasible Outputs

  • Position-output methods can produce trajectories that violate non-holonomic constraints, such as a car moving sideways.Many planning constraints and evaluation metrics are defined over spatial coordinates, making position outputs common.Source: A. Producing Dynamically-feasible… · p. 6
  • The dynamically-extended unicycle balances accuracy and efficiency for vehicles while requiring estimates only of position and velocity.It accounts for key vehicular non-holonomic constraints and uses four states and two actions.Source: A. Producing Dynamically-feasible… · p. 6
  • Predicting action distributions and integrating them through agent dynamics propagates action uncertainty into position uncertainty.For linear dynamics, such as a single integrator, the resulting system is linear Gaussian; nonlinear dynamics can be handled approximately by linearization.Source: A. Producing Dynamically-feasible… · p. 6
  • The model learns from ground-truth positions without additional training data, backpropagating through dynamics to produce dynamically feasible trajectory samples.This contrasts with methods that directly output positions.Source: A. Producing Dynamically-feasible… · p. 6

B. Incorporating Heterogeneous Data

Trajectory-only inputs omit other information available from robotic perception systems. The section describes encoding HD maps and other modalities for incorporation into behavior-prediction models.

B. Incorporating Heterogeneous Data

VI. EXPERIMENTS AND PRACTICAL CONSIDERATIONS

The experiments compare Trajectron++ with other trajectory-prediction methods on pedestrian and vehicle benchmarks. The section also offers latent-space guidance, noting that only a subset of latent variables may be needed.

A. Quantitative Performance

  • Trajectron++ significantly outperforms Social GAN and Social BiGAT on BoN ADE, BoN FDE, and KDE NLL on ETH and UCY pedestrian datasets.Table I compares the CVAE-based method with GAN-based methods for pedestrian modeling.Source: A. Quantitative Performance · p. 7
  • Trajectron++ is also evaluated against varied vehicle-modeling approaches on nuScenes, and phenomenological methods won all prizes in the ICRA 2020 nuScenes prediction challenge.One prize winner was Trajectron++.Source: A. Quantitative Performance · p. 7

B. Latent Space Size

  • The authors recommend allocating latent variables for high-level behaviors, starting with a large number when the required count is unclear.Their cited models use 25 latent variables, while assigning significant probability to only a few modes at a time.Source: B. Latent Space Size · p. 7
  • Using evidential theory to analyze CVAE learned weights, found that only 2−12 latent variables had direct evidence in, and the others could be pruned without any loss in performance.Source: B. Latent Space Size · p. 7

C. Online Model Runtime

The model can support real-time online prediction by updating its LSTM encoder with newly observed data. The conclusion highlights CVAE flexibility for heterogeneous inputs and agent dynamics, while identifying interpretability, sensor robustness, and downstream evaluation as areas for further work.

C. Online Model Runtime

  • An update-and-predict scheme reuses the LSTM encoder state so new observations can be incorporated without a full forward pass, supporting real-time online performance in,.Only the last LSTM cells in the encoder need to receive the newly observed data.Source: C. Online Model Runtime · p. 7

VII. CONCLUSIONS AND FUTURE WORK

  • The CVAE approach is described as attractive for multi-agent prediction with heterogeneous data and non-Markovian interaction histories, and flexible for incorporating agent dynamics and different planning algorithms.The conclusion also identifies further model improvements and open evaluation and architecture questions for integration with downstream planning and control.Source: §VII · p. 7
  • Future work includes improving latent-space interpretability, robustifying against upstream sensor noise, and generating more realistic simulation agents for testing and validation.The paper also identifies open questions about evaluation metrics and architecture choices for downstream planning and control.Source: §VII · pp. 7–8
Loading 2008.03880v2…