PaperlayerView exact revision on arXiv

Source-linked AI summary

Diverse Trajectory Forecasting with Determinantal Point Processes

Ye Yuan, Kris Kitani

arXiv:1907.04967v2cs.CVcs.LGcs.RO

TL;DR

Safety-critical trajectory forecasting needs likely yet diverse futures, because likelihood-based sampling can concentrate on a dominant mode. The paper learns a context-conditioned DSF that maps features to latent codes and optimizes decoded trajectories with a DPP-based diversity loss. Experiments on 2D trajectories and human motion report greater diversity than baseline forecasting approaches.

  • Problem

    Safety-critical systems need diverse possible futures, but VAE samples can concentrate around the major mode of an imbalanced training distribution.

  • Method

    A diversity sampling function maps forecasting context features to latent codes, decodes them into trajectories, and optimizes their set diversity with a DPP-based loss.

  • Results

    Experiments on synthetic 2D data and human motion show that DSF generates more diverse trajectories than state-of-the-art generative-model baselines.

  • Takeaways & Limitations

    DPP-based optimization provides a way to generate diverse trajectory sets in the continuous sample space of a generative model.

Abstract

from arXiv · show

The ability to forecast a set of likely yet diverse possible future behaviors of an agent (e.g., future trajectories of a pedestrian) is essential for safety-critical perception systems (e.g., autonomous vehicles). In particular, a set of possible future behaviors generated by the system must be diverse to account for all possible outcomes in order to take necessary safety precautions. It is not sufficient to maintain a set of the most likely future outcomes because the set may only contain perturbations of a single outcome. While generative models such as variational autoencoders (VAEs) have been shown to be a powerful tool for learning a distribution over future trajectories, randomly drawn samples from the learned implicit likelihood model may not be diverse -- the likelihood model is derived from the training data distribution and the samples will concentrate around the major mode that has most data. In this work, we propose to learn a diversity sampling function (DSF) that generates a diverse and likely set of future trajectories. The DSF maps forecasting context features to a set of latent codes which can be decoded by a generative model (e.g., VAE) into a set of diverse trajectory samples. Concretely, the process of identifying the diverse set of samples is posed as a parameter estimation of the DSF. To learn the parameters of the DSF, the diversity of the trajectory samples is evaluated by a diversity loss based on a determinantal point process (DPP). Gradient descent is performed over the DSF parameters, which in turn move the latent codes of the sample set to find an optimal diverse and likely set of trajectories. Our method is a novel application of DPPs to optimize a set of items (trajectories) in continuous space. We demonstrate the diversity of the trajectories produced by our approach on both low-dimensional 2D trajectory data and high-dimensional human motion data.

1 INTRODUCTION

Trajectory forecasting must represent diverse, multi-modal futures because safety-critical systems may need to prepare for less likely behaviors. The paper proposes DSF, which uses DPP-based optimization to generate diverse trajectory sets.

  • Safety-critical forecasting requires diverse possible trajectories, including less likely behaviors such as a vehicle merging into an adjacent lane.
  • The paper learns a diversity sampling function that produces diverse future trajectories from forecasting context features.
  • DPP-based diversity optimization helps the method generate trajectories that are less repetitive and less affected by imbalanced data.
  • Experiments on synthetic data and human motion show that DSF generates more diverse trajectories than state-of-the-art generative models.

2 RELATED WORK

Prior trajectory-forecasting work includes deterministic predictions, distributional models, and broader approaches for generating diverse solutions. These lines of work motivate forecasting multiple plausible futures rather than a single outcome.

  • Trajectory forecasting research includes deterministic models that predict one future and distributional approaches that represent multiple plausible trajectories.
  • Diversity methods include diverse M-Best solutions, multiple choice learning, submodular subset selection, and determinantal point processes.

3 BACKGROUND

VAEs model future trajectories through latent codes and decoder likelihoods, while DPPs evaluate subsets using diversity and quality. The paper uses DPPs because they capture global diversity within trajectory sets.

  • 3.1 VARIATIONAL AUTOENCODERS: VAE training jointly optimizes an encoder and decoder by maximizing the evidence lower bound, which addresses the intractable posterior.
  • 3.1 VARIATIONAL AUTOENCODERS: VAEs model trajectories with latent codes and generate future samples by decoding codes drawn from a prior distribution.
  • 3.2 DETERMINANTAL POINT PROCESSES: DPPs assign probabilities to subsets through a positive semidefinite kernel and favor sets whose elements are less similar.
  • 3.2 DETERMINANTAL POINT PROCESSES: A DPP kernel combines item quality weights with a similarity matrix to evaluate both quality and diversity.
  • 3.2 DETERMINANTAL POINT PROCESSES: The paper selects DPPs to evaluate and optimize future-trajectory sets because they capture global diversity and quality.

4 APPROACH

The approach learns a deterministic diversity sampling function that maps context to latent codes, decodes them into trajectories, and optimizes their diversity with a DPP-based objective. It balances trajectory diversity with sample quality and uses DPP inference to remove redundant trajectories.

  • 4.1 LEARNING A CVAE FOR FUTURE TRAJECTORIES: The cVAE supplies the trajectory model by encoding context-conditioned trajectories into latent codes and decoding sampled prior codes into future trajectories.The cVAE is trained with a reconstruction term and KL divergence, after which prior sampling enables efficient trajectory generation.
  • 4.2 DIVERSITY SAMPLING FUNCTION (DSF): The DSF maps forecasting context to N latent codes, which a cVAE decoder converts into a ground set of future trajectories.The DSF is parameterized by a neural network, and N is the sampling budget.
  • 4.2 DIVERSITY SAMPLING FUNCTION (DSF): A DPP kernel combines trajectory similarity with per-trajectory quality to define diversity-aware sampling over the DSF outputs.Similarity is measured in trajectory space, while quality is measured in latent space.
  • 4.2 DIVERSITY SAMPLING FUNCTION (DSF): The quality metric treats latent samples inside a prior-defined sphere equally, reducing preference for major modes while penalizing samples far from the data manifold.The radius is chosen so 90% of Gaussian prior samples lie within the sphere; its size controls the diversity–quality trade-off.
  • 4.2 DIVERSITY SAMPLING FUNCTION (DSF): The DSF is optimized with expected DPP cardinality rather than DPP log likelihood because repeated trajectories can make the latter numerically unstable.Expected cardinality increases when DPP subsets can select more mutually dissimilar trajectories.
  • 4.2 DIVERSITY SAMPLING FUNCTION (DSF): At inference, greedy MAP estimation selects a diverse subset from the generated ground set by exploiting the submodularity of DPP log probability.Selection continues until marginal gain becomes negative or the ground set is exhausted.

5 EXPERIMENTS

Experiments evaluate whether DSF produces diverse, accurate trajectory sets across synthetic 2D data, human motion, and Human3.6M, using diversity-aware metrics and multiple baselines.

  • 5 EXPERIMENTS: Evaluation uses ADE and FDE for accuracy and ASD and FSD to measure repetition and diversity among forecasted trajectories.The study also compares DSF with cVAE, MCL, R2P2, and cGAN using matched networks and hyperparameter searches.
  • 5 EXPERIMENTS: The experiments test diversity across balanced and imbalanced low-dimensional data and high-dimensional human-motion tasks.Synthetic data models three possible routes, while human-motion experiments forecast pose sequences from motion-capture data.
  • 5.1 SYNTHETIC 2D TRAJECTORY DATA: DSF outperforms baselines in all metrics on balanced and imbalanced synthetic data at N = 10.Its trajectories are more diverse, less affected by imbalanced data, and less repetitive under the DPP formulation.
  • 5.2 DIVERSE HUMAN MOTION FORECASTING: Our method outperforms other methods in all metrics for human motion forecasting at N = 10.Table 2 reports the quantitative comparison, while qualitative results show more diverse future human motions than baselines.

6 CONCLUSION

The paper presents a DSF-based forecasting approach that uses a DPP diversity measure to generate diverse trajectory sets, with experiments showing greater diversity than state-of-the-art baselines.

  • The method generates more diverse vehicle trajectories and human motions than state-of-the-art baseline forecasting approaches.
  • The approach learns a DSF to optimize over a generative model’s sample space and produce diverse trajectory sets.
  • The diversity measure applies DPPs to optimize a set of trajectories in continuous space.

A ALGORITHMS

The algorithms train a conditional VAE and use a DSF with a DPP kernel to construct a forecasted trajectory set.

  • The cVAE training algorithm maps context and trajectory data to encoder and decoder networks through posterior sampling and trajectory reconstruction.
  • The forecasting algorithm generates latent codes with the DSF, decodes them into a trajectory ground set, and forms a DPP kernel from quality and similarity terms.
  • A greedy procedure repeatedly selects the candidate maximizing the log-determinant objective and removes it from the remaining set.

B IMPLEMENTATION DETAILS

The implementation uses different network architectures for synthetic trajectories and human motion, with CNN-based processing for synthetic data and recurrent processing for human motion.

  • Synthetic data uses a CNN for the obstacle map, flattened trajectory features, and an MLP decoder for reconstructed trajectories.
  • Human motion uses Bi-LSTMs to extract temporal features and a forward LSTM to decode reconstructed trajectories.

B.1 NETWORK ARCHITECTURES

The network architectures adapt the context encoder, trajectory decoder, and DSF to the dimensionality and temporal structure of synthetic vehicle data and human motion.

  • Synthetic data: Synthetic trajectories use 2D latent codes, CNN features from obstacle maps, flattened history, and an MLP DSF producing N latent codes.
  • Human motion: Human motion uses 8D latent codes and Bi-LSTMs with mean pooling to encode future and past poses.
  • Human motion: A forward LSTM decodes each human-motion pose from the previous pose, latent code, and temporal context features.
  • Human motion: The human-motion DSF uses a separate Bi-LSTM for past-pose features before producing latent codes through an MLP.

B.2 TRAINING AND EVALUATION

Training uses separate optimization settings for the cVAE and DSF, while evaluation groups ground-truth futures using context-dependent thresholds.

  • B.2 TRAINING AND EVALUATION: The cVAE uses Adam with learning rate 1e-4, batch size 32, and 500 or 100 epochs for synthetic data and human motion, respectively.The KL weighting β is 0.1 for synthetic data and 1e-4 for human motion, with one posterior sample during training.
  • B.2 TRAINING AND EVALUATION: The DSF uses Adam at learning rate 1e-4 for 20 epochs, with similarity-matrix scale k set to 1 for synthetic data and 1e-2 for human motion.
  • B.2 TRAINING AND EVALUATION: Ground-truth future sets use grouping threshold ε=0.1 for synthetic data and ε=0.5 for human motion, with synthetic context distances computed from past trajectories.

B.3 IMPLEMENTATION DETAILS FOR EXPERIMENTS ON HUMAN3.6M

The Human3.6M experiment represents motions with a 17-joint 3D skeleton, trains on five subjects, and evaluates on two held-out subjects using specified cVAE and DSF settings.

  • B.3 IMPLEMENTATION DETAILS FOR EXPERIMENTS ON HUMAN3.6M: Human3.6M motions are represented as 3D joint-position sequences with a 17-joint skeleton, training on S1, S5, S6, S7, and S8 and testing on S9 and S11.
  • B.3 IMPLEMENTATION DETAILS FOR EXPERIMENTS ON HUMAN3.6M: The cVAE uses 128 latent dimensions, β=0.1, 5000 training examples per epoch, 500 epochs, learning rate 1e-4, and batch size 64.
  • B.3 IMPLEMENTATION DETAILS FOR EXPERIMENTS ON HUMAN3.6M: The DSF uses similarity scale k=5, batch size 64, 1000 training examples per epoch, 20 epochs, and Adam with learning rate 1e-3.
  • B.3 IMPLEMENTATION DETAILS FOR EXPERIMENTS ON HUMAN3.6M: The grouping threshold used when computing the metrics is ε=0.1.

C ADDITIONAL VISUALIZATION

Additional human-motion visualizations compare forecast quality and diversity across methods by showing the starting pose and the final poses of 10 forecasted samples.

  • C ADDITIONAL VISUALIZATION: The additional human-motion results visualize forecast quality and diversity, which the paper says are best seen in the accompanying video.
  • C ADDITIONAL VISUALIZATION: Figure 7 shows each method’s final pose for 10 forecasted motion samples alongside the starting pose.
Loading 1907.04967v2…