PaperlayerView exact revision on arXiv

Source-linked AI summary

MG-GAN: A Multi-Generator Model Preventing Out-of-Distribution Samples in Pedestrian Trajectory Prediction

Patrick Dendorfer, Sven Elflein, Laura Leal-Taixé

arXiv:2108.09274v1cs.CV

TL;DR

Pedestrian trajectory prediction must handle future distributions with multiple disconnected modes, because single-generator GANs can produce unrealistic out-of-distribution paths. MG-GAN assigns specialized generators to modes and learns scene-conditioned probabilities over them; its evaluation reports substantially higher precision while maintaining similar recall, reducing OOD behavior relative to single-generator methods.

  • Problem

    Single-generator trajectory models struggle with target distributions containing disconnected modes, producing unrealistic out-of-distribution samples that matter for safety-critical navigation.

  • Method

    MG-GAN uses multiple mode-specialized generators and a second module that learns a scene-conditioned categorical distribution over those generators.

  • Results

    MG-GAN outperforms GAN+L2 by 29%, PECNet by 54% and Trajectron++ by 86% in Precision, while Recall differences are 0.02, 0.06, and 0.07 points, respectively.

  • Takeaways & Limitations

    Precision and recall together expose OOD behavior that minimum-distance metrics alone do not capture, while MG-GAN reduces OOD samples compared with existing state-of-the-art methods.

  • Takeaways & Limitations

    The approach assumes real-world target distributions can be multimodal and disconnected, and its training formulation assumes additive normally distributed error ϵ ∼N(0, σI).

Abstract

from arXiv · show

Pedestrian trajectory prediction is challenging due to its uncertain and multimodal nature. While generative adversarial networks can learn a distribution over future trajectories, they tend to predict out-of-distribution samples when the distribution of future trajectories is a mixture of multiple, possibly disconnected modes. To address this issue, we propose a multi-generator model for pedestrian trajectory prediction. Each generator specializes in learning a distribution over trajectories routing towards one of the primary modes in the scene, while a second network learns a categorical distribution over these generators, conditioned on the dynamics and scene input. This architecture allows us to effectively sample from specialized generators and to significantly reduce the out-of-distribution samples compared to single generator methods.

1. Introduction

Pedestrian trajectory prediction must represent multiple plausible futures while avoiding unrealistic paths between disconnected modes. MG-GAN addresses this with specialized generators and a scene-conditioned generator-selection distribution, evaluated using precision and recall.

  • Future pedestrian motion is multimodal because scene layout and social interactions permit several plausible paths.
  • Single-generator GANs produce out-of-distribution trajectories between disconnected trajectory modes because continuous latent mappings cannot represent disconnected supports.
  • MG-GAN uses multiple continuous generators, each specializing in one trajectory mode, to model disconnected multimodal distributions.
  • A second module estimates scene-conditioned categorical probabilities over generators, enabling mode selection across scenes with different junction structures.
  • MG-GAN evaluates predictive-distribution quality with precision and recall and reports improved OOD behavior over single-generator and state-of-the-art methods.

2. Related Work

Prior trajectory-prediction methods improved multimodal forecasting but often emphasized coverage while overlooking unrealistic out-of-distribution samples. MG-GAN applies conditional mixtures of generators to learn disconnected trajectory distributions and conditions generator probabilities on the scene.

  • Trajectory forecasting progressed from handcrafted and deterministic models toward generative methods that produce distributions of possible futures.
  • Diversity-promoting GAN objectives improve mode coverage but can generate unrealistic out-of-distribution trajectories that minimum-distance metrics overlook.
  • VAEs also use continuous transformations and therefore face limitations when modeling distributions on disconnected manifolds.
  • A single generator preserves continuous latent-space topology and cannot exclusively generate samples on disconnected manifolds.
  • Existing generator mixtures often assume uniform or unconditional probabilities, whereas MG-GAN models a conditional distribution over generators for pedestrian trajectory prediction.

3. Problem Definition

The task is to generate multiple future trajectories for each pedestrian when the target distribution may contain disconnected modes. The problem formulation identifies continuous single-generator mappings as the source of OOD samples and motivates evaluating both coverage and realism.

  • For each pedestrian, the model generates K future trajectories conditioned on the observed trajectory and scene layout.
  • Real-world trajectory distributions can be multimodal and composed of disconnected modes.
  • Standard conditional GANs learn continuous mappings from observations and latent variables to future trajectories.
  • Unrealistic predictions can cause autonomous systems to take harmful actions, so forecasts should cover all modes while keeping unsupported samples few.
  • Preventing OOD samples requires either disconnected latent spaces or discontinuous generator mappings, with the paper finding discontinuous mappings more effective.
  • Recall measures mode coverage, while precision captures generated-sample quality and correlates with the number of OOD samples.

4. Method

MG-GAN models disconnected multimodal trajectory distributions with multiple specialized generators and a scene-conditioned categorical network that selects among them. Its alternating training procedure jointly learns generator distributions and generator-selection probabilities, while evaluation emphasizes both trajectory coverage and avoidance of OOD samples.

  • 4.1. MG-GAN: MG-GAN represents the target distribution as a mixture of distributions modeled by multiple generators, enabling discontinuous trajectory prediction.Each generator specializes in a different trajectory distribution conditioned on encoded scene and pedestrian features.
  • 4.1. MG-GAN: The visual and trajectory encoders combine dynamic, physical-attention, and social-attention features into the conditioning representation c.These features are extracted from observed trajectories and scene image patches before being concatenated.
  • 4.1. MG-GAN: Each generator uses an LSTM decoder initialized by c and random noise z, producing a trajectory distribution with its own parameters.The generators share an architecture but not weights, allowing them to specialize in different modes.
  • 4.2. PM-Net: PM-Net predicts a scene-conditioned categorical distribution p(g|c) over generators, allowing inference to sample or select generators according to probabilities π.Unlike fixed generator distributions, PM-Net adapts generator activation to scene observations and interactions.
  • 4.3.1 GAN Training: The adversarial loss encourages realism, while a classifier loss encourages different generators to model non-overlapping trajectory modes.The model also applies a best-of-many L2 loss to increase predicted-trajectory diversity.
  • 4.3.2 PM-Net Training: PM-Net training estimates generator support for ground-truth trajectories using generated samples, a Gaussian additive-error assumption, Bayes’ rule, and cross-entropy optimization.The resulting network weighs most highly the generator whose predictions are closest to the ground-truth sample.
  • 4.3.2 PM-Net Training: Training alternates between updating PM-Net with sampled trajectories and updating the generators using PM-Net probabilities while excluding PM-Net parameters.This procedure is described as analogous to expectation-maximization and supports joint end-to-end optimization.

5. Experimental Evaluation

The evaluation uses real and synthetic pedestrian datasets, distribution-aware metrics, and comparisons against single- and multi-generator baselines. MG-GAN improves coverage and precision while reducing out-of-distribution samples, with gains under matched resource budgets and robustness to generator count.

  • Experimental setup: The evaluation covers four public datasets, synthetic data with ground-truth trajectory distributions, and ablations of MG-GAN components.Synthetic data enables direct identification of out-of-distribution samples by comparing generated and ground-truth distributions.
  • Metrics: Recall measures coverage of ground-truth modes, while precision measures generated samples within the ground-truth support and therefore relates directly to OOD samples.ADE and FDE use the minimum-over-k procedure with k = 20 and do not evaluate the complete generated set.
  • Synthetic-data results: MG-GAN increases Recall by 0.28 and Precision by 0.32 over single-generator baselines, while multi-generator methods achieve similar recall and MG-GAN obtains 15% higher Precision.The synthetic experiment evaluates multimodal distributions with disconnected support and reports greater efficiency than baselines.
  • Synthetic-data results: MG-GAN covers more ground-truth modes than a single-generator model at the same sample count, with expectation sampling especially improving recall when samples are scarce.The comparison varies the number of samples k and contrasts expectation sampling with random sampling.
  • Efficiency: With matched total parameters, MG-GAN improves ADE/FDE by 50% and recall/precision by 30% over a single-generator GAN, while using fewer MACs during prediction.Each of four generators uses approximately one quarter of the baseline parameters, and only one selected generator runs during the forward pass.
  • Benchmark results: On ETH/UCY and SDD, MG-GAN is competitive on ADE/FDE but does not achieve state-of-the-art performance on these traditional metrics.The paper argues that distance metrics can favor high-variance predictions that produce more OOD samples.
  • Benchmark results: On benchmark distribution metrics, MG-GAN improves Precision by 29% over GAN+L2, 54% over PECNet, and 86% over Trajectron++, with recall differences of 0.02, 0.06, and 0.07 points.The model maintains similar mode coverage while lowering OOD samples and achieving a better overall F1 than current state-of-the-art methods.
  • Ablation studies: PM-Net makes MG-GAN robust to the number of generators, with results deviating at most 7% from the best reported values across nG ∈ {2, . . . , 8}.PM-Net learns to activate generators according to observed scene features because multimodality depends on social interactions and layout.

6. Conclusion

MG-GAN addresses the inability of single-generator GANs to model disconnected multimodal trajectory distributions by using multiple mode-specialized generators and scene-conditioned sampling. The paper reports reduced out-of-distribution samples and emphasizes precision alongside recall.

  • MG-GAN uses multiple generators specialized to different trajectory modes and samples from them conditioned on the scene observation.
  • MG-GAN reduces out-of-distribution samples compared with existing state-of-the-art methods.
  • The paper argues that precision should be considered alongside recall when evaluating pedestrian trajectory prediction.

A. Architecture

The architecture combines scene and trajectory encoders, multiple independently weighted generators, and a Path Mode Network that predicts generator probabilities. A GAN discriminator and classifier support adversarial training and generator specialization, while likelihood-based training estimates mode assignments.

  • Architecture: MG-GAN uses multiple generators and a Path Mode Network to specialize predictions by trajectory mode.The generators use LSTM decoders, while PM-Net outputs a categorical distribution over generators conditioned on encoded scene features.
  • Encoding: Dynamic, visual, social, and physical features are combined into a conditional encoding for trajectory generation.LSTMs encode relative displacements, CNNs encode scene patches, and soft-attention modules model pedestrian-pedestrian and pedestrian-scene interactions.
  • Generators: Each generator uses an LSTM decoder initialized with encoded scene features and Gaussian noise, then predicts trajectories recurrently.
  • Training: The discriminator distinguishes real from predicted trajectories, while the classifier predicts which generator produced a trajectory.Both branches use scene and trajectory encodings and produce outputs through sigmoid or softmax activations.
  • Training: Training alternates between optimizing PM-Net from approximated generator likelihoods and optimizing generators with adversarial, best-of-many, and classification losses.
  • PM-Net objective: The likelihood model assumes normally distributed trajectory errors and uses Monte Carlo samples to estimate generator probabilities.Bayes’ rule yields a posterior distribution over generators, with a uniform prior used at the start of training.

B.2. Hyperparameters

The appendix specifies optimization settings, studies loss and sampling hyperparameters, and defines manifold-based precision and recall for evaluating realism and mode coverage. The analyses link training choices to ADE, precision, recall, and multimodal behavior, though the supplied passages do not include the table values.

  • Hyperparameters: Adam optimization uses learning rate 0.001, β1 = 0.5, β2 = 0.999, q = 20, l = 1, and σ = 1.
  • Loss weighting: Higher λTraj slightly improves ADE, Precision, and Recall by encouraging close samples and further generator specialization.
  • PM-Net training: As σ approaches zero, generator probabilities become one-hot for the generator producing samples closest to the ground truth, while large σ makes them uniform.The appendix states that a suitable σ is necessary for convergence to a solution with high precision.
  • Training samples: A single generator-training sample q = 1 is insufficient for multimodal learning and produces linear straight motion.Increasing q enables more multimodal predictions, according to the supplied analysis.
  • Precision and Recall: Precision tests whether predictions lie inside the ground-truth manifold, whereas recall tests whether ground-truth samples lie inside the prediction manifold.The manifolds are estimated by unions of discs around trajectory points, with Rmax = 2m.

D. Synthetic Dataset

The synthetic dataset simulates pedestrians in the Hyang-4 scene, whose junctions provide separated spatial modes. An occupancy map constrains walkable regions to create multiple possible futures from the same observation.

  • Dataset construction: The synthetic dataset is generated on the Hyang-4 scene of the Stanford Drone Dataset.
  • Dataset construction: Hyang-4 provides separated spatial modes through an upper junction with two modes and a lower junction with three modes.
  • Dataset construction: The dataset simulates approximately 80,000 pedestrians using the Social Force Model.
  • Dataset construction: An occupancy map restricts where pedestrians can walk, controlling the modes of their future trajectories.The scene image and occupation map are shown as the dataset’s visual inputs and constraints.

E. Multimodality of Real Datasets

The datasets differ in estimated multimodality, with SDD less multimodal than FPD and UCY/ETH. This helps explain why MG-GAN shows less advantage over single-generator methods on SDD.

  • Estimation procedure: The multimodality estimate counts disconnected manifold components across timesteps.The manifold is estimated from the collected trajectory set using the procedure described in Appendix C.
  • Estimated multimodality: SDD has fewer estimated modes than FPD and UCY/ETH, with averages of 1.15, 1.36, and 1.34, respectively.Modes are estimated from trajectories that are spatially, directionally, and speed-wise similar, after filtering collisions.
  • Benchmark comparison: MG-GAN achieves state-of-the-art performance on ETH and UCY, which are more multimodal than SDD.The authors note that greater multimodality makes it harder to show benefits over existing methods on SDD.
  • Benchmark comparison: On SDD, the single-generator GAN+L2 baseline obtains ADE 14.6 and FDE 27.5, with similar results to MG-GAN.The comparison uses the same backbone for both models.

F. Toy Experiment

The toy dataset contains multiple trajectory modes from six circularly arranged starting positions. The experiment compares how models represent these modes in predictions and lower-dimensional latent spaces.

  • Dataset and setup: The toy dataset uses six equidistant starting positions on a circle, with three paths generated uniformly from each position.The setup is used to study representation of multiple modes in a lower-dimensional latent space.
  • Results: The simple GAN baseline fails to recover all three modes.The results visualize both predicted trajectories and their corresponding latent-space vectors.
  • Results: Adding an L2 loss produces unrealistic out-of-distribution samples because it encourages samples to spread across the output space.The same spreading behavior is also reflected in the latent space.
  • Results: InfoGAN does not encode different modes in its categorical values.

G. Visualizations

The visualizations examine generated trajectories, latent-space behavior, and the training procedure across datasets and model variants. They show that MG-GAN generators remain associated with specific modes, unlike the single-generator baseline during latent-space interpolation.

  • Generated trajectories: MG-GAN generated trajectories are visualized on ETH, UCY, SDD, and the Forking Path dataset.These visualizations correspond to Figures 11–13.
  • Latent-space behavior: Latent-space interpolation produces out-of-distribution samples for the GAN baseline, whereas each MG-GAN generator stays within a specific mode’s support.The comparison is shown for the single generator and five individual MG-GAN generators.
  • Toy visualization: Figure 10 compares model predictions with corresponding latent-space vectors, using different colors for discrete latent codes or generators.
  • Training procedure: Algorithm 1 describes training MG-GAN by optimizing the generators, classifier, and PM-Net with Adam updates.The training specification includes trajectory-loss and generator-classification regularization weights, along with generator and PM-Net sample counts.
  • Training procedure: The training inputs include observed trajectories, an image crop, and ground-truth observations sampled from real data.
Loading 2108.09274v1…