PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Trajectory Forecasting in Crowds: A Deep Learning Perspective

Parth Kothari, Sven Kreiss, Alexandre Alahi

arXiv:2007.03639v3cs.CV

TL;DR

Human trajectory forecasting needs models that capture subtle social interactions and evaluate whether predictions are socially and physically acceptable. The paper analyzes interaction modules, proposes two domain-knowledge-based designs, and develops TrajNet++ with collision-oriented metrics. Its experiments show that the proposed methods outperform competitive baselines on synthetic data and reduce collisions on real data, while distance-based metrics alone may miss meaningful behavioral differences.

  • Problem

    Existing evaluations use different data subsets and lack definitive, interaction-centric comparisons and metrics for physical feasibility and social behavior.

  • Method

    The paper analyzes neural-network interaction encoders, proposes two domain-knowledge-based interaction models, and introduces TrajNet++ with novel behavior-oriented evaluation metrics.

  • Results

    The proposed models outperform competitive baselines on TrajNet++ synthetic data and show significant collision-reduction gains on real data, although no clear real-data winner emerges on distance-based metrics.

  • Takeaways & Limitations

    Collision-based metrics provide a more complete picture of socially acceptable forecasting than ADE/FDE alone, especially for safety-critical scenarios.

Abstract

from arXiv · show

Since the past few decades, human trajectory forecasting has been a field of active research owing to its numerous real-world applications: evacuation situation analysis, deployment of intelligent transport systems, traffic operations, to name a few. Early works handcrafted this representation based on domain knowledge. However, social interactions in crowded environments are not only diverse but often subtle. Recently, deep learning methods have outperformed their handcrafted counterparts, as they learned about human-human interactions in a more generic data-driven fashion. In this work, we present an in-depth analysis of existing deep learning-based methods for modelling social interactions. We propose two knowledge-based data-driven methods to effectively capture these social interactions. To objectively compare the performance of these interaction-based forecasting models, we develop a large scale interaction-centric benchmark TrajNet++, a significant yet missing component in the field of human trajectory forecasting. We propose novel performance metrics that evaluate the ability of a model to output socially acceptable trajectories. Experiments on TrajNet++ validate the need for our proposed metrics, and our method outperforms competitive baselines on both real-world and synthetic datasets.

I. INTRODUCTION

Human trajectory forecasting must model subtle social interactions and produce socially and physically acceptable futures. This paper analyzes interaction encoders, proposes domain-knowledge methods, and introduces TrajNet++ with behavior-oriented evaluation.

  • Human motion forecasting must account for social interactions because surrounding pedestrians affect an individual’s future trajectory.
  • Physically acceptable predictions, including collision avoidance, are crucial for safety-critical applications.
  • The paper analyzes neural-network interaction encoders and proposes two domain-knowledge-based modules for learning social etiquettes.
  • TrajNet++ is an interaction-centric benchmark with trajectory categorization, extensive evaluation, and metrics for physical feasibility and socially acceptable behavior.
  • II. RELATED WORK: Deep-learning interaction models are motivated by the limitations of handcrafted approaches for diverse and subtle crowd interactions.

III. PROBLEM STATEMENT

The forecasting task maps observed trajectories of all pedestrians to future trajectories through motion encoding, interaction modeling, and decoding. The proposed directional pooling represents neighbors by relative velocity to capture collision avoidance and leader-follower behavior.

  • III. PROBLEM STATEMENT: The network takes trajectories X of all pedestrians as input and forecasts corresponding future trajectories Y.
  • III. PROBLEM STATEMENT: The pipeline combines motion encoding, an interaction module, and a decoder that outputs a trajectory or trajectory distribution.
  • 1) Grid Based Interaction Models:: Grid-based interaction models represent neighbors in cells arranged by spatial position relative to the primary pedestrian.
  • 1) Grid Based Interaction Models:: Occupancy pooling stores neighbor presence, whereas social pooling stores neighbors’ past histories through representations such as LSTM hidden states.
  • 1) Grid Based Interaction Models:: Directional pooling stores each neighbor’s relative velocity, encoding leader-follower and collision-avoidance cues that depend on motion direction.
  • 1) Grid Based Interaction Models:: Directional pooling uses a smaller N×N×2 input than social pooling’s N×N×H_dim representation, making it computationally faster for real-time deployment.

2) Non-Grid Based Interaction Models:

Non-grid interaction modules encode neighbours without an implicit spatial grid, requiring strategies for variable neighbour counts and collective information aggregation. The taxonomy distinguishes input states, embeddings, aggregation mechanisms, and aggregated-vector embeddings, including the proposed DirectConcat design.

  • Non-grid modules handle variable numbers of neighbours and aggregate their state information into an interaction vector.
  • Their design choices cover neighbour input state, input-state embedding, aggregation strategy, and aggregated-vector embedding.
  • Neighbour states may use relative position, hidden state, geometric features, velocity, or relative velocity, with relative velocity proposed to reduce collisions.
  • Aggregation mechanisms include attention, symmetric max- or sum-pooling, and concatenation; attention can use handcrafted or learned neighbour weights.
  • The concatenation strategy performs at par with sophisticated aggregation counterparts when top-k neighbours are selected by a defined criterion.
  • DirectConcat: DirectConcat selects the top-k closest neighbours, concatenates their relative-velocity and relative-position embeddings, and processes them with an LSTM.

B. Forecasting Model

The forecasting model combines motion encoding, interaction encoding, and decoding while holding the architecture constant across interaction-module designs. It trains on the primary pedestrian and predicts future velocity distributions autoregressively.

  • The common architecture uses a sequence encoder, interaction module, and decoder, with the interaction representation combined with motion information.
  • The LSTM sequence encoder handles varying input lengths and captures long-term dependencies, with shared weights across scene participants.
  • Each pedestrian’s state is represented using velocity, embedded with an MLP, and concatenated with the interaction vector before sequence encoding.
  • The decoder predicts the next velocity as a bivariate Gaussian distribution parameterized by its mean and covariance-related terms.
  • Training minimizes negative log-likelihood for the primary pedestrian rather than all trajectories in each scene.
  • During testing, ground-truth positions are used through T_obs, after which predicted positions provide inputs for forecasting future trajectories.

1) Equivalence to Graph Neural Networks:

The paper identifies dynamic-interaction graph neural networks as a special case of its LSTM-based interaction pipeline. This equivalence maps pedestrians to vertices, pairwise interactions to edges, and neighbour aggregation to attention pooling.

  • In the graph formulation, pedestrians are vertices and interacting pedestrian pairs are connected by edges that update over time.
  • A vertex uses an LSTM sequence encoder, while each neighbour edge embeds and updates relative state with an MLP or LSTM.
  • Attention aggregates connected-edge information at each time step, corresponding to GAT-pooling in graph neural network terminology.
  • The resulting aggregated vector may pass through an MLP to form the interaction vector supplied to the primary pedestrian’s sequence encoder.
  • The paper extends layer-wise relevance propagation to trajectory-forecasting regression by reverse-propagating both predicted velocity components and summing input relevance scores.
  • The proposed LRP analysis identifies which neighbours and past velocities influence the next predicted velocity and can be applied to any trajectory forecasting network.

V. TRAJNET++: A TRAJECTORY FORECASTING BENCHMARK

TrajNet++ is an interaction-centric benchmark built by categorizing scenes and sampling mainly interacting trajectories. Its hierarchy separates static, linear, non-linear, interacting, and non-interacting cases, with interacting scenes further subdivided by social behaviour.

  • TrajNet++ addresses inconsistent prior sampling by constructing a benchmark containing largely scenes where social interactions occur.
  • TrajNet++ Construction: The hierarchy supports sampling trajectories and examining model performance across diverse interaction scenarios.
  • Trajectory Categorization: A primary pedestrian is the reference pedestrian whose motion is forecast, while other pedestrians are treated as neighbours.
  • Trajectory Categorization: The categorization hierarchy labels trajectories as Static, Linear, or Non-Linear according to displacement and extended Kalman filter forecastability.
  • Trajectory Categorization: Interacting trajectories are Type III scenes in which the primary pedestrian undergoes social interactions.
  • Trajectory Categorization: Type III is divided into Leader Follower, Collision Avoidance, Group, and Other Interactions subcategories.
  • TrajNet++ Construction: The benchmark samples mainly Type III trajectories and removes many Type-I scenes because they can hamper training and mislead evaluation.

B. Evaluation Metrics

Unimodal evaluation uses ADE and FDE for trajectory distance, while the proposed Col-I and Col-II metrics assess collision-related physical feasibility and mode matching. Multimodal evaluation adds top-k distance metrics and Average NLL for prediction distributions.

  • ADE averages L2 distance across predicted time steps, whereas FDE measures final-destination distance at the prediction horizon.
  • Col-I measures collisions between the primary pedestrian and neighbours in forecasted trajectories, indicating collision avoidance.
  • Col-II measures collisions between the primary pedestrian and neighbours in ground-truth trajectories, indicating whether the predicted mode matches neighbours’ intentions.
  • Col-I complements ADE/FDE because a physically feasible trajectory can differ from ground truth and therefore receive a large distance error.
  • Top-k ADE and Top-k FDE select the prediction closest to ground truth among k outputs, while Average NLL evaluates ground-truth likelihood in the prediction distribution.

A. Implementation Details

Experiments compare interaction modules on controlled synthetic data and interaction-centric real-world trajectories, using collision metrics to expose differences that distance metrics may miss. The analyses emphasize domain-informed inputs and aggregation choices.

  • Synthetic experiments use goal direction alongside velocity because ORCA provides each pedestrian’s goal.
  • Col-I provides a more complete performance picture than distance metrics, especially when collision prevention matters in safety-critical scenarios.
  • D-Grid outperforms O-Grid, especially on Col-I, while explicitly providing neighbours’ relative velocity improves collision avoidance.
  • Concatenating neighbourhood information performs better than attention and max-pooling alternatives in the reported non-grid comparison.
  • On real-world interacting trajectories, Col-I differentiates model designs while ADE/FDE performance remains similar across methods.

1) Classical Methods:

Classical and neural baselines are compared on interaction-centric real-world data, with interaction-based neural models outperforming handcrafted models on distance metrics. Domain-informed designs improve collision avoidance but can trade some distance-based accuracy for safer predictions.

  • Classical Methods:: Interaction-based neural models outperform handcrafted models on distance-based metrics because they learn subtle and diverse social interactions.
  • Interaction Models:: D-Grid performs better than O-Grid in real-world experiments, while its explicit relative-velocity focus supports collision avoidance.
  • Interaction Models:: D-Grid’s simpler design slightly reduces distance-based accuracy by limiting expressibility, but produces safer predictions.
  • Aggregation Strategy:: Max-pooling performs worst among evaluated aggregation strategies, while concatenation is only slightly worse than attention.
  • Comparison to Vanilla LSTM:: Interaction-based models outperform Vanilla LSTM on distance metrics, while Col-I indicates their selected primary-pedestrian modes can yield collision-free scenes.

7) Understanding NN decision-making:

LRP is used to inspect whether interaction modules attend to neighbours in human-intuitive ways. The analysis links D-Grid’s lower Col-I to shifting attention toward potential colliders, while the benchmark and models support more objective comparison.

  • 7) Understanding NN decision-making:: LRP assigns relevance scores to neighbours at each timestep, enabling inspection of neural-network decision-making.
  • 7) Understanding NN decision-making:: In one scene, D-Grid focuses on a distant potential collider while S-Grid focuses on a less relevant neighbour.
  • 7) Understanding NN decision-making:: In a complex scene, D-Grid shifts focus among neighbours as collision risks and proximity change.
  • 7) Understanding NN decision-making:: D-Grid combines lower Col-I with decision-making that satisfies human intuition, and LRP can be applied to trained interaction architectures.
  • VII. CONCLUSIONS: TrajNet++ is interaction-centric and includes collision metrics that measure the physical feasibility of model predictions.
  • VII. CONCLUSIONS: The proposed models outperform competitive baselines on synthetic data and significantly reduce prediction collisions on real-world data.

A Deep Learning Perspective

The benchmark categorizes pedestrian trajectories by observable motion and interaction conditions, while selecting datasets that emphasize agent-agent interactions rather than scene constraints. It combines widely used and newer real-world datasets spanning diverse crowd settings and interaction patterns.

  • Trajectory categorization: Trajectories are categorized as Static, Linear, Leader Follower, Collision Avoidance, Group, Other Interactions, or Non-Interacting using explicit motion and interaction conditions.The categories use distance, prediction error, angular ranges, relative velocity, duration, and neighbor proximity.
  • Interaction categories: Leader Follower, Collision Avoidance, and Group categories identify distinct neighbor relationships through direction, relative velocity, angular position, distance, and temporal persistence.Other Interactions captures a neighbor in front within 5 m, while Non-Interacting covers trajectories satisfying none of the preceding conditions.
  • Dataset selection: Dataset selection explicitly targets crowded settings where agent-agent interactions matter more than scene constraints, using pedestrian spatial coordinates from annotation files.The stated selection uses only pedestrian locations at each time frame from each real-world dataset.
  • Real-world datasets: ETH and UCY provide established benchmarks with approximately 750 and 780 pedestrians, respectively, and capture diverse social or nonlinear trajectories.ETH includes leader-follower, collision-avoidance, and group behaviors; UCY contains three scenes and socially induced nonlinear motion.
  • Real-world datasets: WildTrack, L-CAS, and CFF extend coverage to outdoor, indoor, and high-density train-station settings with diverse crowd interactions.L-CAS contains over 900 pedestrian tracks, while CFF contains 42 million trajectories extracted from real-world train stations.

2) TrajNet++ Synthetic Dataset:

TrajNet++ adds controlled synthetic interaction data by calibrating ORCA to real collision-avoidance motion and filtering scenes that are excessively sensitive to perturbations. The benchmark then splits categorized trajectories into training and testing datasets.

  • Motivation: Interaction-centric synthetic data provides a controlled environment for comparing model components under specified interaction scenarios.The synthetic portion is designed to evaluate performance under controlled conditions.
  • Simulator selection: ORCA is selected over Social Force because it better matches real-world human motion during collision avoidance, including reaction distance and curvature.The chosen ORCA parameters are illustrated as having reaction curvature similar to human trajectories.
  • Dataset generation: Synthetic scenes initialize randomly placed pedestrians on a circle with minimum separation, assigning each pedestrian a goal diametrically opposite its starting position.The simulations randomly choose the number of pedestrians from [4, 7).
  • Sensitivity filtering: ORCA exhibits sensitive dependence on initial conditions because agents approach the same point, so small perturbations can substantially alter future trajectories.Only scenes classified as Type III interacting are selected before filtering.
  • Sensitivity filtering: Each scene is perturbed with bounded uniform noise and simulated k times; scenes are removed when any prediction has significantly higher ADE than ground truth.The filtering procedure is illustrated in Fig. 13.
  • Dataset splits: TrajNet++ divides trajectories into training and testing splits according to the benchmark’s interaction categorization.The split statistics are reported in Tables VI and VII.
Loading 2007.03639v3…