Source-linked AI summary
Review of Pedestrian Trajectory Prediction Methods: Comparing Deep Learning and Knowledge-based Approaches
Raphael Korbmacher, Antoine Tordeux
TL;DR
Pedestrian trajectory prediction must address individual motion, interactions, and collective dynamics across different scenes and application scales. The paper reviews and compares knowledge-based and deep-learning approaches, finding deep learning accurate given enough data while its coverage of scenes and interactions remains open. It identifies hybrid approaches as promising for combining knowledge with learned models.
Problem
Pedestrian trajectory prediction must address individual motion, interactions, and collective dynamics across different scenes and application scales.
Method
The paper conducts a bibliographical review comparing knowledge-based and deep-learning approaches by methodology, application field, phenomena of interest, and scale.
Results
Deep learning predicts trajectories with relatively high accuracy given enough data, whereas its ability to cover all possible scenes and interactions remains an open question.
Takeaways & Limitations
Hybrid approaches can combine knowledge with learned models to compensate for scarce data and reduce deep learning’s missing explainability.
Takeaways & Limitations
Deep learning is mainly used for single-pedestrian prediction, and its ability to support large-scale simulation and collective dynamics remains to be demonstrated.
Abstract
from arXiv · showhide
In crowd scenarios, predicting trajectories of pedestrians is a complex and challenging task depending on many external factors. The topology of the scene and the interactions between the pedestrians are just some of them. Due to advancements in data-science and data collection technologies deep learning methods have recently become a research hotspot in numerous domains. Therefore, it is not surprising that more and more researchers apply these methods to predict trajectories of pedestrians. This paper compares these relatively new deep learning algorithms with classical knowledge-based models that are widely used to simulate pedestrian dynamics. It provides a comprehensive literature review of both approaches, explores technical and application oriented differences, and addresses open questions as well as future development directions. Our investigations point out that the pertinence of knowledge-based models to predict local trajectories is nowadays questionable because of the high accuracy of the deep learning algorithms. Nevertheless, the ability of deep-learning algorithms for large-scale simulation and the description of collective dynamics remains to be demonstrated. Furthermore, the comparison shows that the combination of both approaches (the hybrid approach) seems to be promising to overcome disadvantages like the missing explainability of the deep learning approach.
I. INTRODUCTION
Pedestrian trajectory prediction is valuable but challenging because it must address individual motion, interactions, and collective dynamics. The review compares knowledge-based and deep-learning approaches, highlighting their differing data requirements, scales of application, and prospects for hybridization.
- Trajectory prediction supports facility design, evacuation, autonomous driving, human-robot interaction, assistive technologies, and augmented or virtual reality.
- The task ranges from predicting individual trajectories during local interactions to predicting many pedestrians and emergent collective dynamics.
- Knowledge-based models use interpretable rules and parameters representing physical, social, or psychological factors, whereas deep learning learns from data without interpretable rules or parameters.
- Deep learning methods include LSTM, CNN, and GAN architectures and have become prominent for trajectory prediction after the emergence of social-LSTM.
- Hybrid approaches combine knowledge-based components with deep learning or use deep learning to estimate knowledge-based parameters, addressing data scarcity and limited explainability.
- The review compares knowledge-based and deep-learning approaches technically and by application field, then discusses future directions for their common development.
II. THE KNOWLEDGE-BASED APPROACH
Early pedestrian-dynamics research used observations and visual records to develop behavioral knowledge, facility guidelines, and planning concepts. Researchers later developed simulation models because those concepts were not suited to predicting pedestrian flows or trajectories.
- Researchers initially used direct observations, photographs, and time-lapse films to study pedestrian behavior.
- This knowledge informed level-of-service concepts, pedestrian-facility design elements, and planning guidelines.
- Because these concepts and guidelines were not suited to predicting pedestrian flows or trajectories, researchers began creating simulation models.
A. Microscopic pedestrian models
Microscopic models represent individual pedestrians and their interactions, enabling heterogeneous behavior and trajectory prediction across scales but creating computational costs for large simulations. They include acceleration-based, velocity-based, and decision-based formulations.
- Microscopic models represent individual behavior and interactions, allowing heterogeneous characteristics to be assigned to separate agents.
- Microscopic models can be computationally expensive, limiting their use in large-scale simulation.
- Microscopic models can predict trajectories at any scale, while crowd-level dynamics emerge from mutual influences among individuals.
- Acceleration-based models: Acceleration-based models determine movement through superposed external forces, commonly combining desired-direction relaxation with interactions involving neighbors and obstacles.
- Velocity-based and decision-based models: Velocity-based models emphasize pedestrian dynamics without substantial inertial effects or latency, while decision-based models use rules or decisions rather than differential equations.
- Decision-based models and cellular automata: In synchronous decision-based models, pedestrians choose actions at t + ∆t using the system state at t; cellular automata discretize time, space, and pedestrian state.
B. Trends during the past decades
Pedestrian dynamics has developed substantially over recent decades, with microscopic force-based modeling prominent among knowledge-based approaches. Traditional cellular-automata, queuing, and fluid- or gas-analogy approaches appear to be reaching a plateau in citation trends.
- The field began with investigations in the 1960s and 1970s but became a major research focus during the past three decades.
- Experimental studies intensified during the 2010s and covered uni-directional flow, counter-flow, bottlenecks, and intersecting flow.
- Knowledge-based models developed across microscopic and macroscopic scales, with the microscopic social force model identified as especially prominent.
- Traditional cellular automata, queuing-process, and fluid- or gas-dynamics analogies currently appear to be reaching a plateau, despite lightly increasing citation trends.
C. Knowledge-based models for understanding and predicting
Knowledge-based models seek to explain collective pedestrian dynamics through mechanisms and interpretable parameters, with trajectory prediction emerging mainly as an indirect capability. Deep learning methods instead learn trajectory-related patterns from data, using classes such as LSTM, CNN, and GAN.
- Knowledge-based models: Knowledge-based models identify mechanisms and parameters underlying collective pedestrian dynamics, including self-organisation and macroscopic coordination.Their multiscale perspective links individual walking behaviours to large-scale patterns.
- Knowledge-based models: Trajectory prediction is generally an indirect goal of microscopic knowledge-based models, which primarily aim to understand large-scale dynamics.Such models can nevertheless support trajectory prediction and are implemented in multi-agent simulation tools.
- Deep learning models: Deep learning trajectory methods are commonly classified into LSTM, CNN, and GAN classes, alongside other emerging architectures.Alternative classifications distinguish sequential from non-sequential methods or focus on interactions, destinations, and intentions.
- Literature overview: The reviewed literature includes broad collections of deep-learning trajectory algorithms and citation-based indicators of research activity.The cited materials include algorithm selections and annual citation estimates based on Google Scholar searches.
A. Long short-term memory networks
LSTM networks use temporal memory to model sequences and have become prominent in pedestrian trajectory prediction. Social-LSTM introduced social pooling to incorporate neighbouring pedestrians into each prediction step, inspiring extensions with scene, obstacle, and attention information.
- Long short-term memory networks: LSTM networks use feedback connections across time steps to retain recent input representations for sequence modelling.This temporal memory makes them suited to time-series prediction and sequence labelling.
- Long short-term memory networks: Social-LSTM incorporates nearby pedestrians through a social-pooling layer that gathers their hidden states as input for subsequent prediction.This design explicitly accounts for interactions in crowded scenarios.
- Long short-term memory networks: Later LSTM variants add scene information, static obstacles, scene scale, or attention mechanisms to model pedestrian and environmental influences.These extensions distribute social and scene information across different network components or interaction modules.
- Long short-term memory networks: The literature review includes an overview table of pedestrian trajectory algorithms relying on LSTM networks.The table is presented as a selection of deep-learning trajectory-prediction articles.
B. Convolutional neural network
CNNs are well established for visual perception but are less common for pedestrian trajectory prediction because their non-sequential structure complicates input and output design. Graph CNN variants represent pedestrian interactions as spatio-temporal graphs.
- Convolutional neural networks: CNNs support computer-vision tasks such as classification, detection, tracking, and segmentation, with reduced computational load compared with regular neural networks.Their role in pedestrian dynamics is especially connected to detecting and tracking pedestrians or vehicles.
- Convolutional neural networks: CNNs are not widely used for pedestrian trajectory prediction because their non-sequential structure makes network input and output design difficult.The reviewed applications include vehicle trajectories, pedestrian behaviour for autonomous vehicles, and pose or action recognition.
- Convolutional neural networks: Behavior-CNN was the first CNN described as modelling and predicting pedestrian trajectories, encoding behaviour into sparse displacement volumes.Those volumes serve as network input at the first stage of its three-stage architecture.
- Convolutional neural networks: Graph CNN approaches model pedestrian interactions as edges in a spatio-temporal graph, with pedestrians represented as nodes.Some variants combine graph CNN feature extraction with LSTM processing.
- Convolutional neural networks: The review includes an overview table of trajectory-prediction algorithms based on convolutional neural networks.The table is labelled as a survey of CNN-relying deep-learning algorithms.
C. Generative adversarial networks
GANs address the multimodal nature of future pedestrian motion by generating distributions of possible trajectories rather than only a single prediction. Their promise is tempered by training difficulty and mode collapse, while fair comparison with knowledge-based methods remains difficult.
- Generative adversarial networks: GANs can represent multimodal future pedestrian motion by generating distributions of potential trajectories instead of only one best trajectory.A generator produces samples while a discriminator distinguishes generated samples from training data.
- Generative adversarial networks: GAN training uses competition between generator and discriminator components so the generator learns samples resembling the true data distribution.The process is framed as a two-player min-max game.
- Generative adversarial networks: GAN-based trajectory methods face training difficulty and mode collapse despite promising results.These issues can limit the diversity or reliability of generated trajectory samples.
- Comparison: Knowledge-based and deep-learning approaches are difficult to compare within one fair framework because they use different inputs and quality criteria.Knowledge-based methods simulate from current system states, whereas deep-learning definitions commonly use past trajectories.
- Comparison: Knowledge-based model evaluation requires parameter calibration and validation, while deep-learning coefficients are trained by minimizing a cost function.The approaches therefore differ in both parameter meaning and evaluation procedure.
B. Application oriented comparison
Knowledge-based and deep-learning approaches differ in data needs, application scope, and strengths. Deep learning performs better for local, low-density trajectory prediction, while knowledge-based models offer interpretability and adaptability that numerical accuracy comparisons often omit.
- Application scope: Knowledge-based models mainly support crowded, large-scale simulations, whereas deep-learning methods focus on local interactions involving single pedestrians in low-density situations.Applications include infrastructure design and evacuation for knowledge-based models, and automated mobile systems for deep learning.
- Numerical comparison: Deep-learning trajectory accuracy is commonly evaluated with average and final displacement errors, using social-LSTM and the social force model as benchmark references.Table VII reports ADE and FDE comparisons across selected studies and datasets.
- Numerical comparison: In low-density situations, deep-learning algorithms outperform knowledge-based models in trajectory prediction, although their error metrics show high fluctuation.The fluctuation is identified as mainly affecting deep-learning evaluation.
- Advantages and disadvantages: Knowledge-based models use simple forms and interpretable parameters, enabling reproducibility and adjustment to environmental or behavioral changes without new training data.Their parameters can be changed to represent altered pedestrian preferences, such as increased social distance or desired time gaps.
- Advantages and disadvantages: Knowledge-based models require domain knowledge and may represent only average pedestrian behavior, making complete crowd-behavior coverage difficult with one model.These limitations are not fully reflected in predominantly numerical comparisons.
V. FUTURE DIRECTIONS
The paper identifies future trends for pedestrian-trajectory prediction but does not specify them in the supplied passage.
- Future directions: The work’s final section examines future directions and trends in pedestrian-trajectory prediction.The supplied passage introduces this purpose without listing specific directions.
A. The hybrid approach
Hybrid methods combine knowledge-based and deep-learning approaches to address their complementary weaknesses. They can inject knowledge into learning systems, use learning to improve knowledge-based predictions, and potentially improve accuracy while retaining interpretability.
- A. The hybrid approach: Hybrid approaches are proposed because knowledge-based models provide interpretable, data-efficient parameters while deep-learning coefficients are generally difficult to interpret and data-intensive.The paper identifies this complementarity across pedestrian prediction and other scientific applications.
- A. The hybrid approach: Knowledge-based simulations can generate synthetic data, and their components can be embedded in neural networks to improve deep-learning predictions.Knowledge-guided loss functions can enforce physical consistency and rule out unrealistic outputs.
- A. The hybrid approach: Deep-learning models can improve knowledge-based predictions by learning residuals that correct the errors made by the original model.Residual modeling is described as an established way to address imperfections in knowledge-based approaches.
- A. The hybrid approach: With sufficient data, deep learning can predict trajectories accurately for a given scene, but its coverage of varied interactions and scenes remains an open question.The hybrid approach may compensate for data scarcity with available knowledge.
- A. The hybrid approach: Hybrid approaches are considered promising because they combine the strengths of both methods and reduce shortcomings such as deep learning’s missing explainability.The passage presents this as a supported prospective conclusion rather than a demonstrated universal result.
B. Other directions
Future work considers extending deep learning from single-pedestrian prediction to large-scale collective simulation. It also proposes scenario-aware architectures and richer interaction variables, while noting unresolved requirements for training and generalization.
- B. Other directions: Deep learning is currently used mainly to learn human behavior and predict single-pedestrian trajectories, while large-scale collective simulation remains a future application.The paper notes potential use in crowd dynamics and evacuation scenarios.
- B. Other directions: Reinforcement learning has disadvantages involving reward functions, predefined destinations, and interaction modeling, motivating combinations with supervised or deep reinforcement learning.The supplied passage reports these as proposed development directions.
- B. Other directions: Before supervised deep learning can support large-scale crowd simulation, researchers must determine suitable network types, complexity, and training data for varied geometries and scenarios.The paper explicitly frames these as open questions.
- B. Other directions: Universal networks capable of predicting accurate trajectories across all density levels and facility types remain an expectation rather than an established capability.The passage presents this possibility alongside unresolved questions about scenario dependence.
- B. Other directions: Deep-learning inputs could incorporate relative velocity, time gap, bearing angle, collision cone, and time-to-collision in addition to trajectory history.These variables are drawn from knowledge-based pedestrian-dynamics studies and may be especially useful with limited training data.