PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Pedestrian Path Prediction Using Inverse Reinforcement Learning

Šimon Sukup, Ariyan Bighashdel, Pavol Jancura

arXiv:2608.15929v1cs.AI

TL;DR

Pedestrian path prediction must account for crowd interactions and pedestrian decision-making, yet it was unclear whether unified learning-task formulations would improve advanced deep models. This paper adapts STGAT to a unified framework and finds that PPO-based stochastic-policy and supervised sequential formulations substantially improve performance across all datasets.

  • Problem

    It was unclear whether unified pedestrian path-prediction formulations that improved shallow models would also enhance advanced architectures, despite the importance of accurate prediction for road safety.

  • Method

    The paper adapts STGAT within a unified framework using new state-action definitions and supervised or reinforcement-learning formulations, including PPO and REINFORCE.

  • Results

    Substantial improvement across all datasets was achieved by the ODM stochastic-policy formulation with PPO and the designed supervised SDM formulation.

  • Takeaways & Limitations

    Reformulating the learning task and accounting for stochastic human movement can enhance complex deep-learning architectures for pedestrian path prediction.

  • Takeaways & Limitations

    The stochastic policy did not reach the deterministic policy’s results within 150 training epochs, and further algorithm tuning was necessary.

Abstract

from arXiv · show

Pedestrian path prediction is crucial for enhancing the safety of autonomous vehicles and advanced driver-assistance systems. Previous studies explored different learning-task formulations for pedestrian path prediction and compared these formulations using shallow neural networks, but did not extend this analysis to more complex deep-learning models. This paper adapts the Spatial-Temporal Graph Attention Network (STGAT) to a unified pedestrian path prediction framework and introduces state and action definitions specific to STGAT. The resulting formulations support deterministic and stochastic policies, one-time and sequential decision-making, and reinforcement-learning algorithms including REINFORCE and proximal policy optimization. The proposed learning-task formulations improve prediction performance across the selected benchmark datasets compared with the standard supervised-learning formulation. These results demonstrate that reformulating the decision process and training objective can improve an advanced pedestrian trajectory prediction architecture and may provide a path toward improving other graph-based prediction models.

I. INTRODUCTION · II. RELATED WORK · A. Advances in Pedestrian Path Prediction and Interaction

Pedestrian path prediction is motivated by road-safety needs and the challenges of crowd interaction and pedestrian decision-making. The paper extends a unified MDP-based framework from shallow policies to STGAT, building on advances in deep and graph-based interaction modelling.

  • I. INTRODUCTION: Pedestrian behaviour prediction can help autonomous vehicles and advanced driver-assistance systems reduce human-caused traffic incidents and improve road safety.The prediction challenge includes crowd interactions and each pedestrian’s underlying decision-making process.
  • I. INTRODUCTION: Lemmens et al. introduced a unified pedestrian path prediction framework that formulates prediction as a Markov decision process for objective comparison of learning-task formulations.Their work compared standard supervised learning with inverse reinforcement learning and more advanced reinforcement-learning algorithms.
  • I. INTRODUCTION: The earlier framework improved a shallow neural-network policy but did not extend to advanced deep-learning architectures predominant in pedestrian path prediction.This gap motivates testing whether the unified framework can enhance more advanced models.
  • I. INTRODUCTION: This paper modifies the unified framework for STGAT, using novel state and action definitions to create learning-task formulations specific to the architecture.STGAT is used as both a stochastic and deterministic policy with advanced reinforcement-learning algorithms.
  • II. RELATED WORK: STGAT applies deep learning to time-series forecasting and interaction modelling, motivating review of the preceding work that led to its development.The related-work discussion also explains the motivation for using a unified pedestrian path prediction framework.
  • A. Advances in Pedestrian Path Prediction and Interaction: Early pedestrian path prediction models had limited generalization and scalability, prompting broader adoption of recurrent, long-short-term-memory, gated-recurrent-unit, and transformer architectures.Alahi et al. used LSTMs to address temporal dependencies in pedestrian-path data.
  • A. Advances in Pedestrian Path Prediction and Interaction: Research increasingly modelled social interactions through spatio-temporal graphs, while Social Attention and Trajectron used LSTMs to manage structured sequence data.These methods address social behaviour and intentions alongside temporal dependencies.
  • A. Advances in Pedestrian Path Prediction and Interaction: Graph Convolutional Networks and Graph Attention Networks advanced interaction modelling, with STGAT extending GATs to represent dynamic pedestrian interactions across space and time.STGAT accounts for spatial interactions among individuals and captures their temporal evolution.

B. Unified Pedestrian Path Prediction Framework · III. METHODS · A. Reinforcement Learning Problem Setup

The paper unifies pedestrian path prediction formulations to test whether reinforcement-learning improvements persist with the STGAT architecture. It defines STGAT-specific state, action, policy, and decision-process formulations for deterministic or stochastic prediction.

  • B. Unified Pedestrian Path Prediction Framework: A unified framework separates gains from model architecture from gains caused by alternative learning-task formulations.The earlier unified framework used a shallow neural network, leaving open whether advanced reinforcement-learning improvements would persist with deeper architectures.
  • III. METHODS: The STGAT extension treats path prediction as reinforcement learning rather than standard supervised learning.It required new state-action definitions, value-network architecture, and loss definitions tailored to STGAT.
  • A. Reinforcement Learning Problem Setup: Reinforcement learning models an agent making state-based decisions in a Markov decision process to maximize cumulative reward.The formulation includes state space S, action space A, and reward function R : S × A → R.
  • A. Reinforcement Learning Problem Setup: The framework supports stochastic policy πθ(a | s) and deterministic policy µθ(a | s), with θ representing STGAT’s learnable parameters.The stochastic policy assigns probabilities to state-action pairs, whereas the deterministic policy assigns one action to each state.
  • A. Reinforcement Learning Problem Setup: One-time decision-making represents all 12 predicted pedestrian steps in one action and produces a single reward.An ODM episode contains one decision step, with the action comprising the predicted coordinate sequence.
  • A. Reinforcement Learning Problem Setup: Sequential decision-making treats pedestrian steps as separate actions, producing episodes with Tε = 12 steps.This formulation is used because one-time decision-making does not appropriately reflect pedestrian decision-making.
  • A. Reinforcement Learning Problem Setup: In inverse reinforcement learning, rewards are estimated from expert ground-truth trajectories using a distance-based squared L2 formulation, with future rewards discounted in the return.The policy may be stochastic or deterministic, and its parameters are updated through the policy-gradient objective and stochastic gradient descent with learning rate α.

B. STGAT Architecture and Implementation

STGAT combines separate temporal and spatial interaction modeling with a decoder that predicts each pedestrian’s subsequent steps. Its training uses encoder reconstruction pretraining followed by end-to-end optimization of the full pipeline.

  • Architecture: STGAT uses separate G-LSTM and M-LSTM modules to model temporal and spatial pedestrian interactions within one scene.Noise is added to encoder outputs to support multimodality.
  • Architecture: A single D-LSTM decoder takes intermediate state vectors as sequences and outputs each pedestrian’s next steps separately.
  • Training: 250 epochs of encoder pretraining reconstruct observed trajectories, followed by 150 epochs training the complete pipeline including the D-LSTM.The pretrained checkpoint was reused after the first two training phases.

C. ODM Formulations … D. SDM Formulations (BEP Extension)

The formulations define pedestrian path prediction as either whole-scene observation with single-pedestrian actions or individual trajectory episodes, while extending supervised and stochastic policy variants within STGAT. These definitions support deterministic and stochastic decision processes across ODM and SDM settings.

  • 1) SL-MSE: SL-MSE maps to an ODM process in which the state contains 8 observed steps from every pedestrian in the scene.STGAT uses the whole pedestrian scene as one input to model interactions.
  • 1) SL-MSE: The SL-MSE action is the next 12 steps of one pedestrian, after which the final state contains the complete 20-step trajectory.For SL-MSE, the discount factor is λ = 0.
  • 1) SL-MSE: SL-MSE uses predicted and observed trajectory coordinates at corresponding timestamps in its formulation.The formulation references predicted coordinates ˆx_tτ and ˆy_tτ alongside observed coordinates x_tτ and y_tτ.
  • 2) SL-MSE-SPG: The SL-MSE-SPG variant fixes encoder-output noise for each model initialization to estimate probabilities by sampling sequential pedestrian steps.This modification supports Gaussian sampling of each pedestrian’s next step.
  • 2) SL-MSE-SPG: Its policy-gradient expression uses log πθ(ˆa_tτ | ˆs_tτ), the logarithm of the conditional action probability given the current state.The probability is defined at trajectory time step tτ.
  • D. SDM Formulations (BEP Extension): SDM treats each pedestrian trajectory as an independent episode while retaining surrounding interactions in the observed state.The initial encoded state represents the whole scene, but later predictions rely only on the D-LSTM hidden state.

1) SL-MSE-SDM · IV. EXPERIMENTAL SETTINGS · A. Datasets

The section defines sequential pedestrian actions and state transitions for SDM, then describes the benchmark datasets and trajectory split used for evaluation.

  • 1) SL-MSE-SDM: In SDM, each pedestrian action is defined as one trajectory step, and the next state combines the previous state with the current action.Actions are represented as 2D vectors.
  • 1) SL-MSE-SDM: Each SDM episode contains T_ϵ = 12 steps, with episode time indices satisfying 0 ≤ t_ϵ < T_ϵ.The termination state is reached at t_ϵ = T_ϵ − 1.
  • 1) SL-MSE-SDM: The initial state contains observed x-y trajectory pairs for all pedestrians, with each observed trajectory comprising 8 steps.The initial state is written as s0 ∈ R2N0.
  • 1) SL-MSE-SDM: At each episode step, the state dimension depends on the number of pedestrians present, and transitions append the 2D action to the current state.The transition function maps RN × R2 to RN+2.
  • A. Datasets: ETH and UCY are standard pedestrian path prediction benchmarks containing 2D coordinates and time steps organized into scenes of contemporaneous pedestrians.ETH includes ETH and Hotel, while UCY includes Za… in the supplied passage.
  • IV. EXPERIMENTAL SETTINGS: The experimental split uses four-fifths of the original trajectory training set for training and the remaining fifth for evaluation.The split is defined over trajectories.

B. Evaluation and Metrics · C. Training and Optimization · D. Value function

The paper evaluates pedestrian path predictions with ADE, FDE, and minimum-over-20-run metrics, trains formulations and algorithms through repeated initialized runs and fine-tuning, and uses value-function approximations for REINFORCE and PPO. It also compares stochastic and deterministic policies and different value-network implementations.

  • B. Evaluation and Metrics: ADE and FDE measure prediction error, using the mean of STGAT’s output distribution as the action for comparability across formulations.Predictions are evaluated 20 times per pedestrian, with the minimum run used for mean minFDE and mean m... reporting.
  • B. Evaluation and Metrics: The evaluation reports the minimum error across 20 prediction runs for each pedestrian, averaged over all pedestrians in the set.The passage specifies minimum-over-20-run reporting for mean minFDE and mean m....
  • C. Training and Optimization: STGAT pretraining used five different initializations on each ETH and UCY subset across the first two phases.The third learning phase likewise trained every SL-MSE-SPG formulation and algorithm five times.
  • C. Training and Optimization: Each formulation and algorithm was fine-tuned on the ETH training and validation set with one identical optimal learning rate across all STGAT modules.The reinforcement-learning algorithms included REINFORCE and proximal policy optimization (PPO).
  • D. Value function: REINFORCE with baseline and PPO used value-function approximations that predicted expected returns from state inputs.The value function relied on a network predicting expected returns based on state inputs.
  • D. Value function: A full state baseline combined STGAT’s shared encoder with a shallow 3-layer dense network for large full-scene inputs.Only the shallow layer was updated, avoiding extensive hyperparameter tuning and speeding processing.
  • D. Value function: Table II compares algorithm results under stochastic and deterministic policies.The table is explicitly titled as a comparison of different algorithm results with both policy types.
  • D. Value function: Table III compares different value-network implementations.The table is explicitly titled as a comparison of value network implementations.

V. RESULTS AND DISCUSSION · B. Comparing REINFORCE and PPO variations · C. Value Function Variations

The results compare deterministic and stochastic ODM policies, REINFORCE and PPO variants, and value-function architectures within the unified framework. PPO with a full-state baseline and shared-encoder value functions generally improved prediction performance, while stochastic-policy limitations and baseline design affected outcomes.

  • V. RESULTS AND DISCUSSION: The experiments evaluate stochastic and deterministic ODM policies, REINFORCE and PPO variants, alternative value-function architectures, and SL-MSE-SDM.The evaluation includes REINFORCE, REINFORCE with baseline, PPO with and without baseline, and comparisons of shared-encoder and simplified-state value functions.
  • V. RESULTS AND DISCUSSION: SL-MSE-SPG performed better or similarly on ETH, Hotel, and Zara2 but underperformed on Zara1 and Univ.The passage attributes this variation to the simplicity of REINFORCE and its high-variance gradient steps, which slow convergence.
  • V. RESULTS AND DISCUSSION: The stochastic policy failed to reach the deterministic policy’s results within the fixed 150 training epochs, producing higher overall SL-MSE-SPG averages.Further tuning of parameters such as batch size would be needed to assess the algorithms fairly.
  • B. Comparing REINFORCE and PPO variations: Hyperparameter tuning improved PPO with a full state baseline on all datasets, with ADE decreases of 4% for ETH and over 11% for Hotel.Its FDE decreases ranged from 2% to 10% across all datasets.
  • B. Comparing REINFORCE and PPO variations: REINFORCE with a full state baseline underperformed SL-MSE on several Hotel and Univ metrics but matched or exceeded PPO with a shared encoder baseline on 4 Zara1 and Zara2 metrics.The comparison is reported in Table II.
  • C. Value Function Variations: Using full state through a shared STGAT encoder improved REINFORCE by up to 27% ADE on Univ and by 8% to 13% across Zara2 and Hotel metrics.The passage describes this comparison as an ablation study of value-network choice.
  • C. Value Function Variations: PPO without a baseline performed worse than both the shared-encoder baseline and the full-state baseline, indicating that a separate value-function network supports improved performance.The comparison uses results from Tables II and III.

D. SDM Ablation Study (BEP extension) · VI. CONCLUSION · VII. APPENDICES

The SDM ablation found that deterministic-policy SDM generally reduced performance relative to SL-MSE, while λ = 0.4 remained competitive with PPO on selected datasets. The conclusion emphasizes that the unified STGAT framework, stochastic formulations, and supervised SDM can improve deep-learning pedestrian path prediction, while identifying extensions for future work.

  • D. SDM Ablation Study (BEP extension): SDM with a deterministic policy decreased metrics for all datasets compared with SL-MSE.For λ = 0.4, the fine-tuned setting, decreases ranged from 2% to 13%, while most remaining metrics improved.
  • D. SDM Ablation Study (BEP extension): For λ = 0.4, SDM outperformed PPO with a baseline on ETH.PPO performed better on some Zara2 and Univ metrics by 1%-2%.
  • VI. CONCLUSION: The unified PPP framework implemented STGAT with new state-action definitions, a supervised SDM formulation, and value-network architectures.The study evaluated PPO, PPO without a baseline, REINFORCE, and REINFORCE with a baseline.
  • VI. CONCLUSION: ODM stochastic policy with PPO and a baseline, together with supervised SDM, substantially improved STGAT performance on all datasets.The conclusion attributes this advantage to sophisticated reinforcement-learning algorithms and formulations that account for stochastic human movement.
  • VI. CONCLUSION: The unified framework effectively enhanced the performance of a complex deep-learning pedestrian path prediction architecture.This conclusion follows the reported improvements from the studied stochastic and supervised formulations.
  • VI. CONCLUSION: Future work could combine stochastic STGAT with SDM, use IRL or GAN formulations with a parameterized discriminator, and evaluate the framework on a current SOTA PPP architecture.The paper identifies applying the designed formulations to a current SOTA architecture as a key future direction.

A. Evaluation Metrics · B. Hyperparameters Choice for Each Formulation

The evaluation uses ADE over forecasted and ground-truth trajectories and FDE at the final pedestrian step, with coordinates indexed by trajectory time tτ. Hyperparameter selection focuses on learning rates, while PPO settings include fixed clipping and one value-function iteration, alongside formulation-specific parameter tables.

  • A. Evaluation Metrics: ADE measures the average squared geometric distance between forecasted and ground-truth trajectories.The metric is defined over the trajectories rather than only their endpoints.
  • A. Evaluation Metrics: FDE considers only the final step of each pedestrian.The supplied passage states that the metric simplifies to a final-step expression, but does not provide that expression here.
  • A. Evaluation Metrics: Trajectory coordinates are indexed by the trajectory time tτ.
  • B. Hyperparameters Choice for Each Formulation: Policy learning rate and value function learning rate were the only hyperparameters with a noticeable performance impact and were searched over.The learning rate is denoted as LR.
  • B. Hyperparameters Choice for Each Formulation: The hyperparameter section includes a table for standard learning formulations.
  • B. Hyperparameters Choice for Each Formulation: The hyperparameter section includes a table for algorithms with a value function.
  • B. Hyperparameters Choice for Each Formulation: The hyperparameter section includes a table for PPO-based algorithms.
  • B. Hyperparameters Choice for Each Formulation: The PPO clipping factor was fixed at 0.2, and PPO value function iterations were kept at 1.

C. SL-MSE-SDM Objective Function Derivation · D. SL-MSE-SPG Loss Function Derivation

The SL-MSE-SDM derivation uses a 12-step horizon with a nonzero discount factor, while the SL-MSE-SPG derivation applies the policy gradient theorem with a one-step horizon and λ = 0. The latter uses parametrization and score-function tricks to obtain logarithmic probabilities.

  • C. SL-MSE-SDM Objective Function Derivation: For SL-MSE-SDM, the objective-function derivation uses a time horizon of T ϵ = 12.The passage states that the objective function is calculated with this horizon.
  • C. SL-MSE-SDM Objective Function Derivation: For SL-MSE-SDM, the discount factor is nonzero in the objective-function derivation.This distinguishes the stated setup from the later SL-MSE-SPG derivation.
  • C. SL-MSE-SDM Objective Function Derivation: The SL-MSE-SDM objective function is calculated after defining the objective-function formulation.The passage introduces the calculation directly following the objective-function definition.
  • D. SL-MSE-SPG Loss Function Derivation: The SL-MSE-SPG loss-function derivation follows the policy gradient theorem.The theorem provides the basis for the subsequent derivation.
  • D. SL-MSE-SPG Loss Function Derivation: The policy gradient theorem uses parametrization and score function tricks in the SL-MSE-SPG derivation.These tricks are used to transform the expression into one involving logarithmic probabilities.
  • D. SL-MSE-SPG Loss Function Derivation: The SL-MSE-SPG derivation uses logarithmic probabilities in its resulting expression.The passage explicitly attributes this form to the parametrization and score-function tricks.
  • D. SL-MSE-SPG Loss Function Derivation: For SL-MSE-SPG, the derivation sets T ϵ = 1.The stated horizon is one in the policy-gradient-based derivation.
  • D. SL-MSE-SPG Loss Function Derivation: For SL-MSE-SPG, the derivation sets λ = 0.This parameter value is specified alongside the one-step horizon.
Loading 2608.15929v1…