Source-linked AI summary
Pedestrian Path, Pose and Intention Prediction through Gaussian Process Dynamical Models and Pedestrian Activity Recognition
Raul Quintero, Ignacio Parra, David Fernandez Llorca, Miguel Angel Sotelo
TL;DR
Road fatalities motivate earlier prediction of pedestrian behavior for vehicle safety systems. The paper uses activity-specific B-GPDMs to reconstruct future paths and poses from latent dynamics, reporting recognition and path-prediction results across four activities.
Problem
Earlier pedestrian state and intention estimates are needed to improve the timing and effectiveness of automatic emergency braking interventions.
Method
The method trains B-GPDMs for walking, stopping, starting, and standing, selects a model using activity recognition, and predicts future latent states, paths, poses, and intentions.
Results
Starting intentions are recognized 125ms after gait initiation at 80% accuracy, while stopping intentions are recognized 58.33ms before the event at 70% accuracy.
Takeaways & Limitations
The method provides activity-specific predictions of pedestrian paths, poses, and intentions using only two pedestrian observations.
Abstract
from arXiv · showhide
According to several reports published by worldwide organisations, thousands of pedestrians die in road accidents every year. Due to this fact, vehicular technologies have been evolving with the intent of reducing these fatalities. This evolution has not finished yet since, for instance, the predictions of pedestrian paths could improve the current Automatic Emergency Braking Systems (AEBS). For this reason, this paper proposes a method to predict future pedestrian paths, poses and intentions up to 1s in advance. This method is based on Balanced Gaussian Process Dynamical Models (B-GPDMs), which reduce the 3D time-related information extracted from keypoints or joints placed along pedestrian bodies into low-dimensional spaces. The B-GPDM is also capable of inferring future latent positions and reconstruct their associated observations. However, learning a generic model for all kind of pedestrian activities normally provides less ccurate predictions. For this reason, the proposed method obtains multiple models of four types of activity, i.e. walking, stopping, starting and standing, and selects the most similar model to estimate future pedestrian states. This method detects starting activities 125ms after the gait initiation with an accuracy of 80% and recognises stopping intentions 58.33ms before the event with an accuracy of 70%. Concerning the path prediction, the mean error for stopping activities at a Time-To-Event (TTE) of 1s is 238.01mm and, for starting actions, the mean error at a TTE of 0s is 331.93mm.
I. INTRODUCTION
Pedestrian fatalities motivate earlier and more informed vehicle interventions. The paper proposes activity-specific B-GPDMs to predict pedestrian paths, poses, and intentions up to 1 second ahead.
- More than 275,000 pedestrians died in worldwide road traffic accidents in 2013, motivating improved pedestrian-aware vehicle systems.
- Earlier estimates of pedestrian positions and intentions could improve AEBS timing and active interventions because delayed braking increases impact speed and injury risk.
- The proposed method uses B-GPDMs to predict future pedestrian paths, poses, and intentions up to 1 second in advance.
- Separate models for walking, stopping, starting, and standing are selected by activity because generic activity models can produce less accurate predictions.
A. Predictions using Pedestrian Features
Prior work predicts pedestrian behavior from motion, orientation, infrastructure, and scene context. These approaches combine diverse visual and environmental cues for path and intention prediction.
- Pedestrian motion, positioning, orientation, and head-pose features are used to predict paths, intentions, and future orientations.
- Several approaches combine positioning with dense optical flow or other motion features for path and intention prediction.
- Infrastructure-mounted sensors and roadside units support crossing-intention prediction when vehicle viewpoints are obstructed.
- Scene context such as streets, sidewalks, intersections, crosswalks, distances, and destinations can provide information for pedestrian intention and trajectory prediction.
C. Discussion
The discussion motivates body-part-aware, activity-specific prediction with minimal observation history. The system uses four activity dynamics and a curated CMU dataset with pose and displacement features.
- Pedestrian skeleton motion can reveal activity-specific cues because different body parts may indicate starting or other behaviors differently.
- The method uses four pedestrian actions and only two observations, addressing settings where pedestrians suddenly appear in a vehicle trajectory.
- The CMU-UAH dataset contains 490 sequences and 302,470 poses from 31 subjects, filtered to walking, starting, stopping, and standing without orientation changes.
- Each observation combines joint poses and consecutive displacements, with displacements supporting future path reconstruction and activity classification.
- The labeling protocol defines starting, walking, stopping, and standing intervals using gait and foot-contact events to create reliable ground truth.
B. Pedestrian Skeleton Estimation
The system estimates pedestrian skeletons from stereo-derived point clouds using geometric constraints. The representation includes body-part joints and headings used for prediction.
- A stereo point-cloud skeleton estimator with geometric constraints provides pedestrian observations for testing under noisy conditions.
- The skeleton visualization distinguishes left joints, right joints, head, shoulder center, and hip center, while lines show body-part headings.
C. Learning Pedestrian Activities
The method learns low-dimensional dynamical models from time-related pedestrian-joint observations and uses activity-specific modeling to recognize pedestrian states.
- C. Learning Pedestrian Activities: B-GPDMs transform time-related 3D joint information into low-dimensional latent spaces while supporting future-position inference and observation reconstruction.The models learn observation and dynamics mappings for temporal pedestrian data.
- C. Learning Pedestrian Activities: Each sequence receives its own B-GPDM, with latent coordinates initialized by PCA and kernel parameters initialized from prior values.Figure 4 illustrates a six-step walking model with projected observations and variance visualization.
- C. Learning Pedestrian Activities: Separate dynamical models are trained for pedestrian activity patterns so the recognized activity can determine the model used for future-state estimation.Selecting among models with different dynamics is intended to improve prediction accuracy over a generic model.
- C. Learning Pedestrian Activities: A first-order HMM models transitions among four hidden pedestrian activities from observable 3D joint information.The Viterbi algorithm identifies the most likely activity sequence, using transition and emission probabilities.
- C. Learning Pedestrian Activities: The emission model compares current pose and joint-displacement observations with training subsets using scaled, joint-referenced sum-of-squares errors.The two errors quantify pose and displacement similarity for candidate states.
E. Path, Pose and Intention Prediction
After recognizing the current activity, the method selects and refines a matching dynamical model, then iteratively predicts latent coordinates and reconstructs future pedestrian observations.
- E. Path, Pose and Intention Prediction: The selected activity model is refined around the most similar training observation by gradient descent in latent space.The search minimizes reconstruction discrepancy between the current observation and the observation reconstructed from a latent position.
- E. Path, Pose and Intention Prediction: Iteratively reconstructed future observations provide predicted poses and displacements for computing the pedestrian path.The path accumulates consecutive predicted displacements relative to the right hip, chosen near the center of gravity.
IV. RESULTS
The method achieves strong activity recognition with 11-joint pose and displacement inputs, while transition timing and feature choice affect performance. It also evaluates transition delays and intention recognition under model and labelling constraints.
- Activity Recognition Results: 95.13% maximum activity-recognition accuracy is achieved using poses and displacements from 11 joints, compared with 90.69% using 41 joints.Using body poses alone, accuracy is 91.28% with 11 joints and 88.39% with 41 joints.
- Activity Recognition Results: Displacements distinguish standing from other activities better, whereas poses and displacements are more discriminative for starting and stopping actions.Displacements alone do not reliably distinguish a pedestrian’s first or last step.
- Activity Recognition Results: 622 transitions are correctly detected and 45 incorrectly detected with 11 joints, yielding 93.25% transition accuracy.Most missed transitions are walking-stopping changes, whose stopping samples are comparatively fewer and faster.
- Transition Detection: Starting intentions are recognized 125ms after gait initiation with 80% accuracy when 11 joints are used.The method models standing-starting and starting-walking as separate dynamical changes, which may affect recognition while preserving path-prediction detail.
- Transition Detection: Stopping intentions are recognized 58.33ms before the standing event with 70% accuracy using 11 joints.The mean walking-stopping transition delay before standing is −279.92 ± 158.59ms.
B. Activity Recognition using Vision-based Skeletons
A noisy vision-based skeleton sequence is evaluated for a pedestrian stopping and restarting on a zebra crossing. Standing at the curb is recognized, but the walking-stopping transition is missed.
- Vision-based Skeletons: The sequence shows a pedestrian crossing a road, providing the visual context for the noisy skeleton-based recognition test.The accompanying figure consists of frames from the crossing sequence.
- Vision-based Skeletons: The noisy-observation sequence correctly recognizes the pedestrian’s standing action at the curb but misses the walking-stopping transition.The sequence depicts a pedestrian stopping and then starting to walk again on a zebra crossing.
C. Pedestrian Path Prediction Results
The method predicts pedestrian paths using activity-specific models and evaluates combined longitudinal and lateral error across time-to-event values. Starting and stopping predictions show distinct error patterns linked to activity recognition timing and pedestrian dynamics.
- The evaluation uses 11 joints and measures combined longitudinal and lateral Mean Euclidean Distance across TTE values.Table V and Fig. 10 report the MED and standard deviation for predictions up to 1s.
- Starting-activity errors grow exponentially after the event because recognition is delayed by about 60ms while the pedestrian accelerates.Errors become more linear after the pedestrian finishes accelerating.
- 331.93mm MED is obtained for starting activities at a TTE of 0s and a 1s prediction horizon, similar to 315mm reported at 1.2s in prior work.Starting predictions are affected by assumed zero displacement before recognition and accelerating motion after recognition.
- Stopping-activity errors are linear before the event and decrease logarithmically afterward because the second half of the final step carries the clearest stopping information.The mean stopping-step length is 381.22±78.92ms, delaying model selection until a few hundred milliseconds before the event.
- 238.01mm MED is achieved for stopping activities at a TTE of 1s and a 1s prediction horizon, compared with 224mm at 1.2s in prior work.The paper also reports 314.5mm RMSE at 1s, versus 292mm in another comparison.
D. Path Prediction using Vision-based Skeleton Estimation
The vision-based skeleton evaluation tests path prediction with noisy pose and displacement observations. Lateral error increases with prediction horizon, and orientation mismatch between training and test data limits reconstruction accuracy.
- Lateral MED increases from 131.71±57.89mm at 0.25s to 448.84±157.39mm at 1s with noisy skeleton observations.Intermediate values are 250.95±89.00mm at 0.5s and 355.80±123.37mm at 0.75s.
- The noisy-observation experiment represents poses and displacements computed by a skeleton estimation algorithm for predictions up to 1s.Fig. 11 reports MED in millimetres across the prediction horizon.
- Predicted paths are corrupted when pedestrians are not walking perpendicular to the sensor because the training data have near-zero longitudinal-direction variance.The paper proposes rotating training and test observations to a common sensor orientation.
E. Processing Time
Processing time depends on the selected dynamical model and sequence length. B-GPDM kernel-matrix inversion is the main computational bottleneck and causes processing time to increase exponentially with sequence samples.
- Processing time increases exponentially with sequence length because B-GPDM kernel matrices grow with the number of sequence samples.Kernel-matrix inversion is required to evaluate the likelihood function and its gradient.
- The path-prediction and activity-recognition implementation was evaluated in MATLAB 2016 on an Intel i7-7700K 4.20GHz processor.The reported processing times are hardware- and implementation-specific.
- Path prediction takes longer when the selected model represents a longer sequence because it requires kernel-matrix inversion between test and reconstructed observations.Processing times for each step are reported in Table VI.
V. CONTRIBUTIONS AND FUTURE WORK
The paper combines activity recognition and B-GPDM-based prediction for pedestrian paths, poses, and intentions. It reports strong recognition results for selected body motions while identifying dataset, class-balance, sensing, and deployment limitations for future work.
- 95.13% maximum activity-recognition accuracy is achieved when observations from shoulder and leg motions are used.Shoulder and leg motions are reported as more valuable than other body parts, and poses plus displacements are needed for more than two behaviors.
- 80% accuracy detects starting intentions 125ms after gait initiation, while 70% accuracy recognizes stopping intentions 58.33ms before the event using shoulder and leg joints.The paper identifies these as the reported starting and stopping intention-recognition results.
- Path-prediction errors are similar to prior work, although the proposed method uses two observations rather than a temporal window of n trajectory points.The paper evaluates errors at different TTE values rather than across all time steps.
- Future evaluation should include more sequences, children and elderly pedestrians, moving vehicles, cluttered backgrounds, and more extensive real-world datasets.The authors also propose improved skeleton estimation and methods for imbalanced stopping-activity data.