Source-linked AI summary
SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Prediction
Pu Zhang, Wanli Ouyang, Pengfei Zhang, Jianru Xue, Nanning Zheng
TL;DR
Pedestrian trajectory prediction needs to account for social behavior, but prior LSTM approaches may rely on outdated neighbor states and overlook current intentions. SR-LSTM iteratively refines current participant states through message passing and selects useful neighbor information. The paper reports that the resulting state refinement module with information selection outperforms state-of-the-art approaches.
Problem
Prior LSTM-based methods often use previous neighbor hidden states, which can miss current intentions, while neighbor information is not equally useful for prediction.
Method
SR-LSTM jointly and iteratively refines pedestrians’ current LSTM states through message passing, using motion gating and pedestrian-wise attention to select neighbor features.
Results
The state refinement module with social-aware information selection outperforms state-of-the-art approaches.
Takeaways & Limitations
SR-LSTM can timely capture changes in neighbors’ intentions and model social interactions including group walking, collision avoidance, and group avoidance.
Takeaways & Limitations
The authors note that their reproduced V-LSTM results do not completely match the previously reported V-LSTM* results, possibly because of implementation and training differences.
Abstract
from arXiv · showhide
In crowd scenarios, reliable trajectory prediction of pedestrians requires insightful understanding of their social behaviors. These behaviors have been well investigated by plenty of studies, while it is hard to be fully expressed by hand-craft rules. Recent studies based on LSTM networks have shown great ability to learn social behaviors. However, many of these methods rely on previous neighboring hidden states but ignore the important current intention of the neighbors. In order to address this issue, we propose a data-driven state refinement module for LSTM network (SR-LSTM), which activates the utilization of the current intention of neighbors, and jointly and iteratively refines the current states of all participants in the crowd through a message passing mechanism. To effectively extract the social effect of neighbors, we further introduce a social-aware information selection mechanism consisting of an element-wise motion gate and a pedestrian-wise attention to select useful message from neighboring pedestrians. Experimental results on two public datasets, i.e. ETH and UCY, demonstrate the effectiveness of our proposed SR-LSTM and we achieves state-of-the-art results.
1. Introduction
Pedestrian interactions are difficult to encode comprehensively, and existing LSTM methods can miss neighbors’ current intentions or select irrelevant social information. SR-LSTM addresses these gaps by iteratively refining participants’ states and selecting useful neighbor features.
- Motivation: Pedestrian trajectories reflect scene topology, beliefs, and intricate human-human interactions such as collision avoidance and group cohesion.Examples include strangers avoiding collisions, fellows walking together, and broken groups regrouping.
- Motivation: Existing data-driven LSTM methods learn social behaviors but struggle to comprehensively represent the range of pedestrian interactions.The paper identifies two neglected factors in prior approaches.
- Limitations of prior methods: Previous neighbor hidden states can lag behind sudden intention changes, causing large prediction errors when current states are ignored.Within a 0.4s step, a pedestrian may take one stride and unexpectedly change intentions.
- Social information selection: Neighbor features are not equally useful, so information should be selected according to pairwise motion and relative location.The paper motivates focusing on neighbors whose motion patterns indicate potential collision or interaction.
- Proposed approach: SR-LSTM jointly aligns pedestrians and repeatedly refines their current states through message passing to model deeper interactions.Its social-aware selection mechanism uses an element-wise motion gate and pedestrian-wise attention.
2. Related Work
Related trajectory-prediction methods model pedestrian interactions with RNNs, attention, graph structures, and message passing. SR-LSTM combines constrained message passing with pairwise motion and spatial information for trajectory prediction.
- Human-human interaction: Earlier interaction models include social-force, pedestrian-attribute, walking-group, stationary-group, game-theoretic, and fictitious-play approaches.These methods represent different factors affecting human-human interaction and pedestrian flows.
- RNN-based approaches: RNN-, LSTM-, and GRU-based approaches commonly model each pedestrian with shared parameters and incorporate neighbor information in different ways.The related-work passage identifies current observations and other neighbor representations as primary strategies.
- Attention-based approaches: Attention-based methods assign importance to neighbors using hidden states, soft attention, or pairwise velocity correlation.SR-LSTM differs by selecting motion features from each neighbor rather than only assigning neighbor-level importance.
- Graph-based frameworks: Graph convolution and message-passing frameworks provide inspiration for modeling relational information across entities.These frameworks have been applied in several vision and recognition tasks.
- SR-LSTM relation: SR-LSTM treats walking space as a fully connected graph and guides constrained message passing with pairwise motion and relative spatial location.This design adapts graph-based interaction modeling specifically to trajectory prediction.
3. Method
SR-LSTM extends independently processing LSTMs with a state-refinement module that passes socially selected messages among pedestrians. It refines cell states iteratively using current locations and LSTM states, then predicts the next coordinates from the refined states.
- Vanilla LSTM: The baseline embeds each pedestrian’s location, processes pedestrians independently with shared LSTM parameters, and predicts the next coordinates from the hidden state.The model is trained by minimizing L2 loss, and previously predicted coordinates are fed back during inference.
- State Refinement: SR-LSTM adds a state-refinement subnetwork that passes messages among pedestrians to refine their current LSTM cell states.The refinement module is integrated with the LSTM cells and updates participants’ states before next-step prediction.
- State Refinement: The refinement module takes current locations, hidden states, and cell states of all pedestrians as input and outputs refined cell states.Neighbor messages are integrated with each pedestrian’s cell state; iterations start from the original LSTM states.
- State Refinement: Message passing can be repeated for multiple iterations, enabling deeper interaction modeling before refined states produce predicted coordinates.The framework uses the refined states after L iterations for prediction and associates further refinement with intention negotiation.
- Message Passing: The simple message-passing implementation averages transformed neighboring hidden states before adding them to the pedestrian’s cell state.It treats neighboring features equally and uses a linear transformation for transmitting messages.
- Social-aware Information Selection: Social-aware selection combines an element-wise motion gate with pedestrian-wise attention to select useful neighboring features and regulate message strength.The motion gate uses pairwise motion and relative location, while attention emphasizes important neighbors; equal weighting performs worse.
4.1. Datasets and Metrics
The evaluation uses ETH and UCY pedestrian datasets with five crowd scenes, measuring mean and final displacement errors over 12 predicted time steps after observing eight positions.
- ETH and UCY provide five crowd sets containing 1,536 pedestrians and thousands of non-linear trajectories for evaluation.
- Performance is evaluated using Mean Average Displacement (MAD) and Final Average Displacement (FAD), both measured in meters.
- MAD is the mean Euclidean distance between ground-truth and predicted points across all predicted time steps.
- FAD is the Euclidean distance between the ground-truth and predicted points at the last frame.
- The protocol observes 8 ground-truth positions and predicts the following 12 time steps at 0.4-second intervals.
4.2. Implementation Details
The implementation embeds inputs with a compact MLP and uses a 64-dimensional LSTM state, training in single-step mode and validating or testing in multi-step mode.
- A single-layer MLP embeds input vectors into 32 dimensions, while the LSTM hidden state has dimension 64.
- Training samples use sliding windows of length 20 with stride 1, and trajectory segments within each window are processed as a mini-batch of 8.
- Training uses single-step teaching, whereas validation and testing use multi-step prediction in which outputs feed the next time step.
4.3. Ablation Study
The ablation study shows that preprocessing, selective neighborhood messaging, current neighbor states, and two refinement iterations each contribute to SR-LSTM performance.
- Normalized absolute location outperforms relative position, ETH-Univ frame-rate correction improves performance by about 12.7/9(%), and random rotation reduces overfitting.
- Simple state refinement with a 2-meter neighborhood outperforms V-LSTM by 6.4/6.8(%), while a 10-meter neighborhood changes performance by only 1.4/-0.2(%).
- Without feature selection, treating all pedestrians within 10 meters causes performance deterioration of 5.6/7.5(%) relative to a 2-meter neighborhood.
- With a 10-meter neighborhood, motion gating improves performance by 7.8/12.2(%), pedestrian-wise attention by 6.7/8.3(%), and both together by 11.8/16.4/(%).
- Using current neighbor states outperforms using previous states by 6/8.3(%), highlighting the value of neighbors’ latest features.
- A second refinement layer improves over one layer by 2.8/3(%), while a third layer adds no further promotion.
4.4. Comparison with Existing Works
SR-LSTM is compared with recurrent, social, and multimodal baselines under aligned evaluation settings, achieving lower reported error than V-LSTM with a single prediction.
- The comparison includes Social-LSTM, SGAN, Sophie, and other recent methods evaluated under the same dataset and evaluation methodology.
- SGAN and Sophie select the best match to ground truth from 20 samples, whereas other methods produce one prediction; Sophie also requires a scene image.
- The authors note that their reproduced V-LSTM does not completely match published V-LSTM*, potentially because of hyperparameters, data organization, or teaching mode.
- The authors report that their best model improves performance over V-LSTM by 15.4/18.8(%) using only a single prediction.
4.5. Qualitative Results
SR-LSTM qualitatively improves trajectory prediction when pedestrians change intentions and during common social interactions. Its current-state refinement captures changes that previous-state methods miss.
- SR-LSTM timely captures pedestrians’ changed intentions through current-state refinement and makes suitable trajectory adjustments.V-LSTM ignores interaction, while S-LSTM remains insensitive because it uses previous neighboring states.
- SR-LSTM makes appropriate predictions for consistent group walking, collision avoidance, and group avoidance.The comparisons show weaker interaction modeling in V-LSTM and S-LSTM.
4.6. Social-aware Information Selection
SR-LSTM selects socially relevant neighbor information with motion gating and pedestrian-wise attention. The mechanisms respond to interaction geometry, nearby pedestrians, and longer-range group effects.
- Motion gate: The motion gate selects neighbor hidden features according to pairwise trajectory configurations and potential future interactions.For pedestrian i, selected features of pedestrian j represent trajectories that j might cause to interact with i.
- Motion gate: Close, same-direction trajectory pairs activate features following the shared walking direction.This is the first gate pattern described among the four illustrated elements.
- Motion gate: Opposite-direction neighbors activate features that assess whether they will approach the target pedestrian, including at greater distances.The described patterns cover both relatively close and more distant neighbors.
- Motion gate: A stationary neighbor can receive a selected feature when the target may need to respond if that neighbor starts walking toward them.This pattern indicates sensitivity to possible future interaction with a currently static pedestrian.
- Pedestrian-wise attention: Pedestrian-wise attention prioritizes close neighbors, while later refinement can strengthen farther neighbors’ group-related or longer-range effects.The first refinement often focuses on close neighbors, whereas the second tends to increase attention to farther neighbors.
5. Conclusion
The paper concludes that SR-LSTM jointly predicts crowd trajectories by refining current pedestrian features through message passing and selecting useful neighbor information. The resulting model outperforms state-of-the-art approaches.
- SR-LSTM adaptively refines current features for all pedestrians through a message passing mechanism.The module treats LSTM as a feature extractor for joint trajectory prediction.
- An element-wise motion gate and pedestrian-wise attention select useful features from each neighbor.Together they form the paper’s social-aware information selection mechanism.
- SR-LSTM with information selection outperforms state-of-the-art approaches.