Source-linked AI summary
Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection
Tharindu Fernando, Simon Denman, Sridha Sridharan, Clinton Fookes
TL;DR
The paper addresses modeling complex crowd behavior beyond hand-crafted features. It proposes a neural attention framework with hard-wired attention and reports state-of-the-art performance, including abnormal-event detection without hand-crafted features.
Problem
Modeling complex real-world crowd behavior remains challenging when approaches rely on hand-crafted features.
Method
The paper proposes a neural attention framework combining learned soft attention with hand-crafted hard-wired attention to approximate neighbors’ influence.
Results
The framework achieves state-of-the-art performance and accurately detects abnormal events without handcrafting features.
Takeaways & Limitations
The attention framework supports trajectory modeling in heavily cluttered neighborhoods and abnormal-event detection through hidden-state clustering.
Takeaways & Limitations
The paper notes limitations involving long-term path planning and trajectories with abnormal velocities or changes in walking directions.
Abstract
from arXiv · showhide
As humans we possess an intuitive ability for navigation which we master through years of practice; however existing approaches to model this trait for diverse tasks including monitoring pedestrian flow and detecting abnormal events have been limited by using a variety of hand-crafted features. Recent research in the area of deep-learning has demonstrated the power of learning features directly from the data; and related research in recurrent neural networks has shown exemplary results in sequence-to-sequence problems such as neural machine translation and neural image caption generation. Motivated by these approaches, we propose a novel method to predict the future motion of a pedestrian given a short history of their, and their neighbours, past behaviour. The novelty of the proposed method is the combined attention model which utilises both "soft attention" as well as "hard-wired" attention in order to map the trajectory information from the local neighbourhood to the future positions of the pedestrian of interest. We illustrate how a simple approximation of attention weights (i.e hard-wired) can be merged together with soft attention weights in order to make our model applicable for challenging real world scenarios with hundreds of neighbours. The navigational capability of the proposed method is tested on two challenging publicly available surveillance databases where our model outperforms the current-state-of-the-art methods. Additionally, we illustrate how the proposed architecture can be directly applied for the task of abnormal event detection without handcrafting the features.
1. Introduction
The paper frames pedestrian trajectory prediction and abnormal-event detection as data-driven sequence-to-sequence problems, using neighbouring trajectories to model navigation without relying solely on hand-crafted features. It proposes an LSTM architecture combining soft and hard-wired attention, and evaluates it on surveillance data.
- Crowd behaviour modelling remains challenging for applications including security systems, traffic-flow estimation, abnormal-event detection, and behaviour prediction.
- Human navigation dynamics cannot be captured adequately with only a handful of hand-crafted features, motivating direct learning from pedestrian trajectories.
- Problem Definition: The task predicts a target pedestrian’s future trajectory from their observed history and neighbouring pedestrians’ trajectories as a sequence-to-sequence problem.
- Proposed Architecture: The proposed architecture uses LSTM encoders and combines soft attention with hard-wired attention to process neighbouring trajectories and predict future positions.
- Evaluation: The method learns common human-navigation patterns, improves pedestrian-path predictions over current state-of-the-art methods, and is also applied to abnormal human behaviour detection.
2. Related Work
Prior work models trajectories through clustering, social interactions, force-based mechanisms, and attention, but commonly depends on hand-crafted features or fails to represent agent interactions. The paper motivates combined attention as a way to reduce the computational burden of soft alignment in crowded surveillance scenes.
- Trajectory Clustering: Trajectory-pattern and sub-trajectory methods discover frequent behaviours or clusters by partitioning trajectories into similar line segments.
- Trajectory Clustering: Clustering quality can depend strongly on the distance measure, while the clustering method itself may have little effect on final results.
- Human Behaviour Prediction: Human-behaviour models include social dependencies, attractive and repulsive forces, social affinity features, stationary crowd groups, and topic-based motion patterns.
- Human Behaviour Prediction: Some prior approaches ignore interactions among agents, despite their importance for predicting behaviour in real-world scenarios.
- Human Behaviour Prediction: A recurring drawback of earlier methods is reliance on hand-crafted features that capture abstract environmental semantics and depend heavily on domain knowledge.
- Attention Models: Attention mechanisms align input and output sequences, but soft attention becomes inefficient when it must compute values for every input-output combination.
- Attention Models: Hard attention commonly uses reinforcement learning to approximate focus locations, but learning it can be computationally expensive because it requires sampling.
- Attention Models: In surveillance scenes with hundreds of neighbours, exhaustive soft alignment can become prohibitively expensive, motivating the merger of soft and hard-wired attention.
3. Proposed Approach
The proposed framework predicts pedestrian trajectories by combining learned soft attention for the target pedestrian with distance-based hardwired attention for neighbours. It encodes trajectory sequences with LSTMs, merges the two attention contexts, and supports trajectory clustering and abnormal-event detection.
- Motivation: Soft attention aligns trajectory features with future outputs, but separately applying it to every neighbour is computationally expensive.The model addresses the differing influence of neighbours while avoiding full soft-attention computation for all trajectories.
- Hardwired Attention: Hardwired attention approximates neighbour influence using weights calculated from distance to the pedestrian of interest.Closer neighbours receive higher weights because they are treated as having greater influence on the predicted trajectory.
- Combined Attention: The architecture encodes target and neighbour trajectories with LSTMs, applies soft attention to the target, and applies hardwired attention to neighbouring trajectories.The resulting context vectors are concatenated and used by an LSTM decoder to predict future positions.
- LSTM Decoder: The decoder uses the combined context vector with its previous hidden state and output to predict the next trajectory state.The decoding function is implemented with an LSTM.
- Trajectory Clustering: Neighbourhood preprocessing retains up to ten nearby trajectories per direction, aggregates excess neighbours, and assigns zero weight to dummy trajectories.For more than ten neighbours, the nine closest trajectories and the remaining neighbours’ mean trajectory are retained.
- Trajectory Clustering: Trajectory clusters are used to train separate prediction models, while testing assigns observed trajectories to the closest cluster centre.DBSCAN is used because it can cluster data without specifying the number of clusters.
- Combined Attention: The combined attention model considers the entire sequence of hidden states for the target and neighbouring pedestrians during prediction.This contrasts with approaches that use only a previous hidden state or pooled neighbour information.
4. Experiments
The experiments evaluate trajectory prediction on two surveillance datasets using entry/exit-based clustering, combined attention, and comparisons with established baselines. The proposed models outperform the baselines across datasets and qualitative scenarios, while failures mainly involve sudden destination changes.
- Datasets: The experiments use the Grand Central and Edinburgh Informatics Forum surveillance datasets for human trajectory prediction.Grand Central contains around 12,600 trajectories, while Edinburgh contains around 90,000.
- Experimental setup: Entry/exit-zone clustering separates motion modes while avoiding the many clusters, outliers, and sparse training examples produced by whole-trajectory clustering.The approach was reported to work reasonably well at separating different modes of human motion.
- Quantitative results: The proposed model outperforms Social Force and Social LSTM on both datasets, with larger error reductions on Grand Central's varied crowd patterns.Grand Central includes multiple source and sink positions, different crowd motion patterns, and varied motion paths.
- Ablations: Without clustering, the combined attention architecture still outperforms Social LSTM across all measured error metrics.This result indicates that the combined attention architecture improves prediction independently of the clustering process.
- Ablations: Historical information from pedestrians and neighbours, hard-wired neighbour weights, and cluster-specific models each improve prediction quality and contextual specificity.The clustered combined-attention model produces more accurate predictions than the unclustered version and better captures neighbour influence across contexts.
- Qualitative results: Failure cases mainly arise from sudden destination changes, although the generated trajectories remain plausible despite deviations from ground truth.The paper also reports more spatially specific predictions from the clustered combined-attention model.
- Qualitative results: Qualitative examples show improved handling of motion-mode changes, long-range context, neighbour history, collision avoidance, and crowded areas.The combined-attention mechanism generates more accurate and realistic trajectories than the state-of-the-art baseline in the reported examples.
5. Abnormal behaviour detection
The framework is also applied to abnormal pedestrian behaviour detection by identifying outliers in LSTM hidden states. It detects abnormal trajectories including sudden turns and abnormal velocities, while uncommon normal behaviours can produce false alarms.
- Method: The framework detects abnormal behaviour by extracting encoder and decoder hidden states and passing them through DBSCAN to identify outliers.The method uses hidden states from both the observed trajectory and the decoding period.
- Limitations: Because of the adaptive nature of the neural network, sudden turns, direction changes, and abnormal velocities may not always be classified as abnormal.This limitation applies despite the framework's use of encoded hidden-state representations.
- Experimental setup: The evaluation labels sudden turns, walking-direction changes, and abnormal velocities as abnormal behaviours in 500 Grand Central trajectories.The selected set contains 445 normal and 55 abnormal trajectories.
- Limitations: False alarms occur when uncommon normal behaviours, such as buying tickets or wandering, are poorly represented in the selected trajectory subset.The paper attributes these errors to limited representation of those behaviours in the database subset.
- Results: The proposed approach detects 47 of 55 ground-truth abnormal trajectories, compared with 29 of 55 for the naive approach.The proposed approach detected 441 trajectories as normal and 59 as abnormal.
- Qualitative results: The method can classify sudden circular turns and abnormal velocity even when predicted and observed paths do not differ substantially.The reported examples include direction-change and abnormal-velocity events.
6. Conclusion
The paper proposes an LSTM sequence-to-sequence framework combining soft and hard-wired attention for pedestrian trajectory prediction and abnormal event detection. It achieves state-of-the-art performance in two surveillance datasets, including crowded settings, and detects abnormal events without handcrafted features.
- 6. Conclusion: The framework extends sequence-to-sequence modelling with both soft and hard-wired attention for pedestrian trajectory prediction.Hard-wired weights approximate neighbours’ influence, making attention applicable to real-world scenarios with many neighbours.
- 6. Conclusion: The proposed model demonstrates state-of-the-art performance on two challenging publicly available surveillance datasets.The evaluation tested the model on two surveillance datasets.
- 6. Conclusion: The combined attention architecture improves trajectory prediction for multiple source and sink positions and high crowd densities.The reported improvement includes challenging surveillance settings with heavily cluttered neighbourhoods.
- 6. Conclusion: The framework supports abnormal event detection through hidden-state clustering without handcrafted features.The authors report accurate detection in challenging situations.
- 6. Conclusion: The approach can be extended to applications requiring modelling of multiple co-occurring trajectories, including aircraft, ships, and vehicles.The conclusion identifies aircraft movements, ship trajectories, and vehicle traffic as potential application areas.