Source-linked AI summary
Pedestrian Action Anticipation using Contextual Feature Fusion in Stacked RNNs
Amir Rasouli, Iuliia Kotseruba, John K. Tsotsos
TL;DR
Pedestrian crossing prediction matters for autonomous vehicles because motion history alone is insufficient for understanding variable behavior in complex scenes. The paper introduces SF-GRU, a stacked recurrent architecture that gradually fuses visual, contextual, and dynamic features. Empirical evaluations find that SF-GRU generally outperforms alternative RNN architectures, while performance depends on feature selection, timing, and fusion order.
Problem
Trajectory-based prediction is insufficient because pedestrian actions vary and depend on visual observations, surroundings, and other contextual information.
Method
SF-GRU gradually fuses pedestrian, scene, and vehicle information at different levels of a stacked recurrent network.
Results
SF-GRU performs best on all metrics except recall when using all information, while single-layer GRU has 1.2% higher recall but more than 6% lower precision.
Takeaways & Limitations
Performance is optimal when more complex features enter lower layers and simpler features enter higher layers.
Takeaways & Limitations
The JAAD dataset has few samples, lacks vehicle information, and contains short sequences unsuitable for long-term predictions.
Abstract
from arXiv · showhide
One of the major challenges for autonomous vehicles in urban environments is to understand and predict other road users' actions, in particular, pedestrians at the point of crossing. The common approach to solving this problem is to use the motion history of the agents to predict their future trajectories. However, pedestrians exhibit highly variable actions most of which cannot be understood without visual observation of the pedestrians themselves and their surroundings. To this end, we propose a solution for the problem of pedestrian action anticipation at the point of crossing. Our approach uses a novel stacked RNN architecture in which information collected from various sources, both scene dynamics and visual features, is gradually fused into the network at different levels of processing. We show, via extensive empirical evaluations, that the proposed algorithm achieves a higher prediction accuracy compared to alternative recurrent network architectures. We conduct experiments to investigate the impact of the length of observation, time to event and types of features on the performance of the proposed method. Finally, we demonstrate how different data fusion strategies impact prediction accuracy.
1 Introduction
Pedestrian crossing anticipation is important for safe urban autonomous driving, but motion history alone cannot reliably capture highly variable behavior. The proposed approach combines visual, contextual, and dynamic information to anticipate whether pedestrians will cross.
- Crossing anticipation helps autonomous vehicles choose actions that avoid potential collisions and traffic disruption.
- Trajectory-based methods can fail when pedestrians have no motion history, walk alongside the road, or abruptly change their walking pattern.
- Pedestrian actions are influenced by signals, ego-vehicle motion, road structure, and other environmental factors.
- The proposed method combines pedestrian appearance, pose, dynamics, surrounding context, and ego-vehicle speed for crossing anticipation.
2 Related Works
Prior work uses recurrent architectures for sequential action prediction, but pedestrian crossing prediction requires broader contextual reasoning than trajectory-based approaches alone provide. This paper proposes multilevel feature fusion in a stacked recurrent network and evaluates alternative architectures, feature sources, and fusion orders.
- RNNs, GRUs, and LSTMs are widely used to model sequential data for activity recognition, action prediction, and video captioning.
- Stacked and hierarchical RNNs propagate representations through successive layers, with raw inputs entering at the bottom layer.
- Pedestrian action prediction commonly uses encoder-decoder RNNs that encode observations and infer future predictions from learned representations.
- Existing approaches incorporate social interactions, road structure, head orientation, or signals, but may omit relevant scene dynamics, visual processing, or broader evaluation contexts.
- The paper proposes a stacked recurrent network with multilevel feature fusion and evaluates architectures, observation timing, contextual sources, and feature-input order.
3 Approach
The approach formulates pedestrian crossing anticipation as binary classification and combines visual, pose, pedestrian-motion, and ego-vehicle-speed information in a stacked GRU architecture. Visual features enter lower levels, while dynamics are fused at higher levels to reflect their differing complexity.
- Pedestrian crossing anticipation is defined as determining whether pedestrian i will cross the street from context observed through time m.
- The model uses five information sources, including local visual context, pedestrian pose, bounding-box motion, and ego-vehicle speed.Local context covers the pedestrian and surroundings; bounding-box displacement represents pedestrian velocity, while speed records the ego-vehicle motion.
- A stacked RNN architecture gradually fuses features according to complexity, placing visual scene features at lower levels and trajectories and speed at higher levels.This design uses spatial depth for visual features and higher levels for dynamics features.
- Local context: Local context is extracted from pedestrian crops and expanded surrounding regions that may include streets, pedestrians, signals, or traffic.
- Multimodal feature fusion: The architecture uses GRUs with gated updates, and each level receives the preceding level's representation before producing the final prediction.The final prediction is obtained through a linear transformation of the top-level hidden state.
- The implementation uses 256 hidden units, VGG16-derived 512-dimensional context features, 18-joint pose estimates, and five stacked levels trained with binary cross-entropy.Training uses ADAM for 60 epochs with data augmentation and class balancing.
4 Experiments
Experiments evaluate SF-GRU on pedestrian crossing anticipation using multimodal features, alternative recurrent architectures, and varying observation and time-to-event settings. Results show that contextual feature selection and fusion order substantially affect prediction performance.
- Experimental setup: The experiments use PIE pedestrian tracks recorded from an onboard monocular camera, with bounding boxes, vehicle sensor data, event points, and a 60–40 train-test split.The evaluation reports accuracy, F1 score, precision, recall, and AUC.
- Experimental setup: The main evaluation uses 0.5s observation sequences at 2s time to event and compares SF-GRU with static, GRU, and alternative feature-processing architectures.The selected TTE corresponds to the minimum time within which pedestrians make crossing decisions according to the cited reference.
- Baseline comparison: Approximately 60% accuracy from static local-context images improves by 9% with GRU-based temporal reasoning.The result indicates that temporal modeling improves over single-frame visual information in this evaluation.
- Baseline comparison: SF-GRU performs best on all metrics except recall, where single-layer GRU is 1.2% better but incurs a more than 6% precision drop.Simply adding network layers or separating feature modalities does not improve performance.
- Time to event and observation length: SF-GRU performs best for most time-to-event points, while algorithm performance generally degrades as observations move farther from the crossing event.At early TTE values, when pedestrian intent is clearer, the algorithms perform similarly; single-layer GRU declines rapidly after 2s TTE relative to M-GRU and S-GRU.
- Time to event and observation length: In the 1–2s decision region, increasing observation from 0.3s to 0.5s yields a small gain, while longer observations can reduce performance because accumulated scene-dynamics changes add noise.Longer observations improve precision near 0s or 3s in exchange for lower recall; vehicle-speed changes are given as one source of noise.
- Feature types: Adding contextual information to pedestrian appearance improves overall performance by more than 18%, while decoupling appearance and surround context adds almost 3% accuracy.Using bounding-box coordinates rather than center coordinates improves results by 2%, potentially because scale changes encode motion or distance to the ego-vehicle.
- Feature fusion: Ordering feature fusion so complex visual features enter lower layers and simpler motion features enter higher layers improves performance by up to 9% accuracy, 10% recall, and more than 15% precision.The authors associate this pattern with deeper spatial analysis for visual features and higher-level processing of trajectory coordinates.
5 Conclusion
The paper presents a stacked RNN that gradually fuses pedestrian and vehicle dynamics, appearance, and surroundings at different processing levels. It reports the best performance among evaluated RNN architectures and identifies feature ordering as important for crossing prediction.
- The proposed stacked RNN gradually fuses pedestrian and vehicle dynamics, pedestrian appearance, and surroundings at different processing levels.
- Empirical evaluations show that the proposed approach performs best compared to alternative RNN architectures.
- Performance is optimal when more complex features enter bottom layers and simpler features enter higher levels.
- The authors report that different information sources and fusion strategies affect crossing action prediction.
- The architecture may also benefit related applications such as activity recognition.