Source-linked AI summary
Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention
Dongfang Yang, Haolin Zhang, Ekim Yurtsever, Keith Redmill, Ümit Özgüner
TL;DR
Pedestrian crossing intention prediction needs to use both temporal scene context and heterogeneous sensor-derived features for real-world autonomous driving. The paper introduces a hybrid attention-based fusion architecture and reports state-of-the-art performance on JAAD-related pedestrian action prediction benchmarks.
Problem
Existing vision-based predictors insufficiently use global scene context and have not established an optimal strategy for fusing different features, despite the importance of pedestrian behavior prediction for autonomous driving.
Method
The method combines RGB imagery, semantic segmentation, pedestrian features, and ego-vehicle speed through 2D CNNs, recurrent networks, attention mechanisms, and hybrid fusion.
Results
The proposed model achieves state-of-the-art performance on the pedestrian action prediction benchmark and improves F1 by about 4%.
Takeaways & Limitations
Explicit global context and hybrid fusion provide the paper’s supported route to combining target-pedestrian and scene information for crossing-intention prediction.
Abstract
from arXiv · showhide
Predicting vulnerable road user behavior is an essential prerequisite for deploying Automated Driving Systems (ADS) in the real-world. Pedestrian crossing intention should be recognized in real-time, especially for urban driving. Recent works have shown the potential of using vision-based deep neural network models for this task. However, these models are not robust and certain issues still need to be resolved. First, the global spatio-temproal context that accounts for the interaction between the target pedestrian and the scene has not been properly utilized. Second, the optimum strategy for fusing different sensor data has not been thoroughly investigated. This work addresses the above limitations by introducing a novel neural network architecture to fuse inherently different spatio-temporal features for pedestrian crossing intention prediction. We fuse different phenomena such as sequences of RGB imagery, semantic segmentation masks, and ego-vehicle speed in an optimum way using attention mechanisms and a stack of recurrent neural networks. The optimum architecture was obtained through exhaustive ablation and comparison studies. Extensive comparative experiments on the JAAD pedestrian action prediction benchmark demonstrate the effectiveness of the proposed method, where state-of-the-art performance was achieved. Our code is open-source and publicly available.
I. INTRODUCTION
The paper targets vision-based prediction of whether a pedestrian will cross within 1–2 seconds, motivated by the need for safer autonomous-driving interactions. It combines visual and non-visual spatiotemporal information and evaluates fusion strategies on JAAD.
- Motivation: Pedestrian crossing intention prediction supports autonomous-driving systems’ understanding of vulnerable road-user behavior.The paper focuses on whether a front-facing-camera pedestrian will cross the road shortly.
- Task: The task predicts crossing or not-crossing within a short 1–2-second horizon from video and vehicle information.Inputs are collected from frames preceding the crossing or not-crossing event.
- Approach: The proposed model fuses pedestrian appearance, scene context, bounding boxes, pose keypoints, and ego-vehicle speed.Global context is represented through semantic segmentation, while local context captures enlarged pedestrian appearance.
- Evaluation: The paper compares early, later, hierarchical, and hybrid fusion strategies alongside alternative input configurations and visual encoders.The ablations also compare 3D CNNs with 2D convolution combined with RNNs and attention.
- Results: The proposed method achieves state-of-the-art performance on the recent pedestrian action prediction benchmark and is evaluated on JAAD.The introduction identifies JAAD as the commonly used dataset for demonstrating efficiency.
II. RELATED WORK
Prior work progressed from single-frame CNNs to spatiotemporal models and multimodal feature fusion. The paper identifies missing global scene context and potentially suboptimal fusion strategies as open issues.
- Spatio-temporal modeling: Single-frame CNN approaches omit temporal information, motivating image-sequence models that combine spatial feature extraction with recurrent temporal encoding.Common combinations use 2D CNNs or GCNs followed by LSTM or GRU models.
- Spatio-temporal modeling: 3D CNNs offer an alternative by directly extracting spatiotemporal features from pedestrian image sequences.They replace 2D convolution and pooling kernels with 3D counterparts.
- Feature fusion: Feature-fusion methods separately model pedestrian appearance, pose, location, vehicle motion, and explicit global scene context.The task therefore requires a strategy for combining heterogeneous input channels.
- Feature fusion: SF-GRU hierarchically fuses five feature sources, but it does not incorporate global context.The cited sources include appearance, surrounding context, pose, bounding box, and ego-vehicle speed.
III. PROPOSED METHOD
The proposed method estimates pedestrian crossing probability from sequential visual, pedestrian-state, and ego-vehicle inputs. These sources are extracted as separate channels over m + 1 observations.
- Problem formulation: The task estimates the probability that target pedestrian i will cross the road at future action A_t+n.The action is binary, with t denoting the last observed frame and n the frame offset to the event.
- Input sources: The model uses local image context, bounding-box trajectories, pose keypoints, ego-vehicle speed, and semantic-segmentation global context.These explicit features are extracted and supplied as separate input channels.
- Input sources: Each input source is represented as a sequence of length m + 1.The sequential sources are illustrated in the model overview.
B. Input acquisition
The model acquires visual, geometric, pose, vehicle-speed, and global-scene inputs over time to represent pedestrian motion and road interactions.
- Local and trajectory inputs: Local context consists of RGB image sequences around the target pedestrian, while the 2D trajectory records bounding-box position changes.The trajectory and local context can be obtained through detection and tracking, although this work directly uses dataset ground truth.
- Pose input: Pose keypoints encode the pedestrian’s detailed posture and motion across frames using 18 joints represented by 36-dimensional 2D-coordinate vectors.Because JAAD lacks ground-truth pose annotations, the work extracts keypoints with a pretrained OpenPose model.
- Vehicle input: Ego-vehicle speed is included as a temporal input because it is described as a major factor affecting the pedestrian’s crossing decision.The study directly uses the dataset’s ground-truth speed annotations.
- Global context: Global context is represented by pixel-level semantic masks that classify and localize important scene objects, including roads, pedestrians, and vehicles.DeepLabV3 pretrained on Cityscapes extracts the masks because JAAD does not provide semantic-mask annotations.
C. Model architecture
The architecture combines recurrent encoding and attention across non-visual and visual branches, then fuses their representations for final action prediction.
- Overall architecture: The model contains CNN, RNN, attention, and feature-fusion modules for processing multimodal sequential inputs.Its fusion design is hybrid, with separate non-visual and visual branches.
- Visual encoding: Sequential RGB inputs are encoded with VGG19 features and GRU hidden representations before attention produces a [1,256] feature tensor.Each observed frame produces a [512,14,14] VGG19 feature map, pooled into sequential features of size [16,512].
- Non-visual branch: The non-visual branch hierarchically fuses pose keypoints, 2D location trajectory, and ego-vehicle speed through successive RNN-based encoders.Each later fusion stage combines the previous representation with the next feature source.
- Visual branch: The visual branch encodes local pedestrian appearance and global semantic context with CNN-GRU sequences, applies attention to each, and concatenates the resulting visual vectors.Local context focuses on enlarged pedestrian appearance, while global context covers important objects across the scene.
- Final prediction: Final visual and non-visual feature vectors are concatenated, passed through another attention block, and processed by a fully connected layer to predict the action.The fusion strategy is compared with later, early, and hierarchical alternatives in the study’s ablation design.
A. Dataset and Benchmark
Evaluation uses the JAAD benchmark, including behavioral and all-data subsets, with a standardized configuration and multiple classification metrics.
- Dataset: JAADbeh contains 495 crossing and 191 about-to-cross pedestrian samples, while JAADall adds 2,100 pedestrians with non-crossing actions.The benchmark configuration follows, including 0.8 data-sample overlap and a 1.5 local-context scale.
- Metrics: Performance is evaluated using accuracy, AUC, F1 score, precision, and recall.These metrics are described as widely recognized and commonly used by related works.
B. Implementation
The proposed model is evaluated against established pedestrian-intention baselines using a benchmark implementation and specified training regularization.
- Comparative evaluation: The proposed model is compared with SingleRNN, SF-GRU, and PCPA using the benchmark implementation released with PCPA.The authors developed their model within this benchmark framework.
- Training setup: Training uses dropout 0.5 in the attention module, L2 regularization 0.001 in the fully connected layer, binary cross-entropy loss, and Adam optimization.The implementation also specifies a learning rate of 5×10^-7, although the passage truncates the remaining training details.
C. Ablation study
The ablation studies compare visual encoders, global-context inputs, and feature-fusion strategies across baseline models and seven proposed variants. Results indicate benefits from global context and 2D CNN plus RNN encoding, while hybrid fusion performs best.
- Experimental design: Seven proposed variants were compared with SingRNN, SF-GRU, PCPA, and the proposed model across multiple encoder, input, and fusion configurations.The study tested 2D CNN plus RNN versus 3D CNN, models with or without semantic segmentation, and later, early, hierarchical, or hybrid fusion.
- Quantitative results: The proposed model achieved the best accuracy, F1, and recall on the JAAD behavior subset, with about 4% improvement in F1.F1 balances recall and precision and is identified as an important binary-classification metric.
- Quantitative results: The proposed model achieved the best accuracy, AUC, F1, and precision on JAADall, which contains more non-crossing samples and more closely resembles real-world data distributions.The model also ranked best on the two emphasized metrics, F1 and accuracy.
- Ablation findings: Introducing global context improved model performance when comparing Ours5 with Ours4 and Ours1 with PCPA.The ablation compares variants with and without semantic segmentation as an explicit global feature.
- Ablation findings: Using 2D CNN plus RNN instead of 3D CNN improved spatio-temporal feature extraction and model performance, while hybrid fusion achieved the best performance among fusion variants.Hybrid fusion was compared against later, early, and hierarchical fusion through Ours, Ours5, Ours6, and Ours7.
B. Qualitative Results
Qualitative examples indicate that global context helps the proposed model handle pedestrian direction, occlusion, and poor vision, while some intersection and illumination cases remain difficult.
- Global context helps resolve unknown pedestrian direction, occlusion, and poor vision by modeling interaction between the whole scene and target pedestrian.
- Semantic segmentation of the drivable area helps indicate whether a pedestrian is moving toward or onto the drivable area.
- Both the proposed and PCPA models fail in an intersection case where the pedestrian is near the far road edge after crossing.
- Poor illumination can prevent the model from obtaining sufficient detailed features, producing prediction failures.
VI. CONCLUSION
The paper proposes a vision-based pedestrian crossing-intention predictor that combines global scene context with hybrid feature fusion. Experiments on JAAD report state-of-the-art performance, while future work targets robustness and broader information sources.
- The method represents global context as a channel for target-pedestrian and whole-scene interaction and uses 2D CNNs, RNNs, and attention for hybrid feature fusion.
- Qualitative results include cases where the proposed model succeeds while PCPA fails, alongside cases where both models fail.
- The proposed method achieves state-of-the-art performance against baseline methods on the JAAD pedestrian action prediction benchmark.
- Future work includes improving robustness in poor vision and occlusion, incorporating more information sources, and tuning for specific pedestrian subsets.