Source-linked AI summary
TrajFusionNet+: Transformer-Based Prediction of Pedestrian Crossing Intention via Fusion of Trajectory Representations and Scene Graphs
François G. Landry, Moulay A. Akhloufi
TL;DR
Pedestrian crossing-intention prediction remains difficult because behavior depends on dynamic interactions and scene context, while same-dataset evaluation can overstate generalization. TrajFusionNet+ fuses trajectory, visual, and graph representations through three attention branches. It achieves improved state-of-the-art performance on PIE and JAAD and superior generalization under joint training with separate evaluation.
Problem
Pedestrian crossing-intention prediction is challenging because behavior depends on interactions and environmental context, while conventional same-dataset evaluation can inflate performance.
Method
TrajFusionNet+ combines sequential trajectory and speed features, visual trajectory overlays, and pedestrian-centric scene graphs through SAM, VAM, and GAM.
Results
TrajFusionNet+ achieves improved state-of-the-art performance on PIE and JAAD and superior generalization under joint training with separate evaluation.
Takeaways & Limitations
Joint cross-dataset training with separate evaluation provides a protocol under which TrajFusionNet+ demonstrates stronger generalization than existing approaches.
Abstract
from arXiv · showhide
The pedestrian crossing intention task involves predicting whether pedestrians are likely to cross the road from the point of view of an autonomous vehicle. We introduce TrajFusionNet+, a novel transformer-based model for pedestrian crossing intention prediction. TrajFusionNet+ combines sequential and visual representations of pedestrian trajectory with a graph-based representation of the scene context in order to predict pedestrian crossing intention. The proposed architecture builds upon our previous model, TrajFusionNet, and comprises three branches: a Sequence Attention Module (SAM), which processes a sequential representation of past and predicted pedestrian trajectories; a Visual Attention Module (VAM), which utilizes a visual representation of the pedestrian trajectories by overlaying observed and predicted bounding boxes onto scene images; and a Graph Attention Module (GAM), which extracts pedestrian-centric graphs from segmented scene images and captures the relational dependencies between pedestrians and traffic elements. TrajFusionNet+ achieves improved state-of-the-art performance on the two most widely used pedestrian crossing intention datasets, PIE and JAAD. Furthermore, we introduce a new evaluation protocol in which models are trained jointly on the PIE and JAAD datasets but evaluated separately on each. Under this setting, TrajFusionNet+ demonstrates superior generalization compared to existing approaches.
I. INTRODUCTION
TrajFusionNet+ addresses pedestrian crossing-intention prediction by fusing trajectory, visual, and graph-based scene representations. It also targets cross-dataset generalization through joint training on PIE and JAAD with separate evaluation.
- I. INTRODUCTION: Cross-dataset evaluation is important because same-dataset training and testing can inflate performance and poorly reflect deployment without retraining.
- I. INTRODUCTION: TrajFusionNet+ combines sequential, visual, and graph representations through SAM, VAM, and GAM branches for crossing-intention prediction.SAM processes past and predicted trajectories and vehicle speed; VAM overlays trajectory boxes on scene images; GAM models pedestrian-centric relations with traffic elements.
- I. INTRODUCTION: The proposed protocol jointly trains models on PIE and JAAD, then evaluates them separately to assess generalization across datasets.
- I. INTRODUCTION: Prior work uses sequential, multimodal, visual-transformer, and graph-based approaches, but multimodal modeling can increase inference time.
III. METHOD
The task predicts whether each detected pedestrian will initiate crossing within a future window from observed video frames. It is formulated as binary classification at time t.
- III. METHOD: Crossing intention is predicted as a binary action for each pedestrian from previous observations in a video sequence.
- III. METHOD: The benchmark predicts whether a pedestrian will cross between 1 and 2 seconds after time t.
B. Input Modalities
The model uses pedestrian bounding boxes, vehicle speeds, and scene images as its three input modalities. Speed representation differs between PIE and JAAD because the datasets provide different speed formats.
- B. Input Modalities: The input consists of pedestrian bounding-box sequences, vehicle-speed sequences, and scene-video sequences.Bounding boxes encode the top-left and bottom-right coordinates of each pedestrian box.
- B. Input Modalities: PIE uses raw vehicle-speed values, whereas JAAD's categorical speed labels are encoded ordinally from stopped to accelerating.
C. Architecture
TrajFusionNet+ combines sequence, visual, and graph branches to predict pedestrian crossing intention, using trajectory forecasting, temporal scene representations, and pedestrian-centric graph modeling. Their outputs are integrated through transformer processing and late fusion.
- C. Architecture: Three branches—SAM, VAM, and GAM—process sequential trajectories, visual scene representations, and pedestrian-centric graph relationships.The architecture combines these branch outputs through a late-fusion dense layer.
- C. Architecture: SAM predicts future pedestrian bounding boxes and vehicle speed, then encodes past and predicted trajectories with GAM features for classification.The prediction spans the next 60 timesteps, up to 2 seconds after the current time.
- C. Architecture: GAM constructs pedestrian-centric star graphs from segmented scene images, with the target pedestrian as the central node and other traffic elements as neighboring nodes.Node coordinates are normalized, while edge distances and angles are computed relative to the target pedestrian.
- C. Architecture: TokenGT represents graph nodes and edges as tokens, allowing self-attention to model dependencies beyond explicitly connected graph elements.Graph encodings are produced at four evenly spaced observation timesteps using a shared encoder-only transformer.
- C. Architecture: VAM applies VANs to scene images overlaid with observed and predicted pedestrian bounding boxes, then uses an encoder transformer to model temporal dependencies.The four visual encodings correspond to timesteps t −15, t −10, t −5, and t.
D. Training and Model Settings
TrajFusionNet+ uses staged modular training, independently pretraining lower-level components before freezing them and fine-tuning subsequent modules. Trajectory prediction uses MSE, while crossing classification uses weighted cross-entropy under benchmark evaluation settings.
- D. Training and Model Settings: Lower-level modules are pretrained independently, frozen after training, and then integrated modules are fine-tuned in stages.Temporary classification heads are attached during pretraining and removed after integration into the full model.
- D. Training and Model Settings: The trajectory predictor is pretrained for 60 future timesteps using MSE over four bounding-box coordinates and vehicle speed.PIE training produces 49,527 trajectory-prediction sequences.
- D. Training and Model Settings: Remaining modules are trained as binary crossing classifiers with weighted cross-entropy to address class imbalance.PIE provides 4,770 training sequences under the specified dataset split and evaluation settings.
- D. Training and Model Settings: The SAM, GAM, and VAM components are trained sequentially, with trajectory prediction and GAM frozen while the SAM encoder is trained, followed by modular VAM training and final projection updates.Only SAM and VAM projection layers and final dense layers are updated at the final stage.
- D. Training and Model Settings: Encoder-only transformers use 6 layers, 8 attention heads, d_model = 128, and 1024-dimensional feed-forward layers.These settings apply to the SAM, VAM, and TokenGT encoder-only transformers.
A. Datasets
The study evaluates pedestrian crossing intention on PIE and JAADall using the benchmark protocol, with separate and combined-dataset evaluation settings.
- JAAD: JAAD contains 346 naturalistic pedestrian video clips and behavioral annotations but no numerical vehicle-speed values.Its recordings span North America and Europe under diverse weather conditions.
- PIE: PIE provides nearly ten times more frames than JAAD, longer clips, and richer ego-vehicle information including speed, GPS, and heading angle.PIE recordings were collected in clear weather in Toronto, Canada.
- Evaluation Procedure: PIE and JAADall are evaluated using 0.53-second observations to predict crossing 1–2 seconds later.The protocol uses 16 observation frames and predicts crossing 30–60 frames after observation.
- Evaluation Procedure: Performance is reported with accuracy, AUC, F1-score, precision, and recall for models trained and tested on individual datasets.The study also includes combined PIE–JAAD training and evaluation on the combined and individual datasets.
B. Results
TrajFusionNet+ achieves strong benchmark performance on PIE and JAADall, while maintaining more consistent results across datasets than some competing models.
- Benchmark comparison: TrajFusionNet+ obtains the highest accuracy and F1-score on PIE, matching PedFormer, and the second-best scores on both metrics for JAADall.Compared with TrajFusionNet, it shows small improvements across several metrics.
- Cross-dataset consistency: The TrajFusionNet family performs consistently across datasets and metrics, whereas PedFormer combines strong PIE results with a low JAADall F1-score.The authors interpret this consistency as indicating less susceptibility to dataset-specific overfitting.
- Benchmark comparison: TrajFusionNet+ achieves state-of-the-art results in the comparison of pedestrian crossing intention models.Table III compares the reported models on PIE and JAADall.
C. Training on Data from Combined Datasets
The paper proposes joint training on PIE and JAADall followed by separate evaluation, aiming to measure generalization under fixed weights across datasets.
- Evaluation protocol: Fixed-weight cross-dataset evaluation is presented as a more realistic generalization measure because autonomous-vehicle models are typically not retrained for each environment.JAADall lacks numerical vehicle-speed values, so methods are compared without speed under this protocol.
- Evaluation protocol: The protocol trains models on combined PIE and JAADall data and evaluates them on the combined set and each dataset separately.Only models with publicly available, runnable code are included in the comparison.
- Results: TrajFusionNet+ achieves the highest performance across most metrics under the combined-dataset protocol.It outperforms all other models by a large margin on PIE and remains significantly better than other methods on JAADall while performing comparably to TrajFusionNet.
- Results: Evaluation metrics are substantially lower than in separate train-test evaluation, partly because the speed modality is removed and potentially because of overfitting.The authors note that the performance drop is consistent with observations by Gesnouin et al.
D. Inference Time
TrajFusionNet+ has a relatively expensive model-only runtime but a competitive total runtime when preprocessing is included, with a lighter segmentation backbone reducing latency further.
- Measurement setup: Preprocessing includes computation of model-specific inputs such as pose estimates and segmentation maps, excluding shared pedestrian detection and tracking.This distinction affects comparisons between model-only and model-plus-data-preprocessing runtimes.
- Runtime comparison: 197.24 ms is TrajFusionNet+’s total inference time with preprocessing, lower than every compared approach except the original TrajFusionNet.Measurements use a consumer-grade NVIDIA GeForce RTX 3060 GPU.
- Runtime comparison: 123.16M parameters and higher model-only inference time make TrajFusionNet+ heavier than the other compared approaches before preprocessing.The total-runtime comparison reverses this ordering for most approaches because preprocessing costs are included.
- Lightweight implementation: Replacing SegFormer with DeepLabV3 reduces total inference time by lowering each GAM semantic-map computation from 41.16 ms to 13.95 ms.The lighter implementation provides a runtime-oriented alternative to the proposed SegFormer version.
E. Ablation Study
TrajFusionNet+ integrates SAM, VAM, and GAM, and its component ablations show that graph modeling, temporal visual processing, trajectory information, and vehicle speed contribute to performance. The model also reports improved state-of-the-art performance and generalization across PIE and JAAD.
- Removing GAM causes a small but consistent performance reduction, while replacing TokenGT with a two-layer GCN produces a slight drop.Replacing SegFormer with DeepLabV3 slightly reduces PIE performance but changes little on JAADall.
- Simplifying VAM to one VAN causes a minor PIE drop and a larger JAADall drop, indicating the value of temporal visual modeling.
- Removing trajectory overlays from scene images slightly decreases PIE performance but substantially reduces JAADall performance, supporting trajectory-augmented visual inputs.
- Removing predicted trajectories significantly reduces PIE performance but minimally changes JAADall, while excluding vehicle speed decreases performance across both datasets.
- TrajFusionNet+ achieves improved state-of-the-art performance on PIE and JAAD, with improved generalization under joint training and separate evaluation.