Is the Pedestrian going to Cross? Answering by 2D Pose Estimation

I. INTRODUCTION

• Early knowledge of crossing intent could let a vehicle warn the driver or maneuver more smoothly before a pedestrian enters the road.The authors also connect earlier slowing to passenger comfort, pedestrian confidence, and a reduced chance of injury requiring hospitalization when a crash is not avoidable.Source: §I · p. 1
• The earlier pose-based crossing/not-crossing method used Daimler’s choreographed sequences; JAAD enables evaluation in naturalistic driving conditions.The proposed pipeline combines pedestrian detection, multi-pedestrian tracking, and 2D pose estimation on monocular images.Source: §I · p. 1
• The authors present a JAAD baseline and report that skeleton-based features outperform CNN-based features, with additional time-to-event results.They describe JAAD as the only publicly available dataset then available that was acquired in naturalistic driving and annotated for crossing/not-crossing.Source: §I · p. 1

II. RELATED WORK

• Pedestrian path-prediction models estimate future location, speed, and acceleration, but the cited approaches require dense stereo, optical flow, and ego-motion compensation.The authors’ monocular-image method can be applied in JAAD, which lacks stereo and vehicle data for ego-motion compensation.Source: §II · p. 2
• Head and body orientation have been used to estimate intentions, but how to use those cues is unclear, and one study found head detection unhelpful for crossing classification.The cited orientation approaches include monocular and stereo, images with ego-motion compensation.Source: §II · p. 2
• On JAAD, AlexNet-based appearance features found no performance benefit from focusing specifically on the pedestrian’s head or legs.The authors’ prior pose-based method accumulated skeletons and skeleton-derived features across a sliding time window, and they extend its evaluation to JAAD.Source: §II · p. 2

III. METHOD

• The pedestrian detector is a Faster R-CNN fine-tuned on JAAD training images, while tracking uses an off-the-shelf multiple-object tracker.The tracker associates detections and predictions using CNN-feature cosine distance and Kalman-filtered pedestrian states.Source: §III · p. 2
• Tracking supplies bounding boxes in which the CNN-based pose estimator fits skeletons, because its own pedestrian detection performed less well on JAAD.The pose estimator operates on still monocular images and was not fine-tuned on JAAD because pedestrian body-level annotations would be required.Source: §III · p. 2
• The classifier uses 9 stable leg and shoulder keypoints, normalized by pedestrian height, to derive relative distances and angles.The selected joints are intended to capture leg actions and global body orientation; Fig. 3 illustrates the keypoints and feature construction.Source: §III · p. 2, Figure 3 · p. 3
• The skeleton representation has 396 features per frame, concatenated over T frames; the comparison uses 4096-dimensional-per-frame Faster R-CNN fc6 features.Random Forest performs crossing/not-crossing classification for either feature set.Source: §III · p. 3

A. Dataset

• JAAD contains 346 on-board monocular videos at 30 FPS and 1920 × 1080 resolution, totaling approximately 88,000 frames and 2,624 unique pedestrians.The videos were recorded in North America and Eastern Europe and are mostly 5–10 seconds long.Source: A. Dataset · p. 3
• JAAD includes approximately 390,000 pedestrian bounding boxes with crossing/not-crossing annotations and per-box occlusion tags.Approximately 18% of boxes are partially occluded and 11% heavily occluded.Source: A. Dataset · p. 3
• The dataset also contains contextual annotations such as traffic signs and street width, which this paper does not use.The authors identify these annotations as possible additions for future studies.Source: A. Dataset · p. 3

• The first 250 JAAD videos are used for training, with all remaining videos reserved for testing.Source: B. Evaluation protocol · p. 3
• JAAD crossing labels and specified lateral-motion actions are mapped to C.The included actions are clear-path, moving-fast, moving-slow, slow-down, and speed-up when assigned to a pedestrian with lateral motion direction.Source: B. Evaluation protocol · pp. 3–4
• All other action labels are mapped to NC.Source: B. Evaluation protocol · p. 4

Training:

• The Faster R-CNN was fine-tuned on all training frames for 110,000 iterations, using batches of 256 regions from one image and vertically mirrored inputs to double training samples.It used anchors {8, 16, 32, 64} and a pedestrian-oriented bounding-box aspect ratio of 2.5. The learning rate decreased from 0.001 to 0.0001 after 80,000 iterations.Source: 1) Training · p. 4
• Classifier training excludes pedestrians with bounding boxes under 60 pixels wide or any occlusion, and requires those conditions across more than T frames.Longer tracks provide multiple training samples through a temporal sliding window of size T.Source: 1) Training · p. 4
• For T=14 and T=1, training balances 8,677 C and 8,677 NC samples and selects a Random Forest with 400 trees and maximum depth 15.Each sequence takes its C/NC label from the most recent frame; the 14-frame window is described as roughly below 0.5 seconds.Source: 1) Training · p. 4
• The CNN comparison replaces skeleton features with VGG16 fc6 features inside tracked pedestrian boxes, and also tests combining both feature sets.Each sample uses 396T skeleton-based features or 4096T fc6-based features; the same Random Forest parameters are used for CNN features and the combined features.Source: 1) Training · p. 4

Testing:

• T=1 and T=14 experiments are evaluated on the same tracked pedestrians at the same frames for a fair comparison.Pedestrians lacking required ground-truth information are excluded from quantitative evaluation but remain in the qualitative evaluation.Source: 2) Testing · p. 4
• Tracking starts after detections in 3 consecutive frames and ends after 30 frames without a matched observation.The detector threshold is 5%, non-maximum suppression overlap is 30%, pose estimation uses three scales, and the C/NC threshold is 0.5.Source: 2) Testing · p. 4
• Accuracy is measured on 10,322 testing decisions after randomly balancing the 17,045 positive and 5,161 negative cases.The stated accuracy is (TP + TN)/(P + N); balancing avoids bias toward C.Source: 2) Testing · p. 4
• Table I reports JAAD classification accuracy for skeleton-based and CNN feature types, alongside results reported in [8].Source: Table I · p. 5
• Because JAAD lacks TTE annotations, the authors add them to 9 keep-walking-to-cross and 14 start-walking-to-cross sequences.They plot intention probability and predictability against TTE; predictability ranges from 0 (action cannot be detected) to 1 (it can).Source: 2) Testing · p. 5

Results:

• The reported results are not directly comparable because its training/testing split is unclear and its test excludes heavily occluded pedestrians.The authors’ experiments do not exclude pedestrians due to occlusion.Source: 3) Results · p. 5
• Skeleton-based features outperform CNN fc6 features at fixed T, while T=14 outperforms T=1 and combining feature types does not significantly improve skeleton features.Source: 3) Results · p. 5
• A standing pedestrian’s curbside crossing is predicted with predictability above 0.8 around 8 frames after movement starts, approximately 250 ms in JAAD.Source: 3) Results · p. 5
• C/NC scoring can mislabel crossing elsewhere as an error and roadside walking as correct, while parallel-walking analysis needs more annotations.For the roadside-walking case, the authors suggest location-based reasoning addresses danger rather than predicting the action.Source: 3) Results · p. 5

V. CONCLUSION

• The pipeline combines pedestrian detection, tracking, and pose estimation to address crossing/not-crossing classification in naturalistic JAAD driving conditions.Source: §V · pp. 5–6
• Integrating pedestrian pose-based features over time gives rise to a powerful crossing/not-crossing classifier.Source: §V · p. 6
• The authors report state-of-the-art results for the JAAD dataset.Source: §V · p. 6