Intention Recognition of Pedestrians and Cyclists by 2D Pose Estimation

• Pedestrian crossing prediction before road entry could allow earlier warnings or smoother vehicle maneuvers, with potential to reduce injury severity when crashes cannot be avoided.Source: §I · p. 1
• Cyclist intention recognition targets future turns and stops indicated by arm signals under traffic rules.Source: §I · p. 1
• 2D pose estimation from monocular images is proposed as a shared basis for recognizing pedestrian and cyclist intentions.Source: §I · p. 1
• For JAAD crossing classification, pose-based features reportedly outperform CNN-based features and avoid requiring stereo, optical flow, or ego-motion compensation.Source: §I · p. 1
• The paper extends JAAD pedestrian work by analyzing noisy skeletons and relevant features, and introduces CASR with 219 annotated arm-signal actions plus 10 additional YouTube videos.Source: §I · p. 2

• Pedestrian path-prediction methods may require dense stereo, optical flow, and ego-motion compensation, limiting their applicability to monocular JAAD imagery.Source: §II · p. 2
• Earlier pose-based work used temporal skeleton features for pedestrian crossing classification, and the present study extends it to JAAD and compares skeleton features with CNN appearance features.Source: §II · p. 2
• Cyclist arm-signal recognition lacks a public dataset designed for the task; CASR supplies 219 annotated actions and 10 additional YouTube actions for generalization assessment.Source: §II · p. 2
• Prior cyclist signal work classified a single left-arm-up/down cue using stereo, whereas this paper addresses left turns, two right-turn signals, and stopping without assuming stereo.Source: §II · p. 2
• Other skeleton-based action-recognition studies do not report JAAD results or address cyclist arm signals, and some use sequences that are not publicly available.Source: §II · p. 2

• The pipeline detects and tracks pedestrians or cyclists, fits skeletons within tracked bounding boxes, then classifies intentions from skeleton-based features.Source: §III · p. 3
• Detection uses an object detector that takes a single RGB image and returns bounding boxes for pedestrians or cyclists.Source: §III · p. 3
• Tracking uses Kalman-filtered bounding-box state and cosine-distance matching of CNN features to associate detections with tracks.Source: §III · p. 3
• Pose estimation uses a CNN method on monocular still images, applied inside tracked bounding boxes rather than as the initial detector.Source: §III · p. 3
• Pedestrian features use normalized distances and angles from nine leg and shoulder keypoints; cyclist features add elbow and wrist keypoints, yielding 396 and 1170 features per frame, respectively.Features are concatenated across T frames and classified with a Random Forest.Source: §III · p. 3

A. Dataset

• JAAD contains 346 on-board monocular videos, approximately 88,000 frames, 2,624 pedestrians, and pedestrian tracks with occlusion labels.Source: A. Dataset · pp. 3–4
• Pedestrian features use nine keypoints to produce 396 features per frame, while cyclist features use 13 keypoints to produce 1170.Source: Figure 4 · p. 4

B. Evaluation protocol

• The study uses the first 250 JAAD videos for training and the remainder for testing, and relabels crossing and selected lateral-motion labels as crossing.Other JAAD labels are relabeled as not crossing.Source: B. Evaluation protocol · p. 4

1) Training:

• JAAD training uses 8,677 C and 8,677 NC samples, balancing the classes for both T=14 sequences and T=1 frames.For T=14, only 8,677 of the 36,253 available C sequences are sampled.Source: 1) Training · p. 4
• The RF is selected by five-fold cross-validation, with 400 trees and maximum depth 15; other tested configurations had similar accuracy.The search varied tree counts from 100 to 500 and maximum depths from 7 to 30.Source: 1) Training · p. 4
• CNN comparison features come from VGG16 fc6 activations inside the tracked-pedestrian boxes, and skeleton and CNN features are also combined.Skeleton features comprise 396T values per sample, versus 4096T fc6 values.Source: 1) Training · pp. 4–5

2) Testing:

• Testing compares T=1 and T=14 on the same tracked pedestrians and frames, excluding detected pedestrians lacking required ground-truth annotations from quantitative evaluation.Detection and tracking thresholds and the C/NC probability threshold are specified for testing.Source: 2) Testing · p. 5
• The test set is balanced to 5,161 C and 5,161 NC decisions, yielding 10,322 accuracy evaluations.The original protocol contained 17,045 positives and 5,161 negatives.Source: 2) Testing · p. 5
• TTE annotations cover 9 keep-walking-to-cross and 14 start-walking-to-cross sequences; positive TTE is before the event and negative TTE after it.The event is defined separately for each action: reaching the curbside or beginning to move a leg forward.Source: 2) Testing · p. 5

C. Results

• At fixed T, skeleton-based features outperform CNN fc6 features, combining them adds no significant accuracy, and T=14 outperforms T=1.The reported JAAD accuracy results are in Table I.Source: C. Results · p. 5, Table I · p. 5
• Predictability exceeds 0.8 for a standing pedestrian crossing the curb about 8 frames after starting to move, roughly 250 ms in JAAD.Fig. 6 plots action probability and predictability against time-to-event for two crossing actions at T=14.Source: C. Results · p. 5
• C/NC evaluation can mislabel crossing a road outside the ego-vehicle path and walking along the road in front of the car.The authors identify these cases as future-work issues; more annotations are needed to quantify classification of pedestrians walking parallel to vehicle motion.Source: C. Results · pp. 5–6
• With 20% and 30% keypoint noise, accuracy decreases, with greater robustness at T=14; top-ranked features are mainly three-keypoint angles capturing global pose.For T=14, only one of the top-25 features comes from after frame 9, about 300 ms.Source: C. Results · p. 6, Table II · p. 6, Table III · p. 6

A. Dataset

• CASR contains 40,218 frames and 219 annotated cyclist arm-signal actions, while the YouTube set adds 1,626 frames and 10 actions.The videos are approximately 10 seconds long and include one cyclist.Source: A. Dataset · p. 6
• CASR was recorded with a forward-facing windshield GoPro at 30 fps and 1920 × 1080 resolution.Four cyclists rode on a university campus and were instructed to use arm signals when required.Source: A. Dataset · p. 6
• Arm-signal labels are vehicle-centric: left and right indicate the direction the cyclist will travel as seen in the processed image.For cyclists and vehicles moving in opposite directions, the annotation direction is opposite the cyclist's own indicated turn.Source: A. Dataset · p. 6

B. Evaluation protocol

• Twelve train-validation-test runs vary the roles of four cyclists, and each trained classifier is also tested on annotated YouTube videos.Two cyclists supply training videos; the other two supply validation and testing videos, respectively.Source: B. Evaluation protocol · p. 6
• The results report worst, best, and average metrics across runs, including YouTube generalization; separate results cover clean and noisy keypoints.Table VI reports average Accuracy and F1 for CASR and YouTube under noise-free, 20%, and 30% keypoint conditions.Source: Table V · p. 7, Table VI · p. 7
• Cyclist classification is evaluated per frame with Accuracy and F1, using T=1 and T=14 temporal windows and RF hyperparameter search.The search varies tree count and maximum depth.Source: B. Evaluation protocol · p. 7
• Because the cyclist datasets contain a single cyclist and no pedestrians, detection suffices without tracking; Mask R-CNN replaces Faster R-CNN because the latter often omitted cyclists' arms.Missing detections are replaced with noisy ground-truth boxes, and the detector is not fine-tuned to CASR.Source: B. Evaluation protocol · pp. 7–8

C. Results

• Thirty-percent keypoint noise appreciably reduces T=1 accuracy, while T=14 avoids much of that drop; YouTube testing is more challenging but remains accurate.Temporal aggregation does not significantly boost clean-data performance, though it can stabilize classification output.Source: C. Results · p. 8
• Cyclist errors arise when arms fall outside detection boxes, skeleton fitting is poor, actions are initially ambiguous, or a straight arm makes stop and turn signals look alike.Some onset confusions resolve in later frames; the authors suggest enlarging the detection box at testing time to address arms outside it.Source: C. Results · p. 8
• The authors found no particular class-confusion pattern in CASR, either without keypoint noise or with 20%/30% noise.Source: C. Results · p. 8
• For T=1, relevant features are mainly angles linking arm keypoints with the neck, shoulders, waist, or legs, plus distances between these groups.For T=14, current-frame angles remain prominent, while many relevant distances occur in the middle of the window, where the cyclist has indicated the sign for roughly 200–300 ms; the top 25 features are about 2% for T=1 and 0.15% for T=14.Source: C. Results · p. 8, Table VII · p. 8, Table VIII · p. 8

VI. CONCLUSION

• Across pedestrian crossing recognition on JAAD and cyclist arm-signal recognition on the authors’ CASR dataset, fitted 2D skeleton features support high-accuracy recognition over a roughly 500 ms window.The same procedure is applied to both tasks, and the authors present quantitative and qualitative results.Source: §VI · p. 10