Graph-Based Recognition of Simulated Train-Driver States From Facial and Upper-Body Keypoints

• Long shifts, repetitive duties, and nighttime operation raise fatigue concerns, while dead-man switches use periodic pedal checks that burden drivers.The switch can trigger an alarm and may halt the train if the pedal is not pressed within a specified time.Source: §I · p. 1
• Vehicle-based alertness assessment is constrained by intermittent train-control interactions, while physiological sensing can involve wearability, noise, and cost limitations.Source: §I · p. 2
• Prior vision-based fatigue work often emphasizes facial features, while train operators’ wider, periodically varying movements differ from road-driver patterns.Source: §I · p. 2
• The proposed single-stream framework avoids wearables, train telemetry, and verbal confirmation, learning facial and upper-body landmark dynamics in one graph rather than aggregating handcrafted cue signs.The reduced sensing burden may facilitate cabin integration while limiting interference with normal activities.Source: §I · p. 3
• The paper identifies limited railway-specific visual state recognition, especially for distinguishing reduced alertness from acted emergency-like behaviours, and notes illumination variation as a landmark-estimation challenge.Source: §I · p. 3
• The study formulates alert, not alert, and acted emergency-like behaviour as three classes and introduces a controlled dataset with three illumination settings for feature ablation.The ablation compares facial-only, skeletal-only, and combined representations; the emergency class includes behaviours such as pretending to faint or clutching the chest.Source: §I · p. 3

• RGB images are converted to facial and skeletal keypoints, assembled into a whole-body graph, and processed by a custom DGNN to classify three driver states.Source: §II · p. 3, Figure 1 · p. 4
• The preliminary RGB video dataset covers light, dark, and sudden light/dark conditions and supports ablation and keypoint-occlusion analysis.Source: §II · p. 3
• The study asks whether the graph model detects simulated alertness and emergency states and whether whole-body movements improve three-class performance over facial-only or skeletal-only keypoints.Source: §II · pp. 3–4

• The method uses Mediapipe’s first 17 upper-body keypoints and adds 478 Face Mesh landmarks to capture facial contours, eyebrows, eye openness, and mouth movements.The upper skeleton is selected because drivers are usually seated; the chosen detectors are described as stable for RGB images.Source: A. ESTIMATION OF FACIAL AND SKELETAL… · p. 4
• A masking strategy handles keypoints that were not consistently detected during extraction from RGB video.Source: A. ESTIMATION OF FACIAL AND SKELETAL… · p. 4
• Facial and upper-body 2D coordinates become graph nodes, bones define edges, and coordinates are normalized relative to the nose across the three input configurations.Source: A. ESTIMATION OF FACIAL AND SKELETAL… · p. 4

1) Network architecture

• The customized DGNN uses two graph-temporal convolution blocks, pairing directed graph relational learning with bidirectional temporal convolutions over node and edge features.The architecture processes incoming and outgoing graph connections, then applies temporal convolution along the time dimension.Source: 1) Network architecture · p. 4
• The compact model adds dropout, uses LeakyReLU and label smoothing, then pools learned node and edge features for fully connected output predictions.The authors describe these changes as adaptations for the relatively small dataset and noisy keypoint detections.Source: 1) Network architecture · p. 5

2) Network hyperparameters

• Training used Adam, a learning rate of 0.001, 50 epochs, batch size 512, label smoothing 0.1, and a 30-frame lookback, tuned using validation performance.Participants were seated at a desk and followed on-screen instructions for lever, button, and side-panel tasks while a front-facing RGB camera monitored them.Source: 2) Network hyperparameters · p. 5, Figure 3 · p. 5

1) Ablation study

• Three DGNN variants use whole-body, facial-only, or skeletal-only keypoints to isolate the contributions of facial and skeletal information.Whole-body combines facial and skeletal keypoints; the study examines whether integration enhances performance.Source: 1) Ablation study · p. 5

2) Occlusion Sensitivity Analysis

• Occlusion Sensitivity Analysis hides one or more keypoints and assesses changes in predictions or performance to identify important inputs.A substantial degradation after occlusion indicates that the keypoint is particularly important to the network’s decisions.Source: 2) Occlusion Sensitivity Analysis · p. 5

A. SETUP

• Participants sat before a computer monitor and a front-facing Realsense D435i camera during data collection.The camera captured RGB video at 30 fps and 640×480 resolution.Source: A. SETUP · pp. 5–6

B. DATASET

• The dataset included 25 participants, divided into training, validation, and test sets of 15, 5, and 5 subjects.Participants were 14 males and 11 females aged 23–36; 12 wore glasses and 13 did not.Source: B. DATASET · p. 6
• Each participant recorded nine 2.5-minute videos covering alert, not-alert, and acted emergency-like states under light, dark, and sudden light/dark conditions.The sudden light/dark condition switched the room lights every 15 seconds; emergency behaviours were acted proxies, not clinically validated medical events.Source: B. DATASET · p. 6
• Alert participants followed train-driving prompts, not-alert participants kept their eyes closed and pretended to sleep, and emergency participants acted behaviours such as fainting or clutching their chest.Some participants covered parts of their faces during not-alert and emergency recordings, introducing self-occlusions.Source: B. DATASET · p. 6

1) Accuracy

• Accuracy is the proportion of correct predictions among all samples and measures agreement with ground-truth classes.The section notes that accuracy ranges from 0 to 100% and is particularly useful for balanced classification tasks.Source: 1) Accuracy · p. 6

2) Confusion Matrices

• Confusion matrices compare predictions with actual labels for whole-body keypoints across lighting conditions and three- and two-class tasks.The analysis also includes facial-only and skeletal-only scenarios.Source: 2) Confusion Matrices · p. 6

IV. RESULTS

• 99% test accuracy was achieved for alert/not-alert classification; adding emergency reduced accuracy, while whole-body outperformed facial-only and skeletal-only models.The reported metrics were computed on five test subjects.Source: §IV · p. 6
• Performance was highest in full light, followed by sudden light/dark conditions across all scenarios.Source: §IV · p. 6
• The three-class whole-body confusion-matrix analysis found that emergency states were sometimes misclassified as not alert.Source: §IV · p. 6, Figure 4 · p. 7
• The keypoint-importance plot represents each keypoint as a bar whose height indicates its significance in distinguishing the three driver states.Source: §IV · p. 7

V. DISCUSSION

• 78% overall three-class accuracy was achieved with whole-body keypoints, while skeletal-only or facial-only features were insufficient for appropriate three-class classification, particularly under varying lighting.The study describes whole-body analysis as highly effective for binary driver-alertness classification.Source: §V · p. 7
• Globally, 90.7% of alert and 87.6% of not-alert samples were correctly classified, versus 54.1% of emergency samples, with 34.1% of emergency samples misclassified as not alert.The authors link the emergency/not-alert confusion to similar postures in the two acted conditions.Source: §V · p. 7
• The framework is preliminary evidence from acted scenarios, not a clinically validated detector of real fatigue or medical emergencies; real-world validation and a larger dataset remain future work.Planned dataset expansion includes occlusions and real-world artifacts such as camera vibration.Source: §V · pp. 7, 9
• In light, alert and not-alert recognition reached 94.1% and 93.4%, respectively, while emergency recognition reached 56.2%; in dark, emergency recognition fell to 50.6%.The authors attribute weaker dark-condition performance to RGB-camera and keypoint-estimation limits in low light.Source: §V · p. 8
• Under sudden light/dark changes, 91.8% of alert samples were correctly classified, but 17.1% of not-alert samples and 32.3% of emergency samples were confused across those classes.The authors suggest abrupt illumination changes may add variability to extracted features and hinder that distinction.Source: §V · p. 8
• Occlusion analysis ranked skeletal keypoints, especially the right wrist and both shoulders, among the most influential features, alongside mouth and left-wrist keypoints.Figure 5 presents the overall keypoint ranking; the authors interpret the result as posture cues complemented by facial information.Source: §V · p. 8, Figure 5 · p. 9
• The analysis ranks individual nodes but does not compare alternative graph topologies, root definitions, or connectivity patterns; topology comparison is identified as future work.The authors note that node importance does not isolate the contribution of edge configurations.Source: §V · pp. 8–9
• The study does not make a direct quantitative comparison with prior approaches because datasets, acquisition protocols, class definitions, and evaluation procedures differ.The authors also note that prior studies often use binary classification and more stable illumination settings.Source: §V · p. 9

VI. CONCLUSIONS

• The system uses a customised Directed-Graph Neural Network and whole-body movement to classify alert, not-alert, and simulated emergency conditions.Source: §VI · p. 9
• 81% test accuracy for three-class classification and 99% accuracy for two-class classification were achieved by the whole-body graph in light condition, which had the highest performance among tested configurations.Source: §VI · p. 9
• The study introduces a controlled RGB dataset of alert, not-alert, and acted emergency-like behaviours; planned work includes dataset expansion and thermal or near-infrared sensing for low-light robustness.Source: §VI · p. 9