Source-linked AI summary
Using Eye-tracking Data to Predict Situation Awareness in Real Time during Takeover Transitions in Conditionally Automated Driving
Feng Zhou, X. Jessie Yang, Joost de Winter
TL;DR
Situation awareness is important during transitions from automated to manual driving, yet few studies have predicted it in real time. This paper combines LightGBM and SHAP to model situation awareness from eye-tracking and related data, achieving reported performance of RMSE = 0.121, MAE = 0.096, and correlation coefficient = 0.719 using eye-tracking predictors.
Problem
The paper addresses limited real-time situation-awareness prediction during takeover transitions in conditionally automated driving.
Method
The method combines LightGBM regression with SHAP-based feature selection and interpretation of predictor effects on situation awareness.
Results
Using eye-tracking predictors, the model achieved RMSE = 0.121, MAE = 0.096, and correlation coefficient = 0.719 against normalized situation-awareness ground truth.
Takeaways & Limitations
The identified predictor effects provide domain knowledge for building real-time situation-awareness prediction models from non-intrusive eye-tracking measures.
Abstract
from arXiv · showhide
Situation awareness (SA) is critical to improving takeover performance during the transition period from automated driving to manual driving. Although many studies measured SA during or after the driving task, few studies have attempted to predict SA in real time in automated driving. In this work, we propose to predict SA during the takeover transition period in conditionally automated driving using eye-tracking and self-reported data. First, a tree ensemble machine learning model, named LightGBM (Light Gradient Boosting Machine), was used to predict SA. Second, in order to understand what factors influenced SA and how, SHAP (SHapley Additive exPlanations) values of individual predictor variables in the LightGBM model were calculated. These SHAP values explained the prediction model by identifying the most important factors and their effects on SA, which further improved the model performance of LightGBM through feature selection. We standardized SA between 0 and 1 by aggregating three performance measures (i.e., placement, distance, and speed estimation of vehicles with regard to the ego-vehicle) of SA in recreating simulated driving scenarios, after 33 participants viewed 32 videos with six lengths between 1 and 20 s. Using only eye-tracking data, our proposed model outperformed other selected machine learning models, having a root-mean-squared error (RMSE) of 0.121, a mean absolute error (MAE) of 0.096, and a 0.719 correlation coefficient between the predicted SA and the ground truth. The code is available at https://github.com/refengchou/Situation-awareness-prediction. Our proposed model provided important implications on how to monitor and predict SA in real time in automated driving using eye-tracking data.
I. INTRODUCTION
Conditionally automated driving can leave drivers out of the control loop, reducing situation awareness when takeover is required. This study proposes an explainable eye-tracking model to predict and monitor situation awareness in real time.
- Motivation: Prolonged non-driving-related tasks can reduce drivers’ situation awareness before automated-vehicle takeovers.Reduced awareness may impair successful control transitions when the vehicle reaches its system limit.
- Research gap: Few studies have attempted to monitor or predict situation awareness in real time during automated driving.Existing approaches commonly use SAGAT or SART, which present operational challenges during time-sensitive takeovers.
- Prior evidence: Eye-tracking measures provide dynamic indicators related to visual attention during takeover transitions.Prior work linked eyes-on-road time, attention shifts, and visual sampling to situation awareness or takeover-related behavior.
- Approach: The proposed explainable model combines LightGBM for situation-awareness prediction with SHAP for feature importance and effect interpretation.SHAP-based feature selection identifies important predictor variables and supports model improvement.
- Contributions: The study identifies important eye-tracking measures and demonstrates the potential to monitor situation awareness in real time.The stated contributions target conditionally automated driving and potentially other dynamic environments.
II. RELATED WORK
Situation awareness is important for safe takeover decisions, but conventional subjective and objective measures can interfere with dynamic driving tasks. Eye-tracking offers a nonintrusive behavioral route for assessing situation awareness through visual attention and scanning.
- Importance of SA: Situation awareness supports drivers’ ability to negotiate scenarios safely when conditional automation reaches its operational limit.Drivers may need to maintain or promptly resume awareness within a specific takeover time budget.
- Subjective measures: Subjective measures such as SART are easy to administer but quantify situation awareness through reported demand, supply, and understanding.SART expresses situation awareness as understanding − (demand − supply).
- Objective measures: SAGAT measures task knowledge with freeze-probes, but its interference with ongoing tasks limits its suitability for takeover studies.Its freeze-probe procedure creates operational challenges in time-sensitive and naturalistic driving contexts.
- Process measures: Information-processing indicators, including physiological responses, communication patterns, and visual patterns, have also been used to assess situation awareness.Prior studies examined communication, music, and verbal interaction during partially automated driving.
- Eye-tracking measures: Eye-tracking studies associate visual sampling and eyes-on-road behavior with task performance, situation awareness, or takeover readiness.Reported findings include better task-performance alignment for visual sampling and approximately 60% accuracy for predicting takeover readiness from eyes-on-road gazes.
III. DATASET
The study used simulated conditional-automation videos and reconstructed-scene performance to create a normalized situation-awareness target. LightGBM then modeled this continuous target from predictor variables, including eye-tracking measures.
- Dataset: The dataset included 32 participants who viewed 33 simulated driving videos lasting 1, 3, 6, 9, 12, or 20 seconds.Videos depicted a three-lane scenario with five or six vehicles.
- Dataset: Sixteen scenarios required takeover responses because a vehicle decelerated at 5 m/s2 at the video start.Participants selected the maneuver needed to avoid collision after viewing each video.
- SA ground truth: Situation-awareness ground truth combined vehicle-placement, distance, and relative-speed reconstruction errors.The three inversely normalized error scores received equal weights to produce a continuous score from 0 to 1.
- Predictors: Eye movements were recorded with an EyeLink 1000 Plus at a 2000 Hz sampling rate, alongside other predictor variables.The study used algorithms from prior work to derive eye-tracking measures, with 28 total predictors or 16 eye-tracking-related predictors.
- LightGBM: LightGBM predicted continuous situation awareness as a regression problem by adding decision trees sequentially.Its objective combined mean squared error and mean absolute error, while regularization controlled model complexity.
- LightGBM: The LightGBM configuration used GOSS, feature bundling, and fixed hyperparameters without fine-tuning during training and testing.The listed setup included 100 leaves, a 0.05 learning rate, and up to 5000 boosting rounds.
B. SHAP
SHAP represents each model prediction as a baseline plus additive contributions from input variables, providing mathematically grounded explanations. In this study, SHAP also supported global importance ranking, feature selection, and interpretation of individual predictions, with an efficient tree-ensemble algorithm used for LightGBM.
- SHAP definition: SHAP explains the original model function as an additive model of input-variable contributions around a bias term.The explanation satisfies local accuracy, missingness, and consistency properties.
- SHAP definition: SHAP values quantify each predictor variable’s fair contribution to a prediction and provide a unique additive feature-importance measure.The values are derived from Shapley values in coalitional game theory.
- Uses in this study: SHAP identified global predictor importance, explained main effects on SA, and attributed individual predictions to predictor-value sets.Global importance was also used to select features for the LightGBM model.
- Computational implementation: For tree ensembles such as LightGBM, an efficient SHAP algorithm reduces computation to O(TLD^2).Here, T is the number of trees, L is the maximum number of leaves in a tree, and D = logL.
A. Prediction Results
LightGBM was evaluated against selected regression models using 10-fold cross-validation and RMSE, MAE, and correlation coefficient metrics. It performed best with all predictors and with eye-tracking predictors after SHAP-based selection of the top variables.
- Evaluation: 10-fold cross-validation compared LightGBM with other regression models using RMSE, MAE, and correlation coefficient.The correlation coefficient measured agreement between predicted SA and ground-truth SA.
- All predictors: LightGBM achieved the best performance across RMSE, MAE, and correlation coefficient when all 28 predictor variables were included.LightGBM (all) used all predictors, whereas LightGBM (best) used the top 14 selected by SHAP.
- Eye-tracking predictors: LightGBM achieved the best performance across RMSE, MAE, and correlation coefficient using only eye-tracking-related predictors.LightGBM (all) used all 16 eye-tracking predictors, while LightGBM (best) used the top 9 selected by SHAP.
- Sequential feature selection: Performance stabilized after the first several variables were added, and the best models used the top 14 all-predictor variables or top 9 eye-tracking variables.Variables were added sequentially from the most important to the least important according to their contributions to predicting SA.
1) Feature Importance:
SHAP produced global importance rankings for LightGBM models using either all predictors or only eye-tracking-related predictors. The rankings quantify importance through summed absolute SHAP values and display each variable’s contribution distribution.
- Importance ranking: SHAP ranked predictor variables by global impact, calculated as the sum of their absolute SHAP values across instances.The ranking was produced for models containing all predictors and for models containing only eye-tracking-related predictors.
- Plot interpretation: Each plotted dot represents one SHAP value for a specific predictor variable and sample.The horizontal violin shapes show the distributions of samples for each variable.
- All predictors: In the all-predictor model, videoLength ranked above correctDecision as the most important predictor variables shown in the figure.The vertical axis orders variables from higher to lower global importance.
2) Main effects of predictor variables on SA:
SHAP main-effect plots identified how the most important predictors, including eye-tracking measures, contributed to predicted situation awareness and how their values were distributed. The eye-tracking model reproduced several patterns from the broader predictor set.
- SHAP visualization: SHAP main-effect plots used box plots for predictor effects and histograms for sample distributions, with continuous values grouped into bins.The left y-axis represented SHAP contributions, while the right y-axis represented sample counts.
- Top predictors: videoLength was the most important predictor in the broader model (r = .798, p = .000), with 1-s videos reducing SA by 0.02 to 0.09.Videos longer than 3 s showed little difference in their influence on SA.
- Top predictors: Correct decisions, shorter decision times, more backMirror fixations, and greater driving frequency were associated with better SA, whereas difficulty and road fixations were negatively associated.The reported associations were correctDecision (r = .292), decisionTime (r = −.763), backMirror (r = .745), drivingFrequency (r = −.678), difficulty (r = −.746), and road (r = −.771).
- Eye-tracking predictors: Among eye-tracking predictors, backMirror, pupilStd, fMean, fStd, and sAmpStd were positively associated with SA, while pupilMean, road, and sAmpMean showed negative associations.The corresponding reported correlations included backMirror (r = .753), pupilStd (r = .636), fMean (r = .410), fStd (r = .300), sAmpStd (r = .518), pupilMean (r = −.232), road (r = −.740), and sAmpMean (r = −.477).
- Model explanation: SHAP captured similar patterns for shared predictors in the broader and eye-tracking models, indicating consistency in explaining the LightGBM models.The eye-tracking model used the selected top nine eye-tracking-related predictors, while the broader model used the selected top 14 predictors.
3) Explaining Individual Instances:
Individual SHAP explanations showed how feature values raised or lowered each LightGBM prediction relative to a model baseline. The study also reported that the broader predictor model performed better than the eye-tracking-only model, while the experiment used a low-fidelity video-based setup.
- Individual explanations: For an individual instance, the 14-predictor LightGBM baseline was E[f(x)] = 0.708, producing predicted SA of 0.695 versus ground-truth SA of 0.747.Road, difficulty, decisionTime, pupilMean, and five other features decreased the prediction, while backMirror, pupilChange, videoLength, correctDecision, and fMean increased it.
- Limitation: The experiment’s ecological validity was limited because data were collected in a low-fidelity setup involving videos viewed on a computer monitor.The authors described this as a limitation of the current study.
- Model comparison: The model using the top 14 predictors performed better than the model using only the top nine eye-tracking-related predictors.The authors attributed this difference to the inclusion of variables directly related to SA, including videoLength, correctDecision, decisionTime, and difficulty.
- Runtime: Predicting one sample took less than 1 ms after 10-fold cross-validation of the nine-eye-tracking-measure model.The reported cross-validation computation took about 0.853 s on the stated MacBook Pro system.
- Future direction: The authors suggested that additional behavioral and physiological data could improve model performance and support non-intrusive SA monitoring during conditionally automated driving.Examples included reaction time, eyes-on-road time, and EEG; the authors linked possible interventions to improving takeover performance.
B. Explaining SA Prediction
SHAP explained LightGBM predictions by identifying important factors and their effects on SA. Eye-tracking measures, especially fixations, contributed substantially, while associations involving fixation duration, pupil diameter, and saccades required cautious interpretation.
- Eye-tracking predictors: Fixations were the most important eye-tracking measure type, followed by pupil diameter and saccades.Fixations on the rear-view mirror and road were selected, and more rear-view-mirror fixations were associated with better SA.
- Eye-tracking predictors: Larger fixation-duration measures were generally associated with better SA, but the associations were moderate and sensitive to large values.Correlation coefficients were smaller than 0.5, and exponential-like distributions reduced sample sizes as values increased.
- Eye-tracking predictors: PupilMean was more important than pupilStd, while pupil diameter was influenced by working-memory capacity, individual differences, and lighting fluctuations.The authors noted that more research is needed to understand pupil diameter’s role in SA.
- Eye-tracking predictors: Saccade amplitudes were negatively correlated with SA, consistent with shorter scan paths making vehicle information easier to identify.The paper interprets saccades as indicators of attention shifts during visual search and scanning.
- Individual explanations: SHAP identified important predictor variables and showed how their values increased or decreased individual SA predictions.The most important factors differed across individual instances, including road, difficulty, backMirror, and fixation standard deviation.
- Implications: SHAP-derived domain knowledge could support human-machine systems and real-time SA prediction using non-intrusive eye-tracking measures.The paper frames this knowledge as potentially useful for optimizing joint performance in conditionally automated driving.
VII. CONCLUSION AND FUTURE WORK
The study found that SHAP-enhanced LightGBM predicted takeover-transition SA effectively using eye-tracking measures alone. Future validation should address the low-fidelity setup and participant-sample limitations while testing additional measures and more diverse samples.
- Conclusion: RMSE = 0.121, MAE = 0.096, and correlation coefficient = 0.719 were achieved using only eye-tracking predictors.LightGBM had the best performance among the selected machine learning models after SHAP-based feature selection.
- Future work: The data were collected in a low-fidelity setup, so future studies should replicate the experiment in high-fidelity simulators or naturalistic driving.The authors explicitly identify replication in more realistic settings as future work.
- Future work: Future studies could add more eye-tracking, physiological, and behavioral measures to investigate whether prediction performance improves.Suggested eye-tracking additions include blink rate and nearest-neighbor index.
- Future work: The participant sample mainly comprised engineering students and had an unbalanced gender ratio, motivating broader age and gender-balanced recruitment.The authors recommend recruiting more participants across a large age range with a balanced gender ratio.