PaperlayerView exact revision on arXiv

Source-linked AI summary

EEG-Driven Decoding Framework for Passenger Hazard Perception in Highly Automated Vehicles

Yingkai Yang, Ashton Yu Xuan Tan, Bowen Li, Xiaorong Gao, Sifa Zheng, Jianqiang Wang, Xinyu Gu, Yang Zhao, Yuxin Zhang, Sharon X. Huang, Tania Stathaki, Jun Li, Hong Wang

arXiv:2609.07128v1cs.AIcs.LGeess.SP

TL;DR

AV hazard assessment remains challenged by unforeseen scenarios, while passenger-centered EEG evidence for sequential risk anticipation and danger recognition is limited. The paper proposes PCM, RSL, and a joint 3D-CRNN-based PEDS, achieving strong RP and DI performance with cross-session and cross-subject evaluation. The findings support passenger EEG as an auxiliary signal for AV hazard-perception research, within the studied interaction-based scenarios and controlled passenger-observation setting.

  • Problem

    Existing studies largely focus on drivers, immediate hazard detection, or isolated classification, leaving passenger-centered pre-onset risk prediction and sequential danger recognition underexplored.

  • Method

    The framework combines PCM, RSL, and PEDS, whose joint 3D-CRNN decodes pre-onset RP and near-onset DI from passenger EEG.

  • Results

    3D-CRNN achieved 0.953 BA for RP, 0.850 BA for single-subject DI, 0.770 BA in cross-session DI, and 0.774 BA on seen subjects in cross-subject evaluation.

  • Takeaways & Limitations

    Passenger EEG contains both anticipatory and hazard-related information that can support AV safety research beyond immediate hazard recognition.

  • Takeaways & Limitations

    The framework was evaluated in a controlled passenger-observation setting and uses an interaction-based taxonomy that does not cover all AV hazards.

Abstract

from arXiv · show

Reliable risk assessment remains a central challenge for Autonomous Vehicles (AVs). Despite advances in automation, passenger cognition provides a non-intrusive auxiliary signal that improves both objective and perceived safety without requiring active human intervention. We introduce an Electroencephalogram (EEG)-based Brain-Computer Interface (BCI) that decodes passenger neural responses for both Risk Prediction (RP) and Danger Identification (DI), explicitly modeling humans as passengers to match real-world AV use. To achieve this, we propose the Passenger Cognitive Model (PCM), Risk-aware Sequential Labeling (RSL), and the Passenger EEG Decoding Strategy (PEDS), which integrates a 3D Convolutional Recurrent Neural Network (3D-CRNN) model for joint EEG decoding. Experimental results show that 3D-CRNN achieves a Balanced Accuracy (BA) of $95.3\% \pm 2.7\%$ in RP and improves single-subject DI from $80.9\% \pm 3.9\%$ to $85.0\% \pm 3.2\%$ with RSL. Event-wise analyses further show that 3D-CRNN consistently outperforms other models across different event types in RP and DI. In generalization experiments, 3D-CRNN achieves $77.0\% \pm 5.3\%$ BA in cross-session DI and $77.4\% \pm 1.1\%$ BA on seen subjects in cross-subject evaluation, while maintaining a $64.9\% \pm 8.5\%$ BA on unseen subjects, demonstrating promising generalizability and transferability across both intra-subject and inter-subject variability. These findings establish an Electroencephalogram (EEG) decoding framework for AV passenger hazard perception and suggest that passenger cognitive signals can provide auxiliary supervision for future AV decision-making and Safety of the Intended Functionality (SOTIF) support.

1 Introduction

The paper addresses gaps in passenger-centered AV hazard perception by decoding EEG for both anticipatory risk prediction and instantiated danger identification. It introduces PCM, RSL, and PEDS with a 3D-CRNN, and reports strong decoding and generalization results.

  • SOTIF accidents expose limitations of data-driven AV algorithms in unforeseen, boundaryless scenarios with limited training-data coverage.
  • Existing EEG research largely studies drivers, immediate hazard detection, or isolated classification rather than passenger-centered sequential risk anticipation and danger recognition.
  • The contribution includes a 45-hour dataset from 15 participants covering 14 traffic scenarios.
  • The framework introduces PCM to link traffic stages with passenger neural responses and describe transitions from calm monitoring to risk anticipation and danger recognition.
  • RSL encodes Safe, Low-Risk, and High-Risk progression to couple RP and DI, while PEDS uses an electrode-informed 3D-CRNN for pre-onset and post-onset decoding.
  • 95.3% BA was achieved for RP, while 85.0% BA was achieved for single-subject DI with 3D-CRNN and RSL.

2.1 Problem Formulation

The problem formulation separates pre-onset potential hazard from onset-related instantiated danger, then uses their sequential relationship to define RP and DI. RSL and PEDS jointly map these labels to task-specific EEG decoding outputs.

  • 2.1 Problem Formulation: Passenger EEG is used to decode hazard anticipation before an event and hazard recognition after the event is instantiated.
  • 2.1.1 Operational Event Onset: The event onset t0 is the earliest time when a target may initiate a hazardous maneuver requiring a prompt ego-vehicle response.
  • 2.1.1 Operational Event Onset: The experimental workflow covers simulator settings, EEG preparation and analysis, risk-aware labeling, and construction of PCM and PEDS models.
  • 2.1.2 Input Data: EEG inputs are multichannel segments aligned to event onset, with C channels and T time samples in each analysis window.
  • 2.1.3 Hazardousness Level Definition: Hazardousness uses Risky versus Not Risky for potential pre-onset hazard and Dangerous versus Not Dangerous for instantiated onset or post-onset hazard.
  • 2.1.4 Risk-aware Sequential Labeling: RSL converts the sequential two-axis state into Safe, Low-Risk, and High-Risk levels, excluding Not Risky but Dangerous because Dangerous implies Risky.
  • 2.1.5 Decoding Tasks: RP predicts whether an interaction may evolve into risk during the pre-event phase, whereas DI identifies immediate hazard around onset or immediately afterward.
  • 2.1.6 Objective: PEDS uses a joint 3D-CRNN to produce RP and DI predictions from task-specific EEG windows, integrating RSL, PCM, and model construction.

2.2 Data Collection Setup

The study collected passenger EEG in a controlled driving simulator across diverse traffic events, using event-centered segmentation and task-specific windows for Risk Prediction and Danger Identification.

  • 15 participants were recruited, with data from 11 participants retained for analysis.
  • Participants viewed first-person traffic clips in a simulator while EEG was recorded, covering pedestrian crossings, vehicle cut-ins, cut-outs, and emergency braking.
  • The experiment used four sessions on different days, with 16 unique clips and four clips per session.
  • Risk Prediction used the pre-onset EEG window [−2.0, −0.5] s, whereas Danger Identification used the onset/immediate post-onset window [−0.2, +1.0] s.
  • Each clip contained 25 traffic scenarios drawn from 14 event types, yielding approximately 400 event-related EEG segments and 4 h of EEG per participant.
  • EEG was acquired with 64 electrodes at 1000 Hz and filtered from 0.5 Hz to 50 Hz before mastoid re-referencing.

2.3 Passenger Cognitive Model (PCM)

The Passenger Cognitive Model links traffic conditions to passenger mental states and neural responses, supporting two transitions: Calm to Nervous and observation to decision-making. Comparisons across scenarios reveal timing- and event-dependent EEG signatures that motivate Risk Prediction and Danger Identification.

  • Scenario comparisons: Event 8 produced a 200–300 ms occipital and parietal positive fluctuation consistent with P300, whereas Event 6 showed minor frontal activity without a significant P300.
  • Temporal responses: Event 2 showed a P300 near 300 ms, compared with approximately 400 ms for Event 1, reflecting different detection timing across pedestrian-crossing directions.
  • Limitations and variability: Leading-vehicle events showed delayed or weaker P300 responses, with averaging affected by individual differences in risk perception.
  • The PCM organizes passenger cognition across mental states, time steps, topographic maps, active regions, traffic scenarios, cognition, and classification tasks.
  • Calm state: Safe periods with time-to-event greater than 5 s correspond to a Calm state with no specific active brain region.
  • State transition: Approaching other traffic participants marks a transition toward heightened nervousness as the situation becomes riskier.
  • Task settings: The PCM defines transitions from Calm to Nervous and from active observation to decision-making, yielding RP and DI tasks.

2.4 Passenger EEG Decoding Strategy (PEDS)

PEDS formulates passenger EEG decoding around risk-aware sequential labels and task-specific temporal windows, then uses a task-coupled 3D-CRNN to jointly model pre-onset risk and event-centered danger. The architecture preserves electrode topology and fuses branch features for classification.

  • Risk formulation: PEDS separates Safe, Low-Risk, and High-Risk conditions using Risky and Dangerous terms that quantify surrounding traffic and hazardous behaviors.
  • Risk-aware sequential labeling: RSL assigns distinct labels to time windows within each segment, enabling simultaneous Risk Prediction and Danger Identification.
  • Task-specific windows: Peak-latency analysis selects [−2 s, −0.5 s] for RP and [−0.2 s, 1 s] for DI, balancing discriminative EEG features with prediction timing.RP uses pre-event activity, whereas DI covers event-related peaks around 300 ms in PO3.
  • 3D-CRNN architecture: The task-coupled 3D-CRNN maps separate RP and DI windows into electrode-topology-informed 3D representations, extracts spatial-temporal features, and fuses them with a GRU.The RP branch uses only pre-onset EEG, preserving causality.
  • Task-specific encoders: The RP and DI encoder branches use different temporal and spatial designs to capture gradual anticipatory activity versus stimulus-locked responses.The RP branch uses longer temporal kernels and coarser early downsampling, while DI uses denser sampling and shorter kernels.

3 Results

The experiments evaluate pre-event risk prediction, RSL-enhanced danger identification, event-wise robustness, and generalization across sessions and subjects. Results are reported with balanced accuracy as the primary metric because class distributions are not perfectly balanced.

  • Evaluation scope: The evaluation tests pre-event RP, RSL effects on DI, event-wise consistency, cross-session generalization, and cross-subject generalization.
  • Evaluation protocol: Balanced Accuracy is the primary metric, with results generally averaged across subjects as mean ± standard deviation.
  • Risk Prediction: 95.3% ± 2.7% BA is achieved by 3D-CRNN for RP, with the best overall performance across Precision, Recall, F1-score, and BA.The result indicates that pre-event EEG contains discriminative neural patterns related to upcoming risk.
  • Risk Prediction: 0.947 ± 0.032 BA is achieved by ShallowConvNet as the strongest RP baseline, while XGB-DIM reaches 0.553 ± 0.053 BA.

3.4 Single-Subject Danger Identification

RSL improves single-subject DI for 3D-CRNN and supports strong performance across AEB, cut-in, and pedestrian events. Generalization remains strongest in cross-session and seen-subject settings, while unseen-subject performance exposes inter-subject variability as a continuing challenge.

  • Single-subject DI: 0.850 ± 0.032 BA is achieved by 3D-CRNN with RSL for single-subject DI, improving from 0.809 ± 0.039 without RSL.The absolute BA improvement is 0.041.
  • Single-subject DI: RSL improves most baselines, although its largest sensitivity gains for XGB-DIM do not eliminate lower class-balanced discrimination than CNN-based models.
  • Event-wise analysis: 3D-CRNN achieves the best RP BA across AEB, cut-in, and pedestrian scenarios, with values of 0.880 ± 0.127, 0.946 ± 0.032, and 0.889 ± 0.092.
  • Event-wise analysis: With RSL, 3D-CRNN achieves the strongest DI results across AEB, cut-in, and pedestrian scenarios, including BA values of 0.778 ± 0.129, 0.880 ± 0.026, and 0.799 ± 0.074.
  • Cross-session generalization: 0.770 ± 0.053 BA is achieved by 3D-CRNN in cross-session DI, outperforming all compared baselines despite session-related variability.
  • Cross-subject generalization: 0.774 ± 0.011 BA is achieved on seen subjects, compared with 0.649 ± 0.085 on unseen subjects, where DSC-ConvLSTM reaches slightly higher BA.The seen–unseen gap highlights inter-subject variability in EEG decoding.

4 Discussion

The study presents passenger EEG decoding for both anticipatory risk prediction and danger identification, using PCM, RSL, and PEDS with 3D-CRNN. Results are promising, but validation remains bounded by controlled settings, interaction-focused scenarios, deployment conditions, prediction-timing trade-offs, and ethical requirements.

  • Contributions: The framework addresses both Risk Prediction and Danger Identification, extending EEG-based hazard analysis beyond immediate hazard detection.It targets potential risks before hazardous behavior fully unfolds.
  • Contributions: PCM links traffic conditions with passenger neural responses, while RSL models the sequential relationship between risk prediction and danger classification.PEDS then uses 3D-CRNN to jointly model EEG spatial topology and temporal dynamics.
  • Results: BA reached 0.953 for Risk Prediction and 0.850 for single-subject Danger Identification with the proposed framework.Event-wise analysis found the 3D-CRNN with RSL achieved the best DI performance across AEB, cut-in, and pedestrian scenarios.
  • Limitations: The framework was developed in a controlled passenger-observation setting with a moderate participant count and repeated simulated scenarios.Fatigue, reduced vigilance, habituation, and individual differences may contribute to intra-subject, cross-session, and unseen-subject variability.
  • Limitations: The RP/DI taxonomy covers observable ego–target interactions rather than all AV hazards, leaving non-interactive hazards for future study.Examples include occluded intersections, limited visibility, narrow passages, road-surface anomalies, and aggressive ego-vehicle maneuvers.
  • Limitations: Real-vehicle deployment remains unvalidated because motion, vibration, electrode displacement, and environmental noise may weaken decoding stability.The study also identifies trade-offs between earlier prediction windows and weaker or more ambiguous neural evidence, and notes that full-system latency includes acquisition and preprocessing.
  • Limitations: Future deployment would require safeguards for informed consent, secure data handling, controlled access, and transparent governance of sensitive EEG data.These ethical and privacy issues are beyond the technical study’s scope but are identified as important for responsible translation.
  • Conclusion: Overall, the work is a controlled proof-of-concept rather than deployment-ready validation across users and real-world conditions.The authors call for larger cohorts, realistic environments, improved RP robustness, and multimodal integration.

5 Conclusion

The paper presents a passenger-centered EEG framework targeting both Risk Prediction and Danger Identification through PCM, RSL, PEDS, and 3D-CRNN. Its results support passenger EEG as a source of anticipatory and hazard-related information, while remaining limited by controlled validation and inter-subject variability.

  • Framework: The framework unifies passenger neural mechanisms, task formulation, and EEG decoding for Risk Prediction and Danger Identification.Its components are PCM, RSL, PEDS, and a 3D-CRNN architecture.
  • Results: BA reached 0.953 for Risk Prediction and 0.850 for single-subject Danger Identification, with the highest cross-subject Precision, Recall, and F1-score among evaluated baselines.The results indicate both anticipatory and hazard-related information in passenger EEG.
  • Limitations: The study remains a controlled proof-of-concept with moderate sample size, simulator-based evaluation, and unresolved inter-subject variability.Future work should use larger cohorts, subject-independent modeling, realistic online validation, and multimodal signals.
  • Implications: Passenger EEG is presented as a meaningful cognitive-information source for developing, training, and evaluating safer and more adaptive autonomous vehicles.The supported scope is AV safety research and development rather than established deployment performance.

Ethics Statement

The study reports compliance with the Declaration of Helsinki and approval by Tsinghua University’s Institutional Review Board. Participants gave written informed consent and could ask questions, protect confidentiality, take breaks, or end sessions freely.

  • Ethical approval: The study complied with the Declaration of Helsinki and received Institutional Review Board approval from Tsinghua University, China.
  • Consent and confidentiality: Participants provided written informed consent after receiving study information and were assured that personal and identifiable data would remain confidential.
  • Participant rights: Participants could ask questions, take breaks when necessary, and end the session at any time for any reason.

A Scenario Specifications

The study uses 14 traffic scenarios covering pedestrian encounters, vehicle cut-ins, and lead-vehicle cut-outs with or without emergency braking. Scenario documentation records clip counts, risk levels, descriptions, and example visualizations, while Figure 8 shows settings and event distribution.

  • Scenario coverage: The scenario set contains 14 traffic scenarios involving pedestrian encounters, vehicle cut-ins, and lead-vehicle cut-outs.Lead-vehicle cut-outs are represented with or without emergency braking.
  • Scenario documentation: Figure 8 presents the scenario settings and event distribution for the traffic clips.
  • Scenario documentation: Table 9 organizes each scenario by ID, description, clip count, risk level, and example visualization.

B Data Split and Composition

The evaluation uses 11 valid subjects across sessions 1–4, with explicit partitioning procedures for single-subject, event-wise, cross-session, and cross-subject protocols. Data include all listed event scenarios, with separate Risk Prediction and Danger Identification labels.

  • The study includes 11 valid subjects, identified as 1–8 and 10–12, with data from sessions 1–4.
  • For single-subject and event-wise protocols, each class is split into training and testing sets at a 70:30 ratio, then training data are shuffled and divided using validation_split=0.3.
  • The evaluation covers AEB, cut-in, and left- and right-side pedestrian events, with pedestrian events combined in the scenario specification.
  • Risk Prediction uses Safe/Risk labels, whereas Danger Identification uses Non-danger/Danger labels.
  • Separate tables document all-event single-subject, scenario-wise event-wise, cross-session, and cross-subject LOSO split compositions.
Loading 2609.07128v1…