Source-linked AI summary
A Realistic Dataset and Baseline Temporal Model for Early Drowsiness Detection
Reza Ghoddoosian, Marnim Galib, Vassilis Athitsos
TL;DR
Drowsiness detection needs practical early-warning systems because explicit signs may appear too late for safe intervention. The paper introduces a large realistic dataset and evaluates a low-cost temporal model using sequential blink features. Its baseline outperforms human judgment on the RLDD dataset, while the authors limit claims about driving applicability because the recordings lack driving-specific conditions.
Problem
Early drowsiness detection is needed because explicit signs may appear only shortly before accidents, yet large public realistic datasets are scarce.
Method
The paper introduces the RLDD dataset and a low-computation baseline using sequential blink features with an HM-LSTM for multistage drowsiness detection.
Results
The baseline outperforms human judgment on the RLDD dataset in two designed metrics.
Takeaways & Limitations
Realistic public data and temporal blink modeling support evaluation of subtle drowsiness cues for early detection.
Takeaways & Limitations
The dataset does not represent driving conditions because it lacks night lighting and camera vibration from car motion.
Abstract
from arXiv · showhide
Drowsiness can put lives of many drivers and workers in danger. It is important to design practical and easy-to-deploy real-world systems to detect the onset of drowsiness.In this paper, we address early drowsiness detection, which can provide early alerts and offer subjects ample time to react. We present a large and public real-life dataset of 60 subjects, with video segments labeled as alert, low vigilant, or drowsy. This dataset consists of around 30 hours of video, with contents ranging from subtle signs of drowsiness to more obvious ones. We also benchmark a temporal model for our dataset, which has low computational and storage demands. The core of our proposed method is a Hierarchical Multiscale Long Short-Term Memory (HM-LSTM) network, that is fed by detected blink features in sequence. Our experiments demonstrate the relationship between the sequential blink features and drowsiness. In the experimental results, our baseline method produces higher accuracy than human judgment.
1. Introduction
Early drowsiness detection matters because drowsiness threatens safety in driving and workplaces, while explicit signs may appear only shortly before an accident. This paper addresses the limited availability of large, public, realistic datasets with a low-cost temporal blink-based baseline.
- Drowsiness threatens driving and workplace safety, motivating economical systems that detect its onset early.Driver fatigue is associated with crashes, deaths, injuries, monetary losses, and impaired workplace performance.
- Explicit signs such as yawning, nodding, and prolonged eye closure may emerge only moments before an accident, so subtle cues are relevant for early detection.The dataset includes both subtle facial signs and more explicit signs.
- The RLDD dataset contains around 30 hours of low-frame-rate RGB video from 60 participants, labeled alertness, low vigilance, or drowsiness.The recordings came from indoor real-life environments and various phone or web cameras.
- The baseline combines sequential blink features with an HM-LSTM and voting, using low frame rates while achieving higher accuracy than human judgment.The pipeline targets subtle cues and is described as an end-to-end real-time baseline.
- Previous datasets were often private or acted, leaving a shortage of large, public, realistic drowsiness data.The paper distinguishes realistic recordings from instructed simulations of drowsiness.
- The dataset is presented as larger, more realistic, and more varied than existing public datasets, using real drowsiness and different cameras.Participants recorded themselves with cell phones or web cameras in indoor environments of their choice.
2. Related Work
Prior drowsiness research used private, small, acted, or otherwise mismatched datasets, limiting direct comparison. Existing methods include handcrafted blink features and frame-based deep models, while this paper emphasizes realistic temporal evaluation.
- Prior work lacked datasets that were simultaneously public and realistic, making comparison and state-of-the-art assessment difficult.
- NTHUdriver provides public infrared videos from 36 participants simulating driving, but its acted drowsiness may not represent real early drowsiness.
- DROZY uses real drowsiness data, whereas this paper reports a larger dataset with 60 rather than 14 subjects and recordings across all three alertness classes.
- Other work used 90 hours of real driving, but its dataset was private and unavailable as a benchmark.
- Camera-based approaches use handcrafted blink features or CNN-learned features, but cited studies were not validated on a large public dataset.
- Reported accuracies vary across datasets and protocols, including 82.5% on a private dataset, 73% on NTHU, and 89% on private acted data.These comparisons span different datasets, labels, and evaluation settings.
3. The Real-Life Drowsiness Dataset (RLDD)
The RLDD dataset is a large, public, realistic resource for multistage drowsiness detection, covering subtle as well as extreme states. It contains 180 participant-recorded videos labeled as alert, low vigilant, or drowsy.
- Dataset purpose: The RLDD dataset targets both subtle and extreme drowsiness for early and multistage detection.Its realistic recordings are intended to support detection before drowsiness becomes obvious.
- Data collection: 60 healthy participants contributed three approximately ten-minute videos each, producing 180 RGB videos.Participants recorded one video for each of the three drowsiness states.
- Recording conditions: Recordings were made with personal phones or web cameras, producing varied video resolutions and qualities.The dataset was designed to resemble recordings obtainable in practical settings.
- Labels: Videos were labeled alert, low vigilant, or drowsy using participants’ predominant self-reported state.The labels correspond to increasing sleepiness, from no signs of sleepiness to actively trying not to fall asleep.
- Evaluation structure: The dataset contains 111.3 Gigabytes of video and uses five cross-validation folds of 12 participants.Participants, rather than individual videos, were divided into folds.
- Human baseline: Human judgment was measured by asking 20 volunteers to estimate each muted video’s drowsiness degree on a 0-to-10 scale.Four volunteers evaluated the videos in each cross-validation fold.
4. The Proposed Baseline Method
The baseline detects drowsiness from sequential blink features using an HM-LSTM-based temporal pipeline. It normalizes subject-specific blink patterns, models drowsiness on a continuous scale, and aggregates sequence predictions by voting.
- Blink features: The pipeline extracts blink duration, amplitude, eye opening velocity, and frequency from detected blink events.Each blink is represented as a four-dimensional feature vector derived from eye landmarks and EAR dynamics.
- Preprocessing: Subject-specific normalization uses the first third of alert-state blinks to compute each feature’s mean and standard deviation.The calibration statistics are applied to subsequent states, followed by cross-individual feature normalization.
- Temporal model: An HM-LSTM models temporal and hierarchical patterns across blink sequences.The network is intended to capture relationships among blinks occurring at different timescales.
- Regression and classification: A regression unit outputs drowsiness on a continuous 0-to-10 scale before classification.The continuous output represents the transition from alertness to drowsiness rather than assigning classes directly.
- Regression and classification: The regression output is discretized into alert, low vigilant, and drowsy classes using thresholds at 3.3 and 6.6.Alert covers 0.0 ≤ out < 3.3, low vigilant covers 3.3 ≤ out ≤ 6.6, and drowsy covers 6.6 < out ≤ 10.
- Voting: The final video label is the most frequent class predicted across sliding-window blink sequences.Voting addresses the fact that individual blinks may not all reflect the subject’s overall drowsiness state.
5. Experiments
The experiments evaluate the model with four metrics across cross-validation folds and compare it with simplified architectures and human judgment. HM-LSTM improves accuracy and regression performance, while voting and temporal modeling help translate blink sequences into video-level predictions.
- Evaluation metrics: Four metrics evaluate the model at blink-sequence and video levels, covering classification accuracy and regression error.The metrics are Blink Sequence Accuracy, Blink Sequence Regression Error, Video Accuracy, and Video Regression Error.
- Experimental setup: The five-fold evaluation uses blink sequences of 30 consecutive blinks, zero-padding videos with fewer than 30 blinks.Sequences are generated with a stride of two, and results are averaged across the five folds.
- Baselines: The comparison includes the proposed HM-LSTM pipeline, a typical-LSTM variant, a fully connected variant, and human judgment.The simplified networks test the contribution of HM-LSTM cells and temporal modeling.
- Comparative results: 4% higher accuracy is obtained after switching from typical LSTM cells to HM-LSTM cells, with lower regression error for HM-LSTM.The results also suggest that temporal models outperform simple fully connected layers for drowsiness detection.
- Voting: Voting raises video accuracy because individual blink sequences from one video can represent different drowsiness levels despite sharing one video label.This explains why Blink Sequence Accuracy is lower than the voting-based Video Accuracy.
- Classification results: The middle low-vigilance class is hardest to classify, while alert and drowsy subjects exceed 80% accuracy and are rarely confused with each other.The proposed model detects early and subtle drowsiness better than humans and contains approximately 50,000 trainable parameters.
- Computational requirements: 35–80 frames per second are processed at evaluation time across frame sizes from 568x320 to 1920x1080.The reported speed was measured on a Linux workstation with an Intel Xeon CPU and 16GB of memory.
6. Conclusions
The paper contributes the publicly available RLDD dataset and a low-demand blink-based baseline, reporting performance above human judgment on two designed metrics.
- The baseline method uses temporal relationships between blinks for multistage drowsiness detection.
- The proposed method has low computational and storage demands.
Supplementary Material
The supplementary material identifies the paper’s authors and their affiliation with the Vision-Learning-Mining Lab at the University of Texas at Arlington.
- The paper lists Reza Ghoddoosian, Marnim Galib, and Vassilis Athitsos as authors.
- The authors are affiliated with the Vision-Learning-Mining Lab.
- The Vision-Learning-Mining Lab is at the University of Texas at Arlington.
Blink Retrieval Algorithm
The Blink Retrieval Algorithm processes an EAR time series to identify blink intervals and their characteristic points. It uses extrema, thresholding, and sign changes to retrieve blink starts, ends, and bottoms in linear time.
- The blink pipeline begins with an EAR signal computed from six 2D eye-region landmarks.
- A 13-frame EAR vector is classified by an SVM as open or closed, and consecutive closed labels define a candidate blink sequence.
- For each retrieved blink, the algorithm returns definite start, end, and bottom frames based on the EAR extrema.
- The algorithm runs in Θ(M) time and is reported to operate in real time in practice.
- The algorithm differentiates the EAR sequence, detects local extrema, and uses a threshold to suppress noise-related fluctuations.
- Negative sign changes between threshold-labeled extrema identify downward and upward eye movements that form retrieved blinks.