Source-linked AI summary
A Vision Based System for Monitoring the Loss of Attention in Automotive Drivers
Anirban Dasgupta, Anjith George, S. L. Happy, Aurobinda Routray
TL;DR
Driver drowsiness monitoring matters for road safety, but EEG measurement on board may cause discomfort and artifacts. This paper estimates PERCLOS with a real-time vision system for day and night driving, and reports robust night-time face and eye detection alongside a 98.6% linear SVM hit rate.
Problem
On-board EEG measurement may cause driver discomfort and artifacts, motivating non-contact vision-based monitoring of drowsiness.
Source: §I · p. 1Method
The system estimates PERCLOS using Haar-based face detection and Kalman tracking, day/night eye detection, and SVM eye-state classification with rotation and illumination compensation.
Source: §II · p. 4, §IX · p. 12Results
Night driving tests found face and eye detection robust except for a few malfunctions from other-vehicle and streetlights; linear SVM hit rate was 98.6%.
Source: D. Testing under Night Driving… · p. 12, §IX · p. 12Takeaways & Limitations
The system was implemented on an embedded platform and tested during day and night driving conditions.
Source: §II · p. 4, C. Testing under Day-driving Conditions · p. 11, D. Testing under Night Driving… · p. 12Drivers wearing spectacles were not considered by the algorithm.
Source: §II · p. 4
Abstract
from arXiv · showhide
On board monitoring of the alertness level of an automotive driver has been a challenging research in transportation safety and management. In this paper, we propose a robust real time embedded platform to monitor the loss of attention of the driver during day as well as night driving conditions. The PERcentage of eye CLOSure (PERCLOS) has been used as the indicator of the alertness level. In this approach, the face is detected using Haar like features and tracked using a Kalman Filter. The Eyes are detected using Principal Component Analysis (PCA) during day time and the block Local Binary Pattern (LBP) features during night. Finally the eye state is classified as open or closed using Support Vector Machines(SVM). In plane and off plane rotations of the drivers face have been compensated using Affine and Perspective Transformation respectively. Compensation in illumination variation is carried out using Bi Histogram Equalization (BHE). The algorithm has been cross validated using brain signals and finally been implemented on a Single Board Computer (SBC) having Intel Atom processor, 1 GB RAM, 1.66 GHz clock, x86 architecture, Windows Embedded XP operating system. The system is found to be robust under actual driving conditions.
I. INTRODUCTION
Driver drowsiness is a road-safety concern, and PERCLOS is selected as the alertness indicator. The paper reviews gaps in real-time eye monitoring and presents an embedded vision-based approach for day and night driving.
I. INTRODUCTION
- PERCLOS measures the proportion of time eyelids are at least 80% closed and can be estimated from video after face and eye detection and eye-state classification.The paper describes a closed eye as one in which the eyelid is closed at least 80%.Source: §I · p. 1
- The paper reports that its PERCLOS accuracy was validated against EEG signals in earlier work.The authors cite their previous works, for this validation.Source: §I · p. 1
II. LITERATURE REVIEW
- Earlier monitoring approaches left practical gaps in addressing illumination variation, head rotation, real-time operation, or night driving.The review also notes limitations involving vehicle vibration and eye occlusion in prior approaches.Source: §II · pp. 2–3
- Song et al.’s method achieved 98.7% eye detection on 150 Bern images with view and gaze-direction variation, and 96.6% on 564 AR images with expression and lighting variation, but was not tested in real time.Source: §II · p. 3
- The review identifies real-time frame rate, illumination, face orientation, vibration, and spectacles as issues for in-vehicle eye detection and PERCLOS estimation.The proposed work addresses some of these issues, but does not consider drivers wearing spectacles.Source: §II · pp. 3–4
- The proposed real-time system estimates PERCLOS on an embedded platform, using day- and night-specific eye detection and SVM eye-state classification.It also applies face rotation and illumination compensation; drivers wearing spectacles were not considered.Source: §II · p. 4
III. REAL-TIME FACE DETECTION
The face-detection pipeline uses Haar-like features on downsampled frames, then maps a face-based region of interest back to the original frame for eye detection.
rectangle features
- Haar-like features are rectangular image features computed as the difference between pixel sums in white and black regions.Integral images are used for fast feature computation; AdaBoost selects features that are combined into a cascade of classifiers.Source: A. Face detection using Haar-like… · p. 4, rectangle features · p. 4
III. REAL-TIME FACE DETECTION
- Face detection runs at 3 fps on the stated SBC, motivating frame downsampling to improve real-time performance without compromising accuracy.After integral-image computation, Haar-like features can be calculated at any scale or location in constant time.Source: rectangle features · p. 4
A. Face detection using Haar-like features
- The algorithm saves and downsamples each frame, detects a face with a Haar classifier, selects the upper-face ROI, remaps it, and then detects eyes there.The described method's detection rate depends on its scale factor, with faster processing at lower resolution.Source: Table I · p. 4
rectangle features
- Fig. 2 depicts eye detection through downsampling, ROI extraction, and remapping.The figure accompanies the pipeline that detects the face at reduced resolution and maps ROI coordinates onto the original frame.Source: Table I · p. 4, Figure 2 · p. 5
B. Effect of SF on runtime
Increasing the down-sampling factor (SF) raises average face-detection speed, but accuracy constrains the useful range. Accuracy stays almost constant through SF 6 and then declines, making 6 the reported runtime–accuracy trade-off.
B. Effect of SF on runtime
- Six subjects’ 640 × 480 videos were down-sampled at SF values of 2, 4, 6, 8, and 10 for runtime analysis.Videos were recorded at 30 fps under laboratory conditions; processing was performed offline.Source: B. Effect of SF on runtime · p. 5
C. Effect of SF on accuracy
- Accuracy is evaluated using ROC curves and AUC, with manually marked video frames providing face-presence ground truth.Detection outcomes are compared with ground truth to calculate true and false positives and negatives.Source: C. Effect of SF on accuracy · p. 5
B. Effect of SF on runtime
- Average face-detection execution speed increases almost linearly as SF rises.Fig. 4 plots average speed across subjects for each SF.Source: Figure 4 · p. 5, B. Effect of SF on runtime · p. 5
C. Effect of SF on accuracy
- Face-detection accuracy remains almost constant through SF 6, then decreases; SF 6 is selected as the optimal runtime–accuracy trade-off.The comparison is based on ROC curves and AUC across SF values.Source: Figure 5 · p. 5, Figure 6 · p. 5, C. Effect of SF on accuracy · p. 5
A. Affine Transformation Based Detection of In-plane Rotated Faces
The face-detection procedure uses affine transformation for in-plane rotations and perspective transformation for small off-plane rotations. The latter aligns one facial view to another, while the paper notes that off-plane rotations are rare in normal driving and restricted to ±15°.
A. Affine Transformation Based Detection of In-plane Rotated Faces
- Affine transformation is used for in-plane rotation, and the face detector returns a de-rotated region of interest for eye detection.The transformation is attempted only when the Haar classifier fails to detect a frontal face, so average processing speed is not affected much.Source: A. Affine Transformation Based… · pp. 5–6
- If frontal-face detection fails, the algorithm rotates the image by ±30° and searches the transformed image with the Haar classifier.If those searches also fail, it concludes that no face is found.Source: Table II · p. 6
B. Perspective Transformation Based Detection of Off-plane Rotated Faces
- Perspective transformation maps one facial view to another and is described as robust and computationally inexpensive for aligning the head under small off-plane rotations.Its coefficients are estimated using nonlinear least squares.Source: B. Perspective Transformation Based… · p. 6
- Off-plane rotations are observed very rarely during normal driving and are restricted to ±15°.The perspective transformation is used to compensate for such small rotations.Source: B. Perspective Transformation Based… · p. 6
- Variation in illumination is identified as another problem for detection accuracy.Source: B. Perspective Transformation Based… · p. 6
IV. COMPENSATION OF EFFECT OF LIGHTING CONDITIONS
Bi-Histogram Equalization (BHE) compensates for illumination variation by equalizing image regions split at the mean intensity. The paper also describes Kalman-filter tracking to reduce face-search work.
IV. COMPENSATION OF EFFECT OF LIGHTING CONDITIONS
- BHE splits the image at its mean intensity and equalizes each sub-image separately, preserving the input mean intensity.When face detection fails after geometric transformations, the algorithm applies BHE as preprocessing.Source: §IV · p. 6
- Using BHE, detection accuracy improved from 92% to 94% on 400 onboard images captured under extremely varying lighting conditions.The authors note that illumination compensation still has room for improvement.Source: §IV · p. 6
- Geometric transformations and BHE improve accuracy but reduce frame rate because of their computational burden.Kalman-filter face tracking is used to reduce redundant search using temporal information.Source: §IV · p. 7
V. KALMAN FILTER BASED FACE POSITION ESTIMATION
- Kalman-filter tracking was selected because mean-shift and optical tracking rely on image intensities and are illumination-dependent.The approach uses the assumption that a driver's face position does not change abruptly between consecutive frames.Source: §V · p. 7
A. Defining the Model
The face tracker models position over time with a Kalman filter, then uses Haar detections to define and update the tracking system. This reduces the face-search area and speeds detection without reported accuracy loss.
A. Defining the Model
- The model represents each frame with a state vector, advances it using a state-transition matrix, and models process noise as zero-mean additive white Gaussian noise.The measurement equation relates the measurement vector to the next state and models measurement noise as zero-mean additive white Gaussian noise.Source: A. Defining the Model · p. 7
B. Defining the System
- The system defines the matrices and initial covariance, then uses Haar face detections as measurements for the Kalman tracker.The state includes face-box coordinates and the horizontal and vertical velocities of its center; the measurement error is modeled with a variance of 6.5 pixels per coordinate.Source: B. Defining the System · p. 7
- Kalman-based tracking reduces face-search areas, enabling faster Haar-classifier detection without loss in accuracy.If the Haar classifier fails to detect the face, the tracker reinitializes; Table III presents face-detection speed in frames per second.Source: B. Defining the System · p. 8, Table III
VI. REAL-TIME EYE DETECTION
After face localization, the system detects eyes in the remapped region of interest and classifies eye state to obtain PERCLOS. It uses different eye-detection methods for day and night driving.
VI. REAL-TIME EYE DETECTION
- After face localization, the detected eye is classified as open or closed to obtain PERCLOS.The eye is detected in the remapped region of interest.Source: §VI · p. 8
- For daytime driving, the system uses PCA-based eye detection.Source: §VI · p. 8
- For nighttime driving, the system uses block LBP features to detect eyes.Source: §VI · p. 8
A. PCA based eye detection
The daytime eye detector uses PCA to identify eyes within a face region of interest. Its evaluation is described as showing quite accurate eye detection.
A. PCA based eye detection
- The region of interest is resized to 200x70, then searched with 50x40 windows using 10% overlap.The window with the minimum error is selected as the eye.Source: A. PCA based eye detection · p. 8
- The PCA training algorithm represents training eye images as vectors and computes an average eye vector.The detection procedure resizes the face region to match the training-image resolution and tests overlapping windows.Source: Table IV · p. 8, Table V · p. 8
- The PCA test used 300 face images, including 200 open-eye and 100 closed-eye images.Source: A. PCA based eye detection · p. 8
- PCA is reported to be quite accurate for detecting eyes from the region of interest.The section presents sample Eigen eyes and PCA detection results.Source: A. PCA based eye detection · p. 9
B. Detection of eye in NIR images
PCA was not robust for eye detection at night under passive NIR illumination, so the system uses block-LBP histogram features instead.
B. Detection of eye in NIR images
- PCA was not robust for eye detection under passive NIR lighting at night.A low-cost Gallium Arsenide NIR illuminator was designed for this setting.Source: B. Detection of eye in NIR images · p. 9
- Block-LBP histogram features are used to detect eyes in NIR images.The method is described as illumination-invariant, an advantage for this use.Source: B. Detection of eye in NIR images · p. 9
- The block-LBP eye-detection approach follows prior work.Source: B. Detection of eye in NIR images · p. 9
C. LBP features
LBP encodes local image texture as binary codes, while block histograms retain spatial information for eye detection. For NIR images, block LBP is more robust than PCA, though slower.
C. LBP features
- LBP compares each pixel’s 3×3 neighborhood intensities with the center pixel to produce a binary code.The operator is described as a gray-scale invariant texture measure.Source: C. LBP features · p. 9
D. Block LBP histogram
- Block LBP combines local histograms into a global eye-region description, preserving spatial information lost by a single image-wide histogram.Local feature methods are described as more robust to pose or illumination variation than holistic methods.Source: D. Block LBP histogram · p. 9
E. NIR face database creation
- A facial-image dataset was created under laboratory conditions with varied face orientations and eyelid positions for NIR feature training.Source: E. NIR face database creation · p. 9
F. Eye Detection Algorithm using block LBP Features
- For detection, localized-eye block-LBP histograms are projected into PCA space, and the subwindow with minimum reconstruction error is accepted if it falls below a threshold.The method was tested on the whole database after training with 300 prerecorded NIR eye images.Source: F. Eye Detection Algorithm using block… · pp. 9–10
- For NIR images, block LBP has 98% TPR and 2% FPR, versus PCA at 80% TPR and 10% FPR; PCA runs faster.Source: Table IX · p. 10, F. Eye Detection Algorithm using block… · p. 10
VII. EYE STATE CLASSIFICATION
The system classifies open and closed eyes with an SVM and was evaluated in simulation and on-board. In simulated driving, PERCLOS estimation had about 2% error.
VII. EYE STATE CLASSIFICATION
- SVM classifies the detected eye as open or closed using energy components as input and returns a class number.The weights from eye detection and ground-truth labels are used as training input.Source: A. SVM Training · p. 10, §VII · p. 10
B. Results
- SVM training used 460 images from eight subjects under normal and NIR illumination, with testing on another 1,700 images from the same database.The study evaluated different kernels; the classification results are referenced in Table X.Source: B. Results · p. 10
A. System Description
- The embedded system used a USB camera and NIR lighting, and its algorithm ran at 9.5 fps.The camera was mounted behind the steering wheel; a light-dependent resistor switched on NIR illumination when light was insufficient.Source: A. System Description · p. 10
B. Testing under Simulated Driving
- In simulator testing, PERCLOS estimation achieved about 2% error; to assess robustness in actual driving conditions, 20 on-board tests were performed separately for day and night.Source: B. Testing under Simulated Driving · p. 11
C. Testing under Day-driving Conditions
On-board tests evaluated the system in day and night driving, and the conclusion reports a 9.5 fps algorithm with SVM hit rates of 98.6% for linear and 97.3% for quadratic kernels. Spectacles occluding the eyes remain an area for further research.
C. Testing under Day-driving Conditions
- On-board day tests found about 4.5% PERCLOS estimation error; straight-ahead face and eye detection rates were 98.5% and 97.5%, dropping to 95% and 94% with head rotation.Direct sunlight could saturate the camera sensor; the problem was addressed with a camera with a superior image sensor.Source: Figure 24 · p. 11, C. Testing under Day-driving Conditions · p. 11
D. Testing under Night Driving Conditions
- At night, PERCLOS estimation error was about 4.9%, and overall false alarming was about 5.5%.Vehicle and street lights caused a few malfunctions; the NIR lighting was automatically switched on by an LDR.Source: D. Testing under Night Driving… · p. 12
- With head rotation at night, face and eye detection rates were 94.5% and 94.2%, while eye-state classification was 98% accurate.For a straight-ahead gaze, face and eye detection rates were 98.2% and 97.1%.Source: D. Testing under Night Driving… · p. 12
IX. CONCLUSION
- 98.6% hit rate for linear SVM and 97.3% for quadratic SVM; the algorithm ran at 9.5 fps and was reported robust in day and night tests.The algorithm was cross-validated using EEG signals.Source: §IX · p. 12
- Further research is needed for detecting eyes occluded by spectacles.Source: §IX · p. 12