Source-linked AI summary
Driver Distraction Identification with an Ensemble of Convolutional Neural Networks
Hesham M. Eraqi, Yehya Abouelnaga, Mohamed H. Saad, Mohamed N. Moustafa
TL;DR
Distracted driving contributes substantially to road-traffic harm, motivating reliable posture identification beyond limited phone-use detection. The paper introduces a publicly available dataset and a genetically weighted CNN ensemble, achieving 90% classification accuracy while also exploring real-time simplification.
Problem
Distracted-driving detection research has focused on a small set of distractions, while distracted driving contributes to substantial road-traffic harm.
Method
The paper collects a publicly available distracted-driver dataset and trains a genetically weighted ensemble of CNNs using multiple visual inputs and transfer learning.
Results
90% classification accuracy was achieved by the best genetically weighted CNN ensemble model.
Takeaways & Limitations
The dataset provides a baseline for future distracted-driving research, while a simpler AlexNet model can operate in real time with satisfactory classification accuracy.
Takeaways & Limitations
The authors identify the need for better face, hand, and skin detectors, particularly through improved localization methods.
Abstract
from arXiv · showhide
The World Health Organization (WHO) reported 1.25 million deaths yearly due to road traffic accidents worldwide and the number has been continuously increasing over the last few years. Nearly fifth of these accidents are caused by distracted drivers. Existing work of distracted driver detection is concerned with a small set of distractions (mostly, cell phone usage). Unreliable ad-hoc methods are often used.In this paper, we present the first publicly available dataset for driver distraction identification with more distraction postures than existing alternatives. In addition, we propose a reliable deep learning-based solution that achieves a 90% accuracy. The system consists of a genetically-weighted ensemble of convolutional neural networks, we show that a weighted ensemble of classifiers using a genetic algorithm yields in a better classification confidence. We also study the effect of different visual elements in distraction detection by means of face and hand localizations, and skin segmentation. Finally, we present a thinned version of our ensemble that could achieve 84.64% classification accuracy and operate in a real-time environment.
1 Introduction
Distracted driving is a major road-safety concern, while existing detection research has focused on limited distraction types. The paper addresses this gap with a deep-learning system and a publicly available, broader-posture dataset.
- Motivation: 1.25 million yearly deaths were attributed to road traffic accidents worldwide, with distracted driving accidents steadily increasing.The paper also reports that nearly one fifth of traffic accidents are caused by distracted drivers.
- Practical relevance: Correct distraction identification is positioned as critical for mitigation systems that adapt in-vehicle information systems to driver state.The paper also connects detection with warnings, preventive measures, semi-autonomous driving, and ADAS functions.
- Research gap: Existing distracted-driving detection research covers manual, visual, and cognitive distractions but commonly focuses on limited activities such as mobile-phone use.The paper distinguishes these distraction types by whether attention, vision, manual control, or cognition is diverted.
- Approach: The proposed approach uses dashboard-mounted RGB images, transfer learning, and retrained fully connected layers to recognize distraction postures.Networks are pretrained on ImageNet before being adapted to the target posture-recognition task.
- Approach: The system combines multiple CNN architectures trained on raw, skin-segmented, face, hand, and face-plus-hands images.The paper presents this as a learnable weighted ensemble together with a new skin-segmentation method and an annotation tool.
2 Literature Review
Prior work spans independent phone-use detectors, specialized datasets, activity-recognition systems, and the StateFarm competition. Methods include SVMs, hand-crafted transforms, region-based classifiers, and neural classifiers across varied camera setups and distraction sets.
- Research landscape: Prior distracted-driving research is grouped into phone-use detection, UCSD datasets, Southeast University datasets, and StateFarm’s Kaggle competition.These groups represent the main dataset and publication lines identified by the paper.
- Phone-use detection: SVM-based studies detect mobile-phone use using frontal driver views or transportation cameras, sometimes assuming fixed hand and face locations.The transportation-camera setting is described as more competitive than the frontal-view setup.
- Activity recognition: Kinect-based activity recognition uses frontal and rear driver views to identify hands-on-wheel states and distractions involving the radio, gear, and mirrors.A related fusion framework segments wheel, gear, and instrument-panel regions before classifying activities.
- Specialized datasets: The Southeast University dataset includes four distraction postures and has been evaluated with contourlet, multiwavelet, Random Forest, KNN, and MLP methods.The cited study reports Random Forests as the strongest classifier in that comparison.
- StateFarm dataset: StateFarm introduced ten posture classes for public posture classification, including safe driving, phone use, radio operation, drinking, reaching, grooming, and passenger conversation.The paper identifies StateFarm’s competition dataset as a major inspiration for its own work.
3 Dataset Design
The paper constructs a publicly obtainable distracted-driver dataset because existing alternatives were restricted or limited in posture coverage. Data were recorded from multiple participants, vehicles, cameras, and driving conditions, then converted and annotated as images.
- Dataset rationale: The dataset was created because StateFarm’s data were restricted and the Southeast University dataset contained only four distraction postures and was not publicly available.The resulting dataset is described as publicly obtainable subject to a license agreement.
- Collection setup: Data collection used a fixed single-camera perspective with two cameras across two phases, recording RGB images from a smartphone and a depth camera.The depth camera’s depth information was not used; different cameras were intended to add dataset diversity.
- Collection setup: Videos were recorded in multiple vehicles and converted into individual images at resolutions of 1080 × 1920 or 640 × 480.The camera was mounted above the front passenger seat using the car roof handle.
- Annotation: An open-source, multiplatform web annotation tool was developed to label the collected videos.The tool was implemented with Electron, AngularJS, and JavaScript and was intended to support future dataset extension.
- Dataset composition: The dataset includes 14,478 extracted frames collected from 44 participants across seven countries and five cars.Participants included 29 males and 15 females, with some recording multiple sessions under differing conditions and clothing.
4 Proposed Method
The proposed method combines multiple visual representations and CNN architectures in a genetically weighted ensemble, while adding specialized skin, face, and hand processing.
- CNN ensemble: The ensemble trains CNNs on raw, skin-segmented, face, hands, and face+hands images using AlexNet, InceptionV3, ResNet-50, and VGG-16.The networks are fine-tuned from pre-trained ImageNet models, except AlexNet models trained from scratch.
- Skin segmentation: Skin segmentation models pixel skin probability with a Multivariate Gaussian Naive Bayes classifier fitted to Gaussian likelihoods for skin and non-skin classes.The model assumes equal prior probabilities for skin and non-skin classes and was trained using the UCI Skin Segmentation dataset.
- Pipeline: For each processed image source, the system applies CNN classifiers and combines their class-probability vectors through weighted softmax aggregation.The overview describes face detection, hand detection, and skin segmentation being applied to each frame before classification.
- Skin segmentation: Adding pixel spatial information to color features improves skin segmentation, addressing sensitivity to changing illumination conditions during driving.The augmented feature vector includes RGB values and each pixel’s X and Y coordinates.
- Face & Hands Detection: Face and hand localization provide additional image sources: face detection can misdetect non-frontal faces, while hand localization converts fully connected AlexNet layers into convolutional layers.The modified hand detector accepts variable-size inputs and produces variable-size outputs.
- GA-based ensemble: A genetic algorithm learns classifier weights for combining the softmax outputs because classifiers may contribute unequally to the final decision.The fitness function uses Negative Log Likelihood over a 50% random population sample to help prevent overfitting.
5 Experiments
Experiments evaluate visual inputs, data splits, temporal context, and model size for distracted-driver classification. The ensemble reaches 90% accuracy on driver-separated test data, while a smaller two-model system reaches 84.64% with real-time CPU performance.
- Visual inputs: Hands contribute more to posture recognition than faces in the evaluated visual-input experiments.AlexNet and InceptionV3 perform best on original images, while switching to skin-segmented images does not substantially change their accuracy.
- Visual inputs: Pixel spatial information improves skin detection accuracy, but the overall method remains unsatisfactory under varying driving illumination.The study reports that skin, face, and hand components add system complexity without improving overall distraction identification accuracy.
- Driver-separated evaluation: 90% accuracy is achieved by InceptionV3 on the test split containing unique drivers.The driver-separated split uses different drivers, cars, times of day, lighting conditions, and driving conditions between training and testing.
- Temporal context: Accuracy improves as temporal context increases up to a threshold, then declines when past information becomes irrelevant; the optimal history interval is 3.35 seconds.At M = 1, classification uses only the current frame; for larger M, predictions from multiple frames are averaged.
- Real-time system: Large deep models are unsuitable for real-time deployment, motivating a NasNetMobile reduction from models with 21.8M–134.3M parameters to 4.3M.The smaller architecture supports the real-time system evaluated above.
- Real-time system: 84.64% classification accuracy is achieved by an ensemble of two NasNetMobile models while maintaining real-time CPU performance.The two models use original and skin-segmented inputs.
6 Conclusion
The paper presents a publicly available dataset and a vision-based system for recognizing distracted-driving postures. Its best model achieves 90% classification accuracy, while future work targets improved real-time localization of faces, hands, and skin.
- 90% classification accuracy was achieved by a genetically weighted ensemble of convolutional neural networks.The ensemble is the paper’s best-performing model and provides a baseline for future research.
- A novel publicly available distracted-driver dataset was collected to develop and test the system.The dataset supports recognition of distracted-driving postures and benchmarking future work.
- Face, hand, and skin detection improved ensemble classification accuracy, but their real-time performance overhead exceeded their contribution.
- Future work proposes manually labeling hand and face proposals to train Fast-RCNN or another detector for one-shot localization.The proposed detector would be evaluated against the existing CNN-based localization method.