Source-linked AI summary
DMD: A Large-Scale Multi-Modal Driver Monitoring Dataset for Attention and Alertness Analysis
Juan Diego Ortega, Neslihan Kose, Paola Cañas, Min-An Chao, Alexander Unnervik, Marcos Nieto, Oihana Otaegui, Luis Salgado
TL;DR
Driver Monitoring Systems need larger, more comprehensive datasets to support the transition from SAE Level-2 to Level-3 automated driving. The paper introduces the multi-modal DMD dataset and demonstrates its use in real-time driver behaviour recognition, achieving enhanced accuracy across fusion approaches while maintaining CPU-only real-time operation.
Problem
Insufficiently large and comprehensive datasets limit Driver Monitoring System development for the transition from SAE Level-2 to SAE Level-3 automated driving.
Method
The paper constructs the multi-purpose, multi-modal DMD dataset and extracts dBehaviourMD to train and validate a real-time driver behaviour recognition system.
Results
The DMD offers greater variability, label richness, and data volume than existing datasets, while dBehaviourMD-based fusion approaches achieve enhanced accuracy with real-time CPU-only operation.
Takeaways & Limitations
DMD provides an open dataset and a demonstrated CPU-efficient basis for developing and evaluating driver monitoring systems in SAE L2–3 scenarios.
Takeaways & Limitations
The RGB–IR comparison is discussed for daytime conditions, with easier pattern learning by IR offered only as a possible explanation.
Abstract
from arXiv · showhide
Vision is the richest and most cost-effective technology for Driver Monitoring Systems (DMS), especially after the recent success of Deep Learning (DL) methods. The lack of sufficiently large and comprehensive datasets is currently a bottleneck for the progress of DMS development, crucial for the transition of automated driving from SAE Level-2 to SAE Level-3. In this paper, we introduce the Driver Monitoring Dataset (DMD), an extensive dataset which includes real and simulated driving scenarios: distraction, gaze allocation, drowsiness, hands-wheel interaction and context data, in 41 hours of RGB, depth and IR videos from 3 cameras capturing face, body and hands of 37 drivers. A comparison with existing similar datasets is included, which shows the DMD is more extensive, diverse, and multi-purpose. The usage of the DMD is illustrated by extracting a subset of it, the dBehaviourMD dataset, containing 13 distraction activities, prepared to be used in DL training processes. Furthermore, we propose a robust and real-time driver behaviour recognition system targeting a real-world application that can run on cost-efficient CPU-only platforms, based on the dBehaviourMD. Its performance is evaluated with different types of fusion strategies, which all reach enhanced accuracy still providing real-time response.
1 Introduction
Driver monitoring systems are increasingly important as driving automation shifts responsibility between humans and machines, but progress is constrained by limited comprehensive visual datasets. The paper introduces DMD to support deep-learning analysis of driver alertness and attention across varied scenarios.
- 1 Introduction: Human factors contribute to almost 90% of road accidents, motivating systems that address driver-related safety risks.
- 1 Introduction: DMS are increasingly needed as SAE-L2-to-L3 automation makes driving a shared activity between humans and machines.They measure driver features and readiness to support safer mode transfer.
- 1 Introduction: Computer-vision DMS methods offer robust, non-obtrusive sensing but require large visual datasets for tasks such as distraction and drowsiness detection.
- 1 Introduction: The paper introduces DMD, a large-scale multi-modal dataset for training and evaluating deep-learning models of driver alertness and attention across diverse scenarios.DMD includes varying illumination, subject characteristics, self-occlusions, and three RGB, depth, and IR views.
2 Review of in-vehicle monitoring datasets and methods
Existing driver-monitoring datasets commonly target particular body regions, behaviours, or sensing settings. The review positions DMD as broader in activities, labels, variability, and data volume for SAE-L2–3 monitoring.
- 2 Review of in-vehicle monitoring datasets and methods: Existing datasets span hands, face, fatigue, body actions, gaze, head pose, and RGB-D body monitoring, often emphasizing one body region or task.
- 2 Review of in-vehicle monitoring datasets and methods: Body-focused datasets show that adding temporal information to image-based distracted-driver recognition can significantly increase classification accuracy.
- 2 Review of in-vehicle monitoring datasets and methods: DMD provides wider activity coverage, richer labels, and greater data variability and volume than the compared existing datasets.The paper states that none of the existing datasets match DMD on all three dimensions.
- 2 Review of in-vehicle monitoring datasets and methods: DMD combines three camera views and RGB, depth, and IR streams with annotations for distraction, fatigue, gaze, head pose, and hand interactions.
- 2 Review of in-vehicle monitoring datasets and methods: Drive&Act is closest to DMD in volume and label richness but focuses on automated-driving actions with drivers disengaged from the driving task.The paper frames Drive&Act as especially relevant to future SAE-L4–5 vehicles, whereas DMD covers broader SAE-L2–3 behaviours.
3 The Driver Monitoring Dataset (DMD)
DMD is a multi-modal, multi-view video dataset covering diverse driver-monitoring scenarios for real-world monitoring research. Its dBehaviourMD subset is balanced and structured for fixed-window behaviour recognition, with broader coverage than existing behaviour datasets.
- 3.1 DMD dataset: DMD combines three camera views with RGB, depth, and infrared video across real-car and simulator environments.The recordings capture drivers’ faces, hands, and bodies, with synchronized streams from three cameras.
- 3.1 DMD dataset: The dataset covers distraction, drowsiness, gaze allocation, head pose, hand positions, object interaction, and contextual annotations across 37 volunteers.Annotations include geometric, temporal, and context classes, totaling 93 classes across the full DMD.
- 3.1 DMD dataset: DMD records scripted behaviour protocols while allowing participants to perform actions without detailed execution instructions, preserving personal variability.The recordings include actions such as phone use, drinking, reaching, gear changes, fatigue signs, and fixation on predefined cabin regions.
- 3.3 Driver behaviour monitoring dataset (dBehaviourMD): dBehaviourMD converts side-view driving-action recordings into short clips for real-time recognition and includes balanced sampling with fixed-window-compatible durations.The recognition window is 50 frames, or 1.67 seconds; training, validation, and testing are split roughly 4:1:1 by drivers.
- 3.3 Driver behaviour monitoring dataset (dBehaviourMD): dBehaviourMD contains more average video clips per class and broader driver-behaviour coverage than existing behaviour-recognition datasets.It includes the 10 classes shared with StateFarm and AUC DD plus reaching sideways, hands-free, and gear-switching classes.
4 Real-time driver behaviour recognition
The proposed recognition system targets accurate, fast, resource-efficient driver monitoring on CPU-only hardware. It evaluates efficient CNN architectures, RGB/IR/depth fusion, and temporal consensus modules for real-time inference.
- 4.1 Proposed architecture for real-time deployment: The recognition pipeline builds on TSN-style 2D CNN processing with partial batch normalization and data augmentation, while disabling horizontal flipping.The training configuration uses SGD with a 0.001 initial learning rate, momentum 0.9, and dropout 0.5.
- 4.4 Real-time inference: The real-time inference pipeline uses optimized OpenVINO models as its execution engine.The design combines modality fusion, temporal consensus, and optimized inference for deployment-oriented recognition.
- 4.2 Multi-modal fusion: The approach fuses RGB, infrared, and depth modalities at data level, decision level, or both levels.The evaluated fusion strategies include deterministic confidence averaging, Bayesian weighting, and Dempster-Shafer theory.
- 4.3 Consensus module: The consensus module applies temporal max pooling to segment features before flattening and an MLP, and it outperforms averaging and prior MLP consensus modules.The module is intended to capture temporal correlation across sampled video segments.
5 Experiments and results
The experiments compare CNN architectures, temporal consensus modules, and modality-fusion strategies for driver behaviour recognition. Lightweight models and multimodal fusion provide strong accuracy with real-time CPU deployment.
- 5.1 Comparison of 2D CNN based models and 3D CNNs: Around 90% overall accuracy is achievable with the lightweight MobileNet-v2 feature extractor on RGB inputs.The experiments also report a clearer benefit from high-performance base models for RGB.
- 5.1 Comparison of 2D CNN based models and 3D CNNs: Four temporal segments already reach the performance upper bound, and 2D CNNs achieve nearly the accuracy of corresponding 3D CNNs.The comparison uses ImageNet-pretrained 2D models and Kinetics-600-pretrained 3D models.
- 5.2 Impact of different consensus modules on performance: The proposed MaxP-MLP consensus module outperforms averaging and MLP modules in almost all evaluated cases.Max pooling emphasizes significant features in the spatio-temporal feature group.
- 5.3 Multi-modal fusion: Depth generally performs better than RGB and IR, while fusing IR and depth at data level increases accuracy across validation and test analyses.The improvement holds for both four- and eight-segment settings.
- 5.3 Multi-modal fusion: Decision-level fusion of RGB and IR-depth streams improves performance over single streams, whereas Bayesian weighting and DST do not add further benefit.IR and depth are combined at data level because their modalities have pixel-wise correspondence.
- 5.1 Comparison of 2D CNN based models and 3D CNNs: MobileNet-v2 reaches comparable accuracy to dense architectures with substantially fewer trainable parameters, supporting real-time CPU-only deployment.The models were converted to OpenVINO and evaluated on an Intel Core-i9 7940X inference pipeline.
6 Conclusions
The paper presents DMD as a broad multimodal dataset for developing driver monitoring systems and demonstrates its use through real-time behaviour recognition. The extracted dBehaviourMD training set supports enhanced multimodal performance while retaining CPU-only real-time operation.
- 6 Conclusions: DMD is an extensive multimodal dataset for SAE L2–3 driver monitoring, combining three camera views, RGB, depth, IR, and labels for diverse driver features.Its labels include time-lapse actions, body and head pose, blinking patterns, and hands-wheel interactions.
- 6 Conclusions: DMD includes real driving scenes involving distraction, fatigue, and behaviour, with three cameras and three streams per camera.The dataset was created for training and validating driver monitoring systems in automated-driving contexts.
- 6 Conclusions: The dBehaviourMD subset enables real-time driver behaviour recognition with enhanced accuracy across different multimodal fusion approaches on CPU-only platforms.The system targets a real-world application using cost-efficient computing resources.