Source-linked AI summary
Real-Time Face & Eye Tracking and Blink Detection using Event Cameras
Cian Ryan, Brian O Sullivan, Amr Elrasad, Joe Lemley, Paul Kielty, Christoph Posch, Etienne Perot
TL;DR
Event cameras offer asynchronous sensing with high temporal resolution, but event-based driver monitoring has received limited attention and blink analysis needs fine temporal detail. The paper combines a recurrent face-and-eye tracker, a synthetic annotated event dataset, and event-based blink analysis. It reports that blink events have a temporal structure that can be decomposed, and demonstrates the methods’ applicability in unconstrained driving environments.
Problem
Event-based driver monitoring has received limited attention, while precise blink analysis requires temporal detail beyond typical fixed-rate camera sampling.
Source: §1 · pp. 2–3Method
The paper trains a fully convolutional recurrent face-and-eye tracker with synthetic annotated event data and detects blinks from event distributions in tracked eye regions.
Source: §1 · pp. 2–3, §3 · p. 5Results
The authors report a bimodal event distribution during blinks that permits decomposition into blink components, and demonstrate method efficacy in unconstrained driving environments.
Source: §5.1 · p. 17, §6 · p. 17Takeaways & Limitations
Event-camera DMS can analyze blink components at high temporal resolution while tracking faces and eyes at a rate adapted to scene dynamics.
Source: §1 · p. 3Evaluation is constrained by the lack of annotated event-camera face datasets, limiting statistically significant testing.
Source: §5.1 · p. 14
Abstract
from arXiv · showhide
Event cameras contain emerging, neuromorphic vision sensors that capture local light intensity changes at each pixel, generating a stream of asynchronous events. This way of acquiring visual information constitutes a departure from traditional frame based cameras and offers several significant advantages: low energy consumption, high temporal resolution, high dynamic range and low latency. Driver monitoring systems (DMS) are in-cabin safety systems designed to sense and understand a drivers physical and cognitive state. Event cameras are particularly suited to DMS due to their inherent advantages. This paper proposes a novel method to simultaneously detect and track faces and eyes for driver monitoring. A unique, fully convolutional recurrent neural network architecture is presented. To train this network, a synthetic event-based dataset is simulated with accurate bounding box annotations, called Neuromorphic HELEN. Additionally, a method to detect and analyse drivers eye blinks is proposed, exploiting the high temporal resolution of event cameras. Behaviour of blinking provides greater insights into a driver level of fatigue or drowsiness. We show that blinks have a unique temporal signature that can be better captured by event cameras.
1. Introduction
Event cameras’ asynchronous, high-temporal-resolution sensing motivates driver monitoring, but event-based DMS has received limited attention. The paper introduces a recurrent face-and-eye tracker, a synthetic annotated dataset, and blink analysis at task-specific temporal rates.
1. Introduction
- The paper targets limited event-based DMS research and the challenge of detecting slow-moving or static objects that generate sparse events.GR-YOLO is described as preserving information from previously detected locations when little or no new information is available.Source: §1 · pp. 2–3
- The proposed GR-YOLO uses a fully convolutional recurrent architecture to detect and track faces and eyes, addressing the need for temporal connections in sparse event data.It is based on YOLOv3-tiny and integrates a gated recurrent unit layer.Source: §1 · p. 2
- Blink analysis tracks event distributions in detected eye regions every 5 ms and identifies abnormal event spikes, at an equivalent of 200 fps.Event cameras allow face and eye tracking at an adaptive rate rather than a fixed frame rate.Source: §1 · p. 3
- Neuromorphic-Helen simulates event streams from annotated Helen images, mapping facial landmarks into event space to create face and eye bounding boxes.The method can incorporate other labeled datasets, including near-infrared data.Source: §1 · p. 3
- The contributions include a synthetic training dataset, event-based face and eye tracking, blink-pattern analysis, and DMS tasks operating at different adaptive rates.Blink analysis is performed at an equivalent of 200 fps, while face and eye detection rates depend on scene dynamics.Source: §1 · pp. 3–4
2. Related Work
Prior work addresses event-based object detection and blink analysis, but the reviewed approaches leave challenges around temporal tracking, motion speed, or isolating eye events. Conventional blink methods also require high frame rates or accurate facial landmarks.
2.1 Object Detection and Tracking
- Event-based object detection includes hand-crafted blink-linked tracking, single-image YOLO with correlation tracking, and neural detectors using leaky event surfaces [Lenz et al., 2018; Jiang et al., 2020; Cannici et al., 2019a].Other approaches pool features over time for slower motion or detect moving regions, while multi-rate event integration has also been proposed [Jia Li et al., 2017; Cannici et al., 2019b; Ghosh et al., 2014; Joubert et al., 2019].Source: §2.1 · p. 4
- Reconstructed intensity images enable conventional object detectors, but reconstruction can be computationally expensive and may diminish event-camera advantages [Rebecq et al., 2019].The cited DMS concern is that many events may arise outside the facial region of interest.Source: §2.1 · p. 4
- The proposed GR-YOLO follows feature-level video detection, which integrates image features across time rather than associating detections from separate frames [Liu & Zhu, 2018; Zhu et al., 2017].The related-work discussion contrasts this with box-level tracking-by-detection [Han et al., 2016; Ning et al., 2017].Source: §2.1 · p. 5
2.2 Blink Detection and Pattern Analysis
- Conventional blink analysis uses eyelid landmarks, but precise blink parameters require over 100 fps, beyond the cited typical DMS rate of 30–60 fps.Blink duration, reopening delay, interval, and closure speed are identified as relevant to drowsiness estimation.Source: §2.2 · p. 5
- Event-based blink methods include full-image event-pattern correlation, event filtering and spike detection, and hybrid eye tracking based on blink-induced eye-ellipse deformation [Lenz et al., 2018; Chen, Hong, et al., 2020; Angelopoulos et al., 2020].The reviewed methods respectively face interference from motion elsewhere in the image, use no eye tracking, or rely on a hybrid frame-and-event system.Source: §2.2 · p. 5
3. Dataset
Neuromorphic Helen (N-Helen) is a synthetic event dataset generated from annotated Helen images through simulated camera motion and event conversion. The authors use ESIM and a fixed-duration event window to align simulated events with frame-based annotations; this version is static, though on-the-fly simulation is described as an alternative.
3. Dataset
- N-Helen generates event-based video and face and eye bounding boxes from Helen images by applying simulated 6-DOF homographic camera motion to images and facial landmarks.The approach maps facial landmarks to event space and uses them to obtain bounding boxes for each frame.Source: §3 · pp. 5–6
- ESIM simulates events with positive and negative contrast thresholds sampled from ~N(0.2, 0.05) and a 1 ms refractory period, then aligns events preceding each frame timestamp to annotations.The paper illustrates conversion from RGB images to events and accumulated event representations.Source: §3 · p. 6, Figure 1 · p. 6
- The authors generated one static N-Helen dataset; on-the-fly simulation could vary contrast thresholds and motion parameters to produce unseen inputs each epoch.The paper notes that a single image can generate many unique inputs under this alternative.Source: §3 · p. 6
4. Methodology
The method combines a recurrent YOLO detector for faces and eyes with a separate event-based blink detector. It evaluates the detector on three test videos and analyzes blink dynamics at high temporal resolution.
4.1 Event Representation
- Training uses a fixed-duration event window, with a single-channel event accumulation performing similarly to a five-bin voxel representation at lower computational cost.Events are clipped between -10 and +10 to remove outliers that affected performance.Source: §4.1 · p. 7
4.2 Network Architecture
- The fully convolutional GR-YOLO places a GRU after convolutional and pooling layers to combine current-frame features with recurrent information before detection.Its YOLO layers predict face and eye boxes at two scales using three anchors per cell.Source: §4.2 · pp. 7–9
- YOLO transforms anchor-relative width and height offsets into box dimensions and uses sigmoid-transformed offsets for center coordinates and objectness.The model filters low-objectness boxes before applying non-maximum suppression.Source: §4.2 · pp. 8–9
4.3 Training Details
- Zero-input sequence augmentation encourages the network to retain prior face and eye positions when current event inputs provide little information.Random rotations and horizontal flips are also used for geometric invariance.Source: §4.3 · p. 10
4.4 Blink Detection
- Blink detection thresholds mean polarity in 5 ms windows, filters likely horizontal pupil motion using vertical event spread, and decomposes detected blinks into phases.A typical blink produces event peaks during eyelid closing and opening, with the upward movement typically lasting longer than closing.Source: §4.4 · pp. 10–11, §5.1 · p. 17
5.1 Face/Eye Detection and Tracking
- GR-YOLO performs worse with larger head movements but continues to perform well in test video 3 despite limited face information.The authors attribute test 3 performance to zero-input augmentation and note that post-detection tracking was not used.Source: Table 2 · p. 13, §5.1 · p. 13
- The evaluation is preliminary because no diverse event-camera benchmark with annotated face boxes is available, limiting statistically significant testing.The face and eye detector was tested on three manually annotated videos, using a subject absent from training.Source: §5.1 · pp. 12, 14, §4.4 · p. 12
6. Conclusion
The paper concludes that event-camera methods are applicable to driver monitoring, reporting efficacy in unconstrained driving environments and pointing to further potential applications.
6. Conclusion
- The proposed face-and-eye tracking and blink methods demonstrate applicability to DMS in unconstrained driving environments.They were tested qualitatively and quantitatively on manually collected test datasets; the authors note that benchmark datasets are unavailable.Source: §6 · p. 17
- Event cameras offer high temporal resolution and adaptive frame rates, while their response to motion supports a more direct approach to blink detection.The authors state these characteristics support advanced DMS features beyond fixed-frame-rate solutions.Source: §6 · p. 17
- Potential event-camera DMS applications include low-latency eye-gaze tracking, saccadic eye-motion detection, and collision analysis.Source: §6 · p. 18