PaperlayerView exact revision on arXiv

Source-linked AI summary

Driver Gaze Zone Estimation using Convolutional Neural Networks: A General Framework and Ablative Analysis

Sourabh Vora, Akshay Rangesh, Mohan M. Trivedi

arXiv:1802.02690v2cs.CV

TL;DR

The paper addresses the lack of gaze-zone systems that generalize across drivers, cars, perspectives, and scale, a gap relevant to distraction monitoring and vehicle handoff. It finetunes and compares CNNs on naturalistic driving data. The reported analysis favors finetuned SqueezeNet with upper-face input, and the model is also evaluated without training on the Columbia Gaze Dataset.

  • Problem

    Gaze-zone estimation has made progress for personalized systems, but generalization across drivers, cars, perspectives, and scale remains limited, despite gaze's relevance to distraction monitoring and takeover.

    Source: §I · p. 1

  • Method

    The authors finetune four pretrained CNN architectures and compare input strategies on naturalistic driving data, including tests across different drivers and camera conditions.

    Source: §I · p. 1, B. Input pre-processing block · p. 4

  • Results

    SqueezeNet consistently outperforms VGG16, which outperforms ResNet50 across crop regions; all three improve with higher-resolution eye input, and upper-face input performs best.

    Source: A. Analysis of network architectures… · p. 5

  • Takeaways & Limitations

    Without training on the Columbia Gaze Dataset, the best-performing network is used to assess whether it encodes gaze directions across that dataset's pose-gaze configurations.

    Source: §VI · p. 10

Abstract

from arXiv · show

Driver gaze has been shown to be an excellent surrogate for driver attention in intelligent vehicles. With the recent surge of highly autonomous vehicles, driver gaze can be useful for determining the handoff time to a human driver. While there has been significant improvement in personalized driver gaze zone estimation systems, a generalized system which is invariant to different subjects, perspectives and scales is still lacking. We take a step towards this generalized system using Convolutional Neural Networks (CNNs). We finetune 4 popular CNN architectures for this task, and provide extensive comparisons of their outputs. We additionally experiment with different input image patches, and also examine how image size affects performance. For training and testing the networks, we collect a large naturalistic driving dataset comprising of 11 long drives, driven by 10 subjects in two different cars. Our best performing model achieves an accuracy of 95.18% during cross-subject testing, outperforming current state of the art techniques for this task. Finally, we evaluate our best performing model on the publicly available Columbia Gaze Dataset comprising of images from 56 subjects with varying head pose and gaze directions. Without any training, our model successfully encodes the different gaze directions on this diverse dataset, demonstrating good generalization capabilities.

I. INTRODUCTION

The paper targets gaze-zone estimation that generalizes across drivers, cars, perspectives, and scale, motivated by the role of driver attention in safe takeover and distraction monitoring. It applies finetuned CNNs and systematic comparisons to a varied naturalistic driving dataset.

I. INTRODUCTION

  • 80% of crashes and 65% of near crashes involve driver distraction, while secondary-task engagement is associated with slower and more variable takeover responses.The authors argue that ADAS should capture distraction so drivers can be alerted or guided and the handover process can be safer.Source: §I · p. 1
  • Driver gaze provides a cue to distraction, including reduced mirror-checking during secondary tasks and gaze-based attention-buffer and behavior-prediction frameworks.Prior work also modeled driver behavior and maneuver prediction from gaze fixations and transitions.Source: §I · p. 1
  • Personalized gaze-zone systems have advanced, but generalization across drivers, cars, perspectives, and scale remains limited.The authors finetune four ImageNet-trained CNNs and evaluate them on naturalistic data collected over 11 drives by 10 subjects in two cars.Source: §I · p. 1
  • The work contributes an ablative comparison of CNN architectures and input strategies, comparison with prior approaches, and a large naturalistic driving dataset.The supplied table caption identifies selected vision-based gaze-zone studies, but does not provide their results.Source: §I · p. 1, Table I · p. 2

II. RELATED RESEARCH

Prior gaze-zone approaches use remote eye tracking or vision-based head-pose and eye cues, but face, landmark, and pupil processing can limit robustness across driving conditions. The paper situates CNN approaches in this literature and emphasizes cross-driver and camera-variation testing.

II. RELATED RESEARCH

  • Remote eye tracking relies on near-infrared illumination and specialized hardware, making it sensitive to outdoor lighting and costly to integrate and maintain in a moving car.The passage also notes lengthy calibration requirements amid driving vibrations and jolts.Source: §II · p. 2
  • Vision-based head-pose methods can miss adjacent gaze zones distinguished by subtle eye movements, while static-head, eye-only gaze remains difficult to classify.Prior methods combined head pose with temporal dynamics or eye and facial-landmark features to address these challenges.Source: §II · p. 2
  • Recent work combines head pose with eye features, including systems evaluated on a dataset of 40 drivers and approaches classifying gaze into six zones.The literature passage describes landmark, eye-feature, and car-geometry inputs in these approaches.Source: §II · p. 2
  • Complex face, landmark, pupil, and feature pipelines depend on each module working correctly, and hand-crafted eye-landmark features are not robust to driver, car, and seat-position variation.These vulnerabilities become apparent under cross-condition evaluation.Source: §II · p. 2
  • In a 40-driver analysis, face or pupil detection failed in 40% of annotated frames, and high-confidence pruning limited decisions to 1.3 fps in 30 fps video.The cited work uses a confidence ratio threshold greater than 10; the authors note this rate could miss quick mirror or speedometer glances.Source: §II · pp. 2–3
  • Choi et al. used a five-layer CNN for nine gaze zones, but the authors note that cross-driver testing was not conducted to their knowledge.This study separates subjects across training and test sets and varies camera position and field of view to evaluate generalization.Source: §II · p. 3

III. DATASET

The dataset combines naturalistic drives with varied drivers, vehicles, lighting, and camera perspectives to support gaze-zone evaluation across subjects. Training and testing use separate groups of drivers, with forward-facing frames subsampled to balance the data.

III. DATASET

  • Ten subjects drove two cars across 11 drives, while synchronized cameras recorded color video at 30 frames per second from multiple views.Experiments used the rearview-mirror camera; other views helped a human expert label gaze zones.Source: §III · p. 3
  • The seven gaze zones include windshield, left and right, center console, rearview mirror, speedometer, and eyes-closed states.Frames were collected from events separated across time to capture variation in head pose, pupil location, and illumination.Source: §III · p. 3
  • Table III describes annotated, training, and test frame counts for each gaze zone.The supplied passage gives the table’s scope but not its cell values.Source: Table III · p. 3
  • Drive days and times were varied to capture weather and lighting variation, and subjects chose comfortable seat positions.The dataset also varied fields of view, including wide-angle and normal views.Source: §III · pp. 3–4
  • Seven subjects’ drives were used for training and three others for testing, creating a cross-subject evaluation; forward-facing frames were subsampled to balance the dataset.The training set also includes a validation subset of 5%, separated from training images in time.Source: §III · pp. 3–4

IV. PROPOSED METHOD

The proposed method compares combinations of input preprocessing and CNN architecture for gaze-zone estimation. It fine-tunes four ImageNet-trained networks and evaluates their accuracy and inference time.

IV. PROPOSED METHOD

  • Sixteen CNN models pair four input-preprocessing strategies with four architectures, with each model tuned on a validation set.Test-set accuracy and inference time are reported for each combination.Source: §IV · p. 4

A. Network finetuning block

  • The four fine-tuned ImageNet-trained architectures are AlexNet, VGG16, ResNet50, and SqueezeNet.The comparison is intended to examine which network works best and how architectural characteristics affect gaze-zone classification.Source: A. Network finetuning block · p. 4
  • The architectures differ in depth and convolution design; SqueezeNet uses fire modules and has a small model size that the paper identifies as feasible for FPGA and embedded deployment.ResNet uses skip connections, while SqueezeNet and ResNet50 include global average pooling.Source: A. Network finetuning block · p. 4

B. Input pre-processing block

The method tests four image crops and fine-tunes CNNs on the resulting inputs. Training uses resized crops and network-specific classifier changes, with optimization choices constrained by overfitting and data variation.

B. Input pre-processing block

  • Four input crops are compared: driver surround, detected face, face with added context, and the upper half of the face.The surround crop tests whether face detection can be skipped; added context tests whether surroundings help locate the driver’s head.Source: B. Input pre-processing block · p. 4, Figure 4 · p. 5
  • The extracted crops are resized to 224x224 or 227x227 according to network requirements, then mean-subtracted.The face-context crop extends the detected face bounding box in all directions.Source: B. Input pre-processing block · p. 4

C. Training

  • AlexNet, VGG16, and ResNet50 receive a new seven-neuron fully connected output layer, while SqueezeNet’s final convolution layer is reduced to seven kernels.The networks are fine-tuned end-to-end using a low learning rate.Source: C. Training · pp. 4–5
  • Networks were trained for 50 epochs with Adam and adaptive learning rates; training beyond 50 epochs began to overfit.Image flipping and rotation were omitted because they could change labels or create unrealistic images; illumination augmentation was omitted because the dataset already varied in lighting.Source: C. Training · p. 5

V. EXPERIMENTAL ANALYSIS & DISCUSSION

The experiments report macro- and micro-average accuracy alongside an N-class confusion matrix. The reported system performance uses a face detector with few observed training-set false detections, which were not checked on the test set.

V. EXPERIMENTAL ANALYSIS & DISCUSSION

  • Macro- and micro-average accuracy are the first two evaluation metrics, and the N-class confusion matrix uses rows for true zones and columns for estimated zones.N is the number of gaze zones.Source: §V · p. 5
  • Fig. 5 illustrates subtle eye differences when the driver looks toward three different gaze zones.Source: Figure 5 · p. 5
  • The face detector had fewer than 0.25% false detections on training data, but test-set false detections were not checked.The paper therefore reports the system’s performance without a test-set false-detection check.Source: §V · p. 5

A. Analysis of network architectures and different image crop regions

Across crop regions, finetuned SqueezeNet on upper-face images gives the strongest cross-subject result. The comparisons also link performance to the resolution of eye features and the architectures’ ability to capture subtle gaze cues.

A. Analysis of network architectures and different image crop regions

  • 95.18% accuracy: finetuned SqueezeNet performs best on upper-half-face images, and all three other compared networks also benefit from higher-resolution eye images.The reported comparisons cover sixteen network-and-crop combinations.Source: A. Analysis of network architectures… · p. 5, Table IV · p. 6
  • AlexNet performs especially poorly on Face-embedded FoV images, where small pupil or eyelid movements are difficult to distinguish from a large, low-resolution eye image.The authors attribute this partly to AlexNet’s 11 × 11 first-layer kernel and stride of 4; the model handles large head movements better than subtle eye movements.Source: A. Analysis of network architectures… · p. 5
  • SqueezeNet and VGG16 outperform AlexNet on subtle gaze distinctions, consistent with their smaller convolution kernels capturing slight pupil or eyelid movements.SqueezeNet uses 3 × 3 and 1 × 1 kernels; VGG16 uses 3 × 3 kernels with stride 1.Source: A. Analysis of network architectures… · pp. 5–6
  • ResNet50 is slightly less accurate than SqueezeNet and VGG16 across crop strategies, possibly because of its 7 × 7 first-layer kernel or limited fine-tuning data for a 50-layer network.Source: A. Analysis of network architectures… · p. 6

B. Comparison of our CNN based model with some current state of the art models

The cross-driver comparison favors the CNN over the Random Forest, whose hand-crafted features are sensitive to camera and driver variation. A separate temporal split shows why same-driver evaluation can give a misleadingly high accuracy.

B. Comparison of our CNN based model with some current state of the art models

  • Tawari et al. ’s Random Forest was implemented on the authors’ dataset to enable a fair comparison with the CNN.Source: B. Comparison of our CNN based model… · p. 6
  • 26.42% margin: the finetuned SqueezeNet CNN substantially outperforms the Random Forest on this dataset.Table V reports the SqueezeNet confusion matrix and Table VI the Random Forest confusion matrix.Source: B. Comparison of our CNN based model… · p. 6, Table V · p. 6, Table VI · p. 6
  • The Random Forest uses head pose, gaze angles, facial landmarks, and eyelid area, features that are vulnerable to camera position, face orientation, occlusion, and differences in drivers or seating.The authors state that these factors limit the Random Forest’s generalization on their dataset.Source: B. Comparison of our CNN based model… · p. 6
  • 98.7% accuracy on a temporal split falls to 82.5% on different drivers, indicating that same-driver evaluation can mask driver-specific overfitting.Choi et al. reported 95% on their own dataset, but the authors note that cross-driver testing was not conducted there.Source: B. Comparison of our CNN based model… · p. 6

C. How can we get away without face detection?

Class activation maps show that SqueezeNet localizes driver eyes and can use either one or both eyes, including when an eye is occluded. Higher-resolution Face-embedded FoV input also supports gaze prediction without a separate face-detection step.

C. How can we get away without face detection?

  • Class activation maps localize on the driver’s eyes, even for Face-embedded FoV input where the eyes occupy a small image area and no eye or face boxes were provided.Figure 6 shows maps for seven gaze zones and four crop regions.Source: C. How can we get away without face… · p. 7, Figure 6 · p. 8
  • When one eye is occluded, the network can base its decision on the visible eye; when both eyes are visible, it can use both.The behavior is described in the examples and illustrated in Figure 7.Source: C. How can we get away without face… · p. 7, Figure 7 · p. 9
  • The authors hypothesize that upper-face images perform better because their higher eye resolution preserves finer iris and eyelid features; this motivated testing higher-resolution Face-embedded FoV images.Source: C. How can we get away without face… · pp. 7, 9
  • Finetuning the network on 625 × 625 higher-resolution Face-embedded FoV images achieved 92.13% macro-average accuracy, though performance remained below training on upper-half face images.Skipping separate face detection requires only one forward pass, enabling real-time operation; predictions are not affected by inaccurate face detections.Source: C. How can we get away without face… · p. 9

D. Inference time for gaze estimation using different architectures

Inference speed varies by architecture, with AlexNet and SqueezeNet faster than VGG16 and ResNet50. The standalone high-resolution SqueezeNet system runs in real time, while the best-performing upper-face model’s reported speed includes face detection.

D. Inference time for gaze estimation using different architectures

  • AlexNet and SqueezeNet have faster single-image forward passes than VGG16 and ResNet50, making finetuned SqueezeNet the authors’ choice for its combination of speed and accuracy.Inference times were measured using Caffe’s Matlab interface on a Titan X GPU.Source: D. Inference time for gaze estimation… · p. 9, Table XI · p. 9
  • 166.7 Hz: the 625 × 625 Face-embedded FoV SqueezeNet system runs standalone in real time; the best-performing upper-face model runs at 16 Hz with the cited face detector.The authors note that the detector could be replaced by another real-time detector or a detector-tracker combination.Source: D. Inference time for gaze estimation… · p. 9

VI. GENERALIZATION ON THE COLUMBIA GAZE DATASET

The authors test whether their best model encodes gaze directions on Columbia Gaze images without training, despite differences in camera pose and subjects. Predicted zones vary with gaze direction, but the dataset does not cover every gaze zone.

VI. GENERALIZATION ON THE COLUMBIA GAZE DATASET

  • Normalized entropy summarizes the distribution of predicted gaze zones across subjects for each configuration; lower entropy indicates that gaze direction was encoded successfully.The evaluation uses configurations selected to resemble how drivers look toward gaze zones in a car.Source: §VI · p. 10
  • Across four configurations with substantially varying head pose, the model still predicted “forward” for subjects looking to the right of the camera.The authors interpret this as evidence that the network encoded gaze direction, not just head pose.Source: §VI · p. 10
  • Across 19 configurations where subjects looked to the right of the camera and vertical gaze was 0° or 10°, the network predicted ’forward’ as the gaze zone for more than 70% of the subjects.For rightward gaze at −10° vertical gaze, the network predicted ’Speedometer’; for leftward gaze at −10° vertical gaze, it predicted ’Radio’ for the majority of subjects.Source: §VI · p. 10
  • The dataset did not test majority predictions for “Right,” “Left,” or “Eyes Closed,” because its horizontal gaze range was limited and it contained no closed-eye images.Its grid spanned only ±15° horizontally.Source: §VI · p. 10
  • Without Columbia-dataset training, the model encoded gaze directions across new subjects despite substantial camera-pose differences.The 56 subjects varied in ethnicity, and many wore prescription glasses.Source: §VI · p. 10

VII. CONCLUDING REMARKS

The authors report a cross-subject accuracy of 95.18% for a fine-tuned SqueezeNet using the upper half of the face, and successful gaze-direction encoding on Columbia Gaze without training. They identify finer gaze zones and temporal context as future work, with continuous labels posing a challenge for temporal modeling.

VII. CONCLUDING REMARKS

  • On the collected naturalistic driving dataset, fine-tuned SqueezeNet trained on upper-half-face images achieved 95.18% accuracy, the best result across the evaluated CNN architectures and image-crop regions.The standalone system required no face detection and achieved 92.13% accuracy while running at 166.7 Hz on a GPU.Source: §VII · p. 10
  • The best model encoded gaze directions on the diverse Columbia Gaze Dataset without training on it.The dataset includes 56 subjects with varied head pose and gaze direction.Source: §VII · p. 10, Figure 8 · p. 11
  • The authors identify adding gaze zones for finer estimates and using temporal context to capture transitions between zones as future work.The current implementation predicts each frame independently.Source: §VII · p. 10
  • Figure 8 shows sample pose-gaze configurations alongside predicted-zone histograms and normalized entropy.Its labels identify the subject-percentage histograms and sample images for each configuration.Source: Figure 8 · p. 11, §VII · p. 11
  • Temporal modeling will require continuous gaze-zone labels, whereas the current labels are for discrete events separated across time.The authors identify obtaining continuous labels as a challenge for implementing an LSTM.Source: §VII · pp. 10–11

Related papers

Loading 1802.02690v2…