Source-linked AI summary
Deep Pictorial Gaze Estimation
Seonwook Park, Adrian Spurr, Otmar Hilliges
TL;DR
Single-eye gaze estimation is difficult because the eyeball center is not observable in 2D images. The paper predicts an intermediate pictorial representation before gaze direction, reporting higher accuracy than prior methods and robustness across conditions.
Problem
Estimating 3D gaze direction from a natural eye image is challenging because the eyeball center is unobservable in 2D data.
Method
The network regresses an anatomically inspired pictorial representation of gaze, called gazemaps, before estimating the final gaze direction with intermediate supervision.
Results
18%: the method reduces gaze error on MPIIGaze versus the state of the art, with improvements also reported on Columbia and EYEDIAP.
Takeaways & Limitations
A single eye image can support gaze estimation using an explicit gaze representation, including under variation in gaze, head pose, and image quality.
Takeaways & Limitations
The representation assumes a spherical eyeball and circular iris, using average eyeball and iris diameters.
Abstract
from arXiv · showhide
Estimating human gaze from natural eye images only is a challenging task. Gaze direction can be defined by the pupil- and the eyeball center where the latter is unobservable in 2D images. Hence, achieving highly accurate gaze estimates is an ill-posed problem. In this paper, we introduce a novel deep neural network architecture specifically designed for the task of gaze estimation from single eye input. Instead of directly regressing two angles for the pitch and yaw of the eyeball, we regress to an intermediate pictorial representation which in turn simplifies the task of 3D gaze direction estimation. Our quantitative and qualitative results show that our approach achieves higher accuracies than the state-of-the-art and is robust to variation in gaze, head pose and image quality.
1 Introduction
Appearance-based gaze estimation aims to infer 3D gaze from a single natural eye image, but the task is difficult because the eyeball center is unobservable and eye images vary substantially. The paper addresses this with a gaze-specific pictorial representation and sequential neural architecture.
- Single-camera gaze estimation supports assistive technologies, human-computer interaction, visual attention analysis, consumer research, AR, and VR.
- Directly regressing gaze angles from eye images is difficult because of inter-subject appearance variation and self-occlusions.
- The proposed architecture first estimates a gaze-specific pictorial representation, then predicts 3D gaze direction from that intermediate image.
- Intermediate supervision on the pictorial representation is experimentally shown to improve accuracy while the approach uses a fully convolutional architecture.
- The pictorial representation depicts the eyeball as a circle and the iris as an ellipse, whose relative positioning changes with gaze direction.
- 18%: the reported gaze-error reduction versus the state of the art on MPIIGaze, alongside evaluations on EYEDIAP and Columbia.
2 Related Work
Prior gaze-estimation methods include handcrafted, model-based, and appearance-based approaches, with CNN methods increasingly targeting unconstrained, person-independent settings. This paper introduces gazemaps as an explicit gaze-specific prior incorporated through intermediate supervision.
- Feature-based methods use handcrafted eye-image features, whereas model-based methods fit a known 3D eye model by minimizing an energy.
- Appearance-based methods learn directly from raw eye images but must handle illumination changes, occlusions, head motion, and eye decorations.
- Person-independent gaze estimation avoids user calibration but requires larger, more diverse training data and greater computational cost.
- CNN approaches for unconstrained gaze estimation mainly differ in their input modalities, including eye images and head-pose angles.
- The paper introduces gazemaps, a pictorial gaze representation incorporated into a deep network through intermediate supervision and fully convolutional processing.
- Unlike auxiliary-task methods that impose implicit priors, gazemaps explicitly encode a gaze-specific prior within the network architecture.
3 Method
The method replaces direct gaze regression with gazemaps: two boolean maps representing projected eyeball and iris geometry, followed by gaze-direction regression. A fully convolutional network predicts gazemaps, while additional layers estimate 3D gaze direction with intermediate supervision.
- 3.1 Pictorial Representation of 3D Gaze: Gazemaps are two boolean maps formed by projecting simple eyeball and iris models to represent 3D gaze direction.The eyeball is modeled as a circle and the iris as an ellipse whose position changes with gaze.
- 3.1 Pictorial Representation of 3D Gaze: The conventional mapping from an eye image x directly to gaze g is decomposed into j: x → m and k: m → g.The authors hypothesize that learning these two mappings is simpler than learning the direct mapping f: x → g.
- 3.1 Pictorial Representation of 3D Gaze: Gazemaps reduce each input image to a minimal normalized representation before gaze estimation from that representation.The representation is intended to retain information necessary for gaze estimation while making the mapping from gazemaps to gaze direction simple.
- 3.1 Pictorial Representation of 3D Gaze: The representation assumes a spherical eyeball and circular iris, with projected eyeball diameter 2r = 1.2n and iris shape determined by gaze angles.The iris is drawn as an ellipse with major-axis diameter r and minor-axis diameter r |cos θ cos φ|.
- 3.2 Neural Network Architecture: The network uses a fully convolutional hourglass component for eye-image-to-gazemap regression and a DenseNet component for gazemap-to-gaze regression.Intermediate supervision applies a cross-entropy loss to predicted and ground-truth gazemap pixels, with α set to 10^-5.
4 Implementation
The implementation combines stacked hourglass modules for gazemap prediction with DenseNet regression, using selective intermediate supervision and training procedures designed for stable optimization. Qualitative examples compare learned representations with and without gazemap loss.
- 4 Implementation: The architecture contains fully convolutional Hourglass and regressive DenseNet parts.The Hourglass network predicts gazemaps, while DenseNet performs regression from gazemaps to gaze direction.
- 4.1 Hourglass Network: The input images are 150×90, and the network refines 64 feature maps of size 75×45.Half-scale feature maps are produced by an initial 7-filter convolution with stride 2, followed by batch normalization, ReLU, and residual modules.
- 4.1 Hourglass Network: Three hourglass modules process the images, but gazemap supervision is applied only after the last module.The design allows multi-scale processing before features become aligned with the final gazemap representation.
- 4.1 Hourglass Network: Intermediate gazemaps and preceding feature maps are concatenated back into the network before the next hourglass module.This follows the concatenation pattern used in the original Hourglass architecture.
- 4.3 Training Details: Training uses Adam with batch size 32, learning rate 0.0002, L2 regularization coefficient 10^-4, and 20 epochs.Image translation and scaling augmentations are applied, and the learning rate is multiplied by 0.1 after every 5k gradient updates.
- 4.3 Training Details: Without intermediate supervision, the network still isolates iris regions and learns similar gaze representations across participants.With gazemap loss, the pictorial representation is more consistent, while the hourglass network can account for occlusions.
5 Evaluations
The evaluations test gazemap supervision, cross-person generalization, and robustness across datasets, image quality, head pose, and gaze direction. The proposed approach improves gaze estimation over competitive baselines, including under low-quality inputs.
- Evaluation setup: Cross-person evaluations use leave-one-person-out testing on MPIIGaze and 5-fold validation on Columbia and EYEDIAP.MPIIGaze trains on 14 users and tests on the held-out user; Columbia and EYEDIAP provide different participant diversity and image-quality conditions.
- Gazemap supervision: Gazemap supervision generally improves performance, with especially pronounced gains for DenseNet growth rate k = 32.Table 1 compares cross-person gaze estimation errors with and without Lgazemap.
- Representation analysis: The learned representation can isolate iris-related structure but may also encode irrelevant factors such as glasses edges without explicit gazemap supervision.Explicit gazemaps impose a prior that discourages these unhelpful details.
- Cross-person performance: 1.0° improvement over the state-of-the-art reduces MPIIGaze error from 5.5° to 4.5° (18.2%).The result uses single-eye input and 0.7M trainable parameters versus 90M for the compared architecture.
- Cross-person performance: The model improves over AlexNet by 0.4° on Columbia and over VGG-16 by 0.9° on EYEDIAP, an 8% improvement.EYEDIAP performance is evaluated without head-pose estimation and relies only on single-eye input despite low-resolution, low-quality images.
- Robustness analysis: Robustness analysis evaluates mean angular error across head pose, gaze direction, contrast, and sharpness variations, with improvements over prior work for varying image quality.The reported robustness analysis also considers ranges of head pitch and gaze yaw angles.
6 Conclusion
The paper concludes that gazemaps provide an explicit, anatomically inspired prior for appearance-based gaze estimation. The approach improves accuracy and robustness across datasets and challenging viewing conditions, while suggesting alternative representations and synthetic-data training as future directions.
- Conclusion: Gazemaps are an explicit, anatomically inspired prior for neural-network gaze estimation from eye images.The architecture predicts gazemaps before estimating final gaze direction and uses intermediate supervision.
- Conclusion: The approach improves accuracy by up to 18% on MPIIGaze and also improves results on Columbia and EYEDIAP against competitive baselines.The conclusion reports gains despite comparisons involving models with substantially more parameters or additional input modalities.
- Conclusion: The final model is more robust to extreme head poses, extreme gaze directions, and poor image quality than prior work.These robustness factors are evaluated as part of the paper’s analysis.
- Future work: Future work includes alternative pictorial representations, alternative gazemap-prediction architectures, and synthesized gaze directions for unsupervised training.The proposed synthetic training direction uses corresponding gazemaps to improve the gaze regression function.
A Baseline Architectures
The baseline comparison uses standard VGG-16 and AlexNet architectures that have served as common references in appearance-based gaze estimation. Training uses Adam with regularization, learning-rate decay, and slight geometric augmentation.
- Baseline architectures: VGG-16 and AlexNet serve as baseline architectures for comparison with the proposed gaze-estimation model.The baselines are described as standard or commonly used architectures in prior gaze-estimation work.
- Training: Both baseline models use batch size 32, learning rate 5 × 10^-5, and L2 regularization coefficient 10^-4 with Adam optimization.The learning rate is multiplied by 0.1 every 5,000 training steps.
- Training: Training applies slight augmentation through image translation and scale changes.This augmentation is reported alongside the optimization schedule and regularization settings.
B Image metrics
The paper describes the image metrics used for robustness plots concerning image quality.
- Image metrics: Image metrics are used to analyze robustness plots for image quality in Figures 6e and 6f.The passage identifies the purpose of the metrics but does not enumerate them.
B.1 Image contrast
Root mean contrast measures image contrast as the standard deviation of pixel intensities. The calculation uses pixel values and the image’s average intensity.
- Root mean contrast is defined as the standard deviation of pixel intensities.
- The image contrast calculation uses the pixel intensity Iij at location (i, j).
- The calculation centers pixel intensities around the image-wide average intensity ¯I.
B.2 Image sharpness
Image sharpness is measured after convolving the image with a Laplacian. The standard deviation of the convolved image provides the sharpness metric.
- Image sharpness is computed from the image after convolution with a Laplacian.The Laplacian approximates the image’s second derivative.
- The Laplacian convolution uses a mask to approximate the second derivative.
- The standard deviation of the convolved image IL is used as the image sharpness metric.