Source-linked AI summary
It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation
Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling
TL;DR
Appearance-based gaze estimation has traditionally relied on eye images, leaving the value of full-face information and its applicability to 3D estimation open. The paper proposes a face-only CNN with learned spatial weights and evaluates it on 2D and 3D tasks. It reports state-of-the-art improvements of 14.3% on MPIIGaze and 27.7% on EYEDIAP for person-independent 3D estimation.
Problem
Previous appearance-based methods used eye information, while the benefit of a face-only approach for 2D and 3D gaze estimation remained open.
Method
The method uses only the full face image and applies learned spatial weights to CNN feature maps to encode different facial regions.
Results
14.3% and 27.7% improvements over the state of the art were achieved for person-independent 3D gaze estimation on MPIIGaze and EYEDIAP, respectively.
Takeaways & Limitations
Full-face appearance-based gaze estimation is more robust to variation from extreme head pose, gaze direction, and illumination than eye-only and multi-region methods.
Takeaways & Limitations
The 2D formulation assumes a fixed target plane and fixed camera intrinsic parameters, so trained regressors cannot directly transfer to different cameras without treatment of projection differences.
Abstract
from arXiv · showhide
Eye gaze is an important non-verbal cue for human affect analysis. Recent gaze estimation work indicated that information from the full face region can benefit performance. Pushing this idea further, we propose an appearance-based method that, in contrast to a long-standing line of work in computer vision, only takes the full face image as input. Our method encodes the face image using a convolutional neural network with spatial weights applied on the feature maps to flexibly suppress or enhance information in different facial regions. Through extensive evaluation, we show that our full-face method significantly outperforms the state of the art for both 2D and 3D gaze estimation, achieving improvements of up to 14.3% on MPIIGaze and 27.7% on EYEDIAP for person-independent 3D gaze estimation. We further show that this improvement is consistent across different illumination conditions and gaze directions and particularly pronounced for the most challenging extreme head poses.
1. Introduction
The paper asks whether full-face images can replace eye-only inputs for appearance-based 2D and 3D gaze estimation. It proposes a spatial-weights CNN and reports strong person-independent 3D gains.
- Previous appearance-based methods primarily used information from one or both eyes.
- Full-face gaze estimation remains open because facial regions may encode head pose and illumination information beyond the eyes.
- The paper evaluates a face-only approach for both 2D and 3D appearance-based gaze estimation.
- The proposed CNN takes the full face image as input and directly regresses 2D or 3D gaze estimates.
- 4.8° person-independent 3D gaze estimation accuracy on MPIIGaze improves 14.3% over the state of the art.
2. Related Work
Earlier appearance-based gaze estimators largely relied on eye images, while multi-region methods added face context. This work extends full-face input to end-to-end 2D and 3D estimation with learned spatial weighting.
- Appearance-Based Gaze Estimation: Appearance-based methods directly regress gaze direction from image appearance rather than geometric eye and face models.
- Multi-Region Gaze Estimation: Most prior methods used a single eye image, with fewer methods using both eyes or a region covering both eyes.
- Multi-Region Gaze Estimation: Krafka et al.'s multi-region method combined individual eye images, a face image, and a face grid for 2D gaze estimation.
- Multi-Region Gaze Estimation: This paper is presented as the first to explore full-face information for both 2D and 3D appearance-based gaze estimation.
- Spatial Encoding in CNNs: The spatial weights mechanism suppresses noisy regions and enhances contributions from low-activation regions in full-face feature maps.
3. Gaze Estimation Tasks
The paper distinguishes 2D screen-location regression from 3D gaze-vector regression. The 3D formulation is more general across devices, while normalization handles camera and person variation.
- The paper investigates full-face appearance-based gaze estimation under both 2D and 3D task formulations.
- 3.1. 2D Gaze Estimation: 2D estimation regresses an input image I to a two-dimensional on-screen gaze location p.
- 3.1. 2D Gaze Estimation: 2D regression usually requires head pose or face bounding-box information because gaze location depends on head position.
- 3.1. 2D Gaze Estimation: A fixed camera with fixed intrinsic parameters is an assumption of the 2D formulation, limiting direct transfer across cameras.
- 3.2. 3D Gaze Estimation: 3D estimation regresses an input image I to a 3D gaze vector g, usually defined as a unit vector from a 3D reference point.
- 3.2. 3D Gaze Estimation: 3D gaze vectors can be converted to screen locations by projecting them into the camera coordinate system and intersecting the target plane.
- 3.2. 3D Gaze Estimation: Perspective normalization uses M = SR to rotate and scale inputs into a normalized space with fixed camera parameters and reference-point location.
4. Full-Face Gaze Estimation with a Spatial Weights CNN
The method estimates 2D or 3D gaze directly from full-face images using a CNN with learned spatial weights that suppress irrelevant regions and enhance useful facial information.
- Full-face approach: The spatial weights CNN learns gaze from the full face rather than restricting input to the eye region.The model targets both 2D screen-coordinate gaze and 3D normalized gaze-vector estimation.
- Spatial weights mechanism: The mechanism suppresses background or otherwise unhelpful regions while retaining and enhancing information from facial regions that may support gaze estimation.Unlike spatial dropout, it applies continuous weights and preserves information from different regions.
- Spatial weights mechanism: Spatial weighting generates a H × W map from the final convolutional activations and multiplies it element-wise with every feature channel.The weighted tensor is then passed to subsequent layers for gaze regression.
- Spatial weights mechanism: The learned weights are shared across feature channels, so their spatial values correspond directly to facial regions in the input image.This ties the weighting pattern to the face’s spatial layout rather than to individual channels.
- Network and training: The full-face model uses an AlexNet-based convolutional backbone followed by task-specific linear regression for 2D or 3D gaze outputs.The convolutional layers are initialized with ImageNet pretraining and the whole network is fine-tuned.
5. Evaluation
Experiments on MPIIGaze and EYEDIAP evaluate 2D and 3D gaze estimation, showing that full-face models outperform eye-only baselines and that spatial weighting is especially useful under challenging conditions.
- Evaluation setup: MPIIGaze evaluation used leave-one-person-out cross-validation across all 15 participants, while EYEDIAP used five-fold cross-validation over 14 participants.MPIIGaze used manually annotated facial landmarks for normalization; EYEDIAP used dataset-provided head-pose and eye-centre annotations.
- 2D Gaze Estimation: Full-face methods significantly outperformed the single-eye baseline for 2D gaze estimation on MPIIGaze.The single-face model was competitive with iTracker variants, and the spatial weights network improved performance further.
- 2D Gaze Estimation: The spatial weights network achieved a statistically significant 7.2% improvement over the second-best single-face model on MPIIGaze 2D estimation.The reported paired t-test significance was p < 0.01.
- 3D Gaze Estimation: The proposed model achieved the best 3D gaze estimation performance on EYEDIAP.Overall EYEDIAP performance was worse than MPIIGaze, which the authors associate with lower resolution and fewer training images.
- Head Pose and Facial Appearance: Eye-blocked full-face images produced 5.5-degree angular error, yet outperformed baseline estimators based directly on head pose.This indicates that facial appearance contributes information beyond head pose for inferring gaze direction.
- Importance of Different Facial Regions: The proposed model performed better than the two-eye model across all MPIIGaze illumination conditions.Under strong directional lighting, wider regions around the eyes on the brighter side received greater importance.
- Importance of Different Facial Regions: Different facial regions became important for different gaze directions, with non-eye regions gaining importance as gaze became more extreme.The eye region was most important for straight-ahead gaze.
6. Conclusion
The study proposes a spatial-weights CNN for full-face appearance-based gaze estimation, leveraging information beyond the eyes. It reports robust person-independent 3D performance across challenging appearance variations, with substantial gains over existing methods.
- The proposed spatial weights CNN uses the full face to support appearance-based gaze estimation.The method leverages information from facial regions beyond the eyes.
- The method was more robust to facial appearance variation caused by extreme head pose, gaze direction, and illumination than current eye-only and multi-region methods.
- 14.3% and 27.7% improvements over the state of the art were obtained on MPIIGaze and EYEDIAP, respectively.