Source-linked AI summary
L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments
Ahmed A. Abdelrahman, Thorsten Hempel, Aly Khalifa, Ayoub Al-Hamadi
TL;DR
Unconstrained gaze estimation remains difficult because appearance, lighting, head pose, and gaze direction vary widely. L2CS-Net separately predicts yaw and pitch with combined classification-regression losses, achieving state-of-the-art performance on MPIIGaze and Gaze360.
Problem
Unconstrained gaze estimation remains challenging because eye appearance, lighting, head pose, and gaze direction vary across images.
Method
L2CS-Net uses a CNN with separate yaw and pitch predictors and two angle-specific losses combining softmax classification with regression.
Results
L2CS-Net achieves state-of-the-art performance with 3.92° mean angular error on MPIIGaze and 10.41° on Gaze360 front 180°.
Takeaways & Limitations
The proposed angle-specific multi-loss design supports robust gaze prediction in unconstrained settings across MPIIGaze and Gaze360.
Abstract
from arXiv · showhide
Human gaze is a crucial cue used in various applications such as human-robot interaction and virtual reality. Recently, convolution neural network (CNN) approaches have made notable progress in predicting gaze direction. However, estimating gaze in-the-wild is still a challenging problem due to the uniqueness of eye appearance, lightning conditions, and the diversity of head pose and gaze directions. In this paper, we propose a robust CNN-based model for predicting gaze in unconstrained settings. We propose to regress each gaze angle separately to improve the per-angel prediction accuracy, which will enhance the overall gaze performance. In addition, we use two identical losses, one for each angle, to improve network learning and increase its generalization. We evaluate our model with two popular datasets collected with unconstrained settings. Our proposed model achieves state-of-the-art accuracy of 3.92° and 10.41° on MPIIGaze and Gaze360 datasets, respectively. We make our code open source at https://github.com/Ahmednull/L2CS-Net.
1 Introduction
Appearance-based gaze estimation is important for interactive and augmented-reality applications, but unconstrained conditions remain difficult. L2CS-Net addresses this challenge with independently regressed gaze angles and separate combined losses, achieving state-of-the-art results on two unconstrained datasets.
- 1 Introduction: Gaze is used to indicate engagement in human-robot interaction and dialogue systems and to predict attention in augmented reality.These applications motivate accurate gaze estimation from ordinary cameras.
- 1 Introduction: CNN-based appearance methods improve gaze accuracy but remain insufficiently robust and generalizable in unconstrained environments.Challenges include varied eye appearance, illumination, head pose, and gaze direction.
- 1 Introduction: L2CS-Net independently regresses yaw and pitch with separate combined losses to improve angle prediction and generalization.Each loss combines binary gaze-bin classification and regression, with softmax-based neighborhood estimation.
2 Related Work
Appearance-based gaze estimation includes conventional person-specific regressors and CNN-based models. CNN methods model nonlinear image-to-gaze mappings and have expanded through multistream, statistical, dilated-convolution, asymmetric, adversarial, adaptive, and attention-based designs.
- 2 Related Work: Conventional regressors work reasonably in constrained settings but significantly lose accuracy under unconstrained conditions.Their mappings are typically person-specific and assume fixed head pose and illumination.
- 2 Related Work: CNN-based methods model highly nonlinear mappings from images to gaze and commonly use VGG or ResNet backbones.Architectures vary from single-eye inputs to multichannel combinations of eyes, faces, and face-grid information.
- 2 Related Work: Statistical-deep-learning hybrids, dilated convolutions, gaze decomposition, head-pose features, and ensembles extend CNN-based gaze estimation.These approaches target richer representations or improved accuracy.
- 2 Related Work: FAR-Net uses asymmetric treatment of the two eyes, while adversarial Bayesian learning targets gaze generalization.Both approaches exploit specialized modeling choices beyond standard CNN regression.
- 2 Related Work: AGE-Net combines CNN feature extraction and attention-based weighting for each eye with face-image features.The parallel eye streams are multiplied before refinement with face-CNN output.
3 METHOD
L2CS-Net combines classification and regression for each gaze angle, using shared convolutional features with separate yaw and pitch heads. It is designed and evaluated for unconstrained gaze estimation using Gaze360 and MPIIGaze.
- 3.1 Proposed loss function: Each gaze angle uses a combined cross-entropy and mean-squared-error loss, with softmax bins followed by expectation-based fine-grained prediction.The regression coefficient β controls the mean-squared-error contribution.
- 3.1 Proposed loss function: The proposed combined classification-regression loss is presented as a novel component relative to the cited CNN-based gaze-estimation literature.The paper states that related CNN methods had not used this combination.
- 3.2 L2CS-Net Architecture: L2CS-Net feeds face images through a ResNet-50 backbone and predicts yaw and pitch separately with two fully connected layers sharing convolutional features.Separate loss functions provide two backpropagated learning signals.
- 3.2 L2CS-Net Architecture: Softmax converts each output into a probability distribution, whose expectation yields a fine-grained gaze prediction before mean-square error is added.Figure 1 depicts the detailed architecture and combined-loss flow.
- Datasets: Gaze360 spans 360° with 238 subjects across varied indoor and outdoor conditions, while MPIIGaze contains 213,659 daily-routine images from 15 subjects.Both datasets provide unconstrained gaze data with environmental or temporal diversity.
4 Experiments
L2CS-Net is evaluated on MPIIGaze and Gaze360 using normalized images, binned gaze targets, angular error, and comparisons with state-of-the-art methods.
- 4.2 Training and results: The datasets use normalized images and both continuous and binned pitch-and-yaw targets for the model’s regression and classification objectives.Normalization removes head roll and standardizes distance to a reference point; gaze targets are divided into bins with binary labels.
- 4.2 Training and results: The model uses an ImageNet-pretrained ResNet-50, Adam optimization, 50 training epochs, batch size 16, and gaze angular error as its evaluation metric.Experiments vary the regression coefficient β and compare performance with state-of-the-art gaze estimation methods.
- 4.2 Training and results: 3.92° mean angular error makes L2CS-Net state of the art on MPIIGaze with β = 1.The evaluation uses leave-one-subject-out cross-validation, and Table 1 compares the result with state-of-the-art methods.
- 4.2 Training and results: On MPIIGaze, L2CS-Net achieves better subject-wise gaze accuracy than FARE-Net for 11 of 15 subjects.Figure 2 presents the subject-level comparison.
- 4.2 Training and results: 10.41° mean angular error on front 180° and 9.02° on front-facing make L2CS-Net state of the art on Gaze360 with β = 1.Table 2 compares these results with state-of-the-art methods under the reported Gaze360 evaluation scopes.
5 Conclusion
The paper concludes that L2CS-Net robustly predicts 3D gaze directions in unconstrained environments and achieves state-of-the-art accuracy on two unconstrained datasets.
- 5 Conclusion: L2CS-Net predicts 3D gaze directions in unconstrained environments and achieves the lowest angular error on MPIIGaze and Gaze360.The conclusion presents the model as robust across both evaluated datasets.