Source-linked AI summary
Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings
Seonwook Park, Xucong Zhang, Andreas Bulling, Otmar Hilliges
TL;DR
Conventional gaze-estimation methods struggle in unconstrained imagery, where appearance-based methods have gained an advantage. This paper learns interpretable eye-region landmarks from synthetic images, applies them to real images, and uses them in lightweight feature-based and model-based estimators. The resulting approach improves landmark tasks and gaze estimation across cross-dataset and personalized settings, including personalization with few calibration samples.
Problem
Conventional feature-based and model-based gaze estimation is challenged by illumination changes and visual artifacts in unconstrained real-world imagery.
Method
A CNN trained exclusively on synthetic images detects eye-region landmarks for direct use in feature-based regression and model-based eyeball fitting.
Results
The method improves iris localization, eyelid registration, cross-dataset gaze estimation, and personalized gaze estimation over prior methods.
Takeaways & Limitations
Explicit landmark features make conventional gaze-estimation methods competitive in unconstrained settings and support personalization with few calibration samples.
Takeaways & Limitations
Direct evaluation of real-world eye-region landmark localization remains difficult because current datasets lack high-quality labels.
Abstract
from arXiv · showhide
Conventional feature-based and model-based gaze estimation methods have proven to perform well in settings with controlled illumination and specialized cameras. In unconstrained real-world settings, however, such methods are surpassed by recent appearance-based methods due to difficulties in modeling factors such as illumination changes and other visual artifacts. We present a novel learning-based method for eye region landmark localization that enables conventional methods to be competitive to latest appearance-based methods. Despite having been trained exclusively on synthetic data, our method exceeds the state of the art for iris localization and eye shape registration on real-world imagery. We then use the detected landmarks as input to iterative model-fitting and lightweight learning-based gaze estimation methods. Our approach outperforms existing model-fitting and appearance-based methods in the context of person-independent and personalized gaze estimation.
1 INTRODUCTION
Gaze estimation from ordinary cameras is useful but remains difficult in unconstrained conditions, where illumination and visual artifacts undermine conventional landmark-dependent methods. The paper proposes robust eye-region landmark learning from synthetic data to improve conventional gaze estimation.
- Off-the-shelf-camera gaze estimation can support accessibility, saliency estimation, reading, and gaze-based interaction.
- Existing systems can fail under low image quality and challenging illumination in real-world environments.
- Appearance-based methods significantly outperform model- and feature-based methods on MPIIGaze, especially for person-independent estimation in unconstrained settings.
- The proposed approach explicitly learns interpretable features so traditional methods can address robust feature detection in natural images.
- The method is trained only on synthetic images and improves iris localization, eyelid registration, cross-dataset estimation, and person-specific estimation.
2 RELATED WORK
Related work spans geometric, feature-based, model-based, cross-ratio, appearance-based, and landmark-localization approaches to gaze estimation. The paper positions learned eye landmarks as a bridge between robust representation learning and interpretable gaze models.
- Feature-based Gaze Estimation: Feature-based methods map hand-crafted eye-shape and auxiliary features, such as head pose, to gaze direction.
- Feature-based Gaze Estimation: The PC-EC vector uses pupil-center and eye-corner locations as a feature for estimating horizontal gaze direction.
- Feature-based Gaze Estimation: Rich learned eye-region landmarks combined with features such as the PC-EC vector significantly improve gaze-estimation accuracy.
- Model-based Gaze Estimation: Model-based methods use eyeball geometry, whereas this approach uses a neural network to fit an eyeball directly to an eye image.
- Appearance-based Gaze Estimation: Appearance-based methods evolved from image-based regressors and classical learners toward CNNs and GANs for in-the-wild benchmarks.
- Appearance-based Gaze Estimation: The paper reports personalized gaze estimation with as few as 10 calibration samples and learned landmark features outperforming AlexNet features.
- Human Pose Estimation and Facial Landmark Localization: Stacked hourglass networks provide a low-complexity architecture for learning spatial relations in landmark localization.
3 OVERVIEW
The paper trains a synthetic-data landmark detector based on a pose-estimation CNN and applies it directly to real eye images. Its landmarks support both feature-based and model-based gaze estimation without fine-tuning or domain adaptation.
- The approach trains a robust eye-landmark detector on synthetic images and applies it directly to real images without fine-tuning or domain adaptation.
- The detector uses a CNN architecture originally developed for human pose estimation.
- Synthetic training data provide correct labels under heavy occlusion, while augmentation supports detection on in-the-wild eye images.
- Detected landmarks feed simple learning-based feature methods and enable fitting a 3D eyeball model when camera intrinsics are unknown.
- The paper uses the landmarks in both feature-based and model-based gaze-estimation pipelines.
4 EYE REGION LANDMARK LOCALIZATION
The method adapts an hourglass network to predict eye-region landmark heatmaps, trained only on synthetic images with extensive augmentation. Strong augmentation improves robustness and speeds training, while the detector transfers to real-world imagery.
- Architecture: The network detects eyeball centers and occluded iris-edge landmarks by exploiting large spatial context in eye images.The approach can achieve reasonable localization even under total occlusion.
- Architecture: The adapted hourglass network predicts 18 confidence heatmaps, one for each eye-region landmark, using repeated multi-scale refinement.Intermediate supervision applies a pixel-wise squared-difference loss to each predicted heatmap.
- Training data: Synthetic UnityEyes training data provides variation in iris color, eye shape, head pose, and illumination, enabling transfer to real-world images despite lacking common webcam artifacts and eye decorations.The method relies on training-data augmentation to learn a robust model from synthetic images alone.
- Results: Higher augmentation produces higher gaze-estimation accuracy on real-world images, while augmentation also improves robustness and speeds training.With more training steps, weaker augmentation can approach the performance of stronger augmentation.
- Training data: Strong augmentation varies translation, rotation, intensity, blur, scale, resolution, and artificial occlusions during training.The training scheme also uses curriculum learning, increasing augmentation difficulty over time.
- Model fitting: The model-based fitting stage predicts eight iris-edge landmarks, the iris center, and the eyeball radius from the eye image.The eyeball is represented using two intersecting spheres, and the landmark predictions support iterative fitting.
5 GAZE ESTIMATION
The detected landmarks are converted into normalized geometric features or fitted to a two-sphere eyeball model for gaze estimation. These representations support person-independent and personalized estimation, including calibration-based adaptation.
- Feature-based gaze estimation: Landmark coordinates are normalized by eye width and centered on the inner eye corner, while an eyeball-to-iris center vector supplies a 2D gaze prior.The prior significantly improves performance when only very few person-specific training samples are available.
- Feature-based gaze estimation: Thirty-six landmark-based features train an SVR to estimate 3D gaze direction as eyeball pitch and yaw.The SVR supports either person-independent training or person-specific training from a small calibration set.
- Model-based gaze estimation: The model-based method represents the eyeball as a large sphere intersecting a smaller corneal-bulge sphere and uses eight iris-edge landmarks plus iris and eyeball centers.It also estimates the eyeball radius in pixels from the landmark network.
- Model-based gaze estimation: Without camera parameters, the fitted model remains in pixel units, with the 3D radius set equal to the image-space eyeball radius.The model therefore does not require camera calibration for fitting.
- Model-based gaze estimation: The model jointly estimates gaze angles, angular iris radius, and an eye-roll offset by iteratively minimizing landmark reprojection error.Conjugate gradient optimization can solve for the unknown model variables using predicted landmark coordinates.
- Personalization: Person-specific calibration estimates parameters that adapt the model to an individual, including the angular offset between optical and visual axes.This correction is applied during personalized gaze estimation.
6 EVALUATIONS
The evaluations show that the learned eye-region landmarks support accurate eyelid and iris localization, model-based and feature-based gaze estimation, and personalized cross-dataset performance. Across these tasks, the approach is compared with established model-fitting, feature-based, and appearance-based methods.
- Evaluation scope: The evaluation covers eyelid registration, iris localization, model-based gaze estimation, feature-based gaze estimation, cross-dataset evaluation, and personalized gaze estimation.Experiments use remote-camera datasets and comparisons against prior state-of-the-art methods and an appearance-based baseline.
- 6.1 Eyelid Registration: Our method is more accurate and robust than Wood et al.'s CLNF approach for eyelid registration on challenging remote eye images.The eyelid error is the mean Euclidean distance to annotated eyelids normalized by interocular distance.
- 6.2 Iris Localization: Our method consistently outperforms ExCuSe and ElSe across the full threshold range on all three iris-localization datasets.Success rates use distance to ground truth normalized by horizontal eye width.
- 6.3 Model-Based Gaze Estimation: The model-fitting approach improves gaze estimation on EYEDIAP while achieving results similar to Wang and Ji on Columbia Gaze.The method uses 20 images per person for calibration and requires neither camera intrinsics nor accurate person-specific 3D models.
- 6.4 Feature-Based Gaze Estimation: Combining eyelid and iris landmarks improves feature-based gaze estimation, especially on UT Multiview with large head-pose and gaze-direction variability.The final feature set also includes an iris-center–eyeball-center vector as a gaze-direction prior.
- 6.5–6.6 Cross-Dataset and Personalized Estimation: Both proposed methods outperform the AlexNet baseline in person-independent cross-dataset evaluation, while the landmark method improves personalized error from 5.5° to 4.6° with 20 samples.The landmarks-based SVR performs better than the AlexNet-derived SVRFC7 representation across calibration levels and can improve by up to 2° on EYEDIAP and UT Multiview.
7 CONCLUSIONS AND FUTURE WORK
The paper demonstrates that synthetic-only training can produce useful eye-region landmarks for unconstrained real images, improving several gaze-estimation tasks. It also identifies unresolved evaluation and calibration questions for future work.
- A synthetic-only neural network detects iris and eyelid landmarks in real, unconstrained images and supports eyeball fitting.The same network improves model-based gaze estimation while retaining a compact form suitable for robust real-time use.
- The method improves eyelid registration, iris localization, cross-dataset person-independent estimation, and personalized gaze estimation.Both model-based and feature-based methods outperform an AlexNet baseline with substantially more parameters.
- Landmark-based SVR performs especially well for personalized gaze estimation when calibration samples are few.The paper highlights personalization with low numbers of calibration samples as a particularly important result.
- Direct evaluation of eye-region landmark localization remains difficult because current real-world datasets lack high-quality labels.The authors therefore call for more research to determine which representations yield the most accurate and robust gaze estimation.
- Future work should clarify how to collect and use calibration samples and further improve gaze accuracy under difficult environmental conditions.The authors suggest combining the low-sample personalization result with online calibration for user-facing cameras.