Source-linked AI summary
Marker-free eye-gaze estimation using a single image and depth from defocus
David Hurtubise-Martin, Feriel Fass, Djemel Ziou, Marie-Flavie Auclair-Fortier
TL;DR
The paper addresses marker-free gaze estimation on a computer screen using only a single 2D camera. It combines iris localization, blur-based depth, head-pose features, and VBMLR classification, achieving near-perfect accuracy under the evaluated 50 cm and 70 cm conditions while remaining lightweight and probabilistic.
Problem
Existing gaze systems often require infrared illumination, multiple cameras, dedicated hardware, or external markers, motivating a single-camera marker-free screen-gaze method.
Method
The method estimates iris displacement and head pose from facial images and blur-based depth, then maps an eight-dimensional feature vector to screen areas using VBMLR.
Results
The proposed VBMLR approach achieves near-perfect accuracy under the evaluated acquisition conditions at user-to-screen distances of 50 cm and 70 cm.
Takeaways & Limitations
The approach remains computationally lightweight, probabilistic, and analytically tractable while using only an integrated 2D camera.
Takeaways & Limitations
At 70 cm, performance degrades because webcam optical limitations make blur-based depth less reliable, especially for Tz and its interaction with δx and δy.
Abstract
from arXiv · showhide
This paper presents a marker-free eye-gaze estimation approach using a single 2D camera, such as an integrated laptop webcam. The gaze-related features are estimated from iris localization and head pose estimated by using depth from defocus. A variational Bayesian multinomial logistic regression framework is used as mapping from the estimated features to the position of regard, based on an 8-dimensional feature vector of head-pose and iris-displacement parameters. No external marker is needed. Experiments were conducted by estimating the gaze of people watching a computer screen at different distances and compared against five existing methods. The obtained scores demonstrate the effectiveness of the proposed approach.
1 Introduction
Eye-gaze estimation supports applications across computer vision, cognitive science, marketing, safety, and cinematography. This work focuses on estimating where a user looks on a computer screen from an image captured by an embedded camera.
- Motivation: Eye-gaze estimation is used to study cognition, personalize recommendations, monitor driver vigilance, and support cinematography workflows.Applications include neurological-disorder and attention studies, screen-based personalization, vigilance monitoring, virtual camera control, and viewer-attention analysis.
- Existing approaches: Existing systems include active-illumination, infrared, multi-camera, calibration-dependent, passive-color, and 3D-camera approaches.These approaches differ in hardware, illumination, calibration, and handling of head motion.
- Target setting: The target setting is a user watching a computer screen while an embedded camera captures the user’s face.The system estimates the screen position being watched from image features such as head pose, eye location, and an automatically detected reference point.
- Approach overview: The paper’s processing pipeline comprises feature estimation followed by mapping those features to the user’s screen position.The document describes the paper’s system and its organization around the proposed method, mapping framework, feature extraction, depth estimation, and experiments.
2 Related work
Prior single-camera gaze-estimation methods differ in target, modeling strategy, preprocessing, calibration, and evaluation. The proposed method combines geometric and appearance-related information while explicitly incorporating blur-based user-to-screen distance into AoR classification.
- Comparison framework: Single-RGB-camera methods are organized by approach type, predicted target, inference strategy, preprocessing, and related design criteria.The comparison covers embedded cameras in laptops, smartphones, and tablets without active illumination or additional sensors.
- Prediction targets: Methods may predict a screen point of regard or gaze direction represented by angles or a gaze vector.Point-of-regard prediction aligns directly with screen interaction, whereas gaze-direction estimation is common in computer vision.
- Modeling strategies: Geometric methods are interpretable and frugal, but their assumptions can fail under low image quality, glasses, large head-pose changes, and unconstrained behavior.Appearance-based methods emerged as an alternative to explicit geometric modeling.
- Preprocessing and calibration: Preprocessing and calibration materially affect accuracy and comparability across gaze-estimation methods.Reported choices include detection and cropping, normalization, coordinate transformations, input modality, and explicit per-user calibration or generalization to unseen users.
- Evaluation: Point-of-regard metrics depend on display and viewing setup, making direct comparison between PoR methods challenging.Pixel, millimeter, centimeter, visual-angle, and area-classification measures capture different aspects of performance and require different setup information.
- Proposed positioning: The proposed method adds depth-from-defocus user-to-screen distance to a compact eight-dimensional feature representation for Bayesian AoR classification.It combines appearance-based features with geometric information while targeting low model complexity and interpretability.
3 Proposed approach overview
The proposed system estimates gaze from a single embedded-camera image by extracting iris displacement and head-pose features, then classifying the corresponding screen area of regard. Blur-based depth supports marker-free head-pose estimation, and VBMLR performs the mapping.
- Problem formulation: The target is the location within a displayed document that the user is watching, with resolvable detail depending on visual, camera, distance, illumination, and estimation factors.The system is designed for users viewing content on a screen through an embedded 2D camera.
- System pipeline: The pipeline acquires an image, extracts gaze-related features, and maps them to a predicted area of regard.The interface displays the camera feed, detection markers, and the predicted area in real time.
- Area classification: The screen is divided into discrete areas of regard, which are treated as classes in the mapping problem.For each frame, the extracted feature vector is assigned to one predefined area class.
- Feature representation: The feature representation contains head orientation, head position, and iris displacement parameters.The eight-dimensional vector is d = (Rx, Ry, Rz, Tx, Ty, Tz, δx, δy).
- Marker-free features: Iris displacement is measured relative to the nose, while facial reference points and blur-based depth estimate head position and orientation without physical markers.This reference-frame design reduces dependence on the face’s global image position during head movement.
- Mapping model: The mapping uses variational Bayesian multinomial logistic regression to estimate probabilities for multiple area-of-regard classes.The Bayesian formulation addresses separability concerns in logistic regression and compares in-screen classes with an outside-screen baseline.
4 Mapping estimation
The mapping stage treats each screen area as a class and estimates its probability from the eight-dimensional feature vector. VBMLR extends Bayesian logistic regression to multiple areas using an outside-screen baseline and supports thresholded real-time decisions.
- Problem formulation: The feature vector d = [Rx, Ry, Rz, Tx, Ty, Tz, δx, δy] is mapped to one in-screen area or an outside-screen baseline class.Each observation is associated with a binary indicator for the relevant class, and the model aims to identify the watched class.
- Multiclass model: VBMLR estimates multiple class-specific parameter vectors by discriminating each in-screen area from the outside-screen baseline.The multinomial extension is required because gaze prediction involves multiple AoR classes rather than a single binary decision.
- Operational constraints: Increasing the number of areas or reducing baseline size can lower accuracy and increase computation, although typical screen partitions may keep the class count manageable.The text notes that class imbalance and the number of AoRs affect both performance and real-time feasibility.
- Training data: The training data contain feature-vector sets for each area and the outside-screen baseline, with class labels indicating which area generated each observation.The formulation assumes balanced area datasets for the binary class comparisons.
- Bayesian estimation: Bayesian priors and variational approximation address drawbacks of maximum-likelihood logistic regression, including collinearity, separability, and numerical instability.The posterior is approximated using variational methods, with Gaussian prior and posterior parameter distributions described in the formulation.
- Prediction: At test time, a feature vector is evaluated against the learned class models and a Bayesian decision rule selects the watched area.The threshold η can be chosen by cross-validation to trade off false positives and false negatives.
5 Iris localization and geometric feature extraction
The system estimates iris displacement relative to the nose and combines it with three-dimensional head orientation and position to form an eight-dimensional gaze feature vector. Iris localization uses dark-region detection, visible-contour verification, ellipse fitting, and temporal smoothing to remain usable under partial occlusion and noise.
- Geometric feature extraction: The gaze feature vector d contains three head-orientation components, three head-position components, and two iris-displacement components.Head orientation and position are expressed in the camera frame, while iris displacement is expressed in image coordinates.
- Geometric feature extraction: The iris displacement vector is measured from the detected nose center to the mean position of the two smoothed iris centers.The nose center provides a facial reference that reduces the effect of global head movement in the image.
- Iris localization: Iris localization first thresholds and cleans dark pixels, separates candidate regions, and selects the most plausible region as an initial iris-center estimate.The search is restricted to the darkest part of each eye region because the iris and pupil usually form a dark structure there.
- Iris localization: The initial candidate is verified using visible contour portions sampled along circular contours, retaining only directions corresponding to visible iris boundaries.The retained directions Γ are used to evaluate circular contours and radial intensity changes over candidate radii.
- Iris localization: The radius with the largest smoothed radial intensity change is retained, after which an ellipse is fitted to the contour points to refine the iris center.The radial change is intended to identify a likely transition between the iris and surrounding region.
- Temporal stabilization: Temporal smoothing combines each current iris measurement with the preceding frame, weighting low-confidence detections toward the previous center and high-confidence detections toward the current one.The procedure is applied independently to both eyes before their resulting centers are averaged.
6 Blur-based depth estimation
Head pose is estimated from three facial reference points whose depths are inferred from image blur using a single 2D camera. Back-projection reconstructs the points in the camera frame, from which facial-plane orientation and position are computed.
- Pipeline overview: The depth-estimation pipeline uses image blur, facial-point depths, and reconstructed three-dimensional points to estimate head position and orientation.The three stages are blur estimation, depth estimation for facial reference points, and head-pose estimation.
- Blur-to-depth model: A Gaussian point-spread-function model relates image blur to the depth z of a scene point, using camera parameters such as focal length, f-number, and focus distance.The camera parameters are obtained from specifications or calibration.
- Blur estimation: Local blur is approximated from the change in image-gradient magnitude after applying additional Gaussian smoothing at a known scale.The implementation converts the image to grayscale, adjusts local contrast with CLAHE, smooths it, and compares gradients.
- Facial reference points: The nose center and two eye-box centers define three image-plane reference points, whose local blur values are converted into depths zA, zB, and zC.The points are named A′, B′, and C′, with A′ at the nose-box center and B′ and C′ at the eye-box centers.
- 3D reconstruction: Back-projection combines each reference point’s image coordinates and estimated depth with the camera intrinsics to reconstruct three-dimensional points A, B, and C.Image coordinates determine the projection-ray direction, while depth determines the point’s position along that ray.
- Head-pose estimation: The head orientation is the normalized normal of the facial plane through A, B, and C, while head translation is the centroid of the reconstructed points.These pose components are combined with iris displacement in the gaze feature vector.
7 Experimentations
The experiments evaluate VBMLR and five baselines for nine screen zones at 50 cm and 70 cm, using an 8-dimensional feature vector and cross-validation. VBMLR performs near ceiling at 50 cm and remains strongest among non-pretrained methods at 70 cm, while longer distance degrades performance.
- Experimental setup: The system uses an integrated 640 × 480 RGB webcam, a nine-zone 15-inch screen, and features d = [Rx, Ry, Rz, Tx, Ty, Tz, δx, δy].Each class has 9 learned parameters including a bias, yielding 81 parameters across nine zones.
- Experimental setup: VBMLR is compared with five representative mappings using preliminary hyperparameter selection and cross-validation.The reported metrics include accuracy, false positives, true negatives, error statistics, training time, and parameter count.
- Results at 50 cm: 99.69% mean accuracy: VBMLR achieves near-ceiling performance at 50 cm, with fold errors from 0.14% to 0.83%.Ridge reaches 99.58%, SVM 99.03%, and CNN-1D 98.86%; Linear Regression reaches 66.11%.
- Results at 50 cm: 99.72% mean accuracy: pretrained ResNet18 slightly exceeds VBMLR at 50 cm, but uses 8.4M parameters and substantially higher computational cost.Its input is the same handcrafted 8-dimensional feature vector reshaped into a pseudo-image rather than a natural gaze image.
- Results at 70 cm: 96.81% mean accuracy: VBMLR remains strongest among non-pretrained methods at 70 cm, followed by CNN-1D at 96.64%.All methods degrade at 70 cm as defocusing weakens depth-related information; Linear Regression reaches 63.42%.
- Results at 70 cm: VBMLR’s confusion at 70 cm concentrates in upper adjacent zones, while lower zones remain less affected in the pooled setting.The reported interpretation links longer-range degradation to noisier depth-related features, notably Tz and its interaction with δx and δy.
- Deployment considerations: VBMLR supports the primary-method choice with about 0.002 ms/sample inference time, 81 parameters, no GPU requirement, and calibrated posterior probabilities.These properties provide a lightweight and interpretable alternative to higher-complexity convolutional models.
8 Conclusion
The paper presents marker-free gaze estimation using a single integrated webcam and blur-based facial-relief estimation. Under the evaluated fixed-distance conditions, VBMLR achieves near-perfect accuracy while remaining lightweight, probabilistic, and analytically tractable.
- Contribution: The proposed system uses only a 2D camera and blur-based facial-relief estimation, avoiding infrared illumination, multiple cameras, dedicated hardware, and external markers.The approach targets low-cost webcam-based screen interaction when the embedded camera is fixed or known during a session.
- Conclusion: VBMLR achieves near-perfect accuracy at the evaluated 50 cm and 70 cm distances while remaining computationally lightweight, probabilistic, and analytically tractable.The comparison reports competitiveness with five other methods and a simpler training and deployment profile than CNN-based alternatives.