Source-linked AI summary
Event Based, Near Eye Gaze Tracking Beyond 10,000Hz
Anastasios N. Angelopoulos, Julien N. P. Martel, Amit P. S. Kohli, Jorg Conradt, Gordon Wetzstein
TL;DR
Conventional gaze-tracking cameras face bandwidth and power limitations that restrict acquisition speed, motivating faster near-eye tracking. The paper combines frames and events with online pupil fitting and polynomial gaze regression, achieving event-rate updates beyond 10,000 Hz and reported accuracy from 0.45° to 1.75° across fields of view.
Problem
Bandwidth and power limitations in conventional cameras restrict gaze acquisition speed, hindering low-latency near-eye tracking.
Method
The method combines low-rate frames and high-rate events to update a parametric 2D pupil model and regress gaze in real time.
Results
10 kHz is the claimed conservative peak update rate, with robust pupil fitting every 20 events.
Takeaways & Limitations
The system provides frame-based precision together with event-based high-rate updates for near-eye gaze tracking.
Takeaways & Limitations
The evaluation does not account for slippage, a major source of error in head-mounted eye tracking.
Abstract
from arXiv · showhide
The cameras in modern gaze-tracking systems suffer from fundamental bandwidth and power limitations, constraining data acquisition speed to 300 Hz realistically. This obstructs the use of mobile eye trackers to perform, e.g., low latency predictive rendering, or to study quick and subtle eye motions like microsaccades using head-mounted devices in the wild. Here, we propose a hybrid frame-event-based near-eye gaze tracking system offering update rates beyond 10,000 Hz with an accuracy that matches that of high-end desktop-mounted commercial trackers when evaluated in the same conditions. Our system builds on emerging event cameras that simultaneously acquire regularly sampled frames and adaptively sampled events. We develop an online 2D pupil fitting method that updates a parametric model every one or few events. Moreover, we propose a polynomial regressor for estimating the point of gaze from the parametric pupil model in real time. Using the first event-based gaze dataset, available at https://github.com/aangelopoulos/event_based_gaze_tracking , we demonstrate that our system achieves accuracies of 0.45 degrees--1.75 degrees for fields of view from 45 degrees to 98 degrees. With this technology, we hope to enable a new generation of ultra-low-latency gaze-contingent rendering and display techniques for virtual and augmented reality.
1 INTRODUCTION
Conventional cameras impose bandwidth, power, and latency constraints on near-eye gaze tracking, while event cameras sample motion adaptively. The proposed hybrid system combines low-rate frames with high-rate events to update pupil and gaze estimates beyond 10,000 Hz.
- Motivation: Camera bandwidth and power constraints create a tradeoff between gaze-tracking accuracy and latency.High-speed camera systems require substantial data acquisition, communication, and processing resources.
- Motivation: Event cameras sample pixel-level irradiance changes asynchronously, reducing redundant data when only the pupil moves.Events include location, timestamp, and change sign, and can improve speed and power consumption.
- System: The system combines low-rate frames with high-rate events to update a parametric pupil model online and map it to a gaze vector.Frames provide traditional pupil-detection anchors, while events update the pupil location during motion.
- Contributions: The work introduces a hybrid prototype, an event-rate model-based algorithm, and a binocular dataset from 24 subjects.The dataset includes saccadic motions and smooth pursuits.
- Results: 0.45° is the reported accuracy within EyeLink’s field of view, with the same evaluation protocol and 10× faster speed.The comparison is based on the system overview in Table 1.
2 RELATED WORK
Prior eye-tracking research spans invasive methods, frame-based model and appearance-based trackers, and event-camera applications outside eye tracking. The paper positions itself as the first practical real-time event-based eye-tracking system with a public event-based gaze dataset.
- Event-based vision: Event cameras have supported high-speed tasks including object tracking, visual odometry, pose tracking, reconstruction, SLAM, and hand tracking.These applications established event-based vision across multiple tracking and 3D perception settings.
- Eye tracking: Camera-based eye tracking evolved from Purkinje-reflection methods to frame-based model fitting and appearance-based gaze estimation.Existing pupil-fitting and gaze-estimation methods are rate-limited by camera frames.
- Datasets: Existing gaze datasets include millions of synthetic and real eye images, but no dataset was available for event-based eye tracking.The cited datasets include GazeCapture, UT Multi-view, SynthesEyes, UnityEyes, MPIIGaze, and NVGaze.
- Problem scope: Modern gaze-tracking work commonly targets either full-face tracking in the wild or near-eye infrared tracking in controlled environments.This paper focuses on the near-eye category rather than tracking in the wild.
- Closest prior work: The closest prior DVS eye-tracking project lacked demonstrated real-time performance and had accuracy limited to a third of a 1024×768 screen.The paper presents its system as the first practical real-time framework with online pupil fitting and a public event-based dataset.
3 SYSTEM AND METHODS
The system fuses asynchronous event data with synchronous frames to update a parametric eye model and map it to gaze in real time. Online geometric fitting and a lightweight polynomial regressor support frequent updates while retaining frame-based information.
- Eye model: The eye model contains an elliptical pupil, parabolic eyelid, and circular glint, represented by 11 parameters.The components are fit separately as quadrics, with candidate points supplied by frames and events.
- Candidate extraction: Candidate pupil-edge points are extracted from thresholded, morphologically opened frames and from events near the estimated ellipse border.The event stream contributes only events sufficiently close to the last estimated ellipse.
- System overview: The hybrid tracker concurrently processes events and frames, using both streams to update the same underlying eye model.Events provide asynchronous updates during motion, while frames update the model at a constant rate.
- Online fitting: Updates can occur event by event or in batches, with multiple events improving robustness compared with singleton updates.The event-wise implementation is designed to operate at high event rates while avoiding repeated full matrix inversion.
- Online fitting: The model is updated online by blending accumulated least-squares statistics with new candidate points using a decay parameter.The implementation stores a 5×5 matrix and 5-dimensional vector, and can update their inverse with the Sherman-Morrison-Woodbury identity.
- Gaze mapping: A fifth-order polynomial regressor maps the pupil center or eye-model parameters to screen gaze coordinates with minimal evaluation cost.Its few additions and multiplications make it suitable for event-rate gaze updates, although more complex regressors may improve robustness or generalization.
4 DATASET
The authors collected the first event-based gaze-tracking dataset using synchronized binocular sensors and controlled eye-motion experiments. It includes 24 subjects performing random saccades and smooth pursuit across a wide 64×96° field of view.
- Acquisition setup: The dataset uses two synchronized DAVIS346b sensors with near-infrared illumination to image the left and right eyes.Participants’ heads were stabilized with an ophthalmic head rest during recording.
- Dataset characteristics: The first event-based gaze-tracking dataset contains recordings from 24 subjects performing random saccades and smooth pursuit.The stimulus was a green cross on a black background, and the monitor covered a 64×96° field of view.
- Experiments: The saccade experiment presented fixation targets at 121 locations arranged as an 11 × 11 grid, with each location shown once for 1.5 seconds.The target order was randomized but shared across users.
5 RESULTS
The event-based tracker achieves real-time pupil updates at a conservative peak rate of 10,000 Hz, with accuracy and precision evaluated across fields of view. Results also identify a trade-off between update speed, robustness, and sparsity, with 20 events per fit providing a practical balance.
- Update rates: 10,000 Hz is the system’s conservative peak pupil-update rate during saccadic motion.The estimate uses more than 200 events per millisecond and 20 events per fit.
- Update rates: 20 events per fit provides a balance among smoothness, robustness, sparsity, and update speed.Using every event maximizes rate but can produce unstable fits, whereas hundreds of events make tracking too sparse between frames.
- Pupil tracking: The event-based pupil estimate almost never has IOU below 0.8 or center error above 3 pixels relative to frame-based estimates.The comparison uses the event estimate immediately before each frame and the pupil extracted from that frame.
- Gaze mapping: The system achieves accuracy below 0.5° in a standard field of view and about 2° in a larger field.The paper reports commercial frame-based systems at < 0.5° within a standard field of view and degradation to 2° on a larger field.
- Gaze mapping: Precision is 1.6° for the smaller field of view and 3° for the larger field of view.Precision is computed as the empirical standard deviation of estimated gaze points using held-out calibration positions.
6 DISCUSSION
The system combines frame-based and event-based sensing for near-eye gaze tracking, achieving a conservative 10 kHz update rate while retaining 0.5° precision in a 121-point fixation task. The discussion highlights miniaturization and AR/VR potential alongside calibration, slippage, and latency limitations.
- Performance: 0.5° visual-angle error was achieved in a 121-fixation-point task while retaining event-based tracking advantages.The reported precision is described as state of the art under the evaluation conditions.
- Performance: 10 kHz is the claimed conservative peak update rate, based on fitting the pupil every 20 events at 200 events per millisecond.The method sustains real-time processing of that event rate.
- Limitations: The work does not report eye-movement-to-update latency because it depends heavily on processor and camera communication hardware.The software update is estimated to require about 300 FLOPS in the worst case, but embedded hardware was not built.
- Miniaturization: The head-mounted prototype uses a 22×22 mm DVS and lens connected to a battery-powered Raspberry Pi, demonstrating miniaturization potential.The prototype sensor is lower resolution and does not output frames, whereas the desktop DAVIS346 chip could also fit the mount with a custom PCB.
- AR/VR implications: Event-based sensing can reduce power during fixation and small eye motions because fewer events are generated and processed.The passage presents this as a potential benefit for battery-constrained AR/VR systems.
- Limitations: Camera slippage relative to the face remains a major challenge for head-mounted use, and the evaluations do not account for it.The discussion points to 3D pose modeling, recalibration, and future event-driven slip compensation as possible directions.
A FORMULATION OF THE PROBLEM FOR P AND C
The appendix formulates event- and image-based fitting for eyelash parabolas, pupil ellipses, and glint circles, then maps fitted pupil parameters to screen coordinates with a polynomial regressor. Each model is updated from spatially selected image points and nearby events.
- A.1 Updating and fitting the parabola P: The eyelash parabola is parameterized as P = (a′,g′,d′) under the assumption that y can be expressed as a function of x.This representation is valid while the eye remains sufficiently upright in image space.
- A.1 Updating and fitting the parabola P: Parabola parameters are obtained by solving P∗ = A′−1 b′ from points belonging to the fitted parabola.The formulation parallels the ellipse fit used for the pupil.
- A.1 Updating and fitting the parabola P: Image candidates for the eyelash parabola come from Harris corners after intensity clipping, pupil-center distance filtering, and removal of lower-half points.The filtering uses a radius ρ′ around the current pupil-center estimate.
- A.1 Updating and fitting the parabola P: Parabola-fitting points combine event and image sets, with events selected within radius δ of the currently estimated parabola.Image points are represented through the corresponding D′ construction.
- A.2 Updating and fitting the circle for the glint C: The glint fit is a special case of ellipse regression with a = b = 1 and h = 0.This constrains the quadric model to a circle.
- A.2 Updating and fitting the circle for the glint C: The glint is modeled as a circle with parameters C = (cx,cy,r), using image points above threshold t3 and within distance ρ′′ of the pupil center.Events near the current glint estimate are also selected within distance δ.
- Regressor: A second-order polynomial regressor maps the fitted pupil center (xe,ye) to screen coordinates (xs,ys).The mapping is represented by coordinate polynomials with parameters θ1 and θ2.
- Regressor: The regressor coefficients are fitted by linear least squares using calibration pairs of presented screen points and ellipse-derived pupil centers.The calibration pairs are denoted by screen coordinates (xs,ys) and image-space estimates (xe,ye).
EXPERIMENT
The experiment defines accuracy as ISO 5725 trueness and precision as agreement among gaze estimates, with gaze error computed from estimated and true directions. The supplied figure passage concerns a frame-only pupil-center-error ablation histogram.
- Accuracy: Accuracy is defined using ISO 5725 trueness: agreement between the arithmetic mean of test results and the true or accepted reference value.The appendix gives the formal definition used for the main-text accuracy results.
- Accuracy: Each gaze-error term li is the L2 distance between estimated gaze direction ˆdi = (ˆφi,ˆθi) and true gaze angle di = (φi,θi).The total number of estimates is denoted by n.
- Precision: Precision is quantified as the empirical standard deviation of the estimated gaze directions ˆdi.The empirical mean of all estimates is denoted by ¯ˆd.
- Ablation: The frame-only ablation is evaluated through a histogram of differences in pupil-center error.The figure passage refers readers to Appendix G for further details.
E BLINK DETECTOR
The blink detector removes likely blink frames from accuracy and precision calculations by detecting drastic changes in the fitted pupil ellipse’s eccentricity.
- Blink detection: Blink frames are identified from changes in ellipse eccentricity, the ratio between the fitted ellipse’s major and minor radii.During a blink, the fitted pupil ellipse often becomes long and thin while fitting the eyelashes.
F EMPIRICAL SMOOTHNESS
The paper quantifies empirical smoothness from evenly spaced gaze estimates, with higher values indicating smaller changes in eye center between successive increments. This measure depends on time scale and relies on an assumption about physiological smoothness.
- Empirical smoothness is computed from a sequence of gaze estimates sampled at evenly spaced times.
- Higher smoothness values indicate that the eye center changes little between successive increments.
- The measure depends on time scale: high-rate estimates are expected to appear smoother than 30Hz frame-rate estimates.
- Optimizing parameters using empirical smoothness assumes that the measured smoothness reflects the eye’s true physiological motion.The paper notes that this assumption has limitations and may be addressed in future work.
G FRAME ONLY ABLATION
The frame-only ablation holds the pupil position fixed between frames, matching traditional non-predictive eye tracking. During saccades, its errors are generally larger than those of the full event-based system, while an event-only alternative is left for future work.
- The frame-only ablation assumes that the pupil does not move between frames, equivalent to traditional non-predictive eye tracking.
- The frame-only and full systems differ less during fixation because most frames contain little eye movement.
- During saccades, the frame-only error exceeds the full system’s event-based error for most histogram mass.The comparison plots d_frame − d_event, so positive values indicate degradation from removing events.
- An event-only baseline is not implemented and would require re-initialization after every blink under the described algorithm.