Source-linked AI summary
Eye Tracking for Everyone
Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech Matusik, Antonio Torralba
TL;DR
Eye tracking is broadly useful but has not become pervasive because existing systems can require costly or specialized hardware and may be inaccurate in real-world conditions. This paper introduces the crowdsourced GazeCapture dataset and trains iTracker, an end-to-end CNN for mobile gaze prediction. iTracker achieves robust gaze prediction on mobile devices and learns features that generalize to other datasets.
Problem
Existing eye-tracking solutions can be costly, require custom or invasive hardware, or be inaccurate under real-world conditions, limiting pervasive use.
Method
The paper builds GazeCapture through crowdsourcing and trains iTracker, an end-to-end CNN that predicts gaze from face and eye images without manually engineered features.
Results
iTracker robustly predicts gaze with errors as low as 1.04cm on mobile phones and 1.69cm on tablets, and its learned features generalize well to other datasets.
Takeaways & Limitations
The work advances eye tracking toward practical use on mobile devices without external attachments.
Takeaways & Limitations
The approach assumes that substantial variability in the training data is important for achieving high-accuracy calibration-free eye tracking.
Abstract
from arXiv · showhide
From scientific research to commercial applications, eye tracking is an important tool across many domains. Despite its range of applications, eye tracking has yet to become a pervasive technology. We believe that we can put the power of eye tracking in everyone's palm by building eye tracking software that works on commodity hardware such as mobile phones and tablets, without the need for additional sensors or devices. We tackle this problem by introducing GazeCapture, the first large-scale dataset for eye tracking, containing data from over 1450 people consisting of almost 2.5M frames. Using GazeCapture, we train iTracker, a convolutional neural network for eye tracking, which achieves a significant reduction in error over previous approaches while running in real time (10-15fps) on a modern mobile device. Our model achieves a prediction error of 1.71cm and 2.53cm without calibration on mobile phones and tablets respectively. With calibration, this is reduced to 1.34cm and 2.12cm. Further, we demonstrate that the features learned by iTracker generalize well to other datasets, achieving state-of-the-art results. The code, data, and models are available at http://gazecapture.csail.mit.edu.
1. Introduction
Eye tracking has broad applications but remains costly, hardware-dependent, or inaccurate in real-world settings. The paper addresses this gap with a crowdsourced dataset and an end-to-end CNN designed for robust mobile gaze prediction.
- Eye tracking supports human–computer interaction, medical diagnosis, psychological studies, and computer vision, but existing solutions can be costly, require custom hardware, or lack real-world accuracy.
- GazeCapture is introduced as the first large-scale eye tracking dataset captured through crowdsourcing.
- Crowdsourced GazeCapture contains almost 1500 subjects recorded under variable lighting and unconstrained head motion, addressing the limited scale of earlier datasets.
- The paper trains iTracker, an end-to-end convolutional neural network using eye and face crops without preexisting head-pose systems or manually engineered features.
- A smaller, faster iTracker network enables real-time mobile performance with minimal accuracy loss, while the overall system targets smartphones and tablets without external attachments.
- The work takes a significant step toward making eye tracking available on ordinary mobile devices.
2. Related Work
Prior eye-tracking methods include model-based geometric approaches and appearance-based approaches, while existing datasets often lack scale or sufficient variation. GazeCapture addresses these dataset limitations through crowdsourcing and varied recording conditions.
- Gaze estimation: Gaze estimation methods are categorized as model-based or appearance-based; model-based approaches infer gaze from eye geometry and can rely on corneal reflections or eye shape.
- Gaze estimation: Model-based approaches can suffer with low image quality and variable lighting conditions, which are relevant to mobile eye tracking.
- Gaze datasets: GazeCapture has approximately 30 times as many participants and 10 times as many frames as the largest existing datasets.
- Gaze estimation: GazeCapture is an appearance-based, end-to-end model that avoids hand-engineered features such as head pose or eye-center location.
- Gaze datasets: Crowdsourcing addresses the cost and limited variation of laboratory data while enabling variation in head pose, gaze points, and illumination.
3. GazeCapture: A Large-Scale Dataset
GazeCapture uses a crowdsourced mobile-app workflow to collect eye-tracking data that is scalable, reliable, and highly variable. The resulting dataset contains 1,474 subjects and 2,445,504 frames, with substantial variation in pose and camera placement.
- Scalability: Crowdsourcing combined with a custom iOS application and Amazon Mechanical Turk provided a scalable way to recruit participants and collect mobile eye-tracking data.The application recorded and uploaded gaze data while AMT supplied the workforce.
- Reliability: Reliability was supported by attention-guiding procedures, including airplane mode, pulsating fixation targets, and delayed recording after target movement.These procedures were designed to ensure workers fixated directly on displayed dots.
- Variability: The collection setup explicitly encouraged variability in pose, appearance, illumination, device orientation, and camera-to-user position.Participants moved their heads and phones, and changed device orientation during recording.
- Dataset scale: 1,474 subjects contributed 2,445,504 frames, including approximately 2.1M iPhone frames and 360k iPad frames.The dataset included 1,249 iPhone users and 225 iPad users recruited through multiple channels.
- Dataset characteristics: Compared with existing datasets, GazeCapture has a similar overall head-pose distribution but a significantly larger proportion of outliers and greater camera-position variation.The dataset includes frames captured with the camera below or above the screen, which can support training and evaluation.
4. iTracker: A Deep Network for Eye Tracking
iTracker is an end-to-end CNN that predicts gaze from a single image using eye, face, and face-grid inputs. It uses a unified prediction space across devices and orientations, with device- and orientation-specific fine-tuning and a smaller mobile-oriented network.
- Model design: iTracker predicts gaze end-to-end from a single image without manually engineered features such as head pose.The model uses deep convolutional neural networks to leverage the large-scale GazeCapture dataset.
- Model inputs: The network combines face imagery and location information with separate left- and right-eye inputs to infer head pose, eye pose, and gaze.Individual eye inputs provide higher-resolution views for identifying subtle eye changes.
- Prediction space: A unified prediction space enables one model to train across multiple devices and orientations by representing screen dots relative to the camera.Direct screen-coordinate prediction would not be meaningful across changing device configurations.
- Adaptation: Fine-tuning the jointly trained network for each device and orientation was helpful, especially because mobile-phone and tablet data were unbalanced.The fine-tuned model is denoted iTracker∗.
- Mobile deployment: Dark knowledge was used to reduce model complexity, computation time, and memory footprint for resource-constrained mobile devices.The mobile-oriented network uses tighter 80 × 80 eye crops from facial landmark detections.
5. Experiments
The experiments evaluate iTracker on unconstrained mobile eye tracking, calibration, cross-dataset transfer, and component or data-scale effects. iTracker achieves low error, benefits from calibration and augmentation, generalizes to TabletGaze, and improves with more diverse subjects.
- Unconstrained Eye Tracking: ∼2cm average error without calibration is reduced to 1.8cm through calibration in the overall evaluation.
- Unconstrained Eye Tracking: 1.53cm and 2.38cm are the lowest reported errors for mobile phones and tablets, respectively, substantially outperforming the baseline.Train and test augmentation both reduce prediction error, while temporal averaging lowers dot error.
- Eye Tracking with Calibration: 1.34cm and 2.12cm errors are achieved on mobile phones and tablets, respectively, when all 13 calibration points are used.Performance decreases slightly with fewer calibration points, likely because the subject-specific SVR overfits.
- Cross-Dataset Generalization: 2.58cm error on TabletGaze shows that iTracker features significantly outperform existing approaches on another dataset.Features are extracted from iTracker and used with support vector regression for cross-dataset gaze prediction.
- Analysis: All three inputs—eyes, face, and face grid—contribute to performance, although face plus face grid can approach the full model.The ablation suggests a potentially more efficient approach that omits eye crops.
- Analysis: Increasing the number of subjects decreases error more quickly than increasing samples per subject when total sample count is held constant.The result indicates that subject variability is important for training accurate gaze models.
6. Conclusion
The paper introduces GazeCapture and iTracker as an end-to-end mobile eye-tracking solution. It reports low errors on phones and tablets and shows that the learned features generalize to existing datasets.
- GazeCapture is introduced as the first large-scale mobile eye-tracking dataset, collected through crowdsourcing.
- iTracker is a deep convolutional neural network trained with GazeCapture to predict gaze on mobile devices.
- 1.04cm and 1.69cm errors are achieved on mobile phones and tablets, respectively.
- The learned features generalize well to existing datasets and outperform state-of-the-art approaches by a large margin.