Source-linked AI summary
TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R. Kulkarni, Jianxiong Xiao
TL;DR
Traditional eye tracking is expensive and difficult to scale, limiting gaze-data collection for saliency research. The paper introduces a webcam-based, game-mediated AMTurk system that collects large-scale gaze data, with accuracy comparable to lab tracking at lower cost and effort. It also demonstrates the approach on a large natural-image saliency dataset and varied visual stimuli.
Problem
Traditional eye tracking requires specialized laboratory equipment and individual participant sessions, making gaze-data collection expensive, time-consuming, and difficult to scale.
Method
The paper combines browser-based webcam gaze prediction with calibration, validation, and game-based quality control for crowdsourced experiments on AMTurk.
Results
The system obtains AMTurk eye-tracking data with accuracy comparable to traditional lab data, at lower cost and with little researcher effort.
Takeaways & Limitations
The system supports large-scale gaze-data collection for natural images and can generalize to varied visual stimuli and tasks.
Takeaways & Limitations
Crowdsourced data collection remains constrained by uncontrolled environments and participant attention, which can affect tracking and saliency-map quality.
Abstract
from arXiv · showhide
Traditional eye tracking requires specialized hardware, which means collecting gaze data from many observers is expensive, tedious and slow. Therefore, existing saliency prediction datasets are order-of-magnitudes smaller than typical datasets for other vision recognition tasks. The small size of these datasets limits the potential for training data intensive algorithms, and causes overfitting in benchmark evaluation. To address this deficiency, this paper introduces a webcam-based gaze tracking system that supports large-scale, crowdsourced eye tracking deployed on Amazon Mechanical Turk (AMTurk). By a combination of careful algorithm and gaming protocol design, our system obtains eye tracking data for saliency prediction comparable to data gathered in a traditional lab setting, with relatively lower cost and less effort on the part of the researchers. Using this tool, we build a saliency dataset for a large number of natural images. We will open-source our tool and provide a web server where researchers can upload their images to get eye tracking results from AMTurk.
1. Introduction
Traditional eye tracking is costly and difficult to scale, limiting gaze-data collection. The paper addresses this gap with a webcam-based AMTurk system designed for scalable, accurate saliency data collection at lower cost and effort.
- Eye-tracking data collection traditionally requires laboratory equipment and individually tracked participants, making it time-consuming, expensive, and difficult to scale.
- The paper develops a webcam-based eye-tracking game for large-scale crowdsourcing on Amazon Mechanical Turk.The design targets ubiquitous hardware, easy setup, sufficient saliency-data quality, and real-time performance.
- The system combines browser-based tracking, game scenarios, an experiment framework, and a large gaze database to motivate careful participation and collect gaze data remotely.
- The system obtains AMTurk eye-tracking data with satisfactory accuracy compared with traditional lab data, at lower cost and with little researcher effort.
2. Related work
Prior gaze-collection approaches span specialized eye trackers, proxy crowdsourcing methods, and laboratory datasets. These alternatives differ in real-time fidelity, scalability, and the number of images or observers they can support.
- Feature-based gaze methods use detailed eye features and geometric or eyeball models, but require controlled environments and specialized imaging hardware.
- Appearance-based methods reduce calibration demands, but small head movements can cause gaze-prediction drift during experiments.
- Crowdsourced alternatives collect proxy viewing data rather than real-time gaze data.
- Self-reported gaze can achieve comparable accuracy to in-lab tracking for some tasks, but yields only a single fixation per report and is unsuitable for full video tracking.
- Mouse-contingent tracking produces saliency maps similar by shuffled AUC, yet mouse movements are slower and its inferred fixations may not match eye-tracking fixations.
- Existing laboratory datasets vary by stimuli, subjects, views, equipment, and task, with dataset construction trading off image count against observers per image.
3. Large-scale crowd-sourcing eye tracking
The system addresses uncontrolled crowdsourcing conditions through browser-based gaze prediction, calibration and validation, quality-control games, and procedures for handling attention and tracking failures. It supports multiple stimuli and produces large-scale saliency data.
- 3.1. Design considerations: Crowdsourced experiments must handle uncontrolled lighting, head pose, viewing distance, hardware limitations, browser resources, and participant attention.
- 3.2. Webcam based gaze prediction algorithm: The system trains user-task-specific gaze models from calibration points and uses facial landmark tracking, eye-region features, and regression for online prediction.
- 3.3. Game design for crowdsourcing: The system estimates fixations from raw gaze data and computes saliency maps that can evaluate individual annotated objects in SUN images.
- 3.3. Game design for crowdsourcing: Experiments interleave calibration, image-viewing sessions, updated model training, and validation using online predictions to detect tracking failures.
- 3.2. Webcam based gaze prediction algorithm: Cross-validation rejects calibration outliers before a refined support-vector-regression model is fitted.
- 3.3. Game design for crowdsourcing: Short tasks can reduce the need for recalibration; free-viewing images typically require only 3–5 seconds each, with validation retained for failure detection.
4. Evaluation
The system was evaluated against commercial eye tracking for gaze prediction, fixation estimation, and saliency-map quality. It achieved low gaze-prediction error, while AMTurk fixation data produced saliency maps comparable to state-of-the-art models but showed lower agreement with Judd subjects.
- Gaze prediction accuracy: 1.06° median error was measured across 33 uniformly sampled screen positions for one subject.The authors describe this as comparable to a state-of-the-art webcam-based gaze prediction algorithm.
- Gaze prediction accuracy: 1.32° average error was measured during free viewing across four Judd images and three subjects.The webcam system sampled at 30 Hz, while Eyelink 1000 provided the 1000 Hz comparison signal.
- Fixation estimation and saliency maps: The meanshift fixation detector was evaluated on 1,000 randomly sampled Judd images using position error, precision, recall, and F-score.Figure 9 varies kernel size while reporting these four evaluation measures.
- Fixation estimation and saliency maps: AMTurk subjects were predicted less well than Judd subjects by leave-one-out saliency maps built from the other subjects’ fixations.The authors suggest higher AMTurk positional noise and demographic differences as possible explanations.
- Fixation estimation and saliency maps: AMTurk fixation maps performed similarly to top-performing state-of-the-art saliency models across several AUC metrics when predicting Judd subjects.The comparison used AMTurk fixation data as the saliency map and evaluated it against 15 Judd subjects.
5. iSUN: A large dataset for natural scenes
The authors use their crowdsourced eye-tracking tool to build iSUN, a large free-viewing dataset of fully annotated natural scenes. The dataset supports object-saliency analysis and substantially expands image-scale eye-tracking data.
- Dataset construction: 20,608 fully annotated natural-scene images from SUN were collected with free-viewing eye-tracking data.At submission, every image had a saliency map based on an average of three observers per image.
- Dataset scale: iSUN is described as the largest compared free-viewed image dataset by number of images.Its images are fully annotated scenes from the SUN database.
- Object saliency: Object-saliency statistics measure overlap between thresholded saliency maps and binary masks for object categories.For example, the top 5% salient image area overlaps a head region by around 0.45 on average in the cited example.
6. Conclusions
The paper presents a webcam-based system for crowdsourced eye tracking on Amazon Mechanical Turk and compares it with commercial eye trackers. It also applies the system to natural-scene images and proposes broader use across visual stimuli and tasks.
- Conclusions: The proposed webcam-based system collects crowdsourced eye-tracking data from Amazon Mechanical Turk and is evaluated against commercial eye trackers.The paper reports free-viewing data collection for a large number of natural-scene images.
- Conclusions: The system can be generalized to different visual stimuli and eye-tracking tasks.The authors characterize it as a tool for collecting large-scale eye-tracking data for the research community.