PaperlayerView exact revision on arXiv

Source-linked AI summary

Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction

Moritz Kassner, William Patera, Andreas Bulling

arXiv:1405.0006v1cs.CVcs.HC

TL;DR

Commercial head-mounted eye trackers are costly and closed, limiting access and user-driven extension. Pupil presents an open source headset and software platform for mobile and pervasive eye tracking, achieving 0.6 degree gaze accuracy with 0.08 degree precision and 0.045 seconds of pipeline latency.

  • Problem

    Commercial head-mounted eye trackers are expensive and closed source, limiting mobile eye tracking to expert users and inhibiting customization and extension.

  • Method

    Pupil combines a wearable headset, open source real-time eye tracking software, and modular hardware and software designed for accessibility and extensibility.

  • Results

    0.6 degree of accuracy and 0.08 degree precision were achieved under ideal conditions, with 0.045 seconds total eye-processing pipeline latency.

  • Takeaways & Limitations

    Pupil contributes an accessible, affordable, and extensible tool for pervasive eye tracking research and has been used across a wide range of disciplines.

  • Takeaways & Limitations

    The system's reported accuracy is constrained by human fixation accuracy of 0.5deg, while broader system limitations include parallax error and reduced tracking robustness in IR-rich environments.

Abstract

from arXiv · show

Commercial head-mounted eye trackers provide useful features to customers in industry and research but are expensive and rely on closed source hardware and software. This limits the application areas and use of mobile eye tracking to expert users and inhibits user-driven development, customisation, and extension. In this paper we present Pupil -- an accessible, affordable, and extensible open source platform for mobile eye tracking and gaze-based interaction. Pupil comprises 1) a light-weight headset with high-resolution cameras, 2) an open source software framework for mobile eye tracking, as well as 3) a graphical user interface (GUI) to playback and visualize video and gaze data. Pupil features high-resolution scene and eye cameras for monocular and binocular gaze estimation. The software and GUI are platform-independent and include state-of-the-art algorithms for real-time pupil detection and tracking, calibration, and accurate gaze estimation. Results of a performance evaluation show that Pupil can provide an average gaze estimation accuracy of 0.6 degree of visual angle (0.08 degree precision) with a latency of the processing pipeline of only 0.045 seconds.

INTRODUCTION

Pupil addresses the cost, closed-source constraints, and limited extensibility of commercial mobile eye tracking with an accessible open source platform. It combines a wearable headset, software framework, and modular design for pervasive eye tracking research.

  • Pervasive eye tracking denotes continuous eye monitoring and analysis usable anywhere and anytime in everyday life.
  • Commercial head-mounted eye trackers are expensive and closed source, restricting mobile eye tracking mainly to expert users and inhibiting customization and extension.
  • Accessibility is defined through open source components, modular hardware and software, documentation, user support, affordability, and flexibility for future changes.
  • Pupil was developed as an accessible, affordable, and extensible mobile eye tracking headset and open source software framework.
  • Pupil uses a wearable headset with scene and infrared eye cameras connected by USB to a computer running software for real-time detection, gaze mapping, and recording.
  • The platform uses modular hardware and software, consumer USB cameras and computing hardware, and open source development to support user modification.

PUPIL HEADSET DESIGN AND HARDWARE

The Pupil headset uses a modular, lightweight frame with adjustable camera mounts to accommodate users, cameras, and tracking configurations. Open interfaces support customization and community-developed variations.

  • The headset comprises three modules: a frame, scene camera mount, and eye camera mount.
  • The 9g frame was refined using a human-head scan and finite element deformation analysis to accommodate physiological variation and maintain camera alignment.
  • The frame supports binocular variants with four mount points and versions without nose-bridge support for prescription eyeglasses.
  • Open source mount interfaces let users develop mounts for cameras of their choice, with open source mounts hosted in a Git repository.
  • The scene camera mount provides radial adjustment within a 90 degree range for different tasks and users.
  • The articulated eye camera mount accommodates variation in eye and face geometry through sliding and ball-joint adjustments.

Cameras

Pupil combines interchangeable USB cameras with infrared illumination and user-supplied computing platforms. Its camera choices support portability, visual coverage, extensibility, and further computer-vision processing.

  • Pupil uses UVC-compliant USB cameras, allowing users to substitute compatible cameras, use other UVC software, and add eye or scene cameras.
  • The eye camera is a lightweight 10x45x7 mm package capturing up to 800x600 pixels at 30Hz to reduce visual obstruction and headset weight.
  • Dark-pupil detection requires eye video in a specific infrared spectrum, supplied by an IR bandpass filter and an 860nm surface-mounted IR LED.
  • The scene camera captures a 30Hz video stream with a 90 degree diagonal field of view and maximum 1920x1080 resolution.
  • Pairing the headset with a laptop, desktop, or tablet enables lightweight mobile use and specialized applications including network streaming, geotagging, and multi-user synchronization.

PUPIL SOFTWARE

Pupil software combines real-time capture and processing with visualization tools and a pupil-detection pipeline based on edge filtering, contour extraction, and ellipse fitting.

  • PUPIL SOFTWARE: Pupil Capture processes two or more camera streams in real time, while Pupil Player replays and visualizes recorded video and gaze data.The software is open source and platform-independent across Linux, MacOS, and Windows.
  • Pupil Detection Algorithm: The pupil detector locates dark pupils in infrared eye images without relying on corneal reflections, including for users wearing contact lenses or eyeglasses.Its design is continually improved using user-submitted eye-camera videos.
  • Pupil Detection Algorithm: The detection pipeline converts eye images to grayscale, detects and filters edges, extracts contours, forms candidate ellipses, and selects a final ellipse through augmented combinatorial search.Candidates are evaluated using supporting-edge fit and the ratio of supporting edge length to ellipse circumference.
  • PUPIL SOFTWARE: Pupil Capture presents scene-camera video, gaze position, eye-camera video, camera controls, and plugin interfaces in separate GUI windows.Plugins can also visualize the pupil-detection algorithm.
  • Pupil Detection Algorithm: 80% detection rate is achieved at a 2-pixel error threshold, increasing to 90% at 5 pixels when the extreme-angle p1-right data are excluded.The comparison included Pupil, Swirski et al., the ITU gaze tracker, and Starburst on the Swirski benchmark dataset.

Gaze Mapping

Pupil maps pupil positions from eye space to scene space using adjustable bivariate polynomials whose parameters are obtained through several calibration routines.

  • Gaze Mapping: Pupil maps eye-space pupil positions to scene space with a transfer function consisting of two bivariate polynomials of adjustable degree.The polynomial parameters are user-specific and obtained through calibration.
  • Gaze Mapping: Calibration options include 9-point screen markers, manually moved concentric markers, natural scene features tracked by optical flow, and camera intrinsic calibration.The routines provide different ways to collect calibration samples or estimate camera parameters.
  • Gaze Mapping: Calibration and mapping functions are abstracted so their underlying models can be modified or replaced.This preserves flexibility for changing the mapping approach.
  • Gaze Mapping: Pupil Capture maps gaze into planar reference-surface coordinates using homographic transformations between the scene-camera plane and reference-surface plane.The system detects planar reference surfaces using a set of 64 markers.

Streaming

Pupil supports real-time data streaming, modular plugins, recording, and gaze visualization, with accuracy testing defined using established spatial accuracy and precision metrics.

  • Streaming: Pupil Capture sends real-time gaze, pupil, and plugin-generated data via ZeroMQ to other applications and network-enabled devices.This extends access to processed tracking data beyond the capture application.
  • Plugin Structure: Plugins can launch in world or eye processes, expose GUI elements, and replace even standard functions such as video recording.The modular structure supports runtime method testing and software extension.
  • Streaming: The world window displays scene-camera video and controls, while the eye window displays eye-camera video and controls for pupil detection and visualization.These windows provide separate interfaces for scene and eye processing.
  • Plugin Structure: Pupil Player visualizes the same gaze and scene data using scan paths, gaze circles, polylines, crosses, and light points.The Scan Path plugin uses optical flow to compensate for ego-motion, while other plugins encode gaze positions graphically.
  • Spatial Accuracy and Precision: Accuracy is defined as average angular offset from fixation targets, while precision is the RMS angular distance between successive samples.Both metrics are reported in degrees of visual angle.
  • Spatial Accuracy and Precision: The spatial test was conducted indoors without direct sunlight, and performance in infrared-intense environments requires separate evaluation.The authors describe perfect detection in such environments as an unsolved problem for infrared-based pupil detection.

Error Sources

The accuracy and precision test captures aggregate gaze-estimation error rather than attributing error to individual sources. Potential contributors include physiological, human, calibration, mapping, hardware, and detection factors, while several are negligible or disregarded under the test conditions.

  • Error Sources: The test exposes total gaze-estimation error, but does not attribute that error to individual sources.The authors explicitly state that individual attribution is outside the test’s scope.
  • Error Sources: Other possible contributors include oculomotor noise, human test error, headset movement, image artifacts, pupil-detection error, and reference-data errors.Several reference, conversion, and artifact sources are considered negligible or filtered under the test conditions.
  • Error Sources: 0.5deg human fixation accuracy is a major factor given the achieved system accuracy.The passage identifies fixation limitations as especially important in light of the measured result.
  • Error Sources: Gaze-mapping error may arise from suboptimal fitting parameters and shortcomings in the mapping model.The authors identify both calibration-related fitting parameters and the mapping model as contributing factors.
  • Error Sources: Parallax from changes in target distance can be substantial in real-world applications but is disregarded in this test scenario.The test setup treats this source as negligible despite its potential practical importance.

Results

Under ideal conditions, Pupil achieves 0.6 degrees of accuracy and 0.08 degrees of precision.

  • Results: 0.6deg accuracy and 0.08deg precision are achieved under ideal conditions.These are the reported performance values for the accuracy and precision test.

Temporal Accuracy, Latency and Precision

Pupil addresses temporal synchronization using hardware timestamps when available and software timestamps as a fallback. Hardware timestamps provide low jitter, whereas software timestamps introduce measurable offsets and depend on camera and driver behavior.

  • Temporal Accuracy, Latency and Precision: Timestamping is needed to synchronize independent video streams and correlate them with external experiment or sensor data.The system seeks timestamps close to the camera sensor exposure time.
  • Temporal Accuracy, Latency and Precision: Hardware timestamps are taken at the beginning of sensor exposure and are used when cameras provide valid hardware timestamps.Pupil Capture implements this option on supported Linux configurations and cameras.
  • Temporal Accuracy, Latency and Precision: 0.0004s world-camera jitter and 0.0001s eye-camera jitter are measured from 1400 successive frame times.Jitter is calculated as the standard deviation of successive exposure times.
  • Temporal Accuracy, Latency and Precision: Software timestamps are less accurate and precise because they are recorded after exposure, transfer, and decompression with indeterminate delays.Their accuracy and precision depend on the camera and video-capture driver.
  • Temporal Accuracy, Latency and Precision: +0.119s world-camera and +0.038s eye-camera software-timestamp offsets are measured against hardware timestamps.The corresponding standard deviations are 0.003s for world and 0.002s for eye timestamps on the Linux test machine.

System Latency

Pupil separates eye and world processing pipelines to characterize latency and prioritize recent eye data for real-time use. The reported latencies and CPU results indicate operation on both tested specialized and portable computing platforms, with cross-platform requirements not fully established.

  • System Latency: The eye pipeline is about one-third the latency of the world pipeline, so eye information is broadcast as soon as available for real-time use.This sacrifices synchronicity for recency in real-time applications, without preventing accurate temporal correlation afterward.
  • System Latency: 0.045s eye-pipeline latency is measured from sensor exposure to pupil-position availability.The measurement covers 1400 samples and has a 0.003s standard deviation.
  • System Latency: 0.124s world-pipeline latency is measured from sensor exposure to network broadcast of pupil, gaze, and reference-surface data.The measurement covers 1200 samples and has a 0.005s standard deviation.
  • Minimum Hardware Requirements: 90% maximum dual-CPU load is observed while recording, detecting pupils at 30 fps, and capturing the world camera at 24 fps.The test uses an 11-inch 2010 MacBook Air with 2 GB RAM and a dual-core Intel Core2Duo SU9400 CPU.
  • Minimum Hardware Requirements: The Linux performance test cannot be generalized to Windows, and Windows hardware requirements were not established.Requirements were found similar for MacOS 10.8 and above.

DISCUSSION

Pupil is presented as an actively developed contribution to pervasive eye tracking, with a growing community and planned improvements. The discussion identifies current limitations and directs future work toward mobility, tracking, mapping, and usability.

  • Pupil is used across a wide range of disciplines and has developed a community of researchers and developers.
  • Current limitations include parallax error and reduced tracking robustness in IR-rich environments.
  • Future development will address hardware and software in parallel.
  • Planned next steps include improving mobility, adding real-time pose tracking and scene mapping, simplifying user experience, and improving pupil tracking.
  • The project provides public hardware, software, user, and developer resources for the Pupil platform.These resources include the headset, source repository, application bundles, user and developer guides, and a user forum.
Loading 1405.0006v1…