PaperlayerView exact revision on arXiv

Source-linked AI summary

Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities

Rakshit Kothari, Zhizhuo Yang, Christopher Kanan, Reynold Bailey, Jeff Pelz, Gabriel Diaz

arXiv:1905.13146v1cs.CV

TL;DR

Gaze coordination in natural settings has received limited study, and unconstrained tracking needs both eye-and-head instrumentation and event detectors. This paper introduces a naturalistic dataset and benchmarks Random Forest and recurrent neural network classifiers; fixation and saccade detection approached human performance, while pursuit remained difficult.

  • Problem

    Natural gaze studies often constrain head movement, and tracking coordination in unconstrained settings requires both eye-and-head instrumentation and event detectors.

    Source: §1 · p. 1

  • Method

    The authors collected the Gaze-in-Wild dataset using eye-tracking glasses, an IMU, and a stereo camera, then benchmarked Random Forest and recurrent neural network classifiers.

    Source: §1 · p. 1, §5.1 · pp. 13–14

  • Results

    Classifiers approached human performance for gaze fixations and saccades but had difficulty distinguishing gaze pursuit without additional contextual information.

    Source: §7 · p. 18

  • Takeaways & Limitations

    Gaze-in-Wild provides manually labelled natural-task data and a benchmark for classifying gaze events during unconstrained behavior.

    Source: §7 · p. 18

    Low-cost IMU drift and helmet slippage affect head-pose accuracy, particularly the azimuth and elevation velocity components.

    Source: §6.5 · p. 17

Abstract

from arXiv · show

The interaction between the vestibular and ocular system has primarily been studied in controlled environments. Consequently, off-the shelf tools for categorization of gaze events (e.g. fixations, pursuits, saccade) fail when head movements are allowed. Our approach was to collect a novel, naturalistic, and multimodal dataset of eye+head movements when subjects performed everyday tasks while wearing a mobile eye tracker equipped with an inertial measurement unit and a 3D stereo camera. This Gaze-in-the-Wild dataset (GW) includes eye+head rotational velocities (deg/s), infrared eye images and scene imagery (RGB+D). A portion was labelled by coders into gaze motion events with a mutual agreement of 0.72 sample based Cohen's $\kappa$. This labelled data was used to train and evaluate two machine learning algorithms, Random Forest and a Recurrent Neural Network model, for gaze event classification. Assessment involved the application of established and novel event based performance metrics. Classifiers achieve $\sim$90$\%$ human performance in detecting fixations and saccades but fall short (60$\%$) on detecting pursuit movements. Moreover, pursuit classification is far worse in the absence of head movement information. A subsequent analysis of feature significance in our best-performing model revealed a reliance upon absolute eye and head velocity, indicating that classification does not require spatial alignment of the head and eye tracking coordinate systems. The GW dataset, trained classifiers and evaluation metrics will be made publicly available with the intention of facilitating growth in the emerging area of head-free gaze event classification.

1 Introduction

Studying gaze in natural behavior requires both eye-and-head tracking and event detectors, because prior work often constrained head movement and existing classifiers were designed for fixed-head settings. The paper adopts an exocentric taxonomy that distinguishes fixation, pursuit, and gaze shifts by their functional role and world-frame context.

1 Introduction

  • Unconstrained eye-and-head coordination requires both joint tracking instrumentation and event detectors, while prior studies often suppressed head movement or restricted target motion.Existing gaze classifiers were traditionally designed for fixed-head situations.Source: §1 · pp. 1–2
  • The study develops event detectors for natural behavior alongside a custom eye-and-head tracker and the Gaze-In-Wild dataset.A portion of the collected data was hand-labelled and used for supervised training and assessment.Source: §1 · pp. 1–2

1.1 Classification scheme and nomenclature

  • The taxonomy defines gaze fixation as maintaining stable gaze on a stationary world-frame object, including compensatory eye-and-head movements.Fixation includes tremors, drifts, microsaccades, and vestibular-ocular compensation.Source: §1.1 · pp. 2–3
  • Gaze pursuit tracks a moving world-frame object using the eyes, head, or both, and may be interrupted by catch-up saccades.Pursuit can be difficult to distinguish from fixation when its signal is small relative to retinal-image stabilization.Source: §1.1 · p. 3
  • Gaze shifts are rapid eye-only movements to a new world location, while the taxonomy classifies events by eye function and object motion in an exocentric frame.Vestibular and optokinetic stabilization mechanisms alone do not determine whether an event is fixation or pursuit.Source: §1.1 · pp. 2–3

2 The Gaze-in-wild dataset

The dataset was designed to capture eye-and-head coordination during naturalistic activities. Nineteen participants completed up to four tasks selected to elicit varied gaze behavior and head and eye poses.

2 The Gaze-in-wild dataset

  • Nineteen participants performed up to four activities while wearing an eye tracker, sensor-equipped hardhat, and laptop backpack.Participants received standardized task instructions and completed calibration before each task.Source: §2 · p. 4
  • Indoor navigation elicited coordinated gaze during walking, including shifts to objects such as posters, signboards, and passersby.Participants walked an indoor corridor loop twice.Source: §2 · p. 4
  • Ball catching was designed to elicit pursuit, although participants also used fixations and predictive catch-up or look-ahead saccades.The experimenter varied throws by bouncing, passing, or rolling the ball.Source: §2 · p. 4
  • Object search elicited head-free visual search, while tea making tested a classic paradigm with some relevant information falling outside the field of view.Search participants located and counted geometrically shaped objects along a corridor circuit.Source: §2 · p. 4

2.1 Hardware setup and error categorization

The setup combines eye tracking, head-motion sensing, and stereo scene capture to record gaze in natural tasks. Each component has a distinct measurement role, with IMU drift and helmet slippage affecting head-pose accuracy.

2.1 Hardware setup and error categorization

  • The mobile setup combined binocular eye-tracking glasses, a hardhat-mounted IMU, and a ZED stereo camera, with the camera providing depth mapped to the eye-tracker coordinate system.The single-IMU arrangement avoided a body-worn device; the setup weighed 700 g excluding the laptop.Source: §2.1 · p. 4

2.1.1 Pupil labs eye tracking glasses (ETG)

  • The eye tracker estimates 3D gaze vectors in the Eye-In-Head coordinate system and provides a confidence value used to discard samples below 0.3.The tracker maps eye features to point-of-regard positions on the world camera.Source: §2.1.1 · pp. 4, 6

2.1.2 Inertial Measurement Unit (IMU)

  • The MPU-6050 estimates pose relative to its initial position, but accumulated IMU drift means longer recordings may require external correction or a secondary sensor.Short recordings were calibrated to within 7° of pose error; frequent head turns can also increase error.Source: §2.1.2 · p. 6

2.1.3 ZED Stereo camera

  • The ZED stereo camera supplies a 1080p point cloud at 30 Hz, with depth error below 0.5 m at approximately 10 m.Beyond that distance, the scene was treated as infinity.Source: §2.1.3 · p. 6

2.2 System calibration

The system calibration combines eye-tracker calibration, temporal alignment, and rotational alignment of the eye and head coordinate systems. A checkerboard-based multiview calibration registers stereo-camera depth with the eye-tracker view.

2.2 System calibration

  • Routine 1 uses red-marker fixations to calibrate the eye tracker; Routine 2 induces VOR to align hardware in a common world coordinate system.The checkerboard corners also support two-way multiview calibration between the ZED stereo camera and eye-tracker world camera.Source: §2.2 · p. 6, Figure 5 · p. 7

2.2.1 Pupil Labs eye tracker calibration

  • Within 10° of the calibration-pattern center, eye-tracker calibration error was within 1°.Routine 1 fitted a polynomial mapping between 2D pupil position and gaze PoR.Source: §2.2.1 · p. 7

2.2.2 Temporal alignment

  • Normalized cross-correlation of angular-velocity traces aligns the IMU, eye tracker, and ZED camera, removing temporal offsets including biological VOR latency.When ZED pose estimates are poor, checkerboard-corner velocity estimates provide the ZED signal for alignment.Source: §2.2.2 · p. 7

2.2.3 ETG-IMU calibration

  • Eye and head coordinate systems are rotationally aligned to a world frame, with matrices tuned to minimize head-compensated cyclopean eye velocity.The approach does not correct the 1–2 cm translation between the IMU and cyclopean gaze origin.Source: §2.2.3 · p. 7

2.2.4 ETG-ZED calibration

  • Checkerboard corners in synchronized camera images estimate extrinsic transforms that register ZED depth in the eye-tracker view for 3D PoR estimation.A ray from the eye-tracker world-camera center is intersected with the transformed point cloud to obtain PoR in millimeters.Source: §2.2.4 · pp. 7–8

2.3 Operations

The operations section describes angular-velocity computation and filtering, alongside a GUI for inspecting and labeling synchronized eye, head, and scene data.

2.3 Operations

  • Angular velocity is computed with a modified Two-Point Central Difference algorithm, using angular displacement across samples n−1 and n+1.At fixed sampling rate fs, ωv = fsδθ/2.Source: §2.3 · p. 8
  • A Kaiser low-pass filter suppresses frequencies above 58±2 Hz, while bilateral filtering reduces noise while preserving peak saccade velocity.The bilateral filter uses a 50 ms window; small-angle approximation incurs 1% measurement error at 14°.Source: §2.3 · pp. 8–9
  • The labeling GUI presents eye and head velocity traces with PoR-overlaid scene imagery, event controls, and temporal-window and label-shift sliders.It also provides eye imagery, confidence scores, depth-view toggling, and ten-second clip recording.Source: Figure 7 · p. 9

3 Labelling

The GW dataset was manually annotated by trained coders, with event-label agreement assessed and labels cleaned using duration and separation rules. The annotation scheme distinguishes event categories for subsequent analysis.

3 Labelling

  • Five annotators labeled over 140 minutes of head-free gaze data, including approximately 20,000 fixations, 18,000 saccades, 1,200 pursuits, and 4,000 blinks.Uncertain sequences and samples with confidence below 0.3 were excluded; stationary fixation and fixation under translation were labeled as separate categories.Source: §3 · p. 9
  • Annotators’ average sample-based Cohen’s κ was 0.72 (σ=0.0336, median=0.731).The passage notes that agreement varied significantly by gaze-event type.Source: §3 · p. 9

3.1 Training labellers

  • Annotators received lectures, jointly discussed a small shared subset, then labeled independently with weekly meetings to discuss uncertain data.The shared subset was analyzed and discussed with the authors before manual annotation began.Source: §3.1 · p. 10

3.2 Data cleaning and post-processing

  • Post-processing merged nearby fixations, removed fixations under 50 ms, saccades over 150 ms, and all labeled events under 10 ms.Fixations less than 0.5° apart and within 75 ms were combined.Source: §3.2 · p. 10

4 Error metrics

Sample-level measures can miss temporal structure and event-level errors, motivating an event-based approach. The section introduces ELC, which matches event transitions and supports cross-category evaluation while measuring temporal alignment.

4 Error metrics

  • ELC addresses shortcomings of prior event metrics by matching transition points, retaining cross-category errors, and measuring event alignment.Its window size sets timing tolerance, with ±25 ms for gaze shifts and ±35 ms for other events; larger windows may suit relaxed onset and offset conditions.Source: §4 · pp. 11–13
  • The EER measure does not match events, treats all event sequences as strings, and provides no insight into temporal offsets; it also returns only one overall rating.Source: §4 · p. 11
  • Figure 8 depicts labeler sequences, sample-level comparison, and event-level comparison, with unmatched regions labeled by misclassification type.Source: Figure 8 · p. 12
  • ELC matches reference transition points to eligible test transitions within a window; an event matches only when both endpoints match.Onsets are matched by the event type to their right, offsets by the type to their left; unmatched gaps are treated as none.Source: §4 · p. 12
  • For matched events, ELC reports endpoint-offset distance and overlap ratio, then averages matched transition times to correct timing offsets before forming a confusion matrix.The confusion matrix supports Cohen’s κ and F1; detached events may be treated as matches under application-specific strictness, which inflates performance scores.Source: §4 · pp. 12–13
  • Because ELC matching is asymmetric, applying it in both directions provides an average measure and can reveal event merges in the testing sequence.Source: §4 · p. 13

5 Machine learning for gaze event classification

The paper trains moving-window and recurrent models on eye and head sensor data for gaze-event classification. Event-based results favor biRNN overall, while sample-based results and feature ablations reveal differences by event type and input.

5 Machine learning for gaze event classification

  • The classifiers use temporally discrete eye-tracker and IMU data, combining stationary and translational fixations into one gaze-fixation class.Source: §5 · p. 13

5.1 Classification models

  • RF classifies each sample from aggregated neighboring-window features, while forward and bidirectional GRU-based RNNs learn temporal patterns from velocity streams.The forward model uses past and present data for real-time prediction; the bidirectional model also uses future data for offline processing.Source: §5.1 · p. 13, Figure 9 · p. 14

5.2 Input features

  • Window features combine eye and head angular displacement, velocity deviation, confidence weights, and absolute and directional velocity measurements.The RNN input consists of six eye and head velocity features; its fully connected layers feed a stack of GRU units.Source: §5.2 · p. 14, §5.1 · p. 14

5.3 Results

  • With sample-based metrics, large-window RF outperforms RNN for saccade detection but performs poorly on pursuit samples; all classifiers struggle to distinguish pursuits.RF performance improves with larger windows, with diminishing returns at 30 ms and above.Source: §5.3 · p. 14, Figure 10 · p. 15
  • biRNN outperforms RF on all reported event-based measures, although RF achieves better l2 scores for tighter fits around saccades.The biRNN also produces more detached events than RF; these are omitted from measures to avoid inflating scores.Source: §5.3 · pp. 14, 16, Table 6 · p. 15, Table 7 · p. 15

5.4 Ablation study

  • Removing head information leaves fixation and saccade detection unchanged but reduces pursuit κ by 25% (0.40 to 0.29); absolute velocity alone still provides reasonable classification certainty.Some pursuit events remain detectable using eye-only information.Source: §5.4 · p. 16, Table 8 · p. 16

6 Discussion

The discussion reports near-human fixation and saccade classification but substantially weaker pursuit classification, and examines how head motion, task context, and evaluation choices shape these findings. It also describes limitations from IMU drift and event-metric design.

6 Discussion

  • The discussion frames the work as a labeled in-the-wild dataset, multi-annotator labeling, and a benchmark of Random Forest and recurrent neural network classifiers using common metrics.ELC was introduced to address timing-offset issues in existing event metrics, and classifier baselines used leave-one-subject-out evaluation with human labels as the gold standard.Source: §6 · p. 16

6.1 Lower gaze pursuit classification performance by classifiers

  • Pursuit classification reached sample κ 0.40 versus average human performance of 0.62, while gaze fixation reached 0.72 versus 0.65 and saccade reached 0.75 versus 0.70.The authors attribute fixation–pursuit confusion to low-velocity ambiguity, orientation errors, and the lack of scene imagery with overlaid gaze PoR in their classifiers.Source: §6.1 · p. 16

6.2 Head tracking: A pursuit or fixation?

  • During ball tracking, anticipatory head movement, predictive or catch-up eye motion, and later pursuit or predictive saccades occur across trajectory phases.Sequences in which the head catches up to a fixation location while coupled to the ball trajectory were labeled fixations because their signals resemble VOR.Source: §6.2 · p. 17

6.3 Head and Eye tracking can have different coordinate systems

  • Absolute eye and head velocities performed almost as well as biRNN-3, while removing head information reduced pursuit sample κ from 0.40 to 0.29.Fixations and saccades were distinctly identifiable from eye-in-head information alone, but pursuits were harder to distinguish from fixations without head movement.Source: §6.3 · p. 17

6.4 Gaze-in-world information for classification

  • Head pose was included so classifiers could capture eye and head dynamics, including pursuits dominated by either movement, rather than classify only head-compensated gaze-in-world signals.Classifying gaze-in-world information would lose the distinction between head-dominated and eye-dominated pursuit events.Source: §6.4 · p. 17

6.5 General limitations

  • Low-cost IMU head-pose error accumulated over recordings averaging three minutes, with 7° average orientation error from start to finish.The orientation error shifted azimuth and elevation velocities, while changing absolute velocity by only 0.04°/s on average; helmet slippage contributed to camera misalignment.Source: §6.5 · p. 17

6.6 Limitations of event-based metrics

  • ELC matches events whose transition points fall within a window, but its scores can vary with window size and the metric is asymmetric.The authors propose averaging ELC-derived metrics for inter-coder evaluation; they call for future event metrics that are symmetric and threshold-independent.Source: §6.6 · p. 17

7 Conclusion

The conclusion presents GW as a naturalistic dataset for eye–head coordination and reports classifier, metric, and feature findings. Pursuit classification remained difficult without contextual information, while absolute eye and head velocities yielded comparable results without coordinate alignment.

7 Conclusion

  • GW provides a large-scale dataset for naturalistic eye–head coordination; classifiers approached human fixation and saccade performance but struggled to distinguish pursuits without contextual information.The work also proposes ELC for event-level sequence matching and reports that adding a low-cost IMU benefits head-free pursuit classification.Source: §7 · p. 18
  • Comparable head-free gaze classification results were observed using absolute eye and head velocities, indicating that eye and head coordinate systems need not be aligned.The conclusion describes four tasks performed without constraints and approximately 2 hours and 15 minutes of manually coded gaze behavior.Source: §7 · p. 18

10 Additional information

The additional-information passage reports data and code availability, consent for identifiable imagery, and the visualization software used. It also states that reported metrics are rounded to two decimal places.

10 Additional information

  • Compressed data and codes are stated to be made publicly available, and identifiable people in the manuscript consented to sharing their information as presented.Source: §10 · p. 20
  • Statistical figures were generated with Gramm, and all reported metrics are rounded to the second decimal.Source: §10 · p. 20

11 Supplementary Information

The supplementary information identifies the manuscript and listed tasks, describes dataset-status symbols and an IMU drift measure, and provides captions for dataset and ablation tables. Figure 1 illustrates paired eye/head velocity traces and scene views for gaze events.

11 Supplementary Information

  • The supplementary material lists ball catching, visual search, and tea making as tasks; the manuscript title names the work as a dataset for eye and head coordination in everyday activities.Table 1 is captioned as presenting dataset status and error measures.Source: §11 · pp. 20–21, Table 1 · p. 21
  • The dataset-status legend defines Θ as IMU absolute angular drift between task start and end, and marks recordings by labeling status, missing depth, head-pose correction, or discard status.When an initial pose was not constrained, head pose was estimated from a visually identifiable fixation direction during the task.Source: §11 · p. 21
  • Figure 1 pairs eye and head velocity traces with world-camera views, illustrating fixations, head-dominated pursuits, and cases that contribute to fixation–pursuit confusion.Its caption notes slight IMU and eye-tracker misalignment and examples where scene imagery clarifies behavior that velocity traces alone may suggest is pursuit.Source: Figure 1 · p. 22
  • Table 2 is captioned as reporting ablation results across all metrics, including event-level κ* computed with ELC and mean l2 distance and deviation in milliseconds.Source: Table 2 · p. 23
Loading 1905.13146v1…