PaperlayerView exact revision on arXiv

Source-linked AI summary

GazeFS: Target-Centered Gaze-Trajectory Forecasting and Stabilization from Gaze-Head History

Yaozheng Xia, Zaiping Zhu, Bo Pang, Minghao Xie, Hui Li, Shaorong Wang, Sheng Li

arXiv:2609.03868v1cs.HCcs.AI

TL;DR

Target-centered gaze correction must distinguish local fluctuation from persistent target-relative offset, since filtering alone cannot infer a corrected center. GazeFS forecasts target-center direction and short-horizon Search/Focus from gaze-head history without target information at inference. In grouped out-of-fold evaluation, it reduced Focus episode bias by 0.182°, within-episode dispersion by 0.257°, and P90 error by 0.400° versus raw hold.

  • Problem

    Gaze correction must address both local fluctuation and persistent target-relative offset, which low-pass filtering alone cannot recenter.

    Source: Introduction · p. 1

  • Method

    GazeFS forecasts target-center direction and short-horizon Search/Focus from gaze-head history without target information at inference.

    Source: Introduction · p. 2, Problem Formulation · p. 4

  • Results

    0.182° in Focus episode bias, 0.257° in within-episode dispersion, and 0.400° in P90 error were reduced by Full GazeFS relative to raw hold.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Focus target centering and empirical residual contraction improve, while temporal smoothness remains a separate objective.

    Source: Introduction · p. 1, Conclusion · p. 8

    Evaluation covers segmented successful acquisitions in one HoloLens 2 task; open-stream false activation and cross-layout, session, task, and device generalization remain unevaluated.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Target-centered gaze interaction requires more than suppressing frame-to-frame fluctuations: target acquisition produces task-aligned changes in gaze-head dynamics, while a gaze trace may retain a persistent target-relative residual direction. We formulate gaze correction as online target-centered gaze-trajectory forecasting and stabilization and introduce GazeFS, which maps a variable-length gaze-head history to the next target-center direction and a short-horizon Search/Focus estimate without target information at inference. Across 7,960 acquisition episodes from 30 participants, Search-Focus differences remain stable under quality control, onset exclusion, and duration matching. History windows improve phase decoding over the current endpoint, but explicit task progress remains a strong control. Under the 30-participant, five-fold grouped out-of-fold protocol across three seeds, the reductions relative to raw hold in Focus episode bias, within-episode dispersion, and P90 target error are 0.182 degrees, 0.257 degrees, and 0.400 degrees, with participant-bootstrap 95% confidence intervals excluding zero. Endpoint-free replay from empty history preserves the Focus advantage and yields raw-network phase balanced accuracy/AUPRC of 0.925/0.993; coordinate controls further show that recent history contributes beyond explicit progress metadata. GazeFS therefore improves Focus target centering and empirical residual contraction while leaving temporal smoothness as a separate objective.

Introduction

GazeFS forecasts target-center direction and short-horizon Search/Focus from gaze-head history without target information at inference. It improves Focus target centering relative to raw hold, while temporal smoothness remains a separate objective.

Introduction

  • Gaze correction must distinguish short-timescale fluctuation from persistent target-relative offset: smoothing can reduce fluctuation but cannot alone recenter an off-target signal.The paper defines stabilization as target-relative centering and residual contraction, with temporal smoothness evaluated separately.Source: Introduction · p. 1
  • Search and Focus are confirmation-anchored, task-defined phases, and gaze-head history probes find recent context more informative than one current observation.Simple task-progress cues explain much of phase predictability, and frame-order perturbation causes little loss, so history is treated as acquisition-history context rather than necessary exact ordering.Source: Introduction · pp. 1–2
  • GazeFS uses masked multi-scale history encoding, parallel target-direction and phase heads, phase-aligned coordinate learning, and episode-prefix refinement.The predicted phase does not gate trajectory decoding at inference; target information is used for offline supervision and evaluation, but not in the inference tensor.Source: Introduction · p. 2
  • 0.182°, 0.257°, 0.109°, and 0.400° reductions in Focus episode bias, within-episode dispersion, median error, and P90 error, respectively, were observed versus raw hold.On common valid Focus support, participant-level simultaneous intervals over the four outcomes remained below zero.Source: Introduction · p. 2
  • Focus centering advantage persists in endpoint-free replay, and controls show recent gaze-head history adds spatial information beyond progress cues without requiring endpoint or future samples.Future-suffix, exact-prefix, and endpoint-free tests verify that outputs do not depend on endpoint knowledge or unobserved future samples.Source: Introduction · p. 2

Related Work

GazeFS differs from prior gaze correction and forecasting by predicting a target-centered interaction direction from target-agnostic gaze–head history, with target geometry used only for offline supervision. It also predicts confirmation-anchored Search/Focus phase jointly with direction, treating phase as target-acquisition progress rather than an intrinsic oculomotor regime.

Gaze filtering and spatial correction

  • Existing gaze correction smooths measured direction, combines gaze and head motion, or selects objects using candidates or scene cues.Temporal filters (Plopski et al. 2022; Špakov 2012; Kumar et al. 2008) remain tied to measurements; eye–head pointing (Sidenmark and Gellersen 2019; Sidenmark et al. 2020, 2022; Hou et al. 2024) and candidate-aware or scene-level methods (Pfeuffer et al. 2017; Wei et al. 2023; Yang, Yin, and Lu 2024) address control or object selection. GazeFS forecasts target-centered direction from target-agnostic history; endpoint-only and Endpoint+Progress controls test current-frame and progress-conditioned calibration explanations.Source: Related Work · p. 2

Temporal gaze forecasting

  • Prior gaze forecasting predicts subsequent observed gaze, whereas GazeFS predicts a latent target-center direction whose geometry is available only during training.Prior work uses gaze, head motion, and scene history for anticipation (Hu et al. 2019; Burlingham et al. 2024; Hu et al. 2020, 2021; Rolff et al. 2022); related work includes short-horizon VR forecasting with oculomotor-event prediction (Melnyk, Friedman, and Komogortsev 2026) and multi-scale or non-stationary architectures (Wu et al. 2023; Cai et al. 2024).Source: Related Work · p. 2

Fixation, intent, and task-aligned phase

  • GazeFS jointly predicts target direction and confirmation-anchored Search/Focus phase, which represents target-acquisition progress rather than an intrinsic oculomotor regime.This differs from velocity- and dispersion-based fixation methods and intent or adaptive-dwell models of readiness or activation timing (David-John et al. 2021; Isomoto, Yamanaka, and Shizuki 2022; Narkar et al. 2024).Source: Related Work · p. 2

Data and Task Definition

The study analyzes gaze–head histories from 7,960 episodes across 30 participants and defines Search and Focus as task-progress labels from confirmation-anchored gaze-to-target traces. Target-center direction supports offline labeling and evaluation but is withheld at inference.

Data and Task Definition

  • 7,960 60-Hz episodes from 30 HoloLens 2 participants use a seven-channel gaze–head representation, while target-center direction is reserved for offline supervision and evaluation.The channels are head-local gaze yaw–pitch, their angular rates, and head yaw–pitch–roll rates.Source: Data and Task Definition · p. 3
  • Search and Focus labels mark acquisition progress: a backward scan from participant confirmation locates the earliest stable target-centered segment, whose onset defines the phase boundary.These labels are not physiological fixation or saccade states.Source: Data and Task Definition · p. 3, Figure 2 · p. 3
  • Complete episodes characterize phase-dependent gaze–head structure, while natural history-window probes compare frozen endpoints across endpoint-only, ordered-history, order-invariant, and explicit-progress controls.The complete probe protocol is reported in Supplement Sec. A.2.Source: Data and Task Definition · p. 3

Task-Aligned Gaze–Head Structure

During acquisition, Focus shows lower gaze-speed tails, head motion, and local dispersion than Search, while raw Focus residuals tend to retain their direction. Recent gaze–head history improves phase decoding over a single endpoint, though the probes do not establish exact frame order as necessary.

Task-Aligned Gaze–Head Structure

  • Focus has lower gaze-speed tails, head motion, and local dispersion than Search, with effects retaining direction after boundary exclusion and duration matching.The boundary exclusion removes ±300 ms around the recovered Search/Focus boundary.Source: Table 1 · p. 3, Task-Aligned Gaze–Head Structure · p. 3
  • Early/late Focus residuals show a 0.886 bias-vector cosine and 0.977 persistent-offset ratio, indicating direction persistence rather than purely frame-local fluctuation.Both intervals are well above zero; this motivates history-based target-center forecasting rather than local smoothing alone.Source: Table 1 · p. 3, Task-Aligned Gaze–Head Structure · p. 3
  • A recent gaze–head window raised balanced accuracy from 0.640 for the current 7D observation to 0.927/0.928 at 64/128 frames.Historically observed elapsed-time features were competitive with the full window, while shuffling individual frames or short blocks produced essentially no loss; thus, the probe supports history context, not exact frame order as necessary. The controlled probe is one-seed; the caption notes a three-seed shuffle audit in Supplement Sec. A.2.Source: Task-Aligned Gaze–Head Structure · p. 3, Table 1 · p. 3

Online Target-Centered Gaze-Trajectory · Forecasting and Stabilization

GazeFS maps observed gaze–head history to target-center direction forecasts and short-horizon Search/Focus estimates without target-derived inputs at inference. Its design combines multiscale temporal encoding, parallel trajectory and phase heads, and task-aligned training objectives.

Problem Formulation

  • From observed gaze–head history alone, GazeFS predicts dense target-center directions and next-S Search/Focus logits, without target-derived inputs at inference.The seven inference channels are gaze yaw–pitch, their angular rates, and head yaw–pitch–roll rates. History-length and prefix descriptors are also available online, though they may encode task progress.Source: Problem Formulation · p. 4

GazeFS Architecture

  • Separate gaze and head streams form fine- and coarse-scale temporal tokens, fused to capture rapid acquisition changes and longer local motion context.The fine and coarse patch sizes are 2 and 4 observations, respectively, with a shared stride of 2.Source: GazeFS Architecture · p. 4, Figure 3 · p. 5
  • Two causal residual modules refine the first R dense forecasts, using raw history and prefix descriptors with gates that decay as context accumulates.Outside the short prefix, the gates leave the base forecast unchanged.Source: GazeFS Architecture · p. 4

Task-Aligned Learning

  • Training aligns phase labels to whether Focus occurs in the next S-frame interval, while coordinate targets specify the next-frame target-center direction.The phase estimate is therefore a short-horizon future estimate rather than a current-frame label.Source: Task-Aligned Learning · pp. 4–5, Problem Formulation · p. 4

GazeFS Architecture

  • A causal, padding-masked shared Transformer feeds parallel trajectory and Search/Focus heads; inference reuses the trajectory representation in one encoder pass.Predicted phase never gates or feeds the trajectory forecast.Source: GazeFS Architecture · p. 4, Figure 3 · p. 5

Task-Aligned Learning

  • The coordinate objective combines phase-weighted Huber and directional cosine terms, while phase uses α-balanced focal-weighted crossentropy and training-only geometry penalties.The coordinate Focus weight comes from the ground-truth label, not predicted phase.Source: Task-Aligned Learning · p. 5

Experiments

Under grouped out-of-fold evaluation, Full GazeFS reduces all four Focus spatial-error outcomes relative to Raw, while increasing residual motion, so centering and temporal smoothness remain distinct objectives. Endpoint-free replay and history–progress controls further show that recent gaze–head history contributes to sustained Focus centering, while progress context has complementary benefits.

Protocols, Statistics, and Baselines

  • Evaluation uses participant-grouped five-fold OOF testing, next-frame direction estimates from information through t, and raw network outputs without EMA, dwell, or FSM decisions.Methods include Raw-hold, frozen-parameter One Euro, endpoint-only, endpoint-plus-progress, no-explicit-progress, and Transformer-Causal controls. Focus spatial metrics are participant-weighted and paired effects use 5,000 participant-bootstrap resamples; temporal residual motion is reported separately from target error and dispersion.Source: Protocols, Statistics, and Baselines · pp. 5–6

RQ1: Focus-Stage Target Centering

  • Full GazeFS reduces Raw Focus bias, dispersion, median error, and P90 error by 0.182°, 0.257°, 0.109°, and 0.400°, respectively.Participant-bootstrap and simultaneous intervals remain below zero; all four outcomes also improve relative to One Euro, endpoint-only, Transformer-Causal, and GazeFS without phase alignment on common Focus support.Source: RQ1: Focus-Stage Target Centering · p. 6, Table 2 · p. 6
  • Full GazeFS increases mean frame-to-frame target-relative residual motion by 0.0578° per adjacent physical frame and residual path by 7.003° relative to Raw.These quantify temporal movement of the residual trajectory, not spatial spread, target error, or device-level pointer jitter.Source: RQ1: Focus-Stage Target Centering · p. 6

RQ2: Phase Dependence and Mechanism Boundaries

  • Only Focus improves jointly across the four spatial outcomes; phase-dependent coordinate training, rather than an isolated classifier benefit, is supported as the mechanism.Search and transition do not improve consistently across centering, spread, tail error, and residual motion. Development-only ablations indicate gaze orientation carries target bearing, while gaze/head rates mainly improve endpoint-free upper-tail error.Source: RQ2: Phase Dependence and Mechanism… · p. 6, Table 3 · p. 6

RQ3: Online Phase, Progress Controls, and Replay

  • Full obtains 0.903 balanced accuracy, 0.936 Focus F1, and 0.990 AUPRC, with median/P90 absolute onset errors of 248/585 ms, from raw phase outputs.All 60 learned OOF checkpoints satisfy future-suffix invariance, with a global maximum absolute difference of zero. Exact-prefix results improve over Raw at 8, 16, 64, and 128 frames, while the 32-frame interval crosses zero.Source: RQ3: Online Phase, Progress Controls,… · p. 6
  • In endpoint-free replay from empty history, Full retains its centering advantage and has higher balanced accuracy and AUPRC and lower onset-error P90 than Transformer-Causal.Focus F1 is slightly lower and median threshold crossing later than Transformer-Causal; replay context grows to 128 frames and then slides without endpoint or future input, EMA, or FSM decisions.Source: RQ3: Online Phase, Progress Controls,… · p. 6, Table 6 · p. 7
  • Without explicit progress metadata, history improves Endpoint+Progress bias, dispersion, median, and P90 by 0.159°, 0.132°, 0.098°, and 0.303°, respectively.Progress alone improves all four spatial outcomes over Endpoint-only. At ≥128 frames, Full remains better than Endpoint+Progress by 0.139°, 0.100°, 0.076°, and 0.284° in bias, dispersion, median, and P90; these advantages localize to sustained Focus, as P90 is +0.360° in the first 250 ms after onset but −0.455° after 500 ms relative to Raw.Source: RQ3: Online Phase, Progress Controls,… · pp. 6–7, Table 7 · p. 7

Secondary Closed-Loop Feasibility

  • In a 12-participant HoloLens 2 study, GazeFS is 0.196 s faster than One Euro in the all-trial min(T, 5 s) sensitivity, while comparisons with the native pointer and Weighted Pointing cross zero.The study compares deployed conditions, and only GazeFS resets history/state after selection or timeout, so it does not isolate learned correction or reset. Recent history adds spatial information beyond progress descriptors; progress/position context chiefly benefits dispersion, upper-tail error, residual motion, and phase/onset reliability, while residual-motion cost motivates explicit centering–smoothness optimization.Source: Secondary Closed-Loop Feasibility · p. 7, Table 8 · p. 7

Limitations

The evidence is limited to confirmation-anchored phases and segmented successful acquisitions in one HoloLens 2 task, with target information used for supervision but absent at inference. Independent-cohort, open-stream, and broader generalization evidence remains outstanding.

Limitations

  • Evaluation covers segmented successful acquisitions from one HoloLens 2 task; open-stream false activation and cross-layout, session, task, and device generalization remain untested.A single-seed held-quadrant screen supports extrapolation only within the recorded geometry.Source: Limitations · p. 7
  • Grouped out-of-fold evaluation is internal validation, not independent-cohort confirmation, because development used a fixed split from the same corpus.Search and Focus are confirmation-anchored phases, and target information is used for supervision although it is absent at inference; the evidence supports complementary history and progress context, not a strict order law.Source: Limitations · p. 7
  • The closed-loop comparison does not isolate reset from learned correction, centering retains a Raw-relative motion cost, and PC-side ONNX latency excludes sensing, transport, and Unity rendering.Latency exclusions are noted in Supplement Sec. C.8.Source: Limitations · pp. 7–8

Conclusion

GazeFS predicts the next target-center direction and short-horizon Search/Focus phase from gaze–head history without target geometry at inference. Grouped OOF and endpoint-free replay support sustained-Focus error reductions, while broader continuous-stream generalization and temporal smoothness remain open.

Conclusion

  • Participant-grouped OOF and endpoint-free replay show sustained-Focus reductions in bias, dispersion, median, and upper-tail error.Controls indicate complementary roles for recent history and observable progress.Source: Conclusion · p. 8
  • GazeFS forecasts the next target-center direction and a short-horizon Search/Focus phase from gaze–head history without target geometry at inference.Source: Conclusion · p. 8
  • A secondary closed-loop study supports feasibility and a narrow duration benefit over One Euro, while temporal smoothness and broader continuous-stream generalization remain open.Source: Conclusion · p. 8

Related papers

Loading 2609.03868v1…