Source-linked AI summary
EyeTAG: Eye Trajectory-Aware Gaze Estimation
Jungmin Lee, Niamat Ullah, Yoseob Han
TL;DR
Single-frame gaze estimates can jitter, while multi-frame approaches do not explicitly represent gaze trajectories. EyeTAG feeds recent prediction differences back as a motion prior; it lowers Gaze360 MAE by about 1.0° over the strongest baseline and is on par with the strongest multi-frame baselines on EVE.
Problem
Single-frame predictions can fluctuate as jitter, and multi-frame methods model motion implicitly rather than carrying the gaze trajectory as an explicit variable.
Source: §1 · p. 2Method
EyeTAG feeds differences between its recent gaze predictions back as a translation-invariant, subject-invariant motion prior, fused with face and eye evidence through causal attention.
Source: §1 · pp. 2–3Results
About 1.0° lower MAE than the strongest baseline on Gaze360; on EVE, EyeTAG MAE is 2.56° versus STAGE 2.58° and Gaze360 2.73°.
Source: §4.3 · p. 10Takeaways & Limitations
EyeTAG models gaze kinematics explicitly, with temporally coherent and stable predictions under unconstrained head-eye motion.
Source: §6 · p. 14EyeTAG estimates individual gaze direction without scene context, and real-time performance has been verified only on a desktop GPU.
Source: §6 · p. 14
Abstract
from arXiv · showhide
Gaze estimation under natural head-eye motion underpins applications from driver monitoring to human-computer interaction. Single-frame methods predict each frame independently, so consecutive outputs fluctuate as jitter. Multi-frame methods reduce this, but they learn motion implicitly inside appearance features, so the gaze trajectory is never an explicit variable. We propose EyeTAG (Eye Trajectory-Aware Gaze Estimation), a causal multi-frame framework built around an explicit first-order gaze prior: at each step it differentiates its own recent predictions and feeds the resulting trajectory back as a compact kinematic token. Because differencing is translation-invariant in gaze space, this token carries subject-invariant motion rather than personal gaze offsets. Face and eye streams supply visual evidence, fused by cross-attention and a causal Transformer decoder. EyeTAG reduces the mean angular error by about 1.0$^\circ$ on Gaze360 and performs on par with the strongest baseline on EVE (2.56$^\circ$ vs. 2.58$^\circ$). Within-model ablations, which keep the encoder and the rest of the architecture fixed and vary only the gaze history, show that the differential formulation, rather than temporal context alone, removes the systematic saccade bias that persists even with an absolute gaze-history prior. Our code is available at https://github.com/peter8366/EyeTAG.
1 Introduction
Gaze estimation remains challenging under real-world variation, while single-frame methods can jitter and multi-frame methods typically encode motion implicitly. EyeTAG instead feeds back an explicit differential gaze trajectory alongside face and eye evidence in a causal model.
1 Introduction
- EyeTAG feeds back first-order differences of its own recent gaze predictions as a subject-invariant motion token, and within-model ablations link the differential prior to removing systematic saccade bias.The ablations keep the encoder and rest of the architecture fixed, distinguishing the differential formulation from temporal smoothing alone.Source: §1 · p. 3
2.1 Gaze Estimation
- Appearance-based gaze estimation maps visual appearance to 3D gaze, while geometric approaches can require calibration and remain sensitive to illumination and occlusion.Large in-the-wild datasets such as GazeCapture [19] and Gaze360 [18] have supported generalization across diverse subjects and scenes.Source: §1 · p. 2, §2.1 · p. 3
2.2 Temporal and Multi-frame Modeling
- Prior temporal methods use recurrent layers, temporal attention, or sequence models, but EyeTAG explicitly feeds back past prediction differences as a compact kinematic prior.The introduction identifies implicit motion representation in appearance features as the gap; the prior uses first-order changes rather than appearance alone.Source: §1 · p. 2, §2.2 · pp. 3–4
2.3 Cross-modal Fusion of Face and Eye Features
- EyeTAG combines face and eye representations with its kinematic prior, addressing the difficulty of aligning global facial context and fine-grained ocular cues.Prior approaches included feature combination and coarse-to-fine refinement; EyeTAG uses cross-modal attention and causal temporal aggregation.Source: §2.3 · p. 4
3 Method
EyeTAG combines face and eye appearance with an explicit autoregressive prior formed from recent predicted-gaze differences. Cross-modal attention and a causal Transformer fuse these signals to predict the current gaze direction.
3 Method
- EyeTAG encodes face, eye, and differential gaze-trajectory inputs, fuses them with cross-modal attention and a causal Transformer, then regresses the current 3D gaze vector.Source: §3 · pp. 4–5, Figure 3 · p. 6
3.1 Dual-Stream Visual Encoder (DSVE)
- The DSVE extracts per-frame facial context and localized eye appearance as separate visual streams for later temporal fusion.The face and eye streams use separate ResNet-based branches; temporal aggregation occurs later in CCAF.Source: §3.1 · pp. 5–6
3.2 Trajectory-Aware Kinematic Encoder (TAKE)
- TAKE converts recent predicted-gaze differences into a compact kinematic token, providing an autoregressive prior based on relative motion rather than absolute gaze coordinates.The differential representation removes subject-specific gaze offsets and is described as supporting stability as context length increases.Source: §3.2 · pp. 6–7
3.3 Cross-Modal Causal Attention Fusion (CCAF)
- CCAF aligns facial and ocular features with cross-attention, then causally aggregates the aligned visual tokens with the kinematic prior for gaze prediction.The causal mask restricts each frame's representation to past observations and the kinematic prior, preventing future-information leakage during autoregressive inference.Source: §3.3 · pp. 7–8
4 Experiments
EyeTAG is evaluated on Gaze360 and EVE using their official protocols, with Gaze360 results reported across three head-pose subsets. It improves Gaze360 accuracy and is comparable to the strongest multi-frame baselines on EVE; controlled temporal analyses examine fixation jitter and saccade responsiveness.
4.1 Dataset
- The evaluation uses subject-disjoint Gaze360 splits and reports All, Semi-Front, and Front subsets; EVE evaluation uses its official validation split and webcam_c stream.EVE test annotations are not publicly available; the datasets contain natural-condition video and desktop-interaction sequences, respectively.Source: §4.1 · p. 8
4.2 Experimental Setup
- Inference takes approximately 20 ms for a 48-frame input window on an RTX A6000, below the 33.3 ms interval of 30 Hz video.This supports feasibility on the evaluated desktop GPU, not embedded real-time deployment.Source: §4.2 · p. 9
4.3 Experimental Results
- EyeTAG achieves 9.29° MAE on Gaze360 All, 9.12° on Semi-Front, and 8.08° on Front, improving MAE by about 1.0° over the strongest baseline.It has the lowest MAE on all three subsets.Source: §4.3 · p. 10, Table 2 · p. 10
- 2.56° mean MAE on EVE places EyeTAG on par with STAGE (2.58°) and ahead of Gaze360 (2.73°).EyeTAG’s error standard deviation is 1.89 versus STAGE’s 2.11; Gaze360’s 1.77 uses future frames.Source: §4.3 · pp. 10–11
- EyeTAG’s fixation jitter is 3.68, comparable to 3.51 for the visual-only and 3.43 for the normal-prior variants, while its overall temporal-behavior score is lowest at 1.87.The score is 1.99 for visual-only and 1.98 for the normal-prior variant.Source: §4.3 · pp. 11–12, Table 3 · p. 11
- For rapid gaze transitions, EyeTAG reduces saccade bias to 0.05, while visual-only and normal-prior variants under-shoot at −0.46 and −0.53.The variants keep the encoder and remaining architecture fixed, isolating the gaze-history component.Source: §4.3 · pp. 11–12, Table 3 · p. 11
5 Ablation Studies
The ablations compare visual fusion, gaze-history representations, and temporal window lengths. Unidirectional cross-attention and the kinematic gaze prior perform best in their respective comparisons, while longer windows improve accuracy until performance saturates.
5.1 Effect of Cross-Modal Fusion Strategy
- Unidirectional cross-attention achieves the lowest error on all subsets among the three face-eye fusion strategies.It uses face embeddings as queries and eye embeddings as keys and values, injecting ocular cues into the face representation without redundant fusion paths.Source: §5.1 · p. 12
5.2 Influence of Gaze History Embedding
- The Kinematic Gaze Prior performs best across all subsets, outperforming both the Normal Gaze Prior and the visual-only variant.Removing gaze history gives the highest error on every subset, while normal gaze history improves over visual-only.Source: §5.2 · p. 13
- Differential yaw has a compact, zero-centered distribution (σ≈12°), unlike normal yaw’s broad ±180° range and higher variance (σ≈39°).The authors state that the differential representation reduces subject-dependent offsets and helps distinguish gaze transitions from noise.Source: §5.2 · p. 13, Figure 5 · p. 13
- Replacing autoregressive gaze history with ground truth improves mean MAE only from 8.83° to 8.65°, with a 0.16–0.24° gap across clip positions.The authors report that scheduled sampling keeps autoregressive error from accumulating over time.Source: §5.2 · p. 13
5.3 Effect of Temporal Window Length
- MAE decreases as the window grows from T=4 to T=48, then saturates at T=64 while inference time and computational cost continue increasing.The study adopts T=48 as the accuracy–latency balance; on an RTX A6000, its window latency is 19.97 ms versus 25.23 ms at T=64.Source: §5.3 · pp. 13–14, Table 6 · p. 14
6 Conclusion
EyeTAG models gaze kinematics with differential trajectories alongside face and eye appearance cues. The paper reports reduced angular error on Gaze360, parity with the strongest multi-frame baseline on EVE, and identifies limits in scene reasoning and deployment scope.
6 Conclusion
- EyeTAG combines differential gaze dynamics with face-eye cross-modal attention and causal temporal aggregation to estimate gaze under unconstrained head-eye motion.The conclusion also reports low fixation jitter and reduced velocity bias during rapid transitions, attributing near-zero saccade bias to the differential prior rather than temporal context alone.Source: §6 · p. 14
- The framework estimates individual gaze direction without scene context or reasoning about what a person is looking at or how multiple people interact.Real-time performance is verified only on a desktop GPU; embedded deployment remains future work.Source: §6 · p. 14
- Future work includes extending the framework to multi-person and scene-aware gaze understanding, including social gaze interactions and contextual intent.Source: §6 · p. 14
S1 Dataset Details
The supplementary visualization locates three Gaze360 evaluation subsets on the unit sphere, distinguished by their yaw ranges.
S1 Dataset Details
- Front covers |yaw| ≤20°, Semi-Front covers |yaw| ≤90°, and All covers |yaw| ≤180° on the unit sphere.The figure colors Front green, Semi-Front blue, and All red; the ranges progress from a narrow frontal cone to the entire sphere.Source: S1 Dataset Details · p. 18, Figure S1 · p. 18
S2 Effect of Face-Encoder Pretraining
The pretraining ablation varies only the face encoder’s initialization while holding encoder architectures fixed. Face-domain VGGFace2 pretraining performs best, and its benefit is separate from EyeTAG’s architectural contributions.
S2 Effect of Face-Encoder Pretraining
- The comparison changes only face-encoder pretraining, holding the ResNet-50 face encoder and ResNet-18 eye encoder fixed.The three settings are no pretraining, ImageNet pretraining, and VGGFace2 pretraining.Source: S2 Effect of Face-Encoder Pretraining · p. 18, Table S1 · p. 19
- Face-domain VGGFace2 pretraining performs best, reducing Gaze360 mean angular error to 8.83° with consistent gains across All, Semi-Front, and Front subsets.The authors report that it transfers better than generic ImageNet features to Gaze360’s extreme head poses and diverse identities.Source: S2 Effect of Face-Encoder Pretraining · pp. 18–19, Table S1 · p. 19
- The pretraining gains are separate from EyeTAG’s architectural contributions because the main-paper fusion and gaze-history ablations keep the VGGFace2 backbone fixed.The temporal-stability analysis therefore mainly reflects the kinematics-aware design rather than backbone initialization.Source: S2 Effect of Face-Encoder Pretraining · p. 19
S3 Effect of Eye-Encoder Backbone
The eye-backbone comparison keeps EyeTAG’s configuration fixed and changes only the eye encoder; stronger spatial capacity and explicit temporal modeling do not improve performance over ResNet-18.
S3 Effect of Eye-Encoder Backbone
- Only the eye backbone changes; all experiments retain T=48, the kinematic gaze prior, and the VGGFace2-pretrained face encoder.The main model uses a shared ImageNet-pretrained ResNet-18 eye encoder that processes frames independently.Source: S3 Effect of Eye-Encoder Backbone · p. 19
- TSM and the 3D CNN perform worse than the 2D ResNet-18 baseline.The comparison tests explicit temporal modeling inside the eye branch.Source: S3 Effect of Eye-Encoder Backbone · p. 19
- Replacing ResNet-18 with EfficientNet-B0 also gives no improvement.The authors suggest the downstream CCAF module already captures enough ocular dynamics at T=48.Source: S3 Effect of Eye-Encoder Backbone · p. 19
S4 Effect of Scheduled Sampling
The scheduled-sampling ablation isolates how gaze history is constructed during training. It outperforms teacher-forcing-only and fully autoregressive training, with the latter 0.53° worse in mean error.
S4 Effect of Scheduled Sampling
- The ablation compares teacher forcing, fully autoregressive training, and scheduled sampling with the normal gaze prior at T=32, holding other components fixed.The schedule anneals the teacher-forcing ratio from 1.0 to 0.0 over epochs 1–5.Source: S4 Effect of Scheduled Sampling · p. 20
- 36.71° Mean error results from teacher forcing only at test time.The authors attribute this to the model not seeing its own prediction errors during training, creating train–test mismatch and rapidly accumulating autoregressive errors.Source: S4 Effect of Scheduled Sampling · p. 20
- 0.53° lower mean error separates scheduled sampling from fully autoregressive training, and scheduled sampling has the lowest error across all subsets.The authors suggest noisy predictions in early epochs make optimization harder for fully autoregressive training.Source: S4 Effect of Scheduled Sampling · p. 21
S5 Effect of Window Length with Normal Gaze Prior
With the normal absolute gaze-history prior, longer windows help through T=32, then accuracy plateaus or slightly degrades; expanding subject-specific bias is offered as the reason.
S5 Effect of Window Length with Normal Gaze Prior
- The controlled sweep uses the model architecture and varies only temporal window length T under the normal gaze prior.The architecture includes a ResNet-50 face encoder, shared ResNet-18 eye encoder, unidirectional cross-attention, and causal Transformer decoder.Source: S5 Effect of Window Length with Normal… · p. 21
- MAE decreases steadily from T=4 to T=32, then accuracy plateaus and can degrade slightly beyond T=32.This shows longer temporal context is useful up to that range under the normal gaze prior.Source: S5 Effect of Window Length with Normal… · p. 21
- 9.46° at T=32 rises to 9.65° at T=48 and 9.91° at T=64 for Semi-Front error.The passage attributes saturation to absolute pitch–yaw ranges expanding with subject-specific bias, making consistent motion patterns harder to extract.Source: S5 Effect of Window Length with Normal… · p. 21
S6 Robustness to Incomplete Gaze History
EyeTAG is evaluated on early Gaze360 clip frames, when its autoregressive gaze history is incomplete and zero-padded. The kinematic prior retains an advantage in this cold-start setting.
S6 Robustness to Incomplete Gaze History
- The cold-start analysis examines the first T frames, when the full window of past predictions does not yet exist and the differential history is zero-padded.The kinematic prior is weakest in this setting.Source: S6 Robustness to Incomplete Gaze History · p. 21
- 8.11° Mean error is achieved by EyeTAG on early frames, compared with 8.48° for the normal-prior variant.The evaluation covers positions 0 ≤ t < T with T=48 across all 68 Gaze360 test clips, with incomplete histories zero-padded.Source: S6 Robustness to Incomplete Gaze History · pp. 21–22, Table S5 · p. 22
- The kinematic prior retains its advantage over the normal prior and all baselines even when history is incomplete.The absolute-prior EyeTAG variant is second in this evaluation, and the authors describe the kinematic prior as degrading gracefully.Source: S6 Robustness to Incomplete Gaze History · p. 22