PaperlayerView exact revision on arXiv

Source-linked AI summary

EyeMakeYou: Identity-, Task-, and Subjective-State-Conditioned Diffusion for High-Frequency Gaze Synthesis

Kamrul Hasan, Mehedi Hasan Raju, Oleg V. Komogortsev

arXiv:2609.04501v1cs.HCcs.CV

TL;DR

High-frequency gaze recordings are costly and existing synthesis approaches do not jointly represent identity, task, and subjective state, limiting the available data for gaze applications. EyeMakeYou uses conditional diffusion with a low-frequency reference and those three conditions to synthesize gaze velocity. It achieves strong spatial and biometric fidelity in GazeBase, with remaining scope and variability limitations.

  • Problem

    High-frequency gaze recordings are resource-intensive and often unavailable at scale, while existing generators do not jointly condition on identity, task, and subjective state.

    Source: §1 · p. 2

  • Method

    EyeMakeYou uses conditional diffusion to generate 5-s, 1000-Hz gaze-velocity sequences from a low-frequency reference conditioned on identity, task, and subjective-state embeddings.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across seven tasks, EyeMakeYou has the highest real–synthetic EKYT similarity and the lowest median spatial errors for HSS and RAN; its embedding cosine similarities range from 0.91 to 0.95.

    Source: §6 · p. 16

  • Takeaways & Limitations

    EyeMakeYou is most suited to generating subject-specific samples for augmenting gaze-based biometric and interactive-system datasets, particularly when real data are limited.

    Source: §6 · p. 17

    Evaluation uses 5-s GazeBase segments from one 1000-Hz eye tracker and does not establish generalization across devices, laboratories, longer sessions, or naturalistic environments.

    Source: §6 · p. 17

Abstract

from arXiv · show

Eye movement biometrics (EMB) is an emerging behavioral modality for user authentication, particularly in virtual- and augmented-reality systems, where gaze dynamics contain distinctive subject-specific features. However, robust EMB systems require diverse, high-quality gaze recordings that are expensive to collect and often unavailable at the scale needed for model development. Generative models can mitigate data scarcity, but existing methods either synthesize generic gaze behavior or personalize signals primarily by identity, without jointly representing the user's task and subjective state. Consequently, generated signals may appear visually realistic while failing to retain the behavioral properties required for biometric applications. To address this limitation, we propose EyeMakeYou, a multi-conditional denoising diffusion framework for subject-specific, high-frequency gaze synthesis. EyeMakeYou generates 5-s, 1000-Hz bivariate gaze-velocity sequences from an identity-removed reference trajectory and conditions the denoising process on an identity embedding, a task embedding, and self-reported ratings of overall difficulty, mental tiredness, and eye tiredness. Its objective combines diffusion noise prediction and identity preservation with multi-resolution spectral, drift-consistency, and event-weighted local-smoothness losses. Experiments on GazeBase show that EyeMakeYou achieves higher median spatial accuracy and greater real--synthetic similarity in the embedding feature space than the existing generative approaches, while retaining selected task-dependent associations between subjective reports and oculomotor features. These findings support conditional diffusion as a practical approach for augmenting gaze datasets for biometric and interactive applications.

1 Introduction

Eye-movement signals are useful for behavioral biometrics and interactive applications, but high-frequency recordings are costly and often unavailable. EyeMakeYou addresses the gap in existing synthesis methods by conditioning gaze generation jointly on identity, task, and subjective state, with losses designed to preserve biometric and signal properties.

Motivation

  • High-frequency gaze recordings capture rapid oculomotor events but are costly to collect, often unavailable at scale, and may expose identity or sensitive attributes.These constraints motivate subject-specific synthetic gaze generation for augmenting limited real datasets.Source: §1 · p. 2

Research gap

  • Existing gaze generators do not jointly condition on user identity, task context, and subjective state, limiting explicit specification of these influences on behavior.SP-EyeGAN starts from random noise, whereas DiffEyeSyn conditions on an identity-removed reference and user embedding [16].Source: §1 · p. 2

Contribution

  • EyeMakeYou generates subject-specific, high-frequency gaze-velocity signals using identity, task, and subjective user-state conditions.Its identity-removed reference is made by downsampling gaze positions to 20 Hz and interpolating them back to 1000 Hz; state inputs are difficulty, mental tiredness, and eye tiredness ratings.Source: §1 · p. 2

Evaluation

  • Evaluation assesses spatial accuracy and precision, EKYT embedding similarity, and associations between subjective-state reports and oculomotor features.User-percentile operating points include U50|E50 and U95|E95.Source: §1 · p. 2

Contribution

  • The training objective combines noise prediction and identity preservation with multi-resolution spectral, drift-consistency, and event-weighted local-smoothness losses.The losses target spectral fidelity, stable integrated displacement, and local temporal curvature around fixation and post-saccadic oscillations.Source: §1 · p. 2

2 Related Work

Earlier statistical approaches modeled gaze behavior but generally did not target individualized, high-frequency oculomotor dynamics. Machine-learning methods added data-driven synthesis, and later personalized diffusion work motivates EyeMakeYou’s joint identity, task, and state conditioning.

2.1 Statistical Models

  • Statistical gaze models progressed from behavioral and graphics-based synthesis to explicit oculomotor modeling, yet often produced generic behavior or simplified high-frequency dynamics.Examples include EyeCatch’s simulated fixations, saccades, and smooth pursuits, and EyeSyn’s physics-based events with Gaussian drift and pink-noise jitter.Source: §2.1 · p. 3

2.2 Machine-learning Models

  • Machine-learning gaze synthesis spans image-conditioned scanpaths and stimulus-agnostic generation, but approaches may omit continuous trajectories or explicit user identity.PathGAN predicts fixation locations rather than continuous trajectories [33], while SP-EyeGAN generates reading-related sequences without conditioning on a specific user.Source: §2.2 · p. 3
  • Prior personalized diffusion uses an identity-suppressed reference and EKYT user embedding [16], while EyeMakeYou jointly conditions denoising on identity, task, and subjective state.The overview shows the reference and noised velocity entering denoising alongside user, task, and state conditions, with predicted noise converted to synthesized velocity.Source: §2.2 · p. 3, Figure 1 · p. 4

3 Proposed Architecture

EyeMakeYou is a conditional diffusion model for 5-s, 1000-Hz two-dimensional gaze-velocity segments. It uses a low-frequency reference trajectory and separately encoded identity, task, and subjective-state conditions to guide synthesis.

3.1 Overview

  • EyeMakeYou uses a 50-step conditional DDPM with a 30-layer dilated residual denoiser to synthesize two-dimensional eye-movement velocity signals.Each segment contains 5000 temporal samples and is globally conditioned on identity, task, and subjective state.Source: §3.1 · p. 4

3.2 Gaze preprocessing and velocity representation

  • The reference trajectory is formed by downsampling gaze position to 20 Hz and interpolating it back to 1000 Hz, retaining coarse movement while diminishing higher-frequency identity features.The cited PVAF analysis reports that the 0–25 Hz band retains nearly all variance in saccade trajectories while diminishing higher-frequency components.Source: §3.2 · p. 4

3.3 Conditional Information

  • Identity conditioning averages and L2-normalizes four EKYT encoder embeddings extracted from the original velocity signal to guide synthesis toward the intended user’s characteristics.The reference velocity and original velocity are differentiated from their position signals with a Savitzky–Golay filter.Source: §3.3 · p. 5, §3.2 · p. 5
  • Task embeddings specify the experimental task, while a two-layer MLP encodes ratings of overall difficulty, mental tiredness, and eye tiredness as the user-state condition.The conditions retain distinct meanings and are combined for FiLM modulation; missing or invalid ratings are replaced by 1 before normalization.Source: §3.3 · pp. 5–6

3.4 Forward Diffusion Process

The forward diffusion process gradually adds Gaussian noise to the clean normalized velocity signal over 50 scheduled steps.

3.4 Forward Diffusion Process

  • The forward process gradually adds Gaussian noise to the clean normalized velocity signal using 50 steps and a linear beta schedule from 10^-4 to 5 × 10^-2.The clean signal is denoted x0.Source: §3.4 · p. 6

3.5 Conditional Diffusion Denoiser

The denoiser predicts added noise from a concatenation of noisy velocity and an identity-removed reference. Identity, task, user state, and diffusion step inform the residual network, with task also providing an input-level bias.

3.5 Conditional Diffusion Denoiser

  • The denoiser concatenates noisy velocity with identity-removed reference velocity, then projects them into a 64-channel temporal representation.The task embedding also supplies a direct task-dependent bias at the input.Source: §3.5 · p. 6
  • The denoiser uses 30 gated dilated residual blocks with 64 channels, and combines their skip features to predict diffusion noise.The diffusion step is represented by a sinusoidal embedding transformed through two SiLU-activated linear layers.Source: §3.5 · pp. 6–7
  • Identity, task, and subjective-state embeddings jointly produce FiLM scale and shift parameters that modulate temporal features in each residual block.The same global conditions are provided to every residual block.Source: §3.5 · p. 7

3.6 Reverse Diffusion and Gaze Synthesis

Reverse sampling begins with Gaussian noise and iteratively denoises the velocity signal. The final normalized velocity is converted to degrees per second and integrated from an initial seed position to produce gaze positions.

3.6 Reverse Diffusion and Gaze Synthesis

  • Starting from Gaussian noise, reverse sampling uses the predicted noise to estimate clean normalized velocity and compute the reverse-process mean.The final denoising step is deterministic.Source: §3.6 · p. 7
  • The generated normalized velocity is converted to degrees/s and integrated from the initial seed position to obtain the synthetic gaze-position trajectory.Source: §3.6 · p. 7

3.7 Loss Functions

The objective combines noise prediction and identity preservation with three signal-fidelity losses targeting frequency structure, cumulative displacement, and event-relevant local dynamics.

3.7.1 Noise-Prediction Loss

  • The diffusion objective minimizes mean-squared error between true and predicted Gaussian noise.Source: §3.7.1 · p. 7

3.7.2 Identity-Preservation Loss

  • The identity loss aligns EKYT representations of generated velocity with those of real velocity.Source: §3.7.2 · p. 8

3.7.3 Proposed Signal-Fidelity Losses

  • Three added losses target distinct signal properties: multi-resolution frequency structure, integrated displacement, and event-weighted local temporal dynamics.Source: §3.7.3 · p. 8

3.7.4 Multi-Resolution Spectral Loss

  • Multi-resolution STFT-magnitude comparisons constrain frequency characteristics beyond pointwise reconstruction, with shorter windows emphasizing rapid changes and longer windows broader patterns.Source: §3.7.4 · p. 8

3.7.5 Drift-Consistency Loss

  • Drift consistency compares final integrated displacement, addressing how small persistent velocity bias can accumulate into substantial position error.Non-finite samples and Basic Noise or Event Detection Noise samples are excluded; noise annotations are not denoiser inputs.Source: §3.7.5 · pp. 8–9

3.7.6 Event-Weighted Local Smoothness Loss

  • Event-weighted smoothness matches generated and real second-order velocity differences, emphasizing fixation, weighting post-saccadic oscillations less, and excluding saccades and noise.It targets local curvature rather than simply minimizing signal roughness.Source: §3.7.6 · p. 9

3.7.7 Combined Objective

  • The combined objective weights identity, spectral, drift, and smoothness losses at 0.5, 0.1, 0.003, and 0.001 alongside noise prediction.Source: §3.7.7 · p. 9

4 Experiments

Experiments use GazeBase recordings and an unseen-user test split to train and evaluate EyeMakeYou. The section specifies implementation, training, and evaluation procedures but supplies no experiment outcomes.

4.1 Dataset

  • GazeBase provides 12,334 left-eye records from 322 volunteers across seven tasks, recorded at 1,000 Hz; only RAN and HSS support spatial accuracy and precision calculations.The 263-participant training set and 59-subject test set have no user overlap.Source: §4.1 · pp. 9–10

4.2 Implementation Details

  • The denoiser concatenates noisy and low-frequency reference velocity, using identity, task, and three-dimensional user-state embeddings as conditions.The embeddings are injected through FiLM modulation, with task also added as an input bias.Source: §4.2 · p. 10

4.3 Training Details

  • Training uses non-overlapping 5-s windows at 1,000 Hz and a 50-step DDPM, optimized for 300 epochs with AdamW.The training objective combines noise prediction, identity preservation, spectral similarity, drift consistency, and event-weighted smoothness.Source: §4.3 · p. 10

4.4 Evaluation Metrics

  • Evaluation covers spatial quality during stable fixation, synthetic–real embedding similarity, qualitative comparison, and physiologically meaningful oculomotor characteristics.Spatial results use error and user percentiles; higher cosine similarity indicates preservation of identity-relevant dynamics.Source: §4.4 · pp. 10–11

5 Results

EyeMakeYou improves median spatial accuracy and EKYT embedding similarity over the compared generators, while precision results vary by task and user percentile. Its signals more closely resemble real traces, and subjective-state associations persist but are task- and round-dependent.

5.1 Spatial Accuracy

  • EyeMakeYou had the lowest median spatial error on HSS (3.60 dva) and RAN (2.81 dva), while its HSS U95|E95 error was 31.15 dva versus 30.85 dva for ground truth.For RAN at U95|E95, EyeMakeYou's error was 28.77 dva, while DiffEyeSyn's was 26.77 dva.Source: Table 1 · p. 11, §5.1 · p. 11

5.2 Spatial Precision

  • At U50|E50, all synthetic models had 0.01 degrees RMS for HSS and RAN; at U95|E95 EyeMakeYou led HSS (0.21 degrees RMS), but the VAE led RAN (0.17 degrees RMS).All synthetic methods had considerably lower U95|E95 dispersion than ground truth, which may indicate smoothing of real high-variability behavior.Source: Table 2 · p. 12, §5.2 · p. 12

5.3 Synthetic Data Similarity

  • EyeMakeYou achieved the highest EKYT embedding similarity across all seven tasks (mean 0.91 for FXS to 0.95 for BLG and HSS), exceeding DiffEyeSyn and the VAE on every task.For BLG, HSS, and RAN, EyeMakeYou scored 0.95 ± 0.03, 0.95 ± 0.01, and 0.94 ± 0.01, respectively.Source: §5.3 · p. 12, Table 3 · p. 13

5.4 Qualitative Evaluations

  • Across HSS, RAN, and TEX, EyeMakeYou more closely reproduced real signals, with clearer fixation plateaus and localized velocity peaks, less jitter than SP-EyeGAN, and fewer abrupt outliers than DiffEyeSyn.In TEX, it maintained smaller-amplitude, denser movement patterns without the excessive drift or oscillation seen in some baseline outputs.Source: §5.4 · p. 13, Figure 2 · p. 14

5.5 User States

  • Synthetic signals retained detectable but exploratory state-feature associations: they were broadest for TEX, sparser for HSS and RAN, and varied across rounds.The analysis evaluates many feature–report pairs and does not establish causal relationships between subjective state and eye-movement behavior.Source: §5.5 · pp. 13–14, Figure 3 · p. 15

6 Discussion

EyeMakeYou offers a stronger overall trade-off than the GAN and VAE baselines and improves on DiffEyeSyn in key settings, with its clearest advantages for typical users and samples. The findings also show that spatial accuracy, fixation variability, identity similarity, and state-related fidelity are distinct dimensions, with limitations in tail behavior and state-relationship preservation.

6 Discussion

  • EyeMakeYou has the lowest median spatial errors for HSS and RAN and the highest real–synthetic EKYT embedding similarity across all seven tasks.Its overall trade-off is stronger than the GAN and VAE baselines and improves on DiffEyeSyn in several key settings.Source: §6 · p. 16
  • For HSS, EyeMakeYou’s U50|E50 error is 3.60 dva and its U95|E95 error is 31.15 dva, close to the real-signal value of 30.85 dva.For RAN, its U50|E50 error is 2.81 dva, but its U95|E95 error of 28.77 dva exceeds DiffEyeSyn’s 26.77 dva.Source: §6 · p. 16
  • All synthetic models have 0.01 degrees RMS precision error at U50|E50, but their U95|E95 dispersion is below real signals, limiting representation of difficult fixation periods.Real-signal U95|E95 values are 0.86 degrees RMS for HSS and 1.14 degrees RMS for RAN; synthetic sequences should complement, not replace, real recordings for worst-case use.Source: §6 · p. 16
  • EyeMakeYou’s cosine similarity ranges from 0.91 to 0.95 across seven tasks, exceeding the VAE and SP-EyeGAN and improving on DiffEyeSyn in every reported task.Because EKYT also supplies the identity condition and preservation loss, this measures consistency in EKYT’s representation rather than independently proving authentication performance.Source: §6 · p. 16
  • Qualitative examples show better preservation of fixation plateaus, major position transitions, and sparse saccade-related velocity peaks than the compared generators.These examples are illustrative and should be read alongside task-wise and user-percentile metrics, not as standalone evidence of general performance.Source: §6 · p. 16
  • Synthetic signals retain selected subjective-report associations, especially for TEX, but these correlations are weaker, sparser, and less stable across recording rounds than in real data.The 58-feature Spearman analysis captures marginal feature–report relationships, not all multivariate or temporal manifestations of state.Source: §6 · pp. 16–17
  • The evaluation covers 5-s GazeBase segments from one 1000-Hz eye tracker and unseen users, but does not establish generalization across devices, labs, longer sessions, or naturalistic environments.The method also requires a real-derived low-frequency reference and an identity embedding from the original velocity signal, so it is intended for conditional augmentation or reconstruction, not unconstrained generation from labels alone.Source: §6 · p. 17
  • EyeMakeYou’s strongest use case is subject-specific synthetic samples for augmenting gaze-based biometric and interactive-system datasets when real data are limited.Remaining targets include tail accuracy, extreme variability, event-level feature equivalence, and state-correlation preservation.Source: §6 · p. 17

7 Conclusion

EyeMakeYou is a diffusion framework for synthesizing subject-specific, high-frequency eye-movement signals using identity, task, and subjective-state conditions. On GazeBase it shows strong spatial and biometric fidelity against evaluated baselines, while user-state associations remain limited and variable.

7 Conclusion

  • EyeMakeYou generates 5-s, 1000-Hz gaze-velocity sequences conditioned on a low-frequency reference, EKYT-derived identity embeddings, task embeddings, and subjective-state embeddings.Training combines noise prediction and identity preservation with multi-resolution spectral consistency, drift consistency, and event-weighted local smoothness.Source: §7 · p. 17
  • On GazeBase, EyeMakeYou has the lowest median spatial errors for HSS and RAN, the lowest high-percentile fixation jitter for HSS, and the highest real–synthetic EKYT similarity across evaluated tasks.Qualitative results indicate preservation of fixation plateaus, task-dependent position transitions, and saccade-related velocity dynamics.Source: §7 · p. 17
  • Synthetic user-state analysis finds task- and round-dependent associations, most distributed for TEX, but these relationships remain limited and variable across rounds.Future work will test additional eye trackers, longer recordings, and naturalistic tasks; report comprehensive authentication and identification performance; and ablate model components.Source: §7 · p. 17

Privacy and Ethics Statement

The study uses anonymized, publicly available eye-tracking data and reports minimal societal risk.

Privacy and Ethics Statement

Loading 2609.04501v1…