Source-linked AI summary
EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning
Yue Jiang, Zixin Guo, Hamed Rezazadegan Tavakoli, Luis A. Leiva, Antti Oulasvirta
TL;DR
Existing models predict average attention patterns, but scanpath prediction for individuals remains an open challenge, despite the importance of fixation order and duration across different stimuli. EyeFormer uses a Transformer-guided reinforcement-learning policy to predict personalized and population-level scanpaths, and the paper reports favorable population-level comparisons and personalized GUI layout applications.
Problem
Existing models can predict average attention patterns, but no scanpath model predicts an individual’s scanpath, while static saliency maps omit fixation order and duration.
Source: §1 · pp. 1–2Method
EyeFormer uses a Transformer as a reinforcement-learning policy to predict scanpaths, including spatial and temporal fixation information, and can personalize predictions from a few user samples.
Source: §1 · p. 2Results
EyeFormer performs better or on par with prior state-of-the-art models on population-level prediction and models individual scanpath variability from a few samples.
Source: §7 · p. 12Takeaways & Limitations
The paper demonstrates personalized GUI layouts based on a user’s scanpath samples and a single optimized layout with minimal variability across individuals.
Source: §1 · p. 2The model is limited to fixed-length scanpaths and fixation prediction, using a seven-second free-viewing window based on UEyes.
Source: §7 · p. 12
Abstract
from arXiv · showhide
From a visual perception perspective, modern graphical user interfaces (GUIs) comprise a complex graphics-rich two-dimensional visuospatial arrangement of text, images, and interactive objects such as buttons and menus. While existing models can accurately predict regions and objects that are likely to attract attention ``on average'', so far there is no scanpath model capable of predicting scanpaths for an individual. To close this gap, we introduce EyeFormer, which leverages a Transformer architecture as a policy network to guide a deep reinforcement learning algorithm that controls gaze locations. Our model has the unique capability of producing personalized predictions when given a few user scanpath samples. It can predict full scanpath information, including fixation positions and duration, across individuals and various stimulus types. Additionally, we demonstrate applications in GUI layout optimization driven by our model. Our software and models will be publicly available.
1 INTRODUCTION
GUI attention is difficult to predict for a given person because interfaces are visually complex and viewing strategies differ. EyeFormer predicts spatial and temporal scanpath characteristics at both individual and population levels, and the paper demonstrates personalized GUI layout optimization.
1 INTRODUCTION
- GUI attention prediction is challenging because interfaces combine text, images, animations, and interactive objects, while viewing strategies vary between people.The paper notes that large individual differences make an “average scanpath” unrepresentative of actual viewers’ behavior.Source: §1 · pp. 1–2
- Unlike static saliency maps, scanpaths preserve fixation order and duration, and prior scanpath research has predominantly focused on natural scenes.The introduction motivates unified modeling across stimuli because viewing patterns can differ between websites and mobile GUIs.Source: §1 · p. 2
- EyeFormer compares favorably with state-of-the-art models on most population-level scanpath metrics for GUIs and natural scenes.The paper also reports an application of personalized scanpath prediction to GUI layout optimization.Source: §1 · p. 2
- EyeFormer uses deep reinforcement learning with a Transformer to predict scanpath spatial and temporal characteristics, including fixation coordinates and durations.It can generate personalized scanpaths from a few samples from the relevant viewer.Source: §1 · p. 2
- EyeFormer predicts an individual’s viewing behavior from a few scanpath samples, reflecting that viewer’s behaviors and preferences.The introduction also describes personalized GUI optimization as an application of personalized scanpath prediction.Source: §1 · p. 2
2 RELATED WORK
Prior scanpath prediction methods derive sequences from saliency maps, predict fixation sequences directly, or frame prediction as sequential control with reinforcement learning. The review describes limitations involving temporal information, fixation ordering, differentiability, and continuous-action precision.
2 RELATED WORK
- Scanpath prediction retains the order of discrete fixations, unlike dense saliency-map prediction, and prior models mainly represented average users rather than individuals.The review groups prior approaches by whether they derive sequence information post-hoc, predict sequences directly, or use reinforcement learning.Source: §2 · p. 2
2.1 Saliency Map-based Scanpath Prediction
- Saliency-map methods using inhibition of return can derive sequences, but neglect fixation duration, lack coherent fixation ranking, and are non-differentiable.The review cites Itti et al. for the inhibition-of-return mechanism and notes its subsequent development in other work [2] [29].Source: §2.1 · pp. 2–3
2.2 Predicting Fixation Sequences
- Direct sequence methods include parameterized Gaussian distributions, grid-based representations, and GANs, whose reported challenges include accumulated errors or reduced fixation-point accuracy.The review also describes NeVA [43] as relying on pretrained models for downstream visual tasks rather than simulating human scanpaths.Source: §2.2 · p. 3
2.3 Reinforcement Learning for Scanpath Prediction
- Grid discretization enables optimization over finite actions but coarsens fixation representation, whereas continuous control has infinitely many feasible actions.EyeFormer formulates fixation prediction as continuous value generation and uses parametric Gaussian action distributions with designed rewards.Source: §2.3 · p. 3
- Figure 2 shows inhibition areas being scaled from display dimensions to the input image and rendered as ellipses after model resizing.The display inhibition diameter is commensurate with human visual angle; the input-map radii account for the resizing dimensions.Source: Figure 2 · p. 3
3 METHOD
EyeFormer formulates sequential fixation prediction as a reinforcement-learning problem, using a Transformer policy to predict the next fixation from the image and preceding fixations. This approach accommodates spatial and temporal fixation information and non-differentiable rewards.
3 METHOD
- EyeFormer uses a Transformer to capture long-range dependencies from previous fixations and reinforcement learning to optimize scanpaths with non-differentiable rewards.The reward examples include salient values computed with inhibition of return.Source: §3 · p. 3
3.1 Problem Formulation
- A scanpath is an ordered sequence of T fixations, each represented by normalized x and y coordinates and a positive fixation duration.The sequence captures spatial and temporal information about human gaze.Source: §3.1 · p. 3
3.2 Environment, State, and Action
The model’s policy predicts the next fixation from the input image and previously predicted fixations. Figure 3 shows how the environment, Transformer policy, action, and reward are connected during training and testing.
3.2 Environment, State, and Action
- The policy samples the next fixation from a distribution conditioned on the image and preceding predicted fixations.The model interacts with an environment whose state includes the input image and previous fixation points.Source: §3.2 · p. 3
- Figure 3 shows the Transformer policy generating the next-point action, with ground-truth-based reward updating the policy during training; testing generates scanpaths directly from the policy.For individual-level prediction, the viewer encoder supplies a viewer embedding alongside the image embedding and previous fixations.Source: Figure 3 · p. 4
3.3 Reward Function
EyeFormer’s reward combines whole-scanpath similarity to the ground truth with rewards for salient fixation locations. The components account for fixation position and duration, while inhibition discourages repeated attention to previously predicted locations.
3.3 Reward Function
- The reward combines DTWD similarity between predicted and ground-truth scanpaths with the summed salient-value rewards for predicted fixations.Training minimizes negative expected reward, equivalent to maximizing positive reward.Source: §3.3 · p. 4
3.3.1 Dynamic Time Warping with Duration (DTWD).
- DTWD spatially aligns scanpaths by fixation position and compares them as 3D vectors of position and duration.It is computed over the full scanpath to encourage predictions closer to ground-truth trajectories and durations.Source: §3.3.1 · p. 4
3.3.2 Salient Value.
- Inhibition of return sets the salient-value reward to zero for points inside areas around previously predicted fixations.The inhibition areas are resized from circles to ellipses to account for image resizing; the chosen radius still permits returning to an already visited element.Source: §3.3.2 · pp. 4–5
3.4 Policy Network
The policy network encodes an image and then generates a sequence of fixation points. For personalized predictions, the decoder can additionally receive a viewer embedding.
3.4 Policy Network
- The policy network first learns an image representation and then generates a scanpath with a fixation decoder.Population-level prediction uses the image representation alone; individual-level prediction also supplies a viewer embedding.Source: §3.4 · p. 5
3.4.1 Vision Encoder.
- A 12-layer Vision Transformer encodes image patches and combines their relationships into a final image embedding.The image is resized and split into non-overlapping patches before encoding.Source: §3.4.1 · p. 5
3.4.2 Fixation Decoder.
- The fixation decoder uses the image embedding and previously generated points to generate the next fixation.The first fixation is set at the screen center, and the policy is represented as a Gaussian distribution.Source: §3.4.2 · p. 5
3.5 Predicting Personalized Scanpaths
EyeFormer adapts its viewer encoder using a few scanpaths from a new viewer, then uses the resulting viewer representation to predict that person’s scanpaths.
3.5 Predicting Personalized Scanpaths
- At test time, a few scanpaths from a new viewer update the viewer encoder through backpropagation, enabling predictions specific to that viewer.The viewer encoder is trained to represent distinct viewing behaviors in separate embedding spaces; its output is sent to the fixation decoder.Source: §3.5 · p. 5
3.6 Policy Gradient
EyeFormer trains its policy with REINFORCE, estimating gradients from sampled scanpaths. A baseline estimates expected reward and is used to reduce gradient-estimation variance.
3.6 Policy Gradient
- REINFORCE updates policy parameters using samples from episodes to estimate the gradient of expected cumulative reward.The method uses a single Monte Carlo scanpath sample for each training example in a minibatch.Source: §3.6 · pp. 5–6
- A baseline estimates expected reward without actions, and subtracting it reduces gradient-estimation variance.In continuous space, the baseline is determined by the reward associated with the mean of multiple samples drawn from the policy.Source: §3.6 · p. 6
4 EXPERIMENTS
The experiments evaluate EyeFormer on GUI and natural-scene scanpaths using multiple measures of spatial, temporal, and sequence similarity, including a test of viewer-specific prediction.
4.2 Metrics
- The evaluation covers GUI and natural-scene datasets with multiple scanpaths per image and uses commonly employed scanpath metrics.Source: §4.1 · p. 6, §4.2 · p. 6, §4 · p. 6
4.1.1 GUIs and Information Graphics.
- UEyes contains eye-tracking data from 62 participants viewing 1,980 GUI and information-graphic images; OSIE contains 700 natural-scene images viewed by 15 participants.UEyes uses training/test splits of 1,872/108 images; OSIE follows an 80%/10%/10% split.Source: §4.1.1 · p. 6, §4.1.2 · p. 6
4.2.1 Dynamic Time Warping (DTW).
- DTW compares whole scanpaths as temporal sequences and can match sequences of different lengths.It identifies an optimal match while preserving essential scanpath features.Source: §4.2.1 · p. 6
4.2.2 Time Delay Embedding (TDE).
- TDE compares subscanpaths, providing a more localized assessment than DTW’s whole-scanpath comparison.Source: §4.2.2 · p. 6
4.2.3 Eyenalysis.
- Eyenalysis matches each fixation to its spatially closest fixation in the other scanpath and averages the paired distances.Source: §4.2.3 · p. 6
4.2.4 Dynamic Time Warping with Duration (DTWD).
- DTWD extends DTW to include fixation duration alongside position, using spatially aligned (x, y, t) fixation vectors.Source: §4.2.4 · p. 6
4.2.5 MultiMatch.
- MultiMatch evaluates shape, direction, length, position, and duration, extending beyond DTWD’s spatial and temporal comparison.Source: §4.2.5 · p. 6
4 EXPERIMENTS
- For individual-level evaluation, the study compares a model personalized to the target test viewer with models personalized to other test viewers on UEyes.Figure 4 illustrates personalized scanpaths generated from a few samples per viewer; its Viewer 1 and Viewer 2 labels do not identify the same people across examples.Source: Table 1 · p. 7, Figure 4 · p. 7
5 RESULTS
EyeFormer’s population-level predictions outperform baseline models on most reported metrics across GUIs and natural scenes, while also supporting viewer-specific prediction. Qualitative comparisons and ablations examine its trajectories and the contributions of reinforcement learning and reward components.
5 RESULTS
- Population-level scanpaths are reported as capturing spatial and temporal characteristics across GUIs and natural scenes, alongside the individual-level results.Figure 5 illustrates population-level predictions against ground truth for fixation positions, ordering, and duration.Source: §5 · p. 7, Figure 5 · p. 8
5.1 Individual-Level Scanpath Prediction
- Target-viewer personalization yields smaller errors than models personalized to other test viewers, and qualitative examples show scanpaths tailored from a few viewer samples.No established baseline method was available for this individual-level comparison.Source: §5.1 · p. 7
5.2.1 Quantitative Evaluation.
- EyeFormer outperforms baseline models on most scanpath metrics for both GUIs and natural scenes, with strong results on duration-sensitive DTWD and MultiMatch Duration.Table 2 reports mean ± SD by metric; dashes denote methods unable to predict duration.Source: §5.2.1 · p. 8, Table 2 · p. 10
5.2.2 Qualitative Evaluation.
- Qualitative comparisons place EyeFormer’s predictions closer to ground truth; its salient-value and DTWD rewards support salient fixations and more accurate fixation ordering.The comparisons discuss baseline weaknesses including clustered points and incorrect fixation order; Figures 6 and 7 show comparisons with prior models, including Chen et al. [7].Source: §5.2.2 · pp. 8, 10, Figure 6 · p. 9, Figure 7 · p. 9
5.3 Ablation Study
- Removing either reward type or inhibition of return reduces overall accuracy, while reinforcement learning improves fixation and duration prediction over a Transformer-only model.For population-level prediction, RL improves TDE by 14.7%, Eyenalysis by 26.5%, and DTWD by 4.8%.Source: §5.3 · p. 10
6 APPLICATION: PERSONALIZED VISUAL FLOWS
The application uses predicted population-level or personalized scanpaths to optimize GUI layouts around a designer-specified fixation order. In tests on 62 viewers, personalized layouts increased selected-element fixation durations compared with population-optimized layouts.
6 APPLICATION: PERSONALIZED VISUAL FLOWS
- The system produces population-optimized and viewer-specific layouts from a GUI design and the designer’s ordered list of important elements, using 50 scanpaths per viewer in these experiments.Individual layouts use personalized scanpath predictions.Source: §6 · pp. 10–11
6.2 Results
- For Design 1, personalized layouts raised average selected-element fixation duration from 1.29 to 1.86 seconds (+44.19%); all 62 viewers followed the desired order.Figure 8 shows example layouts and attention outcomes.Source: Figure 8 · p. 11, §6.2 · p. 12
6.1 Formulation of Optimization Problem
- Layout optimization selects element positions and sizes using predicted personalized scanpaths, building on an integer-programming optimizer for packing, alignment, and preferential positioning while incorporating the designer’s order and EyeFormer-derived objective.The formulation adds a constraint for the designer-specified fixation order.Source: §6.1 · p. 11
6.1.1 Fixation Order Constraint.
- The order constraint requires the predicted sequence to visit the designer’s elements in order; among layouts satisfying it, the objective maximizes their summed fixation duration.The specified order can extend beyond three elements.Source: §6.1.1 · p. 11, §6.1.2 · p. 12
6.2 Results
- For Design 2, personalized layouts raised average selected-element fixation duration from 2.75 to 3.19 seconds (+16%); 61 viewers followed the desired order, versus 46 on the population layout.The comparison tests both duration and adherence to the designer’s requested order.Source: §6.2 · p. 12
7 DISCUSSION
EyeFormer combines a Transformer policy with reinforcement learning, reporting state-of-the-art individual scanpath modeling and performance at or above prior methods for population prediction. The discussion also identifies practical data acquisition and modeling-scope constraints.
7 DISCUSSION
- EyeFormer reports state-of-the-art individual scanpath modeling from a few samples and better-or-on-par population prediction; Transformers capture long-range fixation dependencies, while reinforcement learning optimizes non-differentiable objectives.The authors describe the Transformer as central but insufficient by itself for optimal results.Source: §7 · p. 12
- The model covers spatial and temporal scanpath characteristics across stimuli and individual viewing behaviors, and the authors identify personalized GUI flows as a potential accessibility application.They suggest optimizing interfaces for people with viewing difficulties.Source: §7 · p. 12
- Obtaining individual scanpath data remains a practical challenge for personalization; proposed options include permitted webcam use, calibration sessions, or mouse-movement proxies.These are presented as possible ways to acquire or infer user data.Source: §7 · p. 12
- The model predicts fixed-length scanpaths within a seven-second free-viewing window and fixations only; the authors also flag 50 samples per viewer and natural-scene-oriented metrics as limitations.Future work could address variable lengths, other eye dynamics, fewer samples, and metrics better suited to GUIs.Source: §7 · p. 12