Source-linked AI summary
OAT: Object-Level Attention Transformer for Gaze Scanpath Prediction
Yini Fang, Jingling Yu, Haozheng Zhang, Ralf van der Lans, Bertram Shi
TL;DR
Visual-search gaze prediction needs to reflect object-guided attention and distinct gaze behaviors. OAT predicts scanpaths as object fixations, outperforming baselines and generalizing to unseen layouts and targets.
Problem
Visual-search gaze prediction needs to reflect object-guided attention, while common string-matching metrics miss distinct search, revisit, and refixation behaviors.
Source: §1 · p. 2, §4.3 · p. 8Method
Results
OAT outperforms all baselines across metrics, with a 3.12% behavioral difference from human scanpaths on the collected dataset, and generalizes to unseen layouts and targets.
Source: §4.4 · p. 10, §5 · p. 14Takeaways & Limitations
OAT demonstrates that object-level modeling can predict human scanpaths for unseen layouts and target objects.
Source: §5 · p. 14
Abstract
from arXiv · showhide
Visual search is important in our daily life. The efficient allocation of visual attention is critical to effectively complete visual search tasks. Prior research has predominantly modelled the spatial allocation of visual attention in images at the pixel level, e.g. using a saliency map. However, emerging evidence shows that visual attention is guided by objects rather than pixel intensities. This paper introduces the Object-level Attention Transformer (OAT), which predicts human scanpaths as they search for a target object within a cluttered scene of distractors. OAT uses an encoder-decoder architecture. The encoder captures information about the position and appearance of the objects within an image and about the target. The decoder predicts the gaze scanpath as a sequence of object fixations, by integrating output features from both the encoder and decoder. We also propose a new positional encoding that better reflects spatial relationships between objects. We evaluated OAT on the Amazon book cover dataset and a new dataset for visual search that we collected. OAT's predicted gaze scanpaths align more closely with human gaze patterns, compared to predictions by algorithms based on spatial attention on both established metrics and a novel behavioural-based metric. Our results demonstrate the generalization ability of OAT, as it accurately predicts human scanpaths for unseen layouts and target objects.
1 Introduction
The paper frames visual search as a sequence of gaze shifts and argues that attention is guided by objects, not just pixel intensities. It introduces OAT to predict object fixations and reports stronger alignment with human scanpaths.
1 Introduction
- Predicting gaze during everyday target searches may inform user-intention estimation, robot guidance, VR and AR experiences, and advertising strategies.The examples include hierarchical-task robot guidance in HCI and VR/AR.Source: §1 · p. 1
- Visual attention in goal-directed behavior is increasingly understood as object-directed, while prior scanpath models largely predict pixel-level salience.The introduction describes pixel-based methods that select high-salience locations over time, often with inhibition of return.Source: §1 · p. 2
- OAT predicts the next fixated object from past fixations and encoder representations of the target and scene.Its object-embedding module extracts visual and geometrical features for each object, and predictions can be sampled from the decoder’s probability distribution.Source: §1 · pp. 2–3
- OAT achieves state-of-the-art performance against baselines, with predicted scanpaths differing from human scanpaths by only 3.1%.The comparison uses previously proposed standard metrics and a new behavioral metric.Source: §1 · p. 3
- The paper proposes object-level scanpath prediction, a spatial positional encoding, scalable object attention for varying object counts without retraining, and a behavioral comparison metric.The metric compares distributions of task-relevant gaze behaviors based on psychological processes of visual attention.Source: §1 · p. 3
2 Related Work
Prior scanpath research includes pixel-level free-viewing models and goal-directed methods that condition sequences on a target. OAT differs by predicting object fixations and is presented as better suited to challenging scenes with clutter or similar-looking distractors.
2 Related Work
- Goal-directed scanpath models include target-specific inverse reinforcement learning, its COCO-Search18 extension, and transformer-based approaches [35].The cited approaches generate fixation sequences while modeling correlations with the target.Source: §2 · p. 3
- Gaze target detection by Tononi et al. [31] predicts a single point inside an image, not an external observer’s scanpath.This distinguishes point-of-regard prediction from the task of predicting a sequence of fixations.Source: §2 · p. 4
- 8.95 fixations on the Amazon book cover dataset versus 3.5 on COCO-Search18; the paper reports object-level modeling outperforms pixel-level modeling in challenging scenarios.The paper characterizes the challenging scenarios as involving random clutter, disorganization, or similar-looking distractors.Source: §2 · p. 4
- Most prior goal-directed search models predict pixel-level rather than object-level scanpaths, a distinction the paper highlights for cluttered scenes.The paper notes that pixel-based models may suffice when objects are distinct and likely locations can be anticipated.Source: §2 · p. 4
- OAT differs from Mondal et al. by encoding the target as an extra token, feeding past fixations to the decoder, and predicting objects rather than pixels.The paper says the target token lets objects similar to the target be encoded differently from dissimilar objects.Source: §2 · p. 4
3 OAT Architecture
OAT embeds the target and scene objects, then predicts the next fixation by combining encoder representations with decoder fixation history.
3 OAT Architecture
- OAT processes target-and-scene object embeddings with transformer encoder and decoder blocks, then predicts a probability distribution over the next fixated object.Generation continues until the end token is most probable.Source: §3 · p. 4, Figure 2 · p. 5
- Each object receives a CNN-based visual descriptor and positional embedding, while the target is included as an additional object embedding.The visual descriptors are extracted from rectangular image patches.Source: §3 · pp. 4–5
- Its learnable positional encoding concatenates 1D encodings for image coordinates x and y and target indicator z, designed for cosine similarity to decay approximately as a Gaussian with spatial distance.The encoding uses three spatial dimensions: x, y, and z.Source: §3 · p. 6
- Decoder cross-attention uses encoder outputs as keys and values and decoder inputs as queries, passing the last-fixation state and encoder states to object attention.Source: §3 · p. 6
- Object attention combines encoder keys with the last-fixation query into object scores; Softmax adds an end-token probability, terminating generation when EOS is most probable.OAT averages outputs from identical object-attention blocks.Source: §3 · pp. 6–7
4 Experimental Results
OAT was evaluated on two visual-search datasets using established scanpath metrics and a new behavioral measure. It outperformed the baselines across metrics, with a 3.12% behavioral difference from human scanpaths on the collected dataset.
4.1 Datasets
- The evaluation used the Amazon book cover dataset and a collected dataset of yogurt and wine product arrays.The Amazon dataset contains 102 images of 6 by 14 shuffled book covers; the collected arrays contain wine and yogurt packages.Source: §4.1 · pp. 7–8
4.2 Implementation Settings
- OAT was trained with four encoder and decoder blocks, two object-attention blocks, and a random 8:1:1 training/validation/testing split.Each layer used four attention heads; the hidden-state size was 256.Source: §4.2 · p. 8
4.3 Metrics
- Evaluation averaged 100 sampled trajectories and used Sequence Score, Fixation Edit Distance, behavioral measures, and Multi Match.Multi Match comparisons convert OAT's object-level predictions to pixel-level fixations at object centers.Source: §4.3 · pp. 8–9
- The new behavioral measure compares search, revisits, and refixations, alongside target accuracy, average scanpath length, and overall percentage difference from human scanpaths.Search covers objects not viewed before; revisits cover previously viewed objects other than the last fixation; refixations repeat the last-fixated object.Source: §4.3 · pp. 8–9
4.4 Quantitative Performance Comparison
- OAT outperformed all baselines across metrics on both datasets, with a 3.12% behavioral difference from human scanpaths on the collected dataset.The comparisons are reported in the Amazon book cover and collected-dataset tables.Source: Table 2 · p. 10, Table 3 · p. 10, §4.4 · p. 10
- Despite coarse object-level predictions, OAT outperformed other baselines in average pixel-level Multi Match scanpath similarity across the two datasets.The authors attribute this to more accurate estimates of fixated objects, which yield closer pixel-level predictions despite object-level quantization.Source: Table 4 · p. 10, §4.4 · p. 11
4.5 Scanpath Visualization in Spatial Dimension
OAT's predicted object-level attention heatmaps closely aligned with human heatmaps. A held-out layout-target pair also showed comparable patterns to human scanpaths, supporting generalization to unseen layouts and target objects.
4.5 Scanpath Visualization in Spatial Dimension
- Predicted object-level heatmaps closely aligned with human heatmaps, with only minor differences in probability distributions.OAT heatmaps average 100 predicted scanpaths and show the percentage of time each object was viewed.Source: §4.5 · p. 11
4.6 Generalization to Unknown Categories
- With one layout-target pair held out for testing, OAT showed patterns comparable to human scanpaths and generalized to unseen layouts and target objects.The remaining collected data were used for training.Source: §4.6 · pp. 11–12, Figure 6 · p. 12
- Table 5 reports parameter analysis for DPE, E2E, and RPE.Source: Table 5 · p. 12
4.7 Parameter Analysis of Distance-based PE v.s. End-to-End Learnable PE and Relative PE
The analyses favor distance-based positional encoding and object-level fixation units, while showing that cross-attention and the proposed components contribute to more human-like scanpaths.
4.7 Parameter Analysis of Distance-based PE v.s. End-to-End Learnable PE and Relative PE
- DPE outperforms E2E PE and RPE by 16% in behavioural metric, and its Manhattan-distance saccade distribution is closer to human behaviour than E2E PE.E2E PE produces more long saccades; the authors suggest geometry priors are especially useful when gaze-training data are limited.Source: §4.7 · pp. 12–13, Figure 7 · p. 13
4.8 Importance of object-centric modelling
- Performance degrades substantially when patches cover parts of objects, but only slightly when each fixation represents a group of objects.The comparison tests grids finer than the object level and coarser grids containing multiple objects.Source: §4.8 · p. 13, Table 6 · p. 13
4.10 Ablation Study
- Removing DPE and OA made behavioural difference 17 times worse; removing DPE alone caused 3.84 times degradation and removing OA caused 5.45 times degradation.The authors associate DPE with modelling shelf geometry and OA with capturing dependencies and improving flexibility.Source: Table 7 · p. 14, §4.10 · p. 14
4.9 Improving revisit/refix behaviour by the cross attention
- Replacing fixation 2’s object with a random object changed fixation probabilities by −3.42%, 18.14%, 21.53%, and 33.22% at fixations 3–6, bringing revisit and refixation behaviour closer to human behaviour.The authors identify cross-attention as critical to modelling revisits and refixations.Source: §4.9 · pp. 13–14
5 Conclusion
The conclusion presents OAT as an object-level transformer for gaze scanpaths. It reports stronger baseline performance and generalization to unseen layouts and targets, while identifying non-uniform object shapes as a possible extension.
5 Conclusion
- OAT is a transformer-based architecture for modelling gaze scanpaths at the object level.Source: §5 · p. 14
- OAT outperforms all prior baselines in modelling human behaviour.Source: §5 · p. 14
- OAT demonstrates generalization to unseen layouts and targets.Source: §5 · p. 14
- The authors identify searching for objects with non-uniform shapes as a potential extension.Source: §5 · p. 14
Supplementary Material
The supplementary material provides additional details about the collected dataset.
Supplementary Material
- The supplementary material provides additional details about the collected dataset.Source: Supplementary Material · p. 18
1 Preparation
The preparation section describes the visual-search stimuli and participants used for the collected dataset.
1 Preparation
- The study selected 22 wine products from 21 brands and 27 yogurt products from 13 brands.Source: §1 · p. 18
- For each product type, 20 items were designated as targets, and 100 shelf images were created by arranging packages in permutations.Source: §1 · p. 18
- Wine shelves used 2 rows and 11 columns, while yogurt shelves used 3 rows and 9 columns; images were 1680 × 1050 pixels.Source: §1 · p. 18
- The study collected eye-tracking data from 39 participants aged 18–30 using a Tobii T60XL at 60 Hz.Source: §1 · p. 18
2 Collection Process
The study collected gaze scanpaths while participants searched for target products in shelf images. The collected data were clustered into fixations and cleaned to form the final dataset.
2 Collection Process
- Participants were instructed to find a specific target product in a shelf image, with the study instruction shown in Figure 3.Figures 1 and 2 show example wine and yogurt stimuli.Source: §2 · p. 18, Figure 1 · p. 19, Figure 2 · p. 19
- Each subject received 30 unique target-product and shelf-image combinations; each trial showed the target for five seconds before a three-second search-and-click task.Compensation depended on both speed and accuracy.Source: §2 · p. 18
- After cleaning, the dataset contained 858 scanpaths averaging 7.7 fixations, down from 1,170 collected during the experiment.Researchers removed 312 scanpaths with less than 80% record integrity [?] and eliminated redundant fixations after subjects clicked the mouse.Source: §2 · pp. 18–19
3 Consent From Subjects
The study received IRB approval and obtained signed informed consent from all subjects. The consent provisions restricted data use and protected participant confidentiality.
3 Consent From Subjects
- Before the study, the researchers obtained IRB approval and signed informed consent from all subjects.The consent form outlined the study's data-use and confidentiality provisions.Source: §3 · p. 19
- Participant data were designated for research use only, kept confidential, and not linked to identifying information.The consent provisions specified that names, email addresses, or other personally identifiable information would not be collected or stored in a way that linked responses or eye-tracking data to identities.Source: §3 · p. 19
- Publications using the data were limited to aggregated data, and the consent form is included in the appendix.Names and institutions were anonymized for initial review.Source: §3 · p. 20