Source-linked AI summary
EEGaze: Multimodal Attention Prediction in Visual Search via EEG-Based Saliency Refinement
Mansi Sharma, Dinar Orazgaliyev, Antonio Krüger, Philipp Müller
TL;DR
The paper addresses gaps in visual-search attention prediction: models largely overlook cognitive state, and realistic public datasets combining EEG and eye tracking are scarce. It introduces EEGaze-140 and an EEG-guided saliency-refinement method; evaluations report consistent gains over baselines and ablations.
Problem
Visual attention models largely overlook general cognitive state, while large public datasets combining EEG and eye tracking in naturalistic search scenarios are lacking.
Source: §1 · p. 2Method
EEGazeNet uses EEG recorded before search to refine the output of an image-based saliency model, and EEGaze-140 provides synchronized EEG and eye-tracking data from 36 participants across 140 realistic scenes.
Source: §1 · p. 2Results
EEGazeNet consistently outperforms all baselines and ablation conditions in evaluations on EEGaze-140.
Source: §7 · p. 8Takeaways & Limitations
The authors present EEGaze-140 as a valuable resource for future research on EEG-informed attention prediction.
Source: §7 · p. 9The dataset is limited to relatively young university-recruited participants and two visual-search scenarios, while real-world settings may introduce EEG movement artifacts.
Source: §6.3 · p. 8
Abstract
from arXiv · showhide
Visual search is ubiquitous in daily life, from locating keys in a messy apartment to finding a tool in a cluttered workshop or an application icon on a desktop screen. While visual attention is to a large extent driven by image content, research in psychology indicates that user cognitive states have a significant influence on attention allocation. A promising way to access information about cognitive states is via Electroencephalography (EEG) recordings. Despite this potential, previous works in computational attention modeling have largely disregarded EEG, and related studies in psychology operate with abstract, simplistic stimuli, or even artificially guided search procedures. We bridge this gap with two contributions. First, we present EEGaze 140, the largest publicly available multimodal dataset of free visual search with realistic stimuli. The dataset contains synchronized EEG and eye tracking recordings from 36 participants across 140 naturalistic scenes, covering two scenarios: searching for desktop icons and finding tools in cluttered workshop scenes. Second, we introduce EEGazeNet, the first method for human attention prediction in visual search that integrates EEG signals. Our approach consists of a EEG guided refinement stage that adapts the output of a state of the art saliency prediction method to the user current cognitive state. In comprehensive evaluations on EEGaze 140, we show that EEGazeNet consistently outperforms all baselines and ablation conditions.
1 Introduction
The paper argues that attention prediction for visual search should account for users’ cognitive states as well as image content. It introduces a multimodal dataset and an EEG-guided refinement approach, which consistently improves attention prediction in the reported evaluations.
1 Introduction
- Visual attention reflects cognitive factors such as cognitive load and fatigue, so effective prediction needs user-state information alongside image evidence.The introduction links attention prediction to interactive systems including assistive interfaces, adaptive displays, and human-centered AI.Source: §1 · p. 2
- Existing image-based, personalized, and task-specific approaches largely overlook general cognitive state, while EEG offers a potential source of that information.The paper identifies both a shortage of large public datasets combining EEG and eye tracking in naturalistic search and uncertainty about integrating EEG into attention-prediction pipelines.Source: §1 · p. 2
- EEGaze-140 is a multimodal visual-search dataset of 140 realistic stimuli, with synchronized EEG and eye-tracking from 36 participants across desktop-icon and cluttered-workshop searches, totaling 5,040 trials.EEGazeNet refines a state-of-the-art saliency prediction output using EEG signals reflecting cognitive state before search; EEG incorporation consistently improves attention prediction across all metrics in cross-user and within-user evaluations.Source: §1 · p. 2
- The authors present EEGaze-140 as a foundation for future research on multimodal attention modeling in visual search.Figure 2 shows examples of the desktop and workshop scenes created for the study.Source: §1 · p. 2, Figure 2 · p. 2
2 Related Work
Prior work models attention from image content and has explored task and user information, EEG-attention links, and visual-search datasets. The paper identifies cognitive state and a lack of realistic, publicly available EEG-plus-eye-tracking search data as gaps.
2 Related Work
- The related-work review covers visual attention prediction, EEG links to attention, and existing EEG-plus-eye-tracking datasets for visual search.The table caption specifies that dataset size is the product of users and scenes shown to each user, and marks datasets using only one type of realistic stimulus.Source: §2 · p. 2, Table 1 · p. 3
2.1 Visual Attention Prediction
- Saliency research progressed from pretrained CNNs to Transformer and hybrid models, while other approaches added task characteristics or user personalization.The paper positions its work against models including SUM [21] and task-specific or personalized approaches [39, 45, 50, 54, 57].Source: §2.1 · pp. 2–3
2.2 EEG and Visual Attention
- These models largely overlook internal cognitive states, and prior EEG studies did not integrate EEG as an additional modality into image-based attention prediction.The paper also distinguishes its use of EEG recorded before stimulus presentation from prior EEG-based attention work using signals recorded during viewing.Source: §2.1 · p. 3, §2.2 · p. 3
2.3 Visual Search Datasets
- Visual-search EEG and eye-tracking datasets are scarce; many use abstract stimuli or guided searches, and most are not publicly available.The paper notes that Sharma et al. used realistic workshop scenes but had 1,800 trials from 15 users; EEGaze-140 covers workshops and desktops with 36 users and 5,040 trials.Source: §2.3 · p. 3
3 Dataset
The dataset combines synchronized EEG and eye tracking during searches in realistic desktop and workshop scenes. The section describes how the data were acquired, summarizes search behavior, and explains how gaze was converted into attention maps.
3 Dataset
- The dataset section introduces the recording process, descriptive statistics, and generation of ground-truth attention maps.Figure 4 provides example saliency and binary fixation maps for workshop and desktop stimuli.Source: §3 · p. 3, Figure 4 · p. 5
3.1 Data Acquisition
- The study recruited 36 university volunteers and recorded synchronized EEG and eye tracking during searches across 140 scenes, split between workshop and desktop settings.Participants searched for a shown target, then clicked it only after locating it to reduce mouse-pointer distraction and hand-movement effects during the search phase.Source: §3.1 · pp. 3–4
3.2 Descriptive Statistics
- Across 5,040 visual-search trials, workshop search duration rises as scene object counts increase from 50 to about 450, while desktop scenes generally have 50–90 objects and average search durations below 10 seconds.The authors suggest greater occlusion may contribute to the workshop increase; occlusions between desktop objects were not possible.Source: §3.2 · p. 4
3.3 Attention Map Generation
- Gaze preprocessing identifies fixations, accumulates them into a binary map, then applies Gaussian smoothing to produce a normalized continuous saliency map.The fixation detector discards fixations shorter than 60 ms, and the smoothing uses σ = 40 pixels.Source: §3.3 · p. 5, Figure 4 · p. 5
4 Method
EEGazeNet refines an image-based saliency prediction with EEG-derived spatial information to predict attention maps sensitive to a user’s cognitive state. Its method combines this refinement architecture with a composite loss, staged training, and features extracted from preceding-trial EEG.
4.1 EEGazeNet Architecture
- EEGazeNet starts with SUM saliency, projects EEG features into a spatial map, weights both modalities, refines them with a VSSM, and residual-fuses the output with the initial map.The EEG pipeline uses fully connected layers and bilinear upsampling to align EEG features with the image resolution.Source: §4.1 · pp. 5–6, Figure 5 · p. 5
4.2 Loss Function
- The composite loss is L1 + 7 · L2, combining SUM-adopted attention metrics with a mean-squared-error term.L1 combines CC, KL, SIM, and NSS; L2 penalizes pixel-wise differences between predictions and ground truth.Source: §4.2 · p. 6
4.3 Implementation Details
- Training proceeds in two stages: train SUM on images, then initialize the EEG-integrated refinement model with its pretrained weights.The SUM stage uses 30 epochs with early stopping; the refinement stage uses batch size 8 to accommodate EEG input.Source: §4.3 · p. 6
4.4 EEG Features
- EEG preprocessing filters and cleans the signals, interpolates bad channels, uses common-average referencing, and rejects artifact components identified by SOBI.The artifact rejection threshold is 95%.Source: §4.4 · p. 6
- EEG features come from 20 channels and include 15 descriptive features per channel or four bandpower ratios, extracted from the preceding trial.The descriptive set combines spectral, statistical, complexity, shape, and amplitude measures; the ratios use theta, alpha, and beta powers.Source: §4.4 · p. 6
5 Evaluation
EEGazeNet is evaluated across desktop and workshop scenes in cross-user and within-user settings. It consistently outperforms the baselines and ablations in both settings, with participant-level tests showing significant gains over fine-tuned SUM.
5.1 Evaluation Metrics
- NSS and AUC measure fixation-location prediction, while CC, SIM, and KLD assess distribution alignment; lower KLD and higher scores on the other metrics indicate better performance.The evaluation reports average performance with 95% confidence intervals across users.Source: §5.1 · p. 6, §5.2 · p. 6
5.2 Evaluation Scenarios
- Cross-user evaluation tests on unseen users and images, whereas within-user models are trained and evaluated separately for each user on different images.Workshop and desktop scenes are evaluated separately in both settings.Source: §5.2 · p. 6
5.3 Cross-User Results
- EEGazeNet with EEG Descriptive and Bandpower Ratios features outperforms all baselines and ablations across cross-user metrics, significantly beating fine-tuned SUM on all five metrics (corrected p < 0.05).The advantage over fine-tuned SUM is slightly more pronounced for workshop than desktop scenes.Source: §5.3 · p. 7, Table 2 · p. 7
5.4 Within-User Results
- For desktop KL, EEGazeNet improves over fine-tuned SUM by 0.271 within-user versus 0.196 cross-user; it also beats all within-user baselines and ablations across metrics (corrected p < 0.001 vs fine-tuned SUM).The fine-tuned SUM advantage over non-fine-tuned SUM is smaller within-user than cross-user: 0.074 versus 0.295 for desktop KL.Source: §5.4 · pp. 7–8, Table 3 · p. 7
6 Discussion
EEGazeNet outperformed baselines and ablations in both cross-user and within-user evaluations, with results pointing to utility in data-scarce, single-user settings. The discussion also describes assistive applications and limits involving real-world settings, participant demographics, and scenario coverage.
6.1 Achieved Performance
- EEGazeNet outperformed all baselines and ablation conditions in cross-user and within-user evaluations, with particular utility suggested for data-scarce, single-user settings.Within-user evaluation had one user's attention distribution per image, compared with 36 in cross-user evaluation.Source: §6.1 · p. 8
6.2 Applications
- Predicted attention could guide assistive systems to highlight or verbally direct users toward a search target, and EEG-based saliency refinement may extend beyond search.The dataset may also support scanpath prediction and target-fixation detection.Source: §6.2 · p. 8
6.3 Limitations and Future Work
- The study is bounded by realistic but non-real-world stimuli, a relatively young university-recruited demographic, and two visual-search scenarios.The authors identify movement artifacts in real-world EEG as a challenge requiring dedicated approaches.Source: §6.3 · p. 8
7 Conclusion
The conclusion presents EEGaze-140, a large realistic visual-search dataset with EEG and eye-tracking recordings, and EEGazeNet, an EEG-informed attention-prediction approach. EEGazeNet consistently outperformed baselines and ablation conditions in evaluations on the dataset.
7 Conclusion
- EEGazeNet consistently outperformed all baselines and ablation conditions in comprehensive evaluations on EEGaze-140.The authors describe the dataset as a resource for future research.Source: §7 · pp. 8–9
- EEGaze-140 is presented as the largest dataset of EEG and eye-tracking recordings during realistic visual search.The dataset is publicly available.Source: §7 · pp. 8–9
- EEGazeNet is presented as the first visual-search attention-prediction approach informed by a user's cognitive state through EEG measured before search.Source: §7 · p. 8