PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling human visual search: A combined Bayesian searcher and saliency map approach for eye movement guidance in natural scenes

M. Sclar, G. Bujia, S. Vita, G. Solovey, J. E. Kamienkowski

arXiv:2009.08373v2cs.AIcs.CVq-bio.NC

TL;DR

Saliency maps predict fixation locations but not their sequence, while Bayesian visual-search models had largely been tested with artificial stimuli. This paper combines Bayesian search with saliency-map priors for natural-scene search, where the combined approach performs as well as humans.

  • Problem

    Saliency maps do not model fixation sequences, and Bayesian visual-search models had mainly been tested with artificial stimuli, leaving their application to natural scenes unexplored.

    Source: §1 · p. 2

  • Method

    The paper combines an Ideal Bayesian Searcher with saliency-map priors and modifies the template-response update for natural-scene visual search.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    The saliency-prior IBS performed as well as humans in natural-scene visual search, and its ordered fixation scanpaths were compared quantitatively with human scanpaths.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work extends Bayesian visual-search modeling from constrained artificial stimuli to natural scenes.

    Source: §4 · p. 6

Abstract

from arXiv · show

Finding objects is essential for almost any daily-life visual task. Saliency models have been useful to predict fixation locations in natural images, but are static, i.e., they provide no information about the time-sequence of fixations. Nowadays, one of the biggest challenges in the field is to go beyond saliency maps to predict a sequence of fixations related to a visual task, such as searching for a given target. Bayesian observer models have been proposed for this task, as they represent visual search as an active sampling process. Nevertheless, they were mostly evaluated on artificial images, and how they adapt to natural images remains largely unexplored. Here, we propose a unified Bayesian model for visual search guided by saliency maps as prior information. We validated our model with a visual search experiment in natural scenes recording eye movements. We show that, although state-of-the-art saliency models perform well in predicting the first two fixations in a visual search task, their performance degrades to chance afterward. This suggests that saliency maps alone are good to model bottom-up first impressions, but are not enough to explain the scanpaths when top-down task information is critical. Thus, we propose to use them as priors of Bayesian searchers. This approach leads to a behavior very similar to humans for the whole scanpath, both in the percentage of target found as a function of the fixation rank and the scanpath similarity, reproducing the entire sequence of eye movements.

1 Introduction

The paper frames visual search as goal-directed sampling through eye movements and proposes combining Bayesian search with saliency priors to predict scanpaths in natural scenes.

1 Introduction

  • Visual search samples a scene through eye movements, with gaze strategies aimed at minimizing the steps needed to find a target.The paper characterizes this as active sensing, combining bottom-up information with task-dependent updating across fixations.Source: §1 · p. 1
  • Saliency maps predict likely fixation regions, but do not model fixation sequences; their accuracy degrades after the first few fixations in free exploration.A natural-scene visual-search model using a greedy, goal-related attention map was proposed by Zhang et al..Source: §1 · p. 2
  • Bayesian visual-search models use prior knowledge, visibility, and updated target-location beliefs, but had mainly been tested with artificial stimuli.The introduction identifies incorporating scene information as necessary to extend these models to natural images.Source: §1 · p. 2
  • The proposed IBS model combines state-of-the-art saliency maps as priors with an updated template-response rule, and compares model and human scanpaths.The update uses template-response correlation and may incorporate distractor effects.Source: §1 · p. 2

2 Visual search in natural indoor images: Human data

In the natural indoor-scene experiment, target discovery increased with the number of allowed saccades and plateaued between 8 and 12; participants searched from a shared starting location for each image.

2 Visual search in natural indoor images: Human data

  • Participants searched for an object in a crowded indoor scene, beginning from the same pseudo-random fixation position for each image, at least 300 px from the target.The search image appeared after fixation on the starting dot and disappeared at target fixation or after 2, 4, 8, or 12 saccades.Source: §2 · p. 2
  • The experiment used 134 indoor images without people or text, displayed at 1024 × 768 resolution, with a unique 72 × 72 px target region selected per image.The target region was selected to avoid evaluating memory retrieval accuracy.Source: §2 · p. 2
  • Target discovery plateaued between 8 and 12 allowed saccades among 57 participants.Recorded saccade amplitude and direction, and fixation spatial distributions, followed expected patterns across fixation rank.Source: §2 · p. 2

3 Searcher modeling: combining saliency maps with Bayesian searchers

Saliency maps predict early search fixations but lose predictive power over the sequence, while the Ideal Bayesian Searcher (IBS) had previously been tested on artificial stimuli. This section adapts IBS with saliency priors and distractor-sensitive template responses for natural scenes; cIBS models closely match human search performance and scanpaths.

3.1 Exploring saliency maps

  • Saliency-map AUC peaked at the second fixation and then declined smoothly; DG2 performed best across fixation ranks, while center bias reached 0.5 at fixation four.The human-based saliency map outperformed the other maps overall, and center bias was the worst.Source: §3.1 · p. 3

3.2 Modifying the Ideal Bayesian Searcher (IBS) to handle natural scenes

  • For natural scenes, IBS uses average saliency in each grid cell as its prior and restricts candidate fixations to grid centers.Consecutive fixations within a cell were collapsed to align with the model's behavior.Source: §3.2 · p. 3
  • cIBS models distractors using target-region cross-correlation and uses a shared 2D-Gaussian visibility map with parameters fixed across participants.Its variance remains visibility-dependent, with parameters a and b modulating inverse visibility and preventing 1/d′ from diverging.Source: §3.2 · pp. 3–4

3.2.1 Evaluating searcher models on human data

  • cIBS had the best agreement with human performance as a compromise across metrics; cIBS+DG2 was closest to human mean agreement, and cIBS and IBS scanpaths were almost indistinguishable from humans.The Greedy searcher was also very close in scanpath behavior; only the saliency-based searcher differed significantly from humans.Source: Figure 1 · p. 4, §3.2.1 · pp. 4–5
  • DG2 and Center priors produced scanpaths closer to humans than flat or noisy priors, while DG2 improved target finding relative to Center.The center prior was described as a fair approximation of human priors.Source: Figure 1 · p. 4, §3.2.1 · p. 5

4 Conclusions

The paper extends Bayesian visual search to natural images by incorporating saliency priors and modifying the searcher for natural-scene complexities. Its long-sighted objective also connects the approach to human-like search behavior and Bayesian accounts of cognition.

4 Conclusions

  • Unlike mainly greedy non-Bayesian scanpath approaches, a long-sighted objective can yield human-like behaviors such as inhibition of return without imposing it explicitly [30, 47].Zhang et al. forced inhibition of return, whereas the authors state that it is implicit in their model.Source: §4 · p. 5
  • Saliency models predict fixation patterns across pictures and movies, while search models have also addressed visual-information reliability in fixed-gaze tasks.These examples illustrate related applications of saliency and search modeling beyond the paper's natural-image visual-search setting.Source: §4 · p. 6
  • The model extends the Ideal Bayesian Searcher to natural-image visual search using saliency-map priors, a modified template-response mean, and a visibility shift.The authors describe this as the first Bayesian framework for predicting eye movements during visual search in natural images, to their knowledge.Source: §4 · p. 6
  • The work situates Bayesian models within a broader account of cognition that includes decision making, confidence, learning, and perception.The authors frame this as the brain building abstract models of its environment in novel, noisy, and cluttered scenarios.Source: §4 · p. 6

A Visual search in natural indoor images: Paradigm and human data acquisition and preprocessing

The appendix describes the participants, search-trial procedure, and indoor-image stimuli used for the natural-scene visual-search experiment.

A.1 Participants

  • Fifty-seven adults with normal or corrected-to-normal vision participated, were naïve to the experiment's objectives, and provided informed consent.Participants' mean age was 25.1 ± 5.9 years; 34 were male.Source: §A.1 · p. 9

A.2 Paradigm and procedure

  • Participants viewed a centered target before searching for it in a crowded indoor scene after fixating a pseudo-random starting dot.The target appeared for 3 seconds; trials ended when the target was fixated or the allowed saccade limit was reached, followed by an extra 200 ms for processing. Participants then reported the target location and uncertainty, which were not analyzed.Source: §A.2 · p. 9

A.3 Stimuli

  • Participants searched 134 black-and-white indoor images selected to contain several objects and no people or text.A single nonrepeated target region of 72 × 72 pixels or less was manually selected per image; images were piloted to favor searches requiring several fixations. Images were randomized across three blocks, with calibration before each block.Source: §A.2 · p. 9, §A.3 · p. 10

A.4 Data acquisition

Eye movements were recorded during the visual-search experiment using a monocular EyeLink 1000 setup.

A.4 Data acquisition

  • Eye movements were recorded monocularly at 1000 Hz with an EyeLink 1000 while participants sat 55 cm from a 19-inch monitor with head support.The monitor ran at 60 Hz and displayed at 1280 × 960 resolution.Source: §A.4 · p. 10

A.5 Data preprocessing

The preprocessing pipeline mapped fixations to a grid, consolidated repeated cell visits, and defined which trials counted as correct.

A.5 Data preprocessing

  • Fixations were mapped to a 32 × 24 grid of 32 × 32-pixel cells, with consecutive fixations in one cell collapsed and off-image fixations moved to the nearest cell.Blinks were excluded, and the grid size was also explored in terms of model performance.Source: §A.5 · p. 10
  • A trial counted as correct when the participant fixated within the 72 × 72-pixel target region; only correct trials were analyzed for eye movements.Source: §A.5 · p. 10

B Human behavior results

In the natural-scene search experiment, target detection increased with the number of allowed saccades and eye movements showed changing spatial patterns across fixation ranks. The section also compares saliency maps used to characterize search guidance.

B Human behavior results

  • Target detection increased with allowed saccades, reaching a plateau between 8 and 12 saccades.The experiment allowed 2, 4, 8, or 12 saccades.Source: B Human behavior results · p. 10
  • Saccade amplitude decreased with fixation rank, saccades tended to be horizontal, and fixations spread from a central bias across the image.Fixations first extended over the horizon; the paper notes task, setup, image composition, or processing benefits as possible contributors to the central bias.Source: B Human behavior results · pp. 10–11, Figure S2 · p. 10

C Exploring Saliency Maps

  • Five saliency maps were compared: DeepGaze 2, MLNet, SAM-VGG, SAM-ResNet, and ICF.ICF uses Gaussian filters for low-level intensity and intensity-contrast information; the other models use neural-network architectures.Source: C Exploring Saliency Maps · p. 11
  • A human-based saliency map was formed from observers’ fixation positions for each image, smoothed with a Gaussian kernel.The first fixations were excluded because observers began each trial from the same position.Source: C Exploring Saliency Maps · p. 11

C.1 Prediction of observers’ fixation positions

The section evaluates saliency maps across fixation ranks, then describes the Bayesian searcher and its correlation-based modification. Saliency-map prediction declines across the search, while the Bayesian model uses posterior target-location estimates to guide subsequent fixations.

C.1 Prediction of observers’ fixation positions

  • Human-based saliency performed best, center bias worst, and DeepGaze 2 performed slightly better than the other tested saliency models.DeepGaze 2 was indistinguishable from human performance at the second fixation.Source: §C.1 · pp. 11–12
  • All saliency maps’ AUC values declined smoothly across fixation ranks; center bias reached 0.5 at the fourth fixation.The text interprets the decline as a shift toward top-down mechanisms in eye-movement guidance.Source: §C.1 · p. 11, Figure S3 · p. 12

C.2 Comparing with other definitions of AUC

  • The compared ROC-AUC definitions differ in how they define positive and negative cases; Table S2 reports different AUC metrics for saliency maps on the third fixation.AUC-Judd uses non-fixated pixels as negatives, AUC-Borji uses uniformly sampled pixels, and shuffled-AUC uses fixation locations from other images.Source: §C.2 · p. 12, Table S2 · p. 12

C.3 Prediction of all fixation locations

  • When all fixation locations are considered, AUC is reduced for every model, including the human-based map.Figure S4 presents ROC curves and AUC values for all fixations.Source: §C.3 · p. 12, Figure S4 · p. 13

D.1 Description of the Ideal Bayesian searcher (IBS)

  • The Ideal Bayesian Searcher selects the next fixation to maximize the probability of correctly identifying the target location.Its posterior target-location probability incorporates a prior, a visibility map, and a template response.Source: §D.1 · pp. 12–13

D.1.1 Why does IBS implicitly achieves inhibition of return?

  • IBS implicitly discourages revisiting observed locations because a visible location where the target was absent receives negligible posterior probability.The same intuition applies to nearby locations with high visibility.Source: §D.1.1 · p. 13

D.2 Description of the correlation-based Ideal Bayesian searcher (cIBS)

  • The cIBS search considers fixation locations at grid-cell centers rather than every pixel in the image.The authors cite computational tractability and effectiveness as reasons for this restriction.Source: §D.2 · p. 13
  • The correlation-based IBS uses a template response that incorporates visual similarity between a display location and the target location.For a fairly visible location visually similar to the target, the expected response is higher than in the original definition.Source: §D.2 · p. 14

D.3 Metrics on the Comparison of Performances between Humans and Models

The section defines three complementary comparisons of model and human performance: a saccade-weighted distance, Jaccard Index, and Mean Agreement Score.

D.3.1 Distance weighted by the number of saccades allowed

  • WD(m) averages model–participant differences in target-found proportions at 2, 4, 8, and 12 saccades, weighting each difference by participant variability.Source: §D.3.1 · p. 14

D.3.2 Jaccard Index

  • Jaccard Index compares participant and model binary target-found vectors across images, with the model evaluated under that participant’s per-image saccade limits.Source: §D.3.2 · pp. 14–15

D.3.3 Mean Agreement

  • Mean Agreement Score measures the proportion of trials where participant and model performance agree, calculated as one minus the mean difference between their boolean vectors.Source: §D.3.3 · p. 15

D.3.4 Scanpaths Examples

Only a small fraction of images departed from human scanpaths, but some images elicited two consistent human search patterns that the deterministic model represented with one path.

D.3.4 Scanpaths Examples

  • Only 1% of images departed from human scanpaths under cIBS+DG2’s µ + 3σ criterion.Source: §D.3.4 · p. 15
  • Departures occurred when few participants found the target or viewers followed distinct patterns, such as checking a cupboard versus a table for a cup.Source: §D.3.4 · p. 15
  • Figure S5 places the first four fixations at left and whole scanpaths at right; trace colors encode dissimilarity to the model path.Source: Figure S5 · p. 15
  • For the cup example, the initial choice determined scanpath behavior, although almost all participants eventually explored both candidate regions.Source: §D.3.4 · pp. 15–16

E Data and Code Availability

The authors report that code, data, and parameters needed to reproduce the results and visualizations will be available online.

E Data and Code Availability

Loading 2009.08373v2…