PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent Goal-Directed Attention in Large Vision-Language Models

Han Zhang

arXiv:2609.05517v1cs.CVcs.AIcs.CL

TL;DR

The paper asks whether goal-directed attention can emerge in vision-language models without gaze training, a gap left by computational models often trained for free viewing. It compares two off-the-shelf models’ priorities with human fixations under search and free viewing, finding closer alignment for matching goals even when targets are absent.

  • Problem

    Many computational eye-movement models focus on free viewing and use human fixation training, leaving open whether goal-directed attention can emerge without gaze supervision.

    Source: Introduction · p. 1

  • Method

    The authors compare off-the-shelf VLM predictions with human fixations across search and free-viewing goals, and analyze Qwen’s reasoning traces against semantic anchors.

    Source: Results · pp. 3, 8, Discussion · p. 10

  • Results

    Both models aligned more closely with human fixations under matching goals than mismatched goals, including in target-absent scenes where visible-target localization cannot explain the pattern.

    Source: Results · p. 3, Discussion · p. 10

  • Takeaways & Limitations

    The results suggest that general-purpose VLMs can produce human-aligned goal-directed spatial priorities without direct gaze training, with potential relevance to predicting attention in specific contexts.

    Source: Discussion · pp. 10, 12

    Because the models do not fully disclose their training data, the authors cannot determine whether they encountered the study images during training.

    Source: Discussion · p. 11

Abstract

from arXiv · show

Human observers prioritize visual information according to task goals. Most computational models of naturalistic viewing are gaze-trained for free viewing, leaving open whether goal-directed attention can emerge in systems without gaze supervision. We tested two off-the-shelf vision-language models (VLMs), Qwen3-VL-32B-Thinking and Gemma-4-26B-A4B-it, on 4,887 naturalistic scenes under visual-search and free-viewing instructions. Model predictions were compared with human fixations on the same images under corresponding tasks. Both models aligned more closely with human fixations under matching goals than under mismatched goals. This crossover persisted in target-absent scenes, where alignment could not be explained by simple visual grounding, and appeared in decoder-layer readouts. Furthermore, model-thinking traces were grounded in target semantics during search and in visual prominence during free viewing. These findings show that general-purpose VLMs can generate human-aligned, goal-directed spatial priorities without gaze-specific training, informing theories of goal-directed attention and offering scalable tools for predicting where people look across tasks.

Introduction

Human attention flexibly changes with task goals, drawing partly on scene-semantic knowledge. Existing gaze-trained models largely address free viewing, leaving open whether goal-directed attention can emerge in systems without gaze training; general-purpose VLMs offer a way to test this question.

Introduction

  • Human observers flexibly change eye movements with task goals, drawing partly on scene-semantic knowledge such as likely object locations.For example, searching for a keyboard in an office can prioritize the desk.Source: Introduction · p. 1
  • Many computational eye-movement models target free viewing and are trained on human fixations, leaving open whether goal-directed attention can emerge without direct gaze training.Earlier models used low-level image features, while deep-learning models improved prediction but generally did not account for changing task goals.Source: Introduction · p. 1
  • General-purpose VLMs are not typically trained to predict fixations, but their multimodal training may provide scene-semantic knowledge relevant to goal-directed attention.This motivates testing whether their instruction-following can align with human attention across goals.Source: Introduction · p. 2
  • The study tests two off-the-shelf VLMs against human fixations on the same naturalistic images under search and free-viewing instructions.The authors predict closer model–human alignment when their task goals match, including when a search target is absent.Source: Introduction · p. 2

Results

Both models’ spatial priorities aligned more closely with human fixations when their goals matched, for target-present and target-absent scenes. Additional analyses found goal-related patterns in decoder-layer readouts, scene-level priority shifts, and Qwen’s reasoning, while fixation-order alignment was stronger in free viewing than in search.

Target-present images

  • The study compared zero-shot model predictions and human fixations on identical images under search and free-view instructions, testing whether goal-matched predictions aligned better.Ten human observers participated in each task condition.Source: Results · p. 2
  • Qwen and Gemma both showed a crossover in target-present scenes: search maps aligned better with human search, and free-view maps with human free-view.Qwen’s search-search correlation was .73 versus .15 for search-free; Gemma’s was .70 versus .16.Source: Results · pp. 3–4, Figure 2 · p. 4

Target-absent images

  • With targets absent, both models still showed the matching-goal crossover, including Qwen search correlations of .45 versus .19 and Gemma search correlations of .40 versus .22.The absent-target condition tests alignment beyond locating a visible target.Source: Results · pp. 5–6, Figure 4 · p. 6

Direct readout from decoder layers

  • Decoder-layer priority maps showed weaker human correspondence than final answers, but retained the goal-alignment crossover for both models and target-presence conditions..Source: Results · p. 7

Scene-level alignment

  • Scene-level fidelity was positive on over 99% of target-present images for Qwen and 99% for Gemma, and on 92% and 87% of target-absent images, respectively.Fidelity is the correlation between model and human maps of spatial priority changes between search and free viewing.Source: Results · pp. 7–8, Figure 6 · p. 8

Alignment in model thinking traces

  • Qwen’s search reasoning was more target-aligned, while its free-view reasoning was more aligned with visual prominence across target categories.The analysis compared thinking-trace embeddings with target-word and visual-prominence semantic anchors.Source: Results · pp. 8–9, Figure 7 · p. 9

Alignment in fixation order

  • Both models had stronger fixation-order alignment in free viewing than in search, where alignment was near zero; human observers’ center bias may partly explain the search result.The comparison measured scanpath similarity above a shuffled-order floor.Source: Results · pp. 9–10, Figure 8 · p. 10

Discussion

Across two models, predictions aligned more closely with human fixations under matching goals, including when search targets were absent. The discussion connects this pattern to scene-semantic guidance and considers both its implications and limits.

Discussion

  • Goal-matched predictions aligned more closely with human fixations, and the crossover persisted when targets were absent, where visible-target grounding could not explain it.Qwen’s traces suggest target-relevant scene semantics during search and visual prominence during free viewing, though traces need not faithfully represent internal computation.Source: Discussion · pp. 10–11
  • Zero-shot VLM predictions were competitive with fixation-trained models, although the authors call for a more systematic comparison.On the same dataset, reported correlations for VLMs, DeepGaze II, and a supervised model varied by task and target presence.Source: Discussion · p. 11
  • The findings suggest broad multimodal training may give VLMs scene-semantic knowledge that supports human-aligned priorities and human-readable hypotheses about semantic guidance.Qwen’s traces referred to task-relevant objects and likely locations across thousands of scenes.Source: Discussion · p. 11
  • The authors argue that goal-sensitive behavior in a shared system may inform the possibility that human cognitive control arises from general learning and context-dependent retrieval, without a dedicated control unit.This is presented as a computational possibility, not evidence of mechanistic equivalence.Source: Discussion · p. 11
  • Undisclosed training data prevent determining whether either model encountered the study images, so training-data exposure effects cannot be ruled out.The authors consider simple image memorization unlikely to explain why different instructions produce selectively goal-aligned maps, but do not exclude all exposure effects.Source: Discussion · p. 11
  • Behavioral alignment does not establish mechanistic equivalence: VLMs lack an oculomotor system, and their scene-semantic predictions may diverge from human fixation order and center bias.Humans showed center bias across goals, while VLM predictions showed it only in free viewing; the authors suggest holistic inference may help explain failed fixation-order prediction.Source: Discussion · pp. 11–12

Methods

The study compared two off-the-shelf VLMs’ zero-shot spatial predictions with human fixation data under search and free-viewing goals. It constructed priority maps from emitted points and decoder attention, then evaluated spatial alignment, semantic traces, and fixation order.

Methods

  • Human fixation analyses used the training and validation images from COCO-Search18 and COCO-FreeView: 4,887 images (2,474 target-present, 2,413 target-absent), with 166,868 search and 612,346 free-view fixations.The study tested Qwen3-VL-32B-Thinking and Gemma-4-26B-A4B-it.Source: Methods · pp. 12, 14
  • For each image and prompt, the models generated 20 answers; the analysis retained the top five emitted points to limit exhaustive free-view output and match human scanpath length.Points were aggregated into a frequency matrix, smoothed with a Gaussian filter, and rescaled to form a priority map.Source: Methods · p. 13
  • Decoder-attention priority maps contrasted target-word attention with prompt-baseline attention for search and “remember” attention with the remaining prompt for free viewing.Attention was extracted across predefined layers and heads, combined with greater weight for heads focused on fewer image locations, then post-processed into maps.Source: Methods · pp. 13–14
  • Human priority maps pooled correct search-trial fixations and all free-view fixations, excluding each trial’s initial center fixation.Target-absent search images were selected so the target matched the scene context; fixation durations were not used in map construction.Source: Methods · p. 14
  • Map alignment was measured with Pearson correlation, and repeated-measures analyses tested model goal and human goal across images.Planned comparisons used paired t-tests; scene-level reorganization used one-sample t-tests, with 95% confidence intervals reported for t-tests.Source: Methods · pp. 14–16
  • For target-absent scenes, Qwen’s cleaned thinking-trace sentences were compared with target-word and visual-prominence semantic anchors using embedding cosine similarity.Target words and abstract or very short sentences were removed before embedding the remaining sentences.Source: Methods · pp. 14–15
  • Fixation-order similarity was assessed on target-absent scenes by comparing top-five model scanpaths with human sequences and subtracting a shuffled-model baseline.Human sequences omitted the initial center fixation and were truncated to five; the scanpath metric was converted to a 0–1 similarity score.Source: Methods · p. 15
  • Pilot images were used to select attention layers and check prompts and analysis procedures before parameters were frozen; model and participant data were publicly available or open-weight.The human-data collection procedures were reported as approved by Stony Brook University’s institutional review board.Source: Methods · pp. 12, 15

Supplementary Materials

Decoder-layer priority maps retained goal-alignment crossovers for both models on target-present and target-absent scenes. Supplementary analyses also describe category-level semantic dissociation, an illustrative trace, and differences in center bias.

Supplementary Materials

  • Decoder-layer priority maps were constructed from attention weights as a direct readout for testing whether goal alignment appears in model computations.The supplementary results compare these maps across model and human task goals.Source: Supplementary Materials · pp. 18–19
  • For target-present scenes, decoder-layer maps showed goal-matched alignment for both Qwen and Gemma, with search maps matching human search and free-view maps matching human free viewing.Figures S1–S2 show example maps and goal-alignment results for the two models.Source: Supplementary Materials · p. 18, Figure S1 · p. 18, Figure S2 · p. 19
  • For target-absent scenes, decoder-layer maps also showed the goal-alignment crossover for both models, although search-to-search alignment was comparatively modest.Figures S3–S4 show an example and the corresponding goal-alignment results.Source: Supplementary Materials · pp. 19–20, Figure S3 · p. 20, Figure S4 · p. 21
  • Category-level semantic dissociation is reported as search-minus-free-view thinking Cohen’s d for each semantic anchor; positive values indicate greater target alignment in search thinking.The figure also reports mean fidelity by target category, with 95% confidence intervals; positive fidelity indicates alignment in which regions are prioritized or deprioritized as goals change.Source: Figure S6 · p. 22, Figure S5 · p. 22, Supplementary Materials · p. 23
  • The example Qwen trace for an absent-bottle scene reasons about plausible places to check after noting that no bottle is visible.It ranks the area behind stacked plates, the black pole, and the grassy background as candidate locations.Source: Supplementary Materials · p. 23
  • Humans showed center bias under both goals, with search fixations somewhat more eccentric; VLM predictions showed center bias only during free viewing.Figure S7 plots mean eccentricity by fixation index, where lower values indicate positions nearer the image center.Source: Supplementary Materials · pp. 23–24, Figure S7 · p. 24
Loading 2609.05517v1…