Source-linked AI summary
Human-Like Attention? A Psychophysical Comparison of Visual Search in Humans and MLLMs
Renchi Zhang, Joost C. F. de Winter, Dimitra Dodou, Harleigh C. Seyffert, Yke Bauke Eisma
TL;DR
The paper asks whether MLLMs show human-like visual-search difficulty patterns, a question relevant to understanding their relation to biological vision. It compares humans and baseline MLLMs on identical synthetic 2D/3D searches and finds shared feature/conjunction patterns alongside divergence in complex target-absent decisions.
Problem
It remains unclear whether MLLM–human performance similarities reflect shared cognitive mechanisms or objective visual-signal complexity, complicating understanding of their relation to biological vision.
Source: Introduction · p. 2Method
The study compared humans and unaugmented MLLMs on identical synthetic 2D and 3D visual-search stimuli across feature and conjunction conditions.
Source: Introduction · pp. 2–3Results
Humans and MLLMs performed well in feature searches and degraded with increasing set size in conjunction searches, but MLLMs showed high absent-trial errors in some difficult conjunction conditions.
Source: Discussion · pp. 9–10Takeaways & Limitations
MLLMs show human-like sensitivities to target–distractor similarity, heterogeneity, and feature conjunctions, yet complex target-absent decisions reveal a potential weakness in combined binding and high distractor load.
Source: Discussion · p. 11The stimuli were synthetic, so the evaluation did not test natural cluttered scenes with scene grammar and object co-occurrence.
Source: Discussion · p. 10
Abstract
from arXiv · showhide
Visual search is a fundamental cognitive ability. This study investigates whether Multimodal Large Language Models (MLLMs) exhibit human-like difficulty signatures in visual search tasks. We compared search performance of humans (n = 1,250) and MLLMs using identical 2D and 3D stimuli across different set sizes. Both groups showed efficient performance in feature searches, most clearly when the target had a unique color, but performance degradation in conjunction searches as set sizes increased. Additionally, we found strong correlations between human and MLLM error rates ($\rho = 0.82$), which suggests that MLLMs are sensitive to similar objective complexities, such as stimulus heterogeneity. However, differences were found as well: whereas humans invested extra search time to respond accurately on target-absent trials, MLLMs exhibited extreme present/absent response biases in complex searches. We conclude that MLLMs replicate high-level human performance signatures, yet their underlying computations differ significantly.
Introduction
The introduction asks whether MLLM–human visual-search similarities reflect shared cognitive mechanisms or common stimulus complexity. It motivates testing baseline models against human search theory across feature, conjunction, and 2D/3D conditions.
Introduction
- Feature searches often show shallow set-size effects, whereas conjunction searches require binding and typically incur steeper costs, especially on target-absent trials.The passage notes that classic accounts associate absent-trial costs with self-terminating serial scanning, while alternative capacity-limited models exist.Source: Introduction · p. 1
- Search guidance depends on target–distractor similarity and heterogeneity; 3D depth cues can aid segregation but occlusion can hinder performance.Color and orientation offer stronger guidance than material, and conjunctions within one dimension are described as less efficient than cross-dimensional conjunctions.Source: Introduction · p. 1
- The study asks whether similarities between MLLM and human search reflect shared mechanisms or objective stimulus complexity, a distinction relevant to aligning models with human cognition.The introduction also identifies robotics and assistive technology as possible real-world contexts for that alignment.Source: Introduction · p. 2
- Humans use peripheral vision and gaze repositioning, whereas MLLMs commonly process fixed-resolution image patches feed-forward, so transformer interactions need not act like goal-directed attention.The introduction connects these architectural differences with MLLM difficulty integrating multiple objects and attributes under occlusion, high item counts, or relational binding demands.Source: Introduction · p. 2
- Using identical 2D and 3D stimuli, the study compares a large human sample with MLLMs across set sizes, distractor heterogeneity, target presence, and feature attributes.The stated contributions include separate present/absent analyses, human–model performance correlations, and a benchmark spanning color, shape, and material.Source: Introduction · pp. 2–3
- The experiment tests baseline MLLMs without iterative agents, visual scaffolds, or chain-of-thought augmentation to probe unaugmented search capabilities.The authors note that such augmentation could confound analysis of inherent architectural limitations.Source: Introduction · p. 2
Methods
The methods define nine visual-search conditions and test humans and MLLMs on corresponding 2D and 3D stimuli. They describe model prompting, human trials, and analyses of search performance and human–model relationships.
Experimental Conditions
- Nine conditions distinguish singleton feature searches from conjunction searches that require binding target features.The conditions span seven set sizes, balanced target presence, and 2D/3D stimuli; Table 1 gives example compositions at set size 128.Source: Experimental Conditions · p. 3, Table 1 · p. 3
MLLM Evaluation
- Three instruction-tuned MLLMs answered target-presence prompts under temperature-zero decoding, with ten punctuation variants aggregated for performance estimates.The models were gemini-2.0-flash-lite, gemini-2.5-flash-lite with thinking budget 0, and gemma-3-27b-it; 54 of 188,998 calls failed and were excluded.Source: MLLM Evaluation · pp. 3, 5
Human Evaluation
- A sample of 1,250 participants completed either the 2D or 3D branch, practicing before nine randomized condition blocks of present/absent trials.Participants responded by keypress and received immediate correctness feedback.Source: Human Evaluation · p. 5
Analysis
- Human RT and error rates were analyzed by set size and target presence, MLLM errors were averaged across presence and models, and Spearman correlations compared human and model performance.The shared human-data inclusion mask excluded RTs below 150 ms or above 20,000 ms; correlations used 18 dimensionality-by-condition cells pooled across set sizes.Source: Analysis · p. 5
Results
Humans and MLLMs shared difficulty patterns across feature and conjunction searches, and their error rates covaried strongly. They diverged in target-presence decisions, with MLLMs showing extreme response biases in complex conditions.
Human Performance
- C1, C2, and C5 had near-zero RT slopes, while other searches showed set-size costs; difficulty peaked at triple conjunction C9.Shape-based searches had higher error rates in 3D than 2D, possibly because depth cues reduced silhouette discriminability.Source: Human Performance · p. 5, Figure 2 · p. 6, Figure 3 · p. 6
- Humans were slower but more accurate on target-absent trials in all but the easiest searches; 3D conditions generally elicited longer RTs and more errors.Target-absent trials were consistently more accurate, while 3D conditions carried added costs associated with depth and occlusion.Source: Human Performance · p. 7, Figure 4 · p. 7
MLLM Performance
- MLLM error rates were below 5% in unique-color searches, but reached 40–50% at large set sizes in heterogeneous conjunctions C8 and C9.Unlike humans, MLLMs found C8 harder than C7; adding a third distractor type increased errors.Source: MLLM Performance · p. 7, Figure 5 · p. 8
- MLLMs showed condition-specific target-presence biases: they missed present targets in 3D C9, defaulted to “present” in C8, and often defaulted to “absent” in C4.In material search C4 at set size 256, they missed about half of present targets while rarely false-alarming on absent trials.Source: MLLM Performance · p. 7, Figure 6 · p. 8
MLLMs vs. Humans
- ρ = 0.91 for human RT vs. MLLM error and ρ = 0.82 for human error vs. MLLM error across 18 pooled cells.These positive correlations indicate that objective difficulty factors constrain both groups similarly.Source: MLLMs vs. Humans · p. 7, Figure 7 · p. 9
- Color search was most efficient for both groups, material search least efficient, and target-location effects differed between humans and MLLMs.Humans showed a center advantage and left-side disadvantage, whereas MLLM error rates were largely invariant to target position.Source: MLLMs vs. Humans · p. 9
Discussion
Humans and MLLMs share broad visual-search difficulty signatures, but diverge in complex target decisions and the strategies available to them. The comparison is limited to synthetic stimuli and baseline models.
Discussion
- Humans and MLLMs performed well in feature searches but degraded in complex conjunction searches, with MLLM accuracy falling most in heterogeneous conditions.Human feature searches defined by unique color showed nearly flat reaction-time slopes; MLLM accuracy also decreased as set size increased in conjunction searches.Source: Discussion · p. 9
- At the largest set size, correlations were ρ = 0.88 for human RT vs. MLLM error and ρ = 0.87 for human error vs. MLLM error.The authors interpret the correlations as evidence that objective complexity factors constrain both groups similarly.Source: Discussion · p. 9
- MLLMs found 3D scenes more challenging, but near-perfect target-absent performance in C9-3D accompanied many target-present errors, up to 79%.The authors suggest that the C9-3D failure indicates fragile representations of occlusion, perspective, and shadows.Source: Discussion · p. 10
- Humans responded more slowly but more accurately on target-absent trials, whereas MLLM errors showed extreme present/absent biases in complex conjunction searches.In C7 and C8, MLLMs made more errors on absent trials; in C9-3D, they made many errors when the target was present.Source: Discussion · p. 10
- C8 was easier for humans than C7 but hardest for MLLMs, consistent with humans filtering blue items before serially checking remaining shapes.The authors describe this as a two-template process that reduces effective set size, which MLLMs do not exploit in their single feed-forward pass.Source: Discussion · p. 10
- The study used synthetic stimuli and baseline MLLMs; it did not test scene grammar, object co-occurrence, augmentation methods, or models from other developers.Human data were collected online, and MLLM cognitive effort was assessed with error rate rather than reaction time.Source: Discussion · pp. 10–11
- The authors propose comparing human fixation patterns with MLLM attention maps and investigating why models fail in complex target-absent conditions.They also identify scene context, search history, and reward as additional modulators for future comparative tests.Source: Discussion · p. 11
Appendix A. Construction of Images
The image generator produced configurable 2D and 3D visual-search stimuli, varying object features, display layout, and depth cues across configurations.
Appendix A. Construction of Images
- Stimuli combined shape, color, and material features: circles or squares, yellow or blue, and solid or textured objects.The 2D shapes used specified pixel dimensions, while 3D spheres and cubes were calibrated to match projected area at the center.Source: Appendix A · p. 11
- Images were rendered at 1920 × 1080, with objects placed sequentially in a central display area and spaced to prevent overlap or collision.The target was placed first; 2D objects were upright and unrotated.Source: Appendix A · p. 11
- 3D scenes used a camera angle of 33.7°, three-point lighting, and permitted partial occlusion to provide depth cues.Object sizes varied with distance to account for perspective.Source: Appendix A · p. 11
- The generator created 25 versions per configuration, balancing distractor types and recording target locations for analysis.A configuration crossed condition, target presence, set size, and dimensionality.Source: Appendix A · p. 11
Appendix B. Task Instruction
The human task used practice and test blocks, fixation, and speed-accuracy instructions for binary target-present judgments. The target was a yellow square in 2D and a yellow cube in 3D.
Appendix B. Task Instruction
- Participants completed a nine-image practice block followed by nine blocks of 14 images, with an optional short break between blocks.The instructions provided feedback after responses.Source: Appendix B · pp. 11–12
- Each human trial began with central fixation, then asked participants to decide whether the solid yellow square or cube was present and respond quickly and accurately.The 2D instruction specified a square; the 3D instruction specified a cube.Source: Appendix B · p. 12
- In the instructions, participants were told to press the A key if the object was Absent, and to press the P key if the object was Present.Source: Appendix B · p. 12
Appendix C. Prompt for MLLM Evaluation
MLLM evaluation clarified that the target was an untextured solid yellow square or cube. Each image was queried repeatedly using deterministic prompt variants.
Appendix C. Prompt for MLLM Evaluation
- To address pilot confusion between solid and textured targets, the model prompt explicitly described the target as having no texture on its face.The wording was intended to mirror the human instructions while clarifying texture.Source: Appendix C · p. 12
- Each model received 10 queries per image at temperature 0, with one to ten appended dots creating ten deterministic prompt variants.The prompt adapted the target noun to the stimulus dimension: square or cube.Source: Appendix C · p. 12
- The model was instructed to output 0 for absent and 1 for present.Source: Appendix C · p. 12
Appendix D. Spatial Distribution of Target Locations
Human search difficulty varied modestly with target location, whereas MLLM error rates stayed near the condition average across the display. The analysis standardized per-image performance within condition, set size, and dimensionality groups.
Appendix D. Spatial Distribution of Target Locations
- Per-image metrics were z-scored within sets of 25 images for each condition, set size, and dimensionality; zero-variance sets received z = 0.This applied to 71 of 126 MLLM sets, where models made no errors on any image, and to no human sets.Source: Appendix D · p. 12
- Human errors and response times showed a modest center advantage and left-side bias; right-edge targets were slower and, in 3D, more error-prone.The left-right asymmetry was strongest for 2D error rates and absent for 3D response times.Source: Appendix D · p. 12, Figure 8 · p. 13
- MLLM error rates remained close to the condition average across target locations, unlike the spatial biases observed in humans.Fig. 8 displays human error rate, human response time, and MLLM error rate by target location in 2D and 3D.Source: Appendix D · p. 12, Figure 8 · p. 13
Declarations
The study reports ethics approval and informed consent, declares no competing interests, and provides its open-access reuse terms.
Declarations
- The study received approval from the TU Delft Human Research Ethics Committee, reference 4182.Informed consent was obtained from respondents through an item at the beginning of the questionnaire.Source: Declarations · p. 14
- The authors declare no competing interests.Source: Declarations · p. 14
- The article is licensed under Creative Commons Attribution 4.0 International, subject to attribution and the stated third-party material exceptions.Source: Declarations · p. 14