Source-linked AI summary
An Exam for Active Observers
Jiarui Zhang, Muzi Tao, Shangshang Wang, Ollie Liu, Xuezhe Ma, Willie Neiswanger
TL;DR
Current vision-language benchmarks do not measure whether MLLMs actively revisit visual evidence during reasoning. ActiveVision introduces 17 tasks designed to require iterative perception and finds that frontier models solve at most 10.6% of items, versus 96.1% for humans.
Problem
Whether MLLMs exercise active visual observation remains unanswered because existing vision-language benchmarks do not measure iterative perception.
Method
ActiveVision measures active observation with 17 tasks across distributed scanning, sequential traversal, and visual attribute transfer.
Results
10.6% is the best frontier-MLLM accuracy, while three humans average 96.1% across the benchmark.
Takeaways & Limitations
The results indicate that current MLLMs lack robust active visual observation, a distinct capability not closed by additional reasoning or tooling.
Takeaways & Limitations
The benchmark uses synthetic images with photorealistic re-rendering rather than samples from the natural-image distribution, limiting external validity.
Abstract
from arXiv · showhide
Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot. Decades of psychophysics and cognitive science have argued that this active observation is essential for a wide range of tasks. Whether today's multimodal large language models (MLLMs) exercise active observation is an empirical question that current vision-language benchmarks do not answer. We introduce ActiveVision, a benchmark that makes active observation measurable for MLLMs, comprising 17 tasks across 3 categories. Tasks are designed to force repeated visual perception rather than a single static description. Frontier MLLMs collapse on ActiveVision: the highest-scoring model we evaluate, GPT-5.5 at the highest exposed reasoning-effort tier, solves only 10.6% of items and scores zero on 11 of the 17 tasks, and even Claude Fable 5, despite topping most reasoning and coding leaderboards, solves just 3.5%, far behind three human participants who average 96.1%. Furthermore, much of the gap persists even when models write and run their own vision code: such code is unreliable on realistic imagery, and catching its failures itself requires the active perception the models lack. Together, these results indicate that current MLLMs lack robust active visual observation, motivating architectures and training objectives that close the perception-reasoning loop.
1. Introduction
ActiveVision measures whether MLLMs repeatedly inspect visual evidence while reasoning, a capability existing saturated benchmarks largely leave unmeasured. Across 17 tasks, frontier models perform far below humans, and neither additional reasoning nor autonomous vision-code use closes the gap.
- Motivation: Active observation requires iterative, hypothesis-driven inspection—fixating, predicting, and returning—and is essential to perception across many tasks.The benchmark’s motivating examples require counting, tracing, or matching elements that cannot be solved at a glance.
- Motivation: Existing multimodal benchmarks are nearing saturation, yet applications require models to revisit visual evidence as they reason, leaving active observation largely unmeasured.The introduction identifies robotics, design, manufacturing, computer use, spatial understanding, and scientific discovery as relevant application areas.
- Benchmark: ActiveVision makes iterative visual perception measurable through 17 tasks spanning Distributed Scanning, Sequential Traversal, and Visual Attribute Transfer.Task configurations resist concise language descriptions, requiring models to keep returning to the image rather than answer from a single summary.
- Results: 10.6% is the best model result, achieved by GPT-5.5 xhigh across 85 released items; it scores zero on 11 of 17 tasks, while humans average 96.1%.Claude Fable 5 solves 3.5%, despite topping most reasoning and coding leaderboards.
- Results: 24.7–50.6% is the accuracy range reached by autonomous coding agents, narrowing but not closing the human–model gap because realistic-image tool failures often go undetected.Tool-use gains concentrate on tasks admitting reliable code-based solutions, while traversal remains difficult and verification becomes the bottleneck.
2. Related Work
Related work frames perception as an active, task-directed process, documents persistent MLLM failures on visual perception, and develops benchmarks targeting visual understanding, human–model gaps, or practical applications. ActiveVision distinguishes itself by isolating iterative active observation rather than bundling perception with language, coding, or reasoning.
- Active perception: Task-directed gaze changes visual sampling, while active sensor control can make otherwise ill-posed inverse problems well-posed.Different task prompts elicit different scanpaths for the same painting; active control is linked to shape from shading, structure from motion, and optical flow.
- Active perception: Without iterative sensor redirection, a vision system will fail, according to sensorimotor and computational accounts of gaze-targeted perception.These accounts emphasize local computation targeted by gaze and seeing as mastery of how visual input changes under one’s own movements, with support from everyday-task eye tracking.
- Perception benchmarks: Visual-perception benchmarks such as MMVP, MMStar, and CV-Bench exclude text-solvable items and reward reading the image, but are now largely saturated.This line of work also diagnoses frontier models’ reliance on memorized priors and neglect of queried visual details.
- Human–model gaps: BLINK, BlindTest, ArtQA, BabyVision, ChildBench, and KidGym expose human–model gaps, but do not isolate which capability causes the failures.Their tasks are framed by what people can answer quickly or solve as sighted humans, infants, or children.
- Application benchmarks: Application benchmarks such as OSWorld and BenchCAD track deployment value but bundle coding, reasoning, and visual perception, preventing attribution of failures to perception.ActiveVision instead isolates active observation: the iterative looking loop that prior benchmarks do not explicitly name, using photorealistic inputs resistant to single-language-description shortcuts.
3. ActiveVision
ActiveVision measures active visual observation with 17 reproducible task generators organized around exhaustive enumeration, curve tracing, and fine-grained comparison. Its instances contain discriminative visual information that exceeds a single language summary, requiring repeated gaze shifts and keeping the image in the loop.
- Task families: ActiveVision instantiates three task families from exhaustive enumeration, curve tracing, and fine-grained comparison under working-memory limits.These operations are serial, attention-demanding routines carried out through repeated shifts of gaze.
- Task families: The benchmark’s three families target distributed scanning, sequential traversal, and visual attribute transfer across spatially complex or visually similar regions.Characteristic failures include partial coverage, gestalt interpolation, and substituting a prior instead of measuring both regions.
- Design principle: A single language description cannot losslessly preserve each instance’s discriminative visual state, so solving the tasks requires keeping the image itself in the loop.Twenty scattered dots, for example, carry 190 pairwise spatial relations and twenty real-valued coordinate pairs.
- Task construction: 17 task generators produce reproducible instances from deterministic seeds, with flat per-task answer distributions that prevent modal-answer and task-identity shortcuts.Each generator creates a synthetic scaffold, question, and ground-truth answer.
- Realistic rendering: Procedural scaffolds are re-rendered into noisy, realistic settings with GPT-image-2, and only the rendered image is shown to the model.The pipeline is intended to remove the cartoon-input confound and test MLLMs on realistic imagery.
PHASE A · PROCEDURAL GENERATION PHASE B · PHOTOREALISTIC RE-RENDERING
ActiveVision uses a four-stage pipeline that generates geometric scaffolds with full ground truth and re-renders them photorealistically without changing the benchmark-relevant structure. This makes the imagery resemble downstream applications rather than toy renderings.
- PHASE A · PROCEDURAL GENERATION: A deterministic Python generator emits random smooth closed curves with full ground truth before task-specific photorealistic re-rendering.The pipeline illustrates Tangled Loop Counting.
- PHASE B · PHOTOREALISTIC RE-RENDERING: The photorealistic rendering preserves positions, counts, and topology, so the benchmark question and ground-truth answer apply unchanged.Only the photorealistic image is served to the model during evaluation.
- PHASE B · PHOTOREALISTIC RE-RENDERING: The resulting imagery resembles what downstream applications actually face, making the active-vision diagnosis externally meaningful rather than a toy-rendering artifact.
4. Benchmark Evaluation
Frontier MLLMs perform poorly on ActiveVision, and increasing reasoning effort yields only limited gains. Agentic tools substantially improve accuracy but remain far below humans because realistic visual errors are both common and difficult for models to detect.
- Headline accuracy: 10.6%: GPT-5.5 solves 9 of 85 items, while Claude Fable 5 solves 3.5%; GPT-5.5 scores zero on 11 of 17 tasks.The human mean is 96.1%, roughly nine times the best model’s accuracy.
- Headline accuracy: No item is solved by all six models, and the question-only GPT-5.5 control solves 2 of 85 items, or 2.4%.The control matches the image-and-question run at none effort, indicating that prompt priors alone do not explain performance.
- Reasoning effort: Nearly 100× higher GPT-5.5 reasoning cost raises accuracy only from 2.4% at none effort to 10.6% at xhigh, while failures persist across tiers.Other frontier models show the same narrow low-accuracy pattern, far below the human baseline.
- Task failure patterns: Counting increasingly undercounts crowded scenes, ordered-walk survival collapses within the first one or two moves, and no walk is completed exactly.Difference tasks show high miss rates, low false-alarm rates, and frequent default answers of “same.”
- Agentic tool use: $2.74–$7.63 per item and 12–15 minutes of agent computation still yield only about half the benchmark, while realistic textures cause tool errors that models often fail to audit.Examples include incorrect counts from cobblestone masks, distorted contours, truncated routes, and incorrect maze paths.
5. Discussion
The discussion finds that tool use does not resolve ActiveVision’s visual bottlenecks, while noting synthetic-image and benchmark-design limitations. It frames the benchmark’s quantified human–machine perception gap as motivation for future architectures and training objectives.
- Tool-use gains and remaining bottlenecks: Classical computer-vision tools fail on ActiveVision’s photorealistic images by fragmenting contours, merging objects, and breaking under style variation.These tools work best on clean inputs, whereas humans can revisit the image and recognize when masks belong to the same shape.
- Scope of the agentic evaluation: Agentic evaluation measures tool-orchestrated visual problem solving, including decomposition, noisy-image extraction, and recognizing when outputs need visual verification.This approach offloads iterative visual reasoning to code and reduces verification to simpler visual checks, making it a lower bar than vision-only solving.
- Limitations: ActiveVision uses synthetic, photorealistically re-rendered images from GPT-image-2 rather than samples of the natural-image distribution, trading external validity for exact ground truth and controlled task structure.The benchmark’s external validity therefore rests on the elemental operations measured, including scanning, tracing, and comparing.
- Limitations: As models improve at describing images, the benchmark’s property that no short language description carries the answer may erode, requiring future versions to tighten tasks.The task design currently aims to preserve the need for active visual operations rather than answerable descriptions.
- Outlook: ActiveVision quantifies a human–machine perception gap that current vision-language benchmarks do not surface, providing a target for future architectures and training objectives.The isolated operations are described as substantive to high-stakes visual work, including radiology, cell counting, and inventory inspection.
6. Conclusion
ActiveVision isolates active observation by requiring models to repeatedly return to realistic images, form hypotheses, and check them as reasoning unfolds. Humans solve nearly all items quickly, while frontier MLLMs fail most items, with limited benefit from increased reasoning effort and uneven gains from code and tools.
- 6. Conclusion: ActiveVision measures active observation as repeatedly returning to an image while forming and checking hypotheses during reasoning.Its tasks prevent any single language description from carrying the answer.
- 6. Conclusion: Humans solve nearly all ActiveVision items in about half a minute each, whereas frontier MLLMs fail most items.
- 6. Conclusion: Increasing reasoning effort does little to close the human–MLLM performance gap, while code and tools provide larger but uneven gains concentrated on task.