Source-linked AI summary
PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
Zichao Lin, Yifeng Xie, Bowen Qu, Haiming Wang, Jia Li, Haoning Wu, Yuhao Dong, Zuhao Yang, Jinguo Zhu, Haoyu Lu, Zijia Zhao, Tongtian Yue, Zhangyang Qi, Junwei Yang, Mengfan Dong, Peizhou Cao, Chenzhuang Du, Zaida Zhou, Haotian Yao, Hao Yang, Hongcheng Gao, Lin Sui, Weihong Li, Xinxing Zu, Jia Chen, Yao Wang, Xiaoxue Wu, Yalin Wang, Y. Charles, Yiping Bao, Yangyang Liu, Zhiqi Huang, Xinyu Zhou
TL;DR
Existing evaluations do not cleanly isolate visual perception from reasoning and knowledge. PerceptionBench derives ten atomic capabilities from model failures, builds 3,000 verified questions, and finds that no evaluated model reaches 60% overall accuracy.
Problem
Existing benchmarks rarely isolate visual perception from reasoning and external knowledge while systematically covering the perceptual capabilities models lack.
Method
PerceptionBench derives ten atomic capabilities from failures across 42 benchmarks and constructs 3,000 verified questions isolating perception from reasoning and external knowledge.
Results
No evaluated model reaches 60% overall accuracy, with perception-related hallucination the weakest capability on average and similar aggregate scores masking sharply divergent profiles.
Takeaways & Limitations
PerceptionBench provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs.
Takeaways & Limitations
The taxonomy reflects current model weaknesses and must be re-induced as models improve; failure attribution may also propagate labeling errors.
Abstract
from arXiv · showhide
We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with failures in reasoning or domain knowledge, while application-driven benchmarks only cover narrow, fragmented domains shaped by heuristic designs. To address these limitations, PerceptionBench adopts a bottom-up approach: by diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities. Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge. Benchmark results across sixteen frontier MLLMs reveal that atomic perception remains largely unsolved---no model reaches 60\% accuracy, perception-related hallucination is the weakest capability on average, and similar overall scores conceal sharply divergent capability profiles. PerceptionBench thus provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs.
1 Introduction
PerceptionBench isolates atomic visual perception as a prerequisite for downstream MLLM reasoning and interaction, addressing evaluations that conflate perception with reasoning and external knowledge. It defines ten capabilities from frontier-model failures and evaluates them with 3,000 verified questions, revealing that atomic perception remains largely unsolved.
- No evaluated model surpasses 60% overall accuracy on PerceptionBench, while perception-related hallucination is the weakest capability on average.GPT-5.6-Sol scores 59.7% and Kimi K3 scores 58.5%, the highest reported overall accuracies.
- Visual perception comprises distinct atomic capabilities, including attribute recognition, counting, localization, and text reading, that underpin downstream MLLM reasoning and environmental interaction.
- Existing benchmarks often assess multi-step final answers, making performance dependent on reasoning and external knowledge rather than isolating visual perception.MMMU and MathVista are cited as examples of evaluations where perception errors cannot be cleanly attributed.
- PerceptionBench contains 3,000 verified questions organized around ten atomic perceptual capabilities derived from tracing frontier MLLM failures across 42 existing benchmarks.The taxonomy follows each failure to its earliest erroneous step and clusters the resulting error labels.
- Nearly identical aggregate scores can conceal sharply divergent capability profiles, limiting overall accuracy as a measure of which perceptual abilities models have acquired.The benchmark is designed to diagnose capability-level differences rather than only report a single aggregate score.
2 PerceptionBench
PerceptionBench derives ten atomic perceptual capabilities from empirically attributed failures across 42 visual benchmarks, then constructs and verifies questions that isolate perception from reasoning, knowledge, and ambiguity. Its released benchmark contains 3,000 capability-balanced, difficulty-stratified samples designed to remain diagnostically challenging for frontier MLLMs.
- Taxonomy discovery: Failures across 42 aggregated benchmarks concentrate on a few error types with weak overlap, motivating a benchmark focused on atomic perceptual capabilities.Existing benchmarks therefore expose sparse, application-biased slices of the capability space.
- Taxonomy discovery: A stronger frontier model attributes each candidate failure to the earliest erroneous reasoning stage using the complete question, image, reference answer, and reasoning trajectory.The procedure produces free-form error analyses and open-vocabulary error labels.
- Taxonomy discovery: The perception branch defines ten capabilities: localization, attribute recognition, counting, relation understanding, depth and 3D perception, OCR, comparison, fine-grained recognition, context integration, and perception-related hallucination.Only the perception error class is retained; other classes represent non-perceptual limitations.
- Sample construction: Difficulty screening discards samples solved correctly by every model in a four-model ensemble, stratifies survivors by difficulty, and uses disjoint ensemble members from the sixteen models evaluated later.This avoids trivial items and prevents tailoring the benchmark to an individual model’s failures.
- Sample construction: The released PerceptionBench contains 3,000 samples subsampled from an in-house pool exceeding 17,000 verified samples, with capability-level balancing and difficulty stratification.The public release is drawn from the constructed portion of the full evaluation pool.
- Verification: Multi-stage verification removes or revises samples with unrelated reasoning requirements, ambiguous visual evidence, incorrect annotations, or answers dependent on external knowledge.Capability alignment checks whether visual information acquisition, rather than reasoning complexity, external knowledge, or instruction misunderstanding, is the primary requirement; visual grounding requires answers to derive from the image.
3 Evaluation Results
PerceptionBench shows that atomic visual perception remains a major bottleneck for current MLLMs, with no model reaching 60% overall accuracy. Performance is uneven across capabilities, while released-subset scores closely track the full benchmark and remain stable across repeated runs.
- Overall performance: 59.7% is the highest overall accuracy, achieved by GPT-5.6-Sol, while Kimi K3 reaches 58.5% as the top-performing open-source model.Kimi K3 surpasses the remaining proprietary systems identified in the evaluation, including Gemini-3.1-Pro at 56.2% and GPT-5.5 at 55.8%.
- Capability profiles: 36.7% is the lowest average accuracy among capabilities, recorded for perception-related hallucination, versus 53.2% for visual relation, 52.4% for OCR, and 52.0% for localization.GPT-5.6-Sol reaches 76.7% on localization but only 26.9% on hallucination, illustrating uneven perceptual competence.
- Capability profiles: 50.6% is Gemini-3.5-Flash’s hallucination score, the best among the compared models, despite its 52.0% overall score, while GPT-5.6-Sol records 26.9%.Seed-2.1-Pro follows Gemini-3.5-Flash with 49.8%, showing that capability and reliability can diverge from overall ranking.
- Overall performance: 1.2 points is the gap between Kimi K3 and the best proprietary model, while Qwen3.5-397B-A17B reaches 47.5% versus 32.5% for GLM-4.6V.Even the strongest open-source models leave every capability below 80%.
- Benchmark validity: 0.84 is the Pearson correlation between released-subset and full-benchmark accuracies across the ten capabilities.The full benchmark is slightly harder for most capabilities, while the released subset preserves difficulty structure and relative model performance at lower evaluation cost.
- Evaluation reliability: 0.32 is Kimi K3’s standard deviation across four runs, with run accuracies spanning only 58.2–58.8.The small variation indicates stable scores despite stochastic inference and diverse visual tasks.
4 Related Work
Related work spans broad end-to-end MLLM benchmarks, application-oriented visual benchmarks, and newer perception-centric evaluations. PerceptionBench differs by deriving ten atomic perceptual capabilities bottom-up from frontier-model failures across 42 existing benchmarks, then balancing questions across capabilities and difficulty tiers.
- General-purpose MLLM benchmarks: General-purpose MLLM benchmarks evaluate end-to-end answers across broad task collections, extending early VQA from scene understanding toward holistic evaluation.These benchmarks organize evaluation around tasks rather than perceptual capabilities.
- Application-oriented visual benchmarks: Application-oriented visual benchmarks target individual applications such as text recognition, document and chart understanding, GUI interaction, and spatial interaction.This line of work moves closer to perception but remains organized around specific visual applications.
- Perception-centric benchmarks: Perception-centric benchmarks use short visual questions to reduce reasoning and knowledge demands while testing capabilities that are relatively easy for humans but challenging for MLLMs.Examples include BLINK, TET, VisFactor, and deliberately simple visual-question evaluations.
- PerceptionBench: PerceptionBench derives ten atomic perceptual capabilities from attributed frontier-MLLM failures on 42 existing benchmarks instead of adopting task boundaries or designer-defined categories.Its question pool is balanced across capabilities and difficulty tiers, retaining direct perception evaluation while grounding its capability taxonomy in observed failures.
5 Conclusion
PerceptionBench evaluates atomic visual perception in MLLMs through ten capability categories derived from failures across 42 benchmarks and 3,000 verified questions isolating single capabilities. Evaluations of sixteen frontier MLLMs suggest atomic perception remains largely unsolved: no model reaches 60% overall accuracy, and every capability remains below 80% even for the strongest models.
- Benchmark design: PerceptionBench evaluates atomic visual perception across ten capability categories derived bottom-up from attributed failures on 42 existing benchmarks.Its 3,000 verified questions each isolate a single capability.
- Benchmark design: 3,000 verified questions each isolate a single capability, enabling focused evaluation of atomic visual perception.The questions are part of PerceptionBench’s benchmark design.
- Findings: No model reaches 60% overall accuracy across evaluations of sixteen frontier MLLMs.These evaluations suggest that atomic perception remains largely unsolved.
- Findings: Even the strongest models leave every capability below 80%.This result further indicates that atomic visual perception remains largely unsolved.
Limitations
PerceptionBench’s taxonomy and difficulty calibration reflect current model weaknesses rather than a fixed structure, while analyzer-dependent failure attribution can introduce errors into capability labels.
- Taxonomy and calibration: The taxonomy is induced from current model failures, so it should be re-induced as models improve.The ensemble-based difficulty calibration should likewise be recalibrated.
- Failure attribution: Failure attribution relies on a stronger analyzer model, allowing residual attribution errors to propagate into capability labels.Each question’s capability label is inherited from first-error attribution.
A Contributions
This section's supplied passage consists solely of the paper's author list and does not state any contributions.
- A Contributions: The passage lists the paper's authors but provides no substantive contribution claims.It names the authors, including Zichao Lin, Yifeng Xie, Bowen Qu, and others.
B Full Error Attribution Taxonomy
The complete taxonomy contains five high-level classes and 22 fine-grained error types, with perception-related errors defining PerceptionBench’s atomic capability taxonomy. A first-error priority rule assigns failures to perception whenever a visual fact is initially misread, reserving reasoning or knowledge attribution for correctly extracted visual facts.
- Taxonomy Structure: Five high-level classes and 22 fine-grained error types provide a comprehensive categorization of multimodal visual-understanding failures.Perception-related errors form the capability taxonomy adopted by PerceptionBench; the other classes support error attribution בלבד.
- Taxonomy Structure: Ten perception error types define the atomic perceptual capabilities evaluated by PerceptionBench.The remaining four classes are reasoning, knowledge, premise, and other errors, and are excluded from the released benchmark.
- Attribution Rule: The first-error priority rule attributes a failure to perception when its trajectory contains an initially misread visual fact, even if downstream reasoning is also incorrect.This rule maps the failure to the corresponding perception error type along the perceptual chain.
- Attribution Rule: Only trajectories with all visual facts verifiably extracted correctly can be attributed to reasoning or knowledge classes.This separates non-perceptual failures from perceptual errors during taxonomy induction.
C Source Benchmark List and License
PerceptionBench’s capability-discovery pipeline aggregates 42 open-source benchmarks spanning diverse visual tasks and domains, yielding approximately 9,000 informative failure cases for taxonomy induction. The source artifacts are used under their original licenses, with most permissive, some attribution-required, and a few restrictive or unspecified licensing terms.
- Source benchmark aggregation: 42 open-source benchmarks, including benchmark splits, are aggregated for failure-driven capability discovery across diverse visual tasks and application domains.Table 5 identifies the source benchmarks used by the pipeline; ZeroBench and CharXiv splits are counted under their parent benchmarks.
- Failure-driven discovery: Approximately 9,000 informative failure cases remain after ensemble-based difficulty filtering and support error-taxonomy induction.The induced taxonomy identifies the atomic perceptual capabilities evaluated in PerceptionBench.
- Licensing: All 42 source benchmarks are publicly available research artifacts used in accordance with their original licenses.The two ZeroBench splits and two CharXiv splits are counted under their parent benchmarks.
- Licensing: Most source benchmarks use permissive Apache-2.0 licenses, while nine require attribution under CC-BY-4.0 or CC-BY-SA-4.0.The passage lists the benchmarks assigned to these license categories.
- Licensing: Two benchmarks carry restrictive terms: We-Math uses CC-BY-NC-4.0 and RealWorldQA uses CC-BY-ND-4.0.PointBench has no explicit license and is used strictly for non-commercial research.
D Pairwise Overlap Between Source Benchmarks
The 42 source benchmarks show limited pairwise similarity in their error-type distributions, with overlap clusters largely confined to near-duplicate benchmark splits and groups sharing the same task.
- Pairwise overlap: 0.20 mean off-diagonal weighted Jaccard overlap indicates limited similarity among the error-type distributions of the 42 aggregated benchmarks.Figure 8 reports pairwise weighted Jaccard overlap across all 42 source benchmarks.
- Pairwise overlap: Off-diagonal clusters are limited to same-family splits and same-task groups, including ScreenSpot-v2/ScreenSpot-Pro and math word problems.Other examples include ZeroBench-main/ZeroBench-sub and CharXiv-Reasoning/CharXiv-Descriptive.
E Evaluation Prompt
The evaluation prompt instructs a professional-teacher grader to compare a student answer with a reference answer and return a structured judgment. Grading prioritizes final-answer consistency, including separate judgments for multiple sub-questions.
- Evaluation procedure: The grader must analyze whether the student’s answer correctly answers the question using the question and reference answer.The required output includes a reason for the judgment and a second field containing the judgment itself.
- Evaluation procedure: The evaluation begins after presenting the question through the placeholder {problem}.The prompt explicitly signals the start of grading before displaying the question.
- Evaluation criteria: Grading focuses on whether the student’s final result matches the reference answer, regardless of whether the solution process is correct.For multiple sub-questions, the reason must report the student answer, reference answer, and consistency for each sub-question.
F PerceptionBench Showcases
Figures 9–11 showcase representative PerceptionBench questions spanning its ten atomic perceptual capabilities, with short answers targeting localization, attributes, recognition, depth, OCR, comparison, counting, context integration, and hallucination. Examples include precise text transcription, spatial judgments, hidden-object counting, and absence detection.
- Representative capability examples: Localization and attribute questions test spatial positions, region contents, object appearance, and geometric composition.Examples identify the Gemini symbol at position 6, a tree-free region as Region 2, and compare pen-holder styles.
- Representative capability examples: Recognition, relation, and depth questions assess fine-grained visual distinctions, maze traversal, line direction, and 3D spatial structure.Examples select a matching notch contour, identify the first cat reached as the center cat, determine that the person in red is farther back, and count 26 cubes.
- Representative capability examples: OCR examples require exact transcription of text while preserving case, punctuation, spaces, and specified image regions.Answers include “Marshall B. Clinard” and “Tap Water.”
- Representative capability examples: Comparison and counting questions require selecting the largest or thicker visual element and counting objects under explicit inclusion or exclusion conditions.Examples identify circle N as largest, segment EM as thicker, count 2 flowers inside a box, and count 5 visible potted plants excluding reflections.
- Representative capability examples: Context integration and hallucination questions test cross-image scene changes, map-based relations, and recognizing when queried objects or patterns are absent.Examples identify Dasthagir & Sons as lying on both sides of Road No. 2, detect an added cardboard box, and report zero white tables and zero dog-logo patterns.