PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

Results

• No evaluated model exceeded 60% accuracy (GPT-5.6-Sol 59.7%; Kimi K3 58.5%); perception-related hallucination scored lowest on average, capability profiles diverged, and repeated-run answers were unstable.Similar aggregate scores concealed substantial differences in capabilities acquired, while a substantial fraction of correct answers were not consistently reproduced across runs.Source: Figure 1 · p. 1, §1 · p. 2

Motivation

• Visual perception underpins downstream reasoning and environmental interaction, and missed, misread, or hallucinated content can invalidate subsequent reasoning (Xingyu Fu et al. 2024; Rahmanzadehgervi et al. 2024; J.-T. Huang et al. 2025; Buschoff et al. 2025; Gao, Zihao Huang, et al. 2025).The paper describes perception as distinct atomic capabilities, including attribute recognition, counting, localization, and text reading.Source: §1 · p. 1
• Holistic benchmarks mix perception with reasoning and external knowledge, while application-specific benchmarks measure only designer-defined slices of perception.The introduction contrasts MMMU and MathVista with OCRBench and ScreenSpot as examples of these limitations.Source: §1 · p. 2

Benchmark construction

• PerceptionBench comprises 3,000 verified questions across ten capabilities, derived by tracing frontier MLLM failures on 42 benchmarks to their earliest erroneous steps and clustering error labels.Questions were decomposed from attributed failures or authored against the taxonomy, and verified so difficulty stems from perception rather than reasoning or external knowledge.Source: §1 · p. 2

Failure analysis

• Failure attribution across 42 open-source benchmarks showed that existing suites probe narrow, weakly overlapping slices of the failure space, while perception failure types recur across nearly all benchmarks.The never-observed over-refusal type is omitted from Figure 3.Source: §1 · p. 2, Figure 3 · p. 3

2 PerceptionBench

• Existing benchmarks cover sparse, weakly overlapping slices of perceptual failures, so PerceptionBench uses a bottom-up pipeline grounded in failures across 42 benchmarks.The pipeline discovers capabilities, selects or constructs targeted questions, and verifies each sample; the released set is subsampled from an in-house pool exceeding 17,000 samples.Source: §2 · p. 3, §2.1 · p. 3

2.1.1 Failure-driven Capability Discovery

• Failure attribution prioritizes the earliest erroneous step, producing five error classes and 22 fine-grained types, with perception's ten types adopted as benchmark capabilities.These capabilities are visual localization, visual attribute recognition, visual counting, visual relation understanding, depth and 3D perception, OCR, visual comparison, fine-grained recognition, context integration, and perception-related hallucination.Source: §2.1.1 · p. 4

2.1.2 Benchmark Selection and Construction

• Four frontier MLLMs identify saturated items and stratify retained samples by difficulty; a separate group of sixteen models is reserved for evaluation.Newly constructed samples are screened the same way, removing items solved by the full ensemble to avoid tailoring the benchmark to an individual model.Source: §2.1.2 · pp. 4–5
• Annotators decompose failures into atomic questions and author additional questions on supplemented images to broaden visual diversity and capability coverage.Images are checked against large-scale pre-training corpora, and near-duplicates or likely leaked images above cosine similarity 0.95 are discarded; questions are designed to have unique, visually grounded answers.Source: §2.1.2 · p. 5
• The released 3,000-question subset is drawn from the constructed portion of the in-house pool and is intended to preserve its per-capability difficulty structure and relative model performance at lower evaluation cost.The paper reports that the subset costs less than one fifth of full-pool evaluation.Source: §2.1.2 · p. 5

2.2 Data Verification

• Automated capability-alignment and visual-grounding checks, followed by independent human review, remove or revise samples that rely on reasoning, outside knowledge, ambiguous evidence, or incorrect annotations.The verifier checks whether the target capability is primary; human reviewers assess evidence sufficiency, challenge alignment, and answer unambiguity.Source: §2.2 · pp. 5–6

2.3 Benchmark Statistics

• 3,000 verified samples span ten capabilities; 1,800 (60%) are decomposed source-benchmark questions and 1,200 (40%) are newly authored.The release is capability-balanced and difficulty-stratified, with a distribution based on observed perceptual failure frequencies.Source: §2.3 · p. 6, Figure 5 · p. 6

3.1 Experiment Settings

• All models use unified prompts and official inference settings where available, and the GPT-oss-120B judge agrees with human judgments on 299 of 300 sampled predictions (99.7%).The evaluation covers ten proprietary and six open-source models, uses each model's highest available reasoning budget, and evaluates open-ended questions with short, uniquely determined reference answers.Source: §3.1 · p. 6

3.2 Main Results

• 59.7% is GPT-5.6-Sol's leading overall accuracy, while Kimi K3 is the top open-source model at 58.5%, and overall accuracy spans 32.5% to 59.7%.Kimi K3 trails the best proprietary model by 1.2 points and outperforms every other proprietary system; within open-source models, Qwen3.5-397B-A17B scores 47.5% versus GLM-4.6V at 32.5%, while every capability remains below 80% even for the strongest models.Source: §3.2 · pp. 6–7, Table 1 · p. 7
• Perception-related hallucination averages 36.7%, below visual relation (53.2%), OCR (52.4%), and localization (52.0%); GPT-5.6-Sol scores 76.7% on localization but 26.9% on hallucination.GPT-5.5 (55.8%) and Gemini-3.1-Pro (56.2%) differ by less than one point overall, yet GPT-5.5 leads in localization by 13 points (65.8 vs. 52.7), while Gemini-3.1-Pro leads in OCR by 8 points (64.3 vs. 56.5) and fine-grained recognition by 8 points (54.8 vs. 47.2).Source: §3.2 · p. 7
• Hallucination results invert overall ranking: GPT-5.6-Sol scores 26.9%, while mid-pack Gemini-3.5-Flash scores 50.6%, the best result, followed by Seed-2.1-Pro at 49.8%.The authors hypothesize that aggressive answer commitment can inflate scores on answerable questions while causing plausible but nonexistent visual content to be asserted on false-positive probes; they note this pattern is echoed by pass@4 versus pass4 in Section 3.5.Source: §3.2 · p. 7

3.3 Qualitative Analysis

• In four representative decomposed cases, models already make errors attributable to atomic perceptual capabilities such as counting or localization, before reasoning or knowledge is implicated.The cases use source-benchmark items whose original questions require multi-step solutions and illustrate that failures are often perceptual before cognitive.Source: §3.3 · p. 7, Figure 6 · p. 8

3.4 Benchmark Representativeness

• Released-subset and full-benchmark per-capability accuracies correlate strongly (Pearson r = 0.84), while the full benchmark is slightly harder for most capabilities.The full in-house benchmark contains more than 17,000 verified samples; the authors conclude the released subset preserves per-capability difficulty structure and relative model performance while reducing evaluation cost.Source: §3.4 · p. 8

3.5 Evaluation Reliability

• Across four runs, Kimi K3's accuracy spans only 0.6 points (58.2–58.8, std 0.32), indicating stable benchmark scores despite sampling randomness.However, pass@4 versus pass4 shows that many samples are solved only intermittently, so stable aggregate accuracy does not imply robust per-sample perceptual behavior.Source: §3.5 · p. 9, Table 2 · p. 9

General-purpose MLLM benchmarks

• General-purpose MLLM benchmarks score broad end-to-end tasks, so aggregate performance cannot distinguish perception failures from reasoning or knowledge failures.The passage contrasts early VQA suites (Antol et al. 2015; Hudson et al. 2019; Y. Zhu et al. 2016) with recent holistic benchmarks (Yue, Ni, et al. 2024; Yuan Liu et al. 2024; Lin Chen et al. 2024; Yue, Zheng, et al. 2025).Source: §4 · p. 9

Application-oriented visual benchmarks

• Application-oriented visual benchmarks probe perception more directly, but cover narrow, imbalanced skill slices and may still demand domain reasoning or specialized output formats.The passage discusses text recognition, document and chart understanding, GUI and spatial interaction, and visual grounding benchmarks.Source: §4 · p. 9

Perception-centric benchmarks

• Perception-centric benchmarks use short questions to reduce reasoning demands, yet top-down categories and weak overlap among exposed failures leave perception’s broader structure fragmented.The passage notes that frontier MLLMs still struggle with basic visual perception.Source: §4 · p. 10

PerceptionBench’s distinction

• PerceptionBench derives ten atomic perceptual capabilities from failures across 42 benchmarks, balancing questions across capabilities and difficulty tiers for granular diagnosis.This retains direct perception-focused evaluation while grounding capability coverage in empirically observed model weaknesses.Source: §4 · p. 10

5 Conclusion

• No evaluated model reaches 60% overall accuracy, and even the strongest models score below 80% on every capability.Perception-related hallucination is the weakest capability on average, including for top-ranked systems.Source: §5 · p. 10
• Models with nearly identical overall scores exhibit sharply divergent capability profiles, so aggregate accuracy conceals which capabilities they have acquired.Aggregate scores remain stable across repeated runs, but per-sample behavior does not, suggesting some measured accuracy comes from unstable predictions rather than reliable perception.Source: §5 · p. 10
• Kimi K3 trails the overall leader by only 1.2 points, indicating that the open-source frontier is closing in.Source: §5 · p. 10

• The taxonomy reflects current model weaknesses, so it should be re-induced and difficulty recalibrated as models improve.Capability labels may also change when frontier models’ first erroneous steps shift.Source: Limitations · p. 10
• Failure attribution depends on a stronger analyzer model, whose residual errors may propagate into capability labels.Source: Limitations · p. 10
• PerceptionBench scores do not directly predict end-to-end task performance because the benchmark isolates perception by design.The paper recommends using it alongside task-oriented suites.Source: Limitations · p. 10

B Full Error Attribution Taxonomy

• PerceptionBench evaluates ten atomic perceptual capabilities, defined by the taxonomy’s perception error types.Table 3 summarizes these ten types.Source: B Full Error Attribution Taxonomy · p. 13, Table 3 · p. 13
• The remaining classes—reasoning, knowledge, premise, and other errors—distinguish non-perceptual failures during taxonomy induction but are excluded from the released benchmark.Together with perception, the complete taxonomy comprises five high-level classes and 22 fine-grained error types.Source: B Full Error Attribution Taxonomy · p. 13
• When a trajectory misreads a visual fact, attribution goes to the first erroneous perceptual step, even if downstream reasoning is also incorrect.Reasoning or knowledge attribution applies only when all visual facts are verifiably extracted correctly.Source: B Full Error Attribution Taxonomy · p. 13

C Source Benchmark List and License

• 42 source benchmarks span diverse visual tasks and domains, yielding approximately 9,000 informative failures that underpin taxonomy induction and identification of atomic perceptual capabilities.Table 5 lists the source benchmarks aggregated for failure-driven capability discovery.Source: C Source Benchmark List and License · pp. 13–14, Table 5 · p. 14
• Most source benchmarks use Apache-2.0, MIT, or BSD-3-Clause licenses; others use attribution-required Creative Commons licenses, while We-Math and RealWorldQA have noncommercial and no-derivatives terms, respectively.PointBench has no explicit license and is used strictly for non-commercial research.Source: C Source Benchmark List and License · pp. 14–15
• Every PerceptionBench sample carries source-benchmark and original-index provenance, and source-derived samples remain subject to their origin licenses; RealWorldQA contributes no samples.RealWorldQA is used only for failure analysis, and the authors’ annotations and newly authored questions are released under Apache-2.0.Source: C Source Benchmark List and License · p. 15

D Pairwise Overlap Between Source Benchmarks

• Mean off-diagonal weighted Jaccard overlap was 0.20, with clusters limited to same-family splits and same-task groups, indicating fragmented, complementary failure views.Source: Figure 8 · p. 15, D Pairwise Overlap Between Source… · p. 15

E Evaluation Prompt

• The grader must return a brief justification in [reason] and a single True-or-False verdict in [judge].The output must end with True or False, and the reason must be no more than 100 tokens.Source: E Evaluation Prompt · p. 16
• Judge correctness by comparing the final result with the reference, requiring every sub-question to match and multiple-answer responses to include all correct answers without extras.For multiple-choice questions, any selected option that differs from the reference is incorrect, regardless of the explanation.Source: E Evaluation Prompt · p. 16
• For numerical answers, compare values using the specified significant-figure rules, convert differing units to a common unit, and explain comparisons in scientific notation.Integers and values with at most four significant figures must match exactly; values with more than four must match within four significant figures, comparing order of magnitude before the first four significant figures.Source: E Evaluation Prompt · p. 16
• The prompt also requires English responses for English-writing questions, exact technical terms in biology, chemistry, and physics, and key points when explaining a term.Equivalent simplified answers require the complete simplification process in the reason; vague claims that answers can be made equal are insufficient.Source: E Evaluation Prompt · p. 16

Visual Localization

• Localization samples ask where the Gemini symbol lies on a clock (6 o’clock) and which numbered region contains no trees (region 2).Source: F PerceptionBench Showcases · p. 17

Visual Attribute and Fine-grained Recognition

• Attribute and recognition samples test comparisons of pen-holder appearance (B), character line composition (B), and people’s hair and clothing (A).Another recognition sample asks which option’s contour fills a notch; its answer is D.Source: F PerceptionBench Showcases · p. 17

Visual Comparison and Visual Counting

• Comparison and counting samples ask which circle is largest (N), how many flowers lie inside a red box (2), and how many potted plants are visible (5 pots).Source: F PerceptionBench Showcases · p. 19

Context Integration

• Context-integration samples ask which building lies on both sides of a named road (B) and what was added between two scene images (a cardboard box marked with an arrow, D).Source: F PerceptionBench Showcases · p. 19