PaperlayerView exact revision on arXiv

Source-linked AI summary

PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception

Yana Wei, Hongbo Peng, Yanlin Lai, Liang Zhao, Kangheng Lin, En Yu, Keyu Lv, Han Zhou, Yin Tang, Haodong Li, Mitt Huang, Hangyu Guo, Jianjian Sun, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Vishal M. Patel

arXiv:2606.28322v2cs.CV

TL;DR

Existing multimodal benchmarks can look saturated while missing brittle, human-salient perceptual errors. PerceptionRubrics addresses this with atomic rubrics and gated scoring, revealing a reliability gap between local fact recognition and consistent perception, including an 8% deficit for open-source frontiers.

  • Problem

    Existing multimodal perception benchmarks can remain saturated while failing to reflect genuine perceptual capability and human-sensitive errors.

  • Method

    PerceptionRubrics evaluates 1,038 information-dense images with atomic Must-Right and Easy-Wrong rubrics under a gated scoring mechanism.

  • Results

    Models often pass individual rubric checks yet fail conjunctive mandatory constraints, while open-source frontiers show an 8% perception deficit versus proprietary leaders.

  • Takeaways & Limitations

    Reliable multimodal evaluation should explicitly audit critical visual facts rather than rely on coarse similarity scores.

  • Takeaways & Limitations

    The benchmark assumes that extreme information density and atomic, error-sensitive checks are appropriate proxies for comprehensive perceptual capability.

Abstract

from arXiv · show

We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 10,000 instance-specific rubrics. These criteria are derived from golden captions constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 8% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.

1. Introduction

PERCEPTIONRUBRICS addresses saturated but perceptually brittle MLLM benchmarks by evaluating dense images through atomic, human-refined rubrics and gated scoring. Its evaluation reveals failures in conjunctive consistency, an 8% open-source perception deficit, and stronger human alignment than conventional benchmarks.

  • Motivation: Current benchmarks saturate at high scores while missing salient perceptual failures such as object-counting errors and inverted spatial relations.These failures remain important to users despite high reported metric scores.
  • Benchmark Design: PERCEPTIONRUBRICS curates 1,038 information-dense, distributionally diverse images to create a rigorous caption-based diagnostic testbed.Captioning serves as a proxy for integrated perception, recognition, and reasoning.
  • Rubric Design: Over 10,000 atomic rubrics are organized into Must-Right essential facts and Easy-Wrong details targeting hallucinations, omissions, and misinterpretations.The rubrics are distilled from Golden Captions with human refinement and mined model error patterns.
  • Key Findings: Models often pass fragmented atomic checks yet fail strict conjunctive constraints, exposing a Reliability Gap in information-dense domains such as GUIs.High partial scores therefore do not guarantee perceptual consistency.
  • Key Findings: 8% perception deficit separates the open-source frontier from proprietary leaders, while gated scoring aligns substantially better with human judgment than conventional benchmarks.The evaluation also finds near-perfect correlation between basic perception and hallucination resistance.

2. Related Work

Related work spans broad and task-specific visual perception benchmarks, increasingly detailed image-captioning evaluation, and rubric-based auditing. Despite these advances, leaderboard saturation, sparse references, and linear scoring remain limitations that can obscure localized hallucinations and diverge from human verification.

  • Visual Perception Benchmarks in MLLMs: Visual perception benchmarks divide into holistic suites and task-specific datasets, while broad frameworks increasingly face leaderboard saturation in recent flagship models.Examples include MMBench, MM-Vet, and MME.
  • Evaluation of Image Captioning: Image-captioning evaluation has progressed beyond generic similarity and object-set matching toward detailed reference-based, multi-expert, aspect-based, and human-preference assessments.The passage names methods including DetailCaps, RePer, and CapArena.
  • Evaluation of Image Captioning: Existing captioning methods often rely on sparse, biased references and linear scoring, diluting fatal localized hallucinations with high holistic similarity.This limitation is attributed to a mismatch with the non-linear sensitivity of human verification.
  • Rubric-Based Reward Modeling: Rubric-based reward modeling shifts evaluation from opaque scalar scoring toward structured auditing and reasoning processes built from explicit criteria or chain-of-rubrics.The passage reports that structured criteria have mitigated reward hacking in text generation and cites RM-R1, SPCT, RaR, and ResearchRubrics.

3. PerceptionRubrics

PerceptionRubrics targets human-aligned multimodal evaluation by prioritizing dense, diverse images, atomic rubric construction, and error-sensitive gated scoring. Its caption-centric pipeline combines model consensus, lightweight human verification, dual-stream rubrics, and mandatory-fact gates.

  • Design Principles: The benchmark prioritizes perceptual complexity over scale, using information-dense images that require fine-grained exploration rather than linguistic shortcuts.Its collection emphasizes visual diversity and details that maximize error potential.
  • Data Curation: Data spans seven categories covering natural scenes, documents and OCR, digital interfaces, structured data, STEM and expert imagery, and logic and puzzles.These domains are designed to cover the spectrum of multimodal capabilities.
  • Caption-Centric Pipeline: The caption-centric pipeline transcribes visual information into text before distilling precise, exhaustive rubrics, mitigating visual grounding gaps in direct pixel-to-rubric generation.Golden captions are produced through circular peer review and strict consensus filtering, with humans serving only as final verifiers.
  • Rubric Construction: Dual-stream rubrics combine Must-Right facts distilled from references with Easy-Wrong criteria derived from baseline-model hallucinations and misinterpretations.Domain-adaptive prompts emphasize character precision in text-centric images and spatial relations or object attributes in natural scenes.
  • Gated Scoring: The gated judge assigns boolean rubric outcomes, setting the score to zero when any mandatory Must-Right criterion fails and using Easy-Wrong items for granular differentiation otherwise.This makes high scores require both basic factual correctness and accurate capture of subtle, error-prone details.

4. Experiments

Experiments establish PerceptionRubrics as a dense, atomic benchmark and reveal substantial differences in models’ fine-grained perceptual reliability. Gated evaluation exposes domain-specific failures, a proprietary–open-source gap, and a near-perfect link between essential-fact grounding and detail accuracy.

  • Benchmark and Evaluation Setup: PerceptionRubrics contains 1,038 information-dense images and 10,718 atomic rubrics, including 4,053 Must-Right and 6,665 Easy-Wrong criteria.Each image has a verified golden caption and instance-specific rubrics, averaging 10.33 rubrics per image.
  • Compliance Scores: Seed-2.0-Lite leads with an overall score of 70.07%, exceeding Gemini-3.5-Flash by 0.19%.The evaluation covers 25 models spanning proprietary frontier and leading open-weights systems.
  • Compliance Scores: 61.61%: Qwen3.5 is the best-performing open-source model but trails proprietary state-of-the-art by over 8%.The results indicate a persistent open-source deficit in fine-grained perception and open-world recognition.
  • Domain-Specific Failure Modes: GUI is the dominant source of Must-Right gate failures, whereas Natural and STEM are comparatively easier domains.These failures correspond to cases where predictions do not pass the Must-Right gate, with G = 0.
  • Atomic vs. Holistic Perception: Atomic Accuracy and Must-Right Pass Rate expose a reliability gap between local rubric correctness and satisfying all mandatory visual facts.The gap narrows as model capability increases, suggesting stronger models better maintain the consistency required to keep G open.
  • Consistency of Perceptual Capabilities: R2 ≈0.98: Must-Right Pass Rate has a near-perfect linear correlation with Easy-Wrong accuracy.Models that fail to ground essential visual facts also struggle with subtle details and hallucination resistance.

5. Analysis

The analysis finds that PERCEPTIONRUBRICS closely tracks human preferences while remaining robust to response length, judge choice, and rubric sampling. Its evaluation is stable and preserves model rankings across independent judges and increasing rubric coverage.

  • Human Alignment: Pearson 0.916 and Spearman 1.000: PERCEPTIONRUBRICS shows the strongest alignment with Vision Arena human-preference rankings among compared benchmarks.DOCCI and DetailCaps show substantially weaker agreement with human-preference scores.
  • Length Bias: r = −0.079 and p = 0.0758: Gemini-3.1-Pro shows no statistically significant correlation between predicted caption length and PERCEPTIONRUBRICS performance.Kimi-K2.6 exhibits a weak positive correlation, with r = 0.172 and p = 1.09 × 10−4.
  • Evaluation Robustness: Identical ranking order: GPT-OSS-120B and GPT-5.5 produce the same ordering across three representative models despite a ∼6.0% stricter scoring distribution from GPT-OSS-120B.Standard deviations across independent runs remain consistently low.
  • Rubric Coverage: 20% to 80%: increasing the sampled rubric ratio improves evaluation stability monotonically, as measured by decreasing standard deviations of model scores.The analysis samples both Must-Right and Easy-Wrong rubrics and averages results over three independent runs.

6. Conclusion

PERCEPTIONRUBRICS calibrates multimodal evaluation to human perceptual judgment by decomposing dense image understanding into atomic, verifiable rubrics with gated scoring. Across 25 MLLMs, it exposes reliability gaps and information-dense-domain weaknesses while aligning strongly with human preferences.

  • 6. Conclusion: PERCEPTIONRUBRICS decomposes dense image understanding into atomic, verifiable rubrics and enforces a gated scoring mechanism.The framework calibrates multimodal evaluation to human perceptual judgment.
  • 6. Conclusion: 25 MLLMs reveal a reliability gap between individual fact recognition and consistent conjunctive perception.This exposes perceptual failures often hidden by existing metrics.
  • 6. Conclusion: Information-dense domains such as GUIs remain persistently weak, despite strong alignment between PERCEPTIONRUBRICS scores and human preferences.The findings motivate moving beyond coarse similarity to explicitly audit critical visual facts.

Impact Statement

The work aims to improve the reliability of multimodal evaluation and may influence downstream multimodal large language model development, without identifying specific societal consequences requiring special discussion.

  • Impact Statement: The framework targets more reliable multimodal evaluation, with potential effects on downstream MLLM development.The authors identify no specific societal consequences requiring special discussion.

A. Dataset Statistics

This section presents detailed statistics and comparisons for the PERCEPTIONRUBICS benchmark.

  • The section reports detailed statistics for the PERCEPTIONRUBICS benchmark.
  • It also provides comparisons for PERCEPTIONRUBICS.
  • Together, the statistics and comparisons characterize the PERCEPTIONRUBICS benchmark.

A.1. Comparison with Other Benchmarks … B. Model Roles and Pipeline Details

PerceptionRubrics broadens benchmark evaluation through dense, diverse captions and domains, while its rubric distributions and model-role pipeline support granular, balanced assessment. Caption lengths, rubric counts, and specialized model assignments characterize the benchmark’s construction and evaluation process.

  • A.1. Comparison with Other Benchmarks: PerceptionRubrics differs from existing benchmarks in annotation granularity, data-source diversity, and domain coverage.These three dimensions are identified as the benchmark’s critical distinctions in Table 3.
  • A.1. Comparison with Other Benchmarks: 770.42 words per image, versus 122.1 for DetailCaps-4870 and 135.9 for DOCCI, supports dense captioning of fine-grained details and spatial relationships.The benchmark is presented as a more challenging testbed for evaluating MLLM upper bounds.
  • A.1. Comparison with Other Benchmarks: Seven domains, including GUIs, OCR-heavy documents, and STEM diagrams, extend evaluation beyond predominantly natural-scene benchmarks.The stated purpose is to assess general-purpose capabilities in complex real-world applications beyond simple object recognition.
  • A.1. Comparison with Other Benchmarks: The dataset combines existing visual benchmarks with hybrid annotation using advanced reasoning models and human expert verification.This approach is described as balancing scalability with ground-truth reliability.
  • A.2.1. CAPTION LENGTH DISTRIBUTION: Golden-caption lengths follow a long-tail distribution, with most between 300 and 700 words, a median of 569, and a maximum of 3,461.The distribution is described as covering both concise summaries and highly detailed descriptions.
  • A.2.2. RUBRIC DISTRIBUTION: Total rubric counts primarily range from 8 to 14 and peak at 10, while Must-Right rubrics center around 4 core facts.Easy-Wrong rubrics are contrasted with Must-Right rubrics in the category breakdown.
  • B. Model Roles and Pipeline Details: STEP-3-VL-10B filters complex images, Gemini-3-Pro generates rubrics, Gemini-3-Pro/GPT-5.2/Seed-1.8 validate captions, and GPT-OSS-120B performs final scoring.The models are assigned distinct roles for complexity judging, rubric generation, consensus judging, and evaluation scoring.

C. Prompts

The prompts define a reproducible pipeline for filtering information-dense images and generating grounded, binary gating rubrics. Complexity filtering scores visual complexity and informativeness, while rubric prompts enforce zero-tolerance checks for essential visible content.

  • C.1. Complexity Filtering Prompt: The filtering prompt penalizes blurry, noisy, scribbled, or excessively empty images and requires XML output containing reasoning and integer scores.Text alone does not warrant a high score; density, semantic depth, subject richness, and content—not image dimensions—determine evaluation.
  • C.1. Complexity Filtering Prompt: 1-10 points score images separately for visual complexity and informativeness.Visual complexity measures independent elements, spatial occupancy, and detail clarity; informativeness measures descriptive content, contextual richness, and knowledge value.
  • C.1. Complexity Filtering Prompt: 1-3 points denote low visual complexity, 4-7 medium, and 8-10 high.High-complexity examples include crowds, dense forests, and complex mechanical structures, while low-complexity examples include white space, isolated objects, and blurry snapshots.
  • C.1. Complexity Filtering Prompt: 1-3 points denote low informativeness, 4-7 medium, and 8-10 high.High-informativeness examples include dense documents, complex infographics, narrative-rich historical photos, and scenes requiring long-form descriptions.
  • C.2. Rubric Generation Prompt: 0 is assigned when a candidate fails any checkpoint in the nature-scene rubric task.Checkpoints must be undeniably visible, essential to the image’s core meaning, and verifiable as binary Pass/Fail checks.
  • C.2. Rubric Generation Prompt: Nature-scene rubrics abstract entities, decouple attributes, and retain attributes only when needed to distinguish same-class objects.The grounding check requires each element to appear in both the image and GT caption, discarding hidden or hallucinated details.
  • C.2. Rubric Generation Prompt: 3 to 5 strings are returned as a strictly valid JSON list of generalized rubric criteria.The prompt supplies examples for natural scenes and animal interactions, emphasizing objects, context, interactions, and actions.
  • C.2. Rubric Generation Prompt: 0 is assigned when a candidate fails any GUI checkpoint, which must be visible, functionally critical, and binary Pass/Fail verifiable.GUI rubrics target essential operating or navigation elements such as Submit buttons and Back arrows, excluding decorative banners and advertisements.

D. Human Annotation Feedback

Human annotators refined sophisticated multimodal-model drafts through meticulous verification, focusing on fine-grained visual alignment and deterministic, high-confidence descriptions. They preserved the captions’ natural stylistic diversity to support evaluation of both narrative generation and structured information extraction.

  • Annotation Pipeline: Golden captions averaged 770.42 words and were produced through model voting followed by meticulous human verification.The initial drafts came from Gemini-3-Pro, GPT-5.2, and Seed-1.8.
  • Annotation Feedback: Annotators found the AI-generated drafts surprisingly sophisticated, reducing structural rewriting while increasing vigilance and fine-grained verification demands.
  • Hard Cases and Visual Nuances: Hard Cases and Visual Nuances: Annotators struggled with fine-grained semantic alignment in blurred edges, complex lighting, and severe occlusion.Recurring errors involved material-boundary interpretation, precise spatial relations, and hallucinated objects in low-visibility regions.
  • Annotation Policy: Determinism over Ambiguity: Annotation Policy: Humans deleted unclear or hallucinated content, explicitly describing recognizable objects and retaining only grounded, high-confidence information.Annotators used tool-assisted zooming when needed and otherwise removed ambiguous descriptions.
  • Diversity of Caption Styles: Diversity of Caption Styles: The benchmark preserved literary narratives and structured Markdown representations to assess narrative generation and structured information extraction.

E. Additional Experimental Results · F. Qualitative Examples

The additional results section presents comprehensive PerceptionRubrics evaluations with explicitly defined item-level and sample-level metrics. Qualitative examples span seven categories and demonstrate two rubric tiers: Must-Right core facts and Easy-Wrong fine-grained details, negative constraints, and logical reasoning.

  • E. Additional Experimental Results: Table 4 reports comprehensive evaluation results for models grouped as Open-Source or Proprietary and sorted by Overall Score.All values are percentages.
  • E. Additional Experimental Results: Must-Right Item Accuracy measures essential-item correctness, while Easy-Wrong Item Accuracy measures fine-grained detail correctness.The table also defines Gate Pass as the sample-level rate with all Must-Right items correct, and E-W Avg as mean per-case Easy-Wrong accuracy.
  • F. Qualitative Examples: The benchmark covers seven categories, from daily natural scenes to specialized STEM diagrams and logic puzzles.Examples are presented across Natural Scene, Document & OCR, Digital UI & UX, Structured Data, Logic & Puzzle, STEM & Expert, and Creative & Cultural.
  • F. Qualitative Examples: Must-Right rubrics target core factual elements and primary subjects essential for basic scene understanding.The figures place these items in the top rubric group and mark them with an “OK” icon.
  • F. Qualitative Examples: Easy-Wrong rubrics target challenging details, including specific objects, attributes, text, spatial relations, trends, and technical interface elements.Examples include identifying vegetables, label text and fruit, Android Studio components, plot variables, curve trends, and parameter values.
  • F. Qualitative Examples: Negative constraints require rubrics to reject plausible but incorrect descriptions, such as modern vehicles, people, Brussels sprouts, yellow bulbs, green backgrounds, or Xcode environments.These examples show that Easy-Wrong criteria can test explicit exclusions and fine-grained distinctions.
  • F. Qualitative Examples: The qualitative figures organize each example into two rubric tiers, with Must-Right above Easy-Wrong across all seven categories.Easy-Wrong includes challenging details, negative constraints, and logical reasoning.
Loading 2606.28322v2…