PaperlayerView exact revision on arXiv

Source-linked AI summary

PRISM-BENCH: Diagnosing Grounded and Faithful Multimodal Reasoning Beyond Final-Answer Accuracy

Vansh Bhatia, Mehdi Rezagholizadeh, Parsa Ashrafi Fashi, Igor Gilitschenski, Vikram Appia, Emad Barsoum

arXiv:2610.36554v1cs.CVcs.CL

TL;DR

Final-answer accuracy does not report whether multimodal answers are grounded, faithful to their evidence, or economical to produce. PRISM-BENCH pairs 1,000 questions across five categories with human reasoning traces and evaluates grounding, trace alignment, and reasoning cost. Frontier systems remain below the human baseline, with recent gains concentrated in recognition rather than spatial grounding.

  • Problem

    Final-answer accuracy alone cannot show whether an answer is grounded in the image, whether stated reasoning reflects its evidence, or what it costs to produce.

    Source: §1 · p. 1

  • Method

    PRISM-BENCH pairs 1,000 expert-curated questions across five categories with human-authored reasoning traces, enabling grounding and trace-alignment diagnostics alongside the cost-aware CORE metric.

    Source: §1 · pp. 1–2, §3.1 · p. 3, §4 · p. 4

  • Results

    The best system reaches 65.5% on the four categories with a human baseline, trailing humans by 18.4 points; recent frontier gains are strongest in recognition while spatial grounding remains a deficit.

    Source: §5.1 · p. 4, §7 · p. 9

  • Takeaways & Limitations

    The benchmark’s results show that accuracy alone misses deployment-relevant differences in grounding, reasoning faithfulness, and reasoning cost.

    Source: §4 · p. 4, §7 · p. 9

    CORE-PRM’s scores are bounded by the supplied judge, and a stronger multimodal critic could improve the variant without changing the benchmark.

    Source: §5.5 · p. 9

Abstract

from arXiv · show

Final-answer accuracy reports whether a vision-language model was right, but not whether its answer was grounded in the image, whether its stated reasoning reflects that evidence, or what the answer cost to produce. Those are the properties that decide whether a system can be deployed. We present PRISM-BENCH, a benchmark of 1,000 expert-curated questions spanning five under-represented reasoning categories, in which every item carries an idealized reasoning trace written by one human annotator and validated by another. The traces support two diagnostics that accuracy cannot provide. A perception probe, in which the model answers from a description it wrote itself, attributes roughly 85% of frontier-model failures to misperception and places the bottleneck at grounding. Trace alignment predicts answer correctness for all 25 systems tested (macro AUROC 0.66) and survives length control, so it measures reasoning content and not verbosity. We further introduce CORE, an efficiency-aware metric that reorders 24.2% of model pairs relative to accuracy while remaining uncorrelated with response length. Across 25 open and closed systems, including GPT-5 and Claude Opus 4.6, the best accuracy is 66.0%. On the four categories with a human baseline the best system trails humans by 18.4 points, and recent frontier gains fall on recognition and not on spatial grounding.

1 Introduction

Final-answer accuracy omits whether multimodal answers are grounded, faithful to their evidence, and affordable to produce. PRISM-BENCH pairs questions with human reasoning traces to support diagnostics of grounding, reasoning content, and cost.

1 Introduction

  • PRISM-BENCH pairs 1,000 expert-curated questions across five under-represented categories with human-authored idealized reasoning traces, enabling process diagnostics beyond answer accuracy.The traces provide a per-item reference for the reasoning process as well as the outcome.Source: §1 · pp. 1–2
  • Roughly 85% of frontier-model failures are attributed to misperception, and answering from a model’s own image description reduces accuracy by up to 11.5 points.The perception probe tests whether the model can answer from its own description after the visual input is removed.Source: §1 · p. 2, Figure 1 · p. 2
  • Reference-trace alignment separates correct from incorrect responses across all 25 tested systems, with macro AUROC 0.66 and 0.665 after length control.The length-controlled result supports alignment as a measure of reasoning content rather than verbosity.Source: §1 · p. 2, Figure 1 · p. 2
  • CORE scores accuracy per unit of reasoning cost, reorders 24.2% of model pairs relative to accuracy, and remains uncorrelated with raw length (ρ = −0.05).The evaluation covers 25 open and closed systems, including GPT-5 and Claude Opus 4.6, alongside an expanded human baseline and protocol audits.Source: §1 · p. 2

2 Related Work

Prior multimodal benchmarks cover visual mathematics, spatial reasoning, abstraction, and domain-specific tasks, but largely score final answers without step-level traces or generation cost. PRISM-BENCH’s categories combine functional reasoning patterns with domain-specific grounding requirements.

Multimodal reasoning benchmarks

  • Existing multimodal benchmarks span visual mathematics [11] [24] [15], spatial and relational understanding [8] [5], abstraction [2] [10], and domain-specific settings [12] [13] [7].Broader suites such as MMMU-Pro [23] and MegaBench [4] reduce shortcut exploitation, but benchmarks largely score final answers and rarely release reference reasoning or report generation cost.Source: §2 · p. 2

Reasoning and process supervision

  • Process supervision includes multimodal adaptations of chain-of-thought [26], RL-style training [6], GRPO-style optimization [18] [19], self-reflection [20], and structured captioning [22].Process reward models such as VisualPRM [21] and DreamPRM [3] score intermediate steps, while current PRMs show unstable labelling and weak out-of-distribution generalization.Source: §2 · p. 2

Benchmark scope

  • The five PRISM-BENCH categories balance functional reasoning patterns, such as multi-hop composition, with domain-specific grounding requirements, such as medical interpretation.The figure assigns 200 items to each category.Source: Figure 2 · p. 3
  • Table 1 partitions all 1,000 items across three answer-choice rows; most are 4-way or wider multiple choice, while 11 are open-ended or binary.Source: Table 1 · p. 3

3 PRISM-BENCH

PRISM-BENCH targets five under-represented reasoning categories, with 200 manually curated questions in each. Its construction pairs questions with idealized human reasoning traces and uses separate authoring and validation roles.

3.1 Categories

  • PRISM-BENCH contains 1,000 manually curated questions across five categories, targeting the Perception–Cognition Disconnect while excluding saturated chart and math QA categories.The categories probe cases where a model perceives visual elements but fails to build a coherent world model.Source: §3.1 · p. 3
  • The five 200-item categories cover visual coreference and occlusion, handwriting and OCR, navigation and maps, multi-hop reasoning, and scientific or medical reasoning.Their demands include resolving partially occluded targets, reasoning over handwritten text, composing spatial instructions, chaining clues across image regions, and interpreting medical cues with domain causal knowledge.Source: §3.1 · p. 3
  • Each item includes a ground-truth answer, an idealized human trace, and multi-label reasoning tags; the trace is stored as both a free-form chain and a structured step-by-step rendering.The free-form traces average 43.8 words and the structured renderings 70.1 words; CORE uses the structured form’s length reference, while trace alignment compares against the original.Source: §3.1 · p. 4

3.2 Construction

  • A four-stage construction pipeline separates question design, trace authoring, tag assignment, and validation, with no annotator both authoring and validating the same item.Images come from RefCOCO, schematic layouts derived from OpenStreetMap, handwriting scans and IAM, and MedXpertQA [27]; each item requires 20–30 minutes of annotation effort.Source: §3.2 · p. 4
  • Question, option, answer, and trace text is newly authored for 807 of 1,000 items (80.7%); the 193 medical items reuse an expert-validated image, question, and answer.For those medical items, the trace is authored to the verified answer.Source: §3.2 · p. 4

4 Cost-Aware Metrics

CORE evaluates correctness relative to reasoning cost, using human trace length as a per-item reference. Its critic-augmented variant adds process-reward scores, while the critic-free CORE is the primary metric.

4 Cost-Aware Metrics

  • CORE scores correctness per unit of reasoning cost, using human trace length as the per-item reference.When generated length matches the reference, the denominator is 1 and CORE reduces to accuracy.Source: §4 · p. 4
  • CORE normalizes in log space because length distributions are heavy-tailed, preventing a single verbose response from dominating the average.The human reference is an efficiency floor, not a claim that it is the only valid reasoning path; the log ratio tolerates moderately different-length correct derivations.Source: §4 · p. 4
  • CORE-PRM incorporates step-level process-reward scores to discount correct-but-ungrounded answers.Source: §4 · p. 4
  • CORE is critic-free and serves as the primary metric; CORE-PRM is a critic-agnostic template whose validity is bounded by the supplied judge.Source: §4 · p. 4

5 Experiments

Models generate answers with step-by-step rationales under a shared prompt and answer-extraction procedure. Across the experiments, performance remains far below humans, while grounding, trace alignment, reasoning cost, and protocol reliability reveal distinctions that accuracy alone misses.

5.1 Frontier Models Remain Far Below Humans

  • 66% is the overall accuracy ceiling versus a human average of 83.9%; across four shared categories, the best system reaches 65.5%, 18.4 points behind humans, and navigation is near 47% versus 86% for humans.Table 2 reports accuracy across the 25 systems and human baseline.Source: §5.1 · pp. 4–5, Table 2 · p. 5
  • Recent frontier gains concentrate on OCR, while navigation remains the weakest category and spatial grounding has not moved; category specialization can matter more than model scale.The newest frontier systems have their strongest scores on OCR, while Ovis2.5-9B outperforms several 30B+ systems on navigation.Source: §5.1 · p. 5

5.2 Grounding: Most Failures Are Perceptual

  • About 85% of frontier failures are attributed to perception, and answering from a model’s own description lowers accuracy by 3.7 points for GPT-5 on navigation and 11.5 points for Opus on coreference.The probe targets navigation and coreference, the categories most dependent on grounding; the accuracy drops are independent of the judge’s failure labels.Source: §5.2 · pp. 5–6, Table 3 · p. 6

5.3 Reference-Trace Alignment Predicts Correctness

  • Trace alignment distinguishes correct from incorrect responses for all 25 systems: macro AUROC is 0.66, or 0.665 after response-length control.The association does not establish that a model’s stated reasoning produced its answer; alignment is highest for OCR and lowest for multi-hop.Source: §5.3 · p. 6, Table 4 · p. 7

5.4 Deployment Cost: Longer Reasoning Does Not Pay

  • 24.2% of model pairs (46 of 190) are ranked differently by CORE and accuracy, while CORE is uncorrelated with normalized response length (ρ = −0.050, p = 0.834).CORE is reported for the 20 open-weight models with exact token counts; closed systems whose thinking tokens are hidden are excluded.Source: Table 5 · p. 7, §5.4 · pp. 7–8
  • Reflective phrases do not ensure higher accuracy: Qwen3-VL-8B-Thinking uses 1,812 tokens for 36.0% accuracy, versus 737 tokens and 40.0% for its instruct counterpart.GLM-4.1V-9B-Thinking averages one self-correction every 39 tokens and has the lowest accuracy in the table.Source: Table 6 · p. 8, §5.4 · p. 8

5.5 Reliability of Our Own Protocol

  • Accuracy varied by at most 4.8 points from each frontier system’s mean across prompt and decoding variants, but GPT-5 and Gemini exchanged first place while Claude remained last.The ranking audit covered three systems, four variants, and a 150-item stratified subset.Source: §5.5 · p. 8
  • VisualPRM-8B weakly separates correct from incorrect reasoning, so CORE-PRM is a critic-agnostic template rather than a settled score; the public-source contamination check found the most-reused images hardest.The critic’s mean vi was 0.608 versus 0.596, with negative gaps in multi-hop and coreference/occlusion; handwriting scans were easiest in the contamination check.Source: §5.5 · pp. 8–9

6 Failure Modes

Failures differ by category, from visual misrecognition and spatial errors to fabricated constraints and medical causal attributions.

6 Failure Modes

  • Coreference models over-attend to salient objects and mishandle relative spatial phrases.They must resolve partially occluded targets among distractors using object identity and spatial logic.Source: §6 · p. 9
  • In handwriting and OCR, one misrecognized character can cascade through downstream arithmetic and corrupt an otherwise sound chain.Source: §6 · p. 9
  • Navigation models identify landmarks correctly but fail at path composition, producing off-by-one directional errors and left-right confusions at intersections.Source: §6 · p. 9
  • In multi-hop reasoning, the first hop usually succeeds, while the second introduces fabricated constraints.Source: §6 · p. 9
  • Medical models often perceive the visual feature correctly yet hallucinate a causal attribution; this category has the highest rate of confidently wrong explanations.The passage identifies it as the clearest safety concern for high-stakes deployment.Source: §6 · p. 9

7 Conclusion

PRISM-BENCH evaluates grounding, faithfulness, and cost, using its reasoning traces to support diagnostics beyond accuracy. Its probe points to grounding failures, while trace alignment predicts correctness across systems.

7 Conclusion

  • A perception probe attributes most frontier failures to grounding rather than inference, while length-controlled trace alignment predicts correctness across 25 systems.The conclusion also reports that recent frontier gains concentrate in recognition rather than spatial grounding, and that extra reasoning tokens do not repair a mis-seen scene.Source: §7 · p. 9

A Perception–Reasoning Probe Protocol

  • The probe compares image-based answers with answers from the model’s own question-conditioned description, then attributes baseline failures as perceptual or inferential.The protocol uses both a judge-based attribution step and a judge-free accuracy comparison; their agreement supports reporting approximately 85%, not a single exact figure.Source: A Perception–Reasoning Probe Protocol · p. 12
  • The probe covers navigation and coreference for two frontier systems; its approximately 85% finding is not claimed to transfer to OCR or medical items.Handwriting is excluded from that generalization because perception and inference are entangled by its documented cascade pattern.Source: A Perception–Reasoning Probe Protocol · p. 12

B Trace Alignment Protocol

  • Trace alignment scores coverage of each gold reasoning step by the most similar response step, then averages those maxima rather than requiring verbatim matches.A length-controlled variant removes response length from the score before recomputing AUROC.Source: B Trace Alignment Protocol · p. 12
  • Alignment is higher for correct than incorrect responses in all 25 systems, with positive gaps ranging from +0.0177 to +0.0534.The gap carries the signal because absolute similarities are high for both outcomes.Source: B Trace Alignment Protocol · p. 12

C Ranking Stability Protocol

The stability check tests whether accuracy and leaderboard order hold under four benign prompt and decoding choices. Accuracy varied little, but the top-ranked model changed.

C Ranking Stability Protocol

  • Three frontier systems were tested under four defensible prompt and decoding conditions on a 150-item stratified subset, yielding 1,799 valid runs with no errors.The conditions were the default prompt, a terser prompt, an explicit chain-of-thought instruction, and the default prompt at raised temperature.Source: C Ranking Stability Protocol · pp. 12–13
  • Per-model accuracy deviated by at most 4.8 points from its own mean, but the top-1 ranking changed across conditions.GPT-5 led under P1, Gemini 2.5 Pro under the other three, and Claude Opus 4.6 ranked last in all four.Source: C Ranking Stability Protocol · p. 13
  • Adjacent positions among closely matched systems are not identified by accuracy alone on this benchmark.Table 7 concerns trace alignment, not ranking stability, so it does not establish this protocol’s ranking result.Source: C Ranking Stability Protocol · p. 13

D Critic Validation Protocol

The critic validation finds weak overall separation between correct and incorrect reasoning, limiting how CORE-PRM should be interpreted.

D Critic Validation Protocol

  • The off-the-shelf VisualPRM-8B critic separates correct from incorrect reasoning by only +0.012 overall (0.608 versus 0.596).The gap is negative in multi-hop (−0.037) and coreference/occlusion (−0.056).Source: D Critic Validation Protocol · p. 13
  • CORE-PRM’s validity depends on the supplied judge, and the off-the-shelf 8B critic is weak on unseen domains.Source: D Critic Validation Protocol · p. 13
  • CORE is critic-free and carries every headline result, whereas CORE-PRM is a critic-agnostic template rather than a settled score.A stronger multimodal critic could improve CORE-PRM without changing the benchmark.Source: D Critic Validation Protocol · p. 13

E Category Failure Modes

The five categories exhibit distinct deployment-relevant failure signatures, from visual misidentification and shortcutting to unsupported reasoning and confident medical misattribution. Qualitative examples pair images and ground-truth rationales with verbatim model outputs and token counts.

E Category Failure Modes

  • In visual coreference, weaker systems select the most salient entity or invert figure and ground instead of applying the referring expression.Gemini 2.5 Pro enumerates aircraft and matches the spatial constraint, while three other systems return the foreground plane’s registration.Source: E Category Failure Modes · pp. 13–14, Figure 4 · p. 15
  • In handwriting/OCR, one misrecognized character can propagate through valid downstream reasoning, leaving a confidently wrong but internally consistent answer.In the example, Gemini misreads one word and argues for “based”; the trace’s apparent coherence obscures the perception error.Source: E Category Failure Modes · p. 14, Figure 5 · p. 15
  • Navigation failures include directional-step and left/right errors, as well as inferring orientation from street names rather than map layout.The spatial-orientation example shows systems using “Court” versus “Avenue” despite an explicit instruction to ignore names.Source: E Category Failure Modes · p. 14, Figure 9 · p. 17
  • Multi-hop reasoning often succeeds at the first clue but fabricates a later constraint, and longer chains worsen this pattern.In the Wii example, o3 counts three players correctly but carries that image-count error into the controller count.Source: E Category Failure Modes · p. 14, Figure 7 · p. 16
  • Medical reasoning has the highest rate of confidently wrong explanations and is identified as the clearest safety concern for high-stakes deployment.Models may perceive the visual feature correctly yet hallucinate a plausible, incorrect causal attribution.Source: E Category Failure Modes · p. 14

F Qualitative Examples

  • The figures compare images, options, ground-truth rationales, and verbatim outputs from four systems, with generated token counts.They provide item-level examples for assessing the mechanisms behind aggregate distributions.Source: F Qualitative Examples · p. 14
  • The qualitative figures show that longer, more structured reasoning is not necessarily correct or cheaper than a concise answer.For the map-reading item, Gemma answers correctly in 74 tokens, while three longer responses are incorrect.Source: Figure 6 · p. 16
  • In the medical example, only o3 identifies Beau’s lines; other systems give confident but incorrect clinical explanations.The figure links Beau’s lines to repeated nail-matrix growth arrest and cyclical chemotherapy.Source: Figure 8 · p. 17

G Benchmark Construction

PRISM-BENCH combines five reasoning categories with a staged annotation process designed to produce independently validated answers and idealized reasoning traces. The construction also reports how much item text is newly authored and checks whether source familiarity tracks difficulty.

G Benchmark Construction

  • The benchmark contains 1,000 items, with 200 per category, drawn from sources including RefCOCO, OpenStreetMap-derived layouts, handwriting scans, IAM, and MedXpertQA.The categories cover coreference, navigation, OCR, multi-hop reasoning, and medical questions.Source: G Benchmark Construction · p. 18
  • Three annotators work across four stages, with independent validation; failed items return for revision and are revalidated or removed.Each item takes 20–30 minutes of annotation effort.Source: G Benchmark Construction · p. 18
  • Every trace is an expert-audited idealized minimal chain: the shortest logically complete path to the answer, usable as an efficiency floor.Any language-model-suggested draft is mandatorily audited and edited by an expert before acceptance.Source: G Benchmark Construction · p. 18
  • 807 of 1,000 items (80.7%) have newly authored question, option, answer, and trace text; internal question overlap is low.The mean and median maximum pairwise 5-gram Jaccard similarities are 0.21 and 0.11.Source: G Benchmark Construction · p. 18
  • A behavioral check finds the most-reused image sources are hardest, while handwriting scans are easiest, so difficulty tracks reasoning demands rather than pixel familiarity.The passage reports scores of 46.8/47.8 on OpenStreetMap schematics for GPT-5 and Opus 4.6, versus 82.1/83.6 on handwriting scans.Source: G Benchmark Construction · p. 18
Loading 2610.36554v1…