PaperlayerView exact revision on arXiv

Source-linked AI summary

PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies

Lukas Selch, Yufang Hou, M. Jehanzeb Mirza, Sivan Doveh, James Glass, Rogerio Feris, Wei Lin

arXiv:2510.16505v3cs.CV

TL;DR

It remains unclear whether LMMs can reliably reason across the text, figures, tables, and equations of scientific papers, while existing benchmarks often isolate modalities or use synthetic errors. PRISMM-Bench draws on reviewer-flagged inconsistencies to build three evaluation tasks and structured JSON answer formats. The benchmark shows that even top-performing models struggle with cross-modal reasoning and long-context grounding.

  • Problem

    Existing benchmarks often isolate scientific modalities or use synthetic errors, leaving limited evaluation of real inconsistencies across scientific papers’ text, figures, tables, and equations.

    Source: §1 · p. 2

  • Method

    PRISMM-Bench mines peer reviews, filters and verifies flagged cases, builds identification, remedy, and pair-match tasks, and uses structured JSON answers to mitigate linguistic shortcuts.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 6

  • Results

    Even top-performing models struggle with cross-modal reasoning and long-context grounding; GPT-5 (high reasoning) reached the highest average performance of 53.9%.

    Source: §4.2 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    The findings motivate improvements in filtering pipelines, cross-domain datasets, and debiasing strategies for long-form multiple-choice evaluation.

    Source: §5 · p. 10

    The benchmark focuses on AI-domain papers from ICLR 2024 and 2025 and emphasizes rejected submissions, restricting its scale and domain coverage.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large Multimodal Models (LMMs) are increasingly applied to scientific research, yet it remains unclear whether they can reliably understand and reason over the multimodal complexity of papers. A central challenge lies in detecting and resolving inconsistencies across text, figures, tables, and equations, issues that are often subtle, domain-specific, and ultimately undermine clarity, reproducibility, and trust. Existing benchmarks overlook this issue, either isolating single modalities or relying on synthetic errors that fail to capture real-world complexity. We introduce PRISMM-Bench (Peer-Review-sourced Inconsistency Set for Multimodal Models), the first benchmark grounded in real reviewer-flagged inconsistencies in scientific papers. Through a multi-stage pipeline of review mining, LLM-assisted filtering and human verification, we curate 384 inconsistencies from 353 papers. Based on this set, we design three tasks, namely inconsistency identification, remedy and pair matching, which assess a model's capacity to detect, correct, and reason over inconsistencies across different modalities. Furthermore, to address the notorious problem of choice-only shortcuts in multiple-choice evaluation, where models exploit answer patterns without truly understanding the question, we further introduce structured JSON-based answer representations that minimize linguistic biases by reducing reliance on superficial stylistic cues. We benchmark 21 leading LMMs, including large open-weight models (GLM-4.5V 106B, InternVL3 78B) and proprietary models (Gemini 2.5 Pro, GPT-5 with high reasoning). Results reveal strikingly low performance (27.8-53.9\%), underscoring the challenge of multimodal scientific reasoning and motivating progress towards trustworthy scientific assistants.

1 INTRODUCTION

The paper asks whether LMMs can reason over scientific papers’ multimodal structure, where subtle cross-modal inconsistencies can undermine clarity, reproducibility, and trust. It introduces PRISMM-Bench, a peer-review-grounded benchmark with three tasks and structured JSON answer formats, and reports that current models still perform modestly.

1 INTRODUCTION

  • Existing document QA and standalone visual-element benchmarks miss scholarly multimodal dependencies, while synthetic errors often fail to represent real-world complexity.Open peer reviews provide a source of authentic, human-identified cross-modal inconsistencies.Source: §1 · p. 2
  • PRISMM-Bench curates 384 reviewer-flagged inconsistencies across 353 ICLR 2024 and 2025 papers and evaluates identification, remedy, and pair matching.The benchmark spans 15 categories of visual-textual and inter-visual mismatches.Source: §1 · p. 2
  • Structured JSON answer representations reduce sensitivity to stylistic cues, and a user study found they suppress linguistic shortcuts and better align evaluation with human reasoning.The method converts free-form natural-language answers into uniform structured representations.Source: §1 · p. 2
  • Current LMMs remain limited on the benchmark: even the strongest proprietary models reach only 53.9% accuracy, while large open-weight models achieve around 40%.The authors benchmarked 21 state-of-the-art models.Source: §1 · p. 2

2 RELATED WORK

Prior resources often isolate visual elements, use text-only or abstract-based questions, or evaluate synthetic inconsistencies. PRISMM-Bench instead grounds full-paper multimodal inconsistency tasks in authentic reviewer feedback.

2 RELATED WORK

  • Earlier scientific benchmarks often isolate diagrams, charts, or tables, while multimodal datasets may treat figures and equations outside full-paper context.Whole-paper QA resources are mostly abstract-based and use yes/no or short-span answers.Source: §2 · p. 3
  • PRISMM-Bench is presented as the first benchmark grounded in reviewer-flagged multimodal inconsistencies in scientific papers, unlike synthetic inconsistency evaluation in slides and posters.The benchmark spans textual, visual, and cross-modal errors and includes remedy tasks.Source: §2 · pp. 3–4
  • QASA offers 1.8K expert-written questions on ML papers but remains text-only and does not require reasoning over figures or tables.SciDQA uses reviewer–author QA pairs but remains a text-only LLM benchmark.Source: §2 · p. 3
  • Multiple-choice benchmarks can reward answer-choice shortcuts, including 83% accuracy on TruthfulQA v2 using choices alone and shortcut rates above 70% on HellaSwag.LLM-generated distractors can exacerbate such artifacts.Source: §2 · p. 4
  • PRISMM-Bench uses JSON answer representations to encode key elements of scientific inconsistencies and reduce surface-level answer-choice artifacts.Its structured formats are intended to make models engage with multimodal content rather than exploit linguistic shortcuts.Source: §2 · p. 4

3 PRISMM-BENCH

PRISMM-Bench builds a benchmark from reviewer-flagged inconsistencies, generates tasks to test detection, correction, and visual-visual matching, and applies structured answers to reduce linguistic shortcuts. Evaluation varies how much paper context models receive, from focused excerpts to whole-document page images.

3.1 COLLECTION OF REVIEWER-FLAGGED INCONSISTENCIES

  • The collection pipeline sources ICLR 2024 and 2025 reviews, filters them with an LLM, then manually verifies each case and annotates the relevant text or visual regions.The 18,009 reviews came from rejected or withdrawn submissions without rebuttals, a restriction motivated by reviewer-flagged errors often being resolved during rebuttal.Source: §3.1 · p. 4
  • 384 validated inconsistencies were collected across 353 ICLR submissions, with intra-figure cases (23.7%) and figure-text mismatches (21.9%) most common.The annotation identified 15 inconsistency categories.Source: §3.1 · p. 5

3.2 GENERATION OF BENCHMARK TASKS

  • The benchmark’s three multiple-choice tasks test inconsistency identification, actionable remedy selection, and pair matching between two visual elements; the pair-match subset has 192 samples.Pair matching withholds textual cues to assess visual-visual mismatch detection.Source: §3.2 · p. 6

3.3 ALLEVIATION OF LINGUISTIC BIASES

  • Gemini 2.5 Flash’s no-context Ident accuracy fell from 57.6% to 34.0% after answers were converted to structured JSON formats.The formats are Evidence–Claim for identification and Target–Action for remedy; a 20% subset was manually checked for semantic fidelity.Source: §3.3 · p. 6

3.4 CONTEXTUAL GRANULARITY IN EVALUATION

  • Evaluation provides focused excerpts, full-page images at 144 DPI, or document context supplied as page-image collages for long-range cross-page reasoning.Document context typically uses five images, each arranging npages/5 pages in a three-column grid; some models receive three images instead.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

The experiments compare 21 LMMs across inconsistency tasks and context scopes, finding performance gaps by model, task, and context. CoT and structured JSON answers expose reasoning benefits and linguistic shortcuts, while models remain limited in long-context grounding and correction.

4.1 EXPERIMENTAL SETUP

  • The evaluation covered 21 LMMs across Ident, Remedy, and Match and three context granularities, with Match evaluated only in Focused context.The model set included 16 open-weight models, two high-resolution specialists, and three proprietary models.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • GPT-5 (high reasoning) reached 53.9% average performance, versus 42.5% for the best open-weight model GLM 4.5V 106B.The 11.4-point gap underscores that even the top model remains far from the reliability required of automated scientific assistants.Source: §4.2 · p. 7, Table 1 · p. 8
  • Models performed best in Focused context and often approached random chance as inputs expanded to Page or Document; Remedy scores were lower than Ident, while Match varied from 63.5% for Gemma 3 12B to 45.3% for InternVL3 78B.The authors report a 144 DPI rasterization baseline in an ablation study.Source: §4.2 · pp. 7–8

4.3 ANALYSIS OF CHAIN-OF-THOUGHT REASONING

  • Reasoning-enabled models improved over non-reasoning counterparts, while disabling CoT reduced accuracy by up to 25.7% and CoT traces showed structured-option interpretation, cross-modal grounding, and concept linking.InternVL3.5 8B averaged 37.7%, comparable to large open-weight models and above several 72B non-reasoning models.Source: §4.3 · pp. 8–9, §4.2 · p. 8

4.4 USER STUDY AND LINGUISTIC BIAS ANALYSIS

  • In the user study, Gemini 2.5 Pro exceeded human accuracy in Focused context (81.6% vs. 77.5%) and Whole Document context (83.8% vs. 65.0%), but models reached up to 70.1% without context as human performance fell to 27.5%.JSON formatting reduced shortcut advantage; InternVL3.5 38B fell from 53.7% to 25.3% without context, and human visual reliance (R = 69.0%) exceeded InternVL3.5 8B's (R = 53.5%).Source: §4.4 · pp. 9–10

5 CONCLUSIONS

PRISMM-Bench evaluates LMMs on real-world scientific inconsistencies, and the reported findings show that top models struggle with cross-modal reasoning and long-context grounding. Its scope is currently limited to AI-domain papers from ICLR 2024 and 2025, especially rejected submissions.

5 CONCLUSIONS

  • PRISMM-Bench evaluates LMMs on real-world scientific inconsistencies, and top-performing models struggle with cross-modal reasoning and long-context grounding.The paper also reports that structured answer formats mitigate linguistic shortcuts.Source: §5 · p. 10
  • The authors call for future improvements in filtering pipelines, cross-domain datasets, and debiasing strategies for long-form multiple-choice evaluation.Source: §5 · p. 10
  • The benchmark focuses on AI-domain papers from ICLR 2024 and 2025 and emphasizes rejected submissions to capture unresolved errors.The authors identify restricted scale and domain coverage as consequences of this scope.Source: §5 · p. 10

6 ETHICS STATEMENT

The paper describes its use of publicly available ICLR materials and a small-scale researcher user study, alongside ethical aims for the benchmark. Its appendix outlines qualitative examples, dataset and asset information, and methodological details.

6 ETHICS STATEMENT

  • The study uses publicly available ICLR 2024 and 2025 papers under CC-BY 4.0, and its user study involved consenting experienced researchers without collecting personally identifying information.The authors report no foreseeable participant risks or vulnerable populations.Source: §6 · p. 11
  • The authors frame PRISMM-Bench as supporting transparent, reproducible, ethical scientific-document analysis rather than model misuse.Source: §6 · p. 11

APPENDIX

  • The appendix describes qualitative examples, asset and licensing inventories, ablations, dataset construction, task-question generation, user-study implementation, prompts, and application screenshots.Source: APPENDIX · p. 17

A QUALITATIVE EXAMPLES

  • The qualitative examples include a text–table inconsistency and a figure–equation inconsistency, each paired with evaluation tasks.Source: A QUALITATIVE EXAMPLES · p. 17

B LIST OF ASSETS

  • The authors state that images and annotations come from public datasets and are distributed under their original sources' licensing terms.Source: B LIST OF ASSETS · p. 17

C.1 IMPACT OF RASTERIZATION RESOLUTION

Rasterization resolution affects inconsistency-identification accuracy, but higher DPI does not consistently help. The authors identify 144 DPI as a practical balance of visual clarity, computational cost, and cross-model comparability.

C.1 IMPACT OF RASTERIZATION RESOLUTION

  • The study varied PDF image-extraction DPI while holding other settings fixed, evaluating representative models on Ident with Focused Context.Table 4 reports accuracy and percentage change relative to the 144 DPI baseline.Source: §C.1 · p. 18, Table 4 · p. 18
  • Resolution sensitivity varies by model: most models dropped at 72 DPI, while increasing from 144 to 300 DPI produced mixed outcomes.InternVL3.5 38B fell by up to -20.1% at 72 DPI, whereas Gemini 2.5 Pro declined by -3.3%; at higher resolution, Gemini 2.5 Pro and Ovis2 34B benefited slightly, but InternVL models performed worse.Source: §C.1 · p. 18
  • High-resolution training did not guarantee better results, and the assumption that higher resolution universally improves detection did not hold.VILA HD showed a slight accuracy gain at 72 DPI but failed to leverage the higher fidelity at 300 DPI.Source: §C.1 · p. 18

C.2 CASE STUDY: CHAIN-OF-THOUGHT REASONING

A case study contrasts a non-reasoning response that misdiagnosed a figure with a reasoning-enabled response that identified an error bar violating the metric’s non-negativity. The example illustrates systematic distractor elimination and visual grounding with domain knowledge.

C.2 CASE STUDY: CHAIN-OF-THOUGHT REASONING

  • Without reasoning, InternVL3.5 38B chose a label-overlap distractor despite the labels being legible, relying on a generic critique rather than the visual evidence.The figure places visual context on the right and the Ident question and answer options on the left; natural-language options are shown for ease of comprehension, while the model was tasked using JSON.Source: §C.2 · p. 19, Figure 5 · p. 19
  • Reasoning-enabled InternVL3.5 38B identified error bars below zero as invalid because Unique Successful Jailbreaks is a non-negative count metric.It first ruled out label overlap and a y-axis-range issue, then dismissed legend critiques before applying the metric’s non-negativity.Source: §C.2 · pp. 18–19
  • The case illustrates reasoning through systematic distractor elimination and integration of domain knowledge with visual grounding.The authors report an average of 473 tokens per query and say reasoning substantially improves multimodal consistency and robustness in this case.Source: §C.2 · p. 19

C.3 VALIDITY OF MULTIPLE-CHOICE VS. OPEN-ENDED EVALUATION

The appendix motivates multiple-choice evaluation as more reproducible than open-ended judging, then documents the review-sourcing, annotation, collection statistics, and Ident-task construction behind PRISMM-Bench.

C.3 VALIDITY OF MULTIPLE-CHOICE VS. OPEN-ENDED EVALUATION

  • Open-ended judging produced inconsistent scores and rankings, nondeterministic results for some proprietary reasoning models, and judge-model similarity bias; the authors conclude MCQ offers more reproducible scoring.Gemini 2.5 Pro and GPT-5 shifted some model rankings by two to three positions; the MCQ framework uses controlled distractors and avoids dependence on third-party proprietary judges.Source: §C.3 · pp. 19–20

D.1 REVIEW SOURCING

  • Regex screening of ICLR reviews missed indirect inconsistency descriptions, while rebuttal-updated papers sometimes made the original errors unlocatable in available PDFs.The refined strategy focused on papers without author rebuttals and used Mistral Nemo 2407 to summarize reviews and extract candidate statements, reducing 120,329 reviews to 18,009.Source: §D.1 · p. 21

D.2 ANNOTATION PROCESS

  • Manual verification retained objective, factual contradictions whose two parts were locatable in the PDF and identifiable without deep domain expertise.Annotators recorded textual or visual parts, assigned a category, and wrote a short description; ambiguous cases were discussed with senior researchers until consensus.Source: §D.2 · pp. 21–22

D.3 STATISTICS OF INCONSISTENCY COLLECTION.

  • The collection contains 384 inconsistencies from 353 ICLR papers across 15 categories, with figure–text mismatches and intra-figure inconsistencies most common.Twenty-nine papers (7.6%) had more than one inconsistency.Source: §D.3 · p. 22

D.4.1 INCONSISTENCY IDENTIFICATION (IDENT)

  • For Ident, Gemini Flash generated four-option questions from annotated context and descriptions, followed by manual checks of correct answers and distractors.Verification required the correct answer to reflect the annotator’s description and connect to context; distractors had to be grounded in the inconsistency parts and avoid unsupported elements.Source: §D.4.1 · p. 22

D.4.2 DEBIASING THE INCONSISTENCY IDENTIFICATION TASK

The appendix shows that natural-language answer choices permit substantial no-context accuracy, then introduces evidence–claim JSON to reduce stylistic cues. It also describes a structured target–action format for remedy questions.

D.4.2 DEBIASING THE INCONSISTENCY IDENTIFICATION TASK

  • No-context Ident accuracy was 57.6%, well above the 25% random baseline, and remained 34.0% after adopting evidence–claim JSON.The schema represents a claim and its source alongside supporting evidence, preserving the semantic contradiction while removing stylistic cues.Source: §D.4.2 · pp. 23–24
  • Rephrasing distractors according to MCQ design best practices reduced no-context accuracy to 41.6%, compared with 48.1% after answer-length normalization and 46.2% when the correct answer was shortest.Shortening all options into nominal style yielded 38.2%; the authors report that these interventions reduced but did not eliminate bias.Source: §D.4.2 · p. 23
  • With full context, accuracy fell from 79.4% to 69.5% under the JSON format, while visual-grounding-attributable performance rose from 51.4% to 53.8%.The authors describe the structured format as a regularizer that increases reliance on the provided paper context.Source: §D.4.2 · p. 24

D.4.3 INCONSISTENCY REMEDY TASK

  • The Remedy task uses Target–Action JSON to specify the element at issue, where a change applies, and the proposed correction and reason.The action field can encode modify, remove, add, reposition, or replace.Source: §D.4.3 · p. 24

D.4.4 INCONSISTENCY PAIR-MATCH TASK

The pair-match task asks models to connect one text or visual element with its corresponding inconsistent visual element. Its eligible cases and distractors are selected to require fine-grained matching rather than easy modality-based elimination.

D.4.4 INCONSISTENCY PAIR-MATCH TASK

  • Pair-Match presents one text or visual element as context and asks the model to identify the corresponding inconsistent visual element among four options.The task covers inconsistencies involving two distinct visual parts.Source: §D.4.4 · p. 24
  • Only 135 of 262 inconsistencies were eligible after excluding contradictions contained within a single element.Excluded categories included figure–caption, figure-only, table-only, table–caption, and algorithm-only cases.Source: §D.4.4 · p. 24
  • Distractors were sampled from the same paper and modality, with preference for elements on the same or adjacent pages.This procedure reduced trivial elimination strategies and required models to consider fine-grained inconsistencies.Source: §D.4.4 · pp. 24–25
  • In the example, inline text is the context and extracted-figure image IDs are the options, with the annotated crop serving as the correct answer.The output uses image IDs that refer to figures extracted from the paper.Source: §D.4.4 · p. 25

E.1 IMPLEMENTATION OF THE USER STUDY

The appendix describes the user-study setup, its representativeness check, and prompt and interface materials. The check reports that humans outperformed LMMs even on a subset comparatively easy for some models.

E.1 IMPLEMENTATION OF THE USER STUDY

  • Eight PhD-or-higher participants with advanced AI exposure each received ten randomly sampled tasks, split between focused context and the original document.Median answer times were 45 seconds without context, 145 seconds with focused context, and 169 seconds with whole-document context; participants changed answers in 65% of cases and processed 80 inconsistencies.Source: §E.1 · p. 25

E.2 REPRESENTATIVENESS OF THE USER STUDY SUBSET

  • Even on a randomly sampled 40-instance subset that was comparatively easier for two models, humans outperformed LMMs with focused and whole-document context.The simulation placed the subset in the upper quartile of model difficulty for those models; the authors say this likely understates the human–LMM gap.Source: §E.2 · p. 25

F LLM PROMPTS

F.1 LLM PROMPT FOR REVIEW FILTERING

  • The review-filtering prompt directs the model to analyze visual-inconsistency characteristics systematically, using chain-of-thought instructions and few-shot examples.Source: §F.1 · p. 26

F.2 LLM PROMPT FOR CONVERTING INTO EVIDENCE-CLAIM FORMAT

  • Evidence-claim JSON answer options are used for inconsistency identification to mitigate language biases in multiple-choice evaluation.Source: §F.2 · p. 26

F.3 LLM PROMPT FOR CONVERTING INTO TARGET-ACTION FORMAT

  • Target-action JSON answer options for the remedy task are generated from question-answer pairs in the identification task.Source: §F.3 · p. 26

G SCREENSHOTS OF THE ANNOTATION APP

H SCREENSHOTS OF THE SURVEY APP

Related papers

Loading 2510.16505v3…