OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding

• OCR-grounded meta-reasoning is difficult because models must organize recognized words, layout, fields, and visual correspondences, while existing benchmarks do not jointly control reasoning direction, OCR-object category, answer correctness, and process compliance.The required directions are deduction (applying an explicit rule), induction (inferring a latent rule), and abduction (recovering a missing premise).Source: §1 · pp. 1–2
• OCR-MetaReasoning makes OCR-grounded deduction, induction, and abduction the explicit targets of a benchmark designed to separate OCR perception from reasoning.Samples require models to read relevant text and organize image-grounded evidence for the specified reasoning direction.Source: §1 · p. 2
• The benchmark contains 1,500 single-image samples in a balanced 3 × 5 taxonomy, with reference reasoning steps and separate answer-correctness and process-compliance evaluation.Its five OCR-object categories are transaction analysis, data interpretation, field dependency, document logic, and layout semantics; evaluation includes automatic answer scoring and process criteria for capability match, groundedness, step completeness, and non-hallucination, with MRMS and RPCS.Source: §1 · p. 2
• Evaluations of representative closed- and open-source MLLMs identify persistent weaknesses in grounded rule verification and layout-sensitive inference, indicating that OCR-grounded meta-reasoning remains far from saturated.The introduction also notes that process-compliant rationales can accompany incorrect final answers under exact-match evaluation.Source: §1 · p. 2

2.1 OCR Reasoning Benchmarks

• Existing text-rich benchmarks span reading-oriented VQA and structured OCR reasoning, but are organized mainly by domain, skill, or scenario rather than reasoning direction.OCR-MetaReasoning instead requires each text-rich sample to instantiate deduction, induction, or abduction over OCR-grounded evidence.Source: §2.1 · pp. 2–3
• 205/300 LogicOCR-Real, 457/1,069 OCR-Reasoning, and 69/150 Reasoning-OCR items mapped cleanly to the meta-reasoning and OCR-object axes, versus 48/300 OCRBench v2, 52/300 LogicVista, and 0/300 MME-Reasoning items.Rank correlations across the same eight model versions also varied by benchmark: 0.905 on OCR-Reasoning, 0.695 on LogicOCR-Real, and 0.405 on Reasoning-OCR; the external resources lack the complete controlled 3×5 grid together with process-compliance evaluation.Source: §2.1 · p. 3

2.2 Multimodal Reasoning Evaluation

• Multimodal reasoning benchmarks and the Beyond “Aha!” framework address reasoning abilities, but not the evidence structures of OCR-rich images.OCR-MetaReasoning combines OCR-grounded evidence, deduction, induction, and abduction, process-level evaluation, and diverse OCR-object categories.Source: §2.2 · p. 3

3.1 Task Definition

• Deduction applies visible rules, induction abstracts rules from instances, and abduction reasons backward from observed outcomes to recover hidden premises.Labels follow the direction of the target variable requiring the stated reasoning, even when the solution includes other operations such as arithmetic.Source: §3.1 · pp. 3–4

3.2 Benchmark Construction

• The benchmark contains 1,500 samples balanced across three reasoning types and five OCR-object categories, with 100 samples in each type-category combination.The categories are transaction analysis, data interpretation, field dependency, document logic, and layout semantics.Source: §3.2 · p. 4
• Samples are synthesized from screened text-rich image seeds and human-verified for image-grounded evidence, multiple evidence points, a unique answer, a matching reasoning label, and automatic scorability.When direct OCR extraction sufficed, prompts or images were modified to add challenges such as distractors, masked fields, exceptions, nonadjacent evidence, or layout correspondences.Source: §3.2 · pp. 4–5
• Blinded re-annotation found 96.0% raw agreement for reasoning type and 92.7% for OCR-object category; the final benchmark selected 1,500 samples from 2,326 verified pass candidates.Agreement was measured on 300 samples; after adjudication, all 300 satisfied the validity checklist and matched released labels on both axes.Source: §3.2 · p. 5

3.3 Evaluation Protocol

• Evaluation reports answer correctness with MRMS, the macro average across three reasoning directions, separately from RPCS, which scores process direction, groundedness, completeness, and non-hallucination.Answer scoring uses normalized exact match, numeric match, or micro-F1 according to answer type; paraphrased reasoning can receive RPCS credit when it preserves evidence bindings and direction.Source: §3.3 · pp. 5–6
• The RPCS judge achieved Pearson r = 0.934 and criterion macro-F1 = 0.936 against human ratings, while MRMS remains the judge-independent primary leaderboard metric.Human inter-rater agreement was 90.7% raw; a judge rerun had a 3.1% criterion flip rate and 0.040 RPCS standard deviation.Source: §3.3 · p. 6

4.1 Setup

• The evaluation covers representative closed-source and open-source MLLMs, including GPT, Gemini, Claude, Grok, Doubao, Kimi, GLM, and Qwen3-VL variants.The tested Qwen3-VL scales are 8B, 30B-A3B, and 235B-A22B, with both Instruct and Thinking variants.Source: §4.1 · p. 6

4.2 Main Results

• Layout semantics averages 66.9 across models, 13.1 points below the next-lowest category, while data interpretation leads at 83.3.GPT-5.4-Medium leads layout semantics at 77.6, distinguishing layout-sensitive inference from stronger performance on most non-layout categories.Source: §4.2 · p. 6

Deduction Is the Main Reasoning Bottleneck

• Deduction is the main reasoning bottleneck: its across-model average is 73.6, versus 82.5 for induction and 80.1 for abduction.The passage attributes the difficulty to verifying candidates against visible rules or constraints, compared with aggregating repeated evidence or recovering plausible causes.Source: Deduction Is the Main Reasoning… · pp. 6–7
• Qwen3-VL Thinking improves MRMS over Instruct by 15.5 points at 8B, 9.7 at 30B-A3B, and 4.5 at 235B-A22B.At 235B-A22B, Thinking scores 0.3 points lower than Instruct on layout semantics.Source: Deduction Is the Main Reasoning… · p. 7
• Closed-source models lead overall, but strengths vary: Gemini-3.1-Pro-Preview leads every column except layout semantics, and the largest closed/open reasoning gap is 12.3 points for abduction.The smallest reasoning-type gap is 7.9 points for deduction; Claude-Sonnet-4.6 ranks among the top three for document logic and field dependency despite fourth place in MRMS.Source: Deduction Is the Main Reasoning… · p. 7
• The balanced taxonomy exposes different open-source profiles: Qwen3-VL-235B-A22B-Thinking scores 74.8 on deduction and 69.2 on layout semantics, while Kimi-K2.5 leads open-source models in layout semantics at 70.2.Kimi-K2.5 also leads open-source models in transaction analysis (82.8) and abduction (82.5), but lower document-logic and deduction scores limit its MRMS to 79.6.Source: Deduction Is the Main Reasoning… · pp. 7–8

4.2 Main Results

• 89.3 MRMS makes Gemini-3.1-Pro-Preview the leader, ahead of GPT-5.4-Medium (87.7) and Doubao-Seed-2.0-Pro (86.4); closed-source models average 82.9 versus 73.4 for open-source models.The top-two gap is +1.7 [0.1, 3.1], and only 453 items are solved with full score by all 18 models; 214 items have a mean score below 0.5.Source: §4.2 · p. 6, Table 2 · p. 7, Figure 3 · p. 8

Process Compliance Is High but Uneven

• RPCS spans 72.5–96.2 (mean 87.5), with capability match and groundedness averaging 93.6 and 92.2 versus 81.6 and 82.5 for completeness and non-hallucination.Across reasoning types, RPCS is highest for deduction (89.9), followed by abduction (86.6) and induction (86.0); common weaknesses are omitted intermediate evidence and unsupported rules, entities, fields, or values.Source: §5 · p. 8

Human Baseline

• On the same stratified 300-sample subset, PhD students reach 96.0 MRMS (95% CI [93.7, 98.0]) versus 90.5 [87.2, 93.6] for Gemini-3.1-Pro-Preview, a gap of +5.5 [1.6, 9.2].The human result leaves measurable headroom above the strongest model while confirming practical solvability.Source: §5 · p. 8

Process and Outcome Gaps Identify Training Targets

• Closed-source models lead open-source models by 7.3 RPCS points and 9.5 MRMS points, while average RPCS–MRMS gaps are 7.8 versus 10.1.The largest criterion gap is non-hallucination (87.1 versus 76.8), while capability match has the smallest (95.8 versus 90.9); RPCS–MRMS gaps are 16.4 points for deduction, 3.5 for induction, and 6.5 for abduction. The remaining challenge is verifying OCR-grounded details, covering necessary intermediate steps, and avoiding unsupported decisions.Source: §5 · p. 8

Perception and Reasoning Controls

• On 300 controls, MRMS averaged 80.7 for full image, 16.5 question-only, 71.5 OCR-only, 74.4 layout-aware transcript, 81.3 image plus third-party OCR, and 81.7 answer-format oracle.Question-only input is insufficient; layout-aware text recovers part of the transcript loss, while third-party OCR and the format oracle provide only small gains and do not remove deduction and layout gaps.Source: §5 · pp. 8–9

Strong Processes Do Not Guarantee Correct Answers

• RPCS exceeds MRMS for every model, with gaps of +4.9 to +13.5 points (mean +8.8), and rankings can diverge despite high process scores.For example, Doubao-Seed-2.0-Lite has higher RPCS than Doubao-Seed-2.0-Pro (93.3 versus 93.0) but lower MRMS (84.2 versus 86.4).Source: §5 · p. 8, Table 3 · p. 9

Difficulty Stratification

• Difficulty analysis attributes very-hard and hard-item errors mainly to layout binding and exception handling, while easy items rely on template calculations or bounded candidates.Among 34 very-hard items, layout binding (26.5%), exception/boundary/negation handling (23.5%), and implicit induction (17.6%) dominate; among 44 hard qualitative-analysis items, exception handling (27.3%), implicit induction (22.7%), layout binding (20.5%), and hidden backward dependency (18.2%) dominate. Easy-item step completeness/non-hallucination scores are 92.2/95.9, versus 57.2/53.6 on very-hard items; the 45 easy items mainly use template calculations (77.8%) or bounded candidates (22.2%).Source: §5 · p. 9

• Current MLLMs still struggle with grounded rule verification and layout-sensitive inference, with performance differing across reasoning types and OCR-object categories.Source: §6 · p. 9
• OCR-MetaReasoning adapts deduction, induction, and abduction to OCR-centered document, chart, transaction, form, and layout reasoning tasks.Source: §6 · p. 9
• Its unified protocol measures final-answer correctness and reasoning-process compliance in single-image settings, while future work could address multi-page evidence chains and training for more faithful, direction-aware OCR reasoning.The benchmark uses judge-assisted process evaluation.Source: §6 · p. 9

Limitations

• The benchmark isolates single-image OCR-grounded reasoning and excludes multi-page documents, cross-image evidence, retrieval-augmented reasoning, and interactive clarification.Its results should therefore be interpreted within single-image, text-rich reasoning settings.Source: Limitations · p. 10
• Equal coverage across reasoning types and OCR-object categories enables controlled comparison but does not represent natural task frequencies.Controlled edited or reconstructed images and MLLM-assisted synthesis may also differ from fully natural documents; labels mark dominant bottlenecks despite possible local arithmetic, lookup, comparison, or extraction steps.Source: Limitations · p. 10
• RPCS evaluates visible rationales rather than internal reasoning and cannot fully rule out plausible post-hoc explanations.Proposed extensions include multi-page evidence chains, natural-distribution test sets, compositional annotations, and stronger verification linking intermediate steps to OCR evidence.Source: Limitations · p. 10

A Analysis of Meta-Reasoning Types

• OCR evidence is structured visual information—not merely a transcript—with words, layout, fields, and charts each contributing distinct evidence for reasoning.Source: A Analysis of Meta-Reasoning Types · p. 13
• Deduction applies visible rules to image-grounded cases, where errors can arise from copying salient text without checking all relevant conditions.Source: A Analysis of Meta-Reasoning Types · p. 13
• Induction infers hidden regularities from aligned observations, while shallow field matching is a common source of mistakes.Source: A Analysis of Meta-Reasoning Types · p. 13
• Abduction works backward from an observed result to recover a missing premise, risking explanations that are plausible but insufficiently image-grounded.The benchmark evaluates final-answer correctness and process compliance because the same answer can arise from different ways of organizing visual evidence; Table 4 compares no-hint and task-specific-hint settings across six shared models.Source: A Analysis of Meta-Reasoning Types · p. 13, Table 4 · p. 13

B Effect of Reasoning Hints

• Generic hints ask models to think step by step and verify image evidence; task-specific hints guide abductive backward reasoning, deductive rule-checking, or inductive pattern inference.The type-specific instructions include identifying hidden constraints and eliminating alternatives for abduction, checking boundaries, units, legends, and exceptions for deduction, and testing a contrasting case for induction.Source: B Effect of Reasoning Hints · p. 13
• 70.08 to 71.09: task-specific hints raise average final-answer score by 1.01 percentage points (paired 95% CI [+0.22, +1.78]) across six models.The full-score rate rises from 68.18 to 69.10 (+0.92 points); gains are +2.18 for abduction, +0.91 for deduction, and -0.06 for induction, across 9,000 paired model-sample evaluations.Source: B Effect of Reasoning Hints · p. 13
• 59.03 to 62.95: generic evidence-checking raises Qwen3-VL-8B-Instruct's score by 3.91 points; task-specific hints score 63.35, just 0.40 points higher.The generic-control 95% CI is [+1.96, +5.91], while the task-specific hint's margin over the generic control has 95% CI [-1.57, +2.38].Source: B Effect of Reasoning Hints · pp. 13–14, Table 5 · p. 14
• The results suggest that general evidence-checking guidance accounts for much of the improvement, while hints improve answer-level performance rather than directly demonstrating better reasoning-process quality.The authors interpret the generic-control comparison as evidence that task-specific wording alone is not responsible for most of the improvement.Source: B Effect of Reasoning Hints · p. 14
• -2.19 points: Grok-4.20's score falls under task-specific hints, while Qwen3-VL-8B-Thinking falls by 0.56 points; RPCS also does not improve.Inspected errors include excessive deliberation, incorrect candidate filtering, OCR misreading, and answer-granularity drift; the declines mainly affect data-interpretation and integer-answer cases for Grok-4.20 and exact-match string answers for Qwen3-VL-8B-Thinking.Source: B Effect of Reasoning Hints · p. 14

C Experimental Setup

• Proprietary and open-source models were evaluated using published-configuration decoding settings, including distinct temperature and sampling parameters for the listed model series.Qwen3-VL-Instruct used T = 0.7, top_p = 0.8, top_k = 20; Qwen3-VL-Thinking used T = 1.0, top_p = 0.05, top_k = 20; Kimi-K2.5-Instant used T = 1.0, top_p = 0.95; GLM-4.6V used T = 0.8, top_p = 0.6, top_k = 2.Source: C Experimental Setup · p. 14

D Dataset Statistics

• Table 6 reports benchmark composition by meta-reasoning type, OCR-object category, and sample complexity.The statistics follow the two-axis design shown in Figure 2.Source: Table 6 · p. 14

D.1 Meta-Reasoning Taxonomy and OCR-Object Distribution

• The benchmark contains 1,500 samples, with 500 each for deduction, induction, and abduction, and 100 in each of 15 reasoning-type-by-category cells.Its five categories each contain 300 samples, balancing reasoning direction against visual document genre.Source: §D.1 · p. 14

D.2 Answer Formats and Scoring Diversity

• Integer and string answers predominate, while sample-level scoring uses numeric matching, normalized exact match, or JSON micro-F1.The dataset has 561 integer, 513 string, 280 JSON, and 146 floating-point answers; scoring counts are 707 numeric matching, 513 normalized exact match, and 280 JSON micro-F1.Source: §D.2 · p. 15

D.3 Sample Complexity

• Each single-image sample averages 75.77 question tokens and 4.22 reference reasoning steps, focusing on structured reasoning rather than direct OCR span extraction.Questions range from 21 to 201 tokens, while reference solutions contain 2 to 8 steps.Source: §D.3 · p. 15

E Benchmark Construction

• Each annotated instance includes an image, question, canonical answer, format, scoring rule, taxonomy labels, and reference reasoning path, checked for completeness, compatibility, validity, uniqueness, and taxonomy fit.Structural validity and human verification assess complementary aspects of each item.Source: E Benchmark Construction · p. 15

E.1 Construction Pipeline

• Candidates are built from text-rich visual seeds and assigned a target reasoning type and OCR-object category, with synthesis designed to require deduction, induction, or abduction over image evidence.Seeds are rejected if answers can be copied from one text span, relevant text is illegible, or solutions depend primarily on external knowledge.Source: §E.1 · p. 15

Human verification and candidate filtering

• Three PhD annotators retained candidates only when image-grounded evidence, at least two evidence points, a unique answer, matching taxonomy labels, and automatic scoring after normalization were present.Failed or duplicate-like candidates were filtered or regenerated; rejection cases included leakage, ambiguity, unsupported rules, weak evidence, and mismatched formats.Source: Human verification and candidate… · p. 15

E.2 Benchmark Validity and Scoring Consistency

• All 1,500 instances have complete annotations, unique normalized questions and question–answer pairs, and at least two reference steps (4.22 on average).Scoring is format-specific: 561 integer and 146 floating-point answers use numeric scoring, 513 string answers use normalized exact match, and 280 structured answers use JSON micro-F1.Source: §E.2 · p. 16
• The taxonomy is exactly balanced: 500 samples per reasoning type, 300 per OCR-object category, and 100 in each of its 15 cells.Repeated source images are allowed when instances differ in question or reasoning target; no exact duplicate normalized questions or question–answer pairs are present.Source: §E.2 · p. 16

E.3 Why These Properties Support Benchmark Reliability

• Human verification and complementary structural checks address semantic and item-format failures, supporting a well-formed, deterministically scorable benchmark aligned with its taxonomy.The balanced taxonomy also helps ensure MRMS and category analyses reflect model behavior rather than uneven sampling.Source: §E.3 · p. 16

F Evaluation Process

• Evaluation follows response generation, answer extraction, and score computation, with a separate MLLM-judge stage for reasoning-process compliance.Deterministic answer scoring determines MRMS; the judge evaluates only the visible reasoning process for RPCS.Source: F Evaluation Process · p. 16

F.1 Answer-Level Score Computation

• Answer scoring uses only the extracted final answer, applies format-specific normalization, and assigns zero when a structured response is invalid.Exact-match strings are normalized before comparison; numeric answers require exact equality without tolerance, while structured answers are scored by micro-F1.Source: §F.1 · pp. 16–17

F.2 Reasoning-Process Score Computation

• RPCS judges whether visible reasoning matches the target direction, is image-grounded, includes necessary steps, and avoids hallucinated content.The judge sees the image, question, target type, OCR-object category, format, scoring rule, reference steps, and reasoning trace—but not gold or predicted answers or answer score.Source: §F.2 · p. 17

F.3 Reliability Controls for MLLM-as-Judge

• The MLLM judge is isolated from final-answer outcomes and may credit compressed, reordered, or paraphrased reasoning when evidence bindings and reasoning direction are preserved.It reports four binary criterion decisions, an aggregate process score, and concise rationales.Source: §F.3 · pp. 17–18
• MRMS and RPCS are reported separately, keeping the primary answer-correctness leaderboard independent of MLLM judging while enabling process diagnostics and mismatch analysis.Inference failures, empty outputs, and outputs containing only a final answer receive zero process credit rather than being dropped.Source: §F.2 · p. 17, §F.3 · p. 18

Answer Format Remains a Structured-Output Bottleneck

• Gemini-3.1-Pro-Preview scores 95.2% on float answers but 77.0% on JSON; GPT-5.4-Medium similarly scores 92.5% on float and 78.5% on JSON.Kimi-K2.5 has a narrower range across formats: 78.9% string, 81.3% integer, 80.1% float, and 77.3% JSON. The passage attributes the strongest models’ JSON gap to the requirement for structurally exact outputs under automatic scoring.Source: G More Results · p. 18

Deduction Drives the Main Accuracy Gap

• Gemini-3.1-Pro-Preview scores 80.7% on deduction versus 93.1% induction and 94.1% abduction; GPT-5.4-Medium scores 80.6%, 91.5%, and 90.8%, respectively.Qwen3-VL-235B-A22B-Thinking also scores lower on deduction (74.8%) than induction (87.8%) or abduction (82.1%); the passage characterizes visible-rule application as harder than abstracting patterns or recovering hidden premises.Source: G More Results · p. 18

Layout Semantics Is Consistently Hardest

• Layout semantics scores are 76.2% for Gemini-3.1-Pro-Preview, 77.6% for GPT-5.4-Medium, and 75.1% for Doubao-Seed-2.0-Pro.Qwen3-VL-235B-A22B-Thinking scores 87.3% on data interpretation and 86.3% on document logic, but 69.2% on layout semantics; the passage links this weakness to visually grounded grouping, alignment, and cross-region inference.Source: G More Results · p. 18

Process Scores Are High but Incomplete

• GPT-5.4-Mini-Medium scores 94.9% on capability match and 94.1% on groundedness, but 75.1% on step completeness.Qwen3-VL-8B-Instruct scores 62.2% on step completeness and 63.7% on non-hallucination; models can follow the broad process form while omitting intermediate evidence or adding unsupported details, helping explain why RPCS is generally higher than MRMS.Source: G More Results · p. 18

The Crossed Taxonomy Localizes Failure Modes

• All-model averages are 82.5% for induction, 80.1% for abduction, and 73.6% for deduction; abductive layout semantics is hardest at 63.2%.Inductive document logic reaches 87.3%; closed-source models average 69.6% on abductive layout semantics, versus 55.1% for open-source models. Difficulty depends on the combination of reasoning type and OCR-object category.Source: G More Results · pp. 18–19

H Qualitative Analysis of RPCS and MRMS Mismatches

• In the deductive case, the model used the right constraint-checking structure but wrongly treated March 16, rather than March 22, as meeting the capacity threshold.The failure was an incorrect binding between the capacity condition and its OCR field, not a wrong reasoning direction.Source: H Qualitative Analysis of RPCS and… · p. 19

H Qualitative Analysis of RPCS and MRMS Mismatches; I OCR-MetaReasoning Examples

• The inductive case correctly inferred descending score mappings, but assigned Long Beach’s target cell to the wrong section for one indicator, changing the computed sum.Figures 17–19 separately show examples of the 15 meta-reasoning tasks in text-rich visual scenarios.Source: H Qualitative Analysis of RPCS and… · p. 19, I OCR-MetaReasoning Examples · p. 19

H Qualitative Analysis of RPCS and MRMS Mismatches

• In the abductive case, the model correctly recovered a hidden discount by subtraction but undercounted an explicit coupon subtotal, producing an incorrect answer.The process had the intended backward-recovery structure, but an arithmetic or transcription error changed the recovered premise.Source: H Qualitative Analysis of RPCS and… · p. 19
• High RPCS can coexist with zero MRMS: process scoring checks reasoning direction and grounded steps, while answer scoring checks the canonical final answer.Together, the metrics distinguish choosing the appropriate reasoning mode from executing OCR-grounded operations accurately enough to answer correctly.Source: H Qualitative Analysis of RPCS and… · p. 19

J.1 Human Validation and Construction Evidence

• For reliability validation, three PhD students independently re-annotated 300 samples: 20 from each of the 15 taxonomy cells, with released labels hidden during the separate reliability study.Annotators saw taxonomy labels during construction; Table 8 combines pre-adjudication agreement with post-adjudication outcomes.Source: §J.1 · p. 19
• After adjudication, all 300 validated samples matched the released meta-reasoning and OCR-object labels and satisfied the validity checklist.Disagreements centered on reasoning-direction boundaries, visually similar OCR-object categories, and image-grounding or validity judgments.Source: §J.1 · p. 19
• Construction used synthesis, checklist-based filtering, and balanced selection, yielding 100 samples in every taxonomy cell.Rejected or filtered candidates failed the human checklist, while pass-not-selected candidates passed verification but were omitted during balanced sampling; similar tasks used separate prompt or image regeneration.Source: §J.1 · p. 20

J.2 RPCS Human Agreement and Judge Stability

• Human raw agreement averaged 90.7% across RPCS criteria, with macro Fleiss κ of 0.860.Criterion-level agreement ranged from 89.7% to 91.7%, and Fleiss κ from 0.856 to 0.866; reference-step overlap correlated near zero with judge–human error (Pearson 0.040; Spearman 0.036).Source: §J.2 · p. 20, Table 10 · p. 21

J.3 Uncertainty, Paired Tests, and Item Headroom; K.1 Automatic Difficulty Buckets

• Only 453/1,500 items received full scores from all 18 models, while 214/1,500 averaged below 0.5 and 118/1,500 averaged below 0.25.Difficulty buckets use each item’s mean final-answer score across 18 models; the aggregate hard set contains 214 items, including 118 very-hard and 96 hard-only items.Source: §J.3 · p. 21, §K.1 · p. 23, Table 17 · p. 25

J.4 Human Baseline

• Human majority achieved MRMS 96.0 versus 90.5 for Gemini-3.1-Pro-Preview on the 300-item subset.The majority ambiguity-flag rate was 1.3%, the any-annotator rate was 4.0%, and mean confidence was 0.930; flagged ambiguous items counted as incorrect.Source: §J.4 · p. 21, Table 12 · p. 23

J.5 External Benchmark Construct Comparison

• The construct audit compares selected subsets of six related benchmarks, then reruns the same eight model versions on selected subsets without mapping results across model generations.VisualPuzzles appears in the conceptual comparison but is excluded from the quantitative audit; the supplied passages do not state the audit’s rank-correlation results.Source: §J.5 · pp. 21–22

J.6 OCR, Layout, and Answer-Format Controls

• On the five-model control subset, layout-aware transcripts recover part of the OCR-transcript loss, while third-party OCR and answer-format hints yield only small average gains.The study compares six input conditions and measures layout by contrasting OCR transcript-only with layout-aware transcript; these are subset averages, not full-benchmark results.Source: §J.6 · p. 22, Table 15 · p. 24

J.7 Artifact Provenance and Split Composition

• The final set contains 1,299 unmodified and 201 edited/reconstructed images; all edited/reconstructed samples are abductive.The edits are controlled modifications of public OCR seeds, and the deductive and inductive splits each contain 500 unmodified images; the split comparison is descriptive because editing is confounded with abduction.Source: §J.7 · p. 22

K.2 Qualitative Coding

• The qualitative-analysis subset contains 123 items, including 34 very-hard, 44 hard-only, and 45 easy cases; 107 required image-dependent checks.Its 44 hard-only cases are distinct from the full automatic hard-only bucket of 96; coding distinguishes mechanisms such as layout binding, visual/OCR cues, constraint search, and implicit pattern induction.Source: §K.2 · pp. 23–24

K.3 Eight Representative Case Panels

• Eight representative panels cover three very-hard, three hard, and two easy cases, while three additional cases show RPCS passing despite MRMS failure.The latter involve following a reasoning schema but misbinding a timeline entry, reading a target cell from the wrong section, or undercounting a coupon subtotal.Source: §K.3 · p. 24, Figure 14 · p. 31, Figure 15 · p. 32