SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding

Introduction

• Scientific assistants must locate evidence, recover notation and experimental details, verify relations, compare papers, trace code and data provenance, and produce reusable outputs.The challenges include heterogeneous document structure, compositional and cross-document reasoning, verifiable outputs, and controlled evaluation across bilingual questions and document representations.Source: §1 · p. 2
• Existing benchmarks cover document understanding, scientific question answering, charts, and large-scale scientific resources, but full-document workflows and code or data provenance remain fragmented.This fragmentation makes it difficult to evaluate complete scientific research workflows or identify where they fail.Source: §1 · p. 2
• SciDocBench contains 124 expert-authored, difficulty-screened questions spanning five domains, seven capability groups, and 19 subtasks.It covers detailed single-document analysis and multi-document tasks integrating notation, datasets, evidence, or experimental results across papers.Source: §1 · p. 2
• Four matched bilingual-question and document-representation settings yield 496 instances while holding task semantics, reference answers, and evaluation criteria fixed.Questions use either All Images First page images or Markdown Interleaved text and visuals; outputs are assessed by rule-based matching, LLM-as-a-judge, or execution.Source: §1 · p. 2
• SciDocIR aligns visual and semantic document objects while preserving layout, cross-references, and provenance; SciDocDataset provides approximately 15K supervised fine-tuning and 8K reinforcement-learning samples.Its 14 verifiable training subtasks use targets checkable through structured records, controlled perturbations, exact operations, numerical tolerances, or execution; benchmark questions are not converted into training examples.Source: §1 · pp. 2–3
• Claude-Opus-5 achieves the highest overall score, 62.6/100; 11 of 14 models have a larger document-representation gap than question-language gap.Results also show persistent gaps in scientific information extraction and cross-document synthesis, despite stronger performance on some evidence-verification tasks.Source: §1 · pp. 2–3

Document understanding benchmarks

• SciDocBench builds on document-understanding benchmarks by evaluating perception and long-context reading alongside evidence localization, verification, cross-document synthesis, and provenance-aware reconstruction.Prior resources include PubLayNet for scientific-article layout annotations [124], LayoutLM for text-layout interactions [119], DocVQA and DUE for document-image question answering [68], MMLongBench-Doc for lengthy multimodal documents [66], and ArXivDoc for comparing document representations [43].Source: §2 · p. 3

Scientific question answering and research assistance

• SciDocBench evaluates document operations underlying research workflows, including reading-flow recovery, claim grounding, notation and experiment extraction, consistency checking, and structured result integration.Related resources include QASPER for questions over full NLP papers [20], PubMedQA for abstract-grounded biomedical questions [41], M3SciQA for multimodal, multi-document QA, PaperScope for agentic deep research [116], and PaperBench for end-to-end AI research replication [93].Source: §2 · p. 3

Chart, table, and scientific-object reasoning

• SciDocBench embeds figure, table, and equation tasks in full scientific documents to test value recovery, cross-object consistency, and provenance across documents.PlotQA and ChartQA address plot and chart reasoning [70] [67], while SCITAB focuses on compositional reasoning and claim verification over scientific tables [63].Source: §2 · p. 3

Structured scientific-document resources and multimodal training data

• SciDocIR aligns rendered PDF content with available LaTeX semantics and records block-level layout, relations, and provenance; SciDocDataset uses these records for training data across 14 verifiable subtasks.The framework draws on structured resources such as S2ORC [62] and DocGenome [112], multimodal instruction data [57] [24], and evidence-aware training such as DocSeeker [120].Source: §2 · p. 3

Positioning and scope

• SciDocBench jointly evaluates full-document input, scientific-object reasoning, cross-document understanding, and code or data provenance within research workflows.Its contribution is their composition, not the novelty of each capability; QASPER includes evidence annotations but does not center multimodal scientific-object reasoning [20], ChartQA evaluates charts outside full-paper workflows [67], and MMLongBench-Doc does not center paper–code alignment or cross-document result integration [66].Source: §2 · p. 4

3.1. Benchmark Overview and Design Principles

• SciDocBench contains 124 expert-authored questions across seven capability groups and 19 subtasks; each appears in four matched settings combining two question languages and two document representations, yielding 496 instances.Questions derive from concrete research activities, preserve task-relevant scientific content, and require structured or executable outputs when appropriate.Source: §3.1 · p. 4

3.2. Task Taxonomy

• Its seven capability groups span document perception, scientific information extraction, evidence verification, cross-document understanding, reconstruction and execution, paper–code alignment, and dataset understanding.Together, they cover locating and checking information, integrating notation, datasets, and results across papers, and connecting scientific content to reusable artifacts and computational resources.Source: §3.2 · pp. 4–5, §3.1 · p. 4

3.3. Evaluation Instances and Scoring

• Each instance supplies source materials and a task-specific response contract, then uses rule-based, LLM-judge, or execution-based evaluation; failed or unusable responses score zero.Capability-group and setting scores are means on a 0–100 scale, and the four instances per question preserve content and evaluation criteria for paired sensitivity analysis.Source: §3.3 · p. 5

3.4. Benchmark Construction

• Construction selected 116 information-rich papers across five domains and used expert question authoring, review, difficulty screening, and controlled augmentation.Papers were retained for scientific objects such as figures, tables, equations, and experimental descriptions; questions had to be meaningful, answerable from supplied materials, and unambiguous, while generic or unstable items were filtered out.Source: §3.4 · pp. 5–6

3.5. Quality Control and Evaluation Reliability

• 92.0% exact agreement and quadratic-weighted Cohen’s κ=0.866 were achieved by GPT-5.4-mini and an expert; three-rater ordinal Krippendorff’s α was 0.854 (95% CI [0.757, 0.924]).A binary threshold yielded 70.0% agreement and κ=0.355. Candidates were screened with four models; at least one had to fail, and all-model failures required independent review for sufficiency, answer stability, and unambiguous outputs.Source: §3.5 · pp. 6–7

4.1. SciDocIR Construction

• SciDocIR aligns PDF page blocks with LaTeX-derived semantic records and stores document, page, block, and cross-block relation information.Records preserve provenance, layout, source content, reading order, cross-references, and evidence relations.Source: §4.1 · pp. 7–8

4.2. Verifiable Training-Task Design

• SciDocDataset covers 14 verifiable training subtasks selected by answer-checkability, distinct from SciDocBench’s 19 diagnostic benchmark subtasks.Training tasks span layout and reading flow, evidence verification, cross-document understanding, chart reconstruction, notation, symbol disambiguation, and citation roles; benchmark questions are not converted into training examples.Source: §4.2 · p. 8

4.3. Seeded Generation and Source Authority

• Training data uses seeded, task-specific generation in which GPT supports language and bounded checks while deterministic or traceable sources remain authoritative for ground truth.Sources include retrieved SciDocIR tables, controlled table splits, PlotQA series, rendered charts, dataset lexicons, and notation templates.Source: §4.3 · p. 8

4.4. Controlled Block-Level Re-rendering

• Synthetic evidence is inserted by re-rendering selected figure or table blocks within SciDocIR pages, preserving surrounding page context while controlling the modified evidence.Rendering records link the source page, target block, inserted content, any textual modification, and expected answer to support verification and auditing.Source: §4.4 · p. 8

4.5. Validation and Output Records

• Candidates are validated against task-specific schemas and evidence checks, and accepted samples retain inputs, answers, and construction metadata for auditing.Checks include exact set operations, numeric tolerances, parsing, visual solvability, and execution; candidates are rejected for missing fields or inconsistent or insufficient evidence.Source: §4.5 · p. 9

5.1. Experimental Setup

• Each of 14 proprietary and open multimodal models was evaluated on the same 124 questions across four matched settings, yielding 496 instances per model.The reported scores cover overall performance, four evaluation settings, and seven capability groups.Source: §5.1 · p. 9

5.2. Overall Leaderboard

• 62.60 gives Claude-Opus-5 the highest overall score, ahead of GPT-5.6-Sol at 61.00 and Gemini-3.6-Flash at 59.87; no model reaches 63.The evaluated models’ overall scores span 22.44 points.Source: §5.2 · p. 9

5.3. Capability-Level Analysis

• Capability leadership is distributed: Claude-Opus-5 leads Group A, Claude-Opus-4.8 Group B, GPT-5.6-Sol Groups C and E, and four other models lead Groups D, F, and G.No model leads more than two capability groups; Claude-Opus-5’s overall lead reflects comparative strength across several groups rather than dominance of the taxonomy.Source: §5.3 · p. 10
• Group A has the lowest 14-model mean at 45.92 and Group B follows at 50.59, while model capability profiles vary by as much as 31.62 points.Claude-Opus-5 spans 30.72 points across groups and GPT-5.6-Sol spans 31.47; their weaknesses include dataset understanding and paper–code alignment, respectively.Source: §5.3 · p. 10

5.4. Language and Representation Robustness

• English questions average 52.52 versus 51.83 for Chinese, but individual effects vary in direction, including Qwen3.7-Plus’s 1.24-point gain and GLM-4.6V’s 5.12-point decline.Claude-Sonnet-4.6 also declines by 4.30 points with Chinese questions, showing that language sensitivity is model dependent.Source: §5.4 · p. 10
• All Images First averages 1.52 points above Markdown Interleaved, yet nine models favor the former and five the latter, and the leaders differ across settings.Claude-Opus-5 leads both All Images First settings, Gemini-3.6-Flash leads EN-IL, and GPT-5.6-Sol leads ZH-IL.Source: §5.4 · p. 10
• Language and representation effects can reverse across conditions: Gemini-3.6-Flash’s preferred format switches by question language, and Qwen3.8-27B shows opposing language effects across formats.Across four settings, the average score range is 6.41 points; GPT-5.6-Sol varies by 1.00 point, while Claude-Sonnet-4.6, Qwen3.8-27B, and MiMo-V2.5 each exceed 10 points.Source: §5.4 · p. 10

5.5. Main Findings

• The experiments find no model uniformly strong across workflow capabilities, show that language and representation can alter outcomes and interact, and identify substantial headroom for improvement.The strongest overall score is 62.60, alongside low average scores for document perception and scientific information extraction.Source: §5.5 · p. 10

• 62.60 was the strongest evaluated model's score, and capability winners were distributed across several model families.Source: §6 · p. 11
• SciDocBench contains 124 expert-authored questions across seven capability groups and 19 subtasks, producing 496 instances across two languages and two document representations.Source: §6 · p. 11
• For most models, the document-representation performance gap was larger than the question-language gap.Source: §6 · p. 11
• SciDocIR represents document metadata, page information, and block details, and supports supervised fine-tuning and reinforcement-learning data with verifiable targets.Source: §6 · p. 11
• Scientific document understanding requires layout recovery, evidence grounding, structured extraction, verification, cross-document synthesis, reconstruction, and provenance tracking to operate together.Source: §6 · p. 11

A. Limitations

• The benchmark has 124 underlying questions; its 496 matched variants are not 496 independent scientific problems, and a larger pool could improve reliability and analysis.The current questions are expert-authored and difficulty-screened.Source: A. Limitations · p. 18
• Coverage emphasizes computer science, mathematics, and selected natural and biomedical sciences, underrepresenting social sciences, humanities, and several engineering fields.Source: A. Limitations · p. 18
• Cross-document tasks involve only two to four papers, whereas real literature reviews may span dozens of documents.Source: A. Limitations · p. 18
• Synthetic evidence enables precise control and verification but cannot reproduce every visual convention or failure mode in naturally occurring papers.Source: A. Limitations · p. 18
• SciDocIR depends on layout parsing, source alignment, and OCR, so upstream errors can propagate into task generation and training-data quality.Source: A. Limitations · p. 18

B. Full SciDocBench Task Taxonomy and Data Statistics

• Benchmark source documents are drawn from arXiv, academic journals, and web-crawled sources such as university repositories and Google Scholar.The journal sources include Nature and multiple chemistry and other academic journals.Source: B. Full SciDocBench Task Taxonomy and… · p. 19
• The SFT/RL samples vary in total-token and image counts, rather than consisting only of short, single-image QA pairs.The distributions are described as reflecting realistic scientific-document workflows.Source: B. Full SciDocBench Task Taxonomy and… · p. 19
• SciDocBench’s taxonomy comprises seven capability groups and nineteen subtasks.Figure 4 also summarizes the category proportions.Source: B. Full SciDocBench Task Taxonomy and… · p. 19, Table 4 · p. 20

Question authoring

• Authors submit document-grounded questions with evidence locators only after four verbatim screening responses are recorded and at least one model is incorrect.If the question, ground truth, or locator changes, all four responses must be collected again.Source: C. Benchmark Annotation and… · p. 30
• The authoring instructions require a clear, self-contained prompt with a stable, directly verifiable answer that does not rely on unstated external knowledge.Authors identify the supporting page, section, figure, table, equation, appendix, or other scientific object.Source: C. Benchmark Annotation and… · p. 30

Independent verification

• An independent reviewer accepts an item only after confirming answerability, unambiguous wording and format, a correct ground truth, and at least one incorrect screening response.Reviewers audit every response label; correctable defects are returned for revision, while unstable evidence or irreparable ambiguity leads to rejection.Source: C. Benchmark Annotation and… · p. 30

Audit trail

• Each submission records source-PDF identity, question and evidence data, verbatim screening responses, labels, reviewer audits, timestamps, and the final decision.A SHA-256 digest guards against source-PDF substitution, and screening responses determine item difficulty rather than ground truth.Source: C. Benchmark Annotation and… · p. 30

C.1. LLM-as-a-Judge Reliability Audit

• The semantic-scoring audit covers 100 instances, with 25 from each matched evaluation setting, across 11 models and all seven capability groups.Sampling was approximately balanced across the judge’s initial incorrect, partial, and correct score bands.Source: §C.1 · p. 33
• The audit excludes deterministic rule-based and execution-based evaluations because programmatic scorers establish their correctness.The audit concerns semantic scoring on accepted LLM-as-a-judge evaluation runs.Source: §C.1 · p. 33
• The expert rated anonymized responses using only the question and reference answer, with model identity, GPT-5.4-mini scores, and rationales hidden until ratings were submitted.The expert assigned scores of 0 for incorrect, 0.5 for partially correct, and 1 for fully correct responses.Source: §C.1 · p. 33
• The ordinal comparison maps continuous GPT-5.4-mini and Codex scores to incorrect, partial, or correct using strict endpoint rules.Only 0 maps to incorrect and 1 to correct; values strictly between them map to partial. The audit measures consistency in applying the reference-based rubric, not independent re-annotation of source documents.Source: §C.1 · p. 33
• 91.3% average pairwise three-level exact agreement and ordinal Krippendorff’s α of 0.854 indicate strong agreement across the three rating sources.The α 95% confidence interval is [0.757, 0.924], based on 10,000 bootstrap resamples; thresholding at 0.5 can turn small differences among intermediate scores into binary disagreements.Source: §C.1 · p. 33

D. SciDocIR Schema

• SciDocIR stores document-level and page-level metadata, block records, relations, and task-specific metadata.Figure 8 presents the simplified schema.Source: D. SciDocIR Schema · p. 34, Figure 8 · p. 35
• SciDocIR populates optional fields only when they are relevant to task generation or reward verification.The simplified schema marks these fields as optional when they are useful for generation or verification.Source: D. SciDocIR Schema · p. 34, Figure 8 · p. 35

E. Training-Data Directions and Verifiable Subtasks

• Eight construction directions instantiate fourteen verifiable subtasks in SciDocDataset, reflecting the final data pipeline rather than the broader benchmark taxonomy.Table 15 lists the directions and subtasks.Source: E. Training-Data Directions and… · p. 36, Table 15 · p. 36

E.1. Construction Sources and Model Roles

• Deterministic labels, values, alignments, and perturbation records are locked before GPT-5.4 invocation whenever possible; the model handles bounded language and solvability tasks.Tables 16 and 17 identify the records supplied to each template and the ground-truth authority for subtasks 1–14.Source: §E.1 · p. 36, Table 16 · p. 36, Table 17 · p. 37

E.2. Seed Instruction Templates

• Templates may receive SciDocIR records, a rendered document view, structured source data, and a construction record containing locked labels, values, mappings, perturbations, and tolerances.Angle-bracketed expressions are populated by retrieval, synthesis, or rendering programs.Source: §E.2 · p. 37
• The generated question must use only supplied material, preserve locked fields, be answerable from the document view, specify an answer format, and conceal answers and construction metadata.The model-facing instance contains the question and associated document input; construction metadata and source identifiers are retained for generation, verification, and auditing.Source: §E.2 · p. 37
• The template output includes a user-facing question, task-specific answer, audit source IDs, and checks for answerability and a single interpretation.The model-facing question is generated from supplied records under a shared contract.Source: §E.2 · p. 37
• When supplied, a locked gold answer must be copied exactly; bounded semantic labels must come from the supplied ontology, and incomplete, ambiguous, illegible, or inconsistent evidence triggers rejection.The rejection response includes a reason.Source: §E.2 · p. 37

E.2.6. Citation-Role Analysis

• Inputs include the real LaTeX and text block, citation key, section, page, block ID, local citing context, role ontology, and heuristic role.Source: §E.2.6 · p. 40
• Citation roles are classified from the supplied citing text, not from assumptions about the cited paper’s broader subject.The task selects exactly one role: Background, MethodBasis, DataResource, Baseline, ResultSupport, Critique, or Extension.Source: §E.2.6 · p. 40
• Samples are rejected when citations share a grammatical role span, two roles seem equally plausible, or classification conflicts with the rule-based prior after review.Source: §E.2.6 · p. 40

A1. Layout Parsing & Reading Order Recover

• The layout-parsing prompt constructs adversarial multimodal evaluation items for scientific document parsing and reading-order recovery.Source: A1. Layout Parsing & Reading Order… · p. 41, Figure 9 · p. 41

D3. Cross-Document Result Integration

• The cross-document table-merging prompt grounds tasks in real tables and preserves metrics, entities, and row semantics.Source: D3. Cross-Document Result Integration · p. 43, Figure 11 · p. 43

A3. Citation Role Analysis

• The appendix includes a prompt template for citation-role analysis.Source: Figure 13 · p. 45