Source-linked AI summary
SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding
Shenxi Wu, Yuhong Liu, Haosong Zhang, Tongjin Zou, Yanxun Zhang, Gaochang Chen, Dun Liang, Jiaqi Wang, Zhecan James Wang, Yuhang Zang, Dahua Lin
TL;DR
Scientific-document benchmarks often isolate capabilities that real research assistance requires jointly, including multimodal reasoning, evidence grounding, and provenance tracking. SciDocBench evaluates these workflows under matched language and document-representation conditions, while SciDocIR and SciDocDataset convert the diagnostics into verifiable training resources. The strongest evaluated model scores 62.6 out of 100, and performance varies substantially with document representation.
Problem
Existing benchmarks typically evaluate scientific-document capabilities in isolation, leaving realistic multimodal research workflows insufficiently assessed.
Method
The paper introduces SciDocBench with matched bilingual and dual-representation evaluation, then builds SciDocIR and verifiable SciDocDataset training resources.
Results
The strongest evaluated model scores 62.6 out of 100, while model performance and rankings vary across document representations.
Takeaways & Limitations
The benchmark and evidence-graph pipeline provide an evaluation-to-training framework for diagnosing and improving scientific-document assistants.
Takeaways & Limitations
The benchmark has 124 underlying questions, limited domain coverage, two-to-four-paper cross-document tasks, and four matched variants that are not independent problems.
Abstract
from arXiv · showhide
Scientific papers require models to reason jointly over text, equations, figures, tables, code, and datasets while preserving the provenance of supporting evidence. Existing benchmarks typically evaluate these capabilities in isolation, leaving unclear whether multimodal models can support realistic scientific-reading workflows. We introduce SciDocBench, a workflow-centered benchmark for scientific document understanding. It contains 124 expert-authored and difficulty-screened questions organized into seven research-assistant capability groups and 19 subtasks across five scientific domains. Each question is instantiated under four matched conditions combining English or Chinese questions with all-images-first or interleaved document representations, yielding 496 evaluation instances for controlled analysis. The strongest evaluated system achieves only 62.6/100, with pronounced weaknesses in document perception, evidence grounding, verification, and cross-document reasoning. To translate these diagnostics into scalable training signals, we introduce SciDocIR, a typed evidence-graph representation that preserves scientific document objects, layout and cross-reference relations, and provenance. Building on SciDocIR, we construct SciDocDataset, comprising approximately 15K supervised fine-tuning samples and 8K reinforcement-learning samples across 14 verifiable subtasks. Together, SciDocBench, SciDocIR, and SciDocDataset form an evaluation-to-training framework for diagnosing and improving scientific-document assistants. The project page is available at https://github.com/InternLM/SciDocBench.
1. Introduction
SciDocBench frames scientific document understanding as a workflow requiring multimodal evidence integration, verification, and provenance preservation. It evaluates these abilities under matched conditions and connects diagnosis to verifiable training data.
- Motivation: Scientific assistants must jointly interpret text, equations, figures, tables, code, and datasets while preserving evidence provenance.These capabilities support literature review, claim verification, result comparison, and reproducibility analysis.
- Challenges: Scientific document understanding is challenged by heterogeneous layouts, compositional reasoning across documents, and the need for verifiable outputs.Relevant evidence may span scientific objects, associated repositories, multiple papers, and datasets.
- Benchmark: SciDocBench contains 124 expert-authored questions across five domains, seven capability groups, and 19 subtasks, covering single-document and multi-document research tasks.Its tasks integrate notation, datasets, evidence, and experimental results across papers.
- Evaluation: The matched protocol produces 496 evaluation instances by crossing bilingual questions with All Images First and Markdown Interleaved document representations.Task semantics, reference-answer semantics, and evaluation criteria remain fixed across settings.
- Evaluation-to-training: SciDocIR aligns visual and semantic document objects while preserving layout, cross-reference relations, and provenance for scalable training supervision.SciDocDataset contains approximately 15K supervised fine-tuning samples and 8K reinforcement learning samples across 14 verifiable subtasks.
- Results: Claude-Opus-5 achieves the highest overall score at 62.6 out of 100, while models show persistent extraction and cross-document weaknesses and sensitivity to document representation.For 11 of 14 evaluated models, the document-representation gap exceeds the question-language gap.
2. Related Work
Related benchmarks cover document structure, scientific question answering, visual scientific-object reasoning, and multimodal resources. SciDocBench positions itself by composing these capabilities within full scientific-document workflows with evidence, cross-document, and provenance requirements.
- Document understanding: Document-understanding benchmarks evaluate layout parsing, table extraction, field extraction, visual question answering, and long-context reading.Examples include PubLayNet, LayoutLM, DocVQA, DUE, and MMLongBench-Doc.
- Scientific question answering: Scientific question-answering resources range from evidence-grounded questions over full NLP papers to multimodal, multi-document, and agentic research assistance.QASPER, PubMedQA, M3SciQA, and PaperScope represent different scopes of scientific question answering.
- Scientific-object reasoning: PlotQA, ChartQA, and SCITAB assess visual, logical, numerical, compositional, or claim-verification reasoning over charts and tables.SciDocBench embeds figure, table, and equation tasks within full scientific documents and evaluates evidence distributed across related objects and papers.
- Structured resources: S2ORC and DocGenome provide structured scholarly metadata, full text, multimodal document objects, layout attributes, source code, and relations for analysis or training.These resources support scientific-document structure and multimodal modeling beyond individual question-answering tasks.
- Positioning and scope: SciDocBench jointly targets full-document input, scientific-object reasoning, cross-document understanding, and code or data provenance rather than introducing each capability individually.Existing benchmarks often cover related signals only partially or outside full-paper workflows.
3. The SciDocBench Benchmark
SciDocBench evaluates realistic scientific-reading workflows across scientific objects, capability groups, document representations, and question languages. Its controlled construction and scoring support diagnostic comparisons of model abilities and input sensitivity.
- Benchmark scope: SciDocBench contains 124 expert-authored questions spanning seven capability groups and 19 subtasks, instantiated into 496 matched evaluation instances.The benchmark covers evidence location, scientific-object interpretation, verification, cross-document integration, and paper–code or paper–data connections.
- Workflow realism: Questions target concrete research activities, including claim review, experimental-detail extraction, result comparison, analysis reproduction, dataset tracing, and figure-to-data conversion.The benchmark emphasizes operations that arise when multimodal models act as paper-reading assistants.
- Evaluation protocol: Evaluation instances specify source materials, instructions, response formats, and scoring methods, with outputs ranging from labels and values to graphs, LaTeX, and executable programs.Structured formats are used when outputs must be parsed, verified, or reused directly.
- Controlled evaluation: Each question combines English or Chinese instructions with either All Images First or Markdown Interleaved document inputs.The four settings preserve task semantics, reference-answer semantics, and evaluation criteria for controlled analysis.
- Benchmark construction: Benchmark construction selects information-rich papers, authors and reviews realistic questions, screens difficulty, and applies controlled augmentation.Candidates solved by every screening model are removed, while retained questions undergo checks for ambiguity, evaluability, and answer verification.
- Quality control: Semantic evaluation shows 92.0% exact agreement and quadratic-weighted Cohen’s κ=0.866 between GPT-5.4-mini and human experts on audited response-level instances.Ordinal analysis better represents partial-credit distinctions than a binary threshold, which yielded 70.0% agreement and κ=0.355.
4. SciDocIR and SciDocDataset
SciDocIR normalizes heterogeneous scientific papers into aligned visual and semantic records with layout, cross-reference, and provenance information. SciDocDataset builds verifiable training samples from this representation through seeded generation, controlled rendering, and validation.
- SciDocIR construction: SciDocIR aligns visual PDF parsing with semantic LaTeX parsing into document-, page-, and block-level records.The representation accommodates papers whose scientific objects appear as source environments, rendered regions, or separated visual blocks.
- SciDocIR construction: SciDocIR records metadata, page layout, block content, source representations, captions, reading order, cross-references, and typed relations.Relations connect continuations, parent–child structures, caption–target pairs, and local evidence chains.
- Training task design: SciDocDataset uses 14 verifiable training subtasks selected for supervision through structured fields, perturbations, set operations, numeric tolerances, parsing, or execution.The subtasks cover layout recovery, evidence verification, cross-document understanding, and chart reconstruction.
- Separation of evaluation and training: SciDocBench remains broad and diagnostic, whereas SciDocDataset supplies task-specific supervision without converting held-out benchmark questions into training examples.Accepted samples include instructions, document inputs, verifiable targets, and construction metadata retaining source identifiers and rejection outcomes.
- Seeded generation and authority: Training samples are generated from seed instructions and selected SciDocIR records, while deterministic source fields, locked values, labels, formulas, and construction records remain fixed.GPT performs linguistic naturalization, bounded candidate generation, semantic classification, or solvability checks rather than determining ground truth.
- Controlled rendering: Synthetic figures and tables are inserted into recorded page blocks while preserving the surrounding template, typography, column structure, and nearby content.Rendering records link each insertion to its source page, target block, modified text, and expected answer for verification and auditing.
5. Experiments
SciDocBench evaluation shows uneven multimodal scientific-document capabilities and substantial sensitivity to language and document representation. Strong aggregate scores conceal weaknesses in perception, extraction, cross-document synthesis, and cross-setting robustness.
- Overall performance: 62.60 was the highest overall score, achieved by Claude-Opus-5, while no evaluated model reached 63.Overall scores spanned 22.44 points across systems.
- Capability composition: No model led more than two capability groups, indicating that capability leadership was distributed across model families.Claude-Opus-5 led overall by remaining comparatively strong across several groups rather than dominating one capability.
- Capability composition: 45.92 was the lowest mean capability-group score, for document perception, followed by scientific extraction at 50.59.Group C had the highest mean at 61.09, while capability ceilings also varied across groups.
- Interface sensitivity: Language effects were model dependent: English averaged 52.52 versus 51.83 for Chinese, while individual model changes ranged from a 5.12-point decrease to a 1.24-point increase.The aggregate difference was only 0.69 points, but model-specific effects differed in direction.
- Interface sensitivity: Document representation produced a 1.52-point average gap, with nine models favoring All Images First and five favoring Markdown Interleaved.Neither representation was uniformly preferable, and rankings changed across settings.
- Interface sensitivity: Language and representation effects interacted, producing reversals across models and settings rather than independent performance shifts.For example, Gemini-3.6-Flash favored different representations depending on question language.
- Cross-setting robustness: The average four-setting range was 6.41 points, so reporting only peak accuracy can conflate capability with compatibility with a particular input format.GPT-5.6-Sol varied by 1.00 point, whereas several models exceeded 10 points.
- Main findings: The experiments identify capability composition, interface sensitivity, and remaining headroom as the main conclusions.The strongest score remained 62.60, alongside low average scores for document perception and scientific extraction.
6. Conclusion
The paper presents SciDocBench as a matched, workflow-centered evaluation of scientific-document understanding and uses SciDocIR to generate verifiable training data. Results show limited overall performance, distributed capability leadership, and sensitivity to input format.
- Benchmark: SciDocBench contains 124 expert-authored questions across seven capability groups and nineteen subtasks, evaluated in two languages and two document representations.The matched design produces 496 evaluation instances.
- Findings: The strongest evaluated model scores 62.60, while capability winners are distributed across several model families.For most models, the document-representation gap exceeds the question-language gap.
- Training framework: SciDocIR normalizes document metadata, page information, and block details for constructing supervised fine-tuning and reinforcement-learning data with verifiable targets.The representation supports the paper’s evaluation-to-training pipeline.
- Scope: Scientific-document understanding requires layout recovery, evidence grounding, structured extraction, verification, cross-document synthesis, reconstruction, and provenance tracking together.The conclusion distinguishes this workflow from OCR, long-document question answering, or isolated figure reading.
A. Limitations
The benchmark and training pipeline have limits in statistical breadth, domain and document coverage, realism, and upstream representation quality. These constraints bound how broadly the reported results and generated data should be interpreted.
- Benchmark scope: The benchmark has 124 underlying questions, so the 496 matched instances are not 496 independent scientific problems.A larger question pool would strengthen statistical reliability and enable finer-grained capability analysis.
- Coverage: Domain coverage emphasizes computer science, mathematics, and selected natural and biomedical sciences, leaving social sciences, humanities, and several engineering fields underrepresented.
- Coverage: Cross-document tasks involve two to four papers, whereas real literature reviews may span dozens of documents.
- Data realism: Synthetic evidence enables precise control and verification but cannot reproduce every visual convention or failure mode found in naturally occurring papers.
- Pipeline dependency: SciDocIR depends on layout parsing, source alignment, and OCR, so upstream errors can propagate into task generation and training-data quality.
B. Full SciDocBench Task Taxonomy and Data Statistics
SciDocBench draws documents from arXiv, academic journals, and web-crawled sources, and organizes evaluation around seven capability groups and nineteen subtasks. SciDocDataset statistics emphasize variable-length, multi-image examples rather than short single-image questions.
- Data sources: Source documents come from arXiv, academic journals, and web-crawled sources such as university repositories and Google Scholar.
- Data statistics: SciDocDataset statistics report task-category proportions, total-token distributions, and image-count distributions for SFT/RL samples.These statistics are organized in Figure 4’s three panels.
- Data statistics: The data includes variable-length contexts and multi-image examples reflective of realistic scientific-document workflows.It is not a collection of short single-image QA pairs.
- Task taxonomy: The benchmark task taxonomy contains seven capability groups and nineteen subtasks.
- Source inventories: The benchmark’s source-document inventories are tabulated across categories A1–D3 and E–G.
C. Benchmark Annotation and Verification Protocol
SciDocBench items are authored and screened through a two-stage protocol that records verifiable evidence and independently checks answerability, correctness, and model difficulty.
- Authoring: Authors record each question’s source PDF, metadata, ground truth, and evidence locator identifying the supporting scientific object.The locator may identify a page, section, figure, table, equation, appendix, or other document object.
- Screening: Authors run each finalized item on four designated screening models and preserve every response verbatim for review.
- Authoring: Questions must be answerable from the supplied PDF, use unambiguous wording and output formats, and avoid unstated external knowledge.
- Verification: An independent reviewer verifies answerability, clarity, ground-truth correctness, and that at least one screening model is incorrect before acceptance.The reviewer also audits response labels and may reverse author labels that conflict with the ground truth.
- Auditability: The annotation record stores identifiers, timestamps, task metadata, evidence locators, response labels, and a PDF SHA-256 digest to prevent source substitution.
C.1. LLM-as-a-Judge Reliability Audit
The semantic-scoring audit compares blinded human ratings with automated judgments on a balanced sample spanning evaluation settings, models, and capability groups. Agreement is high under the three-level scoring scheme.
- Audit design: 100 response-level instances were audited, balanced across four evaluation settings and spanning 11 models and all seven capability groups.The audited sample came from accepted LLM-as-a-judge evaluation runs.
- Rating procedure: Human experts rated anonymized responses as 0 incorrect, 0.5 partially correct, or 1 fully correct while model identity and automated judgments remained hidden.
- Agreement: Table 14 reports pairwise agreement using binary scores, quadratic-weighted Cohen’s κ, and mean absolute score difference.Binary results threshold the original scores at 0.5.
- Agreement: 91.3% average pairwise three-level exact agreement was observed across the audited ratings.The ordinal Krippendorff’s α was 0.854, with a 95% confidence interval of [0.757, 0.924].
E. Training-Data Directions and Verifiable Subtasks
SciDocDataset organizes training construction around verifiable subtasks and contracts that separate locked source values from model-assisted linguistic generation. Instances combine SciDocIR records, rendered document views, structured sources, and construction records under explicit rejection rules.
- Subtask inventory: The training pipeline contains eight construction directions and 14 instantiated verifiable subtasks.This inventory is the final data pipeline rather than the broader benchmark taxonomy.
- Model roles: GPT-5.4 performs linguistic naturalization, bounded candidate generation, semantic classification, or solvability checks, while deterministic values and labels remain locked when possible.The supplied construction records identify the authority used for ground truth.
- Inputs: Training instances may combine SciDocIR pages, blocks, relations, metadata, rendered document views, structured sources, and locked construction records.
- Quality controls: Generation must use only supplied material, preserve locked fields, produce a self-contained answerable question, and state the required answer format.
- Quality controls: When evidence is incomplete, ambiguous, illegible, or inconsistent with the construction record, the generator returns a rejection reason instead of an answer.
E.2.6. Citation-Role Analysis
Citation-role analysis classifies how a citation is used in its local scientific context rather than relying on the cited work’s general subject. The task uses a fixed ontology and rejects ambiguous classifications.
- Task definition: A3-lite citation-role classification selects exactly one role from Background, MethodBasis, DataResource, Baseline, ResultSupport, Critique, or Extension.
- Inputs: The classifier uses the citation key, section, page, block, local citing context, role ontology, and heuristic role as inputs.
- Decision rule: The task classifies how the citation key is used in the supplied text, not what the cited paper is generally about.
- Rejection rules: Samples are rejected when roles are equally plausible, citations share the same grammatical role span, or the classification conflicts with the rule-based prior after review.