PaperlayerView exact revision on arXiv

Source-linked AI summary

SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

Abhigya Verma, Khyati Mahajan, Amit Kumar Saha, Shruthan Radhakrishna, Sagar Davasam, Vikas Yadav, Sai Rajeswar Mudumba

arXiv:2607.10400v1cs.CVcs.AI

TL;DR

Existing benchmarks make it difficult to diagnose why VLMs fail on long, visually rich documents. SynthDocBench systematically varies document factors to expose failure patterns, revealing degradation with difficulty and positional sensitivity.

  • Problem

    Existing benchmarks provide limited evidence for diagnosing why VLMs fail when charts, text, tables, and complex layouts span long documents.

  • Method

    SynthDocBench is a synthetic benchmark that independently varies document length, page depth, modality composition, and question type, then evaluates rendered page images against deterministic reference answers.

  • Results

    VLMs showed difficulty-related degradation and positional sensitivity; the middle third was hardest for 5 of 8 models, dropping 5–18 pp below Early.

  • Takeaways & Limitations

    The benchmark reveals failure modes that standard evaluations do not expose, including positional bias and visual hallucination on dense chart-reading tasks.

  • Takeaways & Limitations

    The authors identify broader multimodal settings, longer contexts, and more diverse reasoning tasks as future extensions.

Abstract

from arXiv · show

Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.

1 Introduction

SynthDocBench addresses the diagnostic blind spot in long-context visual document evaluation by independently controlling document length, page depth, modality composition, and question type. Its synthetic design targets failure modes that real-document benchmarks cannot disentangle because answer depth, modality, layout density, and cross-page evidence co-vary.

  • Motivation: Real-world visual documents require long-range retrieval and cross-modal reasoning across text, tables, charts, and complex layouts spanning dozens or hundreds of pages.Existing benchmarks have driven progress, but their results can conceal which document factors cause model failures.
  • Limitations of Existing Benchmarks: 95% on DocVQA and 89% on ChartQA mark near-saturation on single-page evaluation, while harder chart benchmarks show degradations exceeding 30 percentage points.These evaluations still assess charts in isolation rather than within multi-page document contexts.
  • Limitations of Existing Benchmarks: Existing long-document benchmarks prioritize coverage over controlled diagnosis and do not require joint reasoning over charts and textually distributed evidence across distant pages.Because they use real documents, answer depth, presentation modality, layout density, and cross-page evidence integration co-vary and cannot be disentangled.
  • Benchmark Contribution: SynthDocBench is a fully synthetic benchmark that systematically varies document length, page depth, modality composition, and question type as independent axes.The benchmark is designed to decompose VLM failure modes in long-context document understanding.
  • Benchmark Contribution: The benchmark provides three task-specific subsets and probes 24 distinct D3.js chart types, including visually ambiguous forms requiring exact numerical reads.The chart questions are designed so answers cannot be inferred from surrounding text.

2 Related Work

Prior benchmarks evaluate charts and long-context documents largely in isolation or emphasize broad coverage, while single-page document QA is nearing saturation. SynthDocBench addresses this gap with diagnostic evaluation of charts embedded in long, multimodal document contexts.

  • Charts and Visual Reasoning: Over 30 percentage points of degradation separates models saturating ChartQA from their performance on ChartQAPro’s harder, more diverse real-world charts.ChartQAPro increases chart difficulty but retains an isolated-image setting.
  • Charts and Visual Reasoning: Existing chart benchmarks do not evaluate charts within document contexts requiring interpretation alongside surrounding text, tables, and figures.This leaves chart understanding in natural document settings insufficiently assessed.
  • Long-Context Multimodal Benchmarks: Long-context benchmarks progressed from needle-in-a-haystack retrieval and single-page DocVQA to multi-page and multimodal evaluations, including MMLongBench-Doc and MMLongBench.Early retrieval tests have insufficient difficulty ceilings and correlate poorly with downstream reasoning performance.
  • Long-Context Multimodal Benchmarks: LongDocURL and M-LongDoc broaden coverage through cross-element localisation and open-ended responses, but prioritize breadth over diagnostic decomposition.SynthDocBench is distinguished by its explicitly diagnostic design.
  • Vision-Language Models for Document Understanding: Frontier systems approach ceiling performance on DocVQA, whereas long-context visual document reasoning remains substantially harder and MMLongBench-Doc results remain far below single-page benchmarks.Qwen3-VL and InternVL3.5 are cited as examples of systems nearing DocVQA saturation.

3 The SYNTHDOCBENCH Benchmark

SYNTHDOCBENCH is generated through coupled document, QA, and vision-only evaluation stages that preserve structured evidence and deterministic references. It contains 200 long synthetic reports with controlled visual diversity and three question families spanning chart reading, cross-modal reasoning, and multi-hop composition.

  • Benchmark construction: Three coupled stages generate styled reports with aligned manifests, convert manifests into difficulty-controlled QA pairs, and evaluate models only on rendered page images.Reference answers are derived deterministically from the same structured artifacts used for document generation.
  • Question generation: The QA pipeline recovers text, table, and visualization evidence, then generates chart-reading, cross-modal, and complex multi-hop questions with L1–L5 difficulty labels.Multi-hop questions compose 2–4 evidence units, and validation filters malformed, weakly supported, or ambiguous items.
  • Benchmark composition: 200 reports and 1,788 questions comprise the benchmark, distributed across 597 chart-reading, 597 complex multi-hop, and 594 cross_modal questions.The documents average 51.1 pages, 16.7 charts, and ≈20,568 words.
  • Benchmark composition: The benchmark spans 24 distinct chart types across 6 layout archetypes, preventing reliance on narrow visual or structural distributions.This diversity is designed to prevent models from exploiting restricted chart or layout patterns.

4 Evaluation Setup

SynthDocBench evaluates eight vision-language models under a strict vision-only protocol, using judge-based scores and threshold accuracy across question subsets and difficulty levels. Gemini-3.1-Pro leads overall, while OCR comparisons show complementary strengths for complex text evidence and chart decoding.

  • Evaluation protocol: Models receive only rendered page images and questions, with PDFs rasterized at 144 DPI and packaged into five-page vertical strips by default.They cannot access HTML source, embedded metadata, or the underlying document representation.
  • Metrics: Evaluation reports mean judge score and threshold accuracy at τ=6, stratified by question family and difficulty levels L1–L5.GPT-5 judges answers at temperature 0 using a fixed prompt; parse failures are excluded from aggregates.
  • Models: Eight vision-language models are evaluated on SYNTHDOCBENCH, with all responses scored by GPT-5.The models range from Gemini-3.1-Pro and GPT-5.4 to Qwen2.5-VL-7B.
  • Main results: Gemini-3.1-Pro leads overall with ACC 0.725 and mean judge score 7.19, followed by Qwen3.5-VL-122B at ACC 0.655.The ranking remains consistent across chart, complex, and cross-modal question subsets.
  • Main results: Cross-modal questions are consistently hardest, whereas chart-reading is easiest for most models.This indicates that integrating textual and chart evidence remains challenging in long documents.
  • External validity: SYNTHDOCBENCH rankings correlate with MMLongBench-Doc, with Spearman ρ=0.657 and Pearson r=0.683.This supports external validity of the benchmark ordering.
  • OCR baseline: OCR and vision show complementary strengths: OCR reaches complex-question ACC 0.798, while vision reaches chart-reading ACC 0.457 versus OCR’s 0.297.The comparison uses GPT-4o for both modalities and supports text-recoverability of complex evidence alongside pixel-level chart decoding.

5 Analysis and Discussion

Analysis across difficulty, question category, position, error type, and presentation reveals distinct bottlenecks in precise visual reading, cross-modal evidence integration, and long-context handling. Results also identify a rendering-familiarity confound that cautions against interpreting model gaps as purely reasoning differences.

  • Difficulty-Stratified Results: 23 pp: Claude-Sonnet-4.5 drops from L1 to L5, while Gemini-3.1-Pro remains flat at 0.670–0.784.Qwen3.5-VL-122B instead falls from 0.707 at L1 to 0.528 at L5; GPT-4o’s L1 ACC is 0.271.
  • Question-Category Breakdown: 0.643 vs. 0.111: Gemini and GPT-4o show the widest gap on technical/quantitative complex questions.Trend/pattern questions are easiest, whereas value-reading and integrate-sources cross-modal questions are hardest.
  • Positional Bias: 5–18 pp: the middle document third is hardest for 5 of 8 models, with Qwen3.5-VL-122B dropping 18.5 pp from Early to Middle.Claude-Sonnet-4.5 has the steepest monotonic Early-to-Late decline at 11.7 pp, while Gemini-3.1-Pro follows a U-shaped pattern.
  • Hard Failure and Domain Error Analysis: 109 questions: cross-modal failures dominate cases where all eight models score ≤3, while cross-modal ACC trails chart reading by 13–16 pp in every topic domain.Visual hallucination dominates errors, especially on dense value-reading charts, dumbbell plots, and multi-series comparisons, alongside figure-not-found and precision errors.
  • Effect of Image Presentation: 0.369→0.792: Gemini-3.1-Pro ACC rises with pages per strip from 1 to 10, while cross-modal ACC increases from 0.339→0.707.144 DPI is optimal; higher resolution degrades performance under the 4 MB API cap because of JPEG compression.
  • Gemini-3.1-Pro Dominance and Potential Confounds: 13.9 pp: Gemini-3.1-Pro leads Qwen3-VL-235B, but web-rendered HTML/D3.js charts may favor Gemini’s training distribution.The analysis recommends alternative rendering backends to separate rendering familiarity from long-context reasoning effects.

6 Conclusion

SYNTHDOCBENCH is a synthetic long-context visual document understanding benchmark that independently controls key difficulty axes to measure multi-fact retrieval and multi-step reasoning. Future work will broaden its document types, context lengths, and reasoning tasks.

  • Benchmark design: SYNTHDOCBENCH measures VLMs’ ability to locate multiple key information facts in long-context documents and answer multi-step questions.The benchmark targets long-context visual document understanding.
  • Benchmark design: Its combinatorial design independently varies document length, layout complexity, modality composition, and question type.The benchmark is generated synthetically with controlled difficulty axes.
  • Future work: Future extensions will include richer document types such as tables, forms, and mixed-layout reports.These additions are intended to broaden multimodal reasoning settings.
  • Future work: Future work will also cover longer contexts and reasoning tasks including multi-hop aggregation and cross-document grounding.These extensions aim to increase context and task diversity.

Reproducibility Statement

The authors support reproducibility by publicly releasing SynthDocBench’s evaluation code and dataset, while disclosing experiment hyperparameters throughout the paper and appendix.

  • Reproducibility Statement: The evaluation source code and dataset are publicly available, and experiment hyperparameters are disclosed throughout the paper and appendix.Code: https://github.com/ServiceNow/SynthDocBench; dataset: https://huggingface.co/datasets/ServiceNow-AI/SynthDocBench.

Ethics Statement

SynthDocBench uses fully synthetic, privacy-preserving data and controlled generation, while documenting evaluation, environmental costs, benchmark-integrity risks, and limits on intended use.

  • Synthetic data and content generation: Synthetic documents involve no human subjects, personal data, or reproduced real-world documents, and are generated from public topic seeds under structured constraints.Content was reviewed to confirm compliance.
  • Model evaluation and API usage: Evaluations use commercial APIs and open-weight models under their terms, with frozen weights and a GPT-5 judge scoring responses against deterministic references.Rubrics and prompts are disclosed in the appendix for replication.
  • Environmental impact: 150 GPU-hours and 200 API-hours were estimated for document rendering, model inference, and judge scoring.Redundant runs were limited through ablation design, and configurations were reported transparently.
  • Benchmark integrity and Goodhart’s Law: 40% random layout override mitigates spurious topic–layout correlations and reduces overfitting risks from chart types, layouts, and question templates.Programmatic generation also supports straightforward benchmark extensions.
  • Intended use: The benchmark targets research on long-context visual document understanding, not high-stakes deployment, and its controlled failure modes should not be generalized uncritically.The authors encourage diagnostic use and extensions that preserve benchmark integrity.

A Evaluation Pipeline Diagram · B Benchmark Comparison

The evaluation pipeline rasterizes rendered PDFs into concatenated page-image strips, submits them directly to vision-language models, and scores responses with GPT-5. Benchmark comparison shows that strong results on existing benchmarks do not reliably transfer because their heterogeneous corpora lack controlled variation.

  • A Evaluation Pipeline Diagram: 144 DPI rasterization converts rendered PDFs into page images for evaluation.The pipeline then concatenates the images into 5-page vertical strips.
  • A Evaluation Pipeline Diagram: 5-page vertical strips are passed directly to candidate vision-language models at temperature 0 without OCR or metadata.The evaluation is vision-only.
  • A Evaluation Pipeline Diagram: GPT-5 judges each candidate response against a reference answer using the Table 6 scoring rubric.Figure 8 describes the references as deterministic, while Table 6 notes that scores of −1 indicate parse failures excluded from aggregate statistics.
  • B Benchmark Comparison: Table 7 compares SynthDocBench with closely related benchmarks and reports frontier-model scores on each.The comparison includes DocVQA, ChartQA, and MMLongBench-Doc.
  • B Benchmark Comparison: Existing benchmark coverage is sparse and inconsistent across models because the benchmarks use heterogeneous real corpora without controlled variation.This limitation makes controlled attribution of model behavior difficult.
  • B Benchmark Comparison: Strong scores on DocVQA, ChartQA, and MMLongBench-Doc do not transfer reliably to SynthDocBench.The passage states that models approaching saturation on DocVQA and ChartQA still exhibit substantial failure rates on SynthDocBench.

C Judge Validation … G Positional Bias: Bar Chart

The benchmark’s rankings are robust to judge choice and bootstrap uncertainty, while quality controls support question validity. Model score distributions and positional analysis reveal distinct retrieval patterns and chart-reading sensitivity across document positions.

  • C Judge Validation: GPT-5 and Gemini-as-judge agree within 3.5 ACC points, with r ≥0.94 and w1 ≥0.84 across tested candidates.Agreement is highest for chart-reading and cross-modal questions, and lower for complex multi-hop questions.
  • C Judge Validation: ρ = 0.657 (r = 0.683) between SynthDocBench and MMLongBench-Doc rankings indicates moderate positive agreement without identical rankings.The comparison covers six models evaluated on both benchmarks.
  • D Bootstrap Confidence Intervals: All adjacent-ranked model ACC gaps exceed their combined bootstrap 95% CI half-widths, confirming reliable rankings.The intervals use 2,000 resamples with seed 42 and cover each question subset and difficulty level.
  • E Score Distribution per Model: Gemini-3.1-Pro is the only model with a strong right-skewed judge-score distribution, while Qwen2.5-VL-7B is concentrated near 0.Bimodal distributions indicate full recovery or full misses, whereas unimodal distributions centered at 6–7 indicate consistent but imprecise retrieval.
  • F Question Quality Control: Fewer than 2% of chart-reading questions were removed by numeric recomputation against structured metadata using a 5% relative-tolerance threshold.Flagged questions were corrected or dropped when recomputed answers differed beyond the tolerance.
  • F Question Quality Control: κ = 0.81 and a 96% overall acceptance rate were achieved in manual review of 100 questions across answerability, correctness, and clarity.Two authors independently reviewed a stratified random sample.
  • G Positional Bias: Bar Chart: The middle third is hardest for 4 of 6 models in chart-reading ACC, with Claude-Sonnet-4.5 showing the steepest decline at −11.7 pp.Figure 10 uses GPT-5 judging, 597 questions, and equal thirds of relative chart position across 200 reports.

H Domain Analysis … N.3 Prompting Strategy Ablation

The paper documents controlled performance patterns, benchmark taxonomies, implementation choices, evaluation procedures, and ablation findings. Prompting particularly affects complex-question accuracy, while chart-reading and cross-modal performance remain comparatively stable across strategies.

  • H Domain Analysis: Cross-modal accuracy is 13–16 pp below chart-reading and complex in every annotated domain, with the widest gap in AI & Technology.Gemini-3.1-Pro results cover six domains, three question types, and 57 domain-annotated reports containing 513 questions.
  • I Question-Category Breakdown (Full Table); J Performance by Question Subset; K Question Category Taxonomy: Cross-modal integration is consistently the hardest question subset across all six models, identifying modality alignment as the primary bottleneck.The benchmark uses three subsets: chart-reading, complex, and cross-modal.
  • I Question-Category Breakdown (Full Table); K Question Category Taxonomy: The nine mutually exclusive question categories target distinct reasoning capabilities, while difficulty L1–L5 varies independently of category.Each category belongs to one of the three question subsets and may occur at any difficulty level depending on the evidence-chain complexity.
  • L Additional Implementation Details; L.1 Notation Glossary; L.2 Layout Archetypes; L.3 Visualization Metadata Schema; L.4 Question Output Schema; L.5 Rendering Hyperparameters; L.6 Inference and Judge Configuration: The benchmark’s implementation records layout archetypes, visualization semantics, serialized QA evidence, rendering constraints, notation, and deterministic inference settings.The pipeline includes six layout archetypes, structured visualization metadata, evidence traces, page-image rendering controls, and temperature-0 inference for candidates and judge.
  • L.4 Question Output Schema: The corpus contains 24 distinct chart types, spanning common forms such as bar and line charts and specialized forms such as dumbbell and sankey charts.Figure 13 displays the top 20 chart types.
  • N.3 Prompting Strategy Ablation: No-prompt improves complex-question ACC by 5.3, 5.9, and 17.0 points for Gemini, GPT-4o, and Claude, respectively, while CoT modestly raises overall ACC but lowers GPT-4o chart-reading ACC from 0.520 to 0.468.Claude’s complex ACC rises from 0.380 to 0.550; chart-reading and cross-modal ACC vary by at most 5 points across strategies.

N.4 Judge Sensitivity Ablation · O Qualitative Model Response Examples

The judge-sensitivity ablation shows that removing concise-answer prompting improves complex-question accuracy, while chain-of-thought has mixed effects and leaves visual subsets largely stable. Qualitative examples expose chart-reading errors, visual hallucination, figure-localization failures, and incomplete multi-hop retrieval across evaluated models.

  • N.4 Judge Sensitivity Ablation: Claude’s complex-question ACC rises 17.0 points without prompting, from 0.380 to 0.550.Gemini and GPT-4o also improve by 5.3 and 5.9 points, respectively, over default prompting.
  • N.4 Judge Sensitivity Ablation: CoT yields modest overall gains but reduces GPT-4o chart-reading ACC from 0.520 to 0.468.Gemini gains 1.6 points and Claude gains 3.0 points; chart-reading and cross-modal ACC vary by at most 5 points across strategies.
  • O Qualitative Model Response Examples: The qualitative examples present an evidence-page crop, question, ground-truth answer, and annotated model response, with colors marking correct and hallucinated facts.Four representative examples span the three question subsets and dominant failure modes from Section 5.
  • Example 1 — Chart Reading L1 Mixed: Value Reading from Horizontal Bar: In the horizontal-bar example, only Gemini and Claude identify the correct 74% False Negative value; GPT-4o, Qwen, and InternVL3 produce wrong readings.GPT-4o hallucinates 43%, both Qwen models read 26%, and InternVL3 hallucinates 16%.
  • Example 2 — Cross-Modal L2 Failure: Visual Hallucination: No model reads the correct 120→40 file counts in the cross-modal chart question.Every model hallucinates a different plausible-looking pair; Claude and InternVL3 decline to answer, while GPT-4o conflates file counts with axis percentages.
  • Example 3 — Cross-Modal L3 Failure: Figure Not Found: All six evaluated models return “not found” despite Figure 3 being visible and labeling the 180-minute maximum at Dev 6.The failure is attributed to inability to localise and resolve the individual data-point label.
  • Example 4 — Complex Multi-Hop L5 Failure: Incomplete Retrieval: The complex multi-hop question requires six independent statistics from three charts, producing partial retrieval and incorrect values across models.Gemini, Qwen3, GPT-4o, and InternVL3 each retrieve some evidence but make other numerical errors; Claude retrieves none.
Loading 2607.10400v1…