PaperlayerView exact revision on arXiv

Source-linked AI summary

BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks

Juan Rodriguez, Xiangru Jian, Siba Smarak Panigrahi, Tianyu Zhang, Aarash Feizi, Abhay Puri, Akshay Kalkunte, François Savard, Ahmed Masry, Shravan Nayak, Rabiul Awal, Mahsa Massoud, Amirhossein Abaskohi, Zichao Li, Suyuchen Wang, Pierre-André Noël, Mats Leon Richter, Saverio Vadacchino, Shubham Agarwal, Sanket Biswas, Sara Shanian, Ying Zhang, Noah Bolger, Kurt MacDonald, Simon Fauvel, Sathwik Tejaswi, Srinivas Sunkara, Joao Monteiro, Krishnamurthy DJ Dvijotham, Torsten Scholak, Nicolas Chapados, Sepideh Kharagani, Sean Hughes, M. Özsu, Siva Reddy, Marco Pedersoli, Yoshua Bengio, Christopher Pal, Issam Laradji, Spandana Gella, Perouz Taslakian, David Vazquez, Sai Rajeswar

arXiv:2412.04626v2cs.LGcs.CL

TL;DR

Open multimodal document datasets remain limited by restrictive or unclear licensing and insufficient coverage of complex documents and long structured outputs. BigDocs addresses this gap with a curated 7.5M-pair dataset, a 10-task benchmark suite, and supporting tools and models. Models trained with BigDocs outperform models trained on existing datasets, while the authors identify context length and other limitations.

  • Problem

    Existing datasets are often closed-source or restrictively licensed and insufficiently cover complex visual documents and long structured outputs.

    Source: §1 · p. 2

  • Method

    BigDocs curates a 7.5M-pair permissive dataset, introduces BigDocs-Bench and evaluation metrics, and provides tools, metadata, and models for document-related multimodal tasks.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    Models trained on BigDocs outperform those trained on existing datasets, and BigDocs-Bench training enhances long, structured output generation.

    Source: §1 · p. 3, §5.2 · p. 9, §6 · p. 11

  • Takeaways & Limitations

    BigDocs provides openly released resources for developing and evaluating multimodal models on document understanding and structured-output tasks.

    Source: §1 · p. 3, §6 · p. 11

    An 8192-token context limit restricts performance on long structured outputs such as HTML and SVG, while some public benchmarks remain suboptimal.

    Source: §6 · p. 11

Abstract

from arXiv · show

Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding workflows, extracting data from documents, and summarizing reports. Code generation tasks that require long-structured outputs can also be enhanced by multimodality. Despite this, their use in commercial applications is often limited due to limited access to training data and restrictive licensing, which hinders open access. To address these limitations, we introduce BigDocs-7.5M, a high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks. We use an efficient data curation process to ensure our data is high-quality and license-permissive. Our process emphasizes accountability, responsibility, and transparency through filtering rules, traceable metadata, and careful content analysis. Additionally, we introduce BigDocs-Bench, a benchmark suite with 10 novel tasks where we create datasets that reflect real-world use cases involving reasoning over Graphical User Interfaces (GUI) and code generation from images. Our experiments show that training with BigDocs-Bench improves average performance up to 25.8% over closed-source GPT-4o in document reasoning and structured output tasks such as Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a preference for outputs from models trained on BigDocs over GPT-4o. This suggests that BigDocs can help both academics and the open-source community utilize and improve AI tools to enhance multimodal capabilities and document reasoning. The project is hosted at https://bigdocs.github.io .

1 INTRODUCTION

BigDocs addresses limited open access to training data for complex visual-document understanding and long-structured output generation. It introduces an open dataset, benchmark suite, and models spanning document extraction, understanding, creation, and manipulation.

  • 7.5 million image-text pairs support document information extraction, document understanding, and conversion of visual data into HTML, LaTeX, and JSON.Source: §1 · p. 2
  • 80% of 133 surveyed datasets had non-permissive or unclear licensing, motivating BigDocs’ focus on permissive licenses, traceable metadata, and documented transformations.Source: §1 · p. 2
  • BigDocs-Bench provides 10 downstream tasks with train, validation, test, and hidden-test splits for structured output generation and GUI reasoning.Source: §1 · p. 3
  • 88% and 63% of user evaluations preferred BigDocs-trained outputs over Phi3.5 Instruct and GPT-4, respectively.Source: §1 · p. 3
  • BigDocs releases datasets, models, documentation, and tools under an accountability, responsibility, and transparency commitment.Source: §1 · p. 3

2 RELATED WORK

BigDocs addresses gaps in open, permissively licensed multimodal document data by consolidating diverse datasets for document understanding and model development.

  • Existing multimodal datasets often emphasize photographs, while document-focused resources may lack permissive licenses, open release, or public availability.Source: §2 · pp. 3–4
  • BigDocs provides 7.5M permissively licensed image-text pairs from 16 academic datasets and other open platforms for diverse document understanding tasks.Source: §2 · p. 4
  • BigDocs prioritizes transparent curation by documenting licensing complexities and tracking dataset provenance and development decisions.Source: §2 · p. 4

3 BIGDOCS-7.5M

BigDocs-7.5M is a curated, license-permissive dataset assembled through documented acquisition, filtering, metadata tracking, contamination assessment, and quality control.

3.1 DATASET CURATION PROCESS

  • The collection began with 133 public vision-language datasets selected for relevance, quality, diversity, and document-related task coverage.Source: §3.1 · p. 4
  • Datasheets record ownership, sources, annotations, licenses, formats, document properties, and use-case groupings, including separate image and annotation licensing.Source: §3.1 · pp. 4–5
  • Only 20% of considered data was retained under the licensing filter, while 7.5% moderately restrictive and 72.5% non-permissive data was discarded.Source: §3.1 · p. 5

3.2 BIGDOCS TOOLKIT: DATA PREPROCESSING, FILTERING, AND CONSOLIDATION

  • The BigDocs Toolkit modularizes preprocessing, filtering, contamination management, metadata management, and dataset loading for scalable curation.Source: §3.2 · p. 5
  • A unified metadata framework records each sample’s properties, licenses, sources, and transformations to support traceability and transparency.Source: §3.2 · p. 5
  • BigDocs explicitly searches for evaluation-data overlap, reporting contamination findings rather than assuming complete independence from downstream benchmarks.Source: §3.2 · p. 5, Figure 3 · p. 5

3.2.1 THE RESULTING BIGDOCS-7.5M

  • The toolkit includes ten-task benchmark statistics and supports low-quality-data removal plus best-effort licensing compliance for foundational-model training.Source: Table 1 · p. 6, §3.2.1 · p. 6
  • BigDocs-7.5M contains 7.5M training image-text pairs, 500k validation pairs, and 470k test pairs spanning OCR, structured parsing, captioning, and question-answering.Source: §3.2.1 · p. 6

4 BUILDING BIGDOCS-BENCH

BigDocs-Bench introduces ten benchmarks for reasoning over visually rich documents, GUIs, charts, and images that must be converted into structured outputs. Its datasets combine real-world or carefully curated inputs with filtering and verification procedures.

4 BUILDING BIGDOCS-BENCH

  • BigDocs-Bench contains ten specialized tasks with 329k training, 11k validation, 10k testing samples, and an additional hidden test set.Source: §4 · p. 6
  • The suite evaluates visual understanding and structured generation through tasks including GUI2UserIntent, Image2Flow, Chart2Caption, and Screenshot2HTML.Source: §4 · p. 6

4.1 BIGDOCS-BENCH TASKS SUITE

  • Screenshot2HTML uses 10,838 real-world website screenshots paired with HTML, emphasizing layout fidelity and semantic correctness after asset and script normalization.Source: §4.1 · p. 6
  • Table2LaTeX pairs 79,669 rendered table images with source LaTeX code and captions to evaluate table extraction and LaTeX generation.Source: §4.1 · pp. 6–7
  • Image2SVG contains 200k raster images paired with SVG code or descriptions, emphasizing complex vector-image reconstruction and scalability.Source: §4.1 · p. 7
  • Image2Flow maps 10,000 flowchart images to GraphViz or JSON representations, testing structural extraction from raw images rather than preprocessed flowcharts.Source: §4.1 · p. 7
  • Chart2Markdown, Chart2Caption, GUI2Summary, GUI-VQA, and GUI2UserIntent cover chart extraction, chart summarization, screenshot description, visual question answering, and interaction-intent recognition.Source: §4.1 · p. 7

4.2 FILTERING AND QUALITY

  • Filtering removes harmful content, corrupted images, annotation mismatches, and PII; test samples receive at least two-annotator verification, while training checks achieved a 99% pass rate.Source: §4.2 · p. 7

5 TRAINING MULTIMODAL MODELS ON BIGDOCS

The authors train multimodal models with BigDocs and evaluate them on general document benchmarks and BigDocs-Bench tasks involving structured outputs and GUI reasoning. BigDocs-trained models outperform comparison systems across many tasks, with human evaluations favoring BigDocs models on Table2LaTeX and Screenshot2HTML.

5.1 EXPERIMENTAL SETUP

  • The experiments use continual pretraining followed by downstream finetuning, with BigDocs or DocStruct4M for corpus training and task-specific datasets for alignment.Evaluation covers general document benchmarks and BigDocs-Bench, including structured-output metrics such as Tree Edit Distance, TeXBLEU, DINOScore, LST F1, RMSF1, and Rouge-L F1.Source: §5 · p. 8, §5.1 · pp. 8–9

5.2 QUANTITATIVE RESULTS

  • BigDocs-trained models achieve higher average scores than DocStruct4M-trained models across general document benchmarks, despite lower contamination rates on most benchmarks.The results indicate better generalization and robustness without complex task-specific optimizations.Source: §5.2 · p. 9
  • Phi3.5-V4B with BigDocs is the top performer on 8 of 10 BigDocs-Bench tasks, achieving an average score of 50.46.Its performance is weaker on Image2SVG, where it trails LLaVA-NeXT-7B with BigDocs by 25.98 points.Source: §5.2 · p. 9
  • BigDocs models outperform open and closed baselines particularly on flow generation, GUI reasoning, and image-to-LaTeX generation tasks.The performance gap is narrower for Screenshot2HTML, Image2SVG, and Chart2Caption.Source: §5.2 · p. 9, Table 3 · p. 10

5.3 HUMAN EVALUATIONS AND QUALITATIVE RESULTS

  • Phi3.5 BigDocs wins 88% against Phi3.5 Instruct on Table2LaTeX and 65% on Screenshot2HTML in human evaluation.Against GPT-4o, it records a 63% win rate with a 31% draw rate on Table2LaTeX and a 36% win rate with a 7% draw rate on Screenshot2HTML.Source: §5.3 · p. 10, Figure 5 · p. 10
  • Qualitative examples show visually consistent outputs and valid code across Chart2Markdown, Table2LaTeX, Image2SVG, captioning, and VQA tasks.The BigDocs-trained version is highlighted as strong in captioning and VQA compared with GPT-4o.Source: §5.3 · pp. 10–11

6 CONCLUSION

The conclusion presents BigDocs-7.5M as a large-scale, license-permissive dataset and BigDocs-Bench as a ten-task suite for multimodal document and long-format code generation. Experiments report stronger performance than existing datasets, while BigDocs-Bench training improves structured output capabilities.

6 CONCLUSION

  • BigDocs-7.5M is a large-scale, license-permissive dataset, and BigDocs-Bench contains 10 downstream tasks for generating long-format code from images.The artifacts are intended for practical real-world document and code-related applications.Source: §6 · p. 11
  • Models trained on BigDocs outperform models trained on existing datasets, while BigDocs-Bench training enhances long, structured output generation.All BigDocs artifacts are planned for free release under permissive licenses.Source: §6 · p. 11

ETHICS AND REPRODUCIBILITY STATEMENT

The paper emphasizes responsible dataset curation, privacy protection, and reproducibility through filtering, documentation, and public release. It also acknowledges residual risks involving offensive language, malicious code, and human-evaluation data collection.

ETHICS AND REPRODUCIBILITY STATEMENT

  • The authors filter harmful content and remove personally identifiable information from compiled datasets, but cannot guarantee that generated language will be non-offensive.Human evaluations collected no PII.Source: ETHICS AND REPRODUCIBILITY STATEMENT · p. 11
  • Generated code may contain malicious snippets, so developers are encouraged to implement protection protocols.Source: ETHICS AND REPRODUCIBILITY STATEMENT · p. 11
  • All code, datasets, model weights, data sheets, metadata, hyperparameters, experimental setups, and evaluation metrics will be publicly released for replication.The human-evaluation environment and comparison instructions are also documented.Source: ETHICS AND REPRODUCIBILITY STATEMENT · p. 11

A.1 BIGDOCS-7.5M TASKS

BigDocs organizes document understanding into OCR, structured parsing and extraction, captioning and summarization, and question-answering tasks. These tasks map visual document content into text, structured representations, summaries, or answers.

A.1 BIGDOCS-7.5M TASKS

  • OCR-related tasks convert document images into machine-readable text and learn to map textual information within images.They include bounding-box-to-text and text-to-bounding-box transformations.Source: §A.1 · p. 18
  • Structured parsing and extraction tasks transform tables, forms, and charts into formats such as JSON or Markdown while handling complex layouts.Source: §A.1 · p. 18
  • Captioning and summarization tasks generate concise descriptions or overviews of visual or textual content.Examples include figures, charts, and document sections.Source: §A.1 · p. 18
  • Question-answering tasks answer questions over structured or visual data, including multi-turn questions requiring related responses.Source: §A.1 · p. 18

A.2 DATASETS INCLUDED IN BIGDOCS-7.5M

BigDocs-7.5M incorporates datasets spanning OCR, structured parsing, captioning, summarization, and question answering over documents, tables, charts, and diagrams.

A.2 DATASETS INCLUDED IN BIGDOCS-7.5M

  • The included datasets support four core areas: OCR-related tasks, structured parsing and extraction, captioning or summarization, and question answering.These tasks cover document text, layouts, tables, charts, figures, and diagrams.Source: §A.2 · pp. 18–19
  • Document and scene-text datasets provide OCR and layout resources, including CORD, TextOCR, COCO-Text, OCRVQA, CDIP-1M, and ArxivOCR.CORD also supports receipt entity extraction with text, category, and location represented as JSON.Source: §A.2 · pp. 18–19
  • The collection combines datasets for scholarly papers, receipts, business documents, charts, tables, scene text, and scanned document collections.This breadth supports document understanding across varied visual and textual formats.Source: §A.2 · pp. 18–19
  • Table-focused datasets such as PubTables-1M, TableBank, TabFact, WikiTQ, and UniChart support extracting, parsing, questioning, or converting tabular content into structured formats.Their outputs include structured representations such as Markdown and answers grounded in table data.Source: §A.2 · pp. 18–19
  • Other included resources address captioning, summarization, and visual interpretation through Datikz, ArxivTableCap, SVGCap, FigureQA, and related chart or figure tasks.These datasets describe diagrams, tables, figures, and vector graphics or answer questions about charts.Source: §A.2 · pp. 18–19
  • Several datasets are curated or filtered for permissive licensing, including ArxivOCR, ArxivTableCap, and CDIP-1M.ArxivOCR and ArxivTableCap retain papers with CC-BY 4.0 or CC0 licenses, while CDIP-1M is curated from IIT-CDIP.Source: §A.2 · p. 19

A.3 ASSESSING CONTAMINATION

The contamination analysis combines image-embedding retrieval with human judgments to interpret overlap between training and evaluation data. It reports direct training-split dependencies, indirect overlap risks, and lower apparent contamination for BigDocs-7.5M than DocStruct4M on most comparisons.

A.3 ASSESSING CONTAMINATION

  • BigDocs-7.5M directly depends on the training splits of TabFact, WTQ, and TextVQA, while indirect overlap may arise when evaluation and training sets share source material.DocStruct4M reports similar dependencies involving DocVQA, InfoVQA, DeepForm, KLC, ChartQA, TabFact, WTQ, and TextVQA.Source: §A.3 · p. 19
  • The automatic analysis embeds reference and evaluation images with pretrained CLIP and retrieves each evaluation image’s nearest reference image by cosine similarity.Similarity closer to 1.0 makes reference overlap more likely, and exact similarity of 1.0 indicates an exact match.Source: §A.3 · pp. 19, 21
  • Except for MMMU and DudeMini, BigDocs-7.5M appears less contaminated than DocStruct4M under the reported cosine-similarity metrics.Lower values indicate less contamination, while higher similarity thresholds are more pertinent.Source: Table 4 · p. 20, Figure 6 · p. 20
  • Human calibration finds that most DocVQA samples below 0.96 are different, whereas most samples above 0.98 are not.The analysis expects fewer than 10% of samples below 0.97 to be same-template and fewer than 1% to be same-document.Source: Figure 7 · p. 21
  • The human labeling distinguishes different images, same templates, and same documents, treating recurring forms and related document versions as same-template cases.Different scans of the same memorandum are treated as the same document despite visual changes from stamps, crops, rotations, or artifacts.Source: §A.3 · pp. 21–22
  • The authors retain all samples regardless of contamination status and release overlap annotations to support future detection strategies.Cosine-similarity interpretations other than exact matches require grounding in human evaluations.Source: §A.3 · pp. 21–22

A.4 DESCRIPTION OF DOWNSTREAM TASKS PROPOSED IN BIGDOCS-BENCH

BigDocs-Bench is introduced as a suite of downstream tasks that transform visual inputs into structured outputs for multimodal evaluation.

A.4 DESCRIPTION OF DOWNSTREAM TASKS PROPOSED IN BIGDOCS-BENCH

  • BigDocs-Bench formally specifies the downstream tasks proposed for multimodal models.Source: §A.4 · p. 22
  • The suite targets transformations from visual inputs to structured outputs.Source: §A.4 · p. 22
  • The task descriptions provide the formal basis for evaluating the proposed benchmark problems.Source: §A.4 · p. 22

A.4.1 IMAGE2FLOW

Image2Flow converts a flowchart image into structured JSON representing its nodes and directed edges. The dataset is generated through randomized graph specifications, Graphviz rendering, and JSON serialization.

A.4.1 IMAGE2FLOW

  • Image2Flow takes a flowchart image as input and outputs JSON containing the flowchart’s nodes and edges.The output represents the flowchart structure in a machine-readable format.Source: §A.4.1 · p. 22
  • The formal target is a graph G = (V, E), with node attributes and directed edges carrying source, destination, and optional labels.Nodes include identifiers, labels, shapes, and descriptions; edges connect source and destination node IDs.Source: §A.4.1 · p. 22
  • The Image2Flow dataset contains 10,000 samples, each pairing a flowchart image with a Graphviz file and JSON representation.Source: §A.4.1 · p. 22
  • Generation randomizes each graph’s 5–15 node count, node-type distribution, and flow direction before producing Graphviz data with LLaMA 3.1.Directions include bottom-to-top, top-to-bottom, left-to-right, and right-to-left.Source: §A.4.1 · p. 22
  • The generation prompt requires directed, coherent enterprise workflows with outgoing edges for every node and at least two outgoing edges for diamond nodes.Source: §A.4.1 · p. 22
  • Generated Graphviz files are converted into PNG images and JSON files that preserve node attributes and edge labels.Random colors and styles are applied to reduce bias toward specific visual styling.Source: §A.4.1 · p. 23

A.4.2 SCREENSHOT2HTML

This section details BigDocs-Bench’s multimodal document tasks, including dataset construction, training and evaluation procedures, licensing analysis, human evaluation, and metadata traceability. It examines task relationships, curation effects, qualitative behavior, and persistent challenges in structured generation, information extraction, and Screenshot2HTML.

A.4.2 SCREENSHOT2HTML; A.4.3 TABLE2LATEX; A.4.4 IMAGE2SVG & TEXT2SVG

  • The curated tasks include 11,000 website samples averaging 20.3 HTML elements, over 95,000 table images rendered from LaTeX code, and the top 100k image-text examples filtered by image entropy and CLIP Score.Source: §A.4.2 · p. 24, §A.4.3 · p. 24, §A.4.4 · p. 24

A.4.5 CHART2MARKDOWN

  • Chart2Markdown maps chart images to markdown data tables and contains 6,516 recent Statista charts with corresponding tables and human-written summaries.The charts were collected from 2023 and 2024 and were selected to reduce overlap with UniChart and ChartQA while reflecting current facts and trends.Source: §A.4.5 · pp. 24–25

A.4.6 CHART2CAPTION

  • Chart2Caption uses chart images, generation code, and dataset descriptions to produce captions, with 1,496 image-code pairs collected by executing Kaggle analytics notebooks.Captions were generated with InternVL2-26B (Chen et al., 2023; 2024).Source: §A.4.6 · p. 25

A.4.7 GUI2USERINTENT; A.4.8 GUI2SUMMARY; A.4.9 GUI-VQA

  • GUI tasks cover click-based user-intent identification, paragraph-level screenshot summaries, and factual question answering over website interfaces.GUI2UserIntent repurposes clickable text and bounding boxes from SeeClick (Cheng et al., 2024); GUI2Summary uses InternVL2-8B (Chen et al., 2023) for descriptions of content, layout, colors, and style, while GUI-VQA converts sampled summary sentences into factual QA pairs with LLaMA 3.1-8b (Dubey et al., 2024).Source: §A.4.7 · p. 25, §A.4.8 · p. 25, §A.4.9 · p. 26

A.5 DETAILS ON TRAINING VISION-LANGUAGE MODELS ON BIGDOCS

  • BigDocs training uses continual pretraining and supervised finetuning after existing general vision-language checkpoints, with categorical cross-entropy next-token prediction throughout the stages.Experiments use 64 H100 80GB GPUs, a total batch size of 256, learning rate 2e-5, one continual-pretraining epoch, and three finetuning epochs.Source: §A.5 · p. 27

A.6 DETAILS ON BIGDOCS-BENCH’S EVALUATION; A.6.1 PREPROCESSING; A.6.2 METRICS

  • BigDocs-Bench preprocessing normalizes generated structured outputs before task-specific evaluation, while metrics include DOM Tree Edit Distance for Screenshot2HTML, TeXBLEU for Table2LaTeX, RMSF1 for Chart2Markdown, DINOv2Score for Image2SVG, and Rouge-L F1 for summaries and VQA.Flowchart outputs are converted to triplets for Flowchart Triplet F1, and invalid GraphViz, JSON, or SVG generations receive zero scores where specified.Source: §A.6.2 · pp. 28–29

A.7 MAKING BIGDOCS LICENSE-PERMISSIBLE

  • BigDocs retains only sources meeting permissive licensing criteria after reviewing licenses for both images and annotations, and it documents over 100 candidate datasets in a systematic datasheet.The analysis distinguishes permissive licenses such as CC0, MIT, and Apache 2.0 from restrictive terms and cautions that Fair Use is not unrestricted permission.Source: §A.7 · p. 29, Table 5 · p. 30

A.8 HUMAN EVALUATION; A.9 DETAILS OF UNIFIED METADATA FRAMEWORK

  • Human evaluation uses anonymous, randomized pairwise comparisons by diverse evaluators, while BigDocs metadata standardizes license, origin, and optional features keys for transparency and traceability.Key authors were excluded from evaluation, and the comparison interface asked users which model produced the better output for each image.Source: §A.8 · p. 29, §A.9 · p. 30

A.9 DETAILS OF UNIFIED METADATA FRAMEWORK

  • The unified metadata framework records licenses, origins, sample features, and reusable tracking instructions and transformations to support traceability, consistency, and downstream task selection.Mandatory fields include image_license, annotation_license, data_id, and tracking_instruction_id; optional fields capture transformations, URLs, paths, and licensing notes.Source: §A.9 · pp. 30–32

A.10 QUALITATIVE RESULTS

  • Qualitative comparisons found the BigDocs-trained Phi-3.5-Vision model more concise and accurate on VQA, capable of perfect LaTeX tables, and precise at generating HTML from screenshots.Tables 9-28 provide broader model comparisons and error analyses across BigDocs-Bench tasks.Source: §A.10 · p. 32

A.12.1 ABLATIONS ON DATA CURATION

  • Removing VQA formatting reduced average performance by 9.19%, with especially large declines on InfoVQA (-15.13), DeepForm (-30.33), WTQ (-14.28), and TabFact (-15.94).The authors hypothesize that VQA formatting improves instruction tuning and multitask generalization.Source: §A.12.1 · p. 34
  • OCR removal caused a 2.71% average drop, increased hallucinated values in precise extraction, and reduced DeepForm’s true negative rate from 62% to 0%, although TableVQA and TextVQA improved.Declines included InfoVQA (-10.02), DeepForm (-16.78), and TabFact (-7.48), while TableVQA (+3.06) and TextVQA (+3.35) improved.Source: §A.12.1 · p. 34
  • Reducing captioning and other text-image alignment tasks lowered performance by 3.4%, particularly on InfoVQA (-12.07), DeepForm (-17.61), and TabFact (-8.74).The decline primarily reflected errors parsing values for arithmetic operations and converting varied visual or textual formats into percentages.Source: §A.12.1 · p. 35

A.12.2 ABLATION ON CONTEXT LENGTH

  • Longer context did not improve Screenshot2HTML: greedy decoding stayed at or below 10.99, whereas beam search with length penalty reached 15.80 without benefiting from larger contexts.At 8k tokens, HTML was often truncated but remained compilable; repetitive <div> elements and long URLs were common hallucinations, while length penalties improved generation.Source: §A.12.2 · p. 35

A.13 DETAILS IN DATA VERIFICATION ON BIGDOCS-BENCH

  • BigDocs-Bench verification combined automatic filtering, test-set human review, iterative training-split filter refinement, and proportional sampling, yielding a 99% sampled training pass rate.The sampled pass rate was 100% for every task except GUI-VQA, which reached 92%; reviewers checked integrity, synthetic-sample relevance, NSFW content, and PII.Source: §A.13 · p. 36, §A.13.1 · p. 36, §A.13.2 · p. 36

A.14 RESULTS ON LARGER MODELS

  • GPT-4o led larger-model general document benchmarks with an average score of 64.62, followed by Qwen2-VL-72B at 58.40 and GeminiPro-1.5 at 57.05.The comparison is reported for models exceeding 8 billion parameters.Source: Table 8 · p. 37, §A.14 · p. 37

A.15 CORRELATION ANALYSIS OF BIGDOCS-BENCH WITH OTHER VLM BENCHMARKS

  • BigDocs-Bench tasks cluster separately from other document and general VLM benchmarks, indicating distinctive evaluation characteristics.PCA places the aggregated BigDocs-Bench score far from other benchmarks, while most tasks remain separated in the broad VLM space.Source: §A.15.1 · p. 38, §A.15.2 · p. 38

A.15.1 CORRELATION WITH DOCUMENT BENCHMARKS

  • Document-benchmark analysis groups BigDocs-Bench code-generation tasks together, separates VQA tasks, and shows stronger correlations for Image2SVG and Screenshot2HTML with KLC, DeepForm, and TabFact.HTML and SVG cluster together, likely because of their long output sequences, while DeepForm and KLC form a separate cluster.Source: §A.15.1 · p. 38, Figure 14 · p. 39, Figure 15 · p. 39

A.15.2 CORRELATION WITH GENERAL VLM BENCHMARKS

  • General-VLM analysis identifies separate clusters for general vision-language, document understanding, and GUI/table tasks, with BigDocs-Bench tasks showing minimal correlation with other benchmark types.Within BigDocs-Bench, tasks also cluster by type, including VQA, Flows, Charts, and GUIs, while general VLM benchmarks form a dense cluster and document benchmarks are more sparse.Source: §A.15.2 · p. 38, Figure 16 · p. 40, Figure 17 · p. 40

Phi3.5-V-4B

  • BigDocs models still make task-specific errors, including factual mistakes, hallucinations, formatting failures, and incorrect question-mark rendering in Table2Latex.OCR and caption-reduction variants can hallucinate or select unrelated regions, whereas the full BigDocs model answers several extraction questions correctly.Source: Table 9 · p. 41, Table 20 · p. 48, Table 21 · p. 49

GPT4o

  • BigDocs-trained models produce accurate structured outputs in qualitative examples, preserving table structure and improving Screenshot2HTML rendering compared with untuned baselines.Examples report correct GUI-VQA and GUI2Intent answers, while Table2Latex outputs preserve rows, columns, and font styles; BigDocs fine-tuning substantially improves Image2HTML structure.Source: Table 11 · p. 43, Table 13 · p. 44, Table 17 · p. 47
Loading 2412.04626v2…