MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding

• Existing benchmarks either mix few fine-grained samples with broader evaluations or restrict assessment to object-level details in natural images.This leaves comprehensive evaluation of finer-grained visual understanding difficult.Source: §1 · p. 2
• MMDocBench uses document images because they combine multi-granularity content with multi-type information that supports fine-grained perception and reasoning.Documents include elements such as text, tables, charts, and diagrams, often occupying small but important image regions.Source: §1 · pp. 2–3
• MMDocBench defines 15 OCR-free tasks: nine for fine-grained perception and six for reasoning over integrated visual information.Perception tasks cover recognition, localization, detection, and extraction; reasoning tasks include arithmetic, logical, spatial, counting, comparison, and sorting.Source: §1 · p. 2
• Across 16 evaluated LVLMs, MMDocBench reveals significant challenges, especially in region prediction, while answer performance differs between open- and closed-source models, whose region-prediction performance is comparable.Localization and detection tasks are significantly more challenging than other tasks; region prediction is typically harder for documents containing tables, charts, and figures, whereas answer prediction shows no significant variation across document types.Source: §1 · p. 3
• The benchmark contains 2,400 document images, 4,338 QA pairs, and 11,353 supporting regions drawn from varied document types.Supporting regions enable checking whether predictions are grounded in the associated image areas and make responses more interpretable.Source: §1 · pp. 2–3

2.1 LARGE VISION-LANGUAGE MODELS

• LVLMs commonly combine a visual encoder and adapter with language models, and are trained through large-scale pre-training followed by supervised instruction tuning.Representative systems include Flamingo, BLIP-2, LLaVA, MiniGPT-4, mPLUG-Owl, CogVLM, MiniCPM-V, Monkey, and InternVL.Source: §2.1 · p. 3
• Recent document-focused LVLMs improve document understanding through higher-resolution inputs, richer document training data, window patching, and visual-token compression.These approaches target the fine-grained details required for document images, motivating dedicated evaluation.Source: §2.1 · p. 3

2.2 EVALUATION BENCHMARKS FOR LVLMS

• Existing multimodal benchmarks cover broad capabilities, but their fine-grained samples are limited, mixed with other data, and generally lack supporting-region prediction.Natural-image benchmarks such as Visual7W, RefCOCO, GVT-bench, and MMBench mainly assess object-level recognition and interpretation.Source: §2.2 · p. 4

2.3 VISUAL GROUNDING IN LVLMS

• Visual grounding localizes a region relevant to a natural-language description, with prior LVLMs using region-level or pixel-level grounding approaches.These systems and evaluations largely focus on object-level natural-image tasks rather than the varied granularities of document images.Source: §2.3 · p. 4
• MMDocBench extends region-level grounding to OCR, key information extraction, and table recognition through diverse OCR-free document understanding tasks.The benchmark evaluates grounding for document-specific details rather than only natural-image objects.Source: §2.3 · p. 4

3.1 PROBLEM DEFINITION

• LVLMs must output an answer a and supporting bounding-box regions R for each question about a document image.Each region uses top-left and bottom-right coordinates [x1, y1, x2, y2].Source: §3.1 · pp. 4–5

3.2 CONSTRUCTION PIPELINE

• MMDocBench’s taxonomy targets fine-grained discrimination and diversity across capabilities, content granularities, and document types.The design explicitly avoids treating the entire image as a single unit.Source: §3.2 · p. 5
• The benchmark contains 15 tasks and 48 subtasks, covering nine fine-grained perception tasks and six fine-grained reasoning tasks.Covered document types include research papers, book covers, financial reports, receipts, Wikipedia tables, charts, infographics, and industry documents.Source: §3.2 · p. 5
• Construction combines purpose-built and source-dataset subtasks, with document images and QA pairs selected from varied datasets and generally capped at 100 images per subtask.Three subtasks are created from selected document images, while other subtasks transform source input-output pairs into templated QA pairs.Source: §3.2 · pp. 5–6
• Ground-truth regions are normalized to [0, 1000] coordinates and annotate either answer locations or all supporting evidence used for reasoning.OCR assists perception-region matching, while reasoning regions come from existing evidence annotations or manual annotation; three review rounds verify correctness.Source: §3.2 · pp. 6–7

3.3 STATISTICS AND ANALYSIS

• 2,400 document images yield 4,338 QA pairs and 11,353 supporting regions, averaging 10.1 words and about 2.61 regions per question.Supporting-region centers span the image without clear positional bias, while most region areas lie between 1,000 and 10,000, indicating varied granularity.Source: §3.3 · p. 7

4.1 EXPERIMENTAL SETUP

• The evaluation tests 13 open-source and 3 proprietary LVLMs with zero-shot instructions, JSON answer-and-region outputs, and EM, F1, and IOU metrics.The metrics are macro-averaged from sub-task scores to obtain task, capability, and overall performance.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

• GPT-4o reaches 66.40% EM for answer prediction, while Qwen-VL-Max leads region prediction with only 11.44% IOU, exposing the benchmark’s grounding challenge.GPT-4o exceeds the best open-source answer-prediction model, Llava-OV-Chat-72B, at 58.93% EM, whereas TextMonkey and Qwen2-VL-7B-Instruct achieve 8.93% and 9.87% IOU, respectively.Source: §4.2 · pp. 8–9, Table 2 · p. 8
• GPT-4o consistently leads answer prediction across perception and reasoning tasks, while larger models tend to favor reasoning and smaller models tend to favor perception.The reported pattern is attributed to reasoning capabilities improving significantly with model size.Source: §4.2 · p. 9
• Region prediction is especially difficult for reasoning: Qwen-VL-Max reaches only 4.27% IOU, while TextMonkey leads perception at 19.22% but fails to output reasoning-task regions.The authors associate TextMonkey’s failure with training focused on perception-related grounding samples and instructions.Source: §4.2 · p. 9

4.3 ANALYSIS ON DIFFERENT TASKS

• Across tasks, GPT-4o and Qwen-VL-Max usually rank first and second in F1, except that LLaVA-OV-Chat-72B leads Logical and Spatial Reasoning.For IOU, Qwen-VL-Max leads most tasks, while TextMonkey leads Forgery Detection, Infographic QA, and Text Localization; Text Recognition is the exception where Qwen-VL-Max reaches about 71.2% IOU.Source: §4.3 · p. 9, Figure 5 · p. 9, Figure 6 · p. 9
• Document Forgery Detection requires identifying inconsistent word(s) against other words in an image and achieves 1.8% IOU for region prediction.Answer-prediction performance is poor for all LVLMs on Text Localization and Table Cell Localization.Source: §4.3 · p. 10

4.4 ANALYSIS ON DIFFERENT DOCUMENT TYPES

• The benchmark groups images into General, Table-Based, Table-Text, Chart-Based, and Infographic-Based documents for document-type analysis.Each sub-task’s images are kept within the same category.Source: §4.4 · p. 10
• GPT-4o leads answer prediction on General and Table-Based documents, whereas Qwen-VL-Max leads the other three types and most region-prediction categories.TextMonkey leads region prediction on Infographic-Based documents, while Qwen-VL-Max leads the remaining four types.Source: §4.4 · p. 10, Table 3 · p. 10

• GPT-4o achieves the highest answer prediction score at 66.40% EM, while Qwen-VL-Max achieves the best region prediction score at 11.44% IOU.Source: §5 · p. 10
• The benchmark contains 15 tasks and 48 sub-tasks requiring detailed interpretation of image content to answer questions.Source: §5 · p. 10
• Supporting-region annotations evaluate whether models correctly ground their predictions in associated image regions.Source: §5 · p. 10
• Open-source LVLMs perform competitively with proprietary models in region prediction despite a substantial answer-prediction gap.Source: §5 · p. 10

• Table 4 summarizes the key statistics of MMDocBench.Source: Table 4 · p. 17, §A.1 · p. 17

• Tables 5 and 6 compare LVLMs across fine-grained visual perception tasks using F1 and IOU, respectively.Source: Table 5 · p. 18, Table 6 · p. 18
• The perception-task comparisons cover text recognition, table recognition, text localization, table-cell localization, key information extraction, forgery detection, and document, chart, and infographic question answering.Source: Table 5 · p. 18, Table 6 · p. 18
• Tables 7 and 8 compare LVLMs across fine-grained visual reasoning tasks using F1 and IOU, respectively.Source: Table 7 · p. 19, Table 8 · p. 19

• Each sub-task’s document images are divided into General Document, Table-Based Document, Table-Text Document, Chart-Based Document, and Infographic-Based Document categories.Source: §A.3 · p. 20, Table 9 · p. 20

A.4.1 TEXT RECOGNITION

• Text recognition and localization cover reading visible text, finding a phrase, and extracting text from a specified bounding box.Examples include recognizing “Chaos Computer Club” and locating or reading “Miami, Florida.”Source: §A.4.1 · p. 21, §A.4.2 · p. 21

A.4.3 TABLE RECOGNITION

• Table tasks represent tables as cell contents with cell-level boxes and ask for the contents or location of a specified cell.The table example includes structured headers, values, and bounding boxes for each cell.Source: §A.4.3 · p. 22, §A.4.4 · p. 22

A.4.5 KEY INFORMATION EXTRACTION

• Document-oriented tasks target key-value extraction, forged-text detection, and question answering with localized document evidence.Examples ask for a total amount, falsified words, and the document preparation date, with bounding boxes supplied for the relevant regions.Source: §A.4.5 · p. 23, §A.4.6 · p. 23, §A.4.7 · p. 24

A.4.8 CHART QUESTION ANSWERING

• The examples ask how many app publishers were in Apple’s App Store in 2017 and which contact number serves children and teens suffering from mental crisis, with bounding boxes specified.For the crisis question, the listed answer is 1-800-668-6868.Source: §A.4.8 · p. 24, §A.4.9 · p. 25

A.4.10 COUNTING

• Counting, arithmetic, and logical reasoning require combining localized evidence or applying task-specific reasoning to document content.Examples count years exceeding a threshold, subtract church-construction years, and identify the quality-assurance step in a COVID-19 test.Source: §A.4.10 · p. 25, §A.4.11 · p. 26, §A.4.12 · p. 26

A.4.13 SPATIAL REASONING

• Spatial, comparison, and sorting tasks evaluate positional, numerical, and ordering judgments grounded in explicitly identified document text.Examples determine whether a word is at the top, compare a finish value with 12, and identify the model with the least tie rate.Source: §A.4.13 · p. 27, §A.4.14 · p. 27, §A.4.15 · p. 28