PaperlayerView exact revision on arXiv

Source-linked AI summary

MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations

Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, Aixin Sun

arXiv:2407.01523v3cs.CVcs.CL

TL;DR

Long-context document understanding remains insufficiently evaluated because existing resources emphasize single pages or lack challenging cross-page, multimodal evidence. MMLONGBENCH-DOC addresses this gap with a lengthy-document benchmark and expert-reviewed questions, and experiments show that most current LVLMs struggle, often underperforming OCR-based LLM pipelines.

  • Problem

    Existing document-understanding benchmarks provide limited evidence about lengthy, information-dense documents requiring localization and cross-page multimodal reasoning.

  • Method

    MMLONGBENCH-DOC constructs a benchmark from 135 lengthy documents and 1,082 expert-annotated questions spanning multimodal evidence sources and document locations.

  • Results

    Most current LVLMs struggle, with GPT-4o reaching 44.9% F1 and most others performing worse than LLMs given lossy OCR-parsed documents.

  • Takeaways & Limitations

    The benchmark exposes substantial weaknesses in long-context LVLM document understanding while showing that end-to-end multimodal perception can benefit some models.

  • Takeaways & Limitations

    The benchmark’s questions are designed to require access to the relevant documents and exclude questions answerable without document information.

Abstract

from arXiv · show

Understanding documents with rich layouts and multi-modal components is a long-standing and practical task. Recent Large Vision-Language Models (LVLMs) have made remarkable strides in various tasks, particularly in single-page document understanding (DU). However, their abilities on long-context DU remain an open problem. This work presents MMLongBench-Doc, a long-context, multi-modal benchmark comprising 1,062 expert-annotated questions. Distinct from previous datasets, it is constructed upon 130 lengthy PDF-formatted documents with an average of 49.4 pages and 20,971 textual tokens. Towards comprehensive evaluation, answers to these questions rely on pieces of evidence from (1) different sources (text, image, chart, table, and layout structure) and (2) various locations (i.e. page number). Moreover, 33.2% of the questions are cross-page questions requiring evidence across multiple pages. 22.8% of the questions are designed to be unanswerable for detecting potential hallucinations. Experiments on 14 LVLMs demonstrate that long-context DU greatly challenges current models. Notably, the best-performing model, GPT-4o, achieves an F1 score of only 42.7%, while the second-best, GPT-4V, scores 31.4%. Furthermore, 12 LVLMs (all except GPT-4o and GPT-4V) even present worse performance than their LLM counterparts which are fed with lossy-parsed OCR documents. These results validate the necessity of future research toward more capable long-context LVLMs. Project Page: https://mayubo2333.github.io/MMLongBench-Doc

1 Introduction

MMLONGBENCH-DOC targets long-context document understanding, where LVLMs must localize heterogeneous evidence and reason across pages. It introduces a comprehensive benchmark and shows that current LVLM performance remains weak.

  • Motivation: Long documents add localization and cross-page comprehension challenges beyond single-page document-understanding evaluations.The benchmark targets retrieving information from massive heterogeneous content and combining multi-source evidence across pages.
  • Benchmark contribution: MMLONGBENCH-DOC contains 135 PDF documents across 7 domains, averaging 47.5 pages and 21,214.1 textual tokens.The benchmark combines documents from existing datasets with papers, brochures, and other sources, and uses expert annotation with three-round review.
  • Benchmark contribution: Answers require evidence from text, layout, charts, tables, or images, supporting multimodal evaluation across document locations.The benchmark defines evidence pages as pages containing information necessary to answer a question.
  • Results: 44.9% F1 is achieved by GPT-4o, while GPT-4V reaches 30.5%, showing that long-context document understanding remains difficult.The reported experiments evaluate 14 LVLMs and compare them with OCR-based LLM pipelines.
  • Results: Most LVLMs perform worse than LLMs using lossy OCR-parsed documents, although GPT-4o is the best-performing LVLM.The results indicate that current models remain weak at long-context, cross-modality document understanding.

2 Related Work

Prior document-understanding benchmarks and models largely emphasize single components, single pages, or short and low-density documents. MMLONGBENCH-DOC addresses the limited evaluation of lengthy, information-dense, cross-page multimodal understanding.

  • Document-understanding benchmarks: Many prior datasets focus on a single document component or a single page rather than full-document understanding.Examples include table, chart, and single-page question-answering datasets.
  • Document-understanding benchmarks: Existing multi-page datasets remain limited by short context, low information density, crowd-sourced annotations, or few cross-page and unanswerable questions.DUDE averages 5.3 pages, while SlideVQA uses 20-page but relatively low-density slide decks.
  • Models for document understanding: Document-understanding models include OCR-dependent two-stream architectures and OCR-free end-to-end approaches.The latter approach has become dominant with the advancement of LVLMs.
  • Long-context models: Equivalent page number measures generalized-document context length by comparing its pixels with the average page pixel count.The definition applies to website screenshots and posters converted to PNG at resolution 240.
  • Long-context models: Long-context LVLM evaluation remains limited compared with the broader body of work on long-context LLMs.Recent studies examine multi-image understanding, but the area is described as largely unexplored.

3 MMLONGBENCH-DOC

The benchmark collects diverse lengthy documents and expert-written questions, then applies multi-stage quality control. Its statistics explicitly cover document scale, evidence sources, page locations, cross-page reasoning, and unanswerability.

  • Construction: The benchmark uses a three-stage annotation pipeline covering collection, review, and benchmark statistics.The construction combines existing and newly collected documents across diverse sources and topics.
  • Document collection: 135 documents span 7 manually categorized types, with 76 documents from existing datasets and 59 newly collected.Document types include research reports, financial reports, academic papers, brochures, guidelines, industry files, and tutorials or workshops.
  • Question and answer collection: Questions follow standards requiring balanced evidence distributions and document dependence, while expert annotators revise existing questions and add new ones.New annotation emphasizes cross-page, unanswerable, table, chart, and image questions.
  • Quality control: Three-round semi-automatic quality control combines document-relevance detection, LVLM self-reflection, and annotator cross-checking.Cross-checking reports Cohen’s kappa of 0.42 for annotator identifications.
  • Dataset overview and analysis: 1,082 questions cover 135 documents averaging 47.5 pages and 21,214.1 tokens; 494 are single-page, 365 cross-page, and 223 unanswerable.Questions draw evidence from text, layout, tables, charts, and images, and 28.6% of cross-page questions use more than two evidence pages.

4 Evaluation

The evaluation protocol measures long-context document understanding with standardized answer extraction and scoring across multimodal evidence sources and page configurations. Results show that current LVLMs remain weak on lengthy documents, with performance varying by document type, evidence source, and evidence position.

  • Evaluation Protocol: The three-step protocol generates responses, extracts short answers with an LLM, and calculates generalized accuracy and F1 scores.The protocol targets document understanding rather than instruction following and balances answerable and unanswerable questions.
  • Evaluation Setup: 14 LVLMs are evaluated from page screenshots, while 10 LLMs receive lossy Tesseract OCR-parsed documents for comparison.OCR cannot perceive chart or image information, so the text-only documents do not fully preserve document content.
  • Main Results: 44.9% F1 is achieved by GPT-4o, versus 31.4% for GPT-4V, while other LVLMs reach about 20% or lower.The reported results characterize long-context document understanding as challenging and unsolved for current LVLMs.
  • Main Results: 4.2% and 6.4% absolute F1 degradations occur for Gemini-1.5-Pro and Claude-3 Opus when using vision inputs instead of OCR-parsed texts.Conversely, GPT-4o and GPT-4-turbo improve over parsed text by 14.4% and 5.3% F1, respectively.
  • Fine-grained Results: Document type affects model rankings: OCR-based LLMs are comparable or superior on industrial documents, academic papers, and brochures, while LVLMs lead on tutorials, research reports, and guidelines.The latter document types are described as requiring more extensive multimodal information.
  • Fine-grained Results: Only GPT-4o performs relatively consistently across evidence sources; other LVLMs underperform on charts and images relative to text and layout.LLMs are generally better or comparable on text and table questions but worse on questions involving other elements.
  • Evidence Position: Almost all models perform best when evidence appears on initial pages, with performance progressively declining as page index increases.Gemini-Pro-1.5 and Claude-3-Opus show particularly sharp declines.
  • Evidence Pages: All models score higher on single-page than cross-page questions, while models trade helpfulness against safety differently on unanswerable questions.GPT-4o and Claude-3 Opus answer more aggressively, whereas Gemini-1.5-Pro, DeepSeek-VL-Chat, and EMU2-Chat more often refuse uncertain questions.

5 Analysis & Discussion

Additional analyses isolate document length as a major challenge and compare performance when models receive only oracle evidence pages. The results indicate that reducing context helps substantially for some models but does not fully resolve long-context document understanding.

  • Oracle Setting: 820 answerable questions are evaluated using oracle evidence pages rather than whole documents for three representative LVLMs.This comparison tests how much of the benchmark difficulty is attributable to document context length.
  • Oracle Setting: More than 20% absolute performance degradation occurs for Gemini-1.5-Pro and InternLM-XC2-4KHD with lengthy documents compared with oracle-page inputs.For single-page questions, the performance difference reaches up to 30%.
  • Oracle Setting: Oracle-page performance remains about 40% for Gemini-1.5-Pro and 30% for InternLM-XC2-4KHD, while GPT-4o improves by only about 10%.The results show that shorter evidence context helps but does not fully solve the task.
  • Error Analysis: The error analysis categorizes 72 GPT-4o error predictions into seven types, including perceptual, irrelevant-answer, incomplete-evidence, hallucinated-evidence, extractor, reasoning, and knowledge errors.The distribution is presented in Figure 7.

6 Conclusion

MMLONGBENCH-DOC evaluates long-context document understanding in LVLMs and shows that lengthy visual contexts remain difficult. The benchmark is intended to support development of more powerful LVLMs for lengthy documents.

  • Conclusion: MMLONGBENCH-DOC evaluates the long-context document understanding capabilities of LVLMs using extensive experiments on 14 LVLMs and 10 comparison LLMs.The benchmark targets lengthy documents and multimodal perception.
  • Conclusion: GPT-4o demonstrates the benefit of end-to-end multimodal perception, but most LVLMs struggle with long visual contexts and perform worse than OCR+LLM pipelines.The conclusion frames these findings as motivation for developing more powerful LVLMs on lengthy document understanding.

A Benchmark Construction Details

The benchmark combines lengthy, diverse documents with systematic annotation and quality-control procedures designed to emphasize cross-page, unanswerable, and multi-modal questions.

  • Document Collection: Documents from existing datasets are filtered or sampled for length, while ChartQA documents are recovered by matching chart text to original reports.DUDE selects documents over 15 pages, SlideVQA removes a 20-page truncation, FinanceBench samples financial reports, and ChartQA reconstructs source documents through OCR and search.
  • Document Collection: The benchmark selects lengthy documents from four existing datasets and newly collects academic papers, guidelines, and brochures across seven document types.Existing sources include DUDE, SlideVQA, FinanceBench, and ChartQA; newly collected materials diversify the document pool.
  • Document Examples: The benchmark presents examples spanning research reports, financial reports, academic papers, guidebooks, brochures, industrial files, and tutorials or workshops.Some examples display only the first 50 pages of the document.
  • Annotation Quality Control: Ten expert annotators systematically label existing questions as Retain, Revise, or Remove after checking six problems including wrong evidence, ambiguity, shortcuts, and insufficient document relevance.Repetitive and shortcut questions are typically removed, whereas wrongly annotated and decontextualization-required questions are generally revised.
  • Question Annotation: New questions emphasize cross-page and unanswerable cases plus table-, chart-, and image-based evidence to balance predominantly single-page, text-based existing questions.Question-count and evidence-source requirements are specified by document type.

A.6 Potential Bias for LVLM-based Quality Checking

The authors examine whether GPT-4o’s involvement in quality checking biases benchmark results and conclude that any measured bias is small relative to its performance advantage.

  • Potential Bias: The benchmark uses GPT-4o to remove document-agnostic samples and review potentially mislabeled samples, creating a possible bias toward GPT-4o.The authors explicitly identify this as a reasonable concern because GPT-4o substantially outperforms other models.
  • Bias Analysis: Step 1 filtering reduces rather than increases the performance gap because samples answered correctly without documents are removed.The authors attribute the larger performance drops of GPT-4V and Gemini-1.5-Pro in this step to limited document understanding and reliance on internal knowledge.
  • Bias Analysis: Step 2 and 3 quality checking introduces at most a 1.1% absolute performance bias according to the authors.These stages provide inconsistent human and GPT-4o answers to annotators for checking and revision.
  • Bias Analysis: Without GPT-4o’s quality-control involvement, GPT-4o still leads GPT-4V by 7.9% and Gemini-1.5-Pro by 19.8%.The reported scores are 43.1% versus 35.2% and 23.3%, respectively.
  • Annotation Workflow: The annotation workflow includes interfaces for editing existing questions and annotating new questions, with reference answers and metadata.The corresponding interfaces are shown in Figures 21 and 22.

B Experimental Details

The evaluation pipeline prompts models with document representations and extracts answers into typed formats before applying format-specific, rule-based scoring.

  • Prompts: LVLMs receive PNG page screenshots, while LLMs receive TXT documents, and both answer a concise document-grounded question.The response-generation prompt instructs models to read the documents and answer the question concisely.
  • Answer Extraction: Answer extraction maps free-form responses to Integer, Float, String, or List formats, using Not answerable and Fail to answer for distinct failure cases.Not answerable applies when the documents do not support an answer; Fail to answer applies when the model cannot read or understand the documents.
  • Scoring Rules: String answers use exact matching for specified formats or ANLS with threshold τ = 0.5, while integer answers require exact matching.Exact matching covers items such as telephone numbers, email addresses, dates, and filenames.
  • Scoring Rules: Float predictions match references within 1% relative tolerance, whereas list scores require equal lengths and use the minimum ordered element-wise score.Element-wise scoring follows the formats of the list elements.
  • Scoring Limitations: Human evaluation in Appendix B.4 finds that the automatic scorer is imperfect but aligns well with human judgment.The authors state that they will refine the rules to cover additional corner cases.

B.4 Human Evaluation on the Automatic Evaluation Pipeline

The authors manually compare automatic and human evaluations on responses from two representative LVLMs to assess the reliability of their evaluation pipeline.

  • Human Evaluation: The study randomly samples 100 questions and reviews responses from GPT-4o and Gemini-1.5-Pro.Human reviewers assess response correctness and compare it with the automatic evaluation results.
  • Human Evaluation: The comparison evaluates both components of the automatic pipeline: answer extraction and score calculation.The reported results are presented in Table 7.
  • Implementation Details: Model hyperparameters are documented in Table 8, with temperature set to 0.0 and max_new_tokens set to 1024 for all models.The concatenated_images and max_pages parameters control multi-image input and the maximum directly supplied pages for LVLMs.

C.1 Error Analysis

The error analysis categorizes GPT-4o’s incorrect responses into six substantive types beyond extraction errors, spanning perception, evidence completeness, hallucination, knowledge, reasoning, and question intent.

  • Perceptual Error: GPT-4o struggles with visual extraction, including misinterpreting chart axes and colored circles and miscounting green bars.These examples indicate shortcomings in fundamental perceptual capabilities.
  • Incomplete Evidence: GPT-4o sometimes omits evidence, such as a chapter author or an app appearing across two pages.The omissions occur despite stronger global searching abilities than other models on lengthy, multi-modal documents.
  • Hallucinated Evidence: GPT-4o fabricates evidence by referencing a nonexistent page or inventing page content to support incorrect responses.The authors identify this aggressive false-positive behavior as highlighting the importance of further hallucination and safety research.
  • Knowledge Lacking: Some errors reflect missing knowledge, such as using a single-point value instead of an average for fixed asset turnover or misidentifying Tsinghua buildings.These questions require both document information and parametric knowledge within the LVLM.
  • Reasoning Error: Reasoning errors arise when GPT-4o collects the evidence correctly but calculates a relative rather than absolute percentage or miscounts listed quizzes.The analysis describes flawed reasoning as a non-primary cause of incorrect answers.
  • Irrelevant Answer: GPT-4o sometimes answers a different question, discussing button operations instead of functions or summarizing all discrimination types instead of selecting the most discriminative.These cases reflect misunderstanding of question intent.

D Limitations

The benchmark acknowledges limitations involving scale, taxonomy, societal risks, and licensing or document-availability conditions. Despite these constraints, the authors describe it as a step forward and plan iterative expansion and refinement.

  • Scale: The benchmark’s primary limitation is scale: its test set contains 135 documents and 1,082 questions, fewer than previous datasets.The authors prioritize annotation quality because documents average about 50 pages and 20,000 tokens, while expert annotation takes more than one hour per document.
  • Question taxonomy: The benchmark uses only three broad question types because predefined taxonomies could reduce question diversity and posterior taxonomies are difficult for complex document understanding.The categories are single-page, cross-page, and unanswerable questions.
  • Future development: The authors plan to iteratively maintain, refine, and expand the benchmark with more documents and questions.They present MMLONGBENCH-DOC as a significant step forward despite its limitations.
  • Societal implications: Benchmarked models may produce biased or inaccurate outputs that perpetuate stereotypes or inequalities, while long-document processing could enable personal-information surveillance.The authors call for developers and users to recognize these potential societal consequences.
  • Data use: Collected datasets are licensed under Creative Commons or other open-source licenses, and newly collected documents are manually checked for academic availability.The authors state that removal requests for newly collected documents will be honored.
Loading 2407.01523v3…