PaperlayerView exact revision on arXiv

Source-linked AI summary

M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework

Yew Ken Chia, Liying Cheng, Hou Pong Chan, Chaoqun Liu, Maojia Song, Sharifah Mahani Aljunied, Soujanya Poria, Lidong Bing

arXiv:2411.06176v1cs.CL

TL;DR

Existing benchmarks often use shorter documents and simpler extraction questions, leaving open whether models can understand lengthy multimodal documents in depth. This paper introduces M-LongDoc, an automated evaluation framework, and retrieval-aware tuning; the tuning approach yields a 4.6% relative improvement in response correctness over baseline open-source models.

  • Problem

    It remains unclear whether current models can deeply understand lengthy multimodal documents, while existing benchmarks often cover documents under 50 pages and simpler extraction questions.

    Source: §1 · pp. 1–2

  • Method

    The paper builds the 851-sample M-LongDoc benchmark and a judge-based evaluator, and trains models with relevant evidence pages alongside potentially irrelevant pages.

    Source: §1 · p. 2, §3 · p. 7

  • Results

    4.6% relative improvement in the correctness of model responses compared to baseline open-source models.

    Source: §7 · p. 11

  • Takeaways & Limitations

    M-LongDoc offers a resource for more rigorous testing and development of multimodal document-understanding systems.

    Source: §1 · p. 2

    Models may still hallucinate information that is false or not verifiable when answering questions based on multimodal documents.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

The ability to understand and answer questions over documents can be useful in many business and practical applications. However, documents often contain lengthy and diverse multimodal contents such as texts, figures, and tables, which are very time-consuming for humans to read thoroughly. Hence, there is an urgent need to develop effective and automated methods to aid humans in this task. In this work, we introduce M-LongDoc, a benchmark of 851 samples, and an automated framework to evaluate the performance of large multimodal models. We further propose a retrieval-aware tuning approach for efficient and effective multimodal document reading. Compared to existing works, our benchmark consists of more recent and lengthy documents with hundreds of pages, while also requiring open-ended solutions and not just extractive answers. To our knowledge, our training framework is the first to directly address the retrieval setting for multimodal long documents. To enable tuning open-source models, we construct a training corpus in a fully automatic manner for the question-answering task over such documents. Experiments show that our tuning approach achieves a relative improvement of 4.6% for the correctness of model responses, compared to the baseline open-source models. Our data, code, and models are available at https://multimodal-documents.github.io.

1 INTRODUCTION

The paper introduces M-LongDoc to evaluate open-ended question answering over lengthy multimodal documents, alongside an automated evaluator and retrieval-aware tuning. The results report improved response correctness from the tuning approach.

  • M-LongDoc contains 851 samples of recent, lengthy, diverse documents and requires open-ended solutions involving text, figures, or tables.Unlike prior datasets described here, it targets documents spanning hundreds of pages and questions requiring in-depth understanding.Source: §1 · p. 2
  • The automated evaluation framework uses a detailed guide and multiple judge models to score open-ended answers without reference answers or human annotation.Source: §1 · p. 2
  • Preliminary evaluation found models struggle more with figure- and table-based questions than text-based questions and can be distracted by irrelevant document content.The distraction was observed even with retrieval-augmented generation.Source: §1 · p. 2
  • Retrieval-aware tuning includes distracting content from other modalities and pages, and the authors construct a 10,070-sample question-answering corpus for multimodal long documents.Source: §1 · pp. 2–3
  • 4.6% relative improvement in response correctness over the baseline model.Source: §1 · p. 3

2 M-LONGDOC BENCHMARK

M-LongDoc evaluates open-ended multimodal question answering over long, diverse documents, using verified questions and a committee-based scoring framework. A preliminary study finds weaker performance on figure- and table-based questions and reports high agreement between automated and human scores.

2 M-LONGDOC BENCHMARK

  • The benchmark covers academic, financial, and product domains with open-ended questions requiring in-depth analysis over documents averaging more than 200 pages.Academic documents are limited to computer science topics and documents published from January 2024 onward; documents are processed into interleaved text and extracted figure and table images.Source: §2 · p. 3, §2.1 · p. 4

2.1 DATA COLLECTION

  • Documents are sourced as research papers, company reports, and product manuals, then processed to extract page text and figure and table images.The academic domain is limited to computer science, and documents are dated January 2024 or later to reduce contamination or memorization risk.Source: §2.1 · pp. 3–4

2.2 QUESTION GENERATION

  • Questions are generated from category-specific pages with neighboring pages as context, then filtered by automated checklist and human validation.Of generated questions, 80.1% passed automated verification; 80.9% of those passed human verification, yielding 851 retained questions.Source: §2.2 · p. 4

2.3 AUTOMATED EVALUATION

  • Open-ended answers are scored for correctness by multiple multimodal judges using a shared guide, with repeated judgments aggregated into a continuous score.The framework uses three judge models and samples five scores from each; the authors report that its scores largely agree with human preferences with minimal bias.Source: §2.3 · pp. 5–6

2.4 PRELIMINARY STUDY

  • Figure- and table-based questions score significantly lower than text-based questions, and adding retrieved content may fail to improve performance or cause worse results or out-of-memory errors.The automated evaluation scores show a Pearson correlation of 88.9% with human annotations on the preliminary-study samples (p < 0.001).Source: §2.4 · p. 7

3 RETRIEVAL-AWARE MULTIMODAL TUNING

The paper targets multimodal long-document retrieval, where models must identify relevant evidence amid potentially misleading retrieved content. Its tuning approach trains with both gold evidence and distractor pages, using an automatically constructed corpus.

3 RETRIEVAL-AWARE MULTIMODAL TUNING

  • Multimodal long-document retrieval requires models to use pertinent evidence while resisting irrelevant retrieved content, a challenge the authors say existing open-source models do not address.The documents combine unstructured text with images of structured tables and figures, requiring fine-grained perception.Source: §3 · p. 7
  • Training includes the ground-truth evidence page alongside potentially irrelevant retrieved pages so models learn to handle noisy retrieval contexts.The approach is intended to improve text understanding and visual perception while using relevant document content.Source: §3 · p. 7
  • The automatically constructed training corpus contains 10,070 samples across 300 documents, with generated answers based on each gold evidence page.Human verification was omitted for scalability and cost-efficiency; on a random 100-sample subset, answers averaged 4.82 in correctness, and training documents did not overlap with evaluation documents.Source: §3 · p. 8

4 EXPERIMENTS

Experiments use retrieval-based inputs and open-ended answers scored for correctness. The study focuses on selected proprietary and open-source multimodal models, with Qwen2-VL-7B-Instruct used for training experiments.

4.1 TASK SETTING

  • Each model receives the question and the top five retriever-ranked pages, then produces an open-ended solution scored for correctness on a 1-to-5 scale.The retrieval setting avoids processing the full document.Source: §4.1 · p. 8

4.2 MODELS

  • The evaluation includes GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, LLaVA-OneVision-7B, and Qwen2-VL-7B-Instruct.The authors select open-source models supporting interleaved multimodal inputs and fine-grained visual perception, and plan to expand the study as other capable models are released.Source: §4.2 · p. 8

4.3 HYPERPARAMETERS

  • Training uses one epoch, batch size 16, learning rate 1e-4, and LoRA with rank 64 and alpha 32; experiments focus on Qwen2-VL-7B-Instruct.The authors cite training instabilities with other open-source models as the reason for this focus.Source: §4.3 · p. 8

5 RESULTS

Retrieval-aware tuning improves Qwen2-VL answer correctness on M-LongDoc, while open-source models remain weaker on table-related questions. Alternative-input results underscore the value of extracted images for figure- and table-based questions.

5.1 MAIN RESULTS

  • 4.6% relative improvement in answer correctness is reported for retrieval-aware tuning of Qwen2-VL; open-source models perform worse on table-related questions than other categories.Table 3 reports correctness across document domains and question categories and identifies the highest scores among open-source models.Source: §5.1 · p. 8, Table 3 · p. 9

5.2 EFFECT OF ALTERNATIVE SETTINGS

  • Removing images significantly decreases performance on figure- and table-based questions, while text-based performance increases slightly; using only rendered page images may be less optimal than separately extracted text, tables, and figures.The authors note that extracted text may retain partial information about figures and tables, and that page images may make it harder for models to distinguish text from tables.Source: §5.2 · p. 9, Table 4 · p. 9

6 RELATED WORK

Related work surveys multimodal models, document-understanding datasets, and retrieval methods, identifying gaps in long-document QA and the use of retrieved context.

6.1 LARGE MULTIMODAL MODELS

  • Closed- and open-source multimodal models address diverse content, but open-source benchmark performance tends to lag closed-source models (Yue et al., 2023).Examples include reasoning over audio, vision, text, document images, slides, and charts.Source: §6.1 · pp. 9–10

6.2 DOCUMENT UNDERSTANDING DATASETS

  • Document-understanding datasets cover textual reading comprehension, financial and chart QA, and multimodal documents, while M-LongDoc targets hundreds-page documents and open-ended solutions.The section also describes single-page multimodal QA and a benchmark averaging 47.5 pages and 21k textual tokens.Source: §6.2 · p. 10

6.3 RETRIEVAL-AUGMENTED GENERATION

  • Retrieval addresses the cost of processing entire long documents, but multimodal bias and susceptibility to irrelevant retrieved content remain challenges.The paper proposes retrieval-aware tuning for models using retrieval on multimodal long documents.Source: §6.3 · p. 10

7 CONCLUSION

The paper introduces M-LongDoc and a retrieval-aware tuning approach for multimodal long-document QA. The tuning approach improves response correctness relative to baseline open-source models.

7 CONCLUSION

  • M-LongDoc contains 851 samples and evaluates QA over recent, diverse documents often hundreds of pages long, requiring open-ended rather than merely extractive solutions.An automated framework evaluates large multimodal models on the benchmark.Source: §7 · p. 10
  • The paper proposes retrieval-aware tuning as a framework directly addressing retrieval for multimodal long documents.The approach is designed for efficient and effective processing of these documents.Source: §7 · p. 10
  • 4.6% relative improvement in response correctness over baseline open-source models.The paper reports this result for its retrieval-aware tuning method.Source: §7 · p. 11

ETHICS STATEMENT

The ethics statement describes release constraints, annotator status, and the possibility of hallucinated answers.

ETHICS STATEMENT

REPRODUCIBILITY STATEMENT

The reproducibility statement locates training details and hyperparameters in Sections 3 and 4 and describes planned public release of the benchmark, questions, and code.

REPRODUCIBILITY STATEMENT

A.1 DATA VERIFICATION

The appendices describe checks for question quality, retrieval-method selection, and answer scoring, then illustrate benchmark questions and retrieval-aware tuning with examples.

A.1 DATA VERIFICATION

  • Question verification checks category presence, category relevance, answerability, clarity, and whether the answer can be copied, while allowing questions that also require general knowledge or commonsense.The same verification guide is used for automated checks and human annotation.Source: §A.1 · p. 15

A.2 RETRIEVAL METHODS

  • ColPali, which encodes each page as a single image, performs best among the compared retrievers and is selected for the main experiments.Page relevance is based on the highest score among a page’s content pieces, and retrieval methods are compared using mean reciprocal rank of the gold evidence page.Source: §A.2 · p. 15, Table 5 · p. 16

A.3 EVALUATION GUIDE

  • Answers are scored for correctness on a 1–5 scale based on accuracy, comprehensiveness, and relevance to the document and question.The guide calls for comparing each response against the document before assigning its score.Source: §A.3 · p. 16

A.4.1 EXAMPLE OF M-LONGDOC

  • The benchmark’s example asks models to compare trends across two charts in a document.The example is described as testing cross-chart trend analysis and comparison.Source: §A.4.1 · p. 16, Table 6 · p. 17

A.4.2 CASE STUDY OF RETRIEVAL-AWARE TUNING

  • The final case study shows retrieval-aware tuning correcting Qwen2-VL’s mistaken “Cosine” answer to “affine” for the highest lcos across four datasets.The tuned answer reports lcos scores of 0.94 for MNIST and F-MNIST, and 0.96 for CIFAR-10 and CIFAR-100.Source: §A.4.2 · pp. 16, 18, Table 7 · p. 18
Loading 2411.06176v1…