PaperlayerView exact revision on arXiv

Source-linked AI summary

MMDocIR: Benchmarking Multimodal Retrieval for Long Documents

Kuicai Dong, Yujing Chang, Xin Deik Goh, Dexun Li, Ruiming Tang, Yong Liu

arXiv:2501.08828v3cs.IRcs.AIcs.CLcs.CV

TL;DR

Multimodal document retrieval lacks a comprehensive benchmark for finding evidence across visually rich long documents. MMDocIR introduces page- and layout-level retrieval tasks and reports that visual-driven retrievers outperform text-driven ones, while identifying incomplete layout annotations and absent joint text-visual training as limitations.

  • Problem

    Existing multimodal retrieval benchmarks inadequately evaluate question quality, full-document and domain diversity, and fine-grained retrieval of specific visual elements.

  • Method

    MMDocIR provides page- and layout-level retrieval tasks over curated long-document corpora, with expert and constructed evidence labels for evaluation and retriever training.

  • Results

    Visual-driven retrievers consistently outperform text-driven counterparts in multimodal document retrieval.

  • Takeaways & Limitations

    MMDocIR supports more precise, context-aware retrieval by targeting specific document layouts in addition to whole pages.

  • Takeaways & Limitations

    Layout labels are incomplete for 3 of 7 training subsets, and the visual retrievers do not jointly train on text and visual passages.

Abstract

from arXiv · show

Multimodal document retrieval aims to identify and retrieve various forms of multimodal content, such as figures, tables, charts, and layout information from extensive documents. Despite its increasing popularity, there is a notable lack of a comprehensive and robust benchmark to effectively evaluate the performance of systems in such tasks. To address this gap, this work introduces a new benchmark, named MMDocIR, that encompasses two distinct tasks: page-level and layout-level retrieval. The former evaluates the performance of identifying the most relevant pages within a long document, while the later assesses the ability of detecting specific layouts, providing a more fine-grained measure than whole-page analysis. A layout refers to a variety of elements, including textual paragraphs, equations, figures, tables, or charts. The MMDocIR benchmark comprises a rich dataset featuring 1,685 questions annotated by experts and 173,843 questions with bootstrapped labels, making it a valuable resource in multimodal document retrieval for both training and evaluation. Through rigorous experiments, we demonstrate that (i) visual retrievers significantly outperform their text counterparts, (ii) MMDocIR training set effectively enhances the performance of multimodal document retrieval and (iii) text retrievers leveraging VLM-text significantly outperforms retrievers relying on OCR-text. Our dataset is available at https://mmdocrag.github.io/MMDocIR/.

1 Introduction

Multimodal retrieval must recover evidence from visually rich documents, but existing benchmarks inadequately test full-document, diverse-domain, and fine-grained retrieval. MMDocIR addresses these gaps with page- and layout-level tasks and evaluates visual- and text-driven retrievers.

  • Visual elements such as tables, charts, and images convey information that plain text may fail to capture.
  • Existing benchmarks often use relevance-assuming VQA questions, partial documents, narrow domains, and page-only retrieval.
  • MMDocIR supports page-level retrieval of relevant pages and layout-level retrieval of specific paragraphs, headings, equations, tables, figures, or charts.
  • 21.3% of initial page labels from MMLongBench-Doc were corrected during meticulous review.
  • Visual-driven retrievers consistently outperform text-driven counterparts in multimodal document retrieval.

2 Dual-Task Retrieval Definition

MMDocIR defines retrieval over both document pages and extracted layouts, selecting the top k items most relevant to a query. Its system uses offline vector indexing followed by online query-to-item similarity comparison.

  • The task retrieves the top k pages and layouts from a document for query Q, with k much smaller than the total numbers of pages and layouts.
  • Pages and layouts receive relevance scores Sim(Q, p) and Sim(Q, l), respectively.
  • A document page usually contains about 5 to 15 layouts, depending on its complexity.
  • The retrieval pipeline encodes pages and layouts offline, then compares an encoded query with indexed vectors online.

3 MMDOCIR: Evaluation Set

The MMDocIR evaluation set combines curated long-document questions with page- and layout-level evidence annotations. Its protocol includes manual verification, layout detection, overlap-based matching, and cross-validation, while representing multimodal content in OCR- and VLM-derived text forms.

  • 3.1 Document Corpora Collection: The evaluation corpus starts from MMLongBench-Doc and DocBench, yielding 364 documents and 2,193 questions for subsequent annotation.
  • 3.1 Document Corpora Collection: Filtering and revising unsuitable questions produces 1,658 questions for annotation.
  • 3.2 Annotation Process: Page annotation identifies evidence-bearing pages, while layout annotation identifies specific evidence elements with bounding boxes.
  • 3.2 Annotation Process: Cross-evaluation reaches F1 scores of 95.2 for page labels and 87.1 for layout labels before discrepancies are fixed.
  • 3.3 Quality Control: Multimodal layouts are represented as original images, OCR-text, or VLM-text, while pages combine natural text with OCR-text or VLM-text.
  • 3.3 Quality Control: VLM-text averages 1.5 and 3.8 times OCR-text length for tables and figures, respectively.
  • 3.3 Quality Control: The evaluation set contains 313 documents averaging 65.1 pages across 10 domains, with text, image, table, and layout/meta evidence.

4 MMDOCIR: Training Set

The MMDocIR training set aggregates seven DocVQA datasets and reconstructs missing original documents where necessary. A semi-automated pipeline generates page- and layout-level labels, although layout labels are available for only four datasets.

  • The training corpora are collected from seven DocVQA datasets after screening related resources.
  • Because most source datasets lack original documents, the authors trace and recover them for training-set construction.
  • A semi-automated pipeline generates page-level and layout-level labels for datasets that lack them.
  • Layout-level labels were obtained or constructed for only 4 datasets because most existing datasets lack them.

5 Experiment

Experiments evaluate page- and layout-level retrieval with text and visual retrievers, using Recall@k and overlap-based matching for layouts. Visual retrieval generally performs best, while VLM-text improves over OCR-text but introduces efficiency costs and token-level methods trade modest accuracy gains for substantially larger storage.

  • 5.1 Evaluation Metric: Recall@k measures retrieved ground-truth evidence, using page indices for pages and bounding-box overlap for layouts.Overlap-based recall accommodates imperfect alignment between predicted and manually annotated layout boxes.
  • 5.2 Baseline Models and Setting: Six text retrievers and five VLM-based retrievers are adapted to both page- and layout-level retrieval.Text retrievers use OCR or VLM text, whereas visual retrievers directly process page screenshots or visual layouts.
  • 5.3 Main Results for Page-level Retrieval: Visual retrievers consistently outperform text retrievers in page-level retrieval, while MMDOCIR-trained visual models further improve performance.VLM-text outperforms OCR-text methods, indicating that richer visual information survives better in VLM-generated text.
  • 5.3 Main Results for Page-level Retrieval: 10.0GB versus 0.24GB: ColPali requires far more indexing storage than DSE, reflecting the cost of token-level embeddings.Token-level retrievers improve Recall@1 and marginally improve Recall@3/5, but can require ten times more storage than single-vector embeddings.
  • 5.4 Main Results for Layout-level Retrieval: VLM-text approaches perform comparably to visual retrievers for layout retrieval, while pure image sequences generally outperform hybrid image-text inputs.Token-level retrievers marginally outperform dense-level retrievers, but retrieving ten layouts still does not guarantee complete ground-truth coverage.
  • 5.5 Cascade Retrieval: High page-retrieval recall can significantly improve cascade layout retrieval, although cascade errors can propagate from page selection.The cascade first retrieves top-k pages, then reranks layouts within those pages.
  • 5.7 Inference Efficiency: DPR-style retrievers are more efficient than ColBERT-style retrievers, while textual inputs are more efficient than visual inputs across storage, indexing time, and latency.Visual encoders impose substantial computational overhead, whereas hybrid processing can reduce memory and time consumption.

6 Related Work

Prior multimodal retrieval benchmarks differ in question quality, annotation source, document completeness, and retrieval scope. MMDocIR is positioned against datasets with small or synthetic question sets, limited evidence coverage, or incomplete document corpora.

  • 6 Related Work: Wiki-SS uses natural questions and front-page screenshots, while DocMatix-IR uses Phi-3-small-generated questions that are not de-contextualized for retrieval.The cited datasets therefore differ from MMDocIR in question design and evidence construction.
  • 6 Related Work: Over 80% of the examined ViDoRe questions show limitations in complexity, contextual clarity, or complete-document coverage.The analysis covers 2,400 questions and identifies issues after comprehensive review.

7 Conclusion

MMDocIR introduces dual-granularity retrieval for multimodal documents and provides data for training and evaluation. Experiments show visual-driven retrievers outperform text-driven retrievers.

  • MMDocIR evaluates retrieval at page and layout granularity, targeting relevant pages and specific document elements.Layouts include paragraphs, headings, equations, tables, figures, and charts.
  • The benchmark provides expertly annotated labels for 1,685 questions and bootstrapped labels for 73,843 questions.These labels support both training and evaluation of multimodal document retrieval.
  • Visual-driven retrievers significantly outperform text-driven retrievers in multimodal document retrieval.The authors identify this result as evidence of the importance of visual information for retrieval performance.

Limitations

MMDOCIR has incomplete layout annotations in part of its training set and lacks joint training over text and visual passages. These constraints leave room for improved annotation and multimodal training.

  • Three of seven training subsets lack layout labels because the semi-automated pipelines cannot handle complex or cross-modal layouts.The authors propose advanced vision-language models for future annotation of these subsets.
  • Visual retrievers model text passages less effectively than text presented as image screenshots.The current visual retrievers do not explicitly use text query-document pairs.
  • Future work should jointly train or finetune on text and visual passages to improve performance on both retrieval tasks.

A Supplementary Experimental Results

Supplementary material extends the main page- and layout-level retrieval results and documents the benchmark’s source datasets and filtering criteria. The curation emphasizes long, multimodal documents and document-grounded questions.

  • Supplementary experimental results: Supplementary results extend page-level and layout-level retrieval analyses with additional retriever configurations.Table 11 adds OCR-text page results, while Table 12 adds OCR-text and hybrid-input layout results.
  • Dataset construction: Training sources include MP-DocVQA, SlideVQA, TAT-DQA, SciQAG, DUDE, and CUAD.
  • Question filtering: The curation excludes broad summarization, extensive statistical computation, online-search, and unanswerable questions.These filters keep the benchmark focused on retrieving factual information from within documents.

B.5 Training Dataset Label Construction

MMDOCIR constructs page- and layout-level labels from existing multimodal QA datasets, then trains visual retrievers with contrastive supervision and hard negatives. Its corpora span long documents and varied modalities.

  • Label construction: Page labels come directly from MP-DocVQA, SlideVQA, and DUDE, but only DUDE supplies layout labels.
  • Label construction: For SciQAG, layouts become labels when both question-passage and answer-passage similarity scores rank within the top three.MinerU extracts layout chunks, and E5 and BGE provide the similarity scores.
  • Label construction: For arXivQA, brute-force image matching followed by manual verification produces page- and layout-level labels after filtering unmatched images.The process results in 1,579 questions with both label types.
  • Training labels: Hard negatives are non-relevant pages with the highest model scores, selected to provide stronger supervision than random negatives.ColPali scores document pages, and the highest-scoring irrelevant pages are used during contrastive training.
  • Corpus composition: The evaluation set contains 313 documents averaging 65.1 pages across ten domains, with text occupying 60.4% of total modality area.Images occupy 18.8%, tables 16.7%, and other modalities 4.1%.
  • Visual retrievers: DPR-Phi3 uses a single dense embedding, whereas Col-Phi3 uses token-level embeddings for document and query representations.Both visual retrievers are based on Phi3-Vision and encode document pages as screenshots with a text prompt.
  • Query-document similarity: Similarity uses cosine similarity for DPR and sums each query vector’s maximum dot product with document vectors for ColBERT-style retrieval.
  • Training: Training pairs each query with one positive and one hard negative document using contrastive objectives.DPR-Phi3 uses InfoNCE loss with temperature τ = 0.02, while Col-Phi3 uses a softplus loss.

D.3 Implementation Details

This section describes the retrievers, document-domain diversity, and layout detection used to support multimodal retrieval evaluation. The benchmark spans text-heavy, image-rich, and table-rich document types.

  • Implementation Details: Nine off-the-shelf retrievers are evaluated: six text models and three visual models.The text models are DPR, ColBERT, Contriever, E5, BGE, and GTE; the visual models are DSEwiki−ss, DSEdocmatix, and ColPali.
  • Implementation Details: The benchmark covers ten document domains with distinct modality distributions, including image-rich tutorials and brochures, table-rich financial reports, and text-dominant legal documents.Academic papers combine structured text with tables and images, while government and laws documents are primarily textual.
  • Implementation Details: Layout detection identifies spatial locations of images, tables, and text to support precise layout-level retrieval evaluation.Detected layouts allow evidence to be located as a specific image, paragraph, or table rather than only as a page.

E.3 Annotation Examples

The annotation examples illustrate how multimodal retrieval evidence is represented at both page and layout levels. Examples span varied document types and include both benchmark annotations and comparison results.

  • Annotation Examples: Each annotation contains a question, answer, page-level labels, and layout-level labels identifying the evidence page and its specific layout box.Layout labels refine selected pages by locating the evidence region within the page.
  • Annotation Examples: Examples cover research reports, administration and industry documents, tutorials, academic papers, brochures, financial reports, guidebooks, government documents, laws, and news.The figures provide representative screenshots across these document categories.
  • Annotation Examples: RAPTOR outperforms the corresponding SBERT, BM25, and DPR baselines on both QuALITY accuracy and QASPER answer F1 with UnifiedQA-3B.The comparison tables report performance with and without RAPTOR across the two datasets.
  • Annotation Examples: RAPTOR’s QASPER F1 scores are at least 1.8 percentage points higher than DPR and at least 5.3 points higher than BM25 across the tested language models.The controlled comparison includes GPT-3, GPT-4, and UnifiedQA 3B.

F.1 Query and Annotation Analysis

This section analyzes query suitability, corpus construction, licensing, and ethical considerations for multimodal retrieval evaluation. It finds that many VQA-derived queries are poorly matched to information retrieval.

  • Query Analysis: Unsuitable queries are broad, while barely suitable queries may match multiple sections and create non-unique ground truth.A financial query about AMER’s total assets can correspond to several report sections, complicating IR evaluation.
  • Query Analysis: Only 8% of queries adapted from VQA datasets are suitable for information retrieval, compared with 15.5% of queries from industrial documents.The analysis identifies query specificity and discriminative context as important for pinpointing relevant passages.
  • Corpus Construction: The benchmark reconstructs complete document corpora because source VQA datasets often provide only selected pages or cropped figures, tables, and charts.The authors collect original documents to evaluate retrieval over long documents rather than isolated evidence images.
  • Licensing: MMDocIR annotations are released for academic and non-commercial research, while commercial exploitation requires explicit written permission.The stated restriction applies to the new questions and page- and layout-level annotations.
  • Ethical Considerations: The benchmark may expose privacy and fairness risks because long multimodal documents can contain sensitive information and datasets may not represent diverse populations.Suggested mitigations include vetted public data, anonymization or removal of sensitive information, and monitoring retrieval outputs.
Loading 2501.08828v3…