PaperlayerView exact revision on arXiv

Source-linked AI summary

MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrieval

Debanjan Mahata, Atharva Tendle, Daniel Preotiuc-Pietro, Yong Zhuang, Ozan Irsoy

arXiv:2609.01316v1cs.IRcs.AIcs.CLcs.CVcs.LG

TL;DR

Visually rich documents lose important layout-dependent evidence when OCR linearizes pages, motivating retrieval representations beyond extracted text. MIDR moves multimodal reasoning to ingestion, indexes verified textual enrichments for text-centric retrieval, and achieves competitive or superior retrieval with smaller indexes and lower query-time cost, while retaining a complementary role for visual retrieval.

  • Problem

    Tables, charts, figures, and layout relations contain important evidence that plain OCR can linearize, corrupt, or omit, limiting retrieval over visually rich documents.

  • Method

    MIDR is a training-free framework that uses an MLLM to create and verify structured textual enrichments from rendered pages and extracted text, then indexes them for BM25F and dense retrieval.

  • Results

    0.6219 average nDCG@10: MIDR Hybrid across five English ViDoRe V3 domains, while French-domain enrichment lifts BM25 from 0.1532 to 0.5448 nDCG@10 and uses an index roughly 9× smaller than ColQwen2.5’s.

  • Takeaways & Limitations

    Index-time enrichment provides a text-centric deployment alternative to serving-time visual late interaction, with gains concentrated where OCR loses structure and complementary evidence retained by visual retrieval.

  • Takeaways & Limitations

    Ingestion averages 2.1 MLLM calls and approximately 8k tokens per page, scaling with corpus size and requiring reruns when documents or enrichment schemas change.

Abstract

from arXiv · show

Retrieval over visually rich documents has a representation problem: important content often lives in tables, charts, figures, and layout relations that plain OCR linearizes, corrupts, or omits. ColPali-family visual retrievers address this with patch-level multi-vector indexes and late-interaction scoring, keeping image-derived retrieval on the query-time serving path. We introduce MIDR (Multimodal Indexing for Document Retrieval), a training-free framework for enrichment-augmented indexing that shifts multimodal reasoning to index time. During ingestion, a multimodal LLM converts rendered pages into verified textual fields that are indexed with BM25F and optionally fused with dense retrieval, enabling text-centric serving over multimodally grounded evidence. On ViDoRe V3, MIDR Hybrid achieves 0.6219 average nDCG across five English domains, a 23.0% relative gain over BM25, remaining competitive with ColQwen2.5. On two French-document domains, enrichment bridges English queries and French page text, lifting BM25 from 0.1532 to 0.5448 nDCG and outperforming ColQwen2.5. Across all seven domains, MIDR leads ColQwen2.5 on four while using approximately 9x smaller index memory and approximately 2x lower query latency. These results establish index-time multimodal reasoning as a compelling accuracy-deployment alternative to serving-time visual late interaction.

1 Introduction

MIDR addresses the loss of visually grounded evidence in OCR-based retrieval by moving multimodal reasoning to index time and serving verified textual fields. On ViDoRe V3, it improves retrieval substantially, including a strong cross-lingual result on French documents, while reducing deployment costs relative to visual multi-vector retrieval.

  • Motivation: Tables, charts, figures, captions, and layout relations contain retrieval-relevant evidence that plain OCR can flatten or omit.This representation gap is consequential for RAG because retrieval must surface the right evidence before generation.
  • Approach: MIDR performs multimodal enrichment offline, then indexes verified fields for BM25F, dense, or hybrid text-centric retrieval.Document- and page-level fields include structured descriptions and QA pairs, with auditing against rendered pages and extracted text.
  • Results: 0.6219 average nDCG@10: MIDR Hybrid across five English ViDoRe V3 domains, a 23.0% relative gain over raw BM25 at 0.5057.It remains competitive with the ColQwen2.5 reproduction.
  • Results: 0.5448 nDCG@10: MIDR Hybrid on two French-document domains versus 0.1532 for raw BM25, leading ColQwen2.5 on both.English enrichments bridge English queries to French page content.
  • Deployment trade-off: 9× smaller index and 1.1–2.6× lower query latency: MIDR compared with ColQwen2.5’s visual multi-vector index.The design shifts recurring visual retrieval computation into one-time ingestion.
  • Analysis: QA pairs, keyphrases, and table summaries expose distinct retrieval evidence, while visual late interaction remains stronger for equations, diagrams, code layout, and fine-grained visual disambiguation.A per-query oracle reaches 0.7042 nDCG@10, indicating complementary evidence types.

2 Related Work

Related work spans text enrichment, visual multi-vector retrieval, and multimodal RAG. MIDR builds on these directions while distinguishing itself through typed, verified enrichment fields and a changed index rather than query-processing pipeline.

  • Text enrichment: Prior text-centric enrichment systems generate synthetic queries, summaries, QA pairs, or bridging facts before indexing.Examples include doc2query, Doc2Query++, EnrichIndex, IndexRAG, PREMIR, and MLDocRAG.
  • MIDR distinction: MIDR differs by producing a typed multi-field page record whose fields have distinct retrieval roles and are audited against page evidence.Refinement occurs on 9.6% of pages overall and 52% on the hardest domain.
  • Visual retrieval: ColPali-family systems encode rendered pages into patch-level vectors and use late interaction, with ColQwen2.5 providing strong open-weight ViDoRe V3 performance.These methods keep visual representations in the serving path.
  • Multimodal RAG: Multimodal RAG systems typically combine images, retrieval, agents, and generation while spending multimodal computation at query time.Benchmarks such as ViDoRe V3 evaluate retrieval over complex layouts and visually grounded evidence.
  • Positioning: Query-side methods change query processing, whereas MIDR changes what the index contains.This places MIDR as complementary to query refinement and multimodal reranking.

3 Enrichment-Augmented Indexing with MIDR

MIDR constructs multimodally grounded textual page representations through document context, verified page enrichment, and text-centric indexing. Its extract–verify–refine loop limits unsupported fields before retrieval infrastructure consumes them.

  • Design: MIDR represents each page using rendered images, extracted text, document context, and page-level enrichment fields.The task is to rank pages for a text query while deciding whether multimodal computation occurs at indexing or serving time.
  • Document and page enrichment: Document-level enrichment from the first five pages supplies global context for disambiguating repeated entities, acronyms, and domain-specific references.Page enrichment also captures layout, tables, charts, figures, and visual grouping that OCR may flatten.
  • Extract–verify–refine: The extract–verify–refine loop generates structured fields, audits grounding and consistency, and revises only flagged fields.A deterministic postprocessor normalizes tags and keyphrases, removes duplicate QA pairs, and enforces consistency between layout flags and summaries.
  • Indexing and retrieval: Verified enrichments and original page text are indexed as separate BM25F fields, while document-level fields are replicated across corresponding pages.Dense retrieval embeds fields separately and hybrid retrieval fuses BM25F and dense rankings with Reciprocal Rank Fusion.
  • Serving path: Query-time retrieval uses BM25F and dense text indexes without page-image processing, visual multi-vector indexes, or late-interaction scoring.Multimodal reasoning is performed once during ingestion.

4 Experimental Setup

The evaluation uses ViDoRe V3 page retrieval across English-document domains and a French-document cross-lingual stress test. It compares four text-centric configurations with visual multi-vector baselines and analyzes field and query sensitivity.

  • Benchmark: ViDoRe V3 evaluation covers 2,099 English queries over 16,867 pages from 184 documents across five English and two French-document domains.The retrieval unit is a page, with candidate pools restricted to each domain and nDCG@10 reported using official qrels.
  • MLLM sensitivity: MLLM sensitivity is measured by rerunning the full pipeline with alternative backends while holding retrieval, fusion, embeddings, and field weights fixed.Frontier MLLMs span 0.6120–0.6231 nDCG@10 on the English evaluation, with GPT-5.1 strongest among those tested.
  • Retrieval configurations: Four configurations are evaluated: raw-text BM25, enriched-field BM25F, dense retrieval with EmbeddingGemma, and MIDR Hybrid combining lexical and dense rankings.Comparisons include ColQwen2.5 and the larger ColEmbed-3B-v2 visual retriever.
  • Analysis protocol: Ablation analysis includes QA-only, no-QA, semantic-only, visual-only, no-semantic, leave-one-out field, and query-level stratification studies.These analyses group results by visual content type and query type.

5 Results and Analysis

MIDR approaches visual document retrieval by moving multimodal reasoning to ingestion, where enriched textual fields support text-centric retrieval. Across English and French evaluations, it approaches or exceeds visual baselines while reducing serving costs and revealing where enrichment helps most.

  • English retrieval accuracy: 0.6219 average nDCG@10 makes MIDR Hybrid competitive with ColQwen2.5 on five English domains, while improving 23.0% over raw BM25.Enriched BM25F reaches 0.5592 and dense mean-pool retrieval 0.5898, whose RRF fusion yields 0.6219.
  • Accuracy–deployment trade-off: ColEmbed-3B-v2 reaches 0.6730 average nDCG@10, but requires 11.07 MB per page versus 0.038 MB per page for MIDR.MIDR is therefore positioned as an accuracy–deployment operating point rather than the maximum-accuracy retriever.
  • Accuracy–deployment trade-off: 14.0× query latency and 7.5× index memory for MIDR Hybrid compare with 27.9× and 65.0× for ColQwen2.5, respectively.The comparison is normalized to BM25 and reflects the recurring serving cost of visual multi-vector retrieval.
  • Cross-lingual retrieval: 0.5448 average nDCG@10 on French documents with English queries exceeds reproduced ColQwen2.5’s 0.5315 and lifts raw BM25 from 0.1532.English enrichments generated from French content turn cross-lingual retrieval into monolingual matching against English index fields.
  • Enrichment mechanisms: 0.6200 nDCG@10 from QA-only enrichment recovers 94% of the enrichment gain over a markdown-only hybrid baseline, while removing QA lowers retrieval to 0.5788.QA pairs provide the strongest lexical bridge; keyphrases and QA fields also support dense matching.
  • Where enrichment helps: 41.7% relative nDCG@10 gains on numerical queries and 38.6% on mixed-visual pages show that enrichment helps most where OCR loses structure.Text-only pages and boolean queries improve least, at +14.3% and +11.7%, respectively.

6 Conclusion and Future Work

MIDR shifts multimodal reasoning from query time to index time by converting rendered pages into verified textual retrieval fields. It frames enriched text and visual multi-vector indexes as complementary deployment choices.

  • MIDR moves multimodal reasoning from query time to index time, serving verified textual fields through text-centric retrieval infrastructure.
  • MIDR reaches performance comparable to strong visual multi-vector retrievers while using an index roughly 9× smaller and paying ingestion cost before queries arrive.
  • Index-time enrichment supports transformations such as generating English fields from French pages, while different fields contribute distinct retrieval evidence.
  • Enriched text indexes and visual multi-vector indexes are complementary design points for deployments favoring amortized ingestion, text-centric infrastructure, or auditable evidence.
  • MIDR’s schema, prompts, MLLM backend, embedding model, and field boosts remain deployment choices that can improve without changing the serving path.

7 Limitations

The evaluation covers selected ViDoRe V3 domains and a controlled subset of comparable systems, while MIDR trades peak accuracy and some latency advantages against deployment efficiency. Its ingestion cost, backend dependence, language coverage, and field behavior constrain generalization.

  • 7.1 Comparison Scope: The evaluation covers five English-document domains and a targeted study of two French-document domains, excluding some ViDoRe systems such as API-only models.
  • 7.1 Comparison Scope: 0.6730 average nDCG@10 is achieved by ColEmbed-3B-v2 versus MIDR’s 0.6219 on English domains, while MIDR stores roughly 0.038 MB per page versus 11 MB.
  • 7.2 Latency and Index Memory: Optimized MaxSim kernels would narrow the query-latency gap, although they would not reduce the storage cost of patch-level page representations.
  • 7.3 Ingestion Cost: 2.1 MLLM calls and approximately 8k tokens per page are required during ingestion, with costs recurring when documents or the enrichment schema change.
  • 7.4 Backend Dependence: Retrieval quality depends on the index-time backend: the tested open-weight model trails frontier backends by 0.046 nDCG@10 on English and collapses on French.
  • 7.5 Field and Language Coverage: Chart summaries can harm aggregate retrieval, entity strings can confound French queries, and the cross-lingual study covers only one direction of one language pair.

A Dataset and Enrichment Coverage

MIDR evaluates a fielded enrichment schema and fixes embedding and pooling choices for its main results rather than treating them as primary contributions. Group ablations show that QA-only nearly matches full enrichment overall, with stronger non-QA benefits on English than French.

  • A Dataset and Enrichment Coverage: ViDoRe V3 processing covers five English and two French domains using the GPT-5.1 extract–verify–refine pipeline.
  • C Retrieval-Side Design Choices: MIDR’s contribution is the enrichment schema and extract–verify–refine indexing path, not a new dense retriever.
  • C Retrieval-Side Design Choices: EmbeddingGemma was selected after preliminary comparisons because it produced the strongest dense-only English nDCG@10 while matching or exceeding alternatives on French domains.
  • C Retrieval-Side Design Choices: MIDR embeds the original page text and each enrichment field separately, then combines per-field similarities using a pooling strategy.
  • C Retrieval-Side Design Choices: Mean pooling dominated the compared pooling variants across English and French domains and was adopted as the default dense representation.
  • D Group Ablations: 0.5982 hybrid nDCG@10 for qa_only versus 0.5976 for the full schema across all seven domains shows that non-QA fields help English but add offsetting noise on French.
  • D Group Ablations: 0.6231 hybrid nDCG@10 for full enrichment exceeds the 0.5057 markdown-only BM25 baseline by 23.2% in the English group ablation.
  • D Group Ablations: 0.5438 hybrid nDCG@10 for qa_only slightly exceeds 0.5339 for the full schema on the French group ablation.

E Stratified Field-Level Ablations

Field contributions are concentrated in query and page strata rather than distributed uniformly. Table summaries and QA variants provide targeted benefits, while chart and entity fields can hurt aggregate retrieval, especially across languages.

  • E Stratified Field-Level Ablations: 0.004 aggregate hybrid nDCG@10 gain from table summaries becomes a 0.028 loss when removed on table pages, a seven-fold concentration.
  • E Stratified Field-Level Ablations: Coarse QA dominates numerical queries, fine QA dominates extractive queries, and document focus helps open-ended and multi-hop queries.
  • E Stratified Field-Level Ablations: Removing chart summaries improves aggregate hybrid retrieval by 0.003 nDCG@10, making charts the only field with a reported aggregate harm.
  • E Stratified Field-Level Ablations: Broad chart trend descriptions can over-match boolean queries without anchoring the correct page, with the effect amplified on French.
  • E Stratified Field-Level Ablations: Chart and entity fields are deployment knobs for cost-sensitive or multilingual settings, while stricter value extraction and language-aware canonicalization remain follow-up work.
  • E Stratified Field-Level Ablations: Small French strata, including numerical n = 17 and image n = 13, should be interpreted with sample size in mind.

G French Field Behavior

French results show that QA fields provide the main cross-lingual bridge, while chart summaries and main-entity strings can hurt retrieval. The enrichment pipeline shifts multimodal reasoning to ingestion but retains domain-dependent costs and open grounding challenges.

  • French field behavior: French stratified gains exceed 200% relative because raw BM25 is near-floor, with absolute hybrid nDCG@10 values of 0.46–0.75.The largest relative lifts occur on infographics and tables.
  • French field behavior: Removing fine_qa costs 0.0315 nDCG@10 on French versus 0.0162 on English, while removing coarse_qa costs 0.0279 versus 0.0141.QA fields carry essentially all of the cross-lingual bridge signal.
  • French field behavior: qa_only matches the full schema across all seven domains, reaching 0.5982 versus 0.5976 nDCG@10 for hybrid.This exceeds the 94% English-only recovery.
  • French field behavior: Removing chart_summary improves French hybrid retrieval by 0.0104 nDCG@10, while removing main_entities improves it by 0.0075.Chart descriptions can create noisy English representations, and entity strings can retain French forms or transliterations that act as confounders.
  • Offline ingestion cost: The ingestion pipeline averages 2.10 MLLM calls, 6,257 input tokens, 1,914 output tokens, and 23.2 seconds per page.The expense is paid once during index construction and amortized over future queries.

K Enrichment MLLM Sensitivity

MIDR’s gains are relatively insensitive to the choice among frontier enrichment MLLMs, while the open-source model trails substantially, especially on French. The systems also show complementary query-level strengths, motivating mixed extractor–verifier configurations.

  • Frontier model sensitivity: Frontier MLLMs span 0.6120–0.6231 nDCG@10 on English-5, a 2.5% relative range within the 23.2% BM25-to-MIDR gain.This indicates the improvement comes principally from enrichment-augmented indexing rather than a specific frontier MLLM.
  • Open-source model sensitivity: Qwen3-Omni reaches 0.5772 on English-5 versus 0.6120–0.6231 for frontier models and 0.298 on French.Malformed structured outputs and an over-strict verifier compounded the gap.
  • Design implication: A stronger verifier paired with a frugal extractor, or the reverse, offers a way to trade ingestion cost against enrichment quality without retraining.Systematic mixed extractor–verifier configurations remain future work.
  • Complementarity: MIDR and ColQwen2.5 have decisive wins on roughly half of 1,489 English-domain queries, split nearly evenly between MIDR and ColQwen2.5.The partition reports 351 MIDR wins and 362 ColQwen2.5 wins at a gap above 0.1 nDCG@10.
  • Complementarity: French oracle gains of +0.08 nDCG@10 over either system match the English pattern, indicating complementarity extends cross-lingually.The two systems succeed and fail on different queries.

N Implementation Details

MIDR uses fielded lexical and dense retrieval over verified enrichments, with fixed implementation choices and controlled evaluation. Worked examples show that QA and table summaries recover searchable evidence lost through vocabulary mismatch or garbled extraction.

  • Lexical retrieval: BM25F uses Whoosh with default k1 = 1.2 and b = 0.75, applying the same parameters per field.Enrichment fields preserve named entities and domain-specific tokens by using no stemming.
  • Field weighting: All main-paper BM25F results use uniform field weights of 1.0 because ViDoRe V3 has no development split for tuning.A priori role-based weighting is evaluated only as a robustness check.
  • Dense and hybrid retrieval: Dense retrieval embeds page text and enrichment fields with EmbeddingGemma, mean-pools and L2-normalizes them, then searches an exact FAISS IndexFlatIP.Hybrid retrieval fuses BM25F and dense rankings with Reciprocal Rank Fusion using k = 60.
  • Enrichment pipeline: MIDR’s page pipeline generates structured enrichments, normalizes them, verifies grounding and consistency, and conditionally refines flagged fields.Inputs include the rendered page image, extracted page text, document enrichments, and page metadata.
  • Finance example: Enriched BM25F retrieves the finance target at rank 1 with nDCG@10 = 1.0, whereas markdown-only BM25 misses the top 100 with nDCG@10 = 0.0.Coarse QA mirrors the user query and table summaries surface the year-specific total category.
  • Finance example: The finance example’s coarse QA reports $726 million in total restructuring charges, including $716 million in personnel costs.The enrichment exposes the relevant year and total category in searchable language.

P.2 Computer Science Domain: Garbled Table Content

In the computer-science example, PDF-to-text conversion garbles a reference table and omits the queried calls, but MIDR’s table summary and QA fields recover searchable parameterized descriptions.

  • Observed retrieval failure: Baseline BM25 finds no target page in the top 10, with the first target at rank 21 and nDCG@10 = 0.0.The relevant pages contain a reference table whose extracted markdown is garbled and omits the queried calls.
  • Observed retrieval improvement: Enriched BM25F ranks all three target pages first, second, and third, achieving nDCG@10 = 1.0.Three relevant pages are marked in the ground truth.
  • Structured recovery: The table summary converts the garbled reference table into searchable prose describing range(end), range(start, end), and range(start, end, step).It introduces the key terms start, end, step, and pattern.
  • Vocabulary bridging: Fine QA directly states that range(start, end, step) generates a sequence beginning at start until end with the specified step size.This parametric framing bridges to queries comparing two-argument and three-argument calls.
  • Mechanism: MIDR improves lexical retrieval through vocabulary bridging and structured content recovery rather than by adding new information.The finance example addresses query–document vocabulary mismatch, while the computer-science example addresses information loss during extraction.
Loading 2609.01316v1…