PaperlayerView exact revision on arXiv

Source-linked AI summary

VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval

Simon Ott, Yufan Chen, Ruiping Liu, Junwei Zheng, Jiale Wei, Di Wen, Kunyu Peng, Jiaming Zhang, Rainer Stiefelhagen

arXiv:2610.17057v1cs.AIcs.CV

TL;DR

Existing evaluations often focus on short documents and single relevant pages, leaving long-document retrieval with evidence spread across pages or documents less tested. VL-DocIR introduces an evidence-complete page-retrieval benchmark and evaluates visual retrievers and diagnostic settings. Results show that cross-document retrieval remains the hardest setting, while document-context scoring is especially useful for within-document multi-page queries.

  • Problem

    Many page-level evaluations reward retrieving any relevant page rather than recovering the full evidence set, leaving long-document and cross-document evidence retrieval untested by that criterion.

    Source: §1 · pp. 1–2

  • Method

    VL-DocIR pairs natural-language queries with rendered page images and requires retrieval of every annotated evidence page, stratifying evaluation by evidence structure, source, and domain.

    Source: §3 · p. 3, §3.1 · p. 3

  • Results

    Cross-document retrieval remains the hardest setting, while multi-vector retrieval is stronger than single-vector retrieval and document-context scoring is most beneficial for within-document multi-page queries.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VL-DocIR provides an evidence-complete target for evaluating page retrieval across single-page, multi-page, and cross-document configurations.

    Source: §3 · p. 3, §5 · p. 10

    The benchmark is limited to page-level rather than exhaustive element-level grounding and rendered HTML-born documents; its VLM-generated, automatically cleaned queries may also shape the benchmark distribution.

    Source: §5 · p. 10

Abstract

from arXiv · show

Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages. This leaves an important gap between leaderboard performance and realistic long-document retrieval, where the evidence needed to answer a query may span multiple pages or even multiple documents. We present VL-DocIR, a page-level benchmark for evidence-complete visual long-document retrieval. VL-DocIR is built from 29,641 HTML-born documents rendered into 388,548 page images from Wikipedia, arXiv, PubMed, and SEC proxy statements. It contains 271,760 queries across 23 domains and six evidence structures, covering single-page, within-document multi-page, and cross-document evidence configurations. Evidence annotations are grounded to rendered pages and HTML element identifiers, and queries are filtered with a cleaning pipeline targeting clarity, correctness, closedness, and the absence of explicit layout references. We evaluate leading single-vector and multi-vector visual retrievers together with diagnostic retrieval settings such as document-context scoring, query splitting, reranking, and round-robin ensembles. The strongest evaluated configuration reaches 69.3 nDCGAll@10 and 80.6 RecallAll@10 under structure-balanced Macro evaluation, but source- and evidence-structure breakdowns show that cross-document retrieval remains substantially harder than single-page retrieval, and that document-context scoring is especially helpful for within-document multi-page queries. VL-DocIR exposes these failure modes and provides a more demanding target for visual long-document retrieval. The new dataset and code will be made publicly available at https://yufanchen96.github.io/VL-DocIR.

1 Introduction

Page-image retrieval has advanced, but evaluations still leave open whether systems can recover complete evidence distributed across long documents or multiple documents. VL-DocIR benchmarks this setting and shows that performance depends on evidence structure, with cross-document retrieval remaining difficult.

1 Introduction

  • Page-image retrieval addresses information lost in plain-text representations of charts, tables, figures, and layouts, but page-level success alone does not establish recovery of evidence distributed across long or multiple documents.Source: §1 · pp. 1–2
  • VL-DocIR contains 29,641 rendered documents, 388,548 page images, and 271,760 queries across 23 domains and six evidence types, from single-page to cross-document evidence.Each item pairs a natural-language query with multiple evidence pages and the image corpus.Source: §1 · p. 2
  • The benchmark construction pipeline combines HTML-grounded page rendering, evidence-page configuration, VLM-based query generation, and query cleaning to create page-level retrieval supervision at scale.Source: §1 · p. 2
  • Multi-vector retrieval is consistently stronger than single-vector retrieval, while cross-document evidence coverage remains limited and document-context scoring especially helps within-document multi-page queries.The benchmark also evaluates query splitting, reranking, and ensembles.Source: §1 · p. 2

2 Related Work

Related work spans visual page retrieval, document understanding, Document AI, and retrieval benchmarks. These strands establish the relevance of visual and cross-page evidence, while motivating broader evaluation across settings and domains.

2.1 Vision-based Document Retrieval

  • DSE treats screenshots as retrieval units [1], and ColPali adds late interaction over visual page embeddings with ViDoRe [2]; later work explores harder settings and efficient representations [3] [4].Recent work also addresses multilingual and long-document retrieval [8].Source: §2.1 · p. 2

2.2 Long Document and Visual Document Understanding

  • Document VQA and structured-document QA cover document, scene-text, infographic, chart, web-page, and slide settings, while MP-DocVQA extends VQA to multi-page documents.These benchmarks establish that layout, tables, figures, and cross-page evidence matter for downstream understanding.Source: §2.2 · p. 2

2.3 Document AI and Vision-Language Models

2.4 Retrieval and Embedding Benchmarks

  • Retrieval benchmarks span text, multimodal embeddings, long-context reasoning, and document images, motivating heterogeneous evaluation beyond single-domain leaderboards.The cited resources include VLM2Vec, universal multimodal embedding work, and document-image benchmarks [1].Source: §2.4 · p. 3

3 Benchmark

VL-DocIR defines visual long-document retrieval around recovering every page needed to answer a query, rather than retrieving any relevant page. It organizes evaluation by evidence structure, source, and domain.

3 Benchmark

  • A query is evidence-complete at cutoff k only when every ground-truth evidence page appears in the top-k ranking of rendered page images.The benchmark stratifies items by evidence structure, source provenance, and domain coverage.Source: §3.1 · p. 3, §3 · p. 3

3.1 Task Definition and Stratification

  • Six evidence structures distinguish single-page lookup from multi-page and cross-document retrieval, while source and domain slices diagnose performance across document regimes and content areas.The structures are Page-Single, Page-Multi, Doc-Adjacent, Doc-Near, Doc-Far, and Cross-Doc.Source: §3.1 · p. 3

3.2 Visual Document Database Creation Pipeline

  • 29,641 documents yielded 388,548 rendered pages, with average lengths ranging from 6.49 pages for Wikipedia to 41.97 for SEC proxies.HTML-born documents are rendered directly to preserve layout and structure as retrieval signals; pages are split using whitespace and bounding-box heuristics.Source: §3.2 · p. 4

3.3 Benchmark Query Creation Pipeline

  • 206,820 single-page and 64,940 distributed-evidence queries preserve page-level examples while testing complete multi-page or cross-document evidence retrieval.The creation pipeline samples evidence configurations, uses VLMs to generate queries and evidence identifiers, then filters queries for clarity, correctness, closedness, and absence of explicit layout references.Source: §3.3 · pp. 4–5

3.4 Annotation Quality and Adjudication

  • 93.25% of 800 reviewed entries were entirely correct, compared with 88.5% for Cross-Doc entries; 84.23% of Cross-Doc page mappings matched bounding-box-derived assignments.The review was stratified across source and evidence-structure combinations.Source: §3.4 · p. 6

4 Experiment

The experiments test evidence-complete page retrieval across evidence structures and compare visual retrievers with retrieval-stage modifications. Results show that multi-vector retrieval improves over single-vector retrieval, while cross-document evidence remains especially difficult.

4.1 Experimental Setup

  • Evaluation reports evidence-complete RecallAll and nDCGAll alongside standard page-level metrics, with structure-balanced Macro as the primary aggregate.The diagnostic settings include query splitting, document-context scoring, reranking, and round-robin ensembles.Source: §4.1 · pp. 6–7

4.2 Results and Analysis

  • 69.28 nDCGAll@10 and 80.64 RecallAll@10 under Macro for ColNomic (Ensemble); Jina V4 multi-vector gains 11.6 and 10.5 points over single-vector, respectively.The ensemble combines page-local matching, document context, and reranking within the existing retrieval backbone.Source: §4.2 · pp. 7–8, Table 3 · p. 8
  • Reranking generally improves nDCGAll, while document-context scoring often improves RecallAll when relevant pages belong to the same document.The three-list ColNomic ensemble further improves both metrics under Macro and Micro.Source: §4.2 · pp. 8–9, Table 4 · p. 9
  • Evidence-complete RecallAll@5 drops sharply when retrieval must cover multiple pages, with the hardest transition from within-document to cross-document evidence.The source-macro aggregation summarizes the evidence structures; the benchmark's evidence-completion requirement means all required pages must be recovered.Source: Table 5 · p. 9, §4.2 · p. 9

4.3 Qualitative Results and Analysis

  • For a Cross-Doc query, reranking ColQwen2.5 retrieves both relevant evidence pages, whereas the base model retrieves only one and confuses mission numbers.The figure shows top-five results; blue rectangles mark evidence boxes and green rectangles mark correctly retrieved pages.Source: §4.3 · p. 10, Figure 3 · p. 10

5 Conclusion

VL-DocIR evaluates evidence-complete retrieval across page, within-document, and cross-document evidence, finding cross-document retrieval most challenging. The appendix details the source-specific corpus collection, document representation, dataset statistics, and query-generation process.

5 Conclusion

  • Cross-document retrieval remains the most challenging regime; multi-vector retrieval is stronger than single-vector retrieval, and document context is most useful within one document.The strongest evaluated configuration reaches 75.6 nDCGAll@10 and 86.2 RecallAll@10, while aggregate scores can mask cross-document failures.Source: §5 · p. 10
  • The benchmark represents pages with page-level evidence rather than exhaustive element-level grounding, and excludes scanned and handwritten documents.Its VLM-generated, automatically cleaned queries may also reflect generator style and filtering criteria.Source: §5 · p. 10

A.1 Document Collection

  • Collection strategies vary by source: Wikipedia uses recursive article selection, arXiv expands survey seeds, PubMed draws open-access articles, and SEC uses one recent proxy statement per company.The SEC restriction addresses overlapping information across years that could create ambiguous retrieval problems.Source: §A.1 · pp. 17–18

A.2 Document Items

  • Each document item combines equally sized page images, HTML with injected identifiers, and pixel-exact bounding boxes, alongside abstracts, licensing, and citations.Source: §A.2 · p. 18

A.3 Document Dataset Statistics

  • The dataset spans 23 domains with at least 50 documents per domain; Wikipedia articles are usually 1–10 pages, while most SEC statements are 20–60 pages.ArXiv and PubMed articles typically range from 5 to 25 pages.Source: §A.3 · pp. 18–19

A.4 Query Generation

  • Query generation first describes page images, then prompts a VLM for queries using summaries, HTML identifiers, and page images; valid identifiers ground evidence to page-relative boxes.The prompts target clear, unambiguous, closed questions that do not explicitly refer to page layout.Source: §A.4 · pp. 20–21

A.5 Query Cleaning Evaluation Dataset

The query-cleaning evaluation compares removal strategies, then reports dataset statistics and annotation audits that qualify how benchmark results should be interpreted. The retrieval experiments also note the compute required for each model run.

A.5 Query Cleaning Evaluation Dataset

  • 13.37% is the best false omission rate among the tested query-rating thresholds, achieved at 0.75.The evaluation used 1,438 manually labeled queries.Source: §A.5 · pp. 22–23, Table 6 · p. 23
  • The direct LLM classifier and five-neighbor method have false omission rates of 17.82% and 19.77%, respectively.XGBoost's 12.46% rate is slightly below logistic regression's 13.07%, but the authors avoid these classifiers because the small removal dataset may increase variance on unknown query types.Source: §A.5 · pp. 23–24, Table 7 · p. 23, Table 10 · p. 24
  • 13.60% is the false omission rate for combining the LLM classifier and query-rating remover, the selected approach despite slightly lower rates for classifiers.The combination removes a query if either method predicts removal; the LLM approach retained 69.05% of evaluation queries, versus 62.38% for logistic regression and 55.33% for XGBoost.Source: §A.5 · pp. 23–24, Table 8 · p. 23

A.6 Query Dataset Statistics

  • Financial Reports contributes 79,791 queries, and the authors advise caution with grouped domain-by-evidence results for sparsely represented domains.Single-page queries are most common across domain types, followed by Cross-Doc queries.Source: §A.6 · p. 24, Table 11 · p. 25
  • Pure text is the most common evidence type with 90,248 queries, followed by table-row evidence with 40,955 and image-plus-text with 28,932.Figure 7 summarizes evidence-type co-occurrence and distribution.Source: §A.6 · p. 24, Figure 7 · p. 25

A.7 Annotation Validation and Adjudication

  • 746/800 audited entries were fully correct (93.25%); Cross-Doc correctness was 177/200 (88.5%), with PubMed Cross-Doc lowest at 40/50 (80.0%).The audit also found formulation issues, incorrect or unnecessary evidence pages, and non-presence queries without directly retrievable positive evidence; rates describe entries before corrections.Source: Table 12 · p. 26, §A.7 · p. 26
  • A census of 47,013 Cross-Doc queries found 39,600 fully consistent (84.23%); after corrections, method order and the best method stayed unchanged across twelve retrieval runs.The census also found 5,860 queries with an element spanning rendered pages (12.46%).Source: §A.7 · pp. 26–27

B Evaluation Result Details

  • Each retrieval-model experiment consumed approximately 100 hours of compute on A100 and H100 nodes.The evaluation reports results grouped by evidence structure, data source, and evidence type.Source: B Evaluation Result Details · p. 27

B.1 Overall Results

Across the overall benchmark, ColNomic (Ensemble) leads both aggregation protocols, while rankings among component settings and the effects of OCR fusion vary by metric and protocol.

B.1 Overall Results

  • 69.28 nDCGAll@10 and 80.64 RecallAll@10 make ColNomic (Ensemble) the best Macro configuration; its Micro scores are 75.64 and 86.23, respectively.Table 13 reports overall retrieval results across visual, OCR-based, native-text, and hybrid methods.Source: §B.1 · p. 27
  • ColQwen3 has the highest nDCGAll@10 among base visual retrievers; Jina V4 [16] multi-vector exceeds its single-vector counterpart by 11.6 Macro nDCGAll@10 points.For ColQwen2.5, Q-Split reaches 69.19 RecallAll@10 under Macro versus 67.23 for the base model.Source: §B.1 · p. 27
  • OCR-BM25 leads evaluated text baselines under Macro with 50.24 nDCGAll@10 and 61.69 RecallAll@10.OCR-dense (BGE-M3) scores 19.35 and 24.70, while DOM-BM25 scores 46.96 and 57.44, respectively.Source: §B.1 · p. 27
  • Fusing OCR-BM25 with ColNomic (Ctx) raises RecallAll@10 from 78.19 to 78.67 under Macro and from 82.26 to 84.24 under Micro, while nDCGAll@10 falls in both.For ColNomic (Rerank), fusion lowers Macro RecallAll@10 from 74.62 to 74.39 and raises Micro RecallAll@10 from 83.11 to 83.30, with lower nDCGAll@10 under both protocols.Source: §B.1 · p. 27

B.2 Source- and Domain-Averaged Macro Results

ColNomic (Ensemble) leads the reported source- and domain-averaged Macro comparisons. These aggregations weight sources or domains equally while balancing evidence structures within each group.

B.2 Source- and Domain-Averaged Macro Results

  • Each source-averaged score equally weights four sources, and each domain-averaged score equally weights 23 domains, with evidence structures equally weighted within each group.Structure-balanced Macro remains the primary aggregation protocol.Source: §B.2 · p. 27, Table 14 · p. 29, Table 15 · p. 29
  • ColNomic (Ensemble) achieves the highest scores among nine configurations for every reported metric and k under both aggregations.The reported metrics are nDCG, Recall, nDCGAll, and RecallAll at k values 3, 5, and 10.Source: §B.2 · p. 29, Table 14 · p. 29, Table 15 · p. 29
  • 69.78 nDCGAll@10 and 80.85 RecallAll@10 under source-averaged Macro, and 75.26 and 85.93 under domain-averaged Macro, are ColNomic (Ensemble)'s reported scores.Tables 14 and 15 report these source- and domain-averaged results, respectively.Source: §B.2 · p. 29, Table 14 · p. 29, Table 15 · p. 29
  • The higher RecallAll@10 setting differs by aggregation: ColNomic (Ctx) under source-averaged Macro and ColNomic (Rerank) under domain-averaged Macro.The source- and domain-averaged comparisons therefore do not give the same Ctx-versus-Rerank ordering for this metric.Source: §B.2 · p. 29

B.3 Results by Base Model

Across the three base models, document-context approaches often gain more as more pages are retrieved, but the strongest variant depends on the base model and k.

B.3 Results by Base Model

  • ColPali v1.3 (Rerank) leads at k = 3 and RecallAll@5, while ColPali v1.3 (Ctx) surpasses it at RecallAll@10.The faster hierarchical approach is slightly above the base model across k; Ctx and Normal&Ctx have similar RecallAll@k scores.Source: §B.3 · p. 29, Figure 8 · p. 30
  • ColQwen2.5 (Q-Split 3) exceeds the base model in RecallAll@k, unlike Q-Split 1 and 2; document-context variants gain most as k rises.Normal&Ctx improves on Ctx for ColQwen2.5.Source: §B.3 · p. 29
  • The ensemble of ColNomic, ColNomic (Ctx), and ColNomic (Rerank) has the best RecallAll@k among the tested ColNomic variants.ColNomic (Hierarchical) matches the base model while using fewer computing resources during page retrieval.Source: §B.3 · p. 30

B.4 Results by Query Evidence Structure Type and Data Source Type

Retrieval difficulty varies with evidence structure, source, and evidence type: cross-document and table-only evidence are particularly challenging, while model rankings depend on the setting.

B.4 Results by Query Evidence Structure Type and Data Source Type

  • Across sources and models, multi-page queries are harder than single-page queries, Cross-Doc queries harder still, and Doc-Far slightly harder than Doc-Near and Doc-Near than Doc-Adjacent.The reported comparison uses RecallAll@5 by source and evidence structure.Source: §B.4 · pp. 30–31
  • For multi-page queries, document-context embeddings yield higher RecallAll@5 for Wikipedia and SEC Proxy Reports, while reranking is more favorable for arXiv and PubMed.For single-page queries, ColNomic (Normal&Ctx&Rank) is consistently best; the second-best model varies by source.Source: §B.4 · p. 31
  • For Cross-Doc queries, the best RecallAll@5 is 50.9% for Wikipedia, 26.6% for SEC Proxy Reports, 18.0% for arXiv, and 28.0% for PubMed.Document-context embeddings can hurt Cross-Doc retrieval, whereas reranking improves over ColQwen2.5 and ColNomic base models, with the stated exception of non-Wikipedia sources versus Q-Split.Source: §B.4 · p. 31

B.5 Results by Evidence Type

  • Table-only evidence has consistently lower retrieval performance than text-only or image-only evidence, whose results are similar, with image evidence slightly harder to find.ColNomic (Normal&Ctx&Rank) is highest across evidence types; second-best rankings vary with evidence type, metric, and k.Source: §B.5 · p. 31

B.6 Results by Domain Type

Cross-document retrieval performance varies substantially by domain, and the best document-context vector count depends on evidence structure.

B.6 Results by Domain Type

  • Cross-Doc RecallAll@5 reaches 75.6% in the Science domain but remains below 20% in Financial Reports and Astrophysics.Figure 9 reports the maximum across evaluated models for domains with at least 100 Cross-Doc queries.Source: §B.6 · pp. 31, 33, Figure 9 · p. 33

B.7 Document Context Embeddings Vector Count Impact

  • The optimal document-context vector count varies by evidence structure, from about 32 for Cross-Doc queries to 128–256 for single-page queries.Doc-Adjacent and Doc-Near peak between 256 and 512 vectors, while Doc-Distant peaks around 256; for a given count, context embeddings improve RecallAll@10 across structures.Source: §B.7 · p. 34, Figure 10 · p. 34

B.8 Page Count Reduction Analysis

Reducing the page corpus improves retrieval, but substantial gaps remain; cross-document analysis suggests difficulty extends beyond corpus size to complete evidence retrieval.

B.8 Page Count Reduction Analysis

  • Reducing the corpus from 388,548 to 38,551 page images increases RecallAll@5 for every tested model, evidence structure, and source combination.The comparison evaluates the three base models on the full and reduced datasets.Source: §B.8 · p. 34, Table 18 · p. 35
  • Even after reduction, Doc-Adjacent retrieval with the best base model, ColNomic, retains an absolute RecallAll@5 gap of 38.3%.Single-page arXiv and PubMed retrieval approaches perfect performance, while Wikipedia and SEC Proxy Reports retain a gap.Source: §B.8 · p. 35
  • After reduction, ColNomic reaches RecallAll@5 of 20.8% on arXiv and 31.4% on PubMed Cross-Doc queries, while SEC Proxy Reports reaches 19.7%.For reduced-dataset PubMed queries, ColNomic's Recall@5 is 57.7%, compared with RecallAll@5 of 31.4%.Source: §B.8 · pp. 35–36

B.9 Cross-Document Retrieval Error Analysis

  • Cross-Doc models often retrieve at least one correct evidence item at both document and page levels, but complete page-level retrieval is less frequent than document-level success.Document-context embeddings show more partial retrieval, while query splitting has high document-level success; SEC Proxy Statements have the lowest complete evidence retrieval.Source: §B.9 · p. 36

B.10 Difference between Query Generators

Retrieval scores differ between query generators, though comparisons also reflect differences in source and evidence-structure composition. ColNomic (Ensemble) achieves the highest scores among the nine configurations for both generators.

B.10 Difference between Query Generators

  • The 211,777-query Qwen3-VL-32B-Instruct subset was evaluated using retrieval from the complete 388,548-page corpus; Macro equally weights six evidence structures, Micro averages all queries, and both retain the generator’s original query distribution.Differences in retrieval performance may reflect both query formulation and dataset composition; they do not by themselves establish differences in annotation correctness.Source: §B.10 · pp. 36, 38
  • For ColNomic, the generator difference falls to 1.2 percentage points with document-context retrieval, from 5.3 for base retrieval and 4.8 for reranking.Source: §B.10 · p. 38
  • Most configurations have higher Macro RecallAll@10 on GPT-4.1-mini queries, with differences ranging from −0.1 to 7.1 percentage points.Figure 12 compares the generators across configurations on the same full corpus.Source: §B.10 · p. 36, Figure 12 · p. 38
  • At k=10, all nine configurations obtain higher nDCGAll and RecallAll on GPT-4.1-mini queries than on Qwen3-VL-32B-Instruct queries under both Macro and Micro.Source: §B.10 · p. 38, Table 19 · p. 39, Table 20 · p. 39
  • 71.51 Macro nDCGAll@10 and 81.74 Macro RecallAll@10 for ColNomic (Ensemble) on GPT-4.1-mini queries, versus 68.85 and 80.46 on Qwen3-VL-32B-Instruct queries.ColNomic (Ensemble) has the highest scores among the nine configurations for both generators at every evaluated k.Source: §B.10 · p. 38, Table 19 · p. 39, Table 20 · p. 39

Related papers

Loading 2610.17057v1…