Source-linked AI summary
Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracy
Juyeon Kim, Geon Lee, Dongwon Choi, Taeuk Kim, Kijung Shin
TL;DR
Single-vector retrieval is efficient but less accurate, while multi-vector retrieval is more accurate at substantially greater computational cost. HEAVEN combines them in a two-stage framework, and across four multi-document benchmarks it preserves 99.87% of ColQwen2.5’s Recall@1 performance while reducing per-query FLOPs by 99.82%.
Problem
Single-vector retrieval is efficient but less accurate, whereas multi-vector retrieval is more accurate at substantially greater computational cost.
Source: §1 · p. 2Method
Results
99.87% of ColQwen2.5’s Recall@1 performance is retained while per-query FLOPs fall by 99.82% across four multi-document benchmarks.
Source: §1 · p. 2Takeaways & Limitations
HEAVEN and VIMDOC together establish a scalable foundation for efficient and accurate visual document retrieval.
Source: §8 · p. 9HEAVEN relies on pretrained vision-language encoders, and performance may vary with model scale and domain-adaptation quality.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Retrieval over visually rich documents is essential for tasks such as legal discovery, scientific search, and enterprise knowledge management. Existing approaches fall into two paradigms: single-vector retrieval, which is efficient but coarse, and multi-vector retrieval, which is accurate but computationally expensive. To address this trade-off, we propose HEAVEN, a plug-and-play two-stage hybrid-vector framework. In the first stage, HEAVEN efficiently retrieves candidate pages using a single-vector method over Visually-Summarized Pages (VS-Pages), which assemble representative visual layouts from multiple pages. In the second stage, it reranks candidates with a multi-vector method while filtering query tokens by linguistic importance to reduce redundant computations. To evaluate retrieval systems under realistic conditions, we also introduce ViMDoc, a benchmark for visually rich, multi-document, and long-document retrieval. Across four benchmarks, HEAVEN attains 99.87% of the Recall@1 performance of multi-vector models on average while reducing per-query computation by 99.82%, achieving efficiency and accuracy. Our code and datasets are available at: https://github.com/juyeonnn/HEAVEN
1 Introduction
Visual document retrieval matters for applications such as legal discovery, scientific search, and enterprise knowledge management, but single-vector efficiency and multi-vector accuracy come at different computational costs. HEAVEN combines the two paradigms in a two-stage framework and introduces VIMDOC for visually rich, long, multi-document retrieval.
1 Introduction
- Charts, tables, and figures motivate visual document retrieval, while single-vector methods are more efficient and multi-vector methods are more accurate but costlier.OCR or complex layout parsing can increase indexing time and complexity.Source: §1 · pp. 1–2
- HEAVEN first retrieves candidate pages through single-vector filtering over visually summarized pages, then reranks pages with multi-vector retrieval using filtered key query tokens.The first stage aggregates key visual elements across multiple pages before page-level filtering.Source: §1 · p. 2
- HEAVEN’s modular design lets users select or swap each stage’s base encoder without further training.The framework is presented as adaptable to specific domains or upgrades to higher-performing models.Source: §1 · p. 2
- VIMDOC evaluates visual document retrieval in multi-document and long-document settings that existing benchmarks often do not combine.The passage notes that existing benchmarks commonly assume a single document or use short documents.Source: §1 · p. 2
- HEAVEN preserves 99.87% of ColQwen2.5’s Recall@1 performance while reducing per-query FLOPs by 99.82% across four multi-document benchmarks.The comparison is reported in the introduction as the framework’s overall experimental result.Source: §1 · p. 2
2 Preliminaries
The preliminaries define page-level retrieval over document pages and set up single- versus multi-vector scoring in a shared embedding space.
2.1 Problem Definition
- A corpus is a set of documents represented as ordered page sequences, and retrieval seeks ground-truth pages that collectively answer each query.Source: §2.1 · p. 2
2.2 Retrieval Frameworks
- Single-vector retrieval reduces each query and page to one embedding and scores every page with one similarity operation, costing O(d|P|) per query.Source: §2.2 · pp. 2–3
- Figure 3 shows multi-vector models outperforming single-vector models by more at K=1 than at K=200.Source: Figure 3 · p. 3
- Multi-vector retrieval aggregates fine-grained query-page interactions, matching each query token to its best page patch before summing scores; corpus scoring costs O(dnq ∑P∈P nP).Source: §2.2 · p. 3
3 Analysis
The analysis compares the efficiency and accuracy of single- and multi-vector retrieval. Single-vector retrieval is much more efficient, while its accuracy gap narrows as the candidate set grows.
3 Analysis
- Up to 99.94% fewer FLOPs per query in VIMDOC: single-vector retrieval is more efficient than multi-vector retrieval.Source: §3 · p. 3
- Figure 4 depicts the VS-page construction process and example outputs.Source: Figure 4 · p. 3
- In VIMDOC, the accuracy gap is 22.5% at Recall@1 but only 0.63% at Recall@200, narrowing as more candidates are considered.Source: §3 · p. 3
4 Proposed Method: HEAVEN
HEAVEN is a plug-and-play, two-stage framework that combines single-vector retrieval’s efficiency with multi-vector retrieval’s accuracy. It first narrows candidates using visually summarized pages, then reranks them with filtered query tokens.
4 Proposed Method: HEAVEN
- HEAVEN combines single-vector and multi-vector retrieval in a plug-and-play framework whose stage models can be replaced without further training.This makes the framework adaptable to specific domains or upgradeable to stronger models.Source: §4 · p. 3
4.1 Stage 1. Single-Vector Retrieval of Candidate Pages
- VS-pages compress multiple document pages into fewer pages by cropping and assembling representative title layouts.They are constructed at indexing time, so construction adds no query-time inference overhead.Source: §4.1 · p. 4
- Stage 1 ranks VS-pages with single-vector retrieval, expands selected VS-pages to their associated pages, then combines VS-page and page-level scores to select top-K candidates.The combined score weights the two scores using α.Source: §4.1 · p. 4
4.2 Stage 2. Multi-Vector Reranking of Pages
- Stage 2 reranks Stage 1 candidates using multi-vector similarities for key query tokens, which average about 30% of query tokens.The filtered tokens achieve performance comparable to using all tokens and outperform using the same number of random tokens.Source: §4.2 · p. 5
5 Proposed Benchmark: VIMDOC
VIMDOC targets the joint challenge of finding relevant information across multiple long documents, which existing visual document retrieval benchmarks do not capture together. It evaluates visually rich documents with page-level annotations, including cross-page queries.
5.1 Existing Benchmarks
- Existing benchmarks generally cover either short single-document retrieval, long single-document retrieval, or multi-document retrieval over documents averaging 1.0–18.4 pages, not both long and multi-document settings.This leaves the combined challenge of retrieving information across multiple long documents untested.Source: §5.1 · pp. 5–6, Table 2 · p. 6
5.2 Design of VIMDOC
- VIMDOC jointly evaluates multi-document and long-document retrieval over visually rich documents, with page-level ground truth that can span multiple pages.The benchmark is designed for lengthy documents, such as those exceeding 20 pages.Source: §5 · p. 5, §5.2 · p. 6
- VIMDOC retains self-contained queries with distinctive keywords and removes 45.8% of context-dependent queries through heuristic and LLM-based filtering.The retained queries can be searched across the union of all document pages.Source: §5.2 · p. 6
6 Experimental Results
The experiments compare HEAVEN with single-vector and multi-vector retrieval systems across four benchmarks, then examine efficiency, component contributions, hyperparameters, and encoder choices. The results assess whether the hybrid approach can retain retrieval performance while reducing computation.
6.1 Settings
- HEAVEN is evaluated on four benchmarks, using page-level Recall@1 and Recall@3 except for M3DocVQA, which uses document-level metrics; efficiency is measured by per-query FLOPs and latency.The benchmarks are VIMDOC, OpenDocVQA’s SlideVQA and DUDE splits, ViDoSeek, and M3DocVQA.Source: §6.1 · p. 6
6.2 Main Results
- 99.89% of ColQwen2.5’s retrieval performance is preserved on average while FLOPs fall by 99.82%; on M3DocVQA, Stage 1 exceeds DSE by 7.73% in Recall@1 and Stage 2 improves performance by 2.27%.On M3DocVQA, Stage 1 uses 79.67% fewer FLOPs than DSE, and Stage 2 uses 99.81% fewer FLOPs than ColQwen2.5; textual and visual methods also show a clear performance gap on VIMDOC and OpenDocVQA.Source: Table 3 · p. 7, §6.2 · p. 7
6.3 Comparison with Efficiency Variants
- HEAVEN achieves the best efficiency–accuracy trade-off among the four efficiency-oriented variants; pooling generally outperforms pruning, but its performance drops as FLOPs decrease.The comparison includes document patch pooling and pruning, and query token pooling and pruning.Source: §6.3 · pp. 7–8, Figure 7 · p. 7
6.4 Ablation Study
- Removing VS-pages increases FLOPs with only marginal accuracy gains, while skipping candidate refinement causes a severe performance drop; disabling query filtering increases FLOPs, and omitting reranking refinement degrades performance.The ablations assess components in both retrieval stages.Source: §6.4 · p. 8, Table 4 · p. 8
6.5 Efficiency Analysis
- 99.88% faster online retrieval per query than ColQwen2.5 comes with additional offline processing time for VS-page construction.HEAVEN’s accuracy is described as comparable to multi-vector models and substantially better than single-vector models.Source: §6.5 · p. 8, Table 5 · p. 8
6.6 Hyperparameter Analysis
- HEAVEN remains robust across tested reduction factors, filtering ratios, and weighting hyperparameters; the reduction factor has a more pronounced impact on long-document VIMDOC than shorter-document OpenDocVQA.The analysis covers reduction factor r, filtering ratios p1 and p2, and weighting hyperparameters α and β.Source: §6.6 · p. 8
6.7 Plug-and-Play Analysis
- Among tested encoder combinations, DSE for Stage 1 and ColQwen2.5 for Stage 2 achieve the best overall performance on VIMDOC and OpenDocVQA.HEAVEN is designed to allow either stage’s encoder to be replaced with another pretrained model.Source: §6.7 · p. 8
7 Related Work
Prior visual document retrieval methods encode pages as images, sometimes combining visual embeddings with OCR or textual summaries. HEAVEN’s related-work contrast centers on its fully visual indexing and query-token filtering for reducing computation.
Visual Document Retrieval
- Recent visual document retrieval methods encode entire pages as images to bypass OCR, while other visual RAG approaches combine visual embeddings with OCR or textual summaries.HEAVEN instead introduces visually summarized pages for OCR-free, fully visual indexing.Source: §7 · p. 9
Multi-Vector Retrieval
- Multi-vector visual retrieval offers accuracy but incurs high computational overhead; prior efficiency approaches use compression, approximation, pruning, or pooling.HEAVEN contrasts these approaches by filtering query tokens while maintaining performance.Source: §7 · p. 9
- Visual document retrieval has applied patch pruning and pooling to improve efficiency with minimal accuracy loss, whereas HEAVEN reduces computation through query-token filtering while maintaining performance.Source: §7 · p. 9
8 Conclusion
HEAVEN combines single-vector retrieval over visually summarized pages with multi-vector reranking using selective query tokens, aiming to bridge efficiency and accuracy in visual document retrieval. The paper also presents VIMDOC, a benchmark for visually rich, multi-document, and long-document retrieval, and concludes that the framework offers near state-of-the-art accuracy with over 99% lower computation.
8 Conclusion
- HEAVEN combines single-vector retrieval over VS-pages with multi-vector reranking using selective query tokens, achieving near state-of-the-art accuracy with over 99% lower computation.The paper also introduces VIMDOC for visually rich, multi-document, and long-document retrieval.Source: §8 · p. 9
Limitations
HEAVEN has limitations involving pretrained encoders, layout analysis, and its current focus on retrieval. The appendices detail VS-page construction, examples, query-token filtering, and key-token proportions.
Limitations
- HEAVEN relies on pretrained vision-language encoders, whose performance may vary with model scale and domain adaptation quality; Stage 1 layout analysis may be sensitive to noisy or irregular layouts.The framework focuses on retrieval efficiency and does not yet integrate retrieval-augmented generation.Source: Limitations · p. 9
A.1 Detailed VS-page Construction
- VS-pages are constructed by cropping detected title regions and vertically stacking them without resizing or scaling.DocLayout-YOLO classifies layouts into eight categories, but only title layouts are extracted for VS-pages.Source: §A.1 · p. 12
A.2 VS-page Examples
- Examples show constructed VS-pages from VIMDOC, OpenDocVQA, ViDoSeek, and M3DocVQA.The section points to four figures showing outputs across the benchmarks.Source: §A.2 · p. 12
A.3 Detailed Query Key Token Filtering
- Query key tokens are selected using NLTK POS tags NN, NNS, NNP, and NNPS.These tags identify the subset of tokens retained as key tokens.Source: §A.3 · p. 12
A.4 Key Token Distribution
- 30–37% of query tokens are key tokens across benchmarks, with the remainder classified as non-key tokens.Table 7 reports average counts and percentages for both token groups.Source: §A.4 · p. 12, Table 7 · p. 12
B VIMDOC Benchmark
The benchmark appendix details dataset selection and query filtering, alongside experimental model and benchmark statistics and evaluation measures. It defines Recall@K and specifies the assumptions used for FLOPs comparisons.
B.1 Detailed Data Collection
- The benchmark collection includes all four REAL-MM-RAG splits, English VisR-Bench splits with all three query types, only MMDocIR’s evaluation set, and all included LongDocURL and MMLongBench-Doc documents.VisR-Bench’s multilingual split is excluded because baseline models were mainly pretrained on English datasets.Source: §B.1 · p. 12
B.2 Detailed Query Processing
- Two-stage filtering removes 45.8% of queries; heuristics remove unanswerable or explicitly context-dependent queries, while LLM filtering addresses cases heuristics miss.Some context-dependent queries lack explicit keywords and are difficult to catch with rules alone; Figure 14 shows the LLM-filtering prompt.Source: §B.2 · p. 12, Table 8 · p. 12, Figure 14 · p. 14
C.1 Models
- The model appendix records embedding dimensions, base models, batch sizes, and Hugging Face checkpoints used in the experiments.Table 9 distinguishes textual and visual embedding models.Source: §C.1 · p. 14, Table 9 · p. 15
C.2 Benchmark Statistics
- For OpenDocVQA, SlideVQA and DUDE are merged, while the single-page InfoVQA and ChartQA splits are excluded.The selection follows the original paper’s all-pool setting for multi-page documents.Source: §C.2 · p. 14
C.3 Evaluation Metric
- Recall@K measures the share of ground-truth pages retrieved in the top K; FLOPs comparisons assume batch size 1 and float16 embeddings.FLOPs and latency are used to evaluate retrieval efficiency, with batch size 1 chosen because padding makes token-length variation affect batch scoring FLOPs.Source: §C.3 · p. 15
C.4 Hyperparameter for Efficiency Variants
The appendix specifies parameter ranges for HEAVEN and alternative efficiency variants, then presents extended results for cross-page and single-page queries and per-split comparisons.
C.4 Hyperparameter for Efficiency Variants
- Efficiency variants are tested across candidate counts, document and query token pooling, and document and query token pruning ranges.HEAVEN tests Ck = {100,200,400}; document patch pooling spans 4–144 patches, and pruning ratios generally span 0.1–0.9.Source: §C.4 · p. 15
C.5 Extended Experimental Results
- Extended experiments report cross-page and single-page query results, plus per-split comparisons on VIMDOC and OpenDocVQA using Recall and per-query FLOPs.The cross-page setting requires multiple pages; Table 12 reports an average of 2.4 pages for VIMDOC and 2.0 for OpenDocVQA.Source: §C.5 · p. 16, Table 12 · p. 16, Table 13 · p. 17