PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representations

Yibo Yan, Mingdong Ou, Yi Cao, Xin Zou, Shuliang Liu, Jiahao Huo, Yu Huang, James Kwok, Xuming Hu

arXiv:2603.01666v1cs.CLcs.IR

TL;DR

Visual Document Retrieval needs to represent dense layout and content while avoiding the storage overhead of conventional multi-vector systems. ColParse parses pages into a small set of layout-informed sub-images, fuses their embeddings with a global vector, and achieves over 95% storage compression while improving retrieval across models and datasets. The results position document structure as central to practical multimodal retrieval.

  • Problem

    Multi-vector VDR captures fine-grained document semantics but incurs prohibitive storage costs, while existing optimizations can dilute information or ignore layout cues.

  • Method

    ColParse uses document parsing to generate typically fewer than 10 layout-informed sub-image embeddings and fuses them with a holistic global page vector.

  • Results

    Over 95% storage compression is achieved while retrieval performance consistently improves across base models and datasets.

  • Takeaways & Limitations

    ColParse establishes a compact, structurally aware multi-vector representation for practical and interpretable visual document retrieval.

Abstract

from arXiv · show

Harnessing the full potential of visually-rich documents requires retrieval systems that understand not just text, but intricate layouts, a core challenge in Visual Document Retrieval (VDR). The prevailing multi-vector architectures, while powerful, face a crucial storage bottleneck that current optimization strategies, such as embedding merging, pruning, or using abstract tokens, fail to resolve without compromising performance or ignoring vital layout cues. To address this, we introduce ColParse, a novel paradigm that leverages a document parsing model to generate a small set of layout-informed sub-image embeddings, which are then fused with a global page-level vector to create a compact and structurally-aware multi-vector representation. Extensive experiments demonstrate that our method reduces storage requirements by over 95% while simultaneously yielding significant performance gains across numerous benchmarks and base models. ColParse thus bridges the critical gap between the fine-grained accuracy of multi-vector retrieval and the practical demands of large-scale deployment, offering a new path towards efficient and interpretable multimodal information systems.

1. Introduction

Visual Document Retrieval must capture dense interactions among text, layout, and graphics, but multi-vector models face prohibitive storage costs. ColParse addresses this bottleneck by parsing pages into layout-informed sub-images and achieves broad performance gains as a compact, plug-and-play framework.

  • Visual documents combine textual content, intricate layouts, and graphical elements, requiring retrieval beyond natural-image representations.
  • Multi-vector VDR improves fine-grained alignment but storing hundreds or thousands of vectors per page makes large-scale deployment challenging.Existing optimization directions include patch merging and pruning, but merging can dilute fine-grained information and produce unstable performance.
  • ColParse parses each page into typically fewer than 10 semantically meaningful, layout-informed sub-images instead of uniform patches or abstract tokens.The sub-images can represent components such as tables, figures, and paragraphs before being encoded and fused into the representation.
  • Across 24 diverse VDR datasets and 10 mainstream single-vector models, ColParse achieves an average gain of over 10 points in nDCG@5.The experiments are presented as evidence of the framework’s flexibility and robustness.
  • ColParse introduces a layout-informed paradigm for multi-vector construction that uses document parsing to overcome conventional storage-efficiency limitations.
  • The framework is training-free, plug-and-play, and demonstrates robust performance gains across existing single-vector models.
  • ColParse provides interpretable retrieval results by tracing them to specific parsed layout components, supporting practical industrial use.

2. Related Work

Related work develops multi-vector retrieval from token-level text matching toward multimodal visual-document retrieval, while document parsing models provide structured representations of rich document images. These lines of work motivate layout-aware retrieval methods that address visual-document complexity.

  • VLM-based single-vector approaches often struggle to capture the fine-grained semantics required by dense documents.
  • Multi-vector retrieval represents documents as token-level embeddings and uses late interaction for fine-grained matching.
  • ColPali adapted the multi-vector paradigm to multimodal retrieval, targeting visual documents that are more complex than natural images.
  • Document parsing VLMs convert visually rich document images into structured formats such as LaTeX or Markdown.Early sequence-to-sequence models often struggled with the computational cost of high-resolution inputs, motivating newer multi-stage approaches.

3. Methodology

ColParse replaces dense grid-based document representations with a compact set of layout-informed vectors, combining local semantic regions with global page context for retrieval.

  • Compact retrieval representation: The final representation stores only k layout-informed vectors instead of Np grid vectors, reducing storage from O(Np×D) to O(k × D).Because k ≪ Np, the compact representation also focuses MaxSim on semantically salient document regions.
  • Framework overview: ColParse’s three-stage pipeline parses each page, applies dual-stream encoding, and performs global-local fusion.The framework is designed as an offline process for each document image.
  • Layout-informed document parsing: The parser identifies layout regions such as titles, tables, and figures, then crops them into typically fewer than 10 variable-sized sub-images.The number of regions is dynamically determined by document complexity.
  • Dual-stream encoding: Each sub-image receives a local embedding, while the full page receives a global embedding that captures overall context and layout relationships.Both streams use the same standard single-vector encoder.
  • Global-local fusion: Weighted fusion injects global page context into every local vector, producing context-enriched structurally-aware representations.The balancing factor α controls the relative contribution of local and global information.
  • Scoring and theoretical motivation: ColParse computes relevance with MaxSim over its fused vectors and is theoretically motivated as a compression strategy that preserves query-relevant information.The information-bottleneck view links parsing-based disentanglement with contextual refinement.

4. Experiment

Across 24 datasets and ten single-vector models, ColParse consistently improves retrieval while using parsed layout regions and global context to build compact representations. Its gains persist across variants, balancing factors, parsing models, and challenging long-form benchmarks.

  • Main Results: ColParse consistently outperforms single-vector models and multi-vector optimization baselines across diverse benchmarks.The evaluation covers five VDR benchmark suites containing 24 datasets and ten prominent single-vector retrieval models.
  • Main Results: 31.64 points and 42.69 points are the average nDCG@5 gains for VLM2Vec-V1-2B and VLM2Vec-V1-7B on ViDoRe-V1.
  • Main Results: ColParse improves performance across VLM2Vec, GME, UniME, and B3 regardless of architecture or parameter scale.For GME-7B on ViDoRe-V1, semantic chunking reduces performance from 89.36 to 23.21, whereas ColParse maintains state-of-the-art results.
  • Variant Study: 78.41 is VLM2Vec-V2-2B’s performance with ColParse, compared with 24.85 using token-level clustering from a 74.16 reference.The comparison supports preserving visual-semantic alignment within parsed regions rather than aggregating abstract token embeddings.
  • Main Results: 32.07 and 44.21 are ColParse’s average nDCG@5 scores for VLM2Vec-V1-2B and UniME-V2-2B on MMLongBench.The corresponding baselines are 25.93 and 29.31, respectively, and ColParse outperforms other compression baselines.
  • Variant Study: 51.96 on VisRAG with VLM2Vec-V1-2B exceeds the 38.94 achieved by the s2m-add baseline.The passage attributes the gap to weighted fusion retaining both local sub-images and holistic context.

5. Conclusion

ColParse addresses the storage bottleneck in multi-vector VDR by fusing layout-aware sub-image embeddings with a global page vector. Experiments show over 95% storage compression alongside improved retrieval across base models and datasets.

  • ColParse introduces a layout-informed multi-vector paradigm that fuses parsed sub-image embeddings with a holistic global vector.
  • Over 95% storage compression accompanies consistent retrieval improvements across base models and datasets.

Impact Statement

ColParse is presented as a resource-conscious retrieval framework that combines compact, layout-informed representations with improved performance. The paper describes an offline indexing and online retrieval workflow supporting this design.

  • Ethical Considerations: The authors state that ColParse raises no new ethical concerns and follows established ethical guidelines for information retrieval research.They characterize the approach as responsible and resource-conscious while using existing document parsing technologies.
  • Societal Implications: Over 95% lower storage requirements accompany enhanced performance, reducing the barrier to large-scale deployment.The societal-implications passage frames this as resolving the tension between fine-grained retrieval accuracy and deployment cost.
  • Societal Implications: ColParse shifts multimodal retrieval from storage-intensive grid-based representations to a lightweight, layout-informed approach.The method is presented as a new paradigm for more compact and structurally informed representations.
  • Deployment Workflow: The framework separates offline indexing from online retrieval, computing relevance efficiently with a MaxSim operation over compressed embeddings.Offline indexing generates the compact representation, while online retrieval uses that representation for scoring.
  • Layout-Informed Representation: Document parsing produces bounding boxes and content types, crops sub-images, and supports encoding of local regions alongside the entire page.The workflow uses a parser for layout-informed regions and a single-vector encoder for global page context.
  • Layout-Informed Representation: The local and global representations are fused by element-wise addition to form the compact multi-vector representation.The indexing algorithm outputs the resulting representation as a set of vectors in R^k×D.

B. More Theoretical Analysis

The analysis frames VDR representation learning as an information-bottleneck problem and justifies ColParse through semantic decomposition and local-global fusion. Its theoretical claims rely on a primary-region assumption and data-processing arguments.

  • Information Bottleneck Formulation: The information bottleneck seeks a compact document representation that minimizes source information while preserving relevance information.The ideal objective is intractable at indexing time because it requires expectations over an unknown query distribution.
  • Information Bottleneck Formulation: Because the query distribution P(Q) is unknown and potentially infinite, the ideal bottleneck objective cannot be directly optimized.ColParse is presented as a practical surrogate for this objective.
  • Structural Disentanglement: The Semantic Concentration Axiom assumes that a query’s relevance is predominantly determined by one primary semantic region, with other regions adding negligible information.The paper illustrates this with a revenue query whose answer lies in a single financial table.
  • Structural Disentanglement: Under this axiom, the document’s relevance information is approximately represented by the information in its most relevant semantic region.This motivates decomposing a document into multiple region-specific representations.
  • Synergistic Fusion: ColParse fuses each local vector with a global page vector to form a set of final representations, adding page-level context to layout-specific information.The fusion is described as a contextual-information mechanism for each parsed region.
  • Synergistic Fusion: The fusion can improve relevance information only when the fused representation captures non-zero contextual information beyond the local vector.Vector addition exposes both signals during query dot-product scoring, while the data-processing inequality bounds fused information by joint component information.

C.1. Benchmark Details

The evaluation uses five VDR benchmark suites spanning varied document types, query demands, and retrieval settings, together with ten representative single-vector multimodal models. The section also describes benchmark-specific challenges and layout-aware processing procedures.

  • Benchmark Coverage: The experiments cover five benchmark suites integrated into the MMEB visdoc section for multifaceted VDR evaluation.The suites span diverse document types, query complexities, and retrieval scenarios.
  • Benchmark Coverage: ViDoRe-V1 evaluates page-level retrieval on visually rich documents containing complex layouts, tables, and figures.It combines repurposed academic VQA datasets with practical topic-specific tasks.
  • Benchmark Coverage: ViDoRe-V2 introduces long-form, cross-document, and multilingual queries generated through hybrid synthetic and human-in-the-loop processes.Its design addresses performance saturation and reduces extractive bias.
  • Benchmark Coverage: MMLongBench tests long-context multimodal understanding with documents averaging 47.5 pages, cross-page questions, multi-hop reasoning, and unanswerable queries.These properties probe information location, synthesis, and hallucination resistance.
  • Base Models: Ten representative single-vector multimodal retrieval models provide the base-model testbed for evaluating ColParse’s versatility.The models use varied architectures and pre-training paradigms.
  • Layout Processing: The layout-processing procedure first detects structural elements globally, then crops native-resolution semantic regions for parallel local recognition.This coarse-to-fine design preserves recognition accuracy while avoiding redundant full-image computation.

C.4. Main Results

The main-results section compares ColParse variants across five VDR benchmarks and defines a shared offline indexing pipeline. Variants differ in whether they retain local vectors, cluster them by content type, or append global context.

  • Benchmark Results: Results are reported across MMLongBench, ViDoRe-V1, ViDoRe-V2, ViDoSeek, and VisRAG.Tables 2 and 3 compare optimization methods and show relative gains or declines against base models.
  • Variant Construction: The unified indexing framework shares layout parsing and dual-stream encoding before constructing variant-specific multi-vector representations.This separates common processing from representation-choice effects.
  • Variant Construction: The single2multi variant retains raw sub-image vectors produced by layout decomposition.Its output is the local vector set without additional global-vector inclusion or type clustering.
  • Variant Construction: The type-clustered variant aggregates local vectors by semantic content type through averaging, whereas the global-inclusion variant appends the full-page vector.These alternatives isolate content-type aggregation from global-context augmentation.

C.5.2. MORE ANALYSIS

The ablation analysis shows that adding global page context substantially improves local layout-based retrieval, while clustering regions by content type can reduce performance. The comparisons span ColParse variants, five benchmarks, and ten base models.

  • Global Context: Adding global context raised VLM2Vec-V1-2B on ViDoRe-V1 from 34.39 to 49.93.The comparison is between the local-only single2multi baseline and simple global inclusion, s2m-g-i.
  • Global Context: The global-context gain is most pronounced on benchmarks requiring holistic understanding.Figure 5 is cited as highlighting the gap between local-only and globally augmented representations.
  • Type Clustering: Type clustering typically regressed relative to standard single2multi, including a 1.56-point drop for VLM2Vec-V1-2B on ViDoRe-V1.The radar comparisons reportedly show type-clustered variants with the narrowest performance profiles.
  • Evaluation Scope: The ablations compare ColParse and its variants using nDCG@5 across five VDR benchmarks and ten mainstream single-vector multimodal models.Detailed records are referenced in Tables 4 and 5, with visual comparisons in Figure 9.
  • Balancing Factors: Figure 10 compares model-level performance across balancing factors, distinguishing base results from the best-performing factors.Dashed lines denote base results and star points denote the best-performing balancing factors.

C.6.2. EFFECT OF DOCUMENT PARSING MODEL

MinerU2.5 achieves a strong balance of inference efficiency and parsing accuracy against specialized vision-language models. It is runner-up in reported throughput and reaches state-of-the-art performance across six OmniDocBench indicators.

  • Inference efficiency: 2422 tokens/s and 2.25 pages/s place MinerU2.5 as the runner-up for inference efficiency on A100 (80G) hardware.Tokens/sec measures generation speed, while Pages/sec measures end-to-end throughput.
  • Parsing accuracy: MinerU2.5 achieves state-of-the-art performance across all six OmniDocBench parsing indicators.The indicators include Overall, text and reading-order Edit Distances, and formula and table structural similarity metrics.
  • Parsing accuracy: MinerU2.5 records a top Overall score of 90.67 and the lowest reported error rates in text and layout recognition.Table 7 compares parsing performance across multiple tasks on OmniDocBench.

C.6.3. EFFICIENCY ANALYSIS

The efficiency analysis summarizes parsed-vector counts across 24 datasets and five VDR benchmarks, while documenting the parsing and comparison setup used for the reported tables.

  • Parsed-vector efficiency: Table 8 reports the average number of parsed vectors per document across 24 datasets in five VDR benchmarks.The table summarizes parsed-vector counts used in the efficiency analysis.
  • Benchmark setup: Table 7 benchmarks document parsing performance on OmniDocBench across multiple tasks, with best and runner-up results distinguished.The table caption states that the best and runner-up results are bolded and underlined, respectively.
  • Parsed-vector efficiency: The reported values count layout-informed sub-image embeddings generated by the MinerU2.5 document parser.The count is denoted by k in the table description.
Loading 2603.01666v1…