PaperlayerView exact revision on arXiv

Source-linked AI summary

EVIE: Evidence-Vector-Informed Embeddings for Visual Document Retrieval

Zifei Wang, Wei Wen, Qiang Ji, Qian-Wen Zhang, Ruizhi Qiao, Xing Sun

arXiv:2610.11553v1cs.IR

TL;DR

Visual document retrieval needs fine-grained page evidence without the storage burden of large multi-vector indexes. EVIE combines evidence-judged supervision, prefix-wise teacher–student learning, and HAC compression; across 138 tasks, it reports strong retrieval results alongside compact-index performance.

  • Problem

    Visual document retrieval must preserve fine-grained page evidence while addressing storage costs, and mined negatives may contain complete or partial answer evidence.

    Source: §1 · p. 2

  • Method

    EVIE combines evidence-judged supervision, bidirectional teacher–student learning with Prefix-MRL, and HAC semantic-centroid compression for direct MaxSim retrieval.

    Source: §1 · pp. 2–3

  • Results

    Across 138 tasks, EVIE-8B reaches 66.75 nDCG@10 on ViDoRe V3, exceeding the strongest compared external baseline by 1.43 points.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The evaluations support adapting fine-grained visual document retrieval to different storage budgets, including HAC retrieval at 59.58 nDCG@10 with 3.81 GiB per million pages.

    Source: §7 · p. 17

    The public-dataset checks assess validity and relevance, not comprehensive privacy or harmful-content risks; deployment still requires corpus-specific privacy review and access controls.

    Source: ETHICS STATEMENT · p. 18

Abstract

from arXiv · show

Accurate and scalable visual document retrieval (VDR) requires both fine-grained page understanding and efficient indexing, yet existing approaches struggle to achieve both. OCR-based text retrieval adds preprocessing latency and can lose visual and structural cues needed to understand complex pages. Single-vector vision-language models bypass OCR, but compressing an entire page into one vector limits the granularity of query--document matching. Multi-vector retrievers with MaxSim provide finer interactions, yet demand large indexes and still leave room for accuracy improvements. We argue that overcoming these limitations requires preserving query-relevant page evidence throughout representation learning and index construction. To this end, we introduce \textbf{\textit{EVIE}} (Evidence-Vector-Informed Embeddings), a family of native visual document retrievers integrating three key innovations: (1) Evidence-judged data governance, which uses a multimodal judge to identify answer-bearing positives and filter unreliable negatives. (2) Bidirectional teacher--student learning with symmetric listwise distillation and prefix-based Matryoshka representation learning (Prefix-MRL), enabling one student checkpoint to serve six nested embedding dimensions without re-encoding. (3) Hierarchical agglomerative index compression (HAC), which clusters page tokens with spatial regularization and stores semantic centroids for single-stage MaxSim retrieval. Extensive experiments across 138 tasks from ViDoRe V1, V2, V3, and JinaVDR validate EVIE. EVIE-8B achieves 66.75 nDCG@10 on V3, exceeding the best external baseline by 1.43 points, with a four-suite average of 79.51. EVIE-4.5B with HAC retains 59.58 nDCG@10 at only 3.81 GiB per million pages, reducing vector payload by $128\times$. Together, these results improve the accuracy--storage trade-off for visual document retrieval.

1 INTRODUCTION

EVIE targets visual document retrieval’s joint challenge of preserving fine-grained page evidence while keeping indexing compact. It combines evidence-judged supervision, prefix-wise teacher–student learning, and hierarchical token compression, with benchmark results spanning 138 tasks.

1 INTRODUCTION

  • Visual document search must preserve relationships such as chart bar heights, time, and legends that OCR text may omit, while controlling preprocessing and index costs.The paper identifies evidence supervision, representations trained across widths, and compressed tokens for final relevance scoring as coupled requirements.Source: §1 · p. 2
  • 66.75 nDCG@10 on ViDoRe V3 lets EVIE-8B exceed the strongest compared external baseline by 1.43 points.Across 138 tasks from ViDoRe V1, V2, V3, and JinaVDR, EVIE-4.5B with HAC retains 59.58 at 3.81 GiB per million pages, a 128× vector-payload reduction.Source: §1 · p. 3
  • Evidence-judged supervision promotes candidates with complete answer evidence to positives and excludes partial-evidence candidates from the negative loss.A multimodal judge audits mined candidates; the reported audit finds complete- and partial-evidence categories in 14.4% of mined candidate negative slots.Source: §1 · p. 3
  • Bidirectional teacher–student learning with Prefix-MRL trains six nested embedding widths with retrieval and ranking objectives in one checkpoint, without re-encoding.The student receives symmetric bidirectional listwise distillation from the teacher.Source: §1 · p. 3
  • HAC uses spatially regularized cluster assignments but stores semantic centroids, enabling single-stage MaxSim retrieval over compressed page tokens.The centroids are computed from the original semantic vectors and support direct scoring without an uncompressed index or reranking pass.Source: §1 · p. 3

2 RELATED WORK

Related work covers visual document encoders, retrieval supervision and distillation, variable-width multi-vector representations, and index compression. EVIE connects these lines around answer evidence and the accuracy–storage trade-off.

2.1 VISUAL DOCUMENT RETRIEVAL

  • Native visual document retrievers range from compact single-vector encoders to multi-vector late interaction, which matches query tokens to page regions.The section also situates EVIE among bidirectional encoders and broader multimodal retrieval models.Source: §2.1 · p. 4

2.2 SUPERVISION AND DISTILLATION FOR RETRIEVAL

  • EVIE grades mined candidates by answer completeness: full-evidence candidates become positives, partial-evidence candidates are masked from negative loss, and no-evidence candidates remain negatives.The teacher’s query-to-document and document-to-query ranking distributions are transferred to the student across embedding widths.Source: §2.2 · p. 4

2.3 VARIABLE-DIMENSION MULTI-VECTOR REPRESENTATIONS

  • Prefix-MRL applies retrieval supervision and bidirectional listwise distillation at every trained prefix width because truncation changes token similarities used by MaxSim.A shared projection serves six dimensions from 64 to 2048 without rerunning the backbone.Source: §2.3 · p. 4

2.4 MULTI-VECTOR INDEX COMPRESSION

  • HAC uses 2D spatial regularization to guide clustering, while stored centroids average only original semantic vectors for direct MaxSim scoring.Together with Prefix-MRL, this allows embedding width and token count to vary without retaining an uncompressed reranking index.Source: §2.4 · p. 5

3 MODEL ARCHITECTURE AND LATE INTERACTION REPRESENTATION

EVIE independently encodes queries and page images as token vectors, then uses late interaction to connect relevance supervision, variable embedding widths, and compressed indexing.

3 MODEL ARCHITECTURE AND LATE INTERACTION REPRESENTATION

  • Queries and page images are independently encoded into token vectors, with late interaction providing their relevance-scoring interface.The page sequence includes image and context tokens; the figure shows shared affine projection, prefix normalization, and padding-row zeroing before MaxSim.Source: §3 · p. 5, Figure 4 · p. 6

3.1 INPUT SEQUENCES AND CONTEXTUAL ENCODING

  • Eight full-attention layers per 32-layer backbone have causal masking removed, while the other 24 linear/recurrent layers remain unchanged.Bidirectional context is confined to each independently encoded input; query–page interaction occurs only during scoring.Source: §3.1 · p. 5

3.2 MODEL INITIALIZATION AND PREFIX PROJECTION

  • A shared affine projection maps query and page tokens to nested student widths of 64, 128, 256, 512, 1024, or 2048.Each prefix is normalized and padding rows are zeroed; the teacher uses width 4096, and prefixes can be supplied by slicing without rerunning the backbone.Source: §3.2 · p. 5, Figure 4 · p. 6

3.3 LATE INTERACTION SCORING (MAXSIM)

  • MaxSim masks invalid page positions before taking each query token’s maximum, excludes query padding, and uses a query mean in training versus a sum in evaluation.For a fixed query, the mean and sum give the same ranking.Source: §3.3 · p. 6

4 TRAINING AND DISTILLATION FRAMEWORK

EVIE trains evidence-sensitive MaxSim rankings in two stages: retrieval supervision establishes a teacher, then teacher rankings and margins supervise every student prefix while an anchor preserves initial d128 geometry.

4.1 EVIDENCE-JUDGED DATA GOVERNANCE

  • The judge promotes full-evidence pages and excludes partial-evidence pages from negative loss; 14.4% of candidate slots are masked before positive expansion.Source: §4.1 · p. 6

4.2 RETRIEVAL SUPERVISION AND TEACHER CONSTRUCTION

  • Teacher retrieval combines masked-negative ranking and in-batch cross-entropy; two 4096-dimensional branches use sampled-positive and all-positive views before a 0.4/0.6 frozen ensemble.The sampled branch draws one positive per record; the all-positive branch weights records and excludes same-group entries.Source: §4.2 · p. 7

4.3 PREFIX-WISE RETRIEVAL AND DISTILLATION

  • At each of six widths, the student combines retrieval labels with bidirectional teacher-to-student ranking KL, confidence weighting, explicit-negative margins, and an anchor on d128.Source: §4.3 · pp. 7–8

4.4 OPTIMIZATION DETAILS

  • The student retains Preview’s first 128 projection rows, warms added channels alone for 100 steps, then jointly trains LoRA and the full projection.Each teacher branch and the student train for one epoch on 32 GPUs.Source: §4.4 · pp. 8–9

5 HAC: HIERARCHICAL AGGLOMERATIVE INDEX COMPRESSION

HAC compresses page-token indexes by clustering position-guided semantic features while storing semantic centroids for unchanged single-stage MaxSim retrieval.

5.1 POSITION-GUIDED CLUSTERING AND SEMANTIC CENTROIDS

  • Unlike ColParse, HAC pools existing page tokens using the processor grid without region parsing or re-encoding.Source: §5.1 · p. 9
  • HAC uses page-grid positional features with semantic vectors for Ward assignments, then averages original retrieval vectors into centroids without averaging positional features.The positional weight is fixed at 0.1, and assignments are recomputed for each width and centroid-budget configuration.Source: §5.1 · p. 9
  • Queries remain unclustered at the selected width and score directly against stored centroids with MaxSim, though pooling can alter the best-matching vector and score.Source: §5.1 · p. 9, §5 · p. 9

5.2 INDEX STORAGE ACCOUNTING

  • 3.81 GiB per million pages at K=32, d=64 is a 128× reduction versus EVIE’s configured 1024×256 maximum visual-token payload.Storage estimates exclude document IDs, offsets, serialization overhead, and model weights.Source: §5.2 · pp. 9–10

6 EXPERIMENTS AND BENCHMARK RESULTS

Across four benchmark suites and 138 tasks, the experiments assess retrieval quality and storage budgets. EVIE-8B leads the reported V3 comparison, while HAC configurations retain retrieval quality at smaller payloads.

6.1 BENCHMARK SUITE AND EVALUATION PROTOCOL

  • The evaluation covers 138 tasks across ViDoRe V1, V2, V3, and JinaVDR, with suite scores formed by task macro-averages and Avg4 equally weighting the four suites.EVIE runs use single-stage MaxSim without reranking; external model tokenization, resolution budgets, and training data may differ.Source: §6.1 · p. 11

6.2 MAIN RETRIEVAL RESULTS ACROSS BENCHMARKS

  • 66.75 nDCG@10 on V3 puts EVIE-8B 1.43 points above the strongest external baseline; its Avg4 score is 79.51.The V3 comparison covers eight public domains; Avg4 is the unweighted mean of the four suite scores.Source: §6.2 · pp. 11–12, Table 2 · p. 12, Table 3 · p. 12
  • 59.58 nDCG@10 at 3.81 GiB per million pages is HAC’s compact operating point, 2.16 points above WeMM-9B at half its vector payload.At the same 7.63 GiB payload as WeMM-9B, HAC reaches 62.06, a gain of 4.64 points.Source: §6.2 · p. 12, Table 2 · p. 12

6.3 CONTROLLED VALIDATION OF THE THREE CONTRIBUTIONS

  • Evidence-judged governance raises V3 from 62.36 to 66.33 and Avg4 from 75.57 to 79.17 across its full sequence of supervision steps.Masking full-evidence negatives yields the largest incremental gain; excluding partial-evidence candidates adds a further 0.56 V3 points and 0.19 Avg4 points.Source: §6.3 · p. 13, Table 5 · p. 13
  • Bidirectional listwise distillation outperforms query-to-page distillation at all three reported widths, while the complete student objective gains most at the narrowest width.The ablations vary width-wise supervision and objective components under matched training conditions.Source: §6.3 · p. 14, Table 7 · p. 14
  • At d = 64, image-token pooling accounts for 4.81 of HAC’s 4.93-point V3 loss, while spatial guidance adds 1.46 V3 points over semantic-only Ward pooling.Removing context tokens changes V3 by only 0.12 points; HAC uses spatial information for assignment and semantic vectors for stored centroids.Source: §6.3 · pp. 14–15, Table 8 · p. 15

6.4 ACCURACY–STORAGE TRADE-OFFS AND BUDGET ALLOCATION

  • At equal payload, (d = 64, K = 64) gains 0.70 Avg4 points over (d = 128, K = 32); the advantage for narrower, more numerous centroids shrinks at larger budgets.The expanded comparison reports the highest V3 and Avg4 scores for d = 64 at all three tested budgets.Source: §6.4 · p. 16, Table 11 · p. 16, Figure 7 · p. 16

6.5 SYSTEM MEASUREMENTS AND EVIDENCE-LEVEL ANALYSIS

  • HAC (K = 32) reduces median search time from 0.909 to 0.160 ms (5.7×) versus raw d = 64 and cuts GPU index memory by 32×.Across 3,588 queries with a relevant page, 2,521 stay in the top three under both indexes, while 49 fall from raw rank ≤3 to HAC rank >10.Source: §6.5 · p. 17, Table 12 · p. 17, Figure 8 · p. 17

7 CONCLUSION AND FUTURE WORK

EVIE combines evidence-judged supervision, bidirectional teacher–student learning with Prefix-MRL, and HAC. The reported results show an accuracy–storage trade-off, while future work targets image queries that could reuse the compressed document index.

7 CONCLUSION AND FUTURE WORK

  • EVIE-8B reaches 66.75 nDCG@10 on ViDoRe V3, while EVIE-4.5B with HAC retains 59.58 at 3.81 GiB per million pages.The reported evaluations span 138 tasks and support adapting retrieval to different storage budgets.Source: §7 · p. 17
  • Future work aims to align screenshot and document-crop queries with page representations so both modalities can reuse the same compressed index and reduce duplicate vector storage.The stated future direction is to support these image queries alongside text queries.Source: §7 · p. 17
  • The work uses public retrieval datasets but does not claim comprehensive privacy or harmful-content auditing; corpus-specific privacy review and access controls remain necessary for deployment.Public pages may contain personally identifiable information, and dataset licenses or underlying document restrictions still apply.Source: ETHICS STATEMENT · p. 18

A DETAILED MULTI-METRIC RESULTS

The reported results use multiple retrieval metrics across four suites, with scores macro-averaged at the task and suite levels. Tables 13–15 cover the EVIE-8B teacher, the EVIE-4.5B student at d = 2048, and HAC at d = 64, K = 32.

A DETAILED MULTI-METRIC RESULTS

  • V3 results use the same 48 public domain–language tasks, while relevance is defined differently for single-page and graded-qrel suites.V1 and JinaVDR use one binary relevant page per query; V2 and V3 treat qrel gains g > 0 as relevant for Recall, MAP, and MRR, while nDCG retains graded gains.Source: A DETAILED MULTI-METRIC RESULTS · p. 20
  • Scores are averaged over queries within each task, macro-averaged over tasks within each suite, and combined in Avg4 as an unweighted mean.All scores use a 0–100 scale; MAP is normalized by min(k, |Rq|), and MAP and MRR coincide at k = 1.Source: A DETAILED MULTI-METRIC RESULTS · p. 20
  • Tables 13–15 report complete retrieval metrics for three configurations: the EVIE-8B teacher, the EVIE-4.5B student at d = 2048, and HAC at d = 64, K = 32.Source: Table 13 · p. 20, Table 14 · p. 20, Table 15 · p. 21

B DATASET PROVENANCE

The local evaluator uses archived ViDoRe dataset snapshots. Table 16 identifies the repositories, revisions, and configurations used.

B DATASET PROVENANCE

  • The evaluator uses 22 ViDoRe repositories: ten V1 datasets, four V2 datasets, and eight public V3 domain datasets.All listed configurations use the test split.Source: B DATASET PROVENANCE · p. 21, Table 16 · p. 21
  • V3 exports provide six query languages per domain and share the English corpus across those language tasks.Revisions were recovered from per-file Hugging Face download metadata accompanying local snapshots, not current default branches.Source: B DATASET PROVENANCE · p. 21, Table 16 · p. 21

C ADDITIONAL CONTROLLED COMPARISONS

The controlled comparisons examine teacher construction, compression transfer, and governance sensitivity, alongside the audit and tuning procedures used to set them.

C ADDITIONAL CONTROLLED COMPARISONS

  • The human audit samples 900 candidates equally across three judge tiers, with three annotators independently assigning answer-completeness categories.The majority vote defines the human label; each table cell is a majority-vote count divided by 300 candidates from that tier.Source: C ADDITIONAL CONTROLLED COMPARISONS · pp. 21–22
  • The positional weight wpos = 0.1 was selected using an earlier student checkpoint at d = 128, K = 32, then applied unchanged.Table 8 evaluates the fixed setting without retuning for the final checkpoint.Source: C ADDITIONAL CONTROLLED COMPARISONS · p. 22
  • At the evaluated 0.4/0.6 weighting, the teacher mixture exceeds the stronger branch by 0.42 V3 points and 0.34 Avg4 points.The mixture combines sampled-positive and all-positive branches; this single comparison does not establish optimal weights or improvement across training seeds.Source: C ADDITIONAL CONTROLLED COMPARISONS · p. 22, Table 17 · p. 22
  • HAC improves over Ward by 1.46, 1.28, and 0.78 points on EVIE, Jina, and Nemotron, respectively.Raw-to-HAC losses are 4.81, 5.50, and 2.86 points; the comparison uses K = 32, but payloads are not matched across encoders.Source: C ADDITIONAL CONTROLLED COMPARISONS · p. 22, Table 18 · p. 22
  • Full governance raises raw and HAC scores by 3.01 and 4.78 points, while compression loss falls from 6.58 to 4.81 points.The 1.77-point interaction estimate is descriptive, without uncertainty intervals.Source: C ADDITIONAL CONTROLLED COMPARISONS · p. 22, Table 19 · p. 22
Loading 2610.11553v1…