ViSAR: Training-Free Adaptive-$k$ Retrieval for Visual Document Question Answering

Motivation

• DocVQA answers questions over visually rich document pages containing text, images, charts, tables, and document-specific layouts.Multiple-page documents require processing only the relevant subset of pages.Source: §1 · pp. 1–2
• RAG encodes queries and pages in a shared embedding space, then retrieves pages for LVLM answer generation.Late-interaction improves page representation, but the number of retrieved pages remains a critical design choice.Source: §1 · p. 2

Problem

• Fixed top-k retrieval cannot adapt to queries, potentially adding irrelevant pages, increasing LVLM latency, degrading accuracy, or omitting relevant pages.Late-interaction produces independent page relevance scores, which leads to fixed-k selection.Source: §1 · p. 2

ViSAR

• ViSAR is a training-free adaptive-k mechanism that operates on late-interaction multi-vector embeddings and dynamically selects pages at inference time.It constructs a query-conditioned page-level similarity matrix that highlights query semantics.Source: §1 · pp. 2–3, Figure 1 · p. 2

Results

• ViSAR retrieves fewer pages on average and reduces RAG latency by up to 58.7% while maintaining or improving answer accuracy across multiple encoders and LVLMs.The similarity matrix structure reflects semantic localization and correlates with answer accuracy.Source: §1 · p. 3

Late-Interaction Retrieval

• Late-interaction represents queries and pages as independent multi-vector embeddings and aggregates each query embedding’s best page match for relevance.This supports fine-grained semantic matching, offline page encoding, and scalable online retrieval.Source: §2.1 · p. 3

Adaptive-k Retrieval

• Adaptive retrieval has used iterative fixed-k rounds or single-pass score heuristics based on gaps and clustering.These methods estimate how many pages to retrieve rather than always using one fixed k.Source: §2.2 · p. 3

Visual Document Retrieval and DocVQA

• OCR-free DocVQA systems combine page selection with LVLM reasoning, using single-vector retrievers, multi-vector late-interaction retrieval, improved visual encoders, or iterative retrieval.Effective page selection is a key component of the reasoning pipeline.Source: §2.3 · p. 3
• ViSAR addresses prior work’s gap by exploiting semantic structure induced by late-interaction representations while remaining agnostic to the encoder and LVLM.Its approach is distinct from methods that rely on discrete token structures or additional training.Source: §2.3 · p. 3

Method

• ViSAR enables adaptive-k retrieval by weighting Query-to-Page, Page-to-Query, and Page-to-Page interactions without encoder retraining.It uses the MaxSim operator to exploit fine-grained semantic structure in the embedding space.Source: §3 · p. 3

Query-to-Page Interaction Weighting

• The full activation matrix identifies discriminative query semantics instead of aggregating activations into one late-interaction score.Activation scores measure how strongly each query semantic is realized in a page.Source: §3.2 · p. 4
• Aggregating normalized activations across pages penalizes ubiquitous semantic content and highlights sparse activations, producing query-vector weights.Page-level semantic co-activation similarly yields weights for pages.Source: §3.2.1 · p. 4

Page-to-Page Interaction Weighting

• Weighted patch-query and patch-patch similarities produce query-conditioned directional similarities between pages and a page-level similarity matrix.The matrix is computed by averaging the T largest source-to-target interactions, and Sim(p, p′) need not equal Sim(p′, p).Source: §3.2.2 · p. 5, §3.2.3 · p. 5

Adaptive-k Retrieval

• The adaptive-k objective ranks pages by self-similarity, then minimizes a cost combining relevant-set coherence with irrelevant-set leakage.At most N candidate sets are evaluated instead of 2^N, and an additional transition test may accept the next page.Source: §3.2.4 · p. 5

Implementation

• Inactive pages with zero patch weights are excluded, while block-wise evaluation reduces peak memory without changing the mathematical result.These choices avoid unnecessary computation induced by ViSAR's natural sparsity.Source: §3.2.5 · p. 5

Experimental Setup

• Across MMLongBench and LongDocURL, ViSAR is evaluated with three OCR-free visual encoders, ColBERTv2, VisRAG-Ret, and multiple adaptive or fixed top-k baselines.The experiments use answer-evidence pages for ranking evaluation and multi-page visual and textual reasoning scenarios.Source: §4.1 · p. 6
• 7.31% of LongDocURL queries using ColModernVBERT were excluded because numerical instabilities arose during encoding, independently of ViSAR.The excluded samples affected only a few pages and were identified through investigation of the encoder pipeline.Source: §4.2 · p. 6

4.2.1 Adaptive Retrieval

• ViSAR retrieves fewer pages than Oracle on average, follows Oracle at low kOracle, and becomes more compact as kOracle increases, whereas Largest-Gap and Score-Cluster over-retrieve.This behavior favors precision for ViSAR, while the heuristic methods tend toward recall; ViSAR achieves higher F1 particularly at lower kOracle on MMLongBench.Source: §4.2.1 · pp. 6, 8
• ViSAR improves page ranking across Recall and NDCG at ranks 5 and 10, although ColModernVBERT shows more modest gains and weaker overall retrieval performance.The weaker trend is attributed in the passage to ColModernVBERT’s smaller 250M-parameter size versus 3B for the stronger encoder.Source: §4.2.1 · p. 8

4.2.2 Answer Accuracy

• Across 60 encoder–LVLM–budget–dataset configurations, ViSAR improves accuracy in 24 cases and maintains it in the remaining 36, with no reported decreases.Improvements are largest for LVLMs more sensitive to longer contexts; all adaptive methods outperform fixed top-k in the literature-baseline comparison.Source: §4.2.2 · p. 8

4.2.3 RAG Latency

• 58.7% is the maximum MMLongBench end-to-end latency reduction at Max-10, while ViSAR’s retrieval overhead becomes noticeable only for the 468-page document.The reduction is driven by lower generation cost from smaller LVLM contexts; approximations are proposed for extreme document lengths.Source: §4.2.3 · pp. 8–9

4.2.4 Similarity Structure Reflects Accuracy

• Higher similarity-matrix sparsity consistently correlates with higher answer accuracy, while dense matrices produce shallow retrieval minima and can accompany over-retrieval and incorrect answers.The matrix structure is presented as a possible label-free feedback signal, and ablations indicate that both the weighting strategy and objective contribute to performance.Source: §4.2.4 · pp. 9–10, §4.2.5 · p. 10

• ViSAR is a training-free visual embedding weighting mechanism for adaptive-k retrieval in document visual question answering.It operates in the embedding space of late-interaction encoders.Source: §5 · p. 11
• ViSAR constructs a query-conditioned page-level similarity matrix that drives compact, adaptive page selection.The matrix reflects the localization of query-relevant information.Source: §5 · p. 11
• 58.7%: ViSAR reduces RAG latency by up to 58.7% while maintaining or improving answer accuracy.Source: §5 · p. 11
• The similarity matrix structure correlates with answer accuracy and may provide a useful feedback signal for future retrieval strategies.Source: §5 · p. 11

• Figures S1–S14 and Tables S1–S10 are available in the supplementary material PDF.Source: Figure S1