Source-linked AI summary
ViSAR: Training-Free Adaptive-$k$ Retrieval for Visual Document Question Answering
Adrien Mialland, Marc Plantevit, Julien Gallois, Céline Robardet
TL;DR
DocVQA retrieval must choose how many pages to send to an LVLM, since fixed top-k selection may add latency and irrelevant context while omitting relevant pages. ViSAR performs training-free adaptive-k retrieval directly in late-interaction embedding space using a query-conditioned page-level similarity matrix. Across multiple encoders and LVLMs, it reduces RAG latency by up to 58.7% while maintaining or improving answer accuracy, and its matrix structure correlates with accuracy.
Problem
Fixed top-k retrieval cannot adapt page selection to query complexity, creating a trade-off between irrelevant context and omitted relevant pages in multi-page DocVQA.
Method
ViSAR is a training-free retrieval mechanism that weights late-interaction Query-to-Page, Page-to-Query, and Page-to-Page interactions using a query-conditioned page-level similarity matrix.
Results
Up to 58.7% lower RAG latency, with fewer retrieved pages on average and maintained or improved answer accuracy across multiple encoders and LVLMs.
Takeaways & Limitations
Similarity-matrix structure reflects localization of query-relevant information and correlates with answer accuracy, providing a potential feedback signal for retrieval strategies.
Takeaways & Limitations
Numerical-instability cases affecting ColModernVBERT encoding on 7.31% of LongDocURL queries were excluded from evaluation, and ViSAR overhead becomes noticeable on the largest document.
Abstract
from arXiv · showhide
Document Visual Question Answering (DocVQA) often leverages Retrieval-Augmented Generation (RAG), where late-interaction encoders are commonly used to identify document pages relevant to a user query, before answer generation by a Large Vision-Language Model (LVLM). Existing approaches typically retrieve a fixed top-$k$ number of pages regardless of query complexity, which increases LVLM latency and may degrade answer accuracy. We introduce ViSAR (Visual Semantic Activation Retrieval), a training-free adaptive-$k$ retrieval method for late-interaction visual document retrieval. ViSAR operates directly in the embedding space to construct a query-conditioned page-level similarity matrix that highlights query-relevant semantics and dynamically determines the number of pages to retrieve. Across multiple encoders and LVLMs, ViSAR retrieves compact, query-adapted page sets that reduce RAG latency by up to 58.7\%, while maintaining or improving answer accuracy compared with fixed top-$k$ and adaptive retrieval heuristics. Furthermore, we show that the similarity matrix structure correlates with answer accuracy, suggesting future directions for retrieval quality-aware document understanding.
1 Introduction
DocVQA systems must select relevant pages for LVLM reasoning, but fixed top-k retrieval cannot adapt to query complexity and can add latency or irrelevant context. ViSAR uses late-interaction embeddings to construct query-conditioned semantic structure for training-free adaptive-k retrieval.
- Motivation: DocVQA answers questions over visually rich document pages containing text, images, charts, tables, and document-specific layouts.Multiple-page documents require processing only the relevant subset of pages.
- Motivation: RAG encodes queries and pages in a shared embedding space, then retrieves pages for LVLM answer generation.Late-interaction improves page representation, but the number of retrieved pages remains a critical design choice.
- Problem: Fixed top-k retrieval cannot adapt to queries, potentially adding irrelevant pages, increasing LVLM latency, degrading accuracy, or omitting relevant pages.Late-interaction produces independent page relevance scores, which leads to fixed-k selection.
- ViSAR: ViSAR is a training-free adaptive-k mechanism that operates on late-interaction multi-vector embeddings and dynamically selects pages at inference time.It constructs a query-conditioned page-level similarity matrix that highlights query semantics.
- Results: ViSAR retrieves fewer pages on average and reduces RAG latency by up to 58.7% while maintaining or improving answer accuracy across multiple encoders and LVLMs.The similarity matrix structure reflects semantic localization and correlates with answer accuracy.
2 Related Work
Related work has explored late-interaction retrieval, adaptive-k heuristics, and OCR-free visual DocVQA systems. ViSAR differs by exploiting semantic structure in late-interaction visual embeddings without requiring encoder- or LVLM-specific retraining.
- Late-Interaction Retrieval: Late-interaction represents queries and pages as independent multi-vector embeddings and aggregates each query embedding’s best page match for relevance.This supports fine-grained semantic matching, offline page encoding, and scalable online retrieval.
- Adaptive-k Retrieval: Adaptive retrieval has used iterative fixed-k rounds or single-pass score heuristics based on gaps and clustering.These methods estimate how many pages to retrieve rather than always using one fixed k.
- Visual Document Retrieval and DocVQA: OCR-free DocVQA systems combine page selection with LVLM reasoning, using single-vector retrievers, multi-vector late-interaction retrieval, improved visual encoders, or iterative retrieval.Effective page selection is a key component of the reasoning pipeline.
- Visual Document Retrieval and DocVQA: ViSAR addresses prior work’s gap by exploiting semantic structure induced by late-interaction representations while remaining agnostic to the encoder and LVLM.Its approach is distinct from methods that rely on discrete token structures or additional training.
3 Visual Semantic Activation Retrieval (ViSAR)
ViSAR uses weighted multi-level interactions in late-interaction embedding space to construct query-conditioned page similarities and select an adaptive number of pages. Its retrieval objective balances relevant-page coherence against irrelevant-page leakage while exploiting sparsity for efficient computation.
- Method: ViSAR enables adaptive-k retrieval by weighting Query-to-Page, Page-to-Query, and Page-to-Page interactions without encoder retraining.It uses the MaxSim operator to exploit fine-grained semantic structure in the embedding space.
- Query-to-Page Interaction Weighting: The full activation matrix identifies discriminative query semantics instead of aggregating activations into one late-interaction score.Activation scores measure how strongly each query semantic is realized in a page.
- Query-to-Page Interaction Weighting: Aggregating normalized activations across pages penalizes ubiquitous semantic content and highlights sparse activations, producing query-vector weights.Page-level semantic co-activation similarly yields weights for pages.
- Page-to-Page Interaction Weighting: Weighted patch-query and patch-patch similarities produce query-conditioned directional similarities between pages and a page-level similarity matrix.The matrix is computed by averaging the T largest source-to-target interactions, and Sim(p, p′) need not equal Sim(p′, p).
- Adaptive-k Retrieval: The adaptive-k objective ranks pages by self-similarity, then minimizes a cost combining relevant-set coherence with irrelevant-set leakage.At most N candidate sets are evaluated instead of 2^N, and an additional transition test may accept the next page.
- Implementation: Inactive pages with zero patch weights are excluded, while block-wise evaluation reduces peak memory without changing the mathematical result.These choices avoid unnecessary computation induced by ViSAR's natural sparsity.
4 Experiments
Experiments evaluate ViSAR across datasets, visual and text encoders, LVLMs, and adaptive-retrieval baselines. ViSAR generally produces more compact page sets, maintains or improves answer accuracy, reduces latency, and links similarity-matrix structure with answer quality, with document size and encoding stability defining scope boundaries.
- Experimental Setup: Across MMLongBench and LongDocURL, ViSAR is evaluated with three OCR-free visual encoders, ColBERTv2, VisRAG-Ret, and multiple adaptive or fixed top-k baselines.The experiments use answer-evidence pages for ranking evaluation and multi-page visual and textual reasoning scenarios.
- Experimental Setup: 7.31% of LongDocURL queries using ColModernVBERT were excluded because numerical instabilities arose during encoding, independently of ViSAR.The excluded samples affected only a few pages and were identified through investigation of the encoder pipeline.
- 4.2.1 Adaptive Retrieval: ViSAR retrieves fewer pages than Oracle on average, follows Oracle at low kOracle, and becomes more compact as kOracle increases, whereas Largest-Gap and Score-Cluster over-retrieve.This behavior favors precision for ViSAR, while the heuristic methods tend toward recall; ViSAR achieves higher F1 particularly at lower kOracle on MMLongBench.
- 4.2.1 Adaptive Retrieval: ViSAR improves page ranking across Recall and NDCG at ranks 5 and 10, although ColModernVBERT shows more modest gains and weaker overall retrieval performance.The weaker trend is attributed in the passage to ColModernVBERT’s smaller 250M-parameter size versus 3B for the stronger encoder.
- 4.2.2 Answer Accuracy: Across 60 encoder–LVLM–budget–dataset configurations, ViSAR improves accuracy in 24 cases and maintains it in the remaining 36, with no reported decreases.Improvements are largest for LVLMs more sensitive to longer contexts; all adaptive methods outperform fixed top-k in the literature-baseline comparison.
- 4.2.3 RAG Latency: 58.7% is the maximum MMLongBench end-to-end latency reduction at Max-10, while ViSAR’s retrieval overhead becomes noticeable only for the 468-page document.The reduction is driven by lower generation cost from smaller LVLM contexts; approximations are proposed for extreme document lengths.
- 4.2.4 Similarity Structure Reflects Accuracy: Higher similarity-matrix sparsity consistently correlates with higher answer accuracy, while dense matrices produce shallow retrieval minima and can accompany over-retrieval and incorrect answers.The matrix structure is presented as a possible label-free feedback signal, and ablations indicate that both the weighting strategy and objective contribute to performance.
5 Conclusion
ViSAR enables compact, adaptive page selection for DocVQA by operating in late-interaction embedding space. It reduces RAG latency while maintaining or improving answer accuracy, and its similarity structure correlates with answer accuracy.
- ViSAR is a training-free visual embedding weighting mechanism for adaptive-k retrieval in document visual question answering.It operates in the embedding space of late-interaction encoders.
- ViSAR constructs a query-conditioned page-level similarity matrix that drives compact, adaptive page selection.The matrix reflects the localization of query-relevant information.
- 58.7%: ViSAR reduces RAG latency by up to 58.7% while maintaining or improving answer accuracy.
- The similarity matrix structure correlates with answer accuracy and may provide a useful feedback signal for future retrieval strategies.
6 Supporting Information
The supplementary material contains Figures S1–S14 and Tables S1–S10.
- Figures S1–S14 and Tables S1–S10 are available in the supplementary material PDF.