Cascaded Retrieval with Cross-Modal Query Refinement for Efficient Visual Document Search

• Open-domain retrieval must locate fine-grained page evidence across large corpora while keeping query-time cost affordable as the corpus grows.Source: §1 · p. 1
• Using extracted page text instead of page images reduces NDCG@10 by up to 13.5 points with the same embedding backbone.Text extraction may not preserve complex tables, figures, and layout-dependent structures completely.Source: §1 · p. 1
• Existing approaches face cost, recall, and fusion gaps: exhaustive scoring scales with corpus size, filtering risks missed pages, and rank-level fusion discards query-evidence relations.Source: §1 · p. 2
• Across 4 benchmarks, SCOUT matches or exceeds the strongest exhaustive retriever (+3.1% average NDCG@10), with up to 7.8% NDCG@5 gains and 28.8× average latency improvement (up to 80.7×).Source: §1 · p. 2
• SCOUT filters pages with a lightweight visual retriever, reranks a compact pool with late interaction, and adds offline hierarchical summaries through cross-modal query refinement.The two tracks combine before final ranking.Source: §1 · p. 2

• Text-only retrieval can miss layout and visual evidence, while native visual retrievers embed page images and late-interaction models score token- or patch-level representations.Source: §2 · p. 2
• Rank-level fusion combines retrieval outputs without embedding-space geometry; query refinement instead updates a query representation using retrieved evidence.SCOUT applies complementary-text feedback after a lightweight first stage forms the candidate pool.Source: §2 · p. 3
• Cascades pair fast first-stage retrieval with stronger reranking, while prior work also speeds multi-vector indexes through pruning, clustering, or quantization.Source: §2 · p. 3
• Prior approaches address efficiency, recall, or multimodal evidence in isolation; SCOUT uses visual filtering and offline summaries so costly reranking and refinement run on a fixed pool and textual evidence can recover missed pages.Source: §2 · p. 3

3.1 Proposed Framework: SCOUT

• SCOUT’s outputs from both tracks are jointly refined to produce the final ranked list.The framework separates offline summary generation from online retrieval over page images and precomputed summaries.Source: §3.1 · p. 3
• Each page receives a concatenated document- and page-level summary embedding and a single visual embedding.Document summaries capture global topic and structure; page summaries describe local content, including tables, figures, and named entities.Source: §3.1 · p. 3, Figure 2 · p. 4
• Track A filters pages with a lightweight single-vector visual retriever, then reranks its top-K candidates with a multi-vector late-interaction model.Late-interaction scoring is limited to the visual candidate set, bounding that stage’s cost by O(K), regardless of corpus size.Source: §3.1.1 · p. 4

3.1.2 Semantic Retrieval (Track B)

• Track B retrieves pages by scoring their hierarchical document- and page-level summaries against the query, complementing visual matching.It returns the top-T pages by summary score.Source: §3.1.2 · p. 4

3.1.3 Cross-Modal Query Refinement.

• Cross-Modal Query Refinement uses semantic scores over the union of track outputs to form a soft target for refining the visual query embedding.The refined query rescores the union; optimization is limited to at most K + T pages, independent of corpus size.Source: §3.1.3 · p. 5

Datasets

• The evaluation spans ViDoRe v3, ViDoSeek, refined SlideVQA, and REAL-MM-RAG, covering varied document collections and visually rich page types.The page types include tables, figures, charts, slides, and dense layouts.Source: §4 · p. 5

Evaluation

• Retrieval quality is measured with NDCG@5 and NDCG@10, while efficiency is measured as average online latency in milliseconds per query.Latency covers online retrieval, reranking, fusion, and refinement after offline indexing.Source: §4 · p. 5

Baselines

• Baselines include OCR-text dense retrieval models and page-image visual retrievers, including ColNomic-7B and ColEmbed-3B-v2.The text baselines use Linq-Embed-Mistral, Qwen3-Embedding, and Jina-v4 text embeddings; visual baselines include Jina-v4 vision.Source: §4 · p. 5

5.1 Performance Comparison

• 8.4× to 80.7× latency gains, averaging 28.8×, accompany SCOUT’s performance at or above state of the art across four benchmarks.SCOUT’s query latency is 387–553 milliseconds per query, compared with 4.5–32.8 seconds for ColEmbed-3B-v2.Source: §5.1 · p. 5, Table 2 · p. 6
• On ViDoRe 3, SCOUT records NDCG@5 of 58.85 and NDCG@10 of 61.31 versus 56.65 and 58.99 for SOTA, an average improvement of +3.9%.Table 2 covers ViDoSeek, SlideVQA, ViDoRe 3, and REAL-MM-RAG; its Δ(%) reports relative improvement over the best baseline.Source: §5.1 · p. 5, Table 2 · p. 6
• SCOUT achieves the best results across English and French ViDoRe subsets, with improvements up to 5.7%, and on three of four REAL-MM-RAG subsets, with gains up to 7.8%.The reported generalization results cover multilingual and domain-specific retrieval settings.Source: §5.1 · p. 6

5.2 Ablation Study

• Removing the child reranker causes the steepest reported drop, 20.90% in NDCG@10; removing Track A drops performance by 12.35%.The study evaluates average NDCG@10 on ViDoRe 3; the authors attribute visual-track accuracy to the reranker’s late-interaction scoring.Source: §5.2 · p. 6, Table 3 · p. 6
• Removing Track B alone drops NDCG@10 by 7.01%, while replacing CMQR with RRF drops it by 6.28%.The authors report that rank-level fusion discards signal that CMQR preserves at the representation level.Source: §5.2 · p. 6, Table 3 · p. 6
• At K=10, SCOUT is near peak performance and remains within 0.04 points through K=50; reducing K from 50 to 25 cuts reranking latency by 40%.The gap over the variant without CMQR narrows from 12.5 points at K=5 to 3.1 points at K=50.Source: §5.2 · p. 6

Latency Analysis

• CMQR adds +4.40 to +7.78 NDCG@10 points over the cascade alone across the listed datasets, while its refinement dominates SCOUT’s latency at about 287 ms.Refinement and reranking operate on at most Top-K pages, so their cost is independent of corpus size.Source: §6 · p. 7
• SCOUT reaches NDCG@10=67.37 at 468.2 ms/query; ColEmbed-3B-v2 is reported at 13,464.4 ms/query, and ColEmbed’s benchmark has NDCG@10=65.26.ColNomic occupies the faster, lower-quality region at 155.5 ms/query and NDCG@10=64.38.Source: §6 · pp. 6–7, Figure 3 · p. 7

Candidate-Depth Sweep

• Figure 4 examines mean NDCG@10 against latency across different reranker candidate depths K.The supplied passage identifies the sweep but does not report its measured outcomes.Source: Figure 4 · p. 7

Robustness to Query Perturbation

• SCOUT loses 4.6 NDCG@10 points on average under query perturbation, compared with 6.7 for ColEmbed and 6.3 for ColNomic.It has the highest perturbed-query NDCG@10 on all four benchmarks and the smallest drop on three; it loses 0.07 points on ViDoSeek and 2.69 on ViDoRe.Source: §6 · p. 7

RRF vs. CMQR

• CMQR improves final ranking over RRF on every ViDoRe 3 subset and all four benchmarks, with gains growing as the tracks’ top-10 rankings diverge.CMQR’s gain over ColEmbed rises from +1.9% to +5.7%, while RRF falls from −1.2% to −8.9%; CMQR uses semantic scores to refine visual query tokens before final scoring.Source: §6 · p. 8

Summary vs Earlier OCR Approaches

• Replacing VLM-generated hierarchical summaries with raw OCR text reduces average NDCG@10 by −3.6% while latency remains comparable at about 547 ms.The text attributes the gap to lost structure and noise from layout artifacts and extraction errors.Source: §6 · p. 8

Case Study

• In the three-page ViDoRe-Energy case, SCOUT places all gold pages within the top three, while ColEmbed ranks one page 42nd and ColNomic fails to surface all three consistently.The evidence spans three pages among 2,255, and the parent retriever initially ranks two pages 33rd and 42nd; reranking and the summary track improve their positions.Source: §6 · pp. 7–8, Figure 5 · p. 8

7 Conclusion

• SCOUT combines bounded late-interaction visual retrieval, offline hierarchical summaries, and cross-modal query refinement to address exhaustive search costs.Its reported latency reduction reaches 80.7× and averages 28.8× across four benchmarks, while accuracy gains reach 7.8% NDCG.Source: §7 · p. 8
• The child reranker lifts retrieval performance by 20.9% over the parent retriever alone, while CMQR adds up to 7.78 NDCG@10 points over the visual cascade.CMQR refines the query representation using semantic scores.Source: §7 · p. 8
• The authors report robustness to component choice through interchangeable retrievers.Source: §7 · p. 8

8 Limitations and Ethical Considerations

• Pages ranked low by both tracks remain unrecoverable because SCOUT can only rank pages that enter its candidate pool.Track B recovers many pages missed by the parent retriever, but does not eliminate this boundary.Source: §8 · p. 9
• The semantic track relies on proprietary Gemini 2.5 Flash summaries, which may omit or misrepresent page content.Offline indexing is a one-time cost excluded from query-time latency, but may need refreshing for frequently updated corpora.Source: §8 · p. 9
• Sensitive enterprise, financial, legal, or personal collections require access controls, data-retention policies, and privacy-preserving indexing; retrieved pages are evidence candidates, not guaranteed ground truth.The authors encourage human oversight in high-stakes decisions.Source: §8 · p. 9

A Additional Results

• All experiments are run on an NVIDIA RTX 6000 GPU.Source: B Implementation Details · p. 12

B.1 Datasets

• Four benchmarks evaluate retrieval of relevant rendered pages from full corpora, with each page serving as the retrieval unit.Source: §B.1 · p. 12

B.2 Document Retrieval Models

• Visual retrieval encodes page images directly, while text-mediated methods retrieve over OCR, parsed text, or descriptions; multi-vector late interaction preserves layout but makes exhaustive search costly.Source: §B.2 · p. 12

B.3 Offline Indexing

• Rendered pages are the shared index unit, and page embeddings are precomputed offline; SCOUT keeps the lightweight first-stage index active and accesses multi-vector embeddings only for top-K candidates.Source: §B.3 · p. 12

B.4 Offline Summarization Cost

• Hierarchical page and document summaries cost about $0.0023 and 300 ms per page offline; summarizing all four corpora costs under $100, with updates proportional to changed pages.Source: §B.4 · p. 12

B.5 Hyperparameter Selection

• GQR uses a fixed mixture weight α = 0.5 and temperature T = 1.0, with learning rate and optimization steps selected on a held-out validation split.Source: §B.5 · pp. 12–13
• Increasing semantic summary depth from T=1 to T=10 improves mean NDCG@10 by about 1.2 points, after which performance plateaus; T=20 is selected.Source: §B.5 · p. 14
• GQR optimizes the multi-vector query embedding with Adam toward a fixed average-consensus distribution, initialized from the unmodified question embedding.Source: §B.5 · p. 14

B.6 Domain and Language Generalization

• 64.27 NDCG@5 versus 60.81 for SOTA on ViDoRe-Finance is a +5.7% gain at 22× lower latency, and the reported gains remain positive across the listed ViDoRe subsets.Source: §B.6 · p. 14
• 7.8% NDCG@5 improvement on FinReport over ColEmbed is SCOUT’s largest REAL-MM-RAG gain; overall it matches or exceeds the strongest baseline on three of four subsets, with a small TechReport drop.Source: §B.6 · pp. 14–15

B.7 Downstream Performance

• 66.48 average judge score (+4.4% relative) is reported for SCOUT in end-to-end QA, improving over ColEmbed-3B-v2 on every subset.Source: §B.7 · p. 15

C Practical Considerations

• The parent retriever is interchangeable, but model choice trades storage against performance; the child reranker can also be replaced if it supplies embeddings compatible with CMQR’s similarity function.Source: C Practical Considerations · p. 15

D Reciprocal Rank Fusion

• RRF is a non-parametric rank-level fusion baseline over visual and summary retrieval lists, assigning scores to their union of pages using 1-indexed ranks and k = 60.Source: D Reciprocal Rank Fusion · p. 15

D.1 Single-hop vs. Multi-hop Complexity

• On ViDoRe single-hop queries, NDCG@10 rises from 49.37 to 58.05 after reranking and to 62.41 with full SCOUT; ViDoSeek multi-hop is effectively tied with the child reranker.Source: §D.1 · p. 16
• Full-pipeline NDCG@10 drops from 62.41 to 49.67 on ViDoRe and from 95.59 to 93.37 on SlideVQA for multi-hop queries, while ViDoSeek’s multi-hop scores are higher across stages.Source: §D.1 · p. 16
• Robustness testing compares retrieval on original and rephrased queries with gold pages, corpus, and hyperparameters held constant; accepted rewrites preserve numbers and identifiers and pass verification.Source: §D.1 · pp. 16–17, Figure 11 · p. 21

E Generative AI Use

• Generative AI assistants were used for grammatical revision, stylistic polishing, and coding assistance, not substantive outsourcing of intellectual work.Source: E Generative AI Use · p. 17

F Document-Level Summary Prompt

• Document-level summary fields are concatenated and prepended to each page summary before embedding, with global_summary truncated to 300 characters and key_entities to 15 entries.Source: Figure 9 · p. 19

G Page-Level Summary Prompt

• Each page-level semantic summary contains a high-level overview, schema-level descriptions of tables, charts, and figures, and grouped named entities.Source: Figure 10 · p. 20

H Query Paraphrase Prompts

• The query-rephrasing prompts rewrite wording and sentence order while preserving intent; candidates are retained only after number-and-identifier checks and independent verification.Source: Figure 11 · p. 21, H Query Paraphrase Prompts · p. 21