PaperlayerView exact revision on arXiv

Source-linked AI summary

RetrievalRouter: Joint Modality and Architecture Selection for Document Retrieval

Emre Kuru, Mehmet Onur Keskin, Reza Farahbakhsh, Noel Crespi

arXiv:2608.25625v1cs.IR

TL;DR

Document retrieval must balance evidence quality against latency across heterogeneous documents, while static pipelines force one trade-off on every query. RetrievalRouter predicts the modality and architecture of the suitable pipeline from query text alone. It dominates static configurations across the accuracy–latency frontier, including 2.5% higher nDCG@5 and 12.4× faster than the strongest static baseline.

  • Problem

    Retrieval pipelines differ in modality and architecture, creating an accuracy–latency trade-off with no query-level basis for choosing among them.

  • Method

    RetrievalRouter is a lightweight query-aware router that predicts the retrieval pipeline for each query from query text alone and optimizes a reward combining nDCG@5 with normalized latency.

  • Results

    2.5% higher nDCG@5 and 12.4× faster than the strongest static baseline, while outperforming prior adaptive methods in accuracy-oriented settings and matching or numerically outperforming them in latency-oriented settings.

  • Takeaways & Limitations

    Query-level routing can reserve expensive pipelines for queries that genuinely require them instead of imposing one accuracy–latency choice on every query.

  • Takeaways & Limitations

    Query-only routing is underdetermined when the optimal pipeline depends on latent document layout rather than query intent.

Abstract

from arXiv · show

Document retrieval increasingly supports high-stakes information access in finance, healthcare, and law. Modern retrieval pipelines vary both in modality (text or multimodal) and in retrieval architecture (dense or late-interaction). These choices impose a hard compromise: the most effective pipelines are too slow and expensive to run at scale, while the fastest fail to retrieve evidence from complex documents. Practitioners must therefore choose between missed evidence and unusable latency, with no principled basis for adapting that choice at the query level. We show that this compromise is unnecessary. Not every query requires the same pipeline. Across benchmarks spanning financial and scientific corpora, no static pipeline dominates. We introduce RetrievalRouter, a lightweight query-aware router that learns, from the query text alone, which retrieval pipeline best fits each query. A single tunable parameter exposes the full accuracy-latency frontier, and for every static baseline, RetrievalRouter offers an operating point that is simultaneously more accurate and faster. Against the best static baseline, RetrievalRouter is 2.5% more accurate and 12.4 times faster. Furthermore, compared with prior adaptive strategy selection methods, RetrievalRouter achieves significantly higher nDCG@5 across accuracy-oriented settings, while matching or numerically outperforming them on both nDCG@5 and latency in latency-oriented settings. Our code and data are available at https://github.com/emrekuruu/retrieval-router.

1 Introduction

Document retrieval must balance evidence quality against latency across heterogeneous documents. RetrievalRouter addresses this query by query, selecting pipelines according to each query’s needs.

  • Motivation: Across 11 retrieval benchmarks, pipeline failures are query-dependent and asymmetric across modality and architecture.Text-based pipelines degrade on visually complex documents, while multimodal pipelines lag on nuanced textual understanding; dense retrieval is cheaper but late interaction remains necessary for some queries.
  • Motivation: Dense retrievers match late-interaction performance on most queries at a fraction of the cost, but fail on a subset requiring late interaction.
  • Approach: RetrievalRouter predicts both modality and architecture from query text alone, routing each query to a suitable retrieval pipeline.The system jointly selects across both axes for a single underlying corpus.
  • Results: 2.5% higher nDCG@5 and 12.4× faster than the strongest static baseline.RetrievalRouter dominates static configurations on the accuracy–latency frontier and exceeds prior adaptive methods in accuracy-oriented settings.
  • Resources: A query-level pipeline-selection benchmark releases per-query best-pipeline labels across more than 80,000 queries.

2 Related Work

Related work spans static retrieval pipelines and adaptive query-time selection. Prior adaptive methods vary retrieval depth, corpus or modality, or retriever choice, but RetrievalRouter’s setting routes across heterogeneous pipelines on one corpus.

  • Static retrieval: Static retrieval evolved from dense text retrieval toward layout-aware chunking and multimodal representations to address extraction and layout loss.Extraction errors can cause failures on layout-sensitive queries, while multimodal retrieval avoids text extraction but introduces limitations for long-range textual reasoning.
  • Adaptive retrieval: Adaptive retrieval methods condition decisions on query difficulty, uncertainty, learned policies, heterogeneous knowledge bases, or source modalities.
  • Single-corpus routing: Prior single-corpus approaches route among sparse and dense retrieval or experts within a fixed dense architecture.These systems differ from routing jointly across modality and architecture.

3 Background

The evaluated design space combines text or multimodal inputs with sparse, dense, late-interaction, and reranking strategies. These pipelines occupy different accuracy–latency positions because their representations and processing costs differ.

  • Design space: Seven pipelines span two modalities and four retrieval strategies, with five core pipelines defining the design space and two reranking variants representing deployment.
  • Text pipelines: BM25 is the lowest-latency pipeline, while dense text retrieval is slower because it requires embedding generation.Dense representations improve matching beyond exact term overlap.
  • Text pipelines: Late-interaction text retrieval is the most accurate core text pipeline but also the slowest.It preserves per-token embeddings for finer-grained matching.
  • Multimodal pipelines: Multimodal-dense retrieves from full-page images with single visual embeddings, whereas multimodal-late uses patch embeddings and patch-level interaction.Multimodal-dense is the fastest neural pipeline overall.
  • Reranking: Reranking uses a fast dense retriever to retrieve k=100 candidates before applying late interaction, recovering nearly all late-interaction accuracy at lower latency.Text-Rerank and Multimodal-Rerank represent practical deployable static baselines.

4 Proposed Approach

RetrievalRouter treats retrieval selection as a query-level decision, using query text to choose among pipeline arms. Its reward combines per-query nDCG@5 with normalized latency, exposing a tunable accuracy–latency frontier.

  • Proposed Approach: No single pipeline dominates, so a fixed choice overspends on easy queries or underperforms on difficult ones.
  • Router design: RetrievalRouter predicts the best-fitting pipeline from query text alone and uses one router over five direct pipeline arms.The direct action space is BM25, TD, TR, MD, and MR; TL and ML remain static baselines.
  • Problem formulation: The router dispatches each query to arg max over its predicted policy distribution and defines reward from nDCG@5 and normalized latency.The five routing arms form the action space.
  • Problem formulation: λ controls the accuracy–latency trade-off: λ=0 gives a quality-only objective, while λ→1 favors the fastest arm.Sweeping λ traces the accuracy–latency Pareto frontier.
  • Training: Soft targets from the full per-query reward vector avoid arbitrary hard-label choices when multiple pipelines tie.The router trains against reward-derived distributions rather than a single best-pipeline label.
  • Training: The router approximates the per-query oracle decision at inference without running the retrieval pipelines themselves.

5 Experimental Setup

The evaluation uses 11 datasets across financial, scientific, and open-domain corpora, comparing static, adaptive, and oracle retrieval strategies with effectiveness, latency, statistical, and storage measurements.

  • Datasets: 11 datasets span text-heavy scientific papers, mixed-modality financial reports, and chart-dense slide decks.
  • Baselines: Seven static pipelines run identically on every query to establish the accuracy–latency frontier without query-aware routing.
  • Baselines: The adaptive baseline uses a classifier and probability threshold to route queries from BM25 to neural pipelines, with λ = 1 − budget providing the comparison scale.
  • Baselines: A seven-pipeline per-query Oracle selects the pipeline maximizing Equation 1 reward, with latency normalized across the full evaluated set.
  • Metrics: Effectiveness uses nDCG@5, MRR@5, and Recall@5, while latency includes router inference, embedding generation, and vector search.
  • Analysis: Statistical comparisons use paired tests selected by normality, Holm correction, and significance threshold p < 0.001.

6 Evaluation

Across the evaluation, RetrievalRouter adapts pipeline choices to query-level needs and dominates static baselines across the accuracy–latency frontier. Its routing responds to visual complexity and selectively assigns cheaper or more expensive pipelines according to per-query reward.

  • Static baselines: No static pipeline combines late-interaction effectiveness with lightweight-retrieval latency: ML reaches 0.737 nDCG@5 at 8.283s, while BM25 reaches 0.510 nDCG at 0.019s.
  • RetrievalRouter performance: At λ = 0.1, RetrievalRouter reaches 0.755 nDCG@5 at 0.666s, improving 2.5% over ML while running 12.4× faster.
  • RetrievalRouter performance: At λ = 0.5, RetrievalRouter reaches 0.707 nDCG@5 at 0.314s and improves effectiveness over both dense baselines while remaining faster.
  • Efficiency: The 15 ms routing overhead raises BM25-only latency from 0.019s to 0.034s at λ = 1, still below every neural static pipeline.
  • Adaptive comparison: RetrievalRouter has significantly higher nDCG@5 than Arabzadeh et al. in accuracy-oriented settings, while at λ = 0.7 it numerically improves both effectiveness and latency without significant differences.
  • Routing distribution: As λ increases, RetrievalRouter shifts selections toward MD and BM25, adapting allocation to the desired accuracy–efficiency trade-off.
  • Routing distribution: At λ = 0, RetrievalRouter assigns 6.0% of queries to BM25, indicating that cheaper pipelines can be selected when they are the best fit.
  • Modality sensitivity: Under visual complexity, multimodal routing rises from 69.7% to 95.2% at λ = 0, while BM25 declines and MD increases at λ = 0.7.

7 Conclusions

RetrievalRouter frames accuracy–latency selection as a query-level routing problem, jointly choosing modality and architecture to avoid the compromises imposed by static pipelines.

  • RetrievalRouter jointly routes each query across modality and architecture, dispatching it to the cheapest pipeline that can answer it.
  • The work releases per-query best-pipeline labels across more than 80,000 queries alongside its systematic benchmark analysis.
  • 2.5% higher nDCG@5 than the strongest static baseline accompanies 12.4× faster retrieval.
  • 3.0% higher nDCG@5 than the deployment-standard rerank variant comes with 1.7× lower latency.
  • RetrievalRouter achieves significantly higher nDCG@5 than prior adaptive methods in accuracy-oriented settings and matches or numerically outperforms them in latency-oriented settings.

Limitations

The approach trades storage and VRAM for lower inference latency, while its query-only router is evaluated mainly in-domain and remains limited by document ambiguity and distribution shift.

  • Storage Cost: The multimodal-late index is approximately 39 GB, more than 13× the approximately 3 GB text-dense index for the combined corpus.
  • VRAM Cost: Maintaining all pipelines requires 40 GB of VRAM, making the approach best suited to latency-critical applications where hardware costs are secondary.
  • Cross-Domain Generalization: The router is evaluated with in-domain 80/10/10 splits, so zero-shot cross-domain generalization remains an open direction.
  • Query-Only Semantic Ambiguity: Query-only routing is underdetermined when the optimal pipeline depends on latent document layout rather than query intent.
  • Query-Only Semantic Ambiguity: The routing action space covers bi-encoder and late-interaction architectures but not expensive generative LLM-based rerankers or downstream generator sizes.

Ethical Considerations

The paper reports no major ethical concerns, uses publicly released resources under their original licenses, and introduces neither human-subject data nor human evaluation.

  • The authors foresee no major ethical concerns or potential risks in this work.
  • All retrieval models, encoders, and benchmark datasets are publicly released and used under their original licenses.
  • The study introduces no new human-subject data and conducts no human evaluation.
  • The training distribution is restricted to English-language financial, scientific, and open-domain documents, requiring representative in-domain evaluation outside these distributions.

A Experimental Details

Experiments span more than 80,000 queries across financial, scientific, and open-domain datasets, using held-out test splits, oracle pipeline measurements, and a lightweight query router.

  • Dataset Statistics: The benchmark suite covers over 80,000 queries across financial, scientific, and open domains.
  • Data Split: Each source dataset is split independently into 80% training, 10% validation, and 10% test data.
  • Oracle Generation: For each training query, all seven pipelines are executed once to record nDCG@5 and latency, while rewards and soft targets are recomputed per λ objective.
  • Hyperparameters: The decision head projects a 1024-dimensional query representation to five routing logits, with dropout applied to the pooled representation.
  • Inference Cost: The router adds approximately 15 ms per query on an NVIDIA H100, compared with roughly 400 ms for Text-Dense and 8 s for Multimodal-Late.

B Per-Dataset Analysis

At λ = 0, RetrievalRouter allocates pipelines according to dataset- and query-specific retrieval effectiveness rather than a fixed cost ordering, producing gains over static pipelines across multiple datasets.

  • Pipeline allocations: RetrievalRouter routes 60–90% of queries from six financial datasets to MR, whose static performance is strongest on all six.Arabzadeh et al. routes 84–100% of those queries to MD.
  • Pipeline allocations: On Wiki-SS, RetrievalRouter sends 94% of queries to text pipelines because they outperform multimodal pipelines.
  • Effectiveness: RetrievalRouter exceeds the strongest static pipeline on five datasets for nDCG@5 and MRR@5 and on four for Recall@5.
Loading 2608.25625v1…