PaperlayerView exact revision on arXiv

Source-linked AI summary

Less can be More: Relieving RAG Bottlenecks via Evidence Frontloading and Pressure-Adaptive Budgeting

Weibin Cai, Reza Zafarani

arXiv:2608.25115v1cs.CLcs.IR

TL;DR

RAG optimization must account for both reranking and generation, whose bottlenecks can shift with serving conditions. PACE frontloads evidence and adapts reranking budgets; experiments report improved evidence recall and reduced latency under ranking-heavy workloads.

  • Problem

    RAG optimization must coordinate upstream reranking and downstream generation because end-to-end effectiveness and latency span both stages.

    Source: §1 · p. 1

  • Method

    PACE is a training-free framework that frontloads marginal evidence coverage and adapts reranking budgets to relative reranker and LLM pressure.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Across three multi-hop QA datasets and online serving simulations, PACE improves evidence recall and substantially reduces latency under ranking-heavy workloads.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    A larger reranking budget does not always improve final top-K evidence recall, since additional candidates may introduce noise.

    Source: §4.3 · p. 8

    PACE's monotone-submodularity guarantee assumes nonnegative query weights, relevance weights, and document coverage values.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Existing methods for improving Retrieval-Augmented Generation (RAG) efficiency mainly optimize downstream LLM generation, such as context compression or serving optimization. However, RAG is an end-to-end system, and its bottleneck can shift between upstream reranking and downstream generation under different serving loads and reranking budgets.In this paper, we first empirically characterize this shifting-bottleneck behavior and show that upstream reranking can become the dominant bottleneck under high query rates or large reranking budgets. Reducing the reranking budget can relieve this bottleneck, but it may also drop supporting evidence and degrade recall. To address this problem, we propose \textbf{\textsf{PACE}} (\textbf{P}rioritized \textbf{A}daptive \textbf{C}overage of \textbf{E}vidence), a training-free framework that combines \textit{evidence frontloading} with \textit{pressure-adaptive budgeting}. \textsf{PACE} first reorders candidates by marginal evidence coverage, prioritizing documents that are query-relevant, complementary, and useful for forming multi-hop evidence chains. We show that this objective is monotone submodular, giving greedy selection a $(1-1/e)$ approximation guarantee. \textsf{PACE} then dynamically adjusts the reranking budget according to the relative pressure of the reranker and the LLM. Experiments on three multi-hop QA datasets and online serving simulations show that \textsf{PACE} improves evidence recall, reduces p95 latency under ranking-heavy workloads. More importantly, the two components together reveal that \textit{less can be more}: an evidence-dense top-ranked candidates enable higher final recall with fewer reranked documents.

1 Introduction

RAG bottlenecks can shift between reranking and generation as query rates and reranking budgets change. PACE combines evidence frontloading with pressure-adaptive budgeting to reduce upstream workload while preserving evidence recall.

1 Introduction

  • Experiments on three multi-hop QA datasets and online serving simulations show PACE improves evidence recall, reduces latency under ranking-heavy workloads, and achieves higher final recall with fewer reranked documents.The paper attributes this “less can be more” effect to evidence-dense top-ranked candidates.Source: Figure 1 · p. 1, §1 · p. 2
  • Prior approaches mainly accelerate generation or compress context, but cannot restore evidence missed upstream, and evidence-frontloading alternatives have limitations for multi-hop evidence.The introduction cites coarse-grained relevance modeling, missing inter-document dependencies, and limited robustness as limitations of diversity-aware methods.Source: §1 · pp. 1–2
  • Increasing query rates or reranking budgets can shift the dominant bottleneck from generation to upstream reranking, making downstream-only optimization insufficient for ranking-heavy workloads.A small fixed reranking budget may relieve ranking pressure but miss necessary evidence, and is difficult to choose under dynamic serving loads.Source: §1 · p. 2
  • PACE frontloads candidates by marginal evidence coverage, prioritizing direct query relevance, complementarity, and usefulness for forming complete multi-hop evidence chains.Its objective is to cover more query semantics with fewer documents; the paper states this coverage objective is monotone submodular.Source: §1 · p. 2
  • PACE dynamically adjusts the reranking budget according to relative reranker and LLM queueing pressure, reducing reranker workload when upstream ranking becomes the bottleneck.The stated goal is to choose the largest budget that does not make reranking a bottleneck relative to generation.Source: §1 · p. 2
  • Greedy evidence frontloading has a (1 −1/𝑒) approximation guarantee under a cardinality constraint because PACE’s evidence coverage objective is monotone submodular.The objective weights each document’s coverage by its direct query relevance and potential to bridge evidence chains.Source: §1 · p. 2

2 RAG Bottlenecks Shift Across Configurations and Loads

RAG’s bottleneck can shift between reranking and generation as model choices, reranking budget, and serving load change. The section measures that shift through queueing delays and finds that context compression cannot necessarily relieve reranking pressure.

2 RAG Bottlenecks Shift Across Configurations and Loads

  • With a 92.5 parameter ratio, Qwen-3B/MiniLM-L12 shifts from generation-heavy at low QPS to ranking-heavy at QPS 2.5; for Qwen-3B/DeBERTa, D=100 is reranker-dominated at QPS 1, unlike D=50.Model size affects bottleneck tendency, but load and reranking budget determine the actual bottleneck.Source: Figure 2 · p. 3, §2 · p. 3
  • The study identifies the bottleneck by comparing p95 reranker and LLM queueing delays under open-loop arrivals; longer reranker waits indicate ranking-heavy service.It focuses on reranking and generation, using queueing delays to identify which stage limits throughput.Source: §2 · pp. 2–3
  • Context compression reduces LLM queueing delay but not reranker queueing delay, so its end-to-end benefit is limited under high QPS and large D.The compressor also consumes compute, which can further increase reranking pressure and queueing time.Source: §2 · p. 3, Figure 2 · p. 3

P95 LLM queue time · 3 PACE: Prioritized Adaptive Coverage of Evidence

PACE addresses shifting RAG bottlenecks by combining evidence frontloading, which concentrates useful evidence in ranking prefixes, with pressure-adaptive reranking budgets. Its greedy coverage objective has a (1 - 1/e) approximation guarantee, and its budget policy adjusts reranking work to relative reranker and LLM pressure.

3 PACE: Prioritized Adaptive Coverage of Evidence

  • PACE combines evidence frontloading with a dynamically selected reranking budget to balance reranker and LLM pressure while preserving evidence recall.The framework targets the risk that large fixed budgets create a reranker bottleneck and small budgets miss necessary evidence.Source: §3 · p. 3, §3.2 · p. 5

3.1 Evidence Frontloading

  • Greedy frontloading prioritizes documents by marginal coverage of relevant, complementary evidence, so ranking prefixes cover more of the query’s information need.The marginal gain of redundant documents decreases as selected documents cover more evidence dimensions.Source: §3.1 · pp. 3–4
  • Soft anchors raise the relevance of documents connected to query-relevant evidence, allowing weakly query-matched bridge documents to complete multi-hop evidence chains.The refined relevance weight combines direct query-document relevance with anchor-based relevance and can be applied before or after reranking.Source: §3.1 · p. 4
  • Under nonnegative query, document, and refined-relevance values, the evidence-coverage objective is monotone submodular.Greedy selection achieves at least a (1 - 1/e) approximation to optimal evidence coverage under a cardinality constraint.Source: §3.1 · p. 4

3.2 Pressure-adaptive Budgeting

  • The budget policy uses the largest budget Dmax when reranker pressure is no greater than LLM pressure, and reduces the budget when reranker pressure is higher.The reduction is made in full reranker batches, and the policy retains the largest affordable budget.Source: §3.2 · p. 5
  • Together, frontloading and adaptive budgeting reduce upstream reranker workload under ranking-heavy conditions while concentrating useful evidence in the processed prefix.This pairing is intended to retain high evidence recall as the system adapts its reranking budget to changing serving pressure.Source: §3.2 · p. 5

4 Experiments

PACE frontloads evidence to improve recall among top-ranked candidates, then adapts the reranking budget to serving pressure to reduce latency while preserving evidence recall. Its evidence-dense rankings can yield higher final recall with fewer reranked documents.

4.1 Experimental Settings

  • Experiments use SPLADE-v3 retrieval, a DeBERTa-v3 cross-encoder reranker, Provence compression, and Qwen2.5-3B-Instruct generation on three multi-hop QA datasets.The dev evaluations use HotpotQA (1,087 queries), MuSiQue (2,317), and 2WikiMultiHopQA (2,861); evidence recall is measured at varying D and online serving uses Poisson arrivals from 0.5 to 2.5 QPS.Source: §4.1 · pp. 5–6

4.2 Evidence Frontloading Improves Recall

  • Across HotpotQA, MuSiQue, and 2WikiMultiHopQA, PACE improves complete and supporting evidence recall over training-free baselines, especially at small D; on HotpotQA, D=20 matches the best baseline at D=40.At comparable adaptive budgets, PACE maintains higher pre-reranking complete and supporting evidence recall than baselines and ablations, while adaptive methods' recall@D falls as QPS rises.Source: §4.2 · p. 6, Figure 3 · p. 7, Figure 4 · p. 7
  • Combining query and anchor relevance gives the strongest, most stable evidence frontloading; either alone is generally suboptimal, and removing both usually performs worst.Query relevance can miss weakly query-related bridge evidence, while anchor relevance can promote irrelevant documents when anchors are noisy; anchor relevance is relatively more effective on closed-context MuSiQue.Source: §4.2 · pp. 6–7, Figure 5 · p. 7

4.3 PACE Improves Efficiency while Preserving Recall

  • PACE dynamically lowers reranking budgets under pressure, substantially reducing reranker queueing and p95 end-to-end latency despite a slight possible increase in LLM queue time.Under ranking-heavy workloads, fixed D=100 accumulates reranker delay as QPS rises; the upstream bottleneck is relieved by PACE's smaller adaptive budget.Source: §4.3 · pp. 7–8, Figure 6 · p. 8
  • At QPS=1.8, PACE uses nearly half the fixed D=100 budget yet achieves about 20% higher recall@5; it also beats the full-budget baseline at K=5 across fixed-budget tests.After reranking, PACE has the highest recall@5 and remains stable as QPS increases; Table 2 evaluates complete evidence recall under fixed budgets.Source: §4.3 · pp. 8–9, Table 2 · p. 9

5 Related Work

Prior work improves RAG efficiency through generation-side optimization or reranker acceleration, while evidence-recall methods acquire or diversify evidence; PACE instead adapts candidate selection and reranking budget to online pressure while preserving evidence complementarity.

Relieving generation-heavy bottlenecks

  • Serving optimizations and context compression reduce LLM generation cost, but do not address shifting reranker–generation bottlenecks or reranker pressure while preserving evidence recall.Serving systems optimize batching, scheduling, KV-cache management, and prefill [1] [15] [41]; METIS [25] adapts query configurations; RECOMP [37], LLMLingua [11], and Provence [6] compress contexts.Source: §5 · p. 9

Relieving ranking-heavy bottlenecks

  • Early-exit rerankers and Adaptive-K reduce computation or use a heuristic budget, whereas PACE leaves the reranker unchanged and adapts candidate selection and reranking volume to online pressure.PACE controls which candidates enter reranking and how many are processed under serving pressure.Source: §5 · p. 9

Improving evidence recall

  • Multi-hop retrievers MDR [36], Baleen, and IRCoT [33] acquire evidence iteratively or through reasoning-guided search, but require specialized training or extra LLM-retrieval iterations that can increase serving cost.Source: §5 · p. 9
  • Unlike diversity methods MMR, xQuAD, and Dartboard [22], PACE models fine-grained complementarity to preserve similar but jointly necessary evidence, using a monotone submodular objective with a (1-1/e) guarantee.The cited diversity methods select non-redundant documents using coarse distances or manually designed trade-offs; PACE requires no additional training or tuned parameters.Source: §5 · p. 9

6 Conclusion

PACE addresses RAG’s shifting bottleneck by frontloading useful evidence and adapting the reranking budget to real-time system pressure. Across multihop QA datasets and online serving simulations, it improves evidence recall and reduces latency under ranking-heavy workloads.

6 Conclusion

  • PACE improves evidence recall and substantially reduces latency under ranking-heavy workloads, while evidence-dense rankings let smaller reranking budgets yield higher final recall.These findings come from experiments on multihop QA datasets and online serving simulations.Source: §6 · p. 9
  • RAG’s dominant bottleneck can shift between reranking and generation as query arrival rates and reranking budgets change.The paper studies RAG as an end-to-end serving system rather than treating LLM generation as its fixed bottleneck.Source: §6 · p. 9
  • PACE is a training-free framework that ranks useful evidence first using monotone submodular marginal coverage, then adapts reranking budgets to real-time reranker and LLM pressure.This approach aims to relieve upstream reranking bottlenecks while preserving evidence recall.Source: §6 · p. 9

Related papers

Loading 2608.25115v1…