Source-linked AI summary
Less can be More: Relieving RAG Bottlenecks via Evidence Frontloading and Pressure-Adaptive Budgeting
Weibin Cai, Reza Zafarani
TL;DR
RAG efficiency methods often focus on downstream generation even though the dominant bottleneck can shift between reranking and generation with serving load and reranking budget. PACE combines evidence frontloading with pressure-adaptive budgeting, and experiments show improved evidence recall and lower latency under ranking-heavy workloads, including higher final recall with fewer reranked documents.
Problem
Existing RAG efficiency methods mainly optimize downstream generation, leaving end-to-end coordination with upstream reranking unresolved when bottlenecks shift across loads and reranking budgets.
Method
PACE is a training-free framework that frontloads marginally complementary evidence and dynamically adjusts reranking budgets according to relative reranker and LLM pressure.
Results
PACE improves evidence recall and substantially reduces latency under ranking-heavy workloads across three multi-hop QA datasets and online serving simulations.
Takeaways & Limitations
Evidence-dense top-ranked candidates can achieve higher final recall with fewer reranked documents, making less reranking potentially more effective.
Takeaways & Limitations
The monotone-submodular guarantee assumes nonnegative query coverage, relevance weights, and document distances.
Abstract
from arXiv · showhide
Existing methods for improving Retrieval-Augmented Generation (RAG) efficiency mainly optimize downstream LLM generation, such as context compression or serving optimization. However, RAG is an end-to-end system, and its bottleneck can shift between upstream reranking and downstream generation under different serving loads and reranking budgets.In this paper, we first empirically characterize this shifting-bottleneck behavior and show that upstream reranking can become the dominant bottleneck under high query rates or large reranking budgets. Reducing the reranking budget can relieve this bottleneck, but it may also drop supporting evidence and degrade recall. To address this problem, we propose \textbf{\textsf{PACE}} (\textbf{P}rioritized \textbf{A}daptive \textbf{C}overage of \textbf{E}vidence), a training-free framework that combines \textit{evidence frontloading} with \textit{pressure-adaptive budgeting}. \textsf{PACE} first reorders candidates by marginal evidence coverage, prioritizing documents that are query-relevant, complementary, and useful for forming multi-hop evidence chains. We show that this objective is monotone submodular, giving greedy selection a $(1-1/e)$ approximation guarantee. \textsf{PACE} then dynamically adjusts the reranking budget according to the relative pressure of the reranker and the LLM. Experiments on three multi-hop QA datasets and online serving simulations show that \textsf{PACE} improves evidence recall, reduces p95 latency under ranking-heavy workloads. More importantly, the two components together reveal that \textit{less can be more}: an evidence-dense top-ranked candidates enable higher final recall with fewer reranked documents.
1 Introduction
RAG efficiency requires coordinating upstream retrieval and reranking with downstream generation because bottlenecks and evidence needs vary across configurations and serving loads. PACE addresses this by frontloading complementary evidence and adapting reranking budgets to system pressure.
- Motivation: RAG answer quality depends on whether the top-k context covers the evidence needed to answer the query.Increasing k can improve evidence recall, but longer contexts can lose useful information, add noise, and increase inference overhead.
- Motivation: Upstream omissions cannot be recovered by downstream context compression, while generator optimization may not improve latency when queries queue at the reranker.This motivates end-to-end coordination between retrieval/reranking and generation.
- Problem: The dominant bottleneck can shift between reranking and generation as query rates, reranking budgets, and serving loads change.High query rates or large reranking budgets can make upstream ranking dominant, whereas other configurations may remain generation-heavy.
- Approach: PACE combines marginal evidence frontloading with pressure-adaptive budgeting to reduce upstream reranking workload while preserving evidence recall.Its frontloading objective prioritizes query-relevant, complementary documents useful for forming multi-hop evidence chains, while budgeting responds to relative reranker and LLM pressure.
- Results: PACE's evidence-dense top-ranked candidates support higher final recall with fewer reranked documents under ranking-heavy workloads.The paper reports this result from experiments on three multi-hop QA datasets and online serving simulations.
2 RAG Bottlenecks Shift Across Configurations and Loads
RAG's serving bottleneck shifts between reranking and generation as model choices, query load, and reranking budget change. Queueing measurements show why downstream compression helps generation-heavy systems but offers limited end-to-end benefit when reranking dominates.
- Bottleneck characterization: RAG bottlenecks are identified by comparing queueing delays in the reranker and LLM stages.Longer reranker queueing indicates ranking-heavy serving; otherwise, generation is the bottleneck, with p95 queueing latency used to expose tail behavior.
- Load and configuration effects: Increasing QPS generally increases upstream reranking pressure across reranker–LLM model pairs.Relative model size affects bottleneck tendency, but high load can still make reranking dominant despite a much larger LLM.
- Load and configuration effects: With Qwen-3B and DeBERTa, D=100 becomes reranker-queue dominated at QPS 1, whereas D=50 remains LLM-queue dominated.The reranking budget therefore changes which pipeline stage limits end-to-end latency.
- Downstream optimization: Context compression reduces LLM queueing delay but does not reduce reranker queueing delay and may add compute that increases reranking pressure.Its end-to-end benefit is consequently limited under high-QPS, large-D ranking-heavy workloads.
3 PACE: Prioritized Adaptive Coverage of Evidence
PACE addresses shifting RAG bottlenecks by combining evidence frontloading with pressure-adaptive reranking budgets. Its evidence-coverage objective supports greedy selection with a (1-1/e) approximation guarantee, while budgeting reduces reranking work only when reranker pressure exceeds LLM pressure.
- Evidence Frontloading: PACE reorders candidates to frontload relevant, complementary, and multi-hop supporting evidence into the ranking prefix.The objective prioritizes marginal coverage of previously uncovered evidence dimensions, reducing the gain from redundant documents.
- Evidence Frontloading: Soft-anchor refinement raises weights for documents connected to highly query-relevant candidates, recovering weakly query-related bridge evidence.The refined relevance combines direct query-document relevance with anchor-based relevance and can be applied before or after reranking.
- Theoretical Property: The marginal evidence coverage objective is monotone submodular under nonnegative query, document, and relevance values.This property establishes diminishing returns as selected documents cover more evidence dimensions.
- Theoretical Property: Greedy evidence frontloading achieves at least a (1-1/e) approximation to optimal evidence coverage under a cardinality constraint.The greedy rule repeatedly adds the document with the largest marginal gain while remaining efficient enough for reranking-time use.
- Pressure-Adaptive Budgeting: Pressure-adaptive budgeting selects the largest affordable reranking budget, using Dmax when reranking is no more congested than generation and reducing it otherwise.The rule estimates reranker backlog-clearing time and LLM remaining service time, then decreases the budget only when reranker pressure exceeds LLM pressure.
4 Experiments
PACE is evaluated for evidence frontloading and pressure-adaptive serving across three multi-hop QA datasets and online simulations. It improves evidence recall with smaller budgets and reduces latency under ranking-heavy workloads.
- Experimental Settings: Experiments use three multi-hop QA datasets, fixed retriever, reranker, compressor, and generator models, plus online serving simulations on separate GPUs.Evaluation measures evidence recall, p95 end-to-end latency, and queueing times under Poisson arrivals from 0.5 to 2.5 QPS.
- Evidence Frontloading Improves Recall: PACE consistently improves complete and supporting evidence recall@D across datasets, especially at small reranking budgets.On HotpotQA, PACE at D=20 achieves recall comparable to the best baseline at D=40.
- Evidence Frontloading Improves Recall: Combining query and anchor relevance provides the strongest and most stable evidence frontloading across datasets.Removing relevance guidance or using only one relevance source weakens coverage or can promote irrelevant documents when anchors are noisy.
- PACE Improves Efficiency while Preserving Recall: PACE reduces reranker queueing and stabilizes end-to-end latency as QPS increases, unlike the fixed D=100 baseline, although LLM queueing can rise slightly.The adaptive budget relieves the dominant upstream bottleneck under ranking-heavy workloads.
- PACE Improves Efficiency while Preserving Recall: Under comparable adaptive budgets, PACE maintains higher complete and supporting evidence recall as QPS increases.The selection stage accounts for less than 0.007% of p95 end-to-end latency, while at QPS=1.8 most adaptive methods select about 50–60 documents.
- PACE Improves Efficiency while Preserving Recall: After reranking, PACE achieves the highest recall@5, surpassing fixed D=100 and remaining stable as QPS increases.At QPS=1.8, PACE uses nearly half the fixed budget yet achieves about 20% higher recall@5.
5 Related Work
Efficient RAG research has largely targeted downstream generation, while PACE addresses an upstream gap by adapting reranking input and budget to online pressure.
- Serving systems accelerate LLM inference through batching, scheduling, KV-cache management, and optimized prefill execution.
- Context compression methods shorten retrieved contexts to reduce downstream generation cost while preserving core information.
- Adaptive-K chooses a heuristic reranking budget, whereas PACE adapts reranking budgets to real-time reranker–LLM pressure.PACE also controls which candidates enter reranking while leaving the reranker unchanged.
6 Conclusion
The paper frames RAG as an end-to-end serving system whose bottleneck shifts between reranking and generation. PACE combines evidence-dense ranking with adaptive budgeting to improve recall and reduce ranking-heavy latency.
- RAG bottlenecks shift between reranking and generation as query arrival rates and reranking budgets change.
- PACE frontloads useful evidence using a monotone submodular marginal coverage objective, then selects reranking budgets from real-time system pressure.
- Experiments on multihop QA datasets and online serving simulations show improved evidence recall and substantially lower latency under ranking-heavy workloads.
- A smaller reranking budget can produce higher final evidence recall when top-ranked candidates are evidence-dense.