Source-linked AI summary
PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design
Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, Tim Kraska
TL;DR
Periodic retrieval from large databases can slow generation, motivating methods that balance RAG latency and quality. PipeRAG combines pipeline parallelism, flexible retrieval intervals, and performance-model-guided search; it reports up to 2.6× speedup over RETRO without compromising perplexity.
Problem
Method
PipeRAG co-designs the RAG algorithm and retrieval system using pipeline parallelism, flexible retrieval intervals, and performance-model-guided search-space adjustment.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) can enhance the generation quality of large language models (LLMs) by incorporating external token databases. However, retrievals from large databases can constitute a substantial portion of the overall generation time, particularly when retrievals are periodically performed to align the retrieved content with the latest states of generation. In this paper, we introduce PipeRAG, a novel algorithm-system co-design approach to reduce generation latency and enhance generation quality. PipeRAG integrates (1) pipeline parallelism to enable concurrent retrieval and generation processes, (2) flexible retrieval intervals to maximize the efficiency of pipeline parallelism, and (3) a performance model to automatically balance retrieval quality and latency based on the generation states and underlying hardware. Our evaluation shows that, by combining the three aforementioned methods, PipeRAG achieves up to 2.6$\times$ speedup in end-to-end generation latency while improving generation quality. These promising results showcase the effectiveness of co-designing algorithms with underlying systems, paving the way for the adoption of PipeRAG in future RAG systems.
1. Introduction
PipeRAG addresses the latency burden of periodic retrieval from large databases through algorithm-system co-design. It combines overlapping retrieval and generation, flexible retrieval intervals, and performance-model-guided search, with reported gains over RETRO.
1. Introduction
- Periodic retrieval keeps retrieved content relevant as generation context shifts, but retrieval from large databases can substantially slow generation.The paper asks whether RAG system performance can be optimized while preserving or improving generation quality.Source: §1 · p. 1
- PipeRAG targets hardware idleness caused by retrieval-inference dependencies, rising per-token inference latency, and the latency-quality trade-off in approximate nearest-neighbor search.These three performance observations motivate the pipeline, flexible intervals, and performance-model components.Source: §1 · p. 1, Figure 1 · p. 2
- PipeRAG prefetches to overlap retrieval and inference, supports flexible retrieval intervals, and uses a performance model to adjust the search space for upcoming token latency.The model dynamically adjusts the retrieval search space according to expected inference latency.Source: §1 · pp. 1–2
- 2.6× speedup in latency without compromising perplexity, or up to 0.93 lower perplexity than RETRO at the same latency, places PipeRAG ahead on the quality-performance Pareto frontier.The comparisons are reported across various evaluation datasets and databases of up to 200 billion tokens.Source: §1 · p. 2
2. Background and Motivation
Periodic retrieval can improve generation quality and keep retrieved content aligned with changing context. RETRO illustrates this approach, while frequent retrieval from large databases motivates balancing generation quality against latency.
2. Background and Motivation
- Periodic retrieval can improve generation quality and keep retrieved content aligned with contextual shifts; RETRO is a representative model in this category.RETRO integrates retrieval and inference, with an encoder incorporating retrieved tokens and a decoder generating tokens.Source: §2 · p. 2
- RETRO stores token chunks as vector keys and original and following chunks as values, then uses approximate nearest-neighbor search to retrieve close chunks and successors.Each database chunk contains m tokens.Source: §2 · p. 2
- RETRO partitions generated tokens into chunks and uses the preceding chunk to retrieve neighbors for generation of the next chunk.Retrievals occur at regular intervals during generation.Source: §2 · pp. 2–3
- Chunk-wise decoder-to-encoder and encoder-to-decoder attention integrates retrieved information while avoiding attention over all prior retrieval and generation states.Source: §2 · p. 3
- Frequent retrievals from large databases can consume considerable generation time, framing RAG efficiency as a trade-off between perplexity and latency.The section asks whether latency can be reduced at a given quality requirement, or quality improved at a given performance requirement.Source: §2 · p. 3
3. Our Approach: PipeRAG
PipeRAG co-designs the RAG algorithm and retrieval system to improve the performance-quality trade-off. It addresses hardware underutilization with pipeline parallelism and flexible retrieval intervals, then uses performance models to balance retrieval quality and latency.
3.1. Performance-Centric Observations in RAG
- RAG dependencies can leave retrieval or inference idle; inference cost per token rises with sequence length, while ANN search trades retrieval quality against latency.These performance characteristics motivate coordinating both subsystems rather than optimizing either in isolation.Source: §3.1 · pp. 3–4
3. Our Approach: PipeRAG
- PipeRAG combines pipeline parallelism, flexible retrieval intervals, and automatic search-space selection to improve RAG’s performance-quality Pareto frontier.The approach responds to hardware inefficiency and the differing performance characteristics of inference and retrieval systems.Source: §3 · p. 3, §3.2 · p. 4
3.2. Algorithm-System Co-deisgn in PipeRAG
- Pipeline parallelism relaxes dependencies by prefetching with a slightly older query window, then shifting retrieved tokens to approximate results for the latest context.The first retrieval instead uses the initial generation chunk; shifting excludes the first s retrieved tokens while preserving retrieval length.Source: §3.2 · p. 4
- PipeRAG varies retrieval interval m′ and adapts RETRO’s attention; shorter intervals such as 32 reduce query staleness relative to RETRO’s fixed interval of 64.The query window remains 64 tokens in the m′ = 32 example, while retrievals occur twice as often.Source: §3.2 · pp. 4–5
- A performance model uses generation state to select the largest retrieval search space whose latency fits within the next token chunk’s generation time.The retrieval system estimates its performance on deployment hardware and constrains retrieval latency to T(C).Source: §3.2 · p. 5
- PipeRAG assigns overlap construction to the retrieval system because concurrent queries can make retrieval latency difficult for an inference system to predict.The retrieval workload may vary when one retrieval system serves multiple inference systems.Source: §3.2 · p. 5
4. Evaluation
PipeRAG improves the latency–perplexity trade-off over RETRO, while its retrieval intervals and performance-model-driven search can be adjusted to generation and hardware conditions. Its pipeline parallelism also preserves an efficiency advantage over RETRO+ when both support flexible intervals.
4.1. Experimental Setup
- The evaluation uses a 3-billion-chunk C4 token database, a 582M-parameter RETRO checkpoint, and Wikipedia, RealNews, and C4 evaluation sets.Inference used an NVIDIA A100 GPU, while retrieval ran on a 48-core Intel Xeon server with 384 GB memory.Source: §4.1 · p. 6
4.2. Perplexity Evaluation
- Retrieval improves quality across datasets: no retrieval performs worst, one initial retrieval follows, and larger databases lower perplexity.With the largest database, PipeRAG also outperforms RETRO in generation quality.Source: §4.2 · p. 6
- Expanding the vector-search space reduces perplexity, and PipeRAG surpasses RETRO in quality at retrieval intervals of 32 or less.Shorter intervals reduce query staleness and increase content-integration frequency; pipeline parallelism means more frequent retrievals do not necessarily increase latency.Source: §4.2 · pp. 6–7
4.3. RAG Efficiency: Performance-Quality Trade-offs
- Up to 2.6× lower latency while maintaining or reducing perplexity, or up to 0.93 points lower perplexity at the same latency, versus RETRO.The comparison is across evaluated datasets and uses the latency–perplexity Pareto frontier.Source: Figure 6 · p. 7, §4.3 · p. 7
- Performance-model-driven retrieval reduces perplexity by 2.50∼4.82 with 10.6%∼13.2% latency overhead versus the no-retrieval baseline.The table caption describes the resulting latency as comparable to non-retrieval models while significantly reducing perplexity.Source: §4.3 · p. 7, Table 1 · p. 7
- PipeRAG remains more efficient than RETRO across modeled faster-inference and faster-retrieval hardware, but its advantage shrinks when either subsystem becomes much faster.The modeled inference scenarios include 4× and 16× speedups; pipeline parallelism is most effective when subsystem latencies are comparable.Source: §4.3 · pp. 7–8
4.4. Ablation Study
- PipeRAG matches RETRO+ in perplexity but achieves lower latency at the same perplexity by overlapping retrieval with generation.Both systems use retrieval intervals from 8 to 64; RETRO+ supports flexible intervals but lacks PipeRAG’s pipeline parallelism.Source: §4.4 · p. 8
5. Related Work
Related RAG work addresses generation quality through retrieval at different points in generation. PipeRAG differs by focusing on efficiency through algorithm-system co-design.
5. Related Work
- PipeRAG is presented as the first in-depth algorithm-system co-design effort aimed at RAG efficiency, whereas prior RAG research mainly targets generation quality.The paper positions this work as a departure from the dominant focus of existing RAG research.Source: §5 · p. 8
- Periodic retrieval can keep retrieved content relevant as generated text diverges from its initial context, supporting token-generation quality.Prior examples include RETRO, which integrates retrieved content at regular intervals.Source: §5 · p. 8
- Token-level retrieval methods such as kNN-LM retrieve similar passages and following tokens at each generation step, then interpolate retrieved and model token probabilities.The paper notes arguments that integrating longer passages may be more effective than token-level integration.Source: §5 · p. 8
6. Conclusion
Retrieval from large databases creates efficiency challenges for RAG. PipeRAG combines pipeline parallelism, flexible retrieval intervals, and performance-model-driven retrieval, with reported latency and generation-quality improvements.
6. Conclusion
- Retrieval overhead from large databases is a central efficiency challenge for retrieval-augmented generation.The conclusion frames this as an opportunity to optimize RAG efficiency.Source: §6 · p. 8
- PipeRAG combines pipeline parallelism, flexible retrieval intervals, and dynamic adjustment of retrieval quality through performance modeling.The approach is framed as a novel RAG method for improving generation efficiency.Source: §6 · p. 9
- PipeRAG achieves up to 2.6× speedup over RETRO without compromising generation quality.The authors describe algorithm-system co-design as a foundation for future RAG systems and research opportunities.Source: §6 · p. 9
Impact Statements
The impact statement expects minimal ethical concerns or adverse societal impacts because the work does not train new models. A motivating example illustrates why periodic retrieval may help keep context relevant as generated topics shift.
Impact Statements
- The work does not train new models, and the authors anticipate minimal ethical concerns or adverse societal impacts.The stated focus is reducing energy consumption and carbon emissions during large-scale LLM inference.Source: Impact Statements · p. 9
A. A Motivating Example of Periodic Retrievals
- In the example, a response about the Transformer shifts from its introduction to future research trends, making initial retrievals less relevant.The passage presents periodic retrievals as a way to keep retrieved content pertinent to the current generation context.Source: A. A Motivating Example of Periodic… · p. 12, Figure 9 · p. 12
B. Detailed Evaluation Setup
The evaluation setup specifies the database, inference and retrieval software, and hardware; the performance discussion identifies software, model, database, and hardware factors affecting retrieval and inference.
B. Detailed Evaluation Setup
- The evaluation uses a C4 token database, an optimized RETRO implementation, Faiss retrieval, gRPC communication, and separate inference and retrieval servers.The database contains three billion 64-token chunks encoded as 384-dimensional vectors; inference uses an A100 GPU, while retrieval runs on a 48-core CPU server with 384 GB memory.Source: B. Detailed Evaluation Setup · pp. 12–13
C. Performance Trends on Evolving Hardware
- Retrieval and inference hardware performance depends on memory bandwidth and computational capacity.The retrieval discussion also notes specialized retrieval accelerators; inference hardware includes evolving LLM accelerators.Source: §C.1 · p. 13
C.1. Factors Influencing Retrieval and Inference Performance
- Retrieval efficiency also depends on database vector count and representation, dimensionality and compression, index configuration, and optional LLM reranking.Vector count reflects document numbers and encoding granularity; reranking can improve ranking quality over vector similarity alone.Source: §C.1 · p. 13
- Inference software overhead can exceed GPU-kernel time, while lower-precision quantization and sparsity can reduce inference costs.The passage gives 3-bit versus 16-bit precision as an example of a 3–5× speedup and describes mixture-of-experts as activating only a small subset of neurons.Source: §C.1 · pp. 13–14
C.2. Performance Modeling for Future Hardware
The performance model projects PipeRAG efficiency on hypothetical hardware and is checked against experimental data. It accounts for pipeline overlap when estimating generation latency.
C.2. Performance Modeling for Future Hardware
- The model uses measured inference and retrieval performance to project PipeRAG efficiency on hypothetical hardware configurations.It considers configurations with enhanced inference performance, retrieval performance, or both.Source: §C.2 · p. 14
- With PipeRAG, each token chunk's latency is determined by the slower of inference and retrieval, except for the first chunk; RETRO adds the two times.The distinction reflects PipeRAG's retrieval-inference parallelism.Source: §C.2 · p. 14
- 5.7% median difference separates projected performance from actual experimental data, verifying the model within a reasonable range.The model simulates changes such as 4× retrieval improvement or 16× inference enhancement.Source: §C.2 · p. 14
D. Additional Experimental Results
The additional experiments examine stale-query prefetching and compare PipeRAG with a flexible-interval RETRO variant to illustrate pipeline parallelism's applicability and efficiency gains.
D. Additional Experimental Results
- The experiments compare prefetching with stale queries against retrieval using non-stale queries.The stated aim is to demonstrate the fundamental applicability of pipeline parallelism.Source: D. Additional Experimental Results · p. 14
- A separate comparison evaluates PipeRAG against RETRO+, a modified RETRO version that supports flexible retrieval intervals.The comparison is intended to highlight pipeline parallelism's efficiency gains.Source: D. Additional Experimental Results · p. 14
- The additional results are organized around prefetching effectiveness and a flexible-interval baseline comparison.The first examines stale-query content; the second isolates PipeRAG's comparison with RETRO+ as described in the passage.Source: D. Additional Experimental Results · p. 14
D.1. The Effectiveness of Retrievals using Stale Queries
Stale-query retrieval remains similar to non-stale retrieval, while PipeRAG generally compares favorably with RETRO+ at matched retrieval intervals, with a high-staleness qualification.
D.1. The Effectiveness of Retrievals using Stale Queries
- The stale-query experiment compares nearest-neighbor retrievals from stale and non-stale queries using the largest C4 database.It uses three billion token chunks, k = 1, nprobe = 64, and cosine similarity to assess retrieved content.Source: §D.1 · p. 14
- Stale queries yield around 90% cosine similarity to non-stale retrievals across datasets, with similarity gradually declining as staleness increases.The evaluation varies staleness from 1 to 64 tokens while keeping the retrieval interval at 64 tokens.Source: §D.1 · p. 14, Table 2 · p. 14
D.2. PipeRAG versus Baseline Model that Supports Flexible Retrieval Intervals
- For retrieval intervals 8–32, PipeRAG is more efficient than RETRO+; at interval 64, RETRO+ may outperform PipeRAG in low-perplexity scenarios.The comparison holds retrieval intervals, and thus database-request counts, equal; the interval-64 qualification is attributed to high query staleness.Source: §D.2 · p. 15, Figure 10 · p. 15
E. Broader Applicability of PipeRAG
PipeRAG’s pipeline-parallelism idea applies across RAG configurations with periodic retrievals, although this paper evaluates it using RETRO and specific hardware and software setups. The authors identify similar retrieval and inference times as a setting where prefetching is a strong efficiency option, and report stale-query similarity evidence beyond the evaluated checkpoint.
E. Broader Applicability of PipeRAG
- Pipeline parallelism can improve efficiency across RAG configurations that use periodic retrievals.The paper’s evaluation, however, focuses on RETRO and specified hardware and software setups.Source: E. Broader Applicability of PipeRAG · p. 15
- When one retrieval and multiple inference steps take similar time, prefetching database content is presented as a strong option for improving generation efficiency.Source: E. Broader Applicability of PipeRAG · p. 15
- Stale-query windows retrieve content very similar to regular query windows, supporting applicability beyond the RETRO checkpoint used in evaluation.The authors state that prefetching with stale queries is applicable regardless of the generation model.Source: E. Broader Applicability of PipeRAG · p. 15