PaperlayerView exact revision on arXiv

Source-linked AI summary

RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems

Zlatan Feric, Amir Taherin, Bin Ren, Yanzhi Wang, Jennifer Dy, David Kaeli

arXiv:2609.05760v1cs.PFcs.AIcs.ARcs.IR

TL;DR

RAG systems combine heterogeneous stages and a large configuration space, making their system-level trade-offs difficult to characterize. RAGMark benchmarks configurable RAG pipelines with per-stage profiling, and its analyses find that reranking and compression together compound benefits, reducing energy by up to 66% in the reported case. The artifact evaluation reproduces experiments on V100 GPUs, while the paper’s measurements used A100 GPUs, so absolute measurements differ.

  • Problem

    RAG combines heterogeneous subsystems with interacting stages, query-dependent behavior, and a large configuration space that is expensive to evaluate exhaustively.

    Source: §1 · p. 2

  • Method

    RAGMark provides modular per-query, per-stage RAG measurement and analysis, while leveraging existing infrastructure for preprocessing, indexing, and QA evaluation.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Reranking and compression compound benefits, with reranking shrinking compressor input and the combined stages reducing energy by 66% on L3-8B at topk=10.

    Source: §4.5 · p. 9

  • Takeaways & Limitations

    Reranking reduces the compressor’s input, compression densifies retained context, and the generator receives context reduced by both stages.

    Source: §4.5 · p. 9

    Artifact reproduction used 2× NVIDIA V100-PCIE-16GB GPUs rather than the paper’s 2× NVIDIA A100-SXM4-40GB GPUs, so absolute latency, energy, and memory measurements differ.

    Source: §7.3.2 · p. 14

Abstract

from arXiv · show

We present RAGMark, a modular benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems targeting small-scale multi-GPU environments. RAGMark evaluates diverse RAG components, including retrievers, vector databases, prompt-processing methods, and generator models, while collecting detailed per-stage metrics such as latency, GPU utilization, memory consumption, power usage, time to first token (TTFT), throughput, and answer quality. The framework is highly extensible, separating RAG stages, timing, and resource monitoring into modular components, and is designed to efficiently sweep large configuration spaces while minimizing repeated model and database initialization overhead. Using RAGMark, we characterize five RAG workloads on open-domain QA datasets across varying retrieval depths, model scales, reranking, compression methods, and vector database configurations. We show that while autoregressive generation dominates latency in naive pipelines, context-reduction techniques shift bottlenecks across compute, memory bandwidth, and preprocessing stages. Reranking and compression produce compounding benefits: reranking reduces compression workload itself, while both jointly reduce prefill and KV-cache traversal costs, lowering energy consumption by up to 66%. We further observe strong cross-stage interactions, where small upstream context reductions cascade through downstream latency, memory traffic, and energy consumption. The RAGMark source code is publicly available at: https://github.com/zferic/RAGMark.

1. Introduction

RAG supplements language models with retrieved external passages to address hallucinations, outdated knowledge, and limited domain-specific access, but its heterogeneous stages and configuration choices complicate performance characterization. RAGMark addresses the lack of systematic, stage-level benchmarking with an extensible framework for exploring RAG configurations and measuring cross-stage effects.

1. Introduction

  • RAG adds an external retrieval component that supplies relevant passages to the prompt, addressing standalone LLMs’ hallucinations, outdated knowledge, and restricted domain-specific access.In local deployments, understanding computational demands matters where data sensitivity and resource constraints are concerns.Source: §1 · p. 1
  • Existing studies often isolate LLM inference or treat RAG as a monolithic workload, leaving full-configuration, stage-level trade-offs and interactions under-characterized.RAG combines heterogeneous subsystems, query-dependent behavior, interacting stages, and a large configuration space that is expensive to explore exhaustively.Source: §1 · p. 2
  • RAGMark reduces repeated model and database initialization across configuration sweeps and supports query sampling and detailed query-level temporal traces for profiling.The framework reuses loaded models and vector databases and offers sampling strategies intended to preserve representative workload behavior.Source: §1 · p. 2
  • RAGMark provides modular, extensible end-to-end RAG characterization across configurable components and measures per-query, per-stage performance, resource use, and answer quality.Its configuration-driven sweeps also examine how retrieval depth, reranking, compression, model scale, and index placement interact and shift bottlenecks.Source: §1 · p. 2

2. Background

RAG retrieves external passages to augment a generator’s prompt, while its heterogeneous stages create configuration-dependent compute, memory, and latency demands. Index placement and structure, retrieval depth, and added processing stages all shape the trade-offs between retrieval cost, context size, and generation.

2.1. Retrieval-Augmented Generation

  • RAG retrieves relevant external passages at inference time and incorporates them into the generator prompt rather than relying only on knowledge encoded in model parameters.The paper describes this as addressing hallucinations, outdated world knowledge, and restricted access to domain-specific information.Source: §2.1 · pp. 2–3

2.2. RAG Steps and Computational Workload

  • RAG combines data-intensive retrieval with compute-intensive intermediate and generation stages, and the dominant bottleneck depends on configuration.Retrieval depth, chunk size, embedding model size, generator scale, and index placement influence whether retrieval or generation dominates.Source: §2.2 · p. 3
  • GPU-resident indexes can make retrieval fast when they fit, whereas CPU residency can increase latency substantially; index memory scales with corpus size, embedding dimension, and precision.The text estimates roughly 14 GB for 9.2 million chunks with 768-dimensional FP16 embeddings; larger dimensions increase that footprint.Source: §2.2 · p. 3
  • Generation comprises compute-intensive prefill over the full input and memory-bandwidth-bound autoregressive decode using the KV cache, with requirements scaling with model size.Its computational complexity also scales quadratically with input length.Source: §2.2 · p. 3

2.3. Vector Databases and Indexing

  • Index type and precision determine trade-offs among memory occupancy, search speed, and recall, and the paper uses Flat indexes while also examining retrieval-oriented indexing methods.Index construction encodes corpus chunks into vectors before retrieval.Source: §2.3 · p. 3
  • Flat indexes provide exact search and full recall at linear-scan and full-memory cost, while IVF and IVF-SQ trade some recall for search speed and memory efficiency.Scalar quantization reduces embedding memory footprint with minimal recall loss compared with product quantization; approximate structures may retrieve suboptimal chunks and degrade downstream generation quality.Source: §2.3 · p. 3

2.4. Advanced RAG Pipelines

  • Advanced RAG adds pre-retrieval methods such as query rewriting and post-retrieval methods such as reranking and prompt compression, which add inference or encoding overhead.Compression can reduce generator input context and cost, but its impact varies with method, model size, retrieved-token count, and compression rate.Source: §2.4 · p. 4
  • Iterative RAG makes multiple retrieve-generate passes for complex multi-hop questions, while conditional RAG uses a separate model to decide whether retrieval is needed.Both make inference-call counts workload-dependent, complicating bottleneck characterization without representative benchmarking.Source: §2.4 · p. 4

3. Benchmark Implementation and Measurement Methodology

RAGMark combines modular pipeline execution with per-query profiling of system performance and answer quality, and configuration-driven sweeps explore component, device, and workload choices. Its measurements and experimental setup support comparisons across RAG configurations while reducing repeated initialization and using consistent query samples.

3.1. Design Principles

  • The framework’s design principles combine per-query coverage of RAG stages, plug-and-play components, joint performance and accuracy evaluation, query sampling, and deterministic configuration-driven experiments.It uses standard Hugging Face interfaces rather than specialized serving backends to characterize end-to-end behavior; serving-backend support is under development.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2. Framework Implementation

  • Core classes instrument TTFT and token decode latency, normalize compression and reranking controls, and provide standard and iterative pipeline variants.The framework also offers individual-query and batch execution modes and configurable CPU/GPU placement, including distribution of indexes and generation models across devices.Source: §3.2 · pp. 4–5
  • RAGMark’s independent performance pipeline supports per-query, per-stage measurement, stage-interleaved execution, energy integration, memory attribution, unified component interfaces, and configuration-driven sweeps.It uses FlashRAG for dataset preprocessing, FAISS index creation, and QA evaluation, but not for performance profiling because its bulk query processing makes per-query profiling impractical.Source: §3.2 · p. 5

3.3. Measurement Methodology

  • The framework’s measurement categories include per-query metrics, and its configuration space is presented as directly specifiable parameters that can be extended with additional components.The measurement methodology details latency, memory and resource utilization, energy, and answer-quality measures.Source: §3.3 · p. 5, §3.4 · p. 6, Table 2 · p. 6
  • RAGMark measures stage latency, decomposes generation into TTFT and decode time, tracks device and framework memory plus CPU utilization, estimates per-query energy, and evaluates answer quality.Quality metrics include Exact Match, ROUGE, F1, and BERTScore; the reference-based metrics do not require additional LLM inference.Source: §3.3.1 · p. 5, §3.3.2 · p. 5, §3.3.3 · p. 5

3.4. Configuration Space

  • Configuration parameters cover component and index choices, retrieval and reranking depths, compression rates, model configurations, hardware placement, and pipeline settings.To make sweeps practical, RAGMark samples 400 QA pairs per dataset, holds the sample consistent across configurations, and minimizes repeated model and database initialization.Source: §3.4 · p. 6, §3.3.4 · p. 6

3.5. Experimental Setup

  • Experiments cover five primary cases with components on GPU0 and the database sharded across GPU0 and GPU1, using five-query warm-up and averages across multiple runs.The setup is intended to isolate pipeline-configuration effects under a consistent deployment environment.Source: §3.5.1 · p. 6

3.5.1. Configuration Approach.

  • The retrieval corpus is a 2018 English Wikipedia dump chunked to about 9.2 million passages, which are encoded with e5-small-v2, e5-base-v2, or e5-large-v2.The evaluated QA benchmarks include NQ, TriviaQA, SQuAD, WebQuestions, PopQA, and the multi-hop HotpotQA benchmark.Source: §3.5.2 · p. 6, §3.5.3 · p. 6

4. Results

Across the naive pipeline configurations, generation dominates latency, while increasing retrieval depth changes prefill and decode costs and shifts the system’s bottlenecks. The experiments use a two-GPU A100 node and report that retrieval improves ROUGE scores, which stabilize beyond topk=3.

4.1. Platform Setup

  • Experiments were conducted on a multi-GPU node with two NVIDIA A100-SXM4-40GB GPUs (40 GB VRAM each), using NVIDIA driver 545.23.08 and CUDA 12.3.Source: §4.1 · p. 6

4.2. Case 1: Naive Pipeline Analysis

  • LLM TTFT and decode dominate end-to-end latency across models and retrieval depths, while retrieval and embedding contribute under 35 ms even at topk=10.FAISS latency grows from ∼7 ms at topk=1 to ∼27 ms at topk=5, then increases marginally; embedding stays near 7 ms.Source: §4.2 · p. 7
  • For L3-8B, TTFT rises from 64 ms at topk=0 to 322 ms at topk=10, while energy grows 1.8× from topk=5 to topk=10 despite SM utilization plateauing.The authors attribute the larger-context behavior to quadratic attention scaling and increasingly bandwidth-bound decode.Source: §4.2 · p. 7
  • ROUGE scores improve with retrieval depth and stabilize beyond topk=3.Source: §4.2 · p. 7

4.3. Case 2: Reranking Analysis

Reranking adds retrieval-side work but reduces the context passed to generation, yielding lower latency and energy use with little change in ROUGE. The benefit is largest for the L3-8B model at topk=10.

4.3. Case 2: Reranking Analysis

  • At topk=10 on L3-8B, reranking cuts TTFT from 322 ms to 165 ms (49%) and energy from 157.5 J to 75.5 J (52%).Figure 3 compares resource-utilization trends across four configurations; Table 5 covers resource usage across four configurations.Source: §4.3 · p. 7, Figure 3 · p. 8, Table 5 · p. 8
  • At topk=10 on L3-8B, peak GPU0 memory is 24.9 GB with reranking versus 25.5 GB in the naive pipeline.The reranker adds approximately 200 MB, while reduced context lowers context-dependent memory use; the footprint remains largely dominated by model weights.Source: §4.3 · pp. 7–8
  • ROUGE scores differ by less than 0.02 from the naive pipeline across all configurations.Source: §4.3 · p. 8
  • Reranking’s overhead scales linearly with retrieval depth, reaching ∼24 ms at topk=10, while the reduced context lowers prefill time and KV-cache traversal costs.For L3-8B at topk=10, decode falls from 204 ms to 128 ms.Source: §4.3 · pp. 7–8

4.4. Case 3: Compression Analysis

Compression reduces context both across and within retained passages, trading preprocessing overhead for generation-side savings. On L3-8B at topk=10, it lowers TTFT by 66%, while its compute cost can erase gains at smaller models or retrieval depths.

4.4. Case 3: Compression Analysis

  • At topk=10 on L3-8B, compression reduces TTFT from 322 ms to 108 ms (66%) and decode from 204 ms to 111 ms.The authors attribute the larger gains versus reranking to compression shrinking context both across and within retained passages.Source: §4.4 · pp. 8–9
  • Compression overhead grows from ∼22 ms at topk=1 to ∼58–64 ms at topk=10, and Selective-Context latency outweighs its gains in this pipeline.Subsequent analysis therefore uses LLMLingua-2.Source: §4.4 · pp. 8–9
  • For L3-8B at topk=10, energy falls from 157.5 J to 85.7 J (46%), while GPU0 SM utilization drops from 72.4% to 59.5%.Compression adds 58 ms of overhead on the shared GPU, and at smaller model sizes and retrieval depths its compute cost can erase generation savings.Source: §4.4 · p. 9
  • At topk=10 on L3-8B, memory-bandwidth utilization drops from 35.5% to 25.3%, while peak GPU0 footprint remains 25.5 GB.The compressor model occupies VRAM alongside the generator, offsetting the KV-cache reduction.Source: §4.4 · p. 9

4.5. Case 4: Composition Effects: Reranking + Compression

Reranking and compression jointly reduce latency and energy, while iterative RAG repeats retrieval and embedding and changes generation costs across passes.

4.5. Case 4: Composition Effects: Reranking + Compression

  • 52.9 J for L3-8B at topk=10 is 66% below the naive pipeline’s 157.5 J, the lowest energy among the compared configurations.The combined pipeline also lowers TTFT from 179 ms to 108 ms and decode latency from 131 ms to 111 ms versus compression-only.Source: §4.5 · p. 9
  • Reranking shrinks the compressor’s input, roughly halving compression cost from 130 ms to 58 ms, so the combined benefit compounds rather than adding stage costs.Compression then further reduces the context passed to generation.Source: §4.5 · p. 9

4.6. Case 5: Iterative Pipeline Analysis

  • In iterative RAG, retrieval and embedding run twice; for L3-8B at topk=10, decode falls from 867 ms in iteration 0 to 102 ms in iteration 1, while TTFT shifts from 291 ms to 281 ms.The first prompt requests an expanded answer, whereas the second asks for a short factual response.Source: §4.6 · p. 10
  • Iterative RAG amplifies retrieval and generation costs and introduces temporal dependencies absent from single-pass pipelines.The shorter second-pass decode accompanies a different prompt structure.Source: §4.6 · p. 10

4.7. Index Methods

Index latency depends on hardware, batch size, and memory footprint: CPUs suit lightweight or memory-constrained retrieval, while GPUs scale better with concurrency.

4.7. Index Methods

  • At batch=1, CPU IVF1000,SQ8 latency is ∼1.1 ms versus ∼3.9 ms on a single GPU, showing the CPU advantage for lightweight queries.CPU latency is lower across nearly all low-batch configurations, especially for quantized IVF1000 indexes.Source: §4.7 · p. 10, Figure 8 · p. 10
  • At batch=12, CPU IVF300,SQ4 latency rises to ∼56 ms, while the single-GPU equivalent remains near 34 ms; GPUs generally scale better for batched retrieval.A second GPU does not always help because of synchronization overhead.Source: §4.7 · p. 10, Figure 8 · p. 10
  • Compressed IVF indexes require only a few GBs on Wikipedia 2018 with E5-small, while CPUs remain attractive for large databases or when inference uses most VRAM.GPUs become preferable as retrieval concurrency and workload size increase.Source: §4.7 · p. 10

4.8. Evaluating Trade-offs

Retrieval depth and model size trade off answer quality against TTFT, and increasing k does not consistently improve ROUGE.

4.8. Evaluating Trade-offs

  • The 1B model at k=3 or k=5 achieves ∼0.33 ROUGE at ∼0.04s TTFT, while the 8B model at k=8 reaches ∼0.41 ROUGE with ∼5-8x higher latency.Increasing k from 3 to 10 does not consistently improve ROUGE but noticeably increases TTFT, especially for larger models.Source: §4.8 · p. 10

5. Related Work

Prior work addresses RAG execution optimization and accuracy benchmarking, but the cited frameworks leave gaps in reusable, fine-grained system profiling.

5. Related Work

  • PipeRAG and Teola pipeline RAG stages, while other systems study caching and resource-aware scheduling; these efforts emphasize optimization over reusable benchmarking infrastructure.The cited work includes PipeRAG [39], Teola [50], caching [12],, and resource-aware frameworks, [52].Source: §5 · pp. 10–11
  • FlashRAG and BERGEN modularize RAG accuracy evaluation but do not focus on per-stage metrics.Other analyses largely isolate LLM inference rather than modeling RAG’s pre- and post-retrieval stages.Source: §5 · p. 11
  • RAGPerf [58] reports system-level metrics but lacks fine-grained RAG analysis of energy, latency breakdowns, intermediate-stage interactions, compression rates, pipeline configurations, and millisecond-scale queries.RAGMark addresses this gap with profiling across retrieval, compression, generation, and system execution.Source: §5 · p. 11

6. Conclusion

RAGMark characterizes RAG pipeline behavior on GPU systems by decomposing execution into per-stage measures across five canonical pipeline configurations. The framework is publicly released to support reproducible, fine-grained systems analysis.

6. Conclusion

  • The framework characterizes the architectural behavior of RAG pipelines on GPU systems.Source: §6 · p. 11
  • RAGMark decomposes end-to-end RAG execution into per-stage latency, energy, compute utilization, memory bandwidth, and memory footprint measurements.It covers naive, reranking, compression, combined, and iterative pipeline configurations.Source: §6 · p. 11
  • RAGMark is publicly released to support reproducible, fine-grained systems analysis and future exploration of pipeline designs.Source: §6 · p. 11

7. Artifact Appendix

The appendix describes RAGMark’s artifact, its hardware and software requirements, and how to reproduce three experiments. Reproduction uses a remote V100 system rather than the A100 hardware used for the paper results, so absolute measurements differ.

7.1. Abstract

  • The submitted code artifact implements the RAGMark hardware benchmarking framework used to generate the paper’s experimental results.Source: §7.1 · p. 14

7.2. Artifact check-list (meta-information)

  • The artifact supports dense-retrieval RAG configurations and reports quality and hardware metrics; its three experiments take about 1 hour and require about 65 GB of disk space.The disk estimate excludes the approximately 10 GB Docker image.Source: §7.2 · p. 14

7.3.1. How to access.

  • Artifact evaluation uses 2× V100-PCIE-16GB GPUs instead of the paper’s 2× A100-SXM4-40GB, so absolute latency, energy, and memory measurements differ.The experiments remain reproducible on equivalent hardware and demonstrate portability across hardware platforms.Source: §7.3.2 · p. 14

7.3.2. Hardware dependencies.

  • The provided execution environment includes the required software dependencies, while local installation uses dependencies listed in requirements.txt and setup steps in the README.Source: §7.3.3 · p. 14

7.3.3. Software dependencies.

  • The retrieval corpus is the 2018 Wikipedia dump of about 9.2 million passages, and evaluation uses NQ, TriviaQA, SQuAD, WebQuestions, PopQA, and HotpotQA.Source: §7.3.4 · p. 14

7.4. Installation

  • Reviewers can access a remote execution environment over SSH and run a single script; the appendix also gives steps for installing on a new system.The prepared remote environment requires no personal registration.Source: §7.4.1 · p. 14, §7.4.2 · p. 14

7.4.2. From Scratch.

  • A from-scratch setup clones RAGMark, creates a Python virtual environment, installs dependencies, configures data paths, and places a FAISS index and corpus files.Source: §7.4.2 · p. 14

7.5. Experiment workflow

  • The evaluation script runs the experiments and analysis notebook, producing figures for naive-pipeline trade-offs, reranking, and compression.The reproduced figures and aggregated data are saved under the artifact-evaluation output directories.Source: §7.5 · pp. 14–15, §7.6 · p. 15
Loading 2609.05760v1…