Source-linked AI summary
Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation
Zhisheng Qi, Utkarsh Sahu, Li Ma, Haoyu Han, Ryan Rossi, Franck Dernoncourt, Mahantesh Halappanavar, Nesreen Ahmed, Yushun Dong, Yue Zhao, Yu Zhang, Yu Wang
TL;DR
RAG systems create an additional channel for extracting sensitive knowledge, while prior studies lack comparable experimental settings. This paper introduces a unified, reproducible benchmark spanning attacks, defenses, models, knowledge-base configurations, and evaluation stages. The benchmark finds that effective extraction requires both retrieval and generation optimization, while defenses have complementary strengths and no single defense provides complete protection.
Problem
Prior knowledge-extraction studies use heterogeneous datasets, models, knowledge-base configurations, attacker assumptions, and metrics, hindering unified and fair evaluation of RAG security.
Method
The paper builds a comprehensive benchmark with a unified design space, standardized protocols, multiple attack and defense strategies, and stage-specific extraction evaluation.
Results
Effective extraction requires optimization at both retrieval and generation stages, while Query Block, Thresholding, Summary, and System Block show complementary defense strengths.
Takeaways & Limitations
No single defense provides complete protection, so RAG extraction security must be analyzed across pipeline stages and attack–defense configurations.
Takeaways & Limitations
The benchmark assumes an attacker can iteratively submit queries over multiple rounds to probe the knowledge base.
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) has become a cornerstone of knowledge-intensive applications, including enterprise chatbots, healthcare assistants, and agentic memory management. However, recent studies show that knowledge-extraction attacks can recover sensitive knowledge-base content through maliciously crafted queries, raising serious intellectual property and privacy concerns. While prior work has explored individual attack and defense techniques, the research landscape remains fragmented, spanning heterogeneous retrieval embeddings, diverse generation models, and evaluations based on non-standardized metrics and inconsistent datasets. To address this gap, we introduce the first systematic benchmark for knowledge-extraction attacks on RAG systems. Our benchmark covers broad attack/defense strategies, representative retrieval embedding models, open/closed-source generators, (non) graph-based indexing, all evaluated under a unified experimental framework with standardized protocols across multiple datasets spanning diverse languages. By consolidating the experimental landscape and enabling reproducible, comparable evaluation, this benchmark provides actionable insights and a practical foundation for developing privacy-preserving RAG systems in the face of emerging knowledge extraction threats.
1 Introduction
RAG systems support knowledge-intensive applications but expose an additional channel for extracting sensitive knowledge from external knowledge bases. Prior attack and defense studies use inconsistent settings, motivating a unified benchmark and standardized evaluation.
- Motivation: RAG supports knowledge-intensive applications and agentic memory management while introducing knowledge-base extraction vulnerabilities.Unlike traditional data or model extraction, RAG provides adversaries with an additional extraction channel.
- Existing Attacks: Knowledge-extraction attacks jointly craft queries to retrieve sensitive content, induce verbatim reproduction, and evade defenses.Attacks combine an INFORMATION component that steers retrieval with a COMMAND component that directs generation to reproduce retrieved content.
- Existing Attacks: Iterative query–response loops can progressively accumulate sensitive content beyond what single-query attacks recover.
- Benchmark Gap: Prior studies vary in datasets, retrieval embeddings, generators, knowledge-base construction, attacker assumptions, defender assumptions, and evaluation metrics.Examples include MiniLM versus MPNet, Llama versus Gemini, and embedding white-box versus black-box settings.
- Benchmark Contribution: The benchmark unifies the design space, standardizes RAG configurations and metrics, and provides reproducible experiments with actionable security insights.Reported insights include sensitivity of extraction to knowledge format and the value of query-query diversity exploration.
2 Related Work
RAG augments language-model generation with external knowledge, improving factuality, updating, specialization, personalization, and agentic memory capabilities.
- RAG Applications: RAG retrieves external knowledge to support downstream generation and has become a core memory-management component in agentic AI systems.Agents can retrieve, update, and reason over external knowledge during multi-step decision making.
- RAG Benefits: RAG paired with LLMs mitigates hallucinations, supports dynamic knowledge updating, enhances domain specialization, and facilitates personalization.
- RAG Applications: RAG is widely deployed in high-stakes applications, including healthcare decision support.
3 Design Space of Benchmark
The benchmark models attacks as iterative interactions among queries, retrieval, and generation, then evaluates attack and defense choices across architectures, data representations, and pipeline stages.
- RAG Interaction: An attacker submits queries over multiple rounds; retrieval returns knowledge instances that are combined with each query to produce generator outputs.The benchmark assumes iterative probing over a knowledge base and aggregates answers across sequential prompts.
- Design Space: The benchmark design space covers RAG architectures, attack and defense strategies, and evaluation protocols.RAG architectures include the retriever, generator, and knowledge base.
- Knowledge Base Setup: Knowledge bases use Original, Chunking, or Graph Triplet preprocessing over HealthCareMagic, Enron, HarryPotter, and Pokémon datasets.These strategies represent independent documents, segmented long documents, and entity-relation-entity triplets, respectively.
- Attack Query Design: Each attack query combines INFORMATION, which guides retrieval toward target content, with COMMAND, which steers generation to reproduce retrieved content.The benchmark represents attacks as Q_t = concat(I_t, C), covering single- and multi-round settings.
- Attack Query Design: The attack objective maximizes coverage of target knowledge while minimizing irrelevant leakage, although benchmark implementations decouple retriever and generator optimization.The formulation supports targeted and untargeted attacks, including the single-round case T = 1.
- Attack Query Design: Command styles range from direct reproduction requests to role-play or formatting constraints intended to bypass generator safety defenses.
- Defense Strategies: The benchmark includes input restriction, retrieval access, and generation controls, including four representative defenses.Similarity thresholding is motivated by the low relevance of semantically unnatural, diversity-optimized adversarial queries.
- Evaluation Protocols: The evaluation protocol separates extraction into retrieval, generation, and combined metrics, while ASR measures the proportion of queries producing informative, target-grounded responses.ASR requires both an informative generator output and at least one retrieved instance from the target extraction set.
4 Baseline of Benchmark
The benchmark covers six knowledge-extraction attacks that differ in how they construct the INFORMATION component, and evaluates them under four defenses across five metrics and four datasets.
- Six attack baselines differ in their INFORMATION construction strategies.The benchmark includes RandText, RandToken, RandEmb, DGEA, CopyBreak, and IKEA.
- RandText uses syntactically valid but semantically random text, while RandToken concatenates randomly sampled attacker tokens.
- RandEmb greedily aligns queries with target embeddings sampled from an external corpus such as WikiText.
- DGEA selects targets distant from prior extractions, whereas CopyBreak alternates distant exploration with local rewriting around extracted spans.
- IKEA issues human-like information-seeking queries by adaptively sampling topical anchors.
- Figure 3 compares the six attacks under four defenses using five metrics, averaged across four datasets.Transparent bars represent attack performance without defense.
5 Experiments
The benchmark compares extraction attacks and defenses across retrieval, generation, embedding, indexing, and multilingual settings. Results reveal trade-offs between extraction effectiveness, stealth, transferability, and RAG utility.
- Q1 Main Performance Comparison: DGEA consistently outperforms IKEA and CopyBreak in retrieval–extraction effectiveness EER without defenses.DGEA explicitly optimizes query–chunk diversity, whereas IKEA and CopyBreak use less direct diversity mechanisms.
- Q1 Main Performance Comparison: Explicit COMMAND attacks achieve high extraction performance, while IKEA trades lower sensitive leakage for greater stealth by using benign-looking queries.IKEA avoids explicit verbatim instructions and elicits paraphrased responses, reducing detection and reproduction.
- Defense Analysis: Query Block rejects explicit extraction queries, whereas IKEA often evades it; Summary defense consistently reduces EEG SS/LS by discouraging verbatim reproduction.Summary can also produce null or empty outputs for weakly relevant diversity-optimized queries, reducing ASR.
- Embedding Model Analysis: DGEA transfers poorly across embedding models, while IKEA and CopyBreak retain comparable performance in diagonal and off-diagonal settings.DGEA’s optimized, non-natural-language queries depend on shared attacker–retriever embedding spaces; LLM-driven attacks remain effective in black-box settings.
- Defense Analysis: Thresholding at 0.5 or 0.7 drives EER close to zero for nearly all baselines, but increasing the threshold from 0.3 to 0.7 reduces RAG recall.Thresholds near 0.3 affect R-TK, R-TT, R-EB, and DGEA more than CopyBreak and IKEA because their similarity scores are lower.
- Generation Stage Analysis: Closed-source generators outperform open-source generators for explicit verbatim attacks, while CPLX yields the highest ASR among evaluated commands.IKEA shows comparable EEG SS across generator types and stable ASR across command types because it avoids explicit reproduction commands.
- Knowledge Indexing: Fixed-Chunk indexing consistently performs worst, whereas Triplet indexing concentrates private information and enables higher extraction per token than instances or chunks.Fixed-size chunking fragments narratives and introduces redundancy through overlap.
- Query Diversity Optimization: Query diversity consistently improves extraction effectiveness across four datasets, with the largest gains for gradient-based R-EB and DGEA attacks.R-TT and CB improve less because their query optimization relies on indirect LLM-based generation and filtering.
6 Conclusion and Future Work
The benchmark unifies evaluation of knowledge-extraction attacks and defenses in RAG systems, showing that extraction depends on both retrieval and generation and that no single defense fully protects against it.
- The benchmark establishes fair, reproducible protocols for evaluating RAG knowledge-extraction attacks and defenses.
- Effective extraction requires optimization at both the retrieval and generation stages.
- No single defense provides complete protection because existing defenses operate at different RAG stages with complementary strengths.
- Limited query–query diversity causes redundant exploration, while embedding-based attacks exhibit weak cross-model transferability.
- Generator instruction-following capabilities and knowledge-base indexing strategies substantially influence extraction vulnerability.
A.1 Details of Benchmark Baselines
The benchmark covers extraction attack baselines that construct an INFORMATION component to steer retrievers through different embedding regions, ranging from random to adaptive strategies.
- Each baseline constructs the INFORMATION component of an attack query to steer the retriever toward different knowledge-base embedding regions.
- The baselines span purely random and adaptive methods, covering a broad spectrum of attack behaviors.
- RandomText uses a high-temperature LLM to generate syntactically valid but semantically random sentences for diverse embedding-space exploration.
- RandomToken concatenates vocabulary tokens sampled from the attack embedding model as a simple embedding-level randomization baseline.
- RandomEmb samples target vectors from an embedding distribution estimated using English vectors from an external corpus disjoint from the target knowledge.
A.2 Details of Generator Extraction Metrics
Generator extraction effectiveness measures overlap between generated answers and retrieved content, using alignment to handle the absence of direct correspondence between them.
- Generator extraction effectiveness EEG is measured by the overlap between generated output and retrieved content.
- Because generated responses and retrieved items lack explicit correspondence, the benchmark first aligns them before computing similarity.
- The alignment procedure considers separate cases depending on the attack strategy.
A.2.1 Alignment Strategies.
The benchmark uses pair-wise alignment when attacks request verbatim leakage and concatenated alignment when generators blend information from multiple retrieved instances.
- Pair-wise Alignment: Pair-wise alignment applies to DGEA and CopyBreak, which request verbatim leakage and produce corresponding segments for retrieved knowledge instances.
- Pair-wise Alignment: With k retrieved instances and k corresponding generated segments, pair-wise metrics can be applied directly to each aligned pair.
- Concatenated Alignment: Concatenated alignment applies to IKEA because its generator produces one paragraph blending information across retrieved instances.
- Concatenated Alignment: For concatenated alignment, all retrieved instances are joined into one reference text and compared with the generated response.
- Concatenated Alignment: If the generator refuses to answer with a refusal or safety message, the corresponding alignment score is set to 0.
- The unit-level alignment function can be instantiated from semantic or lexical perspectives.
A.2.2 Similarity Instantiations.
Lexical similarity evaluates whether generated output overlaps with retrieved content at the token level, while semantic similarity evaluates extraction at the semantic level.
- Lexical similarity measures token-level overlap between generated output and retrieved content.Common metrics include Exact Match, BLEU, and ROUGE-L; this work uses ROUGE-L.
- High lexical similarity indicates that the generator reproduced retrieved content nearly verbatim.
- Semantic similarity evaluates extraction using embedding-based representations.
A.3 Knowledge Base Setup
The benchmark compares instance, fixed-chunk, and graph-triplet indexing while normalizing extraction measurement around shared private-information targets and retrieved-token length. Rewrite-based defenses can substantially reduce extraction, but their preprocessing costs constrain large-scale evaluation and deployment.
- Knowledge Base Setup: The benchmark compares Instance, Fixed-Chunk, and Graph Triplet indexing strategies.Fixed-Chunk indexing uses fixed-length text chunks with 20% overlap, while Graph Triplet indexing represents sentences as entity-relation-entity triplets.
- Knowledge Base Setup: Raw leaked-item counts are not comparable because indexing strategies differ in granularity and semantic density.Graph indexing produces many fine-grained triplets, whereas chunk-based indexing produces fewer, more information-dense items.
- Knowledge Base Setup: The target-oriented numerator measures extracted key private information shared across indexing strategies rather than leaked indexed items.This anchors evaluation to semantic targets instead of indexing artifacts.
- Knowledge Base Setup: Token-length normalization sums retrieved tokens across attack queries to reduce bias from differing textual verbosity.The normalization accounts for information-density differences between text chunks and graph triplets.
- Rewrite-Based Defense Analysis: SAGE rewrites the corpus before indexing to preserve semantic content while avoiding verbatim reproduction, whereas VAGUE-GATE paraphrases sensitive content during generation.
- Rewrite-Based Defense Analysis: Rewrite-based defenses require substantial preprocessing, restricting evaluation to Harry Potter and Pokémon in the SAGE study.Applying SAGE to HealthCareMagic and Enron would require rewriting approximately 856M and 22B tokens, respectively.
- Rewrite-Based Defense Analysis: SAGE reduced average EEG SS to 53.1 and outperformed Summary and System Block, while VAGUE-GATE provided only moderate protection.VAGUE-GATE preserves surrounding context and high semantic similarity to the original content.
- Rewrite-Based Defense Analysis: Rewrite-based defenses reduce extraction effectiveness but their high preprocessing cost limits practicality for large-scale deployments.