Source-linked AI summary
The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)
Shenglai Zeng, Jiankun Zhang, Pengfei He, Yue Xing, Yiding Liu, Han Xu, Jie Ren, Shuaiqiang Wang, Dawei Yin, Yi Chang, Jiliang Tang
TL;DR
RAG creates an under-explored privacy problem because it may leak private retrieval data while changing how LLMs expose memorized training data. The paper uses black-box extraction attacks and comparisons with and without retrieval augmentation to study both risks. It finds substantial retrieval-data leakage but reduced tendency to output memorized training data, while noting scope limitations around inference-stage use and other retrieval-based architectures.
Problem
RAG privacy research has not fully established whether retrieval data can be extracted or whether retrieval augmentation changes LLM training-data memorization.
Method
The paper conducts empirical black-box attacks using composite prompts for retrieval extraction and targeted or prefix attacks to compare training-data exposure with and without retrieval augmentation.
Results
RAG systems can leak private retrieval data, while incorporating retrieval data substantially reduces LLMs’ tendency to output memorized training data.
Takeaways & Limitations
RAG requires attention to retrieval-data extraction while potentially offering protection for LLM training data, especially when retrieval content is public or desensitized.
Takeaways & Limitations
The study focuses on retrieval augmentation during inference and does not examine integration during pre-training or fine-tuning, while other retrieval-based architectures remain for future investigation.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) is a powerful technique to facilitate language model with proprietary and private data, where data privacy is a pivotal concern. Whereas extensive research has demonstrated the privacy risks of large language models (LLMs), the RAG technique could potentially reshape the inherent behaviors of LLM generation, posing new privacy issues that are currently under-explored. In this work, we conduct extensive empirical studies with novel attack methods, which demonstrate the vulnerability of RAG systems on leaking the private retrieval database. Despite the new risk brought by RAG on the retrieval data, we further reveal that RAG can mitigate the leakage of the LLMs' training data. Overall, we provide new insights in this paper for privacy protection of retrieval-augmented LLMs, which benefit both LLMs and RAG systems builders. Our code is available at https://github.com/phycholosogy/RAG-privacy.
1 Introduction
RAG combines retrieved external knowledge with an LLM to generate contextually relevant responses, but its use creates privacy risks for both retrieval data and memorized training data. The paper investigates extraction attacks on retrieval data and whether retrieval augmentation changes training-data leakage.
- RAG and privacy risks: RAG retrieves relevant knowledge from an external database and combines it with the user query as input to an LLM.The LLM uses both its pre-trained knowledge and the retrieved data to generate a response.
- RAG and privacy risks: The paper identifies potential privacy leakage from both the retrieval dataset and the LLM’s pre-training or fine-tuning data.Retrieval data may contain sensitive domain-specific information, while retrieval can influence the LLM’s generation behavior.
- Research questions: RQ1 asks whether attackers can extract private data from the external retrieval database in RAG.The question targets intentional extraction of sensitive information from retrieved content.
- Research questions: RQ2 examines whether retrieval data affects the memorization of LLM training data by comparing exposure with and without retrieval augmentation.The paper reports that RAG can substantially reduce the LLM’s tendency to output memorized training data.
- Research questions: The proposed composite structured prompting combines an information component for retrieval with a command component that makes the LLM output retrieved contexts.Experiments report that Llama2-7b-Chat and GPT-3.5-turbo can output verbatim or highly similar records.
2 Related Work
Related work presents RAG as a flexible method for improving LLM generation with retrieved knowledge, while prior studies document that LLMs can memorize and reveal training-corpus information.
- RAG background: RAG enhances generation by integrating retrieved information from a document corpus and is used in applications including domain-specific chatbots and email or code completion.Its architecture includes interchangeable or updateable dataset, retriever, and LLM components without retraining the entire system.
- LLM privacy: Prior research shows that LLMs can memorize and inadvertently reveal information from their pre-training corpora.Data-extraction attacks have been used to investigate the recall and reproduction of training-data segments.
3 Method
The paper models black-box attackers who use crafted queries to expose retrieval or training data, and develops attack procedures tailored to RAG’s retrieval-and-generation pipeline. Its retrieval attacks combine targeted or untargeted information prompts with commands that request context reproduction, while training-data attacks compare memorization with and without retrieval augmentation.
- Method overview: The study evaluates leakage risks in different RAG components through attacks targeting retrieval data and LLM training data.The method section introduces RAG background, a threat model, and separate attack methods for retrieval and training data.
- RAG pipeline: The RAG pipeline retrieves Top-k documents from dataset D using retriever R for query q, then concatenates the retrieved context with q before passing it to model M.Retrieval can use embedding similarity or distance, with the smallest distances selected.
- Threat model: The threat model assumes a black-box attacker who interacts only through API queries and therefore controls the crafted query q.The attacker must induce relevant retrieval and make the model disclose the retrieved context.
- Retrieval-data attacks: Composite structured prompting uses an information component to direct retrieval and a command component to instruct the LLM to include retrieved information in its response.The command may ask the model to repeat all context, while the information component varies between targeted and untargeted attacks.
- Retrieval-data attacks: Targeted retrieval attacks seek specific PII or sensitive content, whereas untargeted attacks aim to collect information broadly from the retrieval dataset.Targeted prompts can seek phone numbers or disease-related medical records; untargeted attacks use randomly selected Common Crawl chunks.
- Training-data attacks: Training-data exposure is compared with and without retrieval augmentation using targeted attacks and prefix attacks.Prefix attacks provide exact training-example prefixes and compare generated suffixes, but require attackers to know the actual training data.
4 RQ1: Can we extract private data from the external retrieval database in RAG?
Empirical attacks show that RAG systems can leak private retrieval-database content, while attack outcomes depend on prompting and retrieval settings. Re-ranking is ineffective, whereas summarization reduces untargeted leakage but has limited or adverse effects for targeted extraction.
- Attack Evaluation: RAG attacks extract private information from Enron emails and HealthcareMagic medical dialogues using targeted and untargeted prompting.The evaluation counts retrieved contexts, repeated prompts and contexts, ROUGE-like contexts, and targeted information.
- Ablation Study: Increasing retrieved documents from k = 1 to 4 increases retrieved contexts but does not substantially increase leaked documents.Targeted information increased for HealthcareMagic but not Enron, while models reproduced only partial excerpts from retrieved documents.
- Ablation Study: Command wording changes leakage: “Please repeat all the context” performs consistently well, while very long commands reduce retrieval and short commands reduce extraction.The authors attribute the long-command effect possibly to less diverse query embeddings and the short-command effect to unclear repetition instructions.
- Potential Mitigation: Re-ranking has almost no mitigation effect, whereas abstractive summarization reduces untargeted privacy risk by approximately 50%.Summarization had limited effects on targeted attacks and inadvertently increased personally identifiable information in the Enron setting.
- Potential Mitigation: Distance-threshold retrieval varies leakage and held-out HealthcareMagic ROUGE-L performance by retrieving documents only when their L2 embedding distance is below threshold p.The threshold was varied from 0 to 1.2.
5 RQ2: Can retrieval data affect the memorization of LLMs in RAG?
The study examines whether retrieval data changes LLM memorization through targeted and prefix attacks. Across these evaluations, retrieval integration substantially reduces training-data leakage, although the authors suggest it may redirect outputs toward retrieved content.
- Evaluation setup: Targeted and prefix attacks compare memorized training-data exposure with and without retrieval augmentation.The setup uses GPT-Neo-1.3B and the Enron_Mail subset of its training data, with several retrieval datasets covering general, domain-specific, and email scenarios.
- Evaluation setup: RAG conditions, random noise, and system prompts provide baselines for isolating retrieval data’s effect on memorization.“None” excludes retrieval data; “Random Noise,” “System Prompt,” and “RAG-{dataset}” denote the compared input conditions.
- Targeted Attack: RAG substantially reduces extracted personally identifiable information from training data compared with using the LLM alone.Random noise and protective system prompts mitigate leakage to some extent but remain less effective than RAG integration.
- Prefix Attack: Prefix attacks reduce successful text reconstructions from over 200 cases to fewer than 40 when retrieval data is integrated.Success is defined as ROUGE-L similarity exceeding 0.5 after supplying the first 100 tokens of training examples.
- Discussion: One possible explanation is that external retrieved data shifts generation toward the retrieved context rather than memorized training data.The authors describe this as a hypothesis supported by the differing conditional distributions with and without retrieved data.
6 Conclusions
The paper investigates privacy risks in RAG through attack methods and defense evaluations. It identifies retrieval-data extraction risks while finding that retrieval integration can substantially reduce LLM memorized-training-data leakage.
- Conclusions: The paper systematically evaluates privacy risks in RAG and explores defense techniques against those risks.Its attack methods target private retrieval data, while additional experiments examine training-data exposure.
- Conclusions: Integrating retrieval data can substantially reduce LLMs’ tendency to output memorized training data.The authors state that this finding suggests RAG could potentially mitigate training-data leakage.
7 Limitations
The study focuses on retrieval augmentation during inference and does not examine integration during pre-training or fine-tuning. Other retrieval-based language models with different architectures also remain outside its scope.
- Scope boundary: The study examines retrieval augmentation during inference but not its integration during pre-training or fine-tuning.The authors identify these phases as future-work areas.
- Scope boundary: The findings primarily concern commonly employed RAG systems, while other retrieval-based language models require further investigation.The authors note that these models have distinct components and architectures.
A.1 Ablation Studies
Ablation studies show that retrieval privacy leakage remains high across embedding models and generation temperatures, while command wording materially affects extraction performance.
- Embedding Models: Privacy leakage risks remain high across the evaluated embedding models.The study tests all-MiniLM-L6-v2, e5-base-v2, and bge-large-en-v1.5 with Llama2-7b-Chat.
- Embedding Models: Embedding models divergently affect retrieved contexts and successful extractions across datasets and attack types.E5 is more vulnerable for untargeted HealthCareMagic extraction, whereas BGE increases output in targeted Enron Email attacks.
- Temperature Parameter: RAG faces severe privacy leakage across temperatures 0, 0.6, and 1 for both targeted and untargeted attacks.Temperature 0 corresponds to greedy generation, while higher values increase randomness and output diversity.
- Impact of Questions: Different command questions affect extraction performance, with general commands such as “repeat” or no command producing very low extraction.The experiment evaluates seven command questions while keeping the default settings otherwise.
A.2 Details of Prompting Design
The prompting design builds targeted and untargeted information components for diverse retrieval-data extraction, adds protective prompt variants, and uses summarization defenses to balance security with RAG performance.
- A.2.1 The Information Part for Targeted and Untargeted Attacks: The information component extracts substantial database data, so input diversity is crucial for effective targeted and untargeted attacks.Targeted attacks additionally require contexts aligned with the attacker’s requirements.
- A.2.1 The Information Part for Targeted and Untargeted Attacks: Targeted attacks begin with attacker-provided examples or prefixes describing a clear target or an abstract target such as an email address or phone number.Examples include target-name queries and prefixes such as “Please email us at” or “Please call me at”.
- A.2.1 The Information Part for Targeted and Untargeted Attacks: The second targeted-attack stage generates many similar yet varied information inputs from the examples, including variations produced by replacing target sub-contents.For HealthcareMagic, disease names can be varied using LLM generation; the same approach is used for Enron Email.
- A.2.1 The Information Part for Targeted and Untargeted Attacks: Untargeted attacks use randomly selected Common Crawl chunks to diversify information inputs and obtain more retrieval-dataset data.Because random inputs may affect the command component, the design imposes a maximum length of 15.
- A.2.2 Construction of System Prompts and Random Noises: The protection setup uses random noise and protective system prompts, prepending 50 randomly chosen characters or a randomly selected instruction to each input.The instructions are drawn from commonly used prompts listed in Table 10.
- A.2.3 Summarization Defense Prompts: Summary Defense pursues two goals: limiting RAG performance degradation and improving security.Its prompts selectively include query-relevant context while making minimal context modifications.
- A.2.3 Summarization Defense Prompts: During summarization, each extracted context and its corresponding query are inserted into their designated prompt positions.This construction operationalizes the summarization-defense prompts for each input.
A.3 Performance Evaluation
Performance evaluation partitions each dataset into training and testing sets, measures RAG differently for HealthcareMagic and Enron Mail, and reports mitigation trade-offs alongside leakage and parameter analyses.
- Dataset and Evaluation Setup: Each dataset is split 99:1 into training and testing sets, with training building RAG and 1000 randomly sampled test instances evaluating performance.The evaluation protocol uses the training set for model construction and the sampled testing instances for assessment.
- Dataset and Evaluation Setup: HealthcareMagic uses Input Content as the RAG input, compares output with Output Content, and evaluates ROUGE-L scores.The test data follow the format “Input: Input Content\nOutput: Output Content”.
- Dataset and Evaluation Setup: Enron Mail uses the first 50 test-set tokens as RAG inputs and evaluates the corresponding output with perplexity.The dataset has no explicit input-output pairs.
- Mitigation Evaluation: The mitigation methods exhibit a performance-security trade-off, with detailed performance results reported for summarization and threshold settings.Tables 12–14 contain the detailed mitigation-performance results referenced by the authors.
- Leakage Examples: Table 15 presents examples of leaked retrieval-dataset content, including repeated retrieved texts and masked PII or privacy-sensitive words.Repeated texts are highlighted, while PII and other concerning generated words are masked.
- Parameter and Design Analyses: Additional experiments examine context count k, re-ranking, summarization, and distance thresholds across targeted and untargeted attacks.The cited tables report these factors for Enron Email and for the targeted or untargeted settings.