Source-linked AI summary
Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data
Shenglai Zeng, Jiankun Zhang, Pengfei He, Jie Ren, Tianqi Zheng, Hanqing Lu, Han Xu, Hui Liu, Yue Xing, Jiliang Tang
TL;DR
RAG can expose sensitive information in private retrieval data, and the paper investigates synthetic data as an alternative retrieval context. SAGE generates data from attribute-linked information and iteratively refines it with agents; experiments report comparable performance to original data while substantially reducing privacy risks.
Problem
RAG retrieval involving private data can risk leakage of sensitive information, while existing synthetic-data approaches are not aligned with preserving the key context needed for retrieval.
Source: §1 · pp. 1–2Method
SAGE generates synthetic retrieval data from attribute-linked key information, then uses iterative agent-based refinement to enhance privacy.
Source: §1 · p. 2Results
Takeaways & Limitations
The findings support investigating synthetic retrieval data as a way to mitigate RAG privacy concerns while maintaining utility in the evaluated scenarios.
Source: §5 · p. 8The pipeline's efficacy remains to be validated across a wider range of domains and datasets.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) enhances the outputs of language models by integrating relevant information retrieved from external knowledge sources. However, when the retrieval process involves private data, RAG systems may face severe privacy risks, potentially leading to the leakage of sensitive information. To address this issue, we propose using synthetic data as a privacy-preserving alternative for the retrieval data. We propose SAGE, a novel two-stage synthetic data generation paradigm. In the stage-1, we employ an attribute-based extraction and generation approach to preserve key contextual information from the original data. In the stage-2, we further enhance the privacy properties of the synthetic data through an agent-based iterative refinement process. Extensive experiments demonstrate that using our synthetic data as the retrieval context achieves comparable performance to using the original data while substantially reducing privacy risks. Our work takes the first step towards investigating the possibility of generating high-utility and privacy-preserving synthetic data for RAG, opening up new opportunities for the safe application of RAG systems in various domains.
1 Introduction
RAG can expose sensitive retrieval data, while existing privacy adaptations may leave risks and add utility or inference-time costs. This work proposes SAGE, which uses attribute-based generation followed by agent-based refinement; experiments report comparable utility and substantially reduced privacy risks.
1 Introduction
- RAG privacy attacks can extract retrieval sentences or targeted personal information, putting sensitive-domain uses such as medical chatbots at risk.Existing adaptations may not fully eliminate risk and can introduce privacy-utility trade-offs or inference-time costs.Source: §1 · pp. 1–2
- Synthetic retrieval data addresses a RAG-specific gap: existing synthetic methods are not designed to preserve the key information needed to answer related questions.The paper identifies a lack of exploration into how to generate high-quality synthetic retrieval data while selecting what information to preserve.Source: §1 · p. 2
- SAGE uses attribute-based extraction and generation to retain key contextual information, then iteratively refines the synthetic data with privacy-assessment and rewriting agents.The extraction and generation stage produces data from key information associated with dataset attributes.Source: §1 · p. 2
- Experiments report comparable performance to original retrieval data while substantially reducing associated privacy risks.The paper presents synthetic retrieval data as a data-level alternative that keeps original data from being directly supplied to the LLM.Source: §1 · p. 2
2 Related Works
Related work describes both RAG's utility and its privacy vulnerabilities. Prior synthetic-data methods explore generation and privacy, but the paper says they do not guarantee preservation of the original data's contextual information for RAG.
2.1 Retrieval-augmented generation and its privacy issues
- Prior work describes RAG as improving output accuracy and relevance and mitigating hallucinations, while studies also report privacy leakage and extraction vulnerabilities.Reported concerns include attacks on retrieval-based language models and production RAG systems.Source: §2.1 · pp. 2–3
2.2 Synthetic data generation using large language models
- Prior synthetic-data work explores zero-shot generation, quality improvement, diversity, and privacy, but does not guarantee preservation of original contextual information for RAG.Examples include differentially private in-context demonstrations and private data generation.Source: §2.2 · p. 3
3 Methods
SAGE generates retrieval data in two stages: attribute-based generation aims to retain essential information, and agent-based refinement further mitigates privacy concerns. The process can run offline before corresponding synthetic data is returned at inference.
3 Methods
- SAGE has two stages: attribute-based data generation targets essential information, and agent-based interactive refinement targets further privacy mitigation.Generation can be performed offline once; at inference, the corresponding synthetic data is returned when original data is identified.Source: §3 · p. 3
3.1 Stage-1: Attribute-based data generation
- Stage 1 identifies essential dataset attributes, extracts attribute-linked key information for each sample, then generates synthetic data conditioned on those pairs.Attribute identification uses few-shot examples and is performed once; the identifier, extractor, and generator may use the same or different models.Source: §3.1 · p. 3
3.2 Stage-2: Agent-based private data refinement
- Stage 1 can retain privacy risks because it lacks privacy controls; Stage 2 uses privacy assessment and rewriting agents to iteratively refine generated data.The privacy agent assesses generated data alongside the original for personally identifying information or linkable personal information, provides feedback, and reassesses rewritten data until it deems the result safe.Source: §3.2 · pp. 3–4
4 Experiment
The experiments evaluate SAGE’s utility and privacy, then examine how model choice, attribute count, and retrieval size affect those outcomes. Across medical dialogue and ODQA, synthetic retrieval data maintains utility comparable to original data, while agent refinement further reduces privacy leakage.
4.1 Evaluation Setup
- Experiments address HealthcareMagic-101 medical dialogues, Wiki-PII privacy data, and ODQA benchmarks; compare paraphrasing, ZeroGen, and AttrPrompt; and report retrieval conditions Origin, Stage-1, and Stage-2.Origin uses original data directly as retrieval data; Stage-1 uses attribute-based generation outputs; Stage-2 uses outputs of the complete SAGE pipeline.Source: §4.1 · p. 5
4.2 Utility of using synthetic data
- Synthetic retrieval data achieves utility comparable to, and sometimes better than, original data on medical dialogue and ODQA, while outperforming no retrieval, paraphrasing, and ZeroGen on medical dialogue.The reported ODQA gains over original data occur on NQ and WQ.Source: §4.2 · pp. 5–6, Table 1 · p. 5, Table 2 · p. 6
4.3 Privacy of using synthetic data
- Stage-1 data has lower untargeted attack success than original data and simple paraphrasing, while refined data shows nearly no untargeted attack success.The reported untargeted attacks use HealthCareMagic.Source: Table 4 · p. 6, §4.3 · p. 7
- After stage-1 generation reduces targeted leakage, agent-based refinement lowers targeted attack success to nearly zero.The targeted-attack findings are reported for Wiki-PII and HealthCareMagic.Source: Table 3 · p. 6, §4.3 · p. 7
4.4 Ablation Studies
- Weak Llama3-Chat-8b models yield poor utility, GPT-4 does not outperform GPT-3.5 on stage-1 utility, and GPT-4 yields lower targeted leakage than GPT-3.5.The model-choice comparisons cover both the information extractor and data generator.Source: §4.4 · pp. 7–8, Figure 3 · p. 8
- Very few attributes can impair utility while lowering privacy exposure; adding attributes improves utility but does not guarantee continued gains.The attribute-count experiments use HealthCareMagic.Source: §4.4 · p. 8, Figure 4 · p. 8
- As retrieved-document count increases, stage-1 privacy risks can rise in some cases, whereas agent-refined data maintains minimal leakage.This comparison tests targeted attacks on HealthCareMagic.Source: §4.4 · p. 8, Figure 5 · p. 8
5 Conclusions
The paper investigates synthetic data as RAG retrieval data for mitigating privacy concerns. Its two-stage pipeline combines attribute-based generation with iterative agent-based refinement, and experiments report comparable performance to original data while mitigating privacy issues.
5 Conclusions
- Synthetic retrieval data achieves comparable performance to original data while mitigating associated privacy issues.The conclusion describes this as an initial investigation of synthetic data for RAG.Source: §5 · p. 8
- SAGE combines attribute-based generation to maintain key information with iterative agent-based refinement to further enhance privacy.Source: §5 · p. 8
- The authors identify safe RAG application in sensitive domains as an opportunity opened by this work.Source: §5 · p. 8
6 Limitations
The study evaluates synthetic data for RAG in representative scenarios such as healthcare, but does not claim differential-privacy guarantees. Its privacy scope is common practical attacks, while stricter guarantees remain open.
6 Limitations
- The evaluation covers representative scenarios such as healthcare, with broader domains and datasets left for future validation.Source: §6 · p. 9
- The paper focuses on common practical RAG attacks, whose definitions and settings differ from differential privacy.The authors characterize differential privacy as a stronger form of protection that aims to make each data item indistinguishable.Source: §6 · p. 9
- Providing strict privacy guarantees, such as integrating differential privacy into RAG, remains an open challenge.Source: §6 · p. 9
7 Ethic Statement
The authors frame the work as an effort to reduce privacy risks from sensitive-data leakage in RAG. They emphasize the need for further research into the method’s risks and benefits and responsible deployment.
7 Ethic Statement
- The approach replaces sensitive data during RAG with synthetic data intended to reduce privacy-breach risks.Source: §7 · p. 9
- The authors call for further research to understand the method’s risks and benefits.Source: §7 · p. 9
- The statement connects responsible RAG deployment with balancing language-model benefits and individual privacy rights.Source: §7 · p. 9
A.1.1 Prompts used in stage-1
Stage 1 uses five-shot GPT-4 prompts to identify important attributes, then generates synthetic data from extracted attribute-related information. Default attribute counts differ between the medical-dialog and Wiki-PII datasets.
A.1.1 Prompts used in stage-1
- The attribute identifier uses five-shot GPT-4 samples to summarize important attributes.The medical-dialog dataset uses five attributes for both patient and doctor information; Wiki-PII uses three.Source: §A.1.1 · p. 12
- After attribute-related information is extracted, the data generator uses it to produce synthetic data.Source: §A.1.1 · p. 12
- The appendix points to dataset-specific tables for the extractor attributes and prompts and for the data-generator prompts.Source: §A.1.1 · p. 12
A.1.2 Prompts used in stage-2
Stage 2 iteratively checks generated data for privacy and asks a rewriting agent to revise it when needed. The privacy agent ends the process with a safe signal; the reported average was 3.964 rounds.
A.1.2 Prompts used in stage-2
- The privacy assessment agent checks the generated data alongside the original data and either signals that it is safe or sends suggestions to the rewriting agent.The rewriting agent revises the synthetic data based on the feedback, and the agents repeat this cycle until the privacy agent signals safety.Source: §A.1.2 · p. 12
- The iterative refinement process averaged 3.964 rounds; the authors report that in most cases one refinement round is enough to generate safe data.Source: §A.1.2 · p. 12
- The privacy agent’s system prompt is detailed in Table 16, and the rewriting agent’s system prompt is detailed in Table 15.Source: §A.1.2 · p. 12
A.2 Diverse metric evaluation of model utility
Across the reported utility evaluations, synthetic retrieval data performs comparably to or better than original data. The appendix also details privacy attacks, baseline methods, costs, evaluation metrics, dataset construction, and domain adaptation.
A.2 Diverse metric evaluation of model utility
- Across PopQA and NQ, synthetic retrieval data achieves utility comparable to or better than original data across the reported metrics.Table 5 compares synthetic data, original data, and zero-shot prediction using only an LLM.Source: §A.2 · p. 14
A.3 Details of baseline implementation / A.4 Details of Attack Design.
- Paraphrase filters or rewrites retrieved context, while ZeroGen generates question-answer pairs from named entities and AttrPrompt generates data from selected attributes without original retrieval data.The attack design uses an information component to guide retrieval and a command component to ask the model to include retrieved information; targeted attacks use generated examples, while untargeted attacks use Common Crawl segments limited to 15 tokens.Source: §A.3 · p. 14, §A.4 · pp. 15–16
A.5 Directly Compare Synthetic Data and Original Data
- The authors report nearly no targeted information and almost no untargeted information leaked when comparing synthetic data directly with original data, including in this described extreme case.The comparison covers targeted leakage for Wiki-PII and HealthCareMagic and untargeted leakage for HealthCareMagic.Source: §A.5 · pp. 16–17
A.6 Comparison to paraphrasing with GPT-4o
- One-round paraphrasing variants using GPT-4o still show targeted and untargeted privacy concerns, whereas the authors report GPT-3.5 can achieve good results with their method.The variants add either a general privacy instruction or a specific privacy-protection list.Source: §A.6 · p. 17
A.7 Cost of synthetic data
- Synthetic data is reported to be shorter than original data by 50 tokens for wiki and 86 tokens for chatdoctor, and GPT-3.5 generation costs about $0.003 per sample.The authors describe generation as a one-time offline process that adds no inference-time costs.Source: §A.7 · p. 17
A.8 Details of Evaluation Metrics
- Utility evaluation includes Exact Match and Ragas correctness judgments in addition to ROUGE-L and BLEU; the reported ROUGE-L score is an F-measure balancing precision and recall.Source: §A.8 · pp. 17–18
A.9 Details of Dataset Construction
- Wiki-PII is constructed by extracting authentic PIIs from Enron Mail, segmenting wiki text into 1500-character chunks, and inserting the PIIs at sentence ends.Source: §A.9 · p. 18
A.10 Discussions when adapting SAGE in specific domain application / A.11 Examples of synthetic samples
- SAGE’s attribute count and output format can be adapted to a domain, and its privacy checks can be customized; the authors note that effective domain customization may require domain expertise.Their examples show stage 1 filtering some detail while retaining disease information, followed by stage 2 concealing further details while retaining information for medical advice.Source: §A.10 · pp. 18–19, §A.11 · p. 19