Source-linked AI summary
RAG-CT: Mitigating Privacy Risks on Retrieval-Augmented Generation Systems via Scanning Prompt Distribution
Xingyu Lyu, Jiayimei Wang, Jianfeng He, Ning Wang, Yidan Hu, Yimin Chen
TL;DR
Adversaries can exploit RAG retrieval to extract PII from the underlying knowledge base. RAG-CT detects malicious queries by combining entropy and margin in an anomaly score, and the reported evaluations show reduced ASR across the tested attacks. The authors present it as a lightweight approach for safeguarding RAG systems against PII leakage.
Problem
Adversaries can exploit RAG to extract PII directly from the knowledge base by crafting queries that prompt the model to reproduce retrieved private content.
Source: §1 · p. 1Method
RAG-CT detects malicious queries by combining entropy and margin distributions in a unified anomaly score.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The authors conclude that RAG-CT is a lightweight approach for safeguarding RAG systems against PII leakage.
Source: §6 · p. 4The modeled attacker operates through APIs without direct access to the model’s internal parameters, weights, or knowledge base, though domain knowledge may guide query construction.
Source: §2.2 · p. 2
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for improving the quality of generated contents of Large Language Models (LLMs) by grounding responses in external knowledge, thus reducing hallucinations and factual errors. However, recent studies have highlighted a critical vulnerability: adversaries can exploit the retrieval process to extract personally identifiable information (PII) from the underlying corpus. To mitigate this risk, we propose a novel defense, RAG-CT, that identifies malicious queries by analyzing their entropy and margin distributions and using a score-based detection method. Extensive experiments with four state-of-the-art attack strategies and four defense baselines on two datasets show that our approach significantly reduces PII leakage while outperforming existing defenses. This work provides a lightweight yet effective mechanism to protect RAG systems against PII leakage without requiring modifications to the underlying LLM or retriever.
1. INTRODUCTION
RAG grounds LLM responses in external knowledge, but adversaries can exploit retrieval to extract PII from the underlying knowledge base. RAG-CT detects malicious queries using entropy and margin distributions, and the reported evaluations show reduced attack success rates across multiple attacks.
1. INTRODUCTION
- Adversaries exploit RAG to extract PII by crafting malicious queries that prompt the model to output retrieved private content verbatim.The cited attacks include query expansion and relevance-driven query updates.Source: §1 · p. 1
- RAG-CT detects malicious queries by combining entropy, which measures similarity-score spread, with margin, which measures top-match decisiveness.Low entropy and high margin together signal targeted extraction.Source: §1 · pp. 1–2
- The defense jointly analyzes prompt-query entropy and margin distributions, combining global spread with local decisiveness to score extraction attempts.The authors describe this as a unified anomaly-scoring mechanism.Source: §1 · p. 2
- ASR falls to 0.00 for TBTG and PIDE and below 0.1 for RAG-Thief and Pirate under RAG-CT.The evaluation covers multiple LLMs and datasets.Source: §1 · p. 2
2. SYSTEM AND ADVERSARY MODELS
A typical RAG system retrieves top-ranked chunks from an indexed knowledge base and supplies them with the query to an LLM for a response. The attacker is modeled as a black-box API user seeking PII-bearing chunks, without direct access to the system’s internal parameters, weights, or knowledge base.
2.1. System Model
- The retriever selects the top-k knowledge-base chunks for a query, concatenates them with the query, and sends them to the LLM-RAG to generate a response.The system comprises a retriever, indexed knowledge base, and LLM.Source: §2.1 · p. 2
2.2. Adversary Model
- The attacker seeks PII-bearing chunks and can interact only through APIs, without direct access to model parameters, weights, or the knowledge base.The modeled PII includes phone numbers, email addresses, SSNs, driver’s license numbers, and passport numbers.Source: §2.2 · p. 2
- Across a query sequence, the attacker extracts candidate PII sets from responses and unions them into the recovered PII chunks.The attacker may possess domain knowledge to guide queries toward PII-related targets.Source: §2.2 · p. 2
3. DESIGN OF RAG-CT
RAG-CT relies on the contrast between benign queries, which tend to retrieve multiple related records, and malicious queries, which tend to concentrate retrieval on one private record. It computes entropy and margin from top-k similarity scores, combines them into an anomaly score, and flags queries above a threshold.
3.1. Intuition
- Low entropy and a large margin characterize malicious queries focused on one private record, while benign queries tend to show higher entropy and smaller margins.Entropy measures the spread of normalized similarity scores; margin is the gap between the top two scores.Source: §3.1 · p. 2
3.2. Methodology
- RAG-CT retrieves top-k records, normalizes their similarity scores, computes entropy and margin, and combines the measures into an anomaly score for threshold-based flagging.The five-step workflow begins with retrieval and ends by flagging a query when its score exceeds the threshold.Source: §3.2 · pp. 2–3
- The method uses normalized top-k scores to calculate retrieval entropy and the top-1 versus top-2 similarity gap as its margin measure.The entropy is normalized to [0, 1], and the margin is min-max normalized to [0, 1].Source: §3.2 · p. 3
4. EVALUATION
The evaluation tests RAG-CT against four attack strategies and existing defenses in healthcare and email settings. It reports consistent defense across attacks and examines how retrieval size, threshold, and scoring indicators affect performance.
Experimental setup
- The evaluation uses HealthcareMagic-101 and Enron Email, four LLMs, four attacks, and baselines spanning rewriting, filtering, summarization, and erase-and-check.It measures Unique Leakage Chunks, PII elements, and Attack Success Rate.Source: §4 · p. 3
Main results
- Without defense, attacks achieve high PII leakage in the Healthcare evaluation using 100 prompts.Table 1 reports attack results without defense; the passage does not provide its cell values.Source: §4 · p. 3, Table 1 · p. 3
- RAG-CT reduces ASR to 0.00 for TBTG and PIDE across models and below 0.1 for RAG-Thief and Pirate; similar trends appear on Email.Fig. 5a reports defense results, while Table 2 covers Healthcare results across attacks.Source: §4 · p. 4, Table 2 · p. 4, Figure 5 · p. 4
- Rule-based filtering helps against TBTG and partly against PIDE and RAG-Thief, while erase-and-check helps against TBTG and PIDE but performs poorly on other attacks.Rephrasing and summarization yield only marginal leakage reductions.Source: §4 · pp. 3–4
Ablation studies
- Increasing top-k retrieval size raises ASR; lowering the detection threshold strengthens filtering, and RAG-CT outperforms entropy-only and margin-only strategies.The default threshold is 0.5.Source: §4 · p. 4, Figure 5 · p. 4
5. RELATED WORK
Prior work spans attacks that extract private fragments or adapt queries, and defenses ranging from prompt processing to cryptographic approaches. The paper groups several existing defenses into three broad categories.
Privacy leakage on RAG systems
- Early prompt-injection attacks used static queries, while newer approaches iteratively refine queries to uncover sensitive data.Examples include segment extensions and dynamically updated anchors and queries.Source: §5 · p. 4
Defenses against privacy leakage in LLM/RAG
- Existing defenses include self-processing, auxiliary filtering, and input permutation approaches.Examples are query rewriting, keyword- or semantic-based blocking, and erase-and-check.Source: §5 · p. 4
- Cryptographic frameworks also use differential privacy, secure multi-party computation, and homomorphic encryption to mitigate leakage.Source: §5 · p. 4
6. CONCLUSION
The conclusion presents RAG-CT as a distribution-based defense against PII extraction in RAG systems. It combines entropy and margin analysis to detect and block queries targeting sensitive records, and reports that it outperforms state-of-the-art defenses.
6. CONCLUSION
- RAG-CT combines entropy and margin analysis to detect and block queries targeting sensitive records.Source: §6 · p. 4
- The authors report that extensive experiments show RAG-CT outperforms state-of-the-art defenses.Source: §6 · p. 4
- The paper describes RAG-CT as a lightweight approach for safeguarding RAG systems against privacy leakage.Source: §6 · p. 4