Source-linked AI summary
Mitigating Database Leakage in RAG Systems with Keyword-Grounded Fact Substitution
Ziliang Zhang, Yubo Zhu, Wei Tong, Jingyu Hua, Zijian Wang, Yuan Zhang, Sheng Zhong
TL;DR
RAG systems can leak proprietary database content when prompt injections mislead retrieval or generation. KFS-RAG identifies causally influential keywords in retrieved contexts, converts them into compact grounded facts, and substitutes those facts for raw context; evaluations report reduced leakage while preserving response utility and quality. The study supports sanitized retrieval contexts as a practical basis for more secure RAG systems.
Problem
Prompt injection can mislead RAG retrievers or generators and expose proprietary database content.
Method
KFS-RAG uses attention rollout and causal perturbation to identify relevant keywords, then substitutes retrieved passages with compact keyword-grounded facts.
Results
KFS-RAG reduces proprietary content leakage while preserving generated-answer utility and response quality across evaluated tasks and attacks.
Takeaways & Limitations
Sanitized, causality-aware fact substitution provides a practical foundation for secure and trustworthy RAG systems.
Takeaways & Limitations
The evaluation focuses on single-turn, unimodal text settings and does not establish applicability to multi-turn dialogues, multimodal models, or other model families.
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for combining large language models (LLMs) with external knowledge sources. However, RAG systems remain vulnerable to prompt injection attacks, which may mislead the retriever or generator to expose sensitive database contents. To address this issue, we propose KFS-RAG, a defense that mitigates information leakage by reformulating the retrieved context. Specifically, our method first identifies a small set of influential keywords from the retrieved context via an attention rollout plus a causal perturbation mechanism. These keywords are then used to guide an auxiliary LLM to generate a compact set of keyword-grounded facts from the retrieved passages. Finally, the original context is substituted with these curated facts, ensuring that the generator operates on sanitized evidence rather than the raw retrieved text. Experimental evaluations demonstrate that KFS-RAG significantly reduces the risk of database leakage under injection attacks while maintaining response accuracy and relevance. This work highlights a practical pathway toward building secure and trustworthy RAG systems.
1 Introduction
RAG systems face database-leakage risks from prompt injection, while existing defenses impose trade-offs in detail preservation, assumptions, or usability. KFS-RAG processes retrieved contexts after retrieval, replacing them with relevant distilled facts to reduce leakage while preserving answer utility.
- Prompt injection can mislead RAG retrievers or generators into leaking proprietary database content.
- Existing defenses use synthetic documents, differential privacy, or encryption, but each can lose critical details, require restrictive assumptions, or reduce usability.Synthetic documents may lack fine-grained guidance; differential privacy is difficult for small databases; encryption can limit knowledge-base access.
- KFS-RAG leaves source documents unchanged and intervenes after retrieval by transforming retrieved contexts before generation.It identifies query-relevant keywords and extracts their associated factual information.
- The method replaces raw retrieved context with distilled information, discarding redundant or irrelevant details while retaining evidence relevant to the user query.
- Experiments across open-domain QA, multi-hop QA, and medical dialogue generation reduce proprietary content leakage while preserving generated-answer utility.
- KFS-RAG sanitizes retrieved contexts through causality-aware keyword grounding to reduce information leakage while preserving response quality.
2 Related Work
Research on defending RAG against injection threats centers on data synthesis, differential privacy, and encryption. These approaches face limitations involving omitted source details, dependence on context redundancy, or reduced utility from restricted context use.
- Prior defenses against RAG injection threats primarily pursue data synthesis, differential privacy, or data encryption.
- Synthetic-document methods can omit granular details from the original source text.
- Encryption-based approaches may reduce utility because they cannot fully leverage all retrieved context.
3 Method
KFS-RAG processes retrieved content after retrieval: it identifies query-relevant keywords, extracts bounded keyword-grounded facts, and substitutes those facts for raw context. Its hybrid attention-perturbation selection combines efficient candidate proposal with causal verification, while optional Context Re-synthesis restores stylistic and discourse cues.
- 3 Method: KFS-RAG leaves the document collection unchanged and sanitizes retrieved content by extracting query-relevant facts before generation.The method operates after retrieval, discarding irrelevant or potentially sensitive content while preserving evidence needed for answering.
- 3 Method: The hybrid Attention-Perturbation mechanism uses attention to propose high-recall keyword candidates and loss-based perturbation to validate causal relevance.Attention provides lightweight screening, while masking or modifying candidates tests whether they meaningfully affect model loss.
- 3.2.1 Attention-Guided Candidate Selection: Attention rollout aggregates attention across layers and residual connections, then scores context tokens by their influence on provisional generated answer tokens.At inference, a lightweight provisional generation pass supplies the draft tokens used for rollout and perturbation scoring.
- 3.2.1 Attention-Guided Candidate Selection: Top-ranked words or contiguous phrases become perturbation candidates, because attention alone can highlight incidental tokens rather than causally necessary evidence.Restricting perturbation to this small candidate subset preserves scalability while adding causal verification.
- 3.3 Keyword-Grounded Fact Extraction: An auxiliary LLM converts selected keywords and retrieved passages into compact factual statements that replace the original context for downstream generation.A length constraint limits unintended disclosure while balancing information sufficiency against leakage risk.
- 3.4 Fact Re-Synthesis into Context: Optional Context Re-synthesis rewrites distilled facts using in-domain exemplars to recover stylistic and discourse cues lost by fact-only contexts.This is especially relevant when responses must retain conversational tone and contextual appropriateness, such as doctor–patient dialogue.
4 Experiments
Experiments evaluate KFS-RAG across utility, security, model, keyword-budget, and ablation settings. Results show that KFS-RAG-Fact generally preserves answer quality while reducing leakage, with effectiveness depending on task complexity and fact quantity.
- Experimental setup: Evaluation spans open-domain QA, multi-hop QA, and domain-specific medical dialogue, using utility and leakage-resistance assessments.The setup includes RAG baselines, context-sanitization methods, and KFS-RAG variants.
- Utility: KFS-RAG-Fact achieves the strongest performance on most datasets, outperforming Paraphrased RAG, Random-Fact, and SAGE, while matching or exceeding Vanilla RAG in several ODQA settings.The results attribute this pattern to query-conditioned evidence distillation that concentrates salient facts and reduces redundant noise.
- Security: KFS-RAG-Fact provides the strongest balance between answer quality and leakage resistance among the evaluated defenses.Selective keyword-grounded extraction and a strict length constraint limit reproduced context, whereas KFS-RAG-Full can slightly increase leakage through additional resynthesis.
- Fact budget: For open-domain QA, more facts can add semantic noise and increase leakage, whereas multi-hop reasoning benefits from more facts that help capture complete causal chains.Security gains from smaller fact sets show diminishing marginal returns in attack capability as fact volume increases.
- Model robustness: Across both generators, KFS-RAG consistently improves utility while reducing corpus content disclosure, suggesting effectiveness that does not depend on model-specific behavior.The analysis connects this result to supplying causally relevant facts instead of raw, redundant context.
- Ablations: The two-stage Attention-Perturbation mechanism improves the effectiveness–efficiency trade-off by narrowing candidates with attention before causal perturbation verification.Perturbation-only scoring achieves the highest utility but requires a separate forward pass for each candidate token, so latency grows linearly with context length.
5 Conclusion
The paper concludes that KFS-RAG addresses the privacy-utility dilemma in RAG through a two-stage Attention-Perturbation mechanism. Across benchmarks, it distills causal cores from retrieved documents to improve the balance between response utility and leakage resistance.
- KFS-RAG addresses the privacy-utility dilemma in RAG systems.
- Its two-stage Attention-Perturbation mechanism identifies and distills the causal core of retrieved documents.
- Across several benchmarks, KFS-RAG achieves a superior balance between response utility and leakage resistance.
Limitations
The study is limited to single-turn, unimodal text-based inputs and does not evaluate multi-turn interactions or broader model modalities and families.
- The study focuses on single-turn inputs and does not explicitly address multi-turn or conversational settings.Multi-turn extensions may require additional mechanism design because contextual dependencies can span multiple interactions.
- The approach is evaluated in a unimodal text-based setting and does not investigate multimodal models or model families beyond large language models.The authors identify broader modalities and model families as future-work directions.
A.1 RAG-based LLM Applications
A RAG-based LLM application combines a knowledge database, a retriever, and an LLM. It retrieves relevant context chunks and concatenates them with the question for answer generation.
- A RAG-based LLM application consists of a knowledge database, a retriever, and an LLM.The database is denoted D = {v1, v2, . . . , vn}, where vi is the i-th database item.
- Data Retrieval: During data retrieval, context is partitioned into chunks, embedded as vectors, stored in a vector database, and ranked by distance to the question.The top k most relevant chunks form paired question-context samples.
- Answer Generation: During answer generation, the acquired question-context pairs are queried by concatenating the context and question inputs.
A.2 Injection Attacks in RAG
RAG injection attacks combine anchor information with adversarial instructions to influence retrieval and generation. They include native, formatting-based, context-ignoring, fake-completion, and combined attack patterns, alongside targeted and untargeted strategies.
- An injected query contains anchor information and an adversarial instruction.The anchor steers retrieval toward attacker-chosen passages, while the instruction prompts the generator to reveal or incorporate retrieved context.
- Targeted attacks use domain knowledge or inferred pipeline details to craft plausible bait, whereas untargeted attacks do not rely on that precision.The supplied passage distinguishes the categories by attacker goals and prior knowledge.
- Attack Patterns: Native attacks directly append a malicious instruction as the adversarial instruction segment.
- Attack Patterns: Escape-character attacks use formatting tokens such as \n and \t to make the adversarial instruction appear separate.
- Attack Patterns: Context-ignoring attacks insert phrases that encourage the model to disregard prior constraints.
- Attack Patterns: Fake-completion attacks add a spurious completion fragment, while combined attacks mix multiple techniques to improve success rates and evade detection.The example fake completion begins with “Answer: task complete.”
A.3 Context-side Adversarial Instructions
The adaptive setting embeds adversarial instructions in retrieved documents themselves. KFS-RAG still substantially reduces leakage because it selects query-relevant keywords before substituting length-bounded facts for the generator's context.
- In the adaptive setting, adversarial instructions are embedded directly in the retrieved documents.
- KFS-RAG still substantially reduces leakage compared with Vanilla RAG and SAGE.AP selects query-relevant keywords before fact substitution, and the final generator observes only the length-bounded fact set.
A.4 Auxiliary Model Sensitivity
KFS-RAG remains effective across auxiliary-model choices, but its privacy–utility behavior depends on task demands, fact volume, and candidate-set size. Stronger extractors can improve the trade-off, while excessive context or candidates introduce noise, leakage, or overhead.
- Auxiliary-model sensitivity: KFS-RAG is not tied to a single auxiliary model, although stronger extractors can improve the privacy–utility trade-off.
- Evaluation scope: The evaluation spans open-domain factual retrieval, multi-hop compositional reasoning, and domain-specific medical dialogue.
- Baseline comparison: KFS-RAG offers a stronger privacy–utility balance than summarization, advanced RAG, and prompt-hardening under injection attacks.Summarization and advanced RAG expose more database content, while prompt-hardening sharply reduces answer utility.
- Candidate-set sensitivity: Increasing the AP candidate set does not yield linear performance improvements and adds perturbation and forward-pass costs.Stochastic extraction can overemphasize isolated details, potentially reducing global semantic coherence and ROUGE-L.