Source-linked AI summary
Detecting RAG Extraction Attack via Dual-Path Runtime Integrity Game
Yuanbo Xie, Yingjie Zhang, Yulin Li, Shouyou Song, Xiaokun Chen, Zhihan Liu, Liya Su, Tingwen Liu
TL;DR
RAG extraction attacks can expose retrieved proprietary content and support reconstruction of knowledge bases, while effective defenses remain limited. CanaryRAG embeds canaries in retrieved chunks and checks target and oracle paths for expected canary behavior. It reports substantially lower recovery than prior defenses, with negligible task-performance and latency impact, while focusing on reconstruction attacks rather than individual facts or other privacy threats.
Problem
Black-box prompts can induce RAG systems to disclose retrieved proprietary content, and semantic or lexical matching cannot reliably distinguish unauthorized leakage from legitimate use.
Source: §1 · p. 2Method
CanaryRAG embeds canary tokens in retrieved chunks and detects runtime integrity violations by monitoring target and oracle path behavior.
Source: §1 · p. 2, §4.1 · p. 5, §6 · p. 9Results
CanaryRAG achieves a relative mean CRR of 0.04× against the No Defense baseline, corresponding to an average reduction of approximately 96%.
Source: §5.2 · p. 8Takeaways & Limitations
CanaryRAG is presented as a plug-and-play safeguard for proprietary knowledge that requires no retraining or architectural changes and has negligible task-performance and latency overhead.
Source: §6 · p. 9The work focuses on RAG extraction attacks aimed at reconstructing the knowledge base, not privacy threats such as membership inference or side-channel attacks, or guarantees for individual facts, records, or attributes.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) systems augment large language models with external knowledge, yet introduce a critical security vulnerability: RAG Knowledge Base Leakage, wherein adversarial prompts can induce the model to divulge retrieved proprietary content. Recent studies reveal that such leakage can be executed through adaptive and iterative attack strategies (named RAG extraction attack), while effective countermeasures remain notably lacking. To bridge this gap, we propose CanaryRAG, a runtime defense mechanism inspired by stack canaries in software security. CanaryRAG embeds carefully designed canary tokens into retrieved chunks and reformulates RAG extraction defense as a dual-path runtime integrity game. Leakage is detected in real time whenever either the target or oracle path violates its expected canary behavior, including under adaptive suppression and obfuscation. Extensive evaluations against existing attacks demonstrate that CanaryRAG provides robust defense, achieving substantially lower chunk recovery rates than state-of-the-art baselines while imposing negligible impact on task performance and inference latency. Moreover, as a plug-and-play solution, CanaryRAG can be seamlessly integrated into arbitrary RAG pipelines without requiring retraining or structural modifications, offering a practical and scalable safeguard for proprietary data.
1 Introduction
RAG extraction attacks can reconstruct proprietary knowledge bases through black-box prompting, while existing defenses remain limited and often passive. CanaryRAG detects leakage through runtime canary behavior, with reported protection against state-of-the-art attacks and negligible task-performance and latency impact.
1 Introduction
- Black-box adversarial prompts can induce RAG systems to divulge retrieved content, enabling progressive reconstruction of proprietary knowledge bases and risking legal, economic, and reputational consequences.Prior defenses are described as passive, difficult to deploy plug-and-play, and vulnerable to strong adaptive attacks.Source: §1 · pp. 1–2
- Lexical matching can be bypassed by obfuscation, while semantic similarity cannot reliably distinguish legitimate use of retrieved content from unauthorized disclosure.Both legitimate outputs and intra-class leakage may be grounded in the same source material.Source: §1 · p. 2
- CanaryRAG reframes extraction as a runtime integrity violation, adapting the software-security canary idea to signal when retrieved content is exposed.The paper characterizes canaries as signals of compromised integrity, not as mechanisms that themselves prevent attacks.Source: §1 · p. 2
- CanaryRAG injects canary tokens into retrieved chunks and monitors output, using an oracle task to test for canary suppression by adaptive adversaries.The defense is described as lightweight, model-agnostic, and deployable without retraining or modifying existing RAG pipelines.Source: §1 · p. 2
- CanaryRAG is reported to provide the strongest protection to date against state-of-the-art RAG extraction attacks, with negligible impact on task performance and inference latency.The evaluation also reports robustness against adversaries aware of the canary mechanism.Source: §1 · p. 2
2 Related Work
Prior work documents scalable black-box and iterative attacks for extracting RAG knowledge bases, alongside defenses that address selected leakage pathways. The paper situates CanaryRAG against these attacks and defenses and draws on stack-canary integrity monitoring.
RAG Knowledge Base Extraction Attacks
- Black-box extraction attacks use adversarial prompts and iterative feedback to progressively harvest RAG corpora; one embedding-based approach instead requires white-box access to the embedding model.The cited approaches include instruction following, structured prompting, feedback-guided query refinement, and adaptive prompt optimization.Source: §2 · pp. 2–3
Defenses Against RAG Extraction Attack
- Existing defenses include post-retrieval summarization, synthetic-corpus replacement, RAGFort’s dual-path controls, and implicit watermarks for post-leakage attribution.Prior work reports summarization can halve extraction success, while reranking offers negligible protection.Source: §2 · p. 3
Memory Safety in Software Security
- Stack canaries are integrity sentinels whose alteration signals a boundary violation; CanaryRAG adapts this principle by treating canary appearance in outputs as a retrieval-boundary signal.The passage describes CanaryRAG as the first work to operationalize explicit canaries as an online integrity signal for mitigating extraction during generation.Source: §2 · p. 3
3 Preliminary Work
The paper frames RAG knowledge-base leakage as a confidentiality and runtime-integrity concern, then specifies the adversaries and deployment capabilities considered. Its defense targets large-scale reconstruction rather than eliminating all task-necessary disclosure.
3 Preliminary Work
- A standard RAG system combines a private knowledge base, a retriever that selects relevant chunks, and a generator conditioned on the query and retrieved chunks.The paper identifies confidentiality and integrity as two security perspectives on leakage.Source: §3 · p. 3
- The privacy goal is to prevent unauthorized reconstruction or exfiltration, while allowing some task-necessary semantic disclosure from deployed knowledge bases.The paper notes an inherent privacy-utility tradeoff and cautions against deploying highly confidential assets in RAG systems.Source: §3 · p. 3
- CanaryRAG uses complementary target and oracle expectations: benign target output should suppress canaries, while the oracle probe should expose them.The figure describes a violation in either path as evidence of an adaptive knowledge-base extraction attack.Source: Figure 1 · p. 4
- Runtime confidentiality detection is difficult because adaptive attackers can obfuscate or distribute content across outputs, and leakage may become evident only after substantial corpus exposure.The paper therefore treats runtime integrity violations as a practical proxy for mitigating knowledge-base leakage.Source: §3 · p. 4
Threat Model
- The threat model gives a black-box adversary a query budget and multi-turn interaction, but no direct access to the knowledge base, retriever, or system prompts.The adversary seeks high-fidelity reconstruction and may iteratively refine queries; adaptive adversaries may know the canary tokens.Source: Threat Model · p. 4
- The defender can insert auxiliary markers into the final model input, monitor streaming outputs, and halt generation, but cannot access internal pipeline structures or the knowledge base.The defense is intended as a drop-in mechanism that preserves benign interaction behavior and utility.Source: Threat Model · pp. 4–5
- Direct compromises of the knowledge base, retriever, or infrastructure, as well as side-channel attacks, are outside the considered threat model.Source: Threat Model · p. 5
4 Methodology
CanaryRAG replaces unreliable content-based leakage judgments with runtime integrity checks based on canaries embedded in retrieved chunks. It checks streamed output on a target path and uses a concurrent oracle path to detect adaptive suppression.
4 Methodology
- CanaryRAG defines extraction detection as a runtime integrity problem because legitimate responses use retrieved content and content-based detection cannot reliably separate use from leakage.Canary signals provide a machine-verifiable boundary, and deviations from the expected behavior are treated as evidence of an extraction attack.Source: §4 · p. 5
4.1 Canary Injection in RAG Application
- CanaryRAG inserts non-semantic canary strings into retrieved chunks before the augmented context is supplied to the target model.The strings are chosen not to occur naturally in benign text and are unlikely to appear unless the augmented context is reproduced.Source: §4.1 · p. 5
4.2 Streaming Canary Exposure Detection
- During streaming, CanaryRAG checks sliding output windows for canary exposure and treats any canary emitted on the benign target path as an integrity violation.The windowing accommodates canary strings that may span multiple output tokens; detected exposure can halt streaming before the buffer is released.Source: §4.2 · p. 6
4.3 Dual-Stream Integrity Probing
- The concurrent oracle path probes for canary exposure while the target path should suppress it, so deviations in either path signal extraction or adaptive suppression.The oracle uses a minimum recovery threshold because requiring every canary would be overly strict under generation stochasticity.Source: §4 · p. 5, §4.3 · p. 6
5 Experiments
The experiments evaluate CanaryRAG across representative RAG agents, extraction attacks, benign-user outcomes, runtime latency, and adaptive evasion. Across these settings, CanaryRAG reduces chunk recovery while maintaining benign utility and robust detection.
5.2 RQ1: Effectiveness in Defense
- 0.04× relative mean CRR (about 96% average reduction) versus No Defense at 1.00×; CanaryRAG has the lowest CRR across three agents and two attacks.The reported result is a worst case in which detected users are allowed to continue attacking.Source: Table 1 · p. 7, §5.2 · p. 8
5.1 Experiments Setups
- The evaluation uses three representative agents, Pirates and RAG-Thief with up to 1,500 attempts, and measures chunk recovery, benign quality and FPR, latency, and adaptive-detection TPR.A chunk counts as recovered only when ROUGE-L exceeds 0.5 and embedding cosine similarity exceeds 0.85.Source: §5.1 · pp. 7–8
5.3 RQ2: Impact on User Experience
- Across benign-query settings, CanaryRAG has negligible impact on response quality and an extremely low false positive rate.The evaluation reports response quality using BERTScore and false alarms using FPR.Source: §5.3 · p. 8, Table 2 · p. 9
5.4 RQ3: Runtime Response Latency
- CanaryRAG's latency distributions closely match No Defense, indicating negligible practical runtime overhead.Figure 2 compares the latency distributions with and without the defense.Source: §5.4 · p. 8, Figure 2 · p. 8
5.5 RQ4: Robustness under Adaptive Attacks
- Across suppression, obfuscation, and semantic-transformation attacks, CanaryRAG with oracle maintains high TPR, while removing the oracle substantially degrades adaptive robustness.Table 3 includes canary suppression and output obfuscation combined with deferred task injection.Source: §5.5 · pp. 8–9, Table 3 · p. 9
6 Conclusion
The paper introduces CanaryRAG, a dual-path runtime detection defense for RAG knowledge-base leakage. Experiments report lower extraction success than prior defenses with negligible effects on task performance and latency.
6 Conclusion
- CanaryRAG embeds canary tokens in retrieved documents and uses a dual-path runtime integrity game to detect leakage and adaptive evasion.The paper describes the approach as plug-and-play, requiring no retraining or architectural changes.Source: §6 · p. 9
- Across multiple agents and attack settings, CanaryRAG substantially reduces knowledge-base extraction success compared with prior defenses.Source: §6 · p. 9
- The reported protection has negligible impact on task performance and inference latency, and the method requires no retraining or architectural changes.Source: §6 · p. 9
Limitations
The stated scope is large-scale reconstruction of RAG knowledge bases through extraction attacks, not all RAG privacy threats or individual-fact privacy.
Limitations
- The work focuses on extraction attacks that seek to reconstruct the knowledge base through system interactions, excluding threats such as membership inference and side channels.Source: Limitations · p. 9
- The stated objective is large-scale knowledge-base reconstruction, not a guarantee of privacy for any individual fact, record, or attribute.Source: Limitations · p. 9
- Fine-grained targeted fact extraction is described as extremely inefficient for large-scale reconstruction and as fundamentally different from high-fidelity recovery attacks.Source: Limitations · p. 9
Ethics Statement
The ethics statement frames CanaryRAG as protection against unauthorized knowledge-base extraction. The adaptive-attack discussion tests bypasses of both the target and oracle paths, including deferred extraction instructions.
Ethics Statement
- CanaryRAG is intended to detect and mitigate unauthorized knowledge-base extraction, not introduce new misuse capabilities.The authors frame its contribution as protecting proprietary knowledge and reducing confidentiality risks in deployed RAG systems.Source: Ethics Statement · p. 9
A The Details of Adaptive Attack Tailored to CanaryRAG
- Target-path bypasses suppress canary tokens or obfuscate outputs while still attempting to extract protected context.Obfuscation examples include character separation, reversed ordering, and encoding; suppression directly instructs the model not to emit canary-related tokens.Source: A The Details of Adaptive Attack… · pp. 10–11
- Either target-path bypass alone can still trigger detection when it disrupts the oracle path’s expected canary emission.The oracle path explicitly probes for canaries, so suppression or obfuscation that interferes with its output violates expected behavior.Source: A The Details of Adaptive Attack… · pp. 10–11
- Deferred task injection preserves the oracle’s initial task compliance, then places the malicious extraction instruction at the output’s end.Evaluation on ChatDoctor measures detection TPR for non-adaptive attacks and two adaptive compositions, and compares against a variant without the oracle path.Source: A The Details of Adaptive Attack… · p. 11
B.1 Potential applications in real-world RAG Systems
CanaryRAG detects extraction under both standard and adaptive attacks while keeping benign false positives low. A window-based blocking policy can further tune deployment risk, including under elevated per-query false-positive assumptions.
B.1 Potential applications in real-world RAG Systems
- 99.2% TPR under standard extraction and 94% under strengthened adaptive attacks accompany a worst-case FPR of 0.15% across evaluated agents.The reported FPR is effectively near zero across agents.Source: §B.1 · p. 11
- For practical window sizes and thresholds k = 3–10, the upper bound on window-level false blocking often falls below 10^-6.Figure 3 plots the binomial upper bound using empirically measured, agent-specific per-query FPRs.Source: Figure 3 · p. 12, §B.1 · p. 12
- The elevated-FPR analysis considers distribution shift that could make real-world per-query p higher than observed in controlled evaluation.The authors assess whether window-based blocking remains viable under that concern and conclude it remains practical under conservative assumptions.Source: §B.1 · pp. 12–13
- Increasing threshold k can drive window-level false blocking to negligible levels even when per-query FPR p rises from 0.1% to 1% or 5%.The policy parameters W and k can be tuned after deployment without modifying the detection mechanism.Source: §B.1 · p. 12
- Because extraction requires repeated probes, accumulated integrity violations can trigger blocking and reduce the deployed leakage surface beyond CRR estimates.The reported CRR conservatively assumes attackers may continue querying without restriction after detection, with leakage aggregated over the attack horizon.Source: §B.1 · pp. 11–13
B.2 Discussion on Computational Overhead
CanaryRAG uses a lightweight oracle model and verifies once per query, limiting computational cost as retrieval depth grows. The authors report the strongest empirical protection with the lowest effective computational overhead among evaluated defenses.
B.2 Discussion on Computational Overhead
- CanaryRAG performs oracle verification once per query, while summarization-based defenses operate per chunk and incur approximately k-fold higher FLOPs.The computational gap becomes more pronounced as retrieval depth grows.Source: §B.2 · p. 13
- CanaryRAG avoids RAGFort’s per-agent training and repeated embedding recomputation, and its inference cost remains stable regardless of generator size.Its lightweight oracle model’s cost is decoupled from the main generator and is approximately constant across deployments.Source: §B.2 · p. 13
- CanaryRAG combines the strongest empirical protection with the lowest effective computational overhead among evaluated defenses and negligible additional latency.Reranking and summarization add pre-generation stages, while RAGFort’s cascade decoding can nearly double latency.Source: §B.2 · p. 13
B.3 Summarization and Semantic Paraphrasing Adversaries
Summarization and paraphrasing are treated as auxiliary rather than primary extraction strategies because they lose fine-grained information; the evaluation nevertheless includes them. The paper identifies combining CanaryRAG with other defense layers as future work.
B.3 Summarization and Semantic Paraphrasing Adversaries
- Summaries and paraphrases lose fine-grained structure, ordering, and technical detail, making them poor substitutes for document- or corpus-level reconstruction.The paper frames them as misaligned with the extraction goal, not as defense-specific failures.Source: §B.3 · p. 13
- A valid target chunk must exceed ROUGE-L 0.5 and embedding cosine similarity 0.85 against the original chunk.Source: §B.3 · p. 13
- Summarization and paraphrasing are treated as auxiliary cold-start strategies, and the adaptive evaluation includes them alongside obfuscation instructions.The protocol incorporates initialization because strong initialization can improve downstream Chunk Recovery Rate.Source: §B.3 · p. 14
B.4 Future Work
- Systematically combining CanaryRAG with retrieval-time, decoding-time, and post-generation defenses is left for future research.The paper describes these defenses as complementary components of a possible defense-in-depth architecture.Source: §B.4 · p. 14
C The Philosophy of Canary Design in LLM
CanaryRAG uses semantically irrelevant, high-entropy tokens as runtime integrity markers: benign generation generally ignores them, while faithful extraction exposes them. The paper reports a favorable privacy-utility trade-off and describes design choices for placement and oracle validation.
C The Philosophy of Canary Design in LLM
- High-entropy random strings are ignored in benign generation yet difficult to suppress when extraction forces faithful context reproduction, making them lightweight integrity markers.They need not be secret, document-specific, or semantically meaningful.Source: C The Philosophy of Canary Design in LLM · pp. 14–15
C.1 Canary Placement Strategies
- Intra-chunk placement couples canary exposure more closely to content reproduction; sentence-begin placement is the default because it triggers exposure earlier than sentence-end placement.The two sentence-level strategies showed no significant difference in detection effectiveness.Source: §C.1 · p. 15, Table 6 · p. 15
D Oracle Design Details
- The oracle validates canary exposure using one randomly selected chunk from the top-k results, rather than reproducing the full retrieval context.Its role is to test whether the model can expose canaries when the target path may suppress them.Source: D Oracle Design Details · p. 15
E Privacy-Utility Trade-off
- CanaryRAG maintains near-optimal utility comparable to No Defense and significantly better than Summarize and RAGFort, achieving Pareto optimality across the baselines.Reranker provides negligible privacy improvement despite slightly improving utility.Source: E Privacy-Utility Trade-off · p. 16, Table 7 · p. 16