Protection Is Not One Event: A Survey of RAG Protection Research by Measured Endpoint

• Thirty-two annotators double-coded 118 full-text-confirmed papers using a frozen seven-family endpoint taxonomy, reaching κ = 0.867 agreement.Source: §1 · pp. 1–2
• Only 21 of 118 papers (17.8%) primarily measure emitted or reconstructed protected content; among copyright/ownership records, 73.3% primarily detect corpus use and none measure emitted expression.Source: §1 · p. 2
• Retrieval reranking and context rewriting span five or six endpoint families, so the defense mechanism alone does not identify the protection outcome tested.Source: §1 · p. 2
• Most studies use standard question-answering datasets with little sensitive content and rarely test adaptive attackers who ask follow-up questions.Source: §1 · p. 2
• The released contributions include an item-level evidence map and open-source verification tools, plus a controlled testbed examining copying versus fact recovery.Source: §1 · p. 2

• Prior RAG reviews group studies by threat model, pipeline stage, or defensive module, while broader surveys also organize around architecture and attack surfaces.Source: §2 · p. 2
• An architectural view shows where a defense intervenes but obscures what its evaluation actually tests.Source: §2 · p. 2
• Empirical studies define protection through varied outcomes, including text reproduction, activation steering, proprietary knowledge-base defense, and knowledge extraction benchmarks.Source: §2 · p. 2

3 The Endpoint Taxonomy and Human-Coded Evidence Map

• The taxonomy distinguishes technical events such as verbatim copying, identifier leakage, source reconstruction, privacy proxies, access suppression, corpus-use detection, and pipeline integrity.Source: §3 · p. 3

3.1 A Seven-Family Taxonomy of Measured Endpoints

• A study is assigned by the exact empirical quantity its primary metric evaluates, not by where its defense sits in the RAG pipeline.Source: §3.1 · p. 3
• An unlearning method measured only by refusals is classified as access suppression, while reconstruction from residual index states is source recovery.Source: §3.1 · p. 3
• Families A/A-pii/B/C cover wording, identifiers, source recovery, or formal proxies; D covers suppression, E corpus-use detection, F integrity and attribution, and G underdetermined cases.Source: §3.1 · pp. 3–4
• Thirty-two annotators double-coded the studies and reached κ = 0.867 agreement on primary endpoint family; fixed category boundaries support reproducible comparisons.Source: §3.1 · p. 4

3.2 Literature Search, Human Coding, and Corpus Scope

• Searches across arXiv, OpenAlex, and ACL Anthology returned 11,972 hits, which deduplicated to 8,978 records before screening and full-text recovery.Source: §3.2 · p. 4
• The core map includes studies evaluating RAG systems with interventions and source-verified protection or attribution metrics; each receives one primary endpoint, while mechanisms may receive multiple labels.Source: §3.2 · p. 4
• The map contains 118 confirmed records, and reported proportions describe that analyzed pool rather than absolute field totals; an audit estimates roughly 27.2 additional confirmable studies.Source: §3.2 · p. 4

4.1 Emitted Text and Identifiers (Families A and A-pii)

• Families A and A-pii directly score emitted wording or personal identifiers, but account for only 14 records and face difficult leakage–utility trade-offs.Adaptive attackers recover protected information in 38% of Family A and 33% of Family A-pii records; interventions can reduce leakage while harming task utility, and residual leakage persists under some settings.Source: §4.1 · p. 5

4.2 Source Reconstruction Outside Generation (Family B)

• Family B covers non-generation reconstruction through embeddings, index structure or deletion residue, and parametric residue; its seven studies measure distinct targets that cannot share one scale.A near-zero score is meaningful only when the unprotected baseline attack works; encryption or hidden traversal paths do not establish protection against generated quotations.Source: §4.2 · pp. 5–6

4.3 Mathematical Bounds and Likelihood Proxies (Family C)

• Family C evaluates privacy through formal guarantees or likelihood proxies rather than emitted text, and its operating points depend on protected units, composition rules, and query workloads.These theoretical and statistical measures cannot be pooled with empirical leakage rates.Source: §4.3 · pp. 6–7

4.4 Access and Retrieval Suppression (Family D)

• Family D measures whether restricted documents are retrieved, routed, or answerable; early database-level authorization can prevent leakage, while inferred policies and imperfect metadata leave residual risks.Retrieve-then-filter exposes unauthorized context in 86.1% of base queries; row-level security or predicate pushdown eliminates cross-tenant retrieval leakage in the cited studies. Semantic filters can trail static filters or lose performance when labels or query details are imperfect.Source: §4.4 · p. 7

4.5 Post-Hoc Corpus-Use Detection (Family E)

• Family E asks whether an auditor can prove a corpus was indexed or queried, usually through planted signals, but usage detection does not measure or prevent content disclosure.Detection depends on calibration, query budgets, indexed material, and post-processing; the detector’s reliability must be distinguished from the secrecy of the text.Source: §4.5 · pp. 7–8

4.6 Pipeline Integrity, Attribution, and Recurring Failure Modes (Family F)

• Family F, the largest category with 47 records, tests pipeline integrity and attribution; adaptive attacks expose weaknesses, yet its records do not document attacker recovery of protected source content.Its attack-success measures concern wrong answers, hijacked instructions, or incorrect sources, so a zero score is compatible with verbatim quotation of private or copyrighted sources.Source: §4.6 · p. 8
• Across endpoint families, protection–utility trade-offs, fragile assumptions behind zero-leakage claims, and adaptive attackers limit how readily reported operating points transfer.The cited recurring patterns include reliance on clean permission metadata or explicitly named targets, and static-query results that can change under adaptive attacks.Source: §4.6 · pp. 8–9

5.1 Blind Spot I: Mechanisms Do Not Dictate Endpoints

• Retrieval reranking and context transformation span five and six endpoint families, respectively, while mechanism types predict measured endpoints only 53.4% of the time versus a 39.8% Family F baseline.Metric names are also ambiguous: task utility spans every family, and attack-success or integrity metrics occur across multiple families.Source: §5.1 · p. 9

5.2 Blind Spot II: High-Stakes Claims vs. Skipped Content Disclosure

• Only 21 of 118 studies (17.8%) use content disclosure or reconstruction as their primary endpoint; among copyright or ownership records, 73.3% prioritize usage detection and none prioritize emitted expression.Including formal privacy proxies still leaves 77.1% of records with primary endpoints outside the broader content-protection group.Source: §5.2 · pp. 9–10

5.3 Blind Spot III: Toy Benchmarks and Static Adversaries

• Evaluations rely heavily on general-purpose QA datasets and one-shot or non-interactive attackers, while only 11 studies model feedback-adaptive multi-query access.Although 108 of 118 records report improved protection, many studies lack uncertainty analysis or independent evaluation, so that success rate should not be read as proof that protection is solved.Source: §5.3 · p. 10

6.1 Stopping Surface Copying Does Not Stop Fact Recovery

• The testbed spans 192 documents, two domains and access policies, human-authored fact probes, and four controls evaluated across four language models.It uses fixed MiniLM top-3 retrieval and greedy decoding, and includes an online three-query adaptive controller.Source: §6.1 · p. 11
• 0.0% [0.0, 0.0] copy20 under the normalized 8-gram guard coexists with 48.3% [36.7, 60.8] hidden-fact recovery on Qwen3-8B under three adaptive queries.The attacker also links responses to the exact source document 42.5% of the time; on guarded Qwen3-8B, hidden-fact recall rises from 3.3% after one query to 48.3% on the third adaptive turn.Source: §6.1 · p. 11

6.2 Three Golden Rules for Future RAG Protection

• For secrecy, privacy, or copyright claims, the authors recommend directly measuring disclosure and evaluating on corpora containing sensitive or proprietary spans.They say operational metrics such as access suppression or watermark detection should be paired with a direct disclosure endpoint.Source: §6.2 · p. 12
• The authors recommend naming the target endpoint at subfamily resolution and testing multi-turn adaptive adversaries and imperfect metadata.Single-query evaluations can miss extraction vulnerabilities that emerge over three or more adaptive turns.Source: §6.2 · p. 12
• The authors call for validating evidential sufficiency, including checking that baseline attacks work and that utility metrics are not confounded by response length.Their taxonomy identifies the event measured, not whether a metric is sufficient or unbiased for that event.Source: §6.2 · p. 12

7 Conclusion

• The evidence map finds that mechanisms weakly predict measured outcomes, only 17.8% of studies prioritize content disclosure or reconstruction, and evaluations often use benign benchmarks and non-adaptive adversaries.The conclusion also reports that eliminating verbatim copying can leave source facts recoverable under multi-turn adaptive querying.Source: §7 · p. 12

A Event and Control Schema

• The coding schema separates technical events, protected assets, adversary access, intervention functions, primary endpoints, and evidence relations, assigning each paper one primary endpoint but allowing multiple mechanism labels.It distinguishes observed disclosure from theoretical privacy bounds and classifies claims as concordant, partial, or discordant against the primary endpoint.Source: A Event and Control Schema · p. 19

B.1 Literature Search and Eligibility Criteria

• The search protocol required a retrieval-augmented system, an intervention, a source-verified protection or attribution operating point, and a direct copyright/ownership or neighboring privacy/security goal.Review confirmed 112 studies from the candidate pool and the excluded-record audit added six, yielding 118 records.Source: §B.1 · p. 20

B.2 Human Annotation and Inter-Rater Reliability

• Thirty-two trained annotators double-coded recovered full texts, with disagreements referred to a third reviewer outside the initial pair.The annotators represented NLP and machine learning, security and privacy, and information science or law and technology.Source: §B.2 · p. 20
• All 32 annotators passed a 100-item calibration test at 95% or higher, scoring 95%–98% with a median of 97%.Calibration files and identity-linked records remain restricted under participant confidentiality terms.Source: §B.2 · p. 20
• The released matrix reproduces 89.83% pre-adjudication agreement (κ = 0.8672) on primary endpoint family, while median reliability across 14 categorical fields is κ = 0.87.The public artifact contains 8,024 de-identified rows covering 118 records and 34 coded fields.Source: §B.2 · p. 20

B.3 Copyright-Relation Sensitivity

• The paper verifies citations and operating points against source PDFs and preserves exact query strings and audit logs in the released artifact.The copyright/ownership relation is a technical coding relation rather than a legal classification.Source: §B.3 · p. 21
• Under the frozen copyright/ownership label, 11 of 15 records (73.3%) have a usage-detection primary endpoint, compared with 7 of 103 neighboring-goal records (6.8%).The reported Bonferroni-adjusted p-value is 4.9 × 10^-7 across 14 family comparisons.Source: Table 4 · p. 21, §B.3 · p. 21
• Zero records under the frozen copyright/ownership label have a strict emitted-expression primary endpoint in Family A, with a 95% exact upper bound of 18.1%.The table presents primary endpoints by evidence relation across the 118-record map.Source: Table 4 · p. 21, §B.3 · p. 21

C.1 Design and analysis hierarchy

• The fixed protocol, which was not preregistered, uses a 192-document security corpus across two domains and two policies, plus 64 document-disjoint open-policy utility items.MiniLM retrieval is fixed at top-3 and decoding is greedy.Source: §C.1 · p. 21
• The sole planned confirmatory contrast is Qwen3 Summary-minus-uncontrolled difference-in-differences of exact hidden-probe recall on all 80 held-out answer-only items.The three-query controller observes only released text, redaction, refusal, transport decodability, and released length.Source: §C.1 · p. 21
• The planned inference uses paired source-item bootstrapping and sign randomisation without multiplicity correction for this single contrast.Other models, policies, natural retrieval, shorter spans, and linkage analyses are secondary or exploratory.Source: §C.1 · p. 21

C.2 Natural-policy and adaptive evaluations

• The policy filter excludes inaccessible documents, whereas the normalised 8-gram guard suppresses protected source spans; denied quotation can still leave a question answerable.Authorized quotation is treated as desired release, and adaptive results distinguish raw outcomes from residual outcomes conditional on unguarded copying.Source: §C.2 · p. 22
• Fail-open missing or incorrect metadata admits 12.5% of access-denied targets.Table 7 separately reports guard-minus-unguarded side effects on the same response, excluding authorized target text from the guard set.Source: §C.2 · p. 22
• 48.3% hidden-fact recall remained for Qwen3-8B with a Normalized guard after three adaptive queries, despite 0.0% visible and decoded copy20.Summary + guard reduced hidden-fact recall to 39.2%; Policy + guard reduced it to 22.5%.Source: §C.2 · p. 23
• On natural-policy Qwen3-8B items, the Normalized guard reduced copy20 from 3.1% to 0.0%, while hidden-fact recall remained 12.4%.Policy + Summary + normalized guard reduced hidden-fact recall to 4.2%.Source: §C.2 · pp. 22–23

C.3 Human construct checks

• Human raters achieved 88.0% agreement for expression and 95.3% for recovery; automated source linkage is explicitly not a copying or recovery measure.The linkage calibration ranks Qwen3 answer-only responses against corpus and matched candidate pools.Source: §C.3 · p. 24, Figure 2 · p. 25

C.4 Primary contrast, utility, and sensitivity

• Qwen3-8B token F1 was 46.8 [36.7, 56.5] for Retrieved top-3 short answer, versus 18.9 [15.0, 23.0] for Retrieved top-3 free-form.For Qwen3-8B, Gold source only short answer reached token F1 60.9 [52.6, 68.8].Source: §C.4 · p. 26, Table 14 · p. 26
• For Retrieved top-3, short-answer token F1 was 39.2 for Qwen2.5-3B, 40.0 for Mistral-7B, and 40.3 for Phi-4-14B; free-form (frozen) token F1 was 11.3, 14.3, and 17.3, respectively.Source: §C.4 · p. 26
• The exploratory construct checks report context and answer-format effects across all 80 answer-only items, with long-answer recall using 57 eligible items and equal domain weights.The supplied passage defines these sensitivity-analysis scopes but does not provide their numerical outcomes.Source: §C.4 · p. 27, Table 15 · p. 27

C.5 Seven-query extension

• The exploratory extension fixes Summary, SystemBlock, and threshold configuration, then tests up to six transformations after a complete-source request.It uses a shared attacker for source intervention and control-specific development rankings for natural retrieval; stopped sessions retain their last released state.Source: §C.5 · p. 27
• Qwen3-8B Summary produced seven-query shared-attacker DiD values of −16.3 [−28.6, −4.0] and −36.2 [−48.8, −23.8] percentage points on two endpoints.Table 16 reports endpoint-specific results; these are not pooled across models or attacker designs.Source: Table 16 · p. 27, §C.5 · p. 27

D Survey Synthesis and Record-Level Operating Points

• The survey records 51 single-query, 31 fixed multi-query, 25 non-interactive, and 11 feedback-adaptive settings across 118 papers.Intervention stages and mechanisms are multi-label, so their counts need not sum to 118.Source: D Survey Synthesis and Record-Level… · p. 28, Figure 5 · p. 29
• Figure 6 reports lower, some, and high design-concern counts, with NR equal to zero throughout; these ratings are not efficacy weights.The survey presents this appraisal separately from measured defense outcomes.Source: D Survey Synthesis and Record-Level… · p. 28, Figure 6 · p. 29

Content disclosure and reconstruction

• Repeat-prompt and repeat-context measures count reproduced corpus text, while PII and paraphrase studies report targeted leakage or privacy-extraction outcomes.Examples include Repeat Prompt defined as at least 10 exact matching tokens [Zeng et al., 2025a] and ChatDoctor privacy leakage 0.23 -> 0.17 [Zhu et al., 2026].Source: §D.2 · pp. 29–30

Formal privacy proxies

• Formal privacy studies report guarantees or statistical proxies, including ε = 10 differential privacy and membership-inference accuracy 70.22 -> 53.60% at a 2.6% accuracy cost.The records pair privacy metrics with utility measures, including FID and retrieval or task performance.Source: §D.2 · p. 30

Access control and suppression

• Access-control and suppression examples range from 80% -> 0% cross-tenant leakage to 0.2% -> 0% tenant-isolation leakage over 1,000 queries.Other records measure compliance, unauthorized chunks, denied answers, or sensitive-query filtering rather than emitted-expression copying.Source: §D.2 · pp. 30–31

Corpus-use detection

• Corpus-use detection includes watermark verification, membership audits, and chunk recovery, with Xie et al. (2026b) reporting Chunk Recovery Rate 97.4 -> 1.6.Other reported endpoints include Shadow-RAG membership-audit AUC 0.989 and watermark verification measures.Source: §D.2 · p. 31

Pipeline integrity and attribution

• Pipeline-integrity studies assess poisoning and prompt-injection defenses with attack-success, detection, attribution, and accuracy measures; a certified runtime-memory-poisoning defense with provenance signing reduced ASR from 93% to 5.3% (n=150).Other reported measures include attack pass-through rate and 0% FPR on the benign subset, plus attribution precision/recall alongside ASR with and without defense.Source: §D.2 · pp. 31–32