PaperlayerView exact revision on arXiv

Source-linked AI summary

Rent-a-RAG: Embedding-Space Watermarks for Auditing Third-Party RAG

Alexandr Goultiaev Tolstokorov, Kyriakos Mouratidis, Javad Dogani, Nikolaos Laoutaris

arXiv:2609.03749v1cs.CRcs.CL

TL;DR

Third-party RAG providers lack visibility into whether non-cooperative operators reuse licensed documents, especially when answers are paraphrased and mix multiple sources. The paper proposes DirBucket, which embeds provider-keyed semantic signals through meaning-preserving paraphrases and audits black-box answers using directional evidence. DirBucket detects target reuse without non-target activation, survives evaluated laundering strategies, and transfers to real clinical, cyber-threat-intelligence, and legal corpora.

  • Problem

    Third-party RAG providers need auditable evidence of document reuse because operators can hide caching, paraphrasing, and multi-provider mixing behind black-box interfaces.

  • Method

    DirBucket watermarks provider documents with meaning-preserving paraphrases whose embeddings align with provider- and bucket-specific secret directions, then audits black-box answers.

  • Results

    DirBucket consistently achieves strong target detection with no non-target activation, reaches the reporting threshold by 23 audited answers, and transfers unchanged to real-domain corpora.

  • Takeaways & Limitations

    Embedding-space watermarking with alignment-aware attribution can make document reuse in third-party RAG statistically auditable.

  • Takeaways & Limitations

    DirBucket requires proactively modifiable source documents, making it unsuitable for immutable or wording-sensitive materials such as statutes, filings, quotations, and safety-critical instructions.

Abstract

from arXiv · show

Third-party retrieval-augmented generation (RAG) marketplaces create a new auditing problem: data providers may license corpora to a RAG operator, yet later have no visibility into whether their documents are being reused without compensation. Auditing this misuse is difficult because the operator is non-cooperative, answers are paraphrased by the generator, and one response may combine evidence from many providers. We propose DirBucket, a provider-side semantic watermarking and black-box auditing framework for document-level reuse in multi-provider RAG. DirBucket watermarks documents by meaning-preserving paraphrases whose embeddings are biased toward provider-bucket secret directions, enabling detection from black-box answers while preserving retrieval utility. On a challenging benchmark that reflects mixed-provider reuse under black-box access, DirBucket is the only method that consistently achieves strong target detection with no non-target activation, detecting non-compliance in every audit within 23 audited answers on our primary benchmark. The watermark survives adversarial post-answer laundering, and none of the evaluated evasion strategies simultaneously defeats detection while preserving user-perceived answer quality. Detection transfers unchanged to a second benchmark built from real clinical, cyber-threat-intelligence, and legal provider corpora. These results suggest that embedding-space watermarking can make document reuse in third-party RAG statistically auditable.

1 Introduction

Third-party RAG marketplaces separate corpus providers from operators, leaving providers unable to verify reuse or compensation. DirBucket addresses this with provider-side semantic watermarking and black-box auditing that remains effective under paraphrasing and provider mixing.

  • Marketplace setting: Providers may monetize specialized corpora through RAG operators that handle retrieval, orchestration, model serving, and user-facing applications.
  • Motivation: DirBucket targets document reuse when non-cooperative RAG operators hide caching, queries, retrieval, and source mixing from providers.The audit must produce evidence from queries and final answers without operator self-reports or retrieval instrumentation.
  • Research question: The central question is whether keyed document watermarks can reveal reuse from black-box answers after paraphrasing and cross-provider mixing while preserving utility.
  • Contributions: The paper contributes a threat model, DirBucket, directional attribution and detection, and evaluations on synthetic mixed-provider and real-domain corpora.
  • Headline result: DirBucket retains detection under paraphrasing, stylistic variation, adversarial laundering, and mixing, reaching the reporting threshold by 23 audited answers without non-target activation.

2 Related Work

Prior approaches address provenance, semantic watermarking, or system copying, but the paper focuses on document-level reuse attribution in mixed-provider RAG under black-box access.

  • LLM output watermarking: Generation-time output watermarks assume control over the generator or decoding policy, which providers lack in third-party RAG deployments.
  • RAG provenance auditing: WARD tests dataset presence in a target RAG corpus, whereas DirBucket targets finer-grained document reuse attribution when answers mix multiple providers.
  • Semantic watermarking: SemStamp provides paraphrase-robust semantic watermarking through preferred embedding regions, but it was not designed for black-box multi-provider RAG auditing.
  • RAG watermarking: RAG-WM protects against whole-system copying with intentionally injected watermark text, making it complementary rather than directly comparable to DirBucket.
  • Non-watermarking audits: Non-watermarking methods such as canaries, lexical overlap, similarity, and membership auditing face mismatches with opportunistic natural-document reuse or mixed-provider attribution.

3 Preliminaries and Approach Outline

The framework models a multi-provider RAG operator as a black box and audits final answers by aligning answer windows to provider references, assigning attribution, and pooling directional evidence.

  • Threat model: The threat model allows operators to cache, clone, paraphrase, summarize, reorder, or post-process reused content without cooperation.
  • RAG model: The operator retrieves k documents and generates answers through an unobserved retrieval and generation pipeline, while the auditor sees only queries and final answers.
  • Reference sets: The auditor constructs per-answer reference sets because the true retrieval bundle is hidden, then aligns answer spans to provider-specific reference windows.
  • Attribution: Competitive alignment assigns each answer window to at most one provider only when its match is strong and clearly ahead of the runner-up; otherwise the window is discarded.
  • Statistical testing: Provider evidence is pooled across aligned windows and audited answers, with a minimum-support gate and Benjamini–Hochberg false-discovery-rate control across providers.

4 Directional Bucket Watermark

DirBucket embeds provider-keyed directional signals through meaning-preserving sentence paraphrases and detects them by scoring aligned answer windows against matched reference buckets.

  • Watermark embedding: DirBucket paraphrases each sentence through hill-climbing, filtering candidates for semantic preservation and accepting the highest-gain candidate along a provider-secret bucket direction.
  • Directional signal: Each provider and bucket deterministically define a secret unit direction, while the directional score measures embedding alignment with that direction.
  • Null model: Under the null, directional scores are centered near 0 with variance approximately 1/d, providing the variance model for pooled testing.
  • Reference-bucket detection: The detector anchors each answer window to the bucket of its matched reference window, improving robustness when paraphrase drift changes the answer’s bucket.
  • Audit decision: Pooled anchored scores yield provider-level statistics and p-values, while support gating and BH/FDR testing determine detection across repeated audits.

5 Experimental Evaluation

Across black-box, mixed-provider audits, DirBucket combines strong target detection with zero non-target activation, remains robust to answer laundering, and preserves utility across synthetic and real-domain benchmarks.

  • 5.2 Baseline Frontier in the Primary Setting: DirBucket was the only compared method combining target pooled Z above 42 with near-zero max non-target pooled Z and near-certain detection by about 21 answers.KGW-adapted and SemStamp-adapted baselines did not maintain the required power–specificity combination under the same interface.
  • 5.3 Null Calibration and Key Dependence: Correct-key targets produced strong positive evidence, whereas wrong-key targets and unrelated-provider nulls stayed near zero; Tmin=25 reduced null false-positive rate to 0.At Tmin=10, the null false-positive rate rose to 0.61 because small support counts were consistent with chance fluctuations.
  • 5.5 Robustness to Answer Variation: Across natural and adversarial rewriting, max non-target Z remained 0, while aggressive paraphrase retained 30/30 detection at Z=6.98.The closest-to-threshold conditions were eli5 and 25% compression, where shorter answers starved the audit of aligned windows rather than erasing the signal.
  • 5.5 Robustness to Answer Variation: No tested laundering strategy both reliably evaded detection and preserved user-perceived quality: 25% compression yielded completeness 4.00/5 and 71% preference for the original.This was the only condition that partially evaded detection, but it degraded answer quality.
  • 5.7 Real-Domain Transfer and Utility: Detection generalized to clinical, cyber-threat-intelligence, and legal corpora, while watermarking caused only small retrieval changes: hit@5 fell from 0.973 to 0.962 and MRR@5 from 0.839 to 0.819.The legal corpus produced the least evidence per answer but was still detected in every run; watermarked answers were at least as well supported as source-corpus answers.

6 Conclusion

DirBucket is a provider-side semantic watermarking and black-box auditing framework for detecting document reuse in third-party RAG. Across challenging mixed-provider and real-domain evaluations, it achieved strong target detection without non-target activation and remained robust to laundering with negligible utility loss.

  • 6 Conclusion: DirBucket makes document reuse in third-party RAG statistically auditable under black-box, mixed-provider conditions.The framework combines provider-side semantic watermarking with black-box auditing.
  • 6 Conclusion: Experiments showed strong target detection, no non-target activation, rapid evidence accumulation, laundering robustness, negligible utility loss, and unchanged transfer to real-domain providers.The real-domain benchmark covered clinical, threat-intelligence, and legal corpora.

Limitations

DirBucket is limited to editable textual corpora and shifts rather than eliminates trust assumptions in the audit process.

  • Meaning-preserving paraphrase makes DirBucket unsuitable for immutable or wording-sensitive documents where exact wording carries normative or evidentiary force.The evaluated legal corpus used rewritable editorial case summaries; statutory and filing text was excluded.
  • The method does not directly address multimodal knowledge-base content or source code, where paraphrasing may alter behavior.
  • Auditing still assumes an authorized auditor securely holds provider keys, accesses corpora, and executes audits honestly.The framework removes reliance on operator self-reports but does not study auditor independence, key custody, or dispute resolution.

Ethics Statement

The work frames DirBucket as a mechanism for verifying licensing compliance, while its broader use should remain limited to owned corpora and human-reviewed statistical evidence.

  • DirBucket is intended to help data providers verify compliance with licensing agreements.
  • The mechanism should be applied only to corpora the provider owns, because it could otherwise trace content the provider did not author.
  • Audit outcomes should be treated as statistical evidence for human decision-making rather than automatic proof of infringement.
  • The human evaluation used unpaid volunteer annotators, which the authors identify as a recruitment limitation.

A.2 Watermark Cost–Strength Ablation

DirBucket’s cost–strength sweep shows that fewer hill-climbing rounds sharply reduce cost while retaining detection power and high semantic similarity. The aligned black-box audit aggregates provider-specific directional evidence after answer paraphrasing and attribution.

  • Embedding procedure: The provider embeds watermarks by generating paraphrase candidates, selecting the candidate with the strongest keyed directional score, and accepting it when the gain exceeds δmin.The procedure repeats this process across sentences and hill-climbing rounds before reconstructing the watermarked document.
  • Cost–strength results: 49% gain and target Z=5.85 are achieved by the cheapest configuration, K=6 and ρ=1, while remaining above the Z=4 reporting reference.This setting costs roughly 6× less than the primary configuration.
  • Cost–strength results: Halving rounds from ρ=6 to ρ=3 retains 90% of the directional gain at K=24, making K=24 and ρ=3 a practical cost-sensitive default.The sweep evaluates K ∈ {6, 12, 24} and ρ ∈ {1, 3, 6}.
  • Detection procedure: During auditing, answer windows are competitively aligned to provider reference windows, scored along the matched bucket’s keyed direction, and pooled into per-provider Zdb statistics.The non-oracle procedure then applies support gating and BH/FDR testing to accumulated evidence.
  • Utility: Semantic similarity remains uniformly high at 0.89–0.92 across configurations, indicating that stronger watermarking preserves source meaning.The reported similarity covers all nine K–ρ combinations.

B.4 Direction Scheme Ablation: Naïve vs. Orthogonalized Directions

The default naïve direction scheme outperforms explicit cross-provider orthogonalization because orthogonalization offers little null-side benefit while reducing recovered target evidence. Sensitivity results identify stable operating regions governed by alignment quality, support, and window formation.

  • Direction scheme ablation: Target pooled Z falls from 8.98 to 3.98 for qa, 5.47 to 2.90 for ownwords, and 8.47 to 6.03 for bullets under orthogonalized directions.Correct-key evidence also decreases from 8.03 to 5.80, while unrelated non-target mean pooled Z changes only from 0.152 to 0.148.
  • Direction scheme ablation: Explicit orthogonalization provides limited additional calibration benefit but costs substantial detection power, so the naïve scheme is used in the main experiments.Independent random directions are already close to orthogonal in the m=6, d=768 setup.
  • Hyperparameter sensitivity: The stable τsim region spans 0.78–0.90, with 0.82 near its center; lower values increase false positives, whereas higher values cause evidence starvation.The stable regime requires target detection rate 1.00 and FPR 0.
  • Hyperparameter sensitivity: At τmargin=0.05, the margin widens the τsim stability plateau by approximately 0.04 on each side by discarding ambiguous near-tie assignments.At τmargin=0, τsim values of 0.78 and 0.80 yield FPRs of 0.28 and 0.06, respectively; both fall to zero at τmargin=0.05.
  • Hyperparameter sensitivity: The support gate Tmin=25 eliminates false positives while preserving eligibility, whereas Tmin=10 yields FPR=0.50 and Tmin=50 delays median detection.The cross-benchmark calibration evaluates the complete support-gate and BH decision rule.
  • Hyperparameter sensitivity: Short windows reduce detection to 0.67 with target Z=2.62, while windows beyond roughly 40–80 words risk support starvation because of sentence boundaries.Shorter windows increase embedding variance and bucket flips, turning directional evidence into noise.

B.6 Laundering Utility Evaluation

The laundering evaluation tests whether post-answer transformations can evade DirBucket without harming answer quality. Stronger compression weakens detection only alongside measurable losses in completeness and user preference.

  • Detection under laundering: DirBucket catches 50% compression in all 30/30 trials, while 25% compression partially evades detection in 24/30 trials.The evaluation compares original, neutral, aggressive, compress50, and compress25 conditions.
  • User-perceived quality: At 25% compression, completeness falls to 4.00/5 and 71% of judgments prefer the original answer, despite accuracy remaining high.Accuracy spans 4.85–4.97 across laundering conditions because it measures correctness rather than preservation of all relevant facts.

B.7 Model Robustness

DirBucket remains effective across alternative embedding encoders and answer generators, while non-watermark heuristics and competing watermark baselines fail to maintain calibrated, specific attribution under mixed-provider conditions.

  • Model robustness: 30/30 detection with FPR=0 holds for both alternative embedding encoders and both alternative answer-generation models.MiniLM-L6-v2 achieves Z=6.67 and GTR-T5-base Z=8.21; GPT-3.5-turbo reaches Z=7.27 and Qwen2.5-1.5B Z=8.36.
  • Model robustness: 98.3% of the API model’s mean directional gain is achieved by a locally served open-weight paraphraser at matching semantic fidelity.The comparison uses the same 50 documents and settings, supporting near-parity in audit power.
  • Non-watermark heuristic calibration: At approximately 5% null FPR, semantic similarity and lexical overlap reach only 0.13 and 0.01 TPR, while entailment reaches 0.72 after threshold tuning.The heuristics appear useful at uncalibrated operating points but perform poorly under the target-absent null.
  • Additional watermark baseline diagnostics: 8.23 target pooled Z with 0 max non-target pooled Z in cooperative conditions and 7.78 with 0 under style variation distinguish DirBucket from weaker baselines.KGW-adapted and SemStamp-adapted lose power or specificity in the corresponding diagnostics.
  • Additional watermark baseline diagnostics: 12.07 target pooled Z with 0 max non-target pooled Z is maintained in the oracle-aligned mixed-provider diagnostic, unlike competing watermark baselines.KGW-adapted reaches 13.53 target evidence but also 5.95 non-target evidence, while SemStamp-adapted remains low-power.
  • Empirical null diagnostics and key dependence: 0/120 BH rejections occur for both wrong-key and unrelated-provider null audits under the exact deployed decision rule.Wrong-key support remains high, and full trajectories become BH-significant in only 3/120 cases.

B.11 Robustness to Marketplace Size and Provider Mixing

DirBucket remains robust as provider mixing and marketplace size increase, while preserving utility through substantial paraphrastic rewriting and maintaining low non-target activation.

  • Marketplace size: The controlled-exposure harness holds target exposure constant while non-target providers compete naturally for remaining retrieval slots as marketplace size varies.This isolates detection behavior from changes in retrieval coverage across provider counts.
  • Provider mixing: Median target pooled Z rises from 7.73 at mixing level 3 to 8.50 at level 6, while non-target activation stays below the 5% reference.The benchmark uses the standard m=6 configuration and evaluates realized per-query mixing levels.
  • Marketplace size: 0 max non-target pooled Z analytically keeps the BH-adjusted target p-value below α=0.05 against 999 inactive hypotheses, although empirical large-marketplace evaluation remains necessary.The extension bounds when multiple-testing burden, rather than DirBucket itself, becomes binding.
  • Operating point: Tmin=25 works for all evaluated marketplace sizes, but natural workloads require calibration to expected per-bundle target exposure.Lower exposure can be offset by reducing Tmin proportionally or increasing the number of audited answers.
  • Utility and rewrite profile: The source and watermarked snippets show approximately 0.24 lexical overlap while preserving topical content and the core factual narrative.This qualitative example complements the broader utility results.
  • Utility and rewrite profile: Mean changed fraction 0.978 coexists with high answer-document support, while source-to-watermarked numeric recall remains 0.898 despite lower sentence coverage of 0.622.These diagnostics indicate substantial, compressive rewriting rather than near-copy edits.

B.13 LLM Judge Evaluation Protocol

The evaluation compares source and watermarked text with blinded automated and human judgments, finding high meaning preservation and nearly unchanged answer quality despite compressive paraphrasing.

  • Sentence-level evaluation: Meaning scores 4.86/5, naturalness 4.96/5, factual accuracy 97%, and no important omissions 94% across 1,000 sentence pairs.The 6% omission rate is associated with the compressive paraphrasing pattern.
  • Answer-level evaluation: The answer accuracy gap is Δ=−0.04, with watermarked answers scoring 4.40 versus 4.44 for source answers and no statistically distinguishable difference.Preference is nearly balanced across 1,000 paired questions.
  • Cross-family judge: An unrelated-family judge replicates high sentence quality, scoring meaning 4.69/5 and naturalness 4.95/5.Agreement with GPT-4o is 78% exact match for meaning and 93.5% for naturalness.
  • Human validation: Human ratings find source and watermarked sentences equally natural, with high shared-information ratings and correlations with automated judgments.Correlations are ρ=0.24 for shared information and ρ=0.46 for answer-quality gap.
  • External validity: On the real-domain benchmark, detection is 30/30 with zero non-target activation for same-domain siblings and across-domain providers.The benchmark covers clinical abstracts, cyber-threat-intelligence reports, and legal case summaries, with domain-dependent evidence density.
  • Coverage robustness: At 53% watermark coverage, detection remains 29/30, falling to 19/30 at 30% coverage without any false attribution.Coverage reduces evidence accumulation while preserving the reported false-positive behavior.
Loading 2609.03749v1…