Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documents

Contributions

• A single query-specific blocker document can be retrieved with clean documents and cause a RAG system to fail to answer, ostensibly because information is missing or unsafe.The attack assumes database insertion and editing access but does not require modifying other documents.Source: §1 · p. 1

Motivation

• Jamming is stealthier than incorrect answers because refusals appear plausible, are difficult to fact-check, and resemble routine LLM behavior.The paper describes suppression targets including bad reviews, customer feedback, legal records, and regulatory information.Source: §1 · p. 1

Methods

• The paper compares instruction-based, auxiliary-LLM, and black-box optimization methods for generating blocker documents.The black-box method is the paper’s key technical contribution.Source: §1 · pp. 1–2
• The black-box method uses query-only access, requires no knowledge of the target embedding or LLM, needs only control of the attacker’s document, avoids auxiliary LLMs, and avoids instruction injection.The method outperforms instruction injection in many settings and is therefore less affected by prompt-injection defenses.Source: §1 · pp. 1–2

Evaluation

• Experiments cover NQ and MS-MARCO, multiple embedding models and open-source LLMs, and transfer to larger and proprietary models.The method achieves comparable performance to adversarial input-generation techniques, while its black-box loss is competitive with, and sometimes superior to, white-box alternatives.Source: §1 · p. 2

Findings

• Existing LLM safety metrics do not measure jamming vulnerability, and higher safety scores correlate with higher vulnerability because jamming exploits refusals to unsafe queries.The paper also evaluates perplexity filtering, paraphrasing, larger contexts, and fine-tuning-based defenses.Source: §1 · p. 2

Prompt Injection

• Prompt injection manipulates third-party content or direct prompts to steer an LLM, whereas jamming induces familiar refusal responses rather than unsafe or attacker-chosen outputs.Jailbreaking specifically seeks to bypass safety guardrails, while jamming exploits refusal behavior.Source: §2 · p. 2

Retrieval Poisoning

• Earlier retrieval-poisoning attacks craft documents for retrieval but do not seek to influence the RAG generation component.The paper distinguishes retrieval manipulation from attacks that alter generated answers.Source: §2 · p. 2

RAG Poisoning

• PoisonedRAG adds multiple documents to induce adversary-chosen responses, whereas this paper targets denial of service with one blocker document.PoisonedRAG could be adapted for jamming by selecting refusal as the target response and adding one document.Source: §2 · pp. 2–3
• Chaudhari et al. [8] assume white-box knowledge of both the embedding model and LLM, while this paper does not require either or fixed predefined outputs.Their construction combines retrieval and output-optimization components with a direct instruction, and denial of service may rely only on that instruction.Source: §2 · p. 3
• Xue et al. [59] require multiple documents to dominate retrieval, whereas the proposed method uses a single automatically generated blocker instead of flooding the context.Their denial-of-service documents state that the context contains private information.Source: §2 · p. 3

System Components

• RAG consists of knowledge retrieval and answer-generation modules.Source: §3 · p. 3

Knowledge Retrieval

• The system embeds database documents, embeds the query, computes similarities, and selects the k highest-similarity documents.Query and document encoders may be separate, but the paper denotes them as one embedding model E.Source: §3 · p. 3

Answer Generation

• The generation module queries the LLM with the user query and retrieved documents under a predefined prompt structure.Source: §3 · p. 3

Attacker Objective

• The attacker aims to prevent the RAG system from answering selected queries in systems handling user-generated content.Examples include suppressing bad reviews, customer complaints, news articles, or criminal records.Source: §4 · p. 3
• Refusal is stealthier than an incorrect answer because it is not readily fact-checked and can appear as a normal lack-of-information or safety response.Source: §4 · p. 3

Attacker Capabilities

• The attacker may insert and repeatedly edit personal content but cannot remove or modify other database documents.The attack must therefore jam relevant documents that remain in the retrieved context.Source: §4 · p. 3
• The practical threat model limits the attack to one document because many fake user-generated documents may be detectable or avoidable.Multiple-document attacks are considered only in a limited evaluation.Source: §4 · p. 3
• The attacker has adaptive black-box access to query outputs but does not know the LLM, embedding model, retrieval count k, or output log probabilities.Existing adversarial-document methods often require at least white-box access to the target LLM’s logit layer.Source: §4 · pp. 3–4

Assumptions

• The attack assumes the victim’s exact query is known and the RAG configuration is relatively static, while adaptive systems are left for future work.Static configuration includes the underlying LLM, system prompt, and number of retrieved documents.Source: §4 · p. 4

5 Jamming Attacks Against RAG Systems

• A single blocker document is designed to be retrieved for target query Q and induce a refusal response instead of the clean answer.The poisoned response is optimized to resemble a chosen refusal target while differing from the unpoisoned response.Source: §5 · p. 4

5.1 Target Responses

• The attack targets three plausible refusal reasons: insufficient information, safety concerns, and correctness concerns.These responses are framed as common LLM behaviors and are neither correct nor incorrect answers.Source: §5.1 · p. 4

5.2 Constructing Blocker Documents

• Blocker documents concatenate a retrieval sub-document with a jamming sub-document, separating query matching from response induction.The retrieval component is responsible for being selected, while the jamming component is responsible for generating the desired answer.Source: §5.2 · p. 4

5.3 Retrieval

• Prepending the target query as the retrieval sub-document yields over 97% retrieval accuracy and no retrieval for unrelated queries.This construction pushes the full document’s embedding close to the query embedding.Source: §5.3 · p. 4

5.4 Jamming

• The jamming component is evaluated using instruction injection, oracle-generated text, and black-box optimization.Instruction injection explicitly requests the target refusal, whereas oracle generation asks GPT-4-Turbo to craft a context producing that refusal.Source: §5.4 · pp. 4–5
• Oracle-generated blockers use 30-word documents and up to 10 generation attempts, and they usually operate passively without explicit instructions.The generated document is evaluated by the oracle and regenerated when it fails to induce the desired output.Source: §5.4 · p. 5
• The new BBO method starts from token sequences and iteratively replaces tokens to maximize similarity between the poisoned response and the target refusal.It uses an auxiliary oracle embedding and similarity function while allowing both to differ from the target RAG’s embedding and similarity function.Source: §5.4 · p. 5
• BBO selects each token replacement by querying the poisoned RAG responses and retaining the candidate whose response is most similar to the target response.The search constructs candidate sets, obtains responses for each candidate, and chooses the maximum-similarity candidate.Source: §5.4 · p. 5

6.2 Retrieval

• Over 97% of blocker documents were retrieved in the top five, with top-1 retrieval occurring for 82% of NQ and 50% of MS-MARCO queries.The blocker was retrieved across datasets, embedding models, and target responses.Source: §6.2 · p. 6
• 0% of blocker documents were retrieved for non-target queries, indicating no measured collateral retrieval damage.The documents explicitly include their target queries, which prevents retrieval for other queries.Source: §6.2 · p. 6

6.3 Jamming

• Similarity to target or clean responses did not reliably distinguish jammed from non-jammed queries, despite Pearson correlations of −0.61 on NQ and −0.58 on MS-MARCO.A successful refusal could remain more similar to the clean answer than to the target response, as in a Goliath example with similarities 0.78 and 0.34.Source: §6.3 · pp. 6–7, Figure 2 · p. 7

6.4 Alternative Optimization Methods

• The fully black-box hill-climbing method performed similarly to the AutoDAN hybrid and was competitive with likelihood-based optimization requiring a stronger threat model.The comparison used Llama-2-7b, GTR-base, and 20 NQ queries whose clean responses answered the questions.Source: §6.4 · pp. 7–8, Table 2 · p. 8

6.5 Instruction Injection

• Instruction injection succeeded across most settings, whereas oracle-generated blockers significantly underperformed instruction injection and the proposed approach.Instruction injection was worse on NQ but better on MS-MARCO than the optimization-based method; oracle efficacy also depended strongly on the auxiliary model.Source: §6.5 · p. 8, Table 3 · p. 8, §6.6 · p. 9
• Instruction injection only works against LLMs vulnerable to indirect prompt injection, limiting its robustness when defenses suppress instructions in third-party content.The paper evaluates the proposed method and instruction injection against prompting- and fine-tuning-based defenses.Source: §6.5 · p. 8

6.6 Generating Blockers with Oracle LLM

• Oracle-generated blockers depend on auxiliary-model capabilities, with switching from GPT-4 to Claude-3-Opus halving average NQ success and oracle models sometimes refusing adversarial requests.Manual single-document blockers were completely ineffective because successful poisoning would require many documents to dominate retrieval.Source: §6.6 · p. 9

6.7 Transferability and Larger Models

• Blockers transfer poorly across LLMs, while transfer to larger or proprietary models is limited but non-negligible and direct optimization achieved 30%, 10%, and 30% success on GPT-4o-mini, GPT-4o, and Gemini-1.5-flash.The authors conjecture that multi-model optimization could improve transferability, but leave this for future work.Source: §6.7 · pp. 9–10, Table 4 · p. 10

7 Resistance to Jamming as a Safety Property

• Higher safety scores correlate with greater vulnerability to jamming because attacks exploit LLM reluctance to produce potentially unsafe outputs.Jamming resistance aligns with neither adversarial robustness nor overall trustworthiness in the reported comparisons.Source: §7 · p. 10
• Llama-2-7B is ranked most trustworthy by DecodingTrust yet is the most vulnerable model to jamming in the authors’ experiments.The comparison uses 7B models and excludes compressed variants unavailable in the experiments.Source: §7 · p. 10
• Better toxicity scores are associated with greater jamming vulnerability: Llama-2-7B is least toxic and most vulnerable, while Vicuna-7B and Mistral-7B-OpenOrca score similarly on both measures.The paper links this pattern to blocker documents that exploit refusal of possibly toxic queries.Source: §7 · pp. 10–11
• Jamming-resistance rankings disagree with SALAD-bench, ALERT, and most SafetyBench rankings, which often rate Llama-2 models as safer than the experiments indicate.The reported jamming order is Llama-2-7B, Llama-2-13B, Llama-3.1, Vicuna-7B, Mistral, then Vicuna-13B from most to least vulnerable.Source: §7 · p. 11

8.1 Perplexity-based Detection

• Perplexity filtering separates blockers from clean documents, with ROC-AUC 0.05 and average perplexities of 290.64 versus 15.93.The evaluation used about 680 blockers and 250 clean documents on NQ with GTR-base, measuring perplexity using Llama-2-7B.Source: Figure 3 · p. 11, §8.1 · p. 12

8.2 Paraphrasing

• Query paraphrasing can defend against blockers, but predictable paraphrases let attackers optimize documents for multiple phrasings at P times the LLM-call cost.Paraphrasing can also change query meaning, latency, cost, and answer utility.Source: §8.2 · pp. 12–13

8.4 Defenses Against Prompt Injection

• StruQ defeats instruction injection but leaves black-box optimization stronger, whereas SecAlign affects both methods and makes their performance comparable.SecAlign may also cause the model to ignore benign instructions, and comprehensive quality evaluation is outside the paper’s scope.Source: Table 8 · p. 13, §8.4 · pp. 13–14

8.2 Paraphrasing

• Paraphrasing blocker documents reduces jamming below 10%, but processing every database document is computationally expensive and can harm RAG quality.The defense works by removing or heavily modifying the jamming sub-document, producing mostly natural text.Source: §8.2 · p. 13

8.3 Increasing Context Size

• Increasing retrieved context from k = 3, 7, or 10 reduces attack performance, although jamming remains non-negligible at k = 10.Larger contexts can overflow the LLM window and truncate prompts, even at k = 7 for some queries.Source: §8.3 · p. 13, Table 9 · p. 13

• A single blocker document can jam a RAG system into refusing to answer a specific query.Source: §9 · p. 14
• The black-box optimization method requires only query-only access, but its generated documents are easy to detect.Source: §9 · p. 14
• Future work should examine passive blockers that appear non-anomalous and are difficult to detect without relying on an oracle LLM.Source: §9 · p. 14
• Open questions include universal blockers for whole query classes and attacks under limited queries or changing target databases.Source: §9 · p. 14

• The prompt supplies retrieved context between delimiters before asking the model to answer the query.Source: A RAG System Prompt · p. 18
• The model is instructed to use no prior knowledge beyond the supplied context.Source: A RAG System Prompt · p. 18
• If the context lacks enough information, the model must reply “I don’t know.”Source: A RAG System Prompt · p. 18

• The metric asks an oracle LLM to output YES when a response answers the query and NO otherwise, regardless of answer correctness.Source: B Oracle-based Jamming Metrics · p. 18
• GPT-4-Turbo is used with a fixed random seed and temperature 0, although API behavior remains slightly nondeterministic.Source: B Oracle-based Jamming Metrics · p. 18
• The oracle prompt labels incorrect responses as answering when they still provide an answer, while refusals and explicit inability statements are labeled NO.Source: B Oracle-based Jamming Metrics · p. 18
• Human review found the oracle label consistent with the human label for 90 + % of more than one hundred examined query-response pairs.Source: B Oracle-based Jamming Metrics · p. 18
• The evaluation includes examples of oracle-labeled answering and non-answering responses, illustrating that short refusals can fail to answer while detailed context-based responses can answer.Source: B Oracle-based Jamming Metrics · p. 19
• Human-oracle disagreements include a response naming Johann Sebastian Bach for a composer query and an “I don’t know” response that mentions relevant singers.Source: B Oracle-based Jamming Metrics · p. 19
• Substring matching for “I don’t know” supplements the oracle because the system prompt makes that phrase common in non-answering responses, but matching alone is insufficient.Source: B Oracle-based Jamming Metrics · p. 19

• A query qualifies as jammed only when the clean response answers the query and the poisoned response does not.Source: C Discarded Queries · p. 19
• Queries whose clean responses fail to answer are excluded from evaluation, with discarded counts reported in Table 10.Source: C Discarded Queries · p. 19

Ablation setup

• The study evaluates blocker length, document design, and the number of adversarial documents with GTR-base, Llama-2-7b, Vicuna-7b, Mistral, NQ, and 50 retained queries.Queries for which the unpoisoned RAG system did not respond are discarded.Source: D Ablations · p. 19, Table 10 · p. 20

Number of tokens

• Attack success shows no clear dependence on blocker length from 10 to 100 tokens, while unchanged-token fractions rise from around 40% at n = 10 to 88% at n = 100.The reported intermediate fractions are 69% at n = 30, 78% at n = 50, and 88% at n = 100, suggesting the optimizer underuses longer documents.Source: D Ablations · p. 19, Table 12 · p. 20

Variants of blocker document design

• The design ablation compares an un-optimized query plus 50 exclamation marks, a query-only blocker, and a query plus 50 random tokens.These variants test whether jamming depends on optimization and on the additional blocker text.Source: D Ablations · p. 19, Table 11 · p. 20

Blocker content versus missing context

• Blocker content, rather than removing one clean document, accounts for jamming, and optimization is necessary for effective blockers.The comparison contrasts one blocker with k = 5 against no blocker with k = 4 retrieved documents.Source: D Ablations · p. 20, Table 11 · p. 20

Multiple documents

• Multiple independently optimized blockers reduce attack success, and testing each blocker alone yields similar jamming rates, indicating contradictory context effects.The evaluation uses up to three blocker documents per query on NQ with GTR-base; the study uses k = 5 retrieved documents.Source: D Ablations · p. 20, Table 13 · p. 20