PaperlayerView exact revision on arXiv

Source-linked AI summary

CamoDocs: A Poisoning Attack Against Retrieval-Augmented Language Models Using Camouflaged Documents

Jaewon Jung, Haizhong Zheng, Hongsun Jang, Jaeyong Song, Beidi Chen, Jinho Lee

arXiv:2608.28389v1cs.CRcs.CL

TL;DR

RAG systems can be poisoned through injected documents, and query-inclusion attacks leave artifacts that make them easy to filter. CamoDocs camouflages adversarial content among benign material by dispersing embeddings and filtering replacements for coherence. It remains effective across diverse defenses and models, including proprietary LLMs, while erasure-heavy defenses can reduce utility in retrieval-dependent settings.

  • Problem

    RAG systems depend on external documents that attackers can inject or manipulate, while existing query-inclusion attacks expose lexical and embedding artifacts that defenses can detect.

  • Method

    CamoDocs chunks synthesized benign and adversarial drafts, replaces selected benign tokens with dispersion tokens, applies coherence filtering, and merges the resulting subdocuments.

  • Results

    CamoDocs achieves attack success across seven defenses, three open-weight models, and two proprietary models, including average ASRs of 61.80% on GPT-5.4-mini and 55.09% on Claude-Haiku-4.5.

  • Takeaways & Limitations

    CamoDocs avoids query-overlap and compact-clustering artifacts, while erasure-heavy defenses such as TrustRAG can reduce RAG utility in retrieval-dependent settings.

  • Takeaways & Limitations

    The threat model assumes attackers can inject poisoned documents, which may be difficult in highly restricted environments with strict ingestion controls or manual curation.

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) augments LLMs with external documents, but public or user-editable sources expose RAG systems to data poisoning: attackers can inject malicious documents to steer outputs toward targeted answers. Existing poisoning attacks often rely on query inclusion, inserting the target query into poisoned documents to improve retrieval; however, this creates lexical and embedding-space artifacts that make them easy to filter. We propose CamoDocs, a poisoning attack that avoids direct query inclusion by camouflaging adversarial documents among benign content. CamoDocs chunks synthesized benign and adversarial drafts, replaces selected tokens in benign chunks with dispersion tokens that spread poisoned-document embeddings, and applies coherence filtering to limit readability degradation. Across seven RAG defenses, three open-weight LLMs, and three benchmarks, CamoDocs achieves strong average ASR while avoiding query-overlap artifacts exploited by simple query detection. It also remains effective against proprietary models, achieving average ASRs of 61.80% on GPT-5.4-mini and 55.09% on Claude-Haiku-4.5. Finally, we show that erasure-heavy clustering defenses such as TrustRAG can reduce ASR, but only with substantial utility drops on retrieval-dependent benchmarks such as NeoQA. Code is available at https://github.com/jaewonalive/CamoDocs.

1 Introduction

RAG systems can be poisoned through external documents, while query-inclusion attacks expose lexical and embedding artifacts that defenses can detect. CamoDocs avoids direct query inclusion by camouflaging adversarial content and dispersing poisoned-document embeddings.

  • 1 Introduction: RAG relies on external documents that attackers can poison to steer victim LLMs toward targeted incorrect answers.This risk is especially important for high-stakes applications.
  • 1 Introduction: Existing attacks include target queries to improve retrieval, causing compact query-near embedding clusters that defenses can filter.Query inclusion also makes adversarial documents explicitly contain the target query, enabling simple query-detection defenses.
  • 1 Introduction: CamoDocs generates benign and adversarial drafts, chunks them, and replaces selected benign tokens with dispersion tokens that spread poisoned-document embeddings.The attack avoids direct query inclusion rather than relying on the artifacts exploited by existing defenses.
  • 1 Introduction: CamoDocs applies coherence filtering and merges optimized benign subdocuments with adversarial subdocuments to form final poisoned documents.The filtering step limits readability degradation.
  • 1 Introduction: CamoDocs achieves attack success across seven defenses, three open-weight models, and two proprietary models while avoiding query-inclusion and compact-clustering artifacts.The paper also reports that erasure-heavy clustering defenses can reduce RAG utility.

2 Preliminaries

RAG combines a retriever, a document database, and a generator. The retriever selects relevant documents using dense embeddings, and the LLM generates an output conditioned on the query and retrieved documents.

  • 2 Preliminaries: A RAG system contains a retriever, knowledge database, and generator, usually an LLM.The database stores documents d_i, and the generator produces the final output.
  • 2 Preliminaries: Dense retrievers embed queries and documents, score relevance by dot product or cosine similarity, and return the top-k documents.The embedding model E_θ maps both queries and documents into dense vectors.
  • 2 Preliminaries: The generator conditions on both the query and retrieved documents, formalized as R(q, D, E_θ) = D̃_q and ŷ = LLM(D̃_q, q; ϕ).Here, ϕ denotes the LLM parameters.

3 Method

CamoDocs crafts poisoned documents by combining benign and adversarial subdocuments, then manipulating benign tokens to disperse embeddings while preserving readability and adversarial content. Its gradient-guided, coherence-filtered procedure selects replacements that increase dispersion under a surrogate encoder.

  • Threat Model: CamoDocs assumes black-box access to the victim and the ability to inject malicious documents into the knowledge database.Its objective is to cause targeted incorrect outputs for specific queries.
  • Preparing the Ingredients: For each target query, CamoDocs synthesizes benign and adversarial drafts whose contents respectively provide query-relevant benign material and target-specific adversarial material.The adversarial drafts are conditioned on the correct and target incorrect answers.
  • Preparing the Ingredients: CamoDocs chunks each draft and re-indexes benign–adversarial chunk pairs before token manipulation and merging.Chunking weakens document-level cues that would otherwise make final poisoned documents more similar.
  • Token Manipulation: The dispersion loss increases mean distance from the embedding centroid, pushing poisoned subdocuments apart and reducing compact patterns exploited by defenses.The loss is computed with a surrogate encoder over benign subdocument embeddings.
  • Token Manipulation: At each iteration, CamoDocs uses a first-order gradient score to select candidate replacement tokens from the surrogate vocabulary.The score estimates each candidate’s increase in the dispersion loss.
  • Token Manipulation: A coherence filter re-ranks dispersion candidates by perplexity, retains the lowest-perplexity subset, and exact dispersion-loss evaluation chooses the replacement.The adversarial subdocument remains unchanged during dispersion-oriented optimization to preserve tokens crucial for inducing the target answer.

4 Experiments

Experiments show that CamoDocs remains effective across defenses, models, datasets, retrievers, and ablations, while TrustRAG’s robustness can impose severe utility costs on retrieval-dependent tasks.

  • Results: All baseline attacks achieve less than 12% ASR against query detection across datasets and models because they explicitly include the target query.Benign documents rarely closely match the full user query, so clean accuracy is minimally affected.
  • Results: CamoDocs trades some retrieval advantage for stealth, whereas query inclusion can improve ASR against other defenses but sharply reduces robustness to query detection.Cross-encoder reranking is also less effective against query-inclusion attacks because the target query raises query–document relevance.
  • Results: TrustRAG lowers ASR for all attacks but substantially reduces utility when the LLM depends on retrieved evidence.On NeoQA, it removes 91.48% of retrieved documents and lowers clean accuracy from 29.13% to 5.79%; across thresholds, accuracy remains 4.32%–12.62%.
  • Results: On HotpotQA, TrustRAG removes only 12.78% of retrieved documents, reducing clean accuracy from 49.10% to 43.70%, while no-retrieval accuracy is 37.20%.The contrast indicates that stale benchmarks can obscure defense utility costs when models retain relevant parametric knowledge.
  • Results: Under attack, TrustRAG leaves CamoDocs ASRs of 23.25% on NeoQA and 29.10% on HotpotQA, while NeoQA clean accuracy falls to 9.08%.Thus, TrustRAG does not provide reliable robustness in retrieval-dependent settings without undermining RAG utility.
  • Ablation Study: The dispersion loss produces the largest ablation gain, increasing ASR by 17.20% against TrustRAG and identifying dispersion as the key adaptive mechanism.Random replacement yields lower embedding dispersion and 16.30% ASR versus 29.10% for CamoDocs, despite lower GPT-2 perplexity.
  • Ablation Study: Coherence filtering reduces average GPT-2 perplexity by about 46%, from 738.6 to 401.3, primarily improving adversarial-document readability.Its ASR improvement is marginal at 0.40%.

5 Related work

RAG combines retrieval and generation to ground LLM outputs in external knowledge, but its dependence on external sources creates poisoning vulnerabilities. Prior work includes knowledge poisoning and other adversarial attacks against RAG systems.

  • Retrieval-Augmented Language Models: RAG systems use retrievers, knowledge databases, and generators to fetch documents relevant to a query.Sparse retrievers include BM25, while dense retrievers map queries and documents into vectors and rank them by similarity.
  • Adversarial Attacks for LLMs: Adversarial-attack research on LLMs spans jailbreaking and backdoor attacks on pretraining data.These attack categories provide broader context for security research involving language models.
  • Evaluation Settings: Matched-budget evaluations report ASR values for attacks on HotpotQA using Llama-3.1-8B.The table defines β as the number of adversarial documents injected per target query.
  • Adversarial Attacks for LLMs: RAG vulnerabilities include knowledge poisoning, opinion manipulation, jamming, and joint backdoor attacks.Prior studies also analyze poisoning mechanics and evaluate RAG robustness.

6 Conclusion

CamoDocs avoids query inclusion and disperses adversarial-document embeddings to evade retrieval-time defenses. The paper finds that query-inclusion attacks are easily filtered, while erasure-heavy defenses can substantially damage utility when retrieval is essential.

  • 6 Conclusion: CamoDocs combines document chunking, dispersion-token replacement, and coherence filtering to camouflage poisoned documents.The design avoids query-overlap artifacts and reduces compact-clustering artifacts used by defenses.
  • 6 Conclusion: Query-inclusion attacks are easily filtered, whereas erasure-heavy defenses such as TrustRAG can impose large utility drops in retrieval-dependent settings.The conclusion motivates defenses that detect poisoned documents without aggressively removing useful retrieved evidence.
  • 6 Conclusion: The paper calls for defenses that detect poisoned documents while preserving retrieved evidence.This conclusion follows the reported tension between robustness and RAG utility.

Limitations

The study assumes attackers can inject documents into a RAG knowledge base, incurs additional offline construction computation, and relies on transfer from a surrogate embedding model. These constraints may limit applicability across restricted environments and differing retrieval pipelines.

  • Threat Model: The threat model assumes attackers can inject poisoned documents into the RAG knowledge base.This is realistic for public, user-editable, or web-scraped sources but harder in restricted environments with strict ingestion controls or manual curation.
  • Computational Cost: CamoDocs requires more computation than simple heuristic attacks because it uses gradient-guided token replacement and coherence filtering.The cost occurs once during offline poison-document construction and does not affect victim inference latency.
  • Transferability: CamoDocs relies on transfer from a surrogate embedding model to the victim retriever.Transferability may vary across retrieval architectures, indexing pipelines, and document preprocessing strategies.

Ethical Considerations

The paper studies RAG poisoning to characterize vulnerabilities and support stronger defenses, while limiting misuse through controlled benchmark evaluations. AI assistants were used only for language and manuscript-polish tasks, not for research decisions or results.

  • Ethical Considerations: The work studies adversarial document injection to induce targeted incorrect outputs and improve understanding of RAG vulnerabilities.The stated goal is to help develop stronger defenses and system robustness.
  • Ethical Considerations: The evaluation focuses on controlled benchmark settings and uses the attack to assess existing defense robustness.The authors emphasize detecting poisoned documents without aggressively removing useful retrieved evidence.
  • AI Assistance: AI assistants were used for language polish, grammar correction, wording refinement, and LaTeX troubleshooting.The authors state that assistants did not generate research ideas, design the method, produce results, or make research decisions.
  • Evaluation Practice: The experiments generally follow prior setups but evaluate 1,000 randomly selected queries per dataset because 100 queries were considered insufficient for reliable evaluation.This passage describes an experimental-design choice reported in the appendix.

A.1 Datasets

The evaluation uses four retrieval-dependent QA benchmarks, three open-weight models, and proprietary models, with multiple defenses including query detection. Experiments measure attack success and clean utility under these settings.

  • Datasets: HotpotQA contains 5,233,329 texts and evaluates multi-hop questions on the BEIR test split.
  • Datasets: NQ contains 2,681,468 texts and uses DPR-preprocessed test data because BEIR lacks answers for most evaluated queries.
  • Datasets: NeoQA evaluates evidence-based reasoning with fictional content designed to minimize reliance on parametric knowledge.
  • Datasets: Each NeoQA trial injects 10 adversarial documents, corresponding to a poisoning ratio of approximately 0.7%.
  • Models: The study evaluates Qwen3-8B, Llama-3.1-8B, Mixtral-8x7B, GPT-5.4-mini, and Claude-Haiku-4.5.
  • Defenses: Query detection largely preserves clean accuracy, with negligible effects on HotpotQA and NQ but a 2.5–3.3 percentage-point drop on MS-MARCO.

F Effectiveness against Density-Based Clustering (DBSCAN) Defense

CamoDocs resists DBSCAN-based clustering because its adversarial embeddings are either labeled as noise or form sparse, low-cohesion clusters. Consequently, threshold-based filtering removes no documents across the tested sweep.

  • DBSCAN behavior: DBSCAN’s effectiveness is highly sensitive to ϵ and consistently fails to trigger threshold-based filtering against CamoDocs.
  • Over-segmentation: At small ϵ, DBSCAN labels almost all embeddings as noise, leaving no cluster for the defense to filter.
  • Persistent sparsity: Among settings where clusters form, average intra-cluster cosine similarity ranges from 0.43 to 0.06, below TrustRAG’s 0.88 cohesion threshold.
  • Sweep results: At ϵ = 0.50, the noise label rate is 100%; from ϵ = 0.60 to 1.00, it remains between 99.90% and 94.77%.
  • Attack success: 37.00%–39.30% ASR against DBSCAN exceeds the 29.10% ASR against K-means because no documents are filtered.

G Additional Visualization Results

Additional visualizations and examples support CamoDocs’ embedding-dispersion and coherence-filtering mechanisms. The method’s construction uses synthesized drafts, gradient-guided replacement, coherence screening, and document merging.

  • Additional Visualization Results: CamoDocs adversarial embeddings are more dispersed than those produced by baseline attacks in the additional t-SNE visualizations.
  • Additional Visualization Results: PIA documents lie near the target query because their manipulative instructions directly include the query.
  • Qualitative Example: The coherence-filtered example contains more natural and contextually plausible phrases than the unfiltered adversarial document.
  • Construction: For each query, CamoDocs generates five benign and five adversarial drafts, splits each into two sub-documents, and forms 10 paired sub-document units.
  • Optimization: Each benign sub-document undergoes sequential gradient-guided token replacement, coherence filtering, and exact-loss evaluation before merging with its adversarial counterpart.

I.4 Overall per-query Complexity and Runtime

CamoDocs’ computational costs are incurred offline during poisoned-document construction, leaving inference latency unchanged. Generating the full set of 10 documents requires substantial local computation, while increasing the document count can reduce stealth.

  • Complexity: The per-query cost combines draft synthesis, chunking, token manipulation with coherence filtering, and sub-document merging.
  • Runtime: Offline construction leaves online RAG latency unchanged because inference uses the same retriever and LLM as the clean setting.
  • Runtime: Generating one adversarial document takes approximately 3.22 minutes, so producing 10 documents requires roughly 32.20 minutes of local computation.
  • Sensitivity Study on Poisoning Ratio: At β = 25, only 48.08% of adversarial documents remain after TrustRAG and ASR falls to 19.00%, indicating reduced stealth from denser clustering.
  • Heuristic Defenses: Both query rephrasing and perplexity filtering leave every attack above 60% ASR, while CamoDocs remains above 75%.

L Human Annotation Results for Correct-Answer Judgments

Human annotation was used to assess correct-answer judgments and compare them with the LLM judge. Three annotators independently evaluated each response using the query, answers, and model output.

  • 96.43% average agreement and 0.89 mean Phi correlation were observed between LLM and human correct-answer judgments.All per-defense correlations were statistically significant at p < 0.001.
  • The correct-answer agreement analysis complemented the separate validation of attack-success judgments.The cited analysis used the same subset of HotpotQA test queries as the attack-success validation.
  • Three independent human annotators judged each response for conveying the correct answer and whether the attack succeeded.Annotators received the target query, correct answer, target incorrect answer, and model output.
Loading 2608.28389v1…