Source-linked AI summary
When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapse
Yingtao Ren, Ziyi Zhao, Yiwei Fu, Xiao Luo, Yu-Cheng Chang, Chin-Teng Lin
TL;DR
RAG poisoning attacks can evade output-based detection because they induce false confidence and manipulate generation through retrieved documents. This paper identifies Attention Collapse and introduces D-SCAN, which monitors attention dynamics and outperforms baselines across multiple benchmarks, including when attacks fail to alter the final answer.
Problem
RAG poisoning detection often relies on post-hoc output signals that are inefficient, inaccurate, and ineffective against falsely confident poisoned generations.
Method
The paper identifies Attention Collapse and proposes D-SCAN, a lightweight detector that monitors token- and document-level attention dynamics during inference.
Results
D-SCAN consistently outperforms state-of-the-art baselines across multiple benchmarks and detects poisoning attempts even when attacks fail to alter the final answer.
Takeaways & Limitations
Attention concentration on poisoned documents provides a mechanism-level security signal for detecting RAG poisoning beyond final-output inspection.
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) is indispensable for enhancing large language models. However, RAGs are increasingly susceptible to poisoning attacks, in which adversarial documents are injected to manipulate generator outputs. Previous methods rely on output-side signals such as perplexity and consistency checks to detect such attacks. Nevertheless, our analysis reveals that deliberate attacks often induce false confidence, where poisoned outputs exhibit even lower perplexity than benign ones, rendering uncertainty-based detection ineffective. To address this challenge, we explore the internal dynamics of the generator and identify a distinctive signature termed \textit{Attention Collapse}. Unlike the dispersed attention in benign generations, attacked generations exhibit a decrease in entropy as attention concentrates on poisoned documents. Building on these findings, we propose \texttt{D-SCAN} (Document-level Signal Collapse Analysis), a lightweight detection framework that monitors attention dynamics to identify attacked generations. Extensive experiments on multiple attack benchmarks demonstrate the effectiveness of our method. Moreover, D-SCAN can detect attacks even when they fail to alter the final answer. Code is available at https://github.com/yingtaoren/D-Scan.git.
1 Introduction
RAG poisoning injects adversarial documents to manipulate generator outputs, while existing output-side and black-box detectors are inefficient, opaque, and insensitive to document-level context. The paper identifies Attention Collapse and false confidence as internal signatures of attacks, motivating the lightweight, interpretable D-SCAN detector.
- Motivation: Adversaries inject documents into retrieved databases to induce attacker-specified harmful responses from RAG generators.Retrieved knowledge expands LLM access to current information but introduces poisoning vulnerabilities.
- Limitations: Post-hoc inspection and output-side or black-box detection methods are inefficient, inaccurate, insufficiently granular, and often provide only binary decisions.These approaches overlook information distributed across discrete retrieved chunks and lack interpretability.
- Key observation: Adversarial contexts optimized for generation likelihood create false confidence, causing poisoned responses to exhibit higher average token probabilities than clean responses.This counterintuitive behavior makes uncertainty-based output signals unreliable for poisoning detection.
- Method: D-SCAN formalizes this insight as an interpretable, lightweight framework for real-time poisoning-attack detection.Experiments on three multi-hop QA benchmarks report consistently superior detection accuracy over state-of-the-art baselines with low computational overhead.
- Key observation: Attention Collapse is a distinctive neural signature that exposes adversarial intent within the attention mechanism even when outputs and surface signals appear benign.The phenomenon is more pronounced at the document level than in the broader internal dynamics comparison.
2 Related Work
Prior RAG poisoning work has progressed from heuristic injections to LLM-generated adversarial samples, including retrieval-generation optimization, neuron targeting, and memory corruption in autonomous agents. Existing defenses and detection methods emphasize consistency verification, structural or semantic checks, and reasoning consensus.
- Poisoning attacks for RAG: RAG poisoning attacks have evolved from heuristic injections to sophisticated LLM-generated adversarial samples.
- Poisoning attacks for RAG: PoisonedRAG jointly embeds adversarial targets into retrieval and generation triggers so poisoned documents can induce targeted errors.
- Poisoning attacks for RAG: Poisoning has expanded to autonomous agents, where Agent-Poison corrupts long-term memory to induce self-reinforcing error cycles.
- Attacking defense and detection: Existing RAG defenses rely on deep consistency verification, contradiction graphs, entity-relation triangulation, and reasoning consensus.
- Attacking defense and detection: Reasoning consensus methods isolate retrieved contexts, validate reasoning-path stability, and aggregate diverse generations to reduce malicious outlier influence.
3 Empirical Study and Solution
The empirical study shows that poisoning induces blind confidence and deterministic output behavior rather than uncertainty, while concentrating attention on malicious documents. Based on these internal attention signatures, D-SCAN provides a lightweight detector using token- and document-level attention features.
- Empirical setup: The study evaluates perplexity, consistency, and token- and document-level attention using Llama-3.1-8B-Instruct on the 2Wiki training set.These metrics probe both observable generation behavior and the model’s internal processing under poisoning attacks.
- Blind Confidence: Poisoned samples exhibit higher generation confidence and lower standard deviation than clean samples, revealing the counter-intuitive Blind Confidence phenomenon.Poisoned context increases next-token probabilities, causing the model to prioritize poisoned content over factual information.
- Blind Confidence: Poisoned samples show lower divergence and higher sequence cosine similarity across ten sampling runs, indicating deterministic collapse in the generation space.These consistency metrics further show that poisoning strengthens apparent confidence rather than confusing the model.
- Attention Collapse: Under attack, document attention has lower entropy, higher standard deviation, and greater concentration on poisoned documents than benign attention.The study terms this pathology Attention Collapse: poisoned documents hijack self-attention that is otherwise distributed across retrieved evidence.
- D-SCAN: D-SCAN trains a linear classifier on token- and document-level attention entropy, variance, and density extracted from internal states during inference.The method is designed for super lightweight detection with low computational overhead.
4 Experiment
D-SCAN is evaluated against open-source LLMs and established detectors across three multi-hop benchmarks, where it consistently outperforms competing methods. Additional analyses show robustness to failed attacks, improved detection with multi-sampling, and the importance of document-level attention features.
- Experimental Setup: D-SCAN is compared with open-source LLMs and HaloScope, ReDeep, and RevPRAG on HotpotQA, 2Wiki, and Musique using E5-basev2 retrieval.Clean samples contain five benign documents, while poisoned samples replace two benign documents with poisoning documents.
- Overall Results: D-SCAN consistently outperforms generalized LLM-based detectors and specialized detection methods across the evaluated benchmarks.The results are attributed to monitoring internal attention collapse rather than relying on output-based or semantics-based signals.
- Robustness to Attack Outcomes: D-SCAN maintains high detection fidelity when attacks fail to induce their target answer, while detection is stronger on attack-success samples than on all samples.These findings support attention collapse as an intrinsic poisoning signature and associate successful attacks with prioritization of poisoned content over factual content.
- Sensitivity to Generation Samples: Detection performance consistently improves as the number of generation samples increases, because multi-sampling captures more diverse attention variations.The sensitivity analysis evaluates generation sample sizes from one to ten and frames the trade-off against computational overhead.
- Ablation Study: The full D-SCAN model outperforms variants without token-level or document-level attention metrics, with removing document-level features causing the larger degradation.The results identify cross-document attention hijacking as the dominant attack signal and show that combining token- and document-level features is essential.
5 Conclusion
The paper identifies Attention Collapse as the mechanism through which RAG poisoning hijacks LLM attention, suppressing real evidence in favor of poisoning evidence. It proposes D-SCAN, a lightweight detector that monitors attention dynamics.
- 5 Conclusion: Attention Collapse focuses model attention on poisoning evidence while suppressing real evidence.This finding characterizes how poisoning attacks manipulate the model’s attention allocation mechanism.
- 5 Conclusion: RAG poisoning attacks succeed by hijacking the model’s attention allocation mechanism.
- 5 Conclusion: D-SCAN is a lightweight attack detection method that monitors attention dynamics.