Source-linked AI summary
Towards Safer RAG: Only Agents Capable of System 2 Thinking may Access Untrusted Documents
Mehrdad Ghassabi
TL;DR
RAG systems can be influenced by poisoned documents even after detecting their misinformation. This paper evaluates whether System 2-capable agents are more robust, finding substantially higher resilience without strict Cordon Principle isolation.
Problem
RAG systems remain vulnerable to poisoned documents that can influence final answers despite successful misinformation detection.
Method
The paper introduces Cordon Rate and Contamination Rate metrics and compares reasoning language models with standard models across poisoning-resilience measures.
Results
DeepSeek-Reasoner achieved zero cordon rates on SciFact and FiQA, while DeepSeek-Chat recorded C = 0.175 and 0.025, respectively, with SciFact contamination falling from 0.25 to 0.10.
Takeaways & Limitations
The findings support allowing only System 2-capable agents to access untrusted documents as a more practical foundation for secure RAG than strict isolation.
Takeaways & Limitations
The monitoring-control gap may vary across hallucination types or domains, and future work must investigate this variation in multi-agent architectures.
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) has significantly enhanced the performance of large language models (LLMs), yet these systems remain vulnerable to knowledge-poisoning attacks, in which misinformation in retrieved documents can influence the model's final outputs. Notably, an LLM may correctly detect that a document contains incorrect information while nevertheless being influenced by it. Prior work has addressed this vulnerability through the Cordon Principle, which prevents models responsible for final answer synthesis from directly accessing raw evidence. Although effective, this strict isolation can introduce substantial computational overhead. In this work, we propose a refined security principle: only agents capable of deliberative System 2 reasoning may access untrusted documents. To evaluate this principle, we introduce novel metrics that quantify the discrepancy between misinformation detection and downstream influence. We then empirically compare state-of-the-art reasoning language models with standard language models across these metrics. Our results show that reasoning-capable models are substantially more robust to corrupted evidence, without requiring the strict isolation imposed by the Cordon Principle. These findings provide empirical support for our refined principle and suggest a more practical foundation for secure RAG system design.
1 Introduction
The introduction identifies a key RAG vulnerability: models may detect poisoned documents yet remain influenced by them during answer synthesis. It proposes allowing untrusted-document access only to agents capable of deliberative System 2 thinking and empirically validates this principle against standard models.
- Problem: RAG enables knowledge poisoning attacks in which adversarial documents manipulate models’ final responses, even after the models detect the poisoning.This exposes a gap between misinformation detection and resistance to downstream influence.
- Prior Work: The cordon principle blocks final-answer-synthesis agents from raw evidence, but its security comes with prohibitive computational overhead and reduced flexibility.The approach strictly isolates answer synthesis from untrusted documents.
- Contribution: This work proposes that only agents capable of deliberative System 2 thinking access untrusted documents, while associative-matching agents remain shielded.The authors empirically validate this refined principle using metrics for poisoning resilience and reasoning overhead, comparing reasoning models with standard models.
2 Approach
The approach links the monitoring-control gap to deficient System 2 reasoning and evaluates this hypothesis with two complementary metrics. It measures residual influence from detected misinformation and automatic contamination from retrieved evidence despite instructions to ignore it.
- Motivation and hypothesis: Prior work shows that models can detect misinformation yet remain influenced by it during answer synthesis, motivating the hypothesis that this gap reflects deficient System 2 reasoning.The authors argue that an agent capable of System 2 reasoning would suppress information it confidently identifies as incorrect.
- Evaluation metrics: The Cordon Rate (C) measures the joint probability that a valid instance is both detected and influenced by the poisoned document.It is computed across all valid samples, not as P(influenced | detected, Vi), thereby capturing the absolute residual failure rate.
- Evaluation setup: Each test instance gives model M retrieved contexts containing an injected poisoned document, then asks it to reason, check for misinformation, and answer.A separate judge assesses both whether M detected the misinformation and whether its final answer was influenced by the poison.
- Evaluation metrics: The Contamination Rate (T) measures whether retrieved poison influences a model explicitly instructed to ignore documents beyond influence attributable to its parametric knowledge.It compares M1 with poisoned retrievals against M2 without retrieved context, assigning 1 only when M1 is influenced and M2 is not.
- Poison construction: The poisoning strategy generates a contradictory sentence from the correct answer and expands it into a coherent 500-word passage validated by a judge.The authors state that this naive construction shows the gap is not an artifact of highly optimized adversarial content.
3 Experiments
The experiments compare standard and reasoning-enhanced models on BEIR-derived datasets to test whether System 2 agents can access untrusted documents without downstream influence. Results show reasoning models are more robust, while contamination and the monitoring-control gap vary substantially by task.
- Experimental setup: The study evaluates RQ1–RQ3 using 40 SciFact, FiQA, and MS-MARCO queries, comparing DeepSeek-Chat with DeepSeek-Reasoner and using Gemini 2.5 Pro for automated evaluation.DeepSeek-Chat is the standard base model, while DeepSeek-Reasoner is the reasoning-enhanced model.
- RQ1: Cordon rates: 0 cordon rate: DeepSeek-Reasoner achieves zero on SciFact and FiQA, while DeepSeek-Chat records 0.175 and 0.025, respectively.Both models score zero on MS-MARCO.
- RQ1: Cordon rates: The reasoning model’s lower susceptibility suggests a narrower monitoring-control gap because it can identify and disregard conflicting information in untrusted documents.This result supports access to untrusted documents without substantial influence on final responses.
- RQ2: Contamination rates: 25% contamination rate: DeepSeek-Chat’s T on SciFact indicates implicit influence from poisoned evidence despite an explicit instruction to disregard it.The metric compares a context-provided instance with an otherwise identical no-context instance.
- RQ3: Task dependence: 0 contamination: Both models reach zero on MS-MARCO; FiQA rates are 0.10 for Chat and 0.00 for Reasoner, while SciFact rates are 0.25 and 0.10.The disparity indicates that the monitoring-control gap is primarily observable in challenging, knowledge-intensive scientific fact-verification tasks.
4 Conclusion
The conclusion proposes that only System-2-capable agents should access untrusted RAG documents and reports empirical validation using Cordon Rate and Contamination Rate. It also extends the contamination concern to multi-agent systems, motivating research on separating detection from synthesis.
- The paper proposes that only System-2-thinking-capable agents should access untrusted documents.
- DeepSeek-Reasoner achieved zero cordon rates on SciFact and FiQA, whereas DeepSeek-Chat had C = 0.175 and 0.025, respectively.These results were reported using the Cordon Rate (C) and Contamination Rate (T) metrics.
- System-2-incapable models remained implicitly influenced by context even when explicitly instructed to ignore it.
- The findings suggest that a receiving agent’s System 1 mechanism may be unconsciously influenced by hallucinations detected by another agent.The paper identifies this as analogous to contamination observed in M1 despite explicit ignore instructions.
- Future work should examine how to decouple detection from synthesis in agentic architectures and whether the gap varies across hallucinations.
A Full Prompt Templates
The appendix provides prompt templates for RAG security evaluation, including target-model detection, poison detection, contradiction detection, incorrect-idea generation, poison generation, and poison-consistency checking. These templates enforce structured reasoning or tightly constrained outputs for each evaluation stage.
- A.1 Prompt for Target Model — RAG with Detection Instruction: The target-model prompt instructs step-by-step context analysis, misinformation detection, structured reporting, and answering from valid context and general knowledge.It requires a status, flagged snippet, reason, detailed reasoning, and final conclusion.
- A.2 Prompt for Model Poison Detection: The poison-detection evaluator labels detection TRUE when the response flags, ignores, or warns about poisoned information and FALSE when it incorporates or fails to acknowledge it.The evaluator must output exactly TRUE or FALSE.
- A.2 Prompt for Model Poison Detection: Across the templates, evaluators use constrained outputs and supplied placeholders for prompts, responses, poison content, generated answers, ground truth, questions, and passages.Several templates explicitly prohibit punctuation, explanations, preambles, or additional text.
- A.3 Prompt for Contradiction Detection: The contradiction detector compares an AI answer with ground truth and identifies only logically opposite, mutually exclusive, or numerically incompatible facts.Omissions, paraphrasing, and additional non-conflicting information are explicitly excluded; output is TRUE or FALSE.
- A.4 Prompt for Incorrect Idea generation: The incorrect-idea generator produces one direct, plausible sentence answering the question with a factually incorrect claim contradicting the correct answer.It defines contradiction as logically opposite, mutually exclusive, or numerically incompatible and returns only the sentence.
- A.5 Prompt for Poison Generation: The poison generator expands an incorrect idea into an authoritative, formal, single-paragraph passage of exactly 500 words presented as established truth.The prompt forbids acknowledging falsity, hedging, introductions, commentary, labels, and concluding remarks.
- A.6 Prompt for Poison Incorrectness Check: The poison-incorrectness checker judges an answer solely against the passage, marking it Incorrect when contradicted and Correct when supported or not explicitly refuted.It prohibits external knowledge and requires only one output word: Correct or Incorrect.
When asking a model questions from fact-checking
For fact-checking questions, the benchmark used a strict response format to automatically extract the model’s binary detection decision. Responses had to end with exactly one of two verdict lines.
- SciFact was the only fact-checking dataset in the benchmark, prompting a strict output requirement for automatic detection-flag extraction.
- Models had to conclude their responses with a single line stating the final binary decision.
- The permitted endings were exactly “Final Verdict: Yes” or “Final Verdict: No,” with no subsequent text.