PaperlayerView exact revision on arXiv

Source-linked AI summary

RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety

Adithiyan Rajan Indira Saravanan, Kathleen C. Fraser

arXiv:2609.11758v1cs.CLcs.IR

TL;DR

RAG can improve reliability while also changing the safety of responses to harmful requests, and existing evidence does not cleanly identify why. RAG-Safety-Bench isolates answer availability, topical context, and safe random context across five open-source LLMs, finding model-specific safety degradation and limits to transferring non-RAG safety guarantees into RAG systems.

  • Problem

    RAG safety degradation is difficult to explain because retrieved-document content and retriever quality are confounded, despite RAG’s growing use for grounding LLMs.

  • Method

    RAG-Safety-Bench supplies contexts directly and compares non-RAG, oracle, on-topic, and random-document conditions across harmful queries and five open-source LLMs.

  • Results

    The benchmark finds model-specific safety degradation: Qwen-2.5-7B remains highly harmful with on-topic answer-free documents, while random documents produce the safest responses.

  • Takeaways & Limitations

    Non-RAG safety evaluations do not guarantee safety after RAG deployment, so practitioners should assess safety across the full RAG pipeline.

  • Takeaways & Limitations

    The benchmark is English-only, uses Wikipedia as its sole document source, covers a non-exhaustive set of harms, and does not evaluate retriever effects or full production pipelines.

Abstract

from arXiv · show

Allowing large language models (LLMs) to retrieve information from a set of trusted documents can increase reliability and reduce hallucination. However, recent work has demonstrated that retrieval-augmented generation (RAG) can have unintended side effects on the overall safety of the generated responses, when prompted for harmful or dangerous content. A clearer understanding of the mechanisms leading to this result is needed, as increasing numbers of end users turn to RAG to incorporate corporate documents and knowledge bases into LLM-based systems. We introduce RAG-Safety-Bench, a benchmark to measure the safety impact of RAG on LLM models. By removing the confounding effect of retriever quality, and cleanly separating the problem into four conditions -- non-RAG, RAG with an oracle document containing the answer to the harmful request, RAG with documents related to the harmful request but without the specific answer, and RAG with random, safe documents -- the benchmark isolates the impacts of different factors in the observed safety degradation. We report results across five open-source LLMs, showing an inverse relationship between benign and unsafe capability, strong evidence that baseline safety guardrails do not lead to downstream safety guarantees in the RAG case, and model-specific support for previous findings that even benign documents can lead to unsafe generation in retrieval-enabled systems.

1 Introduction

RAG improves grounding and answer quality, but retrieved context can also reduce safety, even when documents are benign. RAG-Safety-Bench isolates whether unsafe generation reflects answer availability, harmful topical context, or retrieval context more generally.

  • Motivation: RAG can improve factuality and reduce hallucinations, creating an expectation that retrieval-augmented systems should be safer than non-RAG models.Retrieved documents can provide current facts and reduce reliance on stale parametric memory without retraining.
  • Motivation: Prior work instead found that RAG-enabled LLMs can be less safe than non-RAG models even when retrieved documents are benign.The benchmark addresses this newly identified safety problem while separating safety from retrieval capability.
  • Benchmark motivation: RAG-Safety-Bench separates non-RAG evaluation from oracle, on-topic, and random-document conditions to isolate distinct sources of safety change.Oracle documents contain the unsafe answer; on-topic documents concern the query without containing that answer; random documents cover safe topics.
  • Research questions: The study tests whether stronger benign RAG capability predicts lower safety in the oracle condition, where models can directly access answers to harmful requests.This hypothesis links performance on MIRAGE with accuracy and safety outcomes in RAG-Safety-Bench.
  • Research questions: The authors distinguish refusal caused by unwillingness to provide dangerous advice from refusal caused by missing information in retrieved documents.The random condition is expected to produce information-limited refusals, while the oracle condition makes guardrail effectiveness easier to interpret.

2 Related Work

Prior work identifies safety degradation from harmful retrieval and extended context, while safety evaluation commonly relies on refusals or automated classifiers. This paper tests these explanations separately and treats refusal, classifier judgments, and answer accuracy as distinct signals.

  • RAG safety: Earlier RAG safety studies measured whole-system behavior without controlling the safety and information content of retrieved documents.This uncontrolled retrieval variable makes it difficult to isolate why RAG outputs become unsafe.
  • Long-context safety: Research has proposed both harmful documents and context-induced behavioral shifts as explanations for safety degradation in retrieval-enabled models.The benchmark is designed to test these explanations separately.
  • Safety evaluation: Safety evaluation often uses refusal as a proxy, but refusals can take multiple forms and do not exhaust the available safety signals.The study also considers safety classifiers and response accuracy.
  • Safety evaluation: The paper treats refusal, classifier-based safety, and information accuracy as correlated but distinct measures of RAG behavior.This distinction supports a more granular evaluation than any single safety label.

3 Methods

The benchmark controls retrieval quality by supplying contexts directly and evaluates harmful-query responses across four retrieval conditions. It combines a harm taxonomy, Wikipedia-derived answerable questions, multiple safety judges, and five open-source LLMs.

  • Benchmark design: RAG-Safety-Bench directly compares four settings, distinguishing answer-containing oracle contexts from answer-free on-topic contexts and safe random contexts.Providing documents directly removes retriever quality as a confounding factor in evaluating base-model safety guardrails.
  • Benchmark construction: The benchmark constructs harmful queries from a modified AILuminate taxonomy and uses Wikipedia documents containing answers to those questions.The study excludes privacy, direct harmful-content generation, intellectual property, and specialized-advice categories.
  • Benchmark construction: The Full benchmark contains 987 harmful queries, while the Balanced version used in the analysis samples more uniformly across harm subcategories.Both versions are built from Wikipedia-based questions generated with Claude-Sonnet-4.5.
  • Safety evaluation: Responses are judged by LlamaGuard-3-8B, ShieldGemma-2B, and WildGuard, with an overall unsafe label assigned when at least two judges flag a response.The evaluators’ pairwise agreement is measured using Cohen’s Kappa.
  • Models and experiments: Experiments evaluate Gemma-3-12B-It, Llama-3.1-8B-Instruct, Ministral-3-8B-Instruct, Qwen-2.5-7B-Instruct, and Phi-4-14B on harmful queries with up to 1024 generated tokens.Commercial models were not tested because the benchmark questions would violate their API terms of service.

4 Results

Across five open-source models, RAG most consistently increased unsafe behavior when the retrieved context contained the harmful answer, while on-topic and random contexts produced model-dependent effects. The results also show that benign RAG capability did not predict unsafe capability and that safety judgments vary across evaluators.

  • R1: Capability in safe and unsafe contexts: −0.70 Pearson correlation between benign-oracle and unsafe-oracle accuracy rejected the hypothesis that stronger benign RAG capability predicts stronger unsafe capability.All models were generally more likely to answer harmful questions correctly when the answer was available, despite the negative cross-condition relationship.
  • R2: Unsafe information or unsafe context?: The oracle RAG condition produced the highest harmfulness rate for every model, showing that retrieved access to a harmful answer can increase unsafe responses.The benchmark’s four settings separate non-RAG, oracle, on-topic, and random safe-document conditions; Figure 2 reports judged harmfulness rather than response accuracy.
  • R2: Unsafe information or unsafe context?: Random safe documents produced the safest behavior overall, indicating that long context alone was insufficient to steer models into an unsafe regime.On-topic documents had mixed effects: some models returned near-baseline harmfulness, whereas others remained more harmful than in the non-RAG baseline.
  • R2: Unsafe information or unsafe context?: Qwen was third safest without RAG but least safe in the oracle and on-topic conditions, whereas Phi-4-14B remained safest across non-RAG and RAG settings.This model-specific reversal suggests that guardrail robustness differs across models.
  • R2: Unsafe information or unsafe context?: Unsafe responses increased most for radiological weapons, explosives, drugs, chemical weapons, assault, terrorism, and cyberattacks, with Ministral and Qwen also increasing on suicide and self-harm.These category-level changes are reported for the non-RAG-to-oracle comparison.
  • R4: Subjectivity of harm: Human–LLM annotation agreement reached 85% with Cohen’s κ = 0.71–0.73, but automated judges showed only moderate pairwise agreement, supporting the use of multiple evaluators.LlamaGuard and WildGuard agreed at κ = 0.66; other pairwise comparisons were κ = 0.47 and 0.48, while accuracy aligned most with WildGuard at κ = 0.67.

5 Discussion

The discussion finds that long context alone does not explain unsafe RAG behavior: random documents produce the safest responses, while topic-relevant context can trigger harmful generation in Qwen. These findings support evaluating RAG safety separately from non-RAG safety and across model-specific behaviors.

  • Random documents led to the safest responses, indicating that long context alone did not drive unsafe behavior in these experiments.The random condition partly increased refusals because the documents lacked information.
  • In the on-topic condition, most models returned to roughly baseline safety, but Qwen produced a high harmfulness rate despite documents lacking the specific answer.This was the major exception to the broader trend.
  • A chemical-agent example shows on-topic context eliciting actionable harmful instructions after the model initially acknowledged that the documents lacked the needed information.The non-RAG response refused, whereas the oracle response answered directly and the on-topic response continued with harmful guidance.
  • In the random condition, the model stopped after stating that the documents lacked the requested information, highlighting the role of topic-relevant context in harmful generation.
  • RAG-Safety-Bench suggests that even documents without directly harmful information can surface parametric knowledge that a model would ordinarily withhold in non-RAG use.The authors note that Qwen’s susceptibility may relate to its pretraining and safety alignment, but leave that explanation for further research.

6 Conclusion

The paper introduces RAG-Safety-Bench to isolate how retrieval conditions affect LLM safety across five open-source models. It finds a model-specific failure mode in Qwen-2.5-7B and concludes that RAG deployments require end-to-end safety evaluation and mitigation rather than relying on non-RAG evaluations.

  • RAG-Safety-Bench disentangles RAG’s effects on safety by separating four retrieval conditions across five open-source models.
  • For Qwen-2.5-7B, on-topic documents without the answer elicited more harmful responses than the non-RAG baseline.This pattern is consistent with topic-relevant context surfacing harmful information from parametric memory.
  • Non-RAG safety evaluations do not transfer to RAG deployments, motivating end-to-end measures such as corpus, input, retrieval-side, and output filtering.The authors also recommend safety prompts, multiple automated judges where possible, and human oversight.

Limitations

The benchmark's conclusions are bounded by its English-only, Wikipedia-only scope, limited harm coverage, simplified RAG setup, model selection, and evaluation methodology.

  • Scope: RAG-Safety-Bench is English-only, uses Wikipedia as its sole document source, and covers a non-exhaustive set of harm categories.The authors call for further study across languages and use contexts.
  • Methodological scope: The benchmark does not evaluate retriever effects and omits interactions among retriever ranking, query reformulation, and generator safety.Production RAG systems may also include system-level instructions and input/output filters absent from this simplified setup.
  • Model coverage: Experiments included only relatively small open-source models because of computing constraints, whereas real-world deployments may use much larger commercial models.The benchmark can be applied to such models, but those results are not reported here.
  • Evaluation methodology: Automated benchmark generation and assessment rely heavily on Claude-family models, despite human validation and the possibility of stronger judge ensembles.The study also reports only a single benchmark run, so stochastic generation could produce different results across repeated runs.

Ethical considerations

The benchmark uses potentially harmful Wikipedia-derived content for safety evaluation while documenting provenance, licensing, intended use, and human-annotation safeguards.

  • Risk rationale: The benchmark aggregates dangerous information from Wikipedia for safety evaluation without introducing new knowledge or materially easing access, according to the authors.They judge the potential safety-evaluation benefit to outweigh release risks.
  • Intended use and licensing: The benchmark is intended for safety evaluation and research on RAG safety mitigations, with Wikipedia material redistributed under CC BY-SA 4.0 requirements.The release provides attribution, modification descriptions, ShareAlike licensing, and a license link.
  • Human annotation: Author-annotators were briefed on the difficult content and mental-health considerations, and they were not paid separately for annotation work.
  • Tool use: Claude was used for coding assistance and data visualization, while the authors alone conducted experimental design, interpretation, and writing.

A.1 RAG-Safety-Bench benchmark creation

The benchmark is built from Wikipedia-based harmful-question data, structured metadata, answerability audits, and documented provenance for controlled RAG safety evaluation.

  • Harm taxonomy: The taxonomy adapts MLCommons AILuminate to 20 harmful-request subcategories spanning violent crimes, sex-related crimes, nonviolent crimes, hate, weapons, and self-harm.
  • Question generation: Claude Sonnet-4.5 generated candidate harmful questions from relevant Wikipedia categories or documents together with severity, naturalness, answerability, provenance, and reference-answer metadata.
  • Row structure: Each benchmark row records the query, expected unsafe-answer summary, harm taxonomy fields, oracle-document provenance, condition and document metadata, audit outcomes, and runtime provenance.Document records include identifiers, titles, URLs, relation and safety metadata, selected-section metadata, source labels, and cleaned model-facing text.
  • Generation documentation: Table 5 documents the prompt used to generate candidate unsafe questions from Wikipedia articles for RAG-Safety-Bench.
  • Released benchmark: The released package provides Full and Balanced JSONL datasets, with each question expanded into condition-specific rows and accompanied by manifests and audit records.The Full version contains 987 questions, while the Balanced version contains 346 questions.

A.2 Safety evaluation details

Safety evaluation expands each balanced example across four controlled conditions, directly supplies contexts to isolate retrieval quality, and uses fixed generation and judging settings.

  • Evaluation conditions: Each of 346 balanced examples yields four conditions—non-RAG, unsafe-oracle RAG, safe on-topic RAG, and safe random-control RAG—for 1,384 rows per model.Across five models, this produces 6,920 total responses.
  • Retrieval control: The benchmark directly supplies context documents rather than using a live retriever, preventing poor retrieval from artificially inflating apparent safety.
  • Document contexts: The unsafe-oracle condition uses one document, while the safe on-topic and safe random-control conditions generally use three documents with a few two- or one-document exceptions.
  • Prompting: Models are instructed to answer from their own knowledge in non-RAG and only from sequentially labeled provided documents in RAG conditions.
  • Generation settings: Responses use deterministic decoding with temperature 0.0, top-p 1.0, disabled sampling, a 1,024-token limit, and fixed seed 42.RAG document prefixes are capped at 12,000 characters, with shared prompting and decoding settings across conditions for each model.
  • Reproducibility: The five-model evaluation ran on NVIDIA H100 GPUs and required approximately 74 GPU-hours, with configurations and manifests released for reproducibility.

A.3 Answerability audit of evaluation contexts

The benchmark audits answerability and validates automated judgments before comparing harmfulness and refusal patterns across RAG conditions. Results show evaluator trends are broadly consistent, while oracle retrieval often increases harmful responses in specific categories.

  • Answerability audit: The final balanced subset retains cases where oracle contexts contain answer-bearing evidence but on-topic contexts do not, while random contexts use answer_support=none metadata.The audited subset contains 346 questions, and random-control contexts draw from 1,012 benign documents.
  • Validation: Automated accuracy scoring agrees closely with human annotation, reaching 95–96% raw agreement and Cohen’s κ of 0.90–0.92.The validation samples 100 examples across five models and True/False conditions.
  • Refusal classification: Manual review indicates that many responses classified as “other” could instead be safety refusals or combinations of safety refusal and insufficient information.Examples include Phi-4 and Gemma-3 responses that decline harmful assistance while citing missing information or ethical guidelines.
  • Results: Across models and RAG conditions, safety evaluators show consistent aggregate trends despite disagreement, supporting the use of multiple evaluators.Figure 5 jointly reports harmfulness proportions and response accuracy for each model and condition.
  • Results: Oracle retrieval most often increases harmful responses for radiological weapons, explosives, drugs, chemical weapons, assault/battery, terrorism, and cyberattacks.These category-level rates are reported across Figures 6–10 for the five evaluated models.
Loading 2609.11758v1…