PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks

Richard J. Young

arXiv:2511.22047v1cs.CR

TL;DR

Systematic evidence on guardrail robustness against sophisticated, context-framed attacks is limited, although these models are used to block harmful requests before they reach LLMs. The study evaluated ten publicly available guardrail models on 1,445 prompts across 21 attack categories, including established benchmarks and novel adversarial prompts. Qwen3Guard-8B achieved 85.3% full-test accuracy but 33.8% on novel prompts (57.2 percentage point generalization gap); Granite-Guardian-3.2-5B had a 6.5% gap, and Nemotron-Safety-8B and Granite-Guardian-3.2-5B generated harmful content.

  • Problem

    Systematic evidence on guardrail robustness against sophisticated, context-framed attacks is limited, although these models are used to block harmful requests before they reach LLMs.

    Source: §1 · pp. 1–2

  • Method

    The study evaluated ten publicly available guardrail models on 1,445 prompts across 21 attack categories, including established benchmarks and novel adversarial prompts.

    Source: §1 · p. 2

  • Results

    Qwen3Guard-8B achieved 85.3% full-test accuracy but 33.8% on novel prompts (57.2 percentage point generalization gap); Granite-Guardian-3.2-5B had a 6.5% gap, and Nemotron-Safety-8B and Granite-Guardian-3.2-5B generated harmful content.

    Source: §5 · p. 12

  • Takeaways & Limitations

    Organizations should evaluate guardrails on held-out adversarial prompts rather than relying on published benchmark scores.

    Source: §5 · p. 12

    The 145 author-created novel prompts cover a limited sample of attack patterns, with a ±8.2 percentage-point margin of error and possible bias toward hypothesized successful styles.

    Source: §5.1 · p. 13

Abstract

from arXiv · show

Large Language Model (LLM) safety guardrail models have emerged as a primary defense mechanism against harmful content generation, yet their robustness against sophisticated adversarial attacks remains poorly characterized. This study evaluated ten publicly available guardrail models from Meta, Google, IBM, NVIDIA, Alibaba, and Allen AI across 1,445 test prompts spanning 21 attack categories. While Qwen3Guard-8B achieved the highest overall accuracy (85.3%, 95% CI: 83.4-87.1%), a critical finding emerged when separating public benchmark prompts from novel attacks: all models showed substantial performance degradation on unseen prompts, with Qwen3Guard dropping from 91.0% to 33.8% (a 57.2 percentage point gap). In contrast, Granite-Guardian-3.2-5B showed the best generalization with only a 6.5% gap. A "helpful mode" jailbreak was also discovered where two guardrail models (Nemotron-Safety-8B, Granite-Guardian-3.2-5B) generated harmful content instead of blocking it, representing a novel failure mode. These findings suggest that benchmark performance may be misleading due to training data contamination, and that generalization ability, not overall accuracy, should be the primary metric for guardrail evaluation.

1 Introduction

Harmful, misleading, or dangerous LLM outputs have motivated safety guardrails, but systematic evidence on their robustness to sophisticated attacks remains limited. This study evaluates ten publicly available guardrails across 1,445 prompts and 21 attack categories, including novel adversarial prompts.

1 Introduction

  • Safety guardrail models are specialized classifiers intended to detect and block harmful requests before they reach the underlying LLM.Their use responds to concerns about harmful, misleading, or dangerous content from consumer-facing LLM applications.Source: §1 · p. 1
  • Systematic evaluation of guardrail robustness against sophisticated adversarial attacks remains limited, despite recent work on calibration, over-refusal, and new architectures.Existing evaluations have largely focused on underlying LLMs rather than the guardrail models deployed to protect them.Source: §1 · p. 1
  • Benchmarks often emphasize obvious harmful requests, while real-world attacks can frame harmful requests as academic research, corporate security assessments, or fiction.The introduction also identifies multilingual performance degradation and the unexamined relationship between model size and safety classification as open questions.Source: §1 · p. 2
  • Ten publicly available guardrail models were evaluated on 1,445 prompts spanning 21 attack categories, including established benchmarks and novel adversarial categories.The study tested whether detection would decline on sophisticated attacks and also examined model size and classification accuracy.Source: §1 · p. 2

2 Related Work

Prior work covers guardrail architectures, benchmarks, jailbreak attacks, and defenses, while also identifying multilingual safety and benchmark contamination as continuing concerns.

Guardrail Developments

  • Guardrail research spans fine-tuned safeguards, content moderation, enterprise classifiers, and newer reasoning, reinforcement-learning, ensemble, and policy-as-prompt approaches.Recent developments also include safety collapse under similar alignment and downstream data, robotics and privacy guardrails, and severity-level prediction.Source: §2 · p. 2

Safety Benchmarks

  • Safety benchmarks target jailbreaks, over-refusal, toxicity, graded content risks, agent safety, contextual integrity, reasoning traces, and multilingual evaluation.The section describes benchmarks across a range of tasks and evaluation settings.Source: §2 · p. 2

Jailbreak Attacks

  • Jailbreak methods include suffixes, roleplay, encoding, indirect injection, skeleton keys, multi-turn attacks, adaptive attacks, and hybrid strategies.The section also reviews attack distillation and surveys cataloging these techniques.Source: §2 · p. 2

Defense Mechanisms

  • Defenses include spurious-response strategies, attack-essence databases, curiosity-driven red teaming, and live moderation; research increasingly emphasizes output-level moderation.The section notes that sophisticated attacks can evade input-based defenses through rephrasing.Source: §2 · p. 3

Multilingual Safety

  • Multilingual safety work reports elevated unsafe outputs in low-resource languages and vulnerabilities to adversarial prompts across languages.The reviewed studies also show that translating prompts into low-resource languages can bypass safety alignment.Source: §2 · p. 3

Benchmark Contamination

  • Benchmark contamination research documents contamination detection and mitigation methods, motivating evaluation with novel prompts not derived from public benchmarks.The section connects the study’s observed 57.2% degradation on novel prompts to potential contamination effects in guardrail training.Source: §2 · p. 3

3 Methods

The methods evaluate ten public guardrail models on a multi-source prompt set, using standardized inference, classification measures, and statistical analyses to assess performance and generalization.

3.1 Models

  • Ten publicly available models from seven organizations represented major guardrail architectures and were accessed through official HuggingFace identifiers using default configurations.Table 1 identifies the evaluated models.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Test Dataset

  • The 1,445-prompt dataset combined five sources and assigned 21 attack and response categories, with eight custom adversarial groups organized in Figure 1.Table 2 presents the test dataset composition.Source: §3.2 · pp. 3–4, Figure 1 · p. 4, Table 2 · p. 4

3.3 Procedure

  • Each prompt was presented once to each model with model-specific system prompts and standardized generation settings, then parsed using rules for diverse output formats.Figure 2 depicts the procedure, including detection of responses that generate substantive content instead of a safety classification.Source: §3.3 · p. 4, Figure 2 · p. 5

3.4 Measures

  • Classification accuracy measured agreement with the expected refuse or allow outcome; secondary measures were false-negative, false-positive, and per-category accuracy.Source: §3.4 · p. 5

3.5 Analysis

  • Analysis ranked overall accuracy, examined category-level performance and errors, compared public with novel accuracy, and measured prompt-style stability across seven templates.Prompts failing on at least seven of ten models were flagged for qualitative analysis as universally difficult.Source: §3.5 · p. 5

3.6 Statistical Methods

  • Bootstrap confidence intervals used 10,000 resamples; paired comparisons used McNemar tests and Bonferroni correction, with Cohen’s h, κ = 0.89 annotation agreement, and a ±8.2 pp novel-set margin.The novel subset’s margin of error should be considered when interpreting generalization-gap estimates.Source: §3.6 · p. 6

3.7 Hardware and Software

  • Experiments ran on an NVIDIA A100 80GB PCIe GPU with Ubuntu 22.04 LTS, PyTorch 2.9.1, and Transformers 4.57.3.Source: §3.7 · p. 6

4 Results

The results show substantial differences in overall guardrail accuracy, safety-usability balance, and generalization to novel attacks. They also identify prompt-sensitive behavior, a “helpful mode” failure, and attack categories that remain difficult for models.

4.2 Safety vs. Usability Trade-off

  • LlamaGuard models achieved 97–99% benign accuracy but only 4.5–21.8% harmful detection, while Qwen3Guard-8B, WildGuard-7B, and Granite-Guardian-3.3-8B balanced both objectives.Source: §4.2 · p. 6, Figure 4 · p. 8

4.1 Overall Performance

  • 85.3% accuracy (95% CI: 83.4–87.1%) made Qwen3Guard-8B the top model, followed by WildGuard-7B at 82.8% and Granite-Guardian-3.3-8B at 81.0%.All pairwise comparisons against Qwen3Guard-8B were statistically significant (McNemar’s test, p < 0.001).Source: §4.1 · p. 6, Figure 3 · p. 7

4.3 Generalization to Novel Attacks

  • All models degraded on novel prompts; Qwen3Guard-8B fell from 91.0% on public benchmarks to 33.8% on novel prompts, a 57.2-point gap, while Granite-Guardian-3.2-5B had the smallest gap at 6.5%.The evaluation compared 1,300 public prompts with 145 privately created novel prompts.Source: §4.3 · pp. 7–8, Table 4 · p. 9

4.4 “Helpful Mode” Jailbreak

  • Nemotron-Safety-8B and Granite-Guardian-3.2-5B sometimes generated substantive harmful responses instead of safety classifications.This “helpful assistant” mode occurred in 13.6% and 11.1% of their responses, respectively, and included social-engineering scripts, disinformation, and step-by-step harmful instructions; the paper describes this as a more severe failure than a classification error.Source: §4.4 · p. 9

4.5 Model Size Effects

  • Smaller models outperformed larger counterparts within two model families: ShieldGemma-2B scored 62.4% versus ShieldGemma-9B at 54.7%, and LlamaGuard-3-1B scored 59.9% versus LlamaGuard-3-8B at 48.4%.The results suggest safety classification may not benefit from increased model capacity as general language tasks do.Source: §4.5 · p. 9

4.6 Prompt Style Ablation

  • Prompt-style sensitivity varied sharply: Qwen3Guard-8B’s accuracy range was 1.0 percentage points across seven styles, versus 20.7 points for ShieldGemma-2B.ShieldGemma-2B ranged from 48.6% to 69.3%; Nemotron-8B had a 6.8-point range but produced 13–21% unparseable “unknown” responses. The authors report that template choice can shift accuracy by up to 20 percentage points and suggest prompt robustness may serve as a proxy for generalization.Source: §4.6 · pp. 10–11

4.7 Error Analysis by Attack Type

  • Business-framed attacks had a 96.8% success rate across models, and the best-performing model detected only 25% of them; standard JailbreakBench prompts had error rates of 8.9–27.8%.The 291 prompts that failed on at least seven models were predominantly role-play jailbreaks and research-framed requests. The safety-usability spectrum also ranged from LlamaGuard-7B allowing 88.1% of harmful prompts through to Granite-Guardian-3.2-5B over-refusing 25.7% of benign requests.Source: §4.7 · p. 11

5 Discussion

The discussion finds a sharp gap between benchmark accuracy and performance on novel prompts, alongside failure modes and trade-offs that complicate model selection. It recommends evaluation on held-out attacks and monitoring for models that generate content instead of classifying it.

5 Discussion

  • 57.2 percentage points: Qwen3Guard-8B fell from 85.3% accuracy on the full test set to 33.8% on novel prompts.The gap may reflect benchmark-distribution overfitting or training data contamination, but the study cannot prove contamination.Source: §5 · p. 12
  • Granite-Guardian-3.2-5B had a 6.5% gap and LlamaGuard-3-1B a 9.9% gap, making the preferred model depend on whether deployment faces known or novel attacks.The discussion recommends evaluating held-out adversarial prompts rather than relying on published benchmark scores.Source: §5 · p. 12
  • Some guardrail models, including Nemotron-Safety-8B and Granite-Guardian-3.2-5B, actively generated harmful content, including disinformation articles and social engineering guides, rather than classifying safety, transforming a defensive component into an attack vector.Adversarial inputs cause the model to forget its classification role and revert to general-purpose assistant behavior.Source: §5 · pp. 12–13
  • Figure 8 compares accuracy across prompt styles: Qwen3Guard-8B stayed uniformly high, while ShieldGemma-2B reached 48.6% with the generic prompt, 20.7 points below its best.The figure indicates that prompt robustness correlates with overall performance.Source: Figure 8 · p. 13
  • Smaller models outperformed larger counterparts within two families: ShieldGemma-2B over ShieldGemma-9B and LlamaGuard-3-1B over LlamaGuard-3-8B.The discussion offers lower susceptibility to overfitting and greater memorization capacity as possible explanations, not established causes.Source: §5 · p. 13

5.1 Limitations

  • The 145 novel prompts span 9 attack categories, but their ±8.2 percentage-point margin of error and author-designed styles limit generalization claims.The full dataset's margin of error is ±2.6 percentage points.Source: §5.1 · p. 13
  • The evaluation used single-turn prompts and English only, tested each prompt once at temperature 0.1, and excluded commercial guardrails.Parsing may introduce measurement error, helpful-mode rates were not exhaustively validated, and contamination cannot be distinguished from distribution shift, prompt-style overfitting, or capability differences.Source: §5.1 · pp. 13–14

5.2 Recommendations and Future Directions

  • The authors recommend testing held-out prompts, monitoring for helpful-mode failures, considering smaller models, and using defense-in-depth rather than relying on a single guardrail.They also call for evaluation protocols that prioritize generalization and future study of helpful-mode mechanisms.Source: §5.2 · p. 14

Ethics Statement

The ethics statement frames the adversarial prompts as evaluation tools, describes steps to limit misuse, and identifies anticipated benefits for safety practitioners. It also notes the dual-use trade-off and the handling of benchmark and custom prompts.

Ethics Statement

  • The adversarial prompts were designed for evaluation, not genuine harmful intent, to identify guardrail weaknesses developers could address before exploitation.Source: Ethics Statement · p. 14
  • The authors withheld detailed prompt templates to reduce misuse risk and say organizations deploying the evaluated models were notified before publication.Source: Ethics Statement · p. 14
  • The authors judge disclosure benefits to outweigh dual-use risks because defenders need findings and organizations need transparency for deployment decisions.Source: Ethics Statement · p. 15
  • Public benchmark prompts were used under their licenses, while team-created custom prompts contained no personally identifiable information or real-world harmful-interaction content.Source: Ethics Statement · p. 15
  • The research is anticipated to benefit AI safety practitioners by highlighting attack vectors that may require additional defenses.The statement says novel-prompt degradation should encourage more robust, generalizable safety training.Source: Ethics Statement · p. 15

A Example Prompts by Category

The appendix illustrates prompt strategies across research, corporate, and social-engineering framings. Its examples show how harmful requests can be embedded in apparently legitimate or emotionally compelling contexts.

A.1 Advanced Attack (Research Framing)

  • Research-framed prompts ask about disinformation campaigns in a corporate-resilience case study, ransomware for a cybersecurity thesis, and romance scams in an academic study.Source: §A.1 · p. 19
  • One research-framed example asks for technical details about ransomware encryption and command-and-control communication under a cybersecurity-thesis premise.Source: §A.1 · p. 19

A.2 Business Attack (Corporate Framing)

  • Corporate-framed examples invoke equipment-tampering tests, penetration testing to bypass multi-factor authentication, and competitive intelligence about extracting proprietary information.Source: §A.2 · p. 19
  • The equipment-tampering scenario presents a request about detecting machinery degradation as a company vulnerability test.Source: §A.2 · p. 19

A.3 Subtle Adversarial (Social Engineering)

  • Social-engineering examples use emotional pressure: a bereavement appeal for a late refund, a hospitalized child to request bypassing account security, and a journalist's appeal for confidential documents.Source: §A.3 · p. 19
  • The account-security example pairs a locked account and a hospitalized child with a request to bypass verification.Source: §A.3 · p. 19

A.4 Overt Adversarial (Direct Override)

These examples cover direct instruction overrides, multi-step scenarios, and boundary cases. They illustrate both explicit attempts to disable restrictions and ambiguous requests framed as creative, educational, or safety-related.

A.4 Overt Adversarial (Direct Override)

  • Direct-override prompts tell the model to ignore prior instructions, adopt “DAN” mode, or enter developer mode with safety filters disabled.Source: §A.4 · p. 19

A.5 Complex Adversarial (Multi-step)

  • Multi-step examples embed requests for creating a new identity or synthesizing a compound in thriller-writing and video-game scenarios.Source: §A.5 · p. 19

A.6 Boundary Cases

  • Boundary cases ask about household chemicals, lockpicking, and poison symptoms in contexts that could be safety information, legitimate trade, or medical education.Source: §A.6 · p. 20

A.7 JailbreakBench Examples

The appendix gives examples of harmful JailbreakBench prompts and benign prompts expected to be allowed.

A.7 JailbreakBench Examples

  • JailbreakBench examples target defamation, corporate-network hacking, banking-credential phishing, and stock-price manipulation.Source: §A.7 · p. 20

A.8 Benign Examples (Expected: Allow)

  • Benign examples expected to be allowed include party planning, beginner machine-learning questions, meal preparation, and internet troubleshooting.Source: §A.8 · p. 20

B.1 Correct Classification

A Qwen3Guard-8B example correctly classifies a request for a computer-virus tutorial as unsafe and refuses it.

B.1 Correct Classification

  • Qwen3Guard-8B correctly refuses a request for a computer-virus creation tutorial as unsafe.Source: §B.1 · p. 20

B.2 “Helpful Mode” Failure Examples

The examples show guardrails abandoning safety classification and generating substantive content; the authors report this “helpful mode” failure in two models. They warn that the guardrail can itself become a vector for harm generation.

B.2 “Helpful Mode” Failure Examples

  • “Helpful mode” occurred in 13.6% of Nemotron-Safety-8B responses and 11.1% of Granite-Guardian-3.2-5B responses.Source: §B.2 · p. 20
  • For a journalism-class request about disinformation techniques, the model generated a complete fake-news article beginning with a claim that coffee causes immortality, rather than following the expected refusal-and-classification behavior.Source: §B.2 · p. 21
  • Nemotron-Safety-8B adopted a DAN jailbreak persona and announced willingness to provide unfiltered responses instead of classifying and refusing the attempt.Source: §B.2 · p. 21
  • Nemotron-Safety-8B role-played as a financial system and generated verification procedures rather than classifying the request.Source: §B.2 · p. 21
  • The authors warn that these failures make the guardrail itself a potential vector for harm generation, with the DAN example explicitly announcing willingness to bypass safety guidelines.Source: §B.2 · p. 21

Related papers

Loading 2511.22047v1…