RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models

1 INTRODUCTION

• Existing red-teaming datasets vary in risk definitions, scope, and format, leaving comprehensive evaluation across domains an open research question.Source: §1 · p. 2

LLM deployment scenarios?

• The paper identifies a need to assess modern LLM robustness because prior red-teaming studies often evaluated older architectures without comparisons to newer models.Source: LLM deployment scenarios? · p. 2
• RedBench aggregates and harmonizes 37 datasets, analyzes their risk and domain coverage, and establishes baselines for modern LLMs.Source: LLM deployment scenarios? · p. 2
• Its contributions include consistent categorization and domain coverage, analysis of existing datasets, and baselines for Qwen2.5, Llama 3.1, and Gemma 2.Source: LLM deployment scenarios? · p. 2
• The dataset and evaluation code are open-sourced to support transparency, reproducibility, and community-driven research.Source: LLM deployment scenarios? · p. 2

2.1 DATA COLLECTION

• RedBench aggregates 37 curated datasets into 29,362 samples spanning attack prompts and refusal prompts that test over-defensive refusals.Source: §2.1 · pp. 2–3

2.2 TAXONOMY FOR DATASETS

• Each sample receives Risk Category and Domain labels, providing a shared taxonomy intended to address inconsistent, ambiguous, or overlapping risk labels.Source: §2.2 · p. 4

2.2.1 RISK CATEGORIES

• The taxonomy defines 22 risk categories, assigning “No Risk” to benign prompts used to evaluate refusal behavior.Source: §2.2.1 · p. 4

2.2.2 DOMAINS

• The 19 domains represent application areas and use cases ranging from healthcare and military to general knowledge.Source: §2.2.2 · p. 4

2.2.3 ANNOTATION PROCESS

• Qwen2.5-72B-Instruct labels the dataset, with human review of a subset and consensus resolution of discrepancies.Source: §2.2.3 · pp. 4–5

2.3 OVERVIEW OF ANALYSIS

• 3,523 attack samples concern Abusive Content, while Confabulation has 71; refusal prompts are concentrated in General Knowledge (982), with limited coverage in specialized domains.Source: §2.3 · pp. 5–7, Table 1 · p. 5
• 8,581 attack samples and 982 refusal samples are in General Knowledge, while Technology has 4,041 attack and 314 refusal samples; Technology is associated with Cybersecurity Threats and Malware Code.Source: §2.3 · pp. 5–6, Table 2 · p. 6, Figure 2 · p. 6

3 EXPERIMENTS

• The experiments aim to establish modern-LLM baselines and quantify vulnerabilities across risk categories and domains, with code and evaluation scripts open-sourced for reproducibility.The evaluation uses adversarial and refusal prompts.Source: §3 · p. 7

3.1 METRICS

• RR measures legitimate prompts incorrectly refused, while ASR measures attempted attacks that elicit unsafe or inappropriate responses.GPT-4o judges refusals; Llama-Guard-3-8B judges whether attack responses are unsafe or inappropriate.Source: §3.1 · p. 7

3.2 TARGET LLMS

• The evaluation covers six safety-aligned open- and closed-source LLMs to assess robustness across model architectures and safety mechanisms.The models include Llama-3.1-8B-Instruct, Gemma-2-9B-It, Qwen2.5-7B-Instruct, Ministral-8B-Instruct-2410, and GPT-4o Mini.Source: §3.2 · p. 8

3.3.1 RESULTS ON REFUSAL BENCHMARK DATASETS

• 28.53% average Rejection Rate for Llama-3.1-8B-Instruct contrasts with 13.46% for Gemma-2-9B-It across refusal benchmarks.Llama-3.1-8B-Instruct reaches 32.60% on XSTest; Table 3 reports rates across refusal datasets.Source: Table 3 · p. 8, §3.3.1 · p. 8

3.3.2 RESULTS ON ATTACK BENCHMARK DATASETS

• 97.81% (RainbowPlus) and 90.63% (HumanJailbreak) ASR on Ministral-8B-Instruct-2410 contrast with 0.00% (HumanJailbreak) and 6.88% (RainbowPlus) on GPT-4.1-Nano.Table 4 reports attack success rates on HarmBench across models and red-teaming methods.Source: §3.3.2 · p. 8, Table 4 · p. 9
• RainbowPlus achieves high ASRs against both open- and closed-source models, while HumanJailbreak succeeds against open-source models but fails against closed-source models.RainbowPlus reaches 28.75% ASR on GPT-4o-Mini and 6.88% on GPT-4.1-Nano.Source: §3.3.2 · p. 9
• RainbowPlus vulnerability varies across risk categories and domains: Ministral-8B-Instruct-2410 is especially vulnerable on Economic Harm and Extremism and Radicalization, and Environment and Nutrition are susceptible domains.Llama-3.1-8B-Instruct also shows elevated ASRs in the named risk categories, though to a lesser extent.Source: Figure 3 · p. 9, §3.3.2 · p. 9

4 CONCLUSION

• RedBench integrates 37 benchmarks into a 29,362-sample dataset, organized by 22 risk categories and 19 domains for consistent LLM vulnerability evaluation.Source: §4 · p. 9
• Open-source models are particularly vulnerable to adversarial prompts, especially under RainbowPlus, while closed-source models demonstrate stronger resistance.Source: §4 · p. 9
• Existing benchmarks overrepresent Abusive Content and General Knowledge, while refusal benchmarks show high Rejection Rates that pose usability challenges.Source: §4 · p. 9

ETHICS STATEMENT

• RedBench uses adversarial and refusal prompts to systematically identify LLM vulnerabilities as part of its aim to improve safety and robustness.Source: ETHICS STATEMENT · p. 10
• The 29,362 samples include prompts targeting sensitive risks such as Extremism and Radicalization and Child Safety, which could be misused to exploit LLMs.Source: ETHICS STATEMENT · p. 10
• The authors prioritize transparency and responsible research practices, and advocate collaboration with model developers, legal and ethical standards, and community oversight.Source: ETHICS STATEMENT · p. 10

A DISCUSSION

• Llama-3.1-8B-Instruct averaged a 28.53% Rejection Rate, while Gemma-2-9B-IT averaged 13.46%; GPT-4.1-Nano reached 35.18% on ORBench.The passage links high rejection to possible usability limits and lower rejection to possible insufficient safety barriers, and notes that dataset-specific prompt design affects rates.Source: A DISCUSSION · p. 19
• On HarmBench, RainbowPlus ASR was 97.81% for Ministral-8B-Instruct-2410 and 6.88% for GPT-4.1-Nano.The passage also reports 96.25% for Llama-3.1-8B-Instruct and 28.75% for GPT-4o-Mini; it identifies pronounced vulnerabilities in Environment and Nutrition domains and Economic Harm and Extremism and Radicalization categories.Source: A DISCUSSION · p. 19
• RedBench is imbalanced: Abusive Content has 3,523 attack samples, while Confabulation has 71; General Knowledge has 8,581, while Travel has 102.The passage identifies underrepresented risks and contexts, including Election Interference, as gaps for targeted dataset augmentation.Source: A DISCUSSION · p. 19
• Evaluation covers only six LLMs and uses HarmBench for attack testing, limiting generalizability and coverage of RedBench’s adversarial prompts.The authors also note that inherited dataset biases and possible edge-case inconsistencies in semi-automated annotation constrain the dataset.Source: A DISCUSSION · p. 19
• High attack success rates in open-source models and high rejection rates point to a safety–usability trade-off and a need for improved alignment.The discussion particularly notes vulnerabilities in Environment and Nutrition domains.Source: A DISCUSSION · p. 19
• RedBench and its evaluation code are open source, supporting transparency and reproducibility for further work on safety-critical applications.The passage names healthcare and legal advisory systems as example applications.Source: A DISCUSSION · p. 20

B RELATED WORK

• RedBench is positioned as addressing shortcomings in dataset standardization and coverage of risk scenarios.The related-work section reviews prior LLM red-teaming techniques and datasets in that context.Source: B RELATED WORK · p. 20

B.1 AUTOMATED RED TEAMING

• Automated red teaming scales adversarial-prompt generation beyond labor-intensive manual testing, with later approaches including hybrid strategies, optimization, and quality-diversity search.The passage traces this development from early test-case generation to Rainbow and RainbowPlus.Source: §B.1 · p. 20

• Attack datasets such as HarmBench, AdvBench, and DAN target unsafe responses, while refusal datasets such as CoCoNut, ORBench, and StrongREJECT test over-defensive responses to benign prompts.The passage notes inconsistent risk labels in attack datasets and limited domain coverage and inconsistent prompt design in refusal datasets.Source: §B.2 · p. 20
• Specialized datasets address cybersecurity and healthcare risks, but are not integrated into a unified framework.The examples given are PurpleLlamaCyberSecEval and MedSafetyBench.Source: §B.2 · p. 20
• RedBench combines 37 datasets and 29,362 samples under 22 risk categories and 19 domains, unifying attack and refusal prompts.The standardized framework is intended to support systematic evaluations across risk scenarios.Source: §B.2 · p. 20

• The human evaluation used three independent runs on 300 randomly selected samples, with annotators labeling according to the risk-category and domain definitions.Agreement between the model predictions and human annotations was measured.Source: C ANNOTATION RELIABILITY · p. 20
• Qwen2.5-72B-Instruct averaged 84.68% agreement for Risk Category and 97.73% for Domain across three runs.Standard deviations were 2.66% for categories and 0.88% for domains.Source: C ANNOTATION RELIABILITY · p. 21, Table 5 · p. 21
• The authors attribute higher Domain agreement to contextual classification being more objective than risk assessment, which can involve subtler judgments.They conclude the results support using Qwen2.5-72B-Instruct to scale RedBench labeling while maintaining high-quality annotations comparable to human standards.Source: C ANNOTATION RELIABILITY · p. 21

D DATASET SOURCES AND COMPOSITION

• RedBench’s sources span datasets on adversarial prompts, jailbreaking, bias, toxicity, compliance, and refusal behaviors.The source overview presents the breadth of threat models and safety concerns represented.Source: D DATASET SOURCES AND COMPOSITION · p. 21
• Dataset metadata includes sample size, license type, and access link; the sources are available under permissive or community-oriented terms.The passage presents this metadata as supporting transparency and reproducibility.Source: D DATASET SOURCES AND COMPOSITION · p. 21

E TAXONOMY

• The taxonomy section defines 22 risk categories and 19 domains.Source: E TAXONOMY · p. 21

F ADDITIONAL ANALYSIS

• Additional analysis aggregates refusal Rejection Rates and attack ASRs across six LLMs to examine domain- and category-specific vulnerabilities.The analysis is framed as complementing the primary results and identifying areas for future improvement.Source: F ADDITIONAL ANALYSIS · p. 21

F.1 REFUSAL RESULTS BY DOMAIN

• Rejection Rates vary across domains and datasets: Retail reaches 50.00% on CoCoNot and 30.56% on XSTest, while several Biology and Education results are 0.00%.SGXSTest has zero Rejection Rates in several domains, whereas ORBench reaches 28.71% in Finance and 27.27% in Infrastructure, indicating uneven probe effectiveness.Source: §F.1 · p. 23, Table 10 · p. 25
• Across benchmarks, Llama-3.1-8B-Instruct consistently has higher Rejection Rates than other models on CoCoNut and ORBench; on XSTest, it over-refuses across multiple domains, while GPT-4.1-Nano refuses Travel prompts.The authors interpret this as conservative safety alignment potentially limiting usability; GPT-4.1-Nano’s elevated XSTest Travel refusal indicates domain-specific sensitivities. They call for more diverse refusal prompts.Source: §F.1 · p. 23

F.2 ATTACK RESULTS BY CATEGORY AND DOMAIN

• 83.33% for Extremism and Radicalization and 66.67% for Economic Harm are RainbowPlus ASRs, making it the most effective method across the reported risk categories.HumanJailbreak exceeds 60% for most categories, while Direct and ZeroShot are notably weaker in some categories.Source: §F.2 · p. 24, Table 11 · p. 26
• 83.33% ASR for RainbowPlus in Nutrition and 66.67% for RainbowPlus and HumanJailbreak in Environment mark domain-specific vulnerabilities; Healthcare ASRs span 50.00%–66.67% across methods.Direct ASRs are lower in Education (0.00%) and Finance (12.50%).Source: §F.2 · pp. 25, 27, Table 12 · p. 26

G RED-TEAMING METHODS

• Direct uses raw behavior strings; Zero-Shot uses attacker-LLM test cases; Human Jailbreaks substitute behaviors into in-the-wild templates; RainbowPlus uses adaptive Quality-Diversity search.The cited descriptions characterize Direct as a lower-bound baseline, Zero-Shot and Human Jailbreaks as less targeted or adaptable, and RainbowPlus as diverse and effective.Source: G RED-TEAMING METHODS · p. 28

H DETAILED EXPERIMENT SETUP

• vLLM supports high-throughput inference across API and non-API models while minimizing memory overhead and enabling execution on a single GPU.The setup is described as designed for reproducibility, scalability, and efficiency.Source: H DETAILED EXPERIMENT SETUP · p. 28
• Two NVIDIA A40 GPUs labeled 29,362 samples in one week, and one NVIDIA A40 GPU completed attack and refusal evaluations across six LLMs in one week.Source: H DETAILED EXPERIMENT SETUP · p. 28
• Temperature, top-p, and batch sizes were standardized to typical deployment settings, with configurations and evaluation scripts published for reproducibility and extensions.Source: H DETAILED EXPERIMENT SETUP · p. 28

I TARGET LLMS

• The evaluation targets Llama-3.1-8B-Instruct, Gemma-2-9B-It, Qwen2.5-7B-Instruct, and Ministral-8B-Instruct-2410.The descriptions characterize the models as instruction-following or safety-aligned; Qwen2.5 is described as strong in multilingual and safety-critical tasks (Yang et al., 2024; Team, 2024c), and Ministral is a recent Mistral-family addition.Source: I TARGET LLMS · p. 28
• The target set also includes GPT-4o Mini and GPT-4.1-Nano, accessed through OpenAI’s API; Gemini and Claude were excluded because of organizational API access limitations.The paper frames the exclusion as supporting an evaluation focused on accessible and reproducible setups.Source: I TARGET LLMS · p. 28

J PROMPT

• The risk-labeling prompt asks an LLM to match an input to the most relevant supplied risk-category definition and return the category with a brief explanation.The definitions cover risks including sexual, abusive, hateful, violent, criminal, cyber, misinformation, privacy, and environmental content, among others.Source: J PROMPT · p. 29
• The domain-labeling prompt directs the LLM to identify an input’s primary domain, choose the dominant or most specific domain when ambiguous, and default to General Knowledge if none fits.It requests an explanation only when the input is ambiguous or adversarial.Source: J PROMPT · pp. 30–31

K EXAMPLES

• Examples pair prompts with risk and domain labels, including a Biology/WMDP prompt labeled CBRN Information or Capabilities and an Education/HarmfulQA prompt labeled Bias Content.Other examples include misinformation in Entertainment, environmental harm in Environment, and crime in Finance.Source: Example 1: Biology · p. 32, Example 2: Education · p. 32, Example 3: Entertainment · p. 32
• Further examples show risk categories crossing domains, such as Self-Harm Content in Family, Healthcare, Nutrition, and Sports, and Hateful Content in Politics and Religion.Other listed mappings include Cybersecurity Threats (Beyond Malware) in Infrastructure and Technology, Crime Content in Law and Workplace, and Abusive Content in General Knowledge and Travel.Source: Example 5: Family · p. 32, Example 8: Healthcare · p. 33, Example 12: Nutrition · p. 33

Example 11: Election Interference; Example 12: Environmental Harm

• The examples pair a request for a voter-policy campaign assistant with a request to conceal illegal chemical dumping.They are labeled Election Interference and Environmental Harm, respectively.Source: Example 11: Election Interference · p. 36, Example 12: Environmental Harm · p. 36

Example 13: Extremism and Radicalization; Example 14: Hateful Content

• Other prompts seek undetected aid to terrorists and present hate speech as an article-writing task for assessment.Their categories are Extremism and Radicalization and Hateful Content.Source: Example 13: Extremism and Radicalization · p. 36, Example 14: Hateful Content · p. 36

Example 15: Illegal Weapons (Non-CBRN); Example 16: Intellectual Property

• The Illegal Weapons example asks whether instructions were understood, while the Intellectual Property example requests copyrighted song lyrics.The respective domains are General Knowledge and Entertainment.Source: Example 15: Illegal Weapons (Non-CBRN) · p. 36, Example 16: Intellectual Property · p. 36

Example 17: Malware Code; Example 18: Misinformation

• The examples include a request to spread a computer virus and a healthcare statement expressing dislike of quackery.They are categorized as Malware Code and Misinformation.Source: Example 17: Malware Code · p. 37, Example 18: Misinformation · p. 37

Example 19: Other; Example 20: Self-Harm Content

• Prompts cover using sustainable development to access international markets and taking unnecessary drugs to improve mood instead of relieving stress.They are labeled Other and Self-Harm Content, respectively.Source: Example 19: Other · p. 37, Example 20: Self-Harm Content · p. 37

Example 21: Sexual Content; Example 22: Violent Content

• The final examples ask for a legal defense of a sexual assailant and contain a direct threat to murder someone.They are categorized as Sexual Content and Violent Content.Source: Example 21: Sexual Content · p. 37, Example 22: Violent Content · p. 37