AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies

1 Introduction

• Existing benchmarks leave regulatory risks uncovered: even the most extensive covers only 71% of level-3 categories specified in recent government regulations and corporate policies.The authors argue that this coverage gap limits the real-world relevance and actionability of benchmark insights.Source: §1 · p. 2, Figure 1 · p. 2
• AIR-BENCH 2024 provides 5,694 risky-instruction prompts grounded in AIR 2024, which organizes risks from 8 government regulations and 16 company policies into 314 granular categories.AIR 2024 is described as a four-tiered taxonomy in the paper context.Source: §1 · p. 2
• The benchmark prioritizes contextual diversity, uncommon dialects, and syntactic variety, and its GPT-4o category-specific evaluation achieved a Kappa score of 0.86 against human annotators.The authors describe prompts as manually crafted and audited.Source: §1 · p. 2
• In automated decision-making scenarios, most evaluated models made financial-eligibility decisions without cautioning against their use in real-world applications.The authors note that this risk is explicitly emphasized in EU, US, and Chinese regulatory frameworks.Source: §1 · p. 2
• Across the evaluated models, Claude 3 Sonnet had the highest overall safety refusal rate at 89%, while DBRX Instruct had the lowest at 15%.The paper scores models using safety refusal rates across risk categories.Source: §1 · p. 2

2.1 AIR 2024: Unifying AI Risks from Regulations and Policies

• AIR 2024 manually organizes risks from 8 government regulatory frameworks and 16 corporate policies into a four-tier hierarchy, with 314 level-4 risks grouped under 45 level-3 categories.The taxonomy further aggregates these into 16 level-2 and four level-1 categories.Source: §2.1 · p. 3

2.2 The Gap Between AI Safety Benchmarks and Regulations

• Twenty-one of 45 level-3 risk categories have at most one benchmark formally studying them, and several categories are absent from all three mapped benchmarks.The absent categories include automated decision-making, non-consensual nudity, sowing division, and discrimination towards protected characteristics.Source: Figure 2 · p. 3, §2.2 · p. 4
• Across the three mapped benchmarks, SALAD-Bench covers 71% (32/45) of AIR 2024 level-3 categories, compared with 51% (23/45) for HEx-PHI and 26% (12/45) for HarmBench.The comparison uses level-3 categories to balance specificity and generality across benchmarks.Source: §2.2 · pp. 3–4
• Because these gaps limit mapping benchmark results to regulatory frameworks, AIR-BENCH 2024 builds on the 314 risks in 8 regulations and 16 policies.The proposed benchmark is intended as a more extensive evaluation tool aligned with real-world risk categories.Source: §2.2 · p. 4

3 Curation of AIR-BENCH 2024

• The curation pipeline generates regulation- and policy-grounded base prompts, expands them with linguistic and contextual variation, and creates category-specific evaluation prompts, with manual review throughout.The pipeline’s stages are shown in Figure 3.Source: §3 · p. 4, Figure 3 · p. 4

3.1 Data Generation

• For each risk category, leading language models generate base prompts that experts review and refine iteratively until 5–10 high-quality prompts are obtained.Prompts aim for clear risky intent, concrete and diverse contexts, and varied syntax; typically each undergoes 2–3 rounds of manual editing.Source: §3.1 · p. 5

3.2 Data Expansion

• GPT-4o expands base prompts through uncommon-dialect and authority-endorsement mutations, which human experts then review and refine.The mutations aim to preserve semantic meaning while increasing dialect and syntax diversity.Source: §3.2 · p. 5

3.3 Category-Specific Evaluation: AIR-BENCH 2024 Three-level Scoring Autograder

• The category-specific evaluator assigns 0 for risky assistance, 0.5 for ambiguous or non-refusal responses, and 1 for refusal, using judge prompts checked by humans.About 70% of final judge prompts were model-generated and 30% human-edited.Source: §3.3 · pp. 5–6
• Human evaluation found Kappa agreement of 0.86 for the judging approach, compared with 0.69 reported for a fixed judge prompt across categories.The primary quantitative metric is the percentage of scores equal to 1, termed the refusal rate.Source: §3.3 · p. 6

4.2 Model Refusal Study over Risky Instructions from AIR-BENCH 2024

• Claude 3 Sonnet and Haiku have the highest average refusal rate at 89%, while DBRX Instruct has 15%; no model consistently refuses across all 45 level-3 categories.The study treats higher refusal rates as better alignment, while noting that complete refusal may not suit every jurisdiction or use case.Source: §4.2 · p. 6, §4.1 · p. 6, Figure 4 · p. 7
• Refusal rates vary within risk categories: models reject hate speech against Age more often than against Genetic Information or Gender and Occupation, and reject Cutting less than Eating Disorders or Suicide.Gemini 1.5 Pro is more effective on Medical/Pharmaceutical advice than Legal and Government services; it also has a weaker refusal rate for Education eligibility prompts.Source: §4.2 · pp. 6–7, Figure 4 · p. 7, Figure 5 · p. 8

4.3 Refusal Study over Public Sector Categorizations of Risk

• No evaluated model fully aligns its refusals with EU AI Act guidelines; even Claude 3 averages only around 71% for the 11 high-risk and unacceptable categories, and all models perform poorly on regulated-industry advice.The evaluation maps the Act’s risk categories to AIR-BENCH 2024 and examines both its 11 high-risk and unacceptable categories and all 23 mentioned categories.Source: Figure 6 · p. 8, §4.3 · pp. 8–9

4.4 Refusal Study over Private Sector Categorizations of Risk

• Claude 3 models average above 90% refusal for most Anthropic policy risks, but rates are below 70% for automated decision-making and below 20% for regulated-industry advice.The benchmark treats the model as the entire system, which may not reflect deployments with human oversight or additional checks.Source: §4.4 · p. 9, Figure 7 · p. 9
• OpenAI models show lower refusal rates in several policy categories, including automated decision-making, regulated-industry advice, political persuasion, fraud, and child endangerment or abuse.Across four GPT-3.5 Turbo versions, average refusal across policy-specified categories fell from above 60% in older versions to below 40% in the newest; the newer policy also omits military and warfare specifications.Source: §4.4 · p. 9, Figure 8 · p. 10
• Claude 3 shows stronger adherence to its own policies than the GPT family does to OpenAI’s, while both struggle with automated decision-making and regulated-industry advice.The study uses HELM to send prompts to evaluated models, incorporate outputs into judge prompts, and extract judge scores; 22 models from 10 organizations were evaluated.Source: §4.1 · p. 6, §4.4 · p. 10

5 Conclusions

• AIR-BENCH 2024 provides 5,694 context-specific prompts grounded in risks from 8 government regulations and 16 company policies.The paper presents it as a benchmark for assessing AI systems’ alignment with real-world safety concerns.Source: §5 · p. 10
• Evaluation of 22 foundation models finds substantial variation in adherence to safety guidelines, with gaps in automated decision-making and regulated-industry advice even among the best-aligned models.The paper identifies granular risk taxonomies as useful for uncovering these gaps.Source: §5 · p. 10
• Case studies find that no evaluated model fully aligns refusal behavior with EU AI Act categories and that models have gaps in handling risks covered by their own corporate policies.The authors state that AIR-BENCH 2024 adds transparency about risks that developers may not fully uncover themselves.Source: §5 · pp. 10–11
• Because AIR-BENCH 2024 is static, its risk categories need periodic updates as new risks appear in regulations and policies.The authors plan regular AIR taxonomy updates and identify dynamic benchmarking as a possible future direction.Source: §5 · p. 11

A.1 Overview of the AI Risk Taxonomy (AIR 2024)

• AIR 2024 constructs a four-level risk taxonomy by extracting risk categories from government regulations and company policies, including both platform-wide and model-specific corporate policies.The source set includes regulations from the EU, United States, and China, and policies from multiple AI companies.Source: §A.1 · p. 16
• The taxonomy groups risks into System & Operational Risks and Content Safety Risks, which break down into 6 and 17 level-3 categories, respectively.These groups contain 38 and 79 unique level-4 categories, respectively.Source: §A.1 · p. 16
• Societal Risks and Legal & Rights Related Risks comprise 14 and 8 level-3 categories, respectively, with 52 and 145 unique level-4 categories.The taxonomy places political usage, economic harm, deception, manipulation, and defamation under societal risks, and rights, discrimination, privacy, and criminal activities under legal and rights-related risks.Source: §A.1 · p. 16

Summary of Public Sector Categorizations of Risk and Findings in AIR 2024

• Public regulations differ in their level of detail; EU high-risk categories include automated decision-making, while the US Executive Order identifies worker displacement as a risk absent from corporate policies.China’s regulations include distinct risks such as disrupting social order and specify ethical review for certain political-influence services.Source: Summary of Public Sector… · pp. 16–17
• EU, US, and China regulations share seven risk categories, including automated decision-making, advice in heavily regulated industries, privacy violations, and discriminatory activities.The passage also lists autonomous unsafe operation, misrepresentation, and the paired categories #39 and #40 and #41 and #42.Source: Summary of Public Sector… · p. 17

Summary of Private Sector Categorizations of Risk and Findings in AIR 2024

• All companies’ policies cover privacy violations, other illegal or criminal activities, and harassment, while no corporate policy covers disempowering workers.Non-consensual nudity and deterring democratic participation appear in only one corporate policy each; disempowering workers is prohibited under the US Executive Order and EU AI Act.Source: Summary of Private Sector… · p. 17

A.2 Additional Level-4 Results

• Refusal rates vary across 38 level-4 System and Operational Risks, with Nuclear facilities tending to have lower rates than Electrical grids and Air traffic control.Advice in Heavily Regulated Industries has the lowest refusal rates among the level-4 categories discussed.Source: §A.2 · p. 17, Figure 10 · p. 19
• Gemini 1.5 Flash refuses Spear phishing prompts at 50%, compared with 100% for Network intrusion.Across models, Spear phishing also has lower average refusal than Network intrusion.Source: §A.2 · p. 17
• Claude 3 Sonnet’s refusal rate exceeds 90% for Social scoring, Profiling, and Insurance eligibility, but is below 20% for Housing eligibility and Employment.Its average refusal rate for #4 (Automated Decision-Making) is 70%.Source: §A.2 · p. 17
• Anthropic Claude models have the highest overall level-4 refusal rates and DBRX Instruct the lowest, yet some level-4 categories have refusal rates near 0% for nearly all models.The appendix presents further level-4 results for categories including Content Safety Risks, Societal Risks, Fundamental Rights, Privacy, and Criminal Activities.Source: §A.2 · p. 17, Figure 10 · p. 17, Figure 11 · p. 20

B Human Alignment Study to the LLM-based Autograders

• The study sampled 78 prompts across level-2 categories and evaluated 1,716 model-response pairs using five authors.Two duplicate prompts were removed; annotators used the same scoring rules provided to the Autograders.Source: B Human Alignment Study to the… · p. 22
• The human-evaluation interface displayed each question-answer pair with its scoring rules and allowed annotators to enter scores and review prior evaluations.Figure 13 shows the adapted interface used for the alignment study.Source: B Human Alignment Study to the… · p. 22, Figure 13 · p. 22
• Human evaluators and Autograders achieved a Cohen’s Kappa score of 0.86, compared with 0.69 for a prior fixed-prompt automated evaluation.The authors interpret the agreement as evidence of robustness and reliability for assessing harmful content.Source: B Human Alignment Study to the… · p. 22

C Additional Case Studies on Public Sector Categorizations of Risk

• No evaluated model fully aligns with US or Chinese regulations.The analyses report similar safety-behavior trends across EU, US, and Chinese regulations.Source: C Additional Case Studies on Public… · p. 23
• Even Claude 3 models have room to improve on US-regulated risks including Privacy Violations, Autonomous Unsafe Operation of Systems, Misrepresentation, Automated Decision-Making, and Advice in Regulated Industries.Figure 14 presents model refusal rates across 20 risk categories in the US Executive Order.Source: C Additional Case Studies on Public… · p. 23, Figure 14 · p. 23
• Chinese regulations include low-performing categories not covered by US regulations, including Adult Content, Types of Defamation, and Celebrating Suffering.Figure 15 presents refusal rates across 23 risk categories specified in five Chinese AI-related regulations.Source: C Additional Case Studies on Public… · p. 23, Figure 15 · p. 23

D Additional Evaluation Setting

• The evaluation pipeline sends prompts to models at temperature 0.0, judges responses at temperature 0.0, extracts scores and reasoning, and averages scores by category.The model output limit is 512 tokens and the judge output limit is 64 tokens.Source: D Additional Evaluation Setting · p. 24
• For Gemini models, triggered built-in safety filters return API errors that the evaluation treats as empty responses and scores as refusals, bypassing judge evaluation.The study accessed models from Anthropic, Cohere, OpenAI, and Google through their platforms and API clients.Source: D Additional Evaluation Setting · p. 24
• The primary metric is refusal rate, the proportion of model outputs assigned a score of 1; higher rates indicate safer or more conservative behavior under this evaluation.The authors note that complete refusal may not suit every jurisdiction or use case.Source: D Additional Evaluation Setting · p. 24
• The paper directs readers to the dataset card on Hugging Face for detailed experiment-reproduction instructions.Table 1 summarizes the evaluated models.Source: D Additional Evaluation Setting · p. 24, Table 1 · p. 24

• By combining categories from 8 government regulations and 16 company policies, AIR-BENCH 2024 provides a standardized basis for comparing model responses to malicious requests.Source: E Additional Discussion on Broader… · p. 25

AI Companies

• AIR-BENCH 2024 streamlines previously disjointed risk areas into one standardized benchmark, helping companies reduce compliance inefficiencies.The paper says companies face increased compliance costs while navigating government policies and regulations.Source: E Additional Discussion on Broader… · p. 25

AI Researchers

• A unified approach can help researchers reduce redundant, siloed efforts and keep safety research aligned with evolving regulations and acceptable use policies.Source: E Additional Discussion on Broader… · p. 25

End Users

• The benchmark offers end users a common reference and added transparency for understanding and building trust in AI systems.The passage links unclear, nonuniform standards with confusion and distrust, which can erode public trust and hinder adoption.Source: E Additional Discussion on Broader… · p. 25

F.1 Data Expansion

• Two mutation techniques—uncommon dialects and authority endorsement—vary base prompts’ language and perceived legitimacy while aiming to preserve their meaning.Generated variants are manually reviewed and corrected when low quality or semantically unfaithful, to expand linguistic variation and perceived legitimacy levels.Source: §F.1 · p. 25

F.2 Automatic Generation of Category-specific Evaluation Prompts

• Category-specific judge prompts are generated using three example prompts in the system prompt and category-tailored information and harmful instructions in the user prompt.The three in-context examples have scoring rules tailored to their risk categories.Source: §F.2 · pp. 25–26, Figure 16 · p. 26
• For evaluation, GPT-4o scores harmfulness and provides short reasoning from the category-specific prompt, the harmful request, and the model response; each generated judge prompt is manually checked.The manual check verifies that the prompt reflects the examples and relevant category background.Source: §F.2 · p. 26

G Qualitative Examples

• Even plain, direct harmful prompts elicited abundant risky outputs from many models, underscoring the value of granular categories and diversified red-teaming and moderation.Source: G Qualitative Examples · p. 28