PaperlayerView exact revision on arXiv

Source-linked AI summary

Introducing v0.5 of the AI Safety Benchmark from MLCommons

Bertie Vidgen, Adarsh Agrawal, Ahmed M. Ahmed, Victor Akinwande, Namir Al-Nuaimi, Najla Alfaraj, Elie Alhajjar, Lora Aroyo, Trupti Bavalatti, Max Bartolo, Borhane Blili-Hamelin, Kurt Bollacker, Rishi Bomassani, Marisa Ferrara Boston, Siméon Campos, Kal Chakra, Canyu Chen, Cody Coleman, Zacharie Delpierre Coudert, Leon Derczynski, Debojyoti Dutta, Ian Eisenberg, James Ezick, Heather Frase, Brian Fuller, Ram Gandikota, Agasthya Gangavarapu, Ananya Gangavarapu, James Gealy, Rajat Ghosh, James Goel, Usman Gohar, Sujata Goswami, Scott A. Hale, Wiebke Hutiri, Joseph Marvin Imperial, Surgan Jandial, Nick Judd, Felix Juefei-Xu, Foutse Khomh, Bhavya Kailkhura, Hannah Rose Kirk, Kevin Klyman, Chris Knotz, Michael Kuchnik, Shachi H. Kumar, Srijan Kumar, Chris Lengerich, Bo Li, Zeyi Liao, Eileen Peters Long, Victor Lu, Sarah Luger, Yifan Mai, Priyanka Mary Mammen, Kelvin Manyeki, Sean McGregor, Virendra Mehta, Shafee Mohammed, Emanuel Moss, Lama Nachman, Dinesh Jinenhally Naganna, Amin Nikanjam, Besmira Nushi, Luis Oala, Iftach Orr, Alicia Parrish, Cigdem Patlak, William Pietri, Forough Poursabzi-Sangdeh, Eleonora Presani, Fabrizio Puletti, Paul Röttger, Saurav Sahay, Tim Santos, Nino Scherrer, Alice Schoenauer Sebag, Patrick Schramowski, Abolfazl Shahbazi, Vin Sharma, Xudong Shen, Vamsi Sistla, Leonard Tang, Davide Testuggine, Vithursan Thangarasa, Elizabeth Anne Watkins, Rebecca Weiss, Chris Welty, Tyler Wilbers, Adina Williams, Carole-Jean Wu, Poonam Yadav, Xianjun Yang, Yi Zeng, Wenhui Zhang, Fedor Zhdanov, Jiacheng Zhu, Percy Liang, Peter Mattson, Joaquin Vanschoren

arXiv:2404.12241v2cs.CLcs.AI

TL;DR

AI safety evaluation needs benchmarks that reflect the hazards and deployment context of chat-tuned language models. This paper presents MLCommons’ v0.5 proof-of-concept, combining a hazard taxonomy, prompt-based tests, automated grading, and open evaluation infrastructure; it reports that LlamaGuard achieved 70.4% accuracy on a human evaluation set, while emphasizing that v0.5 should not be used to assess AI safety.

  • Problem

    Existing AI safety evaluation needs a principled way to assess hazards in chat-tuned language models across defined systems, use cases, and users.

    Source: §1.1 · p. 5

  • Method

    The paper constructs v0.5 from hazard-focused test items, automated LlamaGuard scoring, and an open benchmark platform informed by a test specification.

    Source: §4.1 · p. 17, §5 · p. 21, §5.2.1 · p. 22

  • Results

    LlamaGuard achieved 70.4% overall accuracy on the human evaluation set used to assess automated safety scoring.

    Source: §5.2.1 · p. 23

  • Takeaways & Limitations

    The benchmark demonstrates an initial approach to structured AI safety evaluation, but v0.5 is presented as a preliminary proof-of-concept rather than a safety assessment tool.

    Source: §7 · p. 26

    The benchmark has limited scope and only identifies a lack of detected safety weaknesses, so strong benchmark performance does not mean that an AI system is safe.

    Source: §2.4 · p. 10, §7 · pp. 25–26

Abstract

from arXiv · show

This paper introduces v0.5 of the AI Safety Benchmark, which has been created by the MLCommons AI Safety Working Group. The AI Safety Benchmark has been designed to assess the safety risks of AI systems that use chat-tuned language models. We introduce a principled approach to specifying and constructing the benchmark, which for v0.5 covers only a single use case (an adult chatting to a general-purpose assistant in English), and a limited set of personas (i.e., typical users, malicious users, and vulnerable users). We created a new taxonomy of 13 hazard categories, of which 7 have tests in the v0.5 benchmark. We plan to release version 1.0 of the AI Safety Benchmark by the end of 2024. The v1.0 benchmark will provide meaningful insights into the safety of AI systems. However, the v0.5 benchmark should not be used to assess the safety of AI systems. We have sought to fully document the limitations, flaws, and challenges of v0.5. This release of v0.5 of the AI Safety Benchmark includes (1) a principled approach to specifying and constructing the benchmark, which comprises use cases, types of systems under test (SUTs), language and context, personas, tests, and test items; (2) a taxonomy of 13 hazard categories with definitions and subcategories; (3) tests for seven of the hazard categories, each comprising a unique set of test items, i.e., prompts. There are 43,090 test items in total, which we created with templates; (4) a grading system for AI systems against the benchmark; (5) an openly available platform, and downloadable tool, called ModelBench that can be used to evaluate the safety of AI systems on the benchmark; (6) an example evaluation report which benchmarks the performance of over a dozen openly available chat-tuned language models; (7) a test specification for the benchmark.

Reader’s guide

The paper is long and organized by purpose: Sections 2 and 5 explain benchmark construction and scoring, Sections 3 and 4 describe hazards and tests, and Section 6 reports model performance.

Reader’s guide

  • Sections 2 and 5 explain how the benchmark was developed and how models were scored.The paper recommends these sections for understanding the benchmark’s process.Source: Reader’s guide · p. 2
  • Sections 3 and 4 describe the hazard taxonomy, tests, and test items, while Section 6 covers model performance.Appendix H also provides a brief datasheet.Source: Reader’s guide · p. 2

1 Introduction

The MLCommons AI Safety Working Group introduces v0.5 as a proof-of-concept benchmark for safety risks in chat-tuned language models, supported by open infrastructure and community-oriented release practices. Its scope is deliberately preliminary, and its public release balances reproducibility and transparency against risks to benchmark integrity.

1.1 Overview of the MLCommons AI Safety Working Group

  • The Working Group brings technical, policy, governance, and trust-and-safety expertise together to develop benchmarks for assessment, tracking, and incentives to improve AI safety.Its work is organized around broad participation and aims to increase transparency and shared knowledge.Source: §1.1 · p. 5

1.2 The AI Safety Benchmark

  • The Working Group introduces v0.5 to assess safety risks in chat-tuned language models as a tractable starting point for broader multimodal and multilingual evaluation.The benchmark focuses on language models because they are widely researched and deployed; future versions may cover other modalities and languages.Source: §1.2 · pp. 5–6

1.2.1 Who is the AI Safety Benchmark for?

  • The benchmark is intended for model providers, model integrators, and AI standards makers and regulators, with future versions considering additional audiences.Their stated objectives include building safer models, comparing and deploying models responsibly, and setting or enforcing safety standards.Source: §1.2.1 · p. 6

1.3 Infrastructure of the v0.5 benchmark

  • ModelBench and ModelGauge provide an open-source, modular platform for standardized, reproducible benchmark runs using versioned tests and systems under test.The infrastructure supports new systems and evolving test versions, and was developed with the HELM team.Source: §1.3 · p. 7

1.4 Release of the v0.5 benchmark

  • The release publishes prompts, annotation guidelines, and taxonomy openly but withholds model responses because they could enable harm or help attackers bypass safety filters.Future releases may restrict or delay some test items to protect benchmark integrity while retaining transparency.Source: §1.4 · pp. 7–8

2 Scope and specification of the Benchmark

The benchmark evaluates specific chat systems in a narrowly defined English-language use case with three bounded personas, using hazard-focused prompts and an openly specified testing process. It measures unsafe responses but does not yet assess false refusals, and its specification is designed to improve reproducibility.

2 Scope and specification of the Benchmark

  • The benchmark evaluates a specific AI system, use case, and persona set rather than the safety of a model in general.Its results are bounded by the selected tests and cannot comprehensively represent all possible hazards.Source: §2 · p. 8

2.1 Systems Under Test (SUTs)

  • The systems under test are English-capable, text-to-text general-purpose chat systems, excluding guardrails, system prompts, safety filters, and user-facing interfaces for simplicity.Examples include Llama-70B-Chat, Mistral-7B-Instruct, and Gemma-7B-Instruct.Source: §2.1 · pp. 8–9

2.2 Use cases

  • The v0.5 use case is an adult chatting with a general-purpose assistant in English within a Western Europe and North America context.Performance in this setting does not automatically extend to specialized uses such as financial advice, education, therapy, or customer service.Source: §2.2 · p. 9

2.3 Personas

  • The benchmark models typical, non-sophisticated malicious, and non-sophisticated vulnerable adult users, while acknowledging that these categories simplify real user intent and behavior.Future work may add bystanders, sophisticated malicious or vulnerable users, and victims’ relations.Source: §2.3 · pp. 9–10

2.4 Unsafe responses and false refusal

  • The benchmark tests unsafe responses but excludes false-refusal assessment because the authors lack resources to determine whether responses to benign prompts constitute false refusals.Benign typical-user prompts could support such testing in principle.Source: §2.4 · p. 10

2.5 Test specification

  • The freely available test specification documents setup choices so test writers can support scalable reproducibility and audiences can understand how results were produced.It addresses sensitivity to prompt formulation, few-shot configurations, and chain-of-thought instructions.Source: §2.5 · p. 10

3 Taxonomy of hazard categories

This section presents a documented, hierarchical taxonomy for organizing and communicating AI hazards, comprising 13 categories with 7 included in v0.5. It also discusses limitations in the taxonomy, including its coverage of behavioral attributes and protected characteristics.

3 Taxonomy of hazard categories

  • The taxonomy was developed iteratively over 10 weeks using Working Group input, interviews with 12 civil society experts, and reviews of 25+ taxonomies, 50+ datasets, 50+ papers, and 10+ guidelines.Definitions and subcategories were refined through writing and reviewing prompts, with comparisons against 17 other taxonomies.Source: §3 · pp. 10–11

3.1 Overview of the taxonomy

  • The taxonomy contains 13 hazard categories, of which 7 are tested in v0.5; categories were prioritized by recognized illegality or heightened societal or personal risks.Each v0.5 category has a single test set comprising multiple test cases.Source: §3.1 · p. 11

3.1.1 Hazard categories

  • The taxonomy is hierarchical, dividing categories into subcategories and sub-subcategories, while category definitions include supporting information, out-of-scope activities, and conflict-resolution guidance.The definitions are not intended to exhaustively list everything captured by each category.Source: §3.1 · p. 11, §3.1.1 · p. 12

3.1.2 Limitations of the taxonomy

  • The taxonomy is currently specific to textual content in English, has not been tested across modalities or languages, and does not comprehensively cover all possible hazard categories.The authors plan to expand and iterate it for v1.0; additional modalities may require categories such as Graphic Violence.Source: §3.1.2 · p. 12, §3.1 · p. 12
  • The taxonomy’s limitations discussion references behavioral attributes and protected characteristics.Source: §3.1.2 · p. 15

4 Test items

This section describes the benchmark’s standardized single-turn test items, generated from expert-validated hazard fragments and interaction templates. It covers their scope, distribution across hazard categories and personas, and supporting dataset documentation.

4 Test items

  • New prompt datasets were created because existing resources lacked complete hazard coverage, consistent quality and format, and sufficiently structured interaction testing.The authors also use v0.5 to stress-test prompt-construction processes needed for future modalities, languages, and hazard categories.Source: §4 · p. 16

4.1 Conceptualisation and design of the test items

  • Prompts are designed as plausible, face-valid examples for exactly one hazard category, but limited ecological validity and crossed hazards can reduce realism and coverage.The benchmark acknowledges that some prompts may fit multiple categories and that important crossed varieties, such as terrorist-related CBRNE, may be missing.Source: §4.1 · p. 16
  • The benchmark uses single-turn free-text chat responses because they best match the defined use case, while multi-turn interactions are harder to assess automatically.Free-text responses resemble real interactions but are difficult to judge consistently; multi-turn testing introduces in-context branching paths.Source: §4.1 · pp. 16–17

4.2.1 Sentence fragments

  • Sentence fragments describe distinct hazards through activities or viewpoints, and two additional experts validated each human-authored fragment for relevance.The fragments omit hazard-specific terms or information that could enable harm, supporting open dataset release and consistent coverage.Source: §4.2.1 · p. 17

4.2.2 Interaction types

  • Templates express each hazard fragment through different communicative goals, such as reporting, requesting advice, or seeking encouragement, using linguistic and human-computer interaction theories.The approach covers 11 activity statement types and 2 viewpoint statement types, but its scalability comes at the cost of diversity.Source: §4.2.2 · pp. 17–18
  • The benchmark excludes dual-use or strongly context-dependent prompts and includes only clear-cut interactions whose risks do not rely heavily on inferred context.This prevents prompts with potentially benign and harmful interpretations from being treated as unambiguous safety tests.Source: §4.2.2 · p. 18

4.3 Dataset overview

  • Test items combine 725 expert-created sentence fragments with 32 templates spanning 13 interaction types, producing 43,090 prompts to vary interactions and test robustness.The fragments comprise 550 activity fragments and 175 viewpoint fragments; templates turn them into grammatically complete prompts.Source: §4.2 · p. 17, §4.2.2 · p. 18, §4.3 · p. 21
  • Prompt counts range from 1,810 to 27,250 per hazard category, averaging 6,156; malicious-user prompts are substantially more numerous for Hate (26,530) and Violent crimes (4,390) because templates cover key terms.Source: §4.3 · p. 21
  • Appendix G breaks down prompts across 13 interaction types by hazard category and persona, while Appendix H provides a brief datasheet intended for expansion in v1.0.Source: §4.3 · p. 21

5 Grading SUTs

The benchmark’s grading system aggregates automated safety judgments from test items into interpretable test and overall grades, while prioritizing consistency, transparency, and avoiding overstated safety. Its v0.5 evaluator and grades remain preliminary and should not be interpreted as meaningful safety signals.

5 Grading SUTs

  • The grading system aggregates item-level unsafe-response judgments into test percentages, five-point risk grades, and an overall benchmark grade.It uses LlamaGuard for item-level classification, converts test-level unsafe percentages into ordinal grades, and aggregates across tests.Source: §5 · p. 21, §5.2 · p. 22

5.1 Requirements of the grading system

  • The system is designed to be consistent, reproducible, interpretable, well-documented, auditable, and useful for targeted safety improvements.Requirements differ across model providers, integrators, and standards makers, including fair scores, granular insights, and meaningful comparisons.Source: §5.1 · pp. 21–22

5.2.1 Scoring SUTs’ performance on test items

  • LlamaGuard achieved 70.4% accuracy on the human evaluation set and 85.6% when weighted to the full response distribution, supporting its use for v0.5 despite errors.It was evaluated against 1,320 human-labelled responses, with higher accuracy on responses classified as not unsafe.Source: §5.2.1 · pp. 22–23

5.2.2 Scoring SUTs’ performance on each test

  • The benchmark compares each SUT with a reference model and uses the lowest-scoring of three candidate references, allowing hazard-specific baselines while retaining interpretability.The reference-model approach is more complex than alternatives and requires selecting a baseline; the grading scale is defined in Table 5.Source: §5.2.2 · p. 24, Table 5 · p. 24

5.2.3 Scoring SUTs’ performance on the benchmark overall

  • The overall benchmark grade is the lowest grade across the seven tests, minimizing the risk of overstating a SUT’s safety.This final aggregation treats the weakest hazard-category performance as decisive.Source: §5.2.3 · p. 24

6 Results

The evaluation tested 13 anonymized, permissively licensed open chat-tuned models under standardized low-temperature settings. Five were graded high risk overall, and unsafe responses were especially common for sex-related crimes and non-typical personas.

6.1 Selection and implementation of SUTs

  • The study evaluated 13 open models from 11 providers, released between March 2023 and February 2024, using chat-tuned language models with permissive licenses.Results were anonymized because v0.5 has numerous limitations and was not intended to spotlight individual model strengths or weaknesses.Source: §6.1 · pp. 24–25
  • Responses were collected at temperature 0.01 with a 500-token maximum, reducing variability and creativity but not making sampling fully deterministic.All models were accessed through Together.AI with remaining parameters at default settings.Source: §6.1 · p. 25

6.2 Performance on the benchmark

  • 5 of 13 SUTs were graded high risk overall, 4 moderate risk, and 4 moderate-low risk; 3 were high risk across every hazard category.Overall performance is defined by the lowest grade across hazard categories, as shown in Figure 1.Source: §6.2 · p. 25, Figure 1 · p. 25
  • Unsafe-response percentages were substantially higher for sex-related-crimes test items than for the remaining hazard categories.Table 6 reports the unsafe-response percentage for the selected reference model by hazard category.Source: §6.2 · p. 25, Table 6 · p. 25
  • SUTs generally responded more unsafely to malicious- and vulnerable-user prompts than to typical-user prompts across most hazards and models.The benchmark therefore exposes variation associated with the three tested personas.Source: §6.2 · p. 25

7 Limitations

The v0.5 benchmark has narrow coverage, simplified tests, automated-evaluation errors, and low-temperature evaluation. Its results can identify observed safety weaknesses but cannot establish that a well-performing system is safe.

The benchmark has limited scope

  • The benchmark covers only English, Western Europe and North America, one use case, and three personas, and it does not test false refusal of benign prompts.Future versions are intended to expand this scope.Source: §7 · pp. 25–26

The taxonomy is incomplete

  • Only seven of the taxonomy’s hazard categories are tested, while other hazards, intersections, subcategories, and LM security issues remain uncovered.Six identified categories were omitted because of feasibility constraints.Source: §7 · p. 26

Tests are designed to be simple

  • The short, clear, single-turn, non-adversarial test items are easy to assess but have limited relevance for sophisticated users and prompting strategies.The tested personas are described as unsophisticated, so adversarial prefixes and prompting tricks were excluded.Source: §7 · p. 26

Automated evaluation introduces some errors

  • Automated LlamaGuard evaluation makes errors that can result in incorrect grades for some SUTs.The paper reports validation but acknowledges that evaluator performance could be improved.Source: §7 · p. 26

SUTs were evaluated at low temperature

  • Low-temperature evaluation improves reproducibility but may underestimate unsafe responses that occur at higher temperatures.The authors plan to test SUTs at different temperatures in future releases.Source: §7 · p. 26

The benchmark can only identify lack of safety rather than safety

  • A good benchmark result means only that the benchmark did not identify safety weaknesses, not that the SUT is safe.The benchmark has negative predictive power and may be misinterpreted without explicit guidance.Source: §7 · p. 26

8 Previous work on AI safety

The paper situates AI safety evaluation within a broad landscape of present and future harms, methodological challenges, directed and exploratory techniques, and benchmark design trade-offs. It presents directed benchmarks as interpretable and standardized but limited in ecological validity and adaptability.

8.1 AI safety

  • AI safety concerns include immediate harms such as scams, disinformation, cyberattacks, toxic content, privacy leakage, and bias, alongside speculative future risks including biowarfare and rogue AI agents.Future-risk evaluation often focuses on dangerous capabilities such as autonomy, deception, self-proliferation, and self-reasoning.Source: §8.1 · pp. 26–27

8.2 Challenges in AI safety evaluation

  • Safety evaluation measures whether models are acceptably safe for a specified purpose and context, but complexity, unpredictability, and socio-technical entanglement make harms difficult to predict and attribute.Model outputs remain probabilistic even at low temperature, while social contexts shape whether and how harms manifest.Source: §8.2 · pp. 27–28
  • Identifying harm likelihood and severity requires extensive production access and resources, while longitudinal, quantitative, representative evidence on AI-caused harms remains limited.Sensitive real-world interaction data are often held by private companies, constraining safety research.Source: §8.2 · p. 28

8.3 Techniques for AI safety evaluation

  • Algorithmic auditing provides independent, systematic evidence about broader sociotechnical systems, including technical properties, governance, ethics, compliance, data, and impacts.External audits additionally require communicating results to stakeholders and are typically more independent.Source: §8.3 · p. 28
  • Directed evaluation uses predefined categorized prompts and offers interpretable, standardized comparisons, but tests may not challenge model-specific capabilities and can become outdated.Exploratory evaluation instead targets novel or poorly understood risks through open-ended methods such as human or AI red teaming, including multi-turn and adversarial interactions.Source: §8.3 · pp. 28–29

8.4 Benchmarks for AI safety evaluation

  • Benchmarks can drive progress, but they may mislead, encourage narrow goals, saturate through overfitting, and lack ecological validity when tests do not approximate real-world data.The paper identifies more ecologically valid and human-and-model-in-the-loop benchmarking as an active development direction.Source: §8.4 · p. 29
  • Existing safety benchmarks differ in hazards, languages, model types, datasets, evaluation methods, and prompt quality, including HarmBench, TrustLLM, DecodingTrust, SafetyBench, and other suites.These projects range from red-teaming and grounded risks to toxicity, fairness, privacy, machine ethics, and broader safety dimensions.Source: §8.4 · pp. 29–30

A Comparison of the AI Safety taxonomy with other taxonomies

This section compares the AI Safety Taxonomy with 17 existing taxonomies, examining differences in coverage, granularity, terminology, and interpretability while clarifying key legal and conceptual terms. It also defines terminology, identifies six v1.0 hazard categories without v0.5 tests, and documents the distribution of templates and prompts across interaction types, hazard categories, and personas.

A Comparison of the AI Safety taxonomy with other taxonomies

  • Four of 17 taxonomies fully cover the AI Safety Taxonomy, while seven miss one category, two miss two, and three miss three to five categories.Most taxonomies also add categories such as Sexual content, Profanity, and Misinformation; broad or ambiguous labels complicate evaluation-result interpretation.Source: A Comparison of the AI Safety taxonomy… · p. 45, Table 7 · p. 46

B Legality of the hazard categories

  • The taxonomy references legality because many hazards are illegal, but legal variation, incomplete laws, assessment difficulty, and company restrictions prevent legality from solely defining its policies.These considerations motivate names such as “Sexual crimes” and “Non-violent crimes.”Source: B Legality of the hazard categories · p. 50

C Terrorist groups and actors

  • Terrorism includes Islamist, ultra-right-wing, ultra-left-wing and anarchist, environmental, single-issue, and Northern-Ireland-related groups, but excludes non-violent advocacy and democratic radical groups.The definition includes examples such as ISIL, Al Qaeda, and the Provisional IRA, while excluding peaceful environmental groups and groups supporting democratic elections.Source: C Terrorist groups and actors · p. 50

D A non-exhaustive list of types of hazardous activities that AI models can enable

  • AI models can enable harmful activities by performing, threatening, planning, recruiting, concealing, justifying, supporting, disseminating instructions, or creating resources for them.Examples include self-harm, bomb detonation, and fraud.Source: D A non-exhaustive list of types of… · p. 50

E Taxonomy terminology definitions

  • The paper provides definitions for key taxonomy terms to standardize their interpretation.These definitions are presented as a terminology resource for applying the AI Safety Taxonomy.Source: E Taxonomy terminology definitions · p. 51
  • The taxonomy defines response properties including containment, description, enablement, encouragement, endorsement, dehumanization, and demeaning, alongside sensitive personal characteristics, advice, and erotica.These definitions distinguish what responses contain, describe, or promote and specify when content concerns protected characteristics or graphic sexual material.Source: E Taxonomy terminology definitions · p. 51

F v1.0 Taxonomy hazard categories

G Number of templates per interaction type

  • The benchmark uses 13 interaction types—two for Viewpoints and 11 for Activities—with 1 to 4 templates per interaction type.Template counts do not linearly determine prompt counts because some hazard categories use special entity templates and templates are handled differently for Viewpoint and Activity classes.Source: G Number of templates per interaction… · p. 51, Table 9 · p. 52
  • Table 10 reports test-item counts by hazard category, persona, and interaction type, with each test’s total combining the items for both personas.The distribution therefore varies across hazard categories, personas, and interaction types rather than following template counts directly.Source: G Number of templates per interaction… · p. 51, Table 10 · p. 53

H Datasheet for AI Safety Benchmark

The dataset evaluates whether generative language-model systems produce hazardous responses, and MLCommons’ AI Safety Working Group created and vetted it.

H.1 Motivation

  • The dataset evaluates safety aspects of systems built on generative language models by assessing whether responses contain hazards.Source: §H.1 · p. 54
  • MLCommons’ AI Safety Working Group created and vetted the dataset with contributions from many researchers, engineers, and practitioners.Source: §H.1 · p. 54
  • MLCommons funded the creation of the dataset.Source: §H.1 · p. 54

H.2 Composition

The benchmark contains labeled English prompts spanning seven high-risk hazard categories, created through templates and sentence fragments for a Western Europe and North America context.

H.2 Composition

  • The seven categories include violent and non-violent crimes, sex-related crimes, child sexual exploitation, CBRNE weapons, suicide and self-harm, and hate.Source: §H.2 · p. 54
  • 43,090 instances cover seven hazard categories, with category totals ranging from 1,810 to 27,250 prompts.Source: §H.2 · p. 54
  • The dataset samples English prompts within Western Europe and North America and does not represent all possible instances.Source: §H.2 · p. 54
  • Each prompt is labeled for hazard category and applicable subcategories, plus persona, speech act, tense, template, and sentence fragment.Source: §H.2 · p. 54
  • The prompts contain no confidential data or identifiable individuals, but many include inappropriate, offensive, or unsafe language.Source: §H.2 · p. 54

H.3 Collection Process

  • Prompts were manually created by combining hazard-indicating sentence fragments with templates representing 13 interaction types and different motivations or aims.Source: §H.3 · p. 54
  • The AI-safety working group and 12 civil-society experts contributed to taxonomy creation, and the dataset was built over 10 weeks.Source: §H.3 · p. 54

H.5 Uses

The dataset supports safety benchmarking and annotation of chat-model responses, with public distribution and planned updates, but it is not intended for model training.

H.5 Uses

  • The dataset benchmarked 14 models from 11 providers for the v0.5 benchmark and is intended solely for testing, not training models.Source: §H.5 · pp. 54–55

H.6 Distribution

  • The dataset is available under a CC-BY license, and MLCommons works with HELM to support, host, and maintain it.Source: §H.6 · p. 55, §H.7 · p. 55

H.7 Maintenance

  • Anyone may join the MLCommons AI Safety Working Group, while future updates may become non-public because public evaluation datasets can have consequences.Source: §H.7 · p. 55

I Annotation guidelines

I.1 Guidelines given to annotators

  • The annotation task covered 2,000 responses and warned workers that the content could be harmful or offensive.Source: §I.1 · p. 55
  • Annotators judge whether chat-model responses are unsafe, defined as creating or increasing harm risk or increasing harm severity.Source: §I.1 · p. 56
  • The annotation scope assumes a general-purpose chat model used as an AI assistant by a user in Northern America or Western Europe.Source: §I.1 · p. 56
  • Responses containing any unsafe element are labeled unsafe, while ambiguity alone is insufficient; garbled responses are still annotated and flagged.Source: §I.1 · p. 56

Related papers

Loading 2404.12241v2…