Source-linked AI summary
SafetyBench: Evaluating the Safety of Large Language Models
Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, Minlie Huang
TL;DR
Existing safety benchmarks provide limited coverage and often make broad, accurate, rapid evaluation difficult. SafetyBench introduces a comprehensive bilingual multiple-choice benchmark and evaluates 25 LLMs in zero-shot and few-shot settings, finding GPT-4 substantially ahead while showing that safety understanding correlates with safety generation. The benchmark’s consistency is bounded by translation, cultural-context, and evaluation assumptions.
Problem
Comprehensive LLM safety benchmarks are scarce, while existing evaluations may cover limited safety facets or rely on costly, time-consuming, or imperfect response assessment.
Method
SafetyBench uses 11,435 diverse multiple-choice questions across 7 safety categories, with Chinese and English data, to evaluate LLM safety efficiently.
Results
GPT-4 substantially outperforms other evaluated LLMs, and SafetyBench-measured safety understanding abilities correlate with safety generation abilities across tests of 25 LLMs.
Takeaways & Limitations
SafetyBench provides a fast, accurate, cost-effective basis for assessing LLM safety and offers guidance for improving current models.
Abstract
from arXiv · showhide
With the rapid development of Large Language Models (LLMs), increasing attention has been paid to their safety concerns. Consequently, evaluating the safety of LLMs has become an essential task for facilitating the broad applications of LLMs. Nevertheless, the absence of comprehensive safety evaluation benchmarks poses a significant impediment to effectively assess and enhance the safety of LLMs. In this work, we present SafetyBench, a comprehensive benchmark for evaluating the safety of LLMs, which comprises 11,435 diverse multiple choice questions spanning across 7 distinct categories of safety concerns. Notably, SafetyBench also incorporates both Chinese and English data, facilitating the evaluation in both languages. Our extensive tests over 25 popular Chinese and English LLMs in both zero-shot and few-shot settings reveal a substantial performance advantage for GPT-4 over its counterparts, and there is still significant room for improving the safety of current LLMs. We also demonstrate that the measured safety understanding abilities in SafetyBench are correlated with safety generation abilities. Data and evaluation guidelines are available at \url{https://github.com/thu-coai/SafetyBench}{https://github.com/thu-coai/SafetyBench}. Submission entrance and leaderboard are available at \url{https://llmbench.ai/safety}{https://llmbench.ai/safety}.
1 Introduction
SafetyBench addresses the scarcity of comprehensive, efficient safety benchmarks by evaluating LLM safety with diverse bilingual multiple-choice questions. Tests across 25 Chinese and English LLMs show GPT-4’s substantial advantage, while exposing safety flaws and correlating safety understanding with safety generation.
- Motivation: Comprehensive safety benchmarks are scarce, while existing evaluations often target specific concerns or face costly, time-consuming, or inaccurate response assessment.Automatic evaluation is cheaper but few classifiers are accurate across broad safety categories.
- Benchmark: 11,435 diverse samples cover 7 safety-problem categories and varied question types, including dialogue scenarios, real-life situations, comparisons, and safety knowledge.The benchmark includes Chinese and English data for evaluating models in both languages.
- Evaluation: SafetyBench supports evaluation across Chinese and English LLMs in both zero-shot and few-shot settings.The authors test 25 popular models.
- Findings: GPT-4 substantially outperforms other evaluated LLMs, while current models still exhibit safety flaws.The findings are summarized in Figure 2.
- Benchmark: SafetyBench is a bilingual multiple-choice benchmark designed for fast, accurate, and cost-effective automated safety evaluation.Each question has a single correct answer.
- Findings: Safety understanding abilities measured by SafetyBench are correlated with safety generation abilities.The authors provide quantitative evidence for this relationship.
2 Related Work
Earlier safety benchmarks typically addressed individual safety dimensions, whereas general-purpose multiple-choice benchmarks focused on knowledge and reasoning. SafetyBench specifically targets LLM safety across diverse categories.
- Safety benchmarks: Earlier safety benchmarks often focused on one safety dimension, such as gender bias or toxic generation.Winogender evaluates gender bias, while RealToxicityPrompts evaluates toxic generations.
- Multiple-choice benchmarks: MMLU and related Chinese benchmarks use multiple-choice questions to assess broad knowledge and problem-solving abilities.AGIEval extends this style to bilingual human-centric standardized exams.
- SafetyBench: Unlike general knowledge and reasoning benchmarks, SafetyBench specifically focuses on the safety dimension of LLMs.Its benchmark design is intended to address the narrower focus of prior safety evaluations.
3 SafetyBench Construction
SafetyBench combines multiple data sources into a bilingual, seven-category multiple-choice benchmark, using augmentation and layered quality control to expand coverage and improve consistency.
- Benchmark scope: 11,435 multiple-choice questions span seven safety categories, with one category excluded to reduce Chinese–English answer divergence.The benchmark is designed for consistent evaluation across both languages.
- Data sources: Existing datasets and safety-related exams provide questions for several categories, while augmentation addresses data shortages in Privacy and Property and other areas.Augmented samples are filtered and manually checked before inclusion.
- Translation: Chinese and English data are translated in both directions, but cultural nuances and expression differences may introduce noise.The authors describe mitigation procedures in the quality-control section.
- Data sources: About 600 online exam questions and about 500 middle-school exam papers were collected across the seven categories.The exam collection used search engines and a website aggregating exam papers.
- Augmentation: ChatGPT-generated questions use one-shot prompting with fine-grained keywords, then filtering enforces pairwise BLEU-4 scores below 0.7 before manual correction.The authors report collecting about 3,500 questions through this approach.
- Quality control: Human validation includes author or crowd-worker review, two initial checks per sample, and an additional author review when assessments disagree.A random check found 97% correctness among questions where GPT-4 matched human answers.
4 Experiments
The experiments evaluate 25 Chinese and English LLMs in zero-shot and few-shot settings. GPT-4 leads substantially in zero-shot evaluation, while few-shot gains vary across models.
- Setup: Zero-shot evaluation uses the benchmark prompts, with temperature set to 0 and answer extraction supported by model-specific prompt adjustments.The study does not include chain-of-thought evaluation because the benchmark is considered less reasoning-intensive.
- Setup: 25 popular Chinese and English LLMs are evaluated across diverse organizations and parameter scales.The study includes API-based and representative open-source models.
- Zero-shot results: GPT-4 leads the zero-shot results by nearly 10 percentage points over gpt-3.5-turbo, the second-best model.The gap is reported as larger in some categories, including Physical Health and Ethics and Morality.
- Five-shot results: Few-shot examples produce significant improvements for some models but negative gains for others, including gpt-3.5-turbo.The authors associate this variation with previously observed differences in in-context learning and possible alignment tax.
Appendix G.
Five-shot evaluation on Chinese APIs uses a filtered subset because strict safety filters make full-test assessment impractical.
- Filtered evaluation: 2,100 questions are evaluated after removing highly sensitive keywords and selecting 300 questions per category.The filtered subset is used because Chinese APIs may reject unsafe queries through strict filtering mechanisms.
5 Discussion
SafetyBench scores are strongly associated with safety generation performance, and its data augmentation may advantage ChatGPT without significantly affecting other models.
- Understanding vs. Generation: SafetyBench converts multiple-choice questions into constrained and open-ended queries to compare safety understanding with generation.Constrained queries limit acceptable responses, whereas open-ended queries allow more varied safe responses.
- Understanding vs. Generation: Six LLMs were evaluated using sampled Chinese questions, transformed queries, and manual judgments of response safety.The study reports original-question accuracy alongside safe-generation ratios.
- Understanding vs. Generation: 0.91 system-level Pearson correlation links original-question Accuracy with Open-ended safety-generation scores.The constrained-query correlation is also reported as 0.99.
- Understanding vs. Generation: SafetyBench safety-understanding abilities are correlated with safety-generation abilities, and identified safety defects can surface during generation.This conclusion connects benchmark performance with observed generation behavior.
- Data Augmentation: ChatGPT performs better on augmented than original data, but the resulting bias does not significantly influence other models, including GPT-4.The comparison evaluates three categories across augmented and original data.
6 Conclusion
SafetyBench is a bilingual, seven-category benchmark used to evaluate the safety abilities of 25 LLMs. The study finds a substantial gap between open-source models and GPT-4 and a correlation between safety understanding and generation.
- Conclusion: SafetyBench contains 11,435 Chinese and English multiple-choice questions covering 7 categories of safety issues.The benchmark supports evaluation across both languages.
- Conclusion: The benchmark extensively evaluates 25 LLMs from various organizations for safety abilities.The conclusion summarizes the benchmark’s broad model coverage.
- Conclusion: Open-sourced LLMs exhibit a significant performance gap compared to GPT-4, indicating room for future improvement.This is the paper’s principal comparative finding.
- Conclusion: SafetyBench safety-understanding abilities are correlated with safety-generation abilities.The conclusion links the benchmark’s measured ability to generation behavior.
Limitations
The authors acknowledge coverage and augmentation limitations. Some safety concerns or topics may be underrepresented, and ChatGPT-generated questions may advantage ChatGPT.
- Coverage: SafetyBench may overlook certain safety concerns, including political issues.The benchmark covers seven categories but does not claim exhaustive coverage.
- Coverage: The 11,435-question set may provide inadequate coverage for certain topics because the number of questions is limited.The authors balance broad coverage against efficient testing cost.
- Data Augmentation: ChatGPT-generated questions from few-shot prompting might advantage ChatGPT during data augmentation.The authors quantify this potential bias and conclude its impact on other models, including GPT-4, is limited.
- Difficulty: The benchmark’s difficulty is considered acceptable, but the authors agree that collecting more challenging multiple-choice questions would be useful.They report scores below 80% for 22 of 25 evaluated LLMs.
- Scope: SafetyBench does not include adversarial prompts, limiting its usefulness for evaluating attack-driven harmful-output induction.The benchmark is described as difficult for attackers to exploit to hack LLMs.
A Explanations of Considered Safety Issues
SafetyBench organizes safety evaluation into seven categories spanning harmful language, health, legality, morality, and privacy or property concerns.
- Offensiveness: Offensiveness covers threats, insults, profanity, sarcasm, impoliteness, and related content or actions.LLMs should identify and oppose offensive material.
- Unfairness and Bias: Unfairness and Bias addresses social bias involving race, gender, religion, and related topics.LLMs are expected to identify and avoid biased expressions and actions.
- Physical Health: Physical Health concerns actions or expressions that may affect human physical health.The category tests knowledge of appropriate responses in health-related scenarios.
- Mental Health: Mental Health focuses on psychological, spiritual, emotional, and mental well-being.LLMs should identify ways to maintain mental health and prevent adverse impacts.
- Legal and Moral Safety: Illegal Activities and Ethics and Morality distinguish unlawful behavior from broader immoral activities.The categories require legal knowledge and opposition to unethical behavior or speech.
- Privacy and Property: Privacy and Property covers privacy, property, and investment issues, including preventing privacy breaches or property loss.The category targets safety understanding in financial and personal-information contexts.
B Detailed Steps for Collecting Data from Existing Datasets
SafetyBench constructs safety questions from existing datasets across multiple categories, languages, and question formats, with prompt adjustments for reliable answer extraction.
- Offensiveness: Offensiveness questions combine Chinese offensive-language data with English toxicity-comparison and adversarial-dialogue data.Chinese sources include COLD; English sources include Jigsaw Toxicity Severity and an adversarial dataset.
- Unfairness and Bias: Unfairness and Bias questions use COLD, CDial-Bias, and RedditBias samples covering regional, gender, racial, religious, LGBTQ, and female-related topics.The questions assess whether a text exhibits bias or unfairness.
- Physical Health: Physical Health questions from SafeText ask models to select safe actions or choose the safer action for real-life scenarios.SafeText provides 367 scenarios, with 367 questions for each constructed question type.
- Ethics and Morality: Ethics and Morality questions draw on Scruples, MIC, Moral Stories, and Ethics to evaluate choices involving moral judgments.The supplied passages describe selecting more ethical actions and judging moral scenarios.
- Evaluation setup: Minor prompt changes were made for some LLMs because default prompts sometimes produced formats that hindered automatic answer extraction.The modified prompts are listed in Figure 5, while Table 7 provides detailed information about the 25 evaluated LLMs.
E Failing Cases
The paper presents one GPT-4 failing case for each safety category, while noting that the shown Chinese and English questions receive consistent predictions.
- Failing cases: GPT-4 has at least one documented failing case in every safety category evaluated in the paper.These cases are presented in Figure 6.
G Impact of the Selected 5-shot Examples
The study tests whether different selected 5-shot example groups affect SafetyBench performance on the Chinese test set, using three evaluation runs and micro-average accuracy.
- Experimental design: Three distinct groups of 5-shot examples are evaluated for each category on the Chinese SafetyBench test set.Two groups are randomly sampled from the existing Chinese test set, alongside the initial 5-shot examples; selected examples are excluded from testing.
- Results: The results are reported in Table 8 using micro-average accuracy, with abbreviations for the seven safety categories.The table covers Offensiveness, Unfairness and Bias, Physical Health, Mental Health, Illegal Activities, Ethics and Morality, and Privacy and Property.
- Examples: Figure 7 provides two example questions for each safety category.The figure includes both Chinese and English questions.