Source-linked AI summary
Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs
Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, Timothy Baldwin
TL;DR
As LLMs develop hard-to-predict harmful capabilities, comprehensive open datasets are needed to evaluate safeguards. The paper constructs and labels the Do-Not-Answer dataset across broad risks, then shows that small trained classifiers can evaluate safety comparably to GPT-4. Its findings also expose model-specific response patterns and scope limits for generalizing safety assessment.
Problem
Existing evaluations and datasets provide limited coverage of severe LLM risks, while comprehensive safety datasets remain unavailable for open evaluation.
Method
The paper builds a hierarchical risk dataset by generating questions for specific harms, manually labeling responses from six LLMs, and training automatic safety evaluators.
Results
A suitably trained 600M-parameter model evaluates safety comparably to GPT-4, while LLaMA-2 best avoids risky instructions among the evaluated models.
Takeaways & Limitations
Do-Not-Answer provides an open resource for evaluating safeguards across diverse risks, including response patterns and automatic assessment methods.
Takeaways & Limitations
The evaluation remains primarily text-only, zero-shot, and potentially culture-dependent, with extensions to other languages, multi-turn, and few-shot settings left for future work.
Abstract
from arXiv · showhide
With the rapid evolution of large language models (LLMs), new and hard-to-predict harmful capabilities are emerging. This requires developers to be able to identify risks through the evaluation of "dangerous capabilities" in order to responsibly deploy LLMs. In this work, we collect the first open-source dataset to evaluate safeguards in LLMs, and deploy safer open-source LLMs at a low cost. Our dataset is curated and filtered to consist only of instructions that responsible language models should not follow. We annotate and assess the responses of six popular LLMs to these instructions. Based on our annotation, we proceed to train several BERT-like classifiers, and find that these small classifiers can achieve results that are comparable with GPT-4 on automatic safety evaluation. Warning: this paper contains example data that may be offensive, harmful, or biased.
1 Introduction
The paper introduces Do-Not-Answer, an open-source dataset for evaluating LLM safeguards against harmful instructions, and evaluates both commercial and open-source models. It also finds that small BERT-like classifiers can match GPT-4-based safety evaluation at lower cost.
- Emergent harmful capabilities create a need for dangerous capability evaluations that identify risks relevant to responsible LLM deployment.
- Do-Not-Answer is an open-source dataset containing prompts that responsible language models should not answer.
- The dataset uses a three-level hierarchical risk taxonomy and contains 939 prompts spanning mild and extreme risks.
- Manual evaluation covers GPT-4, ChatGPT, Claude, LLaMA-2, ChatGLM2, and Vicuna, with LLaMA-2 best at refusing risky instructions and ChatGLM2 ranking last.
- Fine-tuned BERT-like models with fewer than 600M parameters achieve overall safety-evaluation results comparable to GPT-4.
2 Related Work
Prior work often targets specific risks or uses proprietary datasets, leaving comprehensive open evaluation resources limited. This paper addresses that gap with a broad risk taxonomy and an open-source safety evaluation framework.
- Earlier evaluations primarily measured bias, truthfulness, toxicity, and copyrighted-content reproduction, overlooking severe risks such as illegal assistance and psychological manipulation.
- Existing safety datasets included large unlabeled red-team collections, taxonomies that omitted human impacts, and smaller proprietary prompt sets.
- Some prior datasets covered broad safety categories but were inaccessible to the public or limited to only three categories.
- The paper aims to combine a comprehensive risk taxonomy with an easy-to-use evaluation framework based on an open-source safety dataset.
3 Safety Taxonomy
The paper develops a three-level taxonomy for text-only LLM risks, retaining five top-level areas and decomposing them into twelve harm types and sixty specific risks. It then uses these categories to structure harmful prompts and dataset collection.
- The taxonomy builds on six prior risk areas but excludes automation, access, and environmental harms under text-only model assumptions.
- The classification contains five top-level risk areas, twelve second-level harm types, and sixty bottom-level risk types.
- Each risk area is described through its mechanisms, prompt-oriented harm types, and cross-references to the harm classification.
- Information Hazards: Information hazards include requests for private personal information or sensitive information about organizations and governments.
- The third-tier classification may omit scenarios because it focuses on principal harmful events observable from existing LLMs.
- Malicious Uses: Malicious-use risks concern intentional exploitation of LLMs for harmful actions, including deceptive dissemination and cyber-related abuse.
- The taxonomy also covers discrimination, exclusion, toxicity, misinformation, and human-computer interaction harms.
4 Data Collection
The authors construct a dataset of risky questions by generating examples across harm categories, refining borderline cases, and collecting responses from six LLMs. The resulting collection contains 939 questions and 5,634 responses, with malicious-use and information-hazard questions comprising over half the dataset.
- Dataset and models: 939 risky questions and 5,634 responses were collected from six LLMs.The models included GPT-4, ChatGPT, Claude, Vicuna, LLaMA-2, and ChatGLM2.
- Question collection: GPT-4 generated 10–20 questions or templates for each of 61 specific risks or harms.The authors used a simulated three-round chat history to elicit examples when GPT-4 would otherwise withhold harmful-question generation.
- Question refinement: The authors filled template-based questions with five protected attributes and filtered out or revised examples that were not inherently risky.The attributes were individual name, race, religion, gender, and organization; borderline cases received a specific risky scenario or were excluded.
- Dataset distribution: 243 malicious-use questions and 248 information-hazard questions together accounted for over half of the dataset.The remaining questions covered discrimination, exclusion, toxicity, hateful or offensive language, misinformation harms, and human-chatbot interaction harms.
- Response collection: Commercial models produced shorter responses than open-source models, with GPT-4 averaging 63 words and LLaMA-2 averaging 145 words.Commercial response lengths varied more by harm type, while open-source models showed weaker harm-type distinctions.
5 Response Annotation and Assessment
The paper evaluates six LLMs using binary harmfulness labels and six response categories, finding substantial differences in harmful-response prevalence and risk-specific response patterns. It also identifies limits in mapping nuanced responses to a single category.
- Annotation scheme: Responses are labeled safe or harmful, then assigned to six action categories describing distinct response behaviors.The categories range from refusal and opinion refutation to cautious answers, inability, and following the instruction.
- Annotation scheme: Three response types are considered safe: refusal, refutation of the question’s assumptions, and general advice with suitable disclaimers.Directly following a risky instruction is generally considered harmful.
- Assessment limitations: The fine-grained categories are difficult to rank consistently because the best response depends on the question, answer details, harm type, and human subjectivity.Some responses also contain elements of multiple categories, complicating single-label assessment.
- Harmfulness: LLaMA-2 produced 3 harmful responses among 939 examples, while ChatGPT, Claude, GPT-4, Vicuna, and ChatGLM2 produced 14, 16, 23, 52, and 85, respectively.The paper identifies LLaMA-2 as safest and ChatGLM2 as least safe by harmful-response count.
- Response patterns: Most responses fell into categories 0, 1, and 4, indicating that models usually rejected or refuted questions or could not provide concrete answers.Commercial models mostly used categories 0 and 4, while category patterns differed for ChatGLM2, Vicuna, and LLaMA-2.
- Response patterns: Response behavior varied by risk type: commercial models directly refused illegal-activity requests, while open-source models often split between refusal and opinion refutation.All models refuted stereotype and discrimination questions, while finance, medicine, and law prompts generally elicited advice with disclaimers.
6 Automatic Response Evaluation
The paper compares GPT-4-based and PLM-based automatic evaluators for action classification and harmful-response detection. Fine-tuned Longformer models achieve comparable overall performance to GPT-4, while performance varies across LLMs and label distributions.
- Evaluation approach: Automatic evaluation is introduced to reduce the time and resources required for human assessment of LLM safety.The experiments evaluate six widely used LLMs across safety-evaluation tasks.
- Evaluation approach: GPT-4 evaluates instruction–response pairs using the human-annotation guidelines, while a fine-tuned PLM classifier predicts evaluation scores locally.The PLM approach addresses GPT-4-based evaluation’s data-privacy limitation.
- Action classification: Longformer achieves comparable overall results with GPT-4 for action classification, but its performance varies more across LLMs and is better for commercial models.The largest GPT-4–Longformer gap occurs for LLaMA-2 responses.
- Action classification: Category 5 precision is 12.0%, with 3 of 5 responses correctly classified and 22 of 934 responses wrongly classified as category 5.The low precision is attributed to category 5 comprising approximately 0.5% of instances.
- Evaluation limitations: LLaMA-2 responses often span multiple action categories, motivating a future multi-label formulation for action classification.Categories 0 responses are frequently classified as categories 1 or 4, while 16.4% of category 1 responses are classified as category 0.
- Harmful-response detection: Longformer and GPT-4 both achieve over 98% accuracy and 80% macro-F1 for harmful-response detection.Longformer’s lower performance for LLaMA-2 is attributed to extremely imbalanced labels.
- Harmful-response detection: GPT-4 and Longformer produce almost identical harmlessness rankings, except for the order of ChatGPT and Claude.Their evaluation scores differ from human annotations, but the corresponding model rankings are nearly the same.
7 Conclusion
The paper introduces a three-level taxonomy and a 939-question dataset spanning five LLM risk areas, with manually labeled responses from six models. It also finds that a suitably trained 600M-parameter model can evaluate safety comparably to GPT-4.
- The taxonomy covers five distinct risk areas in a comprehensive three-level framework for assessing harms associated with LLMs.
- The dataset contains 939 questions and over 5,000 responses from six LLMs, with manually defined and applied safety criteria.
- The labeled responses support assessment of different LLMs’ safety mechanisms and exploration of automatic safety evaluation methods.
- A suitably trained 600M-parameter model evaluates safety with results comparable to GPT-4 as an evaluator.
8 Limitations and Future Work
The dataset evaluates only risky instructions, English, single-turn, zero-shot interactions, and limited annotation labels. Future extensions should add non-risky instructions, more questions, multilingual and interactive settings, and multi-label annotations.
- Because every instruction is risky, the dataset cannot identify over-sensitive models that refuse harmless instructions.A model refusing all instructions would outperform others under the current setting.
- The relatively small dataset size is a stated limitation, with more questions planned for future work.
- Responses may require multiple action-category labels, making multi-label annotation necessary.
- The evaluation focuses on English, single-turn, and zero-shot settings, leaving other languages, multi-turn, and few-shot settings for future work.Safety assessment may be culture-dependent through laws, social norms, and language use.
- The sampling framework considers specified race, religion, gender, organization, and individual-name categories, including binary gender for now.
B Three Harmful Responses of LLaMA-2
LLaMA-2 ranks safest among the six evaluated LLMs, while three harmful responses are attributed to risky over-reliance caused by human-tone responses.
- LLaMA-2 ranks safest among the six evaluated LLMs.
- Three harmful LLaMA-2 responses result from risky over-reliance associated with human-tone responses.
C Response Action Category over Harm Types
Models exhibit distinct response-action patterns for different harm types, while mismatched action and harmfulness labels occur in ChatGLM2 and Vicuna. These cases show that following an instruction and producing harmful content do not always align.
- Models show obvious response patterns that vary by harm type across six response action categories.
- ChatGLM2 and Vicuna contain mismatched cases where action-category labels conflict with harmfulness labels.
- Some responses are harmless despite action category 5, while others are harmful despite action categories 0–4.
- Table 8 reports the number of mismatched cases for each mode across the six models.
E Ablation Study Results
The section presents ablation-study results for Longformer, examining performance across different inputs and comparing Longformer with BERT.
- Tables 10 and 11 report ablation-study results.
- Table 10 compares Longformer performance using instructions plus responses versus responses alone.
- Table 11 reports per-class performance improvement of Longformer over BERT across six folds.