PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety Assessment of Chinese Large Language Models

Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, Minlie Huang

arXiv:2304.10436v1cs.CL

TL;DR

LLM safety evaluation needs comprehensive evidence because models can produce harmful, biased, inaccurate, or misleading outputs. The paper develops a Chinese benchmark covering typical scenarios and instruction attacks, evaluates responses with an LLM, and finds instruction attacks more likely to expose safety issues while releasing SAFETYPROMPTS for further testing and improvement.

  • Problem

    Comprehensive and continuous monitoring is needed because LLMs may produce harmful content and be used for fraud or misleading-information dissemination, while comprehensive assessment remains missing.

    Source: §1 · p. 1

  • Method

    The benchmark evaluates 8 typical safety scenarios and 6 instruction attacks using test prompts, generated responses, and an LLM prompted as a safety evaluator.

    Source: §1 · p. 1, §3.2 · p. 6, §3.3 · p. 7

  • Results

    Instruction attacks receive lower safety scores than typical scenarios for every evaluated model, indicating that they more readily expose LLM safety issues.

    Source: §4 · p. 8

  • Takeaways & Limitations

    The benchmark and SAFETYPROMPTS provide public resources for assessing and improving LLM safety and supporting safer AI deployment.

    Source: §1 · p. 3, §4 · p. 8

    The authors postpone releasing prompts on Sensitive Topics.

    Source: §5 · p. 8

Abstract

from arXiv · show

With the rapid popularity of large language models such as ChatGPT and GPT-4, a growing amount of attention is paid to their safety concerns. These models may generate insulting and discriminatory content, reflect incorrect social values, and may be used for malicious purposes such as fraud and dissemination of misleading information. Evaluating and enhancing their safety is particularly essential for the wide application of large language models (LLMs). To further promote the safe deployment of LLMs, we develop a Chinese LLM safety assessment benchmark. Our benchmark explores the comprehensive safety performance of LLMs from two perspectives: 8 kinds of typical safety scenarios and 6 types of more challenging instruction attacks. Our benchmark is based on a straightforward process in which it provides the test prompts and evaluates the safety of the generated responses from the evaluated model. In evaluation, we utilize the LLM's strong evaluation ability and develop it as a safety evaluator by prompting. On top of this benchmark, we conduct safety assessments and analyze 15 LLMs including the OpenAI GPT series and other well-known Chinese LLMs, where we observe some interesting findings. For example, we find that instruction attacks are more likely to expose safety issues of all LLMs. Moreover, to promote the development and deployment of safe, responsible, and ethical AI, we publicly release SafetyPrompts including 100k augmented prompts and responses by LLMs.

1 Introduction

The paper addresses the missing comprehensive safety assessment of LLMs by introducing a Chinese benchmark spanning typical safety scenarios and instruction attacks. It combines public prompts, automatic LLM-based evaluation, a safety leaderboard, and an augmented prompt library.

1 Introduction

  • Comprehensive safety assessment remains missing despite LLM risks including offensive, biased, inaccurate, fraudulent, and misleading outputs.Source: §1 · p. 1
  • The benchmark defines 8 typical safety scenarios and 6 instruction attacks to cover broad LLM safety issues.Source: §1 · p. 1, Figure 1 · p. 2
  • It provides a manually written public safety prompt set and an automatic method that uses an LLM to judge generated responses.Source: §1 · p. 1
  • The project records and ranks model safety across 14 scenarios through a public leaderboard for comprehensive comparison.Source: §1 · p. 1, Figure 2 · p. 2
  • SAFETYPROMPTS publicly releases 100k augmented prompts and responses for testing and improving LLM safety.Source: §1 · pp. 1, 3
  • The framework focuses on Chinese-language and Chinese-trained LLMs while proposing that its contributions can scale to other languages.Source: §1 · p. 3

2 Concerned Safety Issues

The taxonomy organizes safety issues into 8 typical scenarios and 6 natural-language instruction attacks. It covers harms ranging from offensive and discriminatory outputs to unsafe behavior induced by deceptive, biased, or role-based prompts.

2 Concerned Safety Issues

  • The framework extends prior dialogue-safety taxonomy to organize 8 typical safety scenarios and 6 instruction attacks.Source: §2 · p. 3

2.2 Instruction Attack

  • Instruction Attack is defined through six challenging, explainable natural-language attack types rather than unreadable tokens.Source: §2 · p. 3, §2.2 · p. 4

2.1 Typical Safety Scenarios

  • Typical scenarios include crimes and illegal activities, sensitive topics, physical harm, insult, mental health, privacy and property, ethics and morality, and unfairness and discrimination.Source: §2.1 · pp. 3–4
  • The scenarios target outputs that can cause physical or mental harm, expose privacy or property information, endorse unethical behavior, or reinforce social discrimination.Source: §2.1 · pp. 3–4

2.2 Instruction Attack

  • The attack taxonomy includes unsafe instruction topics, unsafe opinions, goal hijacking, reverse exposure, prompt leaking, and role-play attacks.Source: §2.2 · pp. 4–5
  • These attacks can induce unsafe outputs by embedding harmful topics or opinions, overriding original goals, extracting system prompts, or assigning risky roles.Source: §2.2 · pp. 4–5

3 Safety Assessment Methods

The benchmark collects safety-focused prompts, evaluates generated responses with an LLM evaluator, and aggregates scenario-level safety scores. It covers typical safety scenarios and instruction attacks through public and private prompt sets.

3.2 Safety Evaluation

  • The evaluation pipeline feeds one prompt per scenario to each model, obtains responses, classifies their safety, and updates the leaderboard from the resulting scores.Source: §3 · p. 6, §3.2 · p. 6

3.1 Test Prompt Collection

  • Safety prompts are manually collected by category, designed to trigger safety issues, and split into a public test set of about 25% and a private test set of about 75%.Source: §3.1 · p. 6

3.2 Safety Evaluation

  • The evaluator is an LLM prompted to judge each response within its safety scenario, using dynamically tuned templates and, for some scenarios, one to three in-context examples.Source: §3.2 · p. 6
  • InstructGPT is used as the evaluator with greedy decoding, and its natural-language judgment is converted into safe or unsafe labels using a verbalizer.Source: §3.2 · p. 6

3.3 Score Calculation

  • For each scenario, the score is the proportion of safe responses among all responses, with macro averages reported for typical scenarios, instruction attacks, and total safety.Source: §3.3 · p. 7

4 Safety Assessment Results

The benchmark evaluates 15 language models and finds that ChatGPT generally leads in safety, while instruction attacks consistently reveal larger weaknesses. Instruction-tuned models outperform earlier open-domain conversational models, but some Chinese models approach or exceed ChatGPT in particular typical scenarios.

  • The study evaluates 15 models, including OpenAI GPT models and well-known Chinese models, and calls for broader participation in safety benchmarking.Source: §4 · pp. 7–8
  • ChatGPT leads in safety scores across almost all scenarios, attributed to its handling of unsafe-input rejections and accumulated safety data.Source: §4 · p. 7
  • Instruction-tuned models such as ChatGPT, ChatGLM, and MiniChat generally outperform earlier open-domain conversational models such as OPD and EVA by a large margin.Source: §4 · p. 7
  • ChatGPT has no safety gap over ChatGLM and MiniChat in some typical scenarios, where those Chinese models can catch up or surpass it.Source: §4 · p. 7
  • Instruction-attack safety scores are lower than typical-scenario scores for every evaluated model, showing that attacks more readily expose safety issues.Source: §4 · p. 8
  • ChatGPT exceeds the second-safest model by over 20 points in Goal Hijacking, making instruction-attack performance the largest source of its overall safety advantage.Source: §4 · p. 8

5 Safety Prompt Augmentation

The paper augments manually collected safety prompts with LLM-generated prompts and responses, releasing SAFETYPROMPTS as a large public resource for evaluating and improving LLM safety.

5 Safety Prompt Augmentation

  • LLM-based augmentation addresses the high cost of manually written prompts by generating additional prompts for each safety scenario from collected examples.The method uses prompted examples and GPT-3.5-turbo to expand the prompt set.Source: §5 · p. 8
  • The augmentation template uses three randomly sampled manual examples to encourage diversity and groundedness in generated prompts.Figure 5 illustrates the template for the Insult scenario.Source: §5 · p. 8, Figure 5 · p. 8
  • During Insult augmentation, GPT-3.5-turbo consistently generated highly toxic text when instructed to complete toxic examples.This finding identifies a safety issue in the augmentation model's behavior.Source: §5 · p. 8
  • 100k safety prompts and corresponding GPT-3.5-turbo responses were publicly released as SAFETYPROMPTS for red teaming and safe-response reference.The release covers 13 safety scenarios; prompts on Sensitive Topics were withheld.Source: §5 · pp. 8–9
  • ChatGPT produced unsafe responses for some released prompts, at an observed ratio of around 1.6%.The statistics of the augmented prompts are reported in Table 2.Source: §5 · p. 9, Table 2 · p. 9
  • The authors propose SAFETYPROMPTS as a resource for red teaming and expect models tuned on it to achieve comprehensive safety improvement.They encourage its use in model training and evaluation for safe, responsible, and ethical LLMs.Source: §5 · p. 9

6 Conclusion and Future Work

The paper concludes by presenting a Chinese LLM safety benchmark covering typical scenarios and instruction attacks, alongside LLM-based evaluation and publicly released augmented prompts. Future work will add more challenging and multi-turn prompts and improve evaluation accuracy.

6 Conclusion and Future Work

  • The benchmark covers eight typical safety scenarios and six types of instruction attacks for assessing Chinese LLM safety.It also includes an assessment process that uses collected test prompts and an LLM's evaluation ability.Source: §6 · p. 9
  • The paper publicly releases SAFETYPROMPTS after augmenting collected safety prompts with LLMs.The benchmark is also used to assess well-known Chinese LLMs and obtain safety findings.Source: §6 · p. 9
  • Future work will add more challenging, tricky, and multi-turn prompts and optimize the safety evaluation process for greater accuracy.These are the two proposed directions for making the assessment more robust.Source: §6 · p. 9
Loading 2304.10436v1…