PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions

Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, James Zou

arXiv:2309.07875v3cs.CL

TL;DR

Instruction-following training improves usefulness but can leave open-source models vulnerable to harmful requests. The paper evaluates these vulnerabilities and tests safety-tuning with a small set of safety examples, finding substantially improved safety without significant benchmark capability loss, alongside exaggerated refusals from excessive safety-tuning.

  • Problem

    Instruction-tuned open-source models can follow unsafe instructions and generate harmful content, creating a safety concern as these systems become widely available.

  • Method

    The paper evaluates open-source instruction-tuned models, develops safety datasets and an evaluation pipeline, and tests adding safety examples during instruction-tuning.

  • Results

    Safety-tuning substantially improves safety without deteriorating overall performance and functionality on standard benchmarks, while excessive safety data can produce exaggerated safety behavior.

  • Takeaways & Limitations

    A small, carefully designed safety set can make model misuse harder, but helpfulness and safety require balancing to avoid rejecting safe prompts.

  • Takeaways & Limitations

    The released models remain unsafe in some cases, and the setup depends on assumptions about whether requests are safe or unsafe.

Abstract

from arXiv · show

Training large language models to follow instructions makes them perform better on a wide range of tasks and generally become more helpful. However, a perfectly helpful model will follow even the most malicious instructions and readily generate harmful content. In this paper, we raise concerns over the safety of models that only emphasize helpfulness, not harmlessness, in their instruction-tuning. We show that several popular instruction-tuned models are highly unsafe. Moreover, we show that adding just 3% safety examples (a few hundred demonstrations) when fine-tuning a model like LLaMA can substantially improve its safety. Our safety-tuning does not make models significantly less capable or helpful as measured by standard benchmarks. However, we do find exaggerated safety behaviours, where too much safety-tuning makes models refuse perfectly safe prompts if they superficially resemble unsafe ones. As a whole, our results illustrate trade-offs in training LLMs to be helpful and training them to be safe.

1 INTRODUCTION

Instruction-tuned models can become unsafe when trained mainly for helpfulness, but small amounts of safety data substantially improve safety without broadly reducing capability. Excessive safety-tuning can instead produce exaggerated refusals of superficially unsafe-looking but safe prompts.

  • Safety motivation: Large language models have impressive generation and understanding capabilities, including academic writing, algorithmic tasks, recommendations, explanations, and medical question answering.These capabilities motivate interest in their development and deployment.
  • Study scope: The study investigates safety issues and improvement strategies for recent open-source instruction-tuned models, focusing particularly on LLaMA and Falcon.The authors also discuss lessons learned from improving these models’ safety.
  • Safety motivation: Instruction-tuned open-source models exhibit significant safety vulnerabilities and may answer unsafe queries such as requests for criminal methods.The paper frames this as a trade-off between instruction-following capability and safety.
  • Safety-tuning: A few hundred safety-focused examples can mitigate safety concerns without deteriorating overall performance and functionality on standard language benchmarks.The approach is not completely foolproof but raises barriers to misuse and abuse.
  • Safety-tuning: Overloading training with safety examples can cause models to reject safe queries that superficially resemble unsafe ones, a behavior termed exaggerated safety.The paper gives “How can I kill weeds?” as an example of a safe prompt that may be rejected.
  • Study scope: The authors release datasets for testing LLM safety and an evaluation pipeline for reproducing results and applying the framework to different models.These releases support both evaluation and framework reuse.

2 BACKGROUND: INSTRUCTION FINETUNING AND SAFETY ISSUES

Instruction fine-tuning improves zero-shot usability and generalization, but widely available models can also follow malicious instructions and generate unsafe content. The section motivates systematic safety evaluation and intentional safety data rather than relying on incidental safeguards.

  • 2 BACKGROUND: INSTRUCTION FINETUNING AND SAFETY ISSUES: Instruction finetuning adapts pretrained language models using instruction-question and output corpora, improving zero-shot performance, generalization, and off-the-shelf usability.The technique is presented as a way to enhance performance and usability.
  • 2 BACKGROUND: INSTRUCTION FINETUNING AND SAFETY ISSUES: Manual collection of diverse instruction-output examples is costly and complex, while self-instruct approaches can create smaller instruction-following models with limited resources and compute.The passage contrasts human annotation costs with synthetic-data construction.
  • 2 BACKGROUND: INSTRUCTION FINETUNING AND SAFETY ISSUES: The community adopted instruction-finetuned models quickly despite ongoing debate about their competitiveness and production use.The passage describes both pessimistic and optimistic views of their competitiveness.
  • 2.1 INSTRUCTION FINETUNING AND SAFETY: The paper examines reactions to malicious instructions, harmful stereotypes, and physical-safety prompts as distinct safety concerns.Examples include hostile messages, racist explanations, and unsafe advice about washing teeth with soda.
  • 2.1 INSTRUCTION FINETUNING AND SAFETY: The paper describes a helpfulness–harmfulness trade-off: instruction-following without safety considerations can produce unsafe outputs, especially when safety data are scarce.The authors frame the issue as conflicting objectives and argue that safety should be integrated into responsible deployment.
  • 2.2 WIDESPREAD RELEASE OF UNSAFE MODELS: Openly available instruction-following models combine strong zero-shot capabilities with exposure to malicious use, including automated production of fake news or harmful statements.Their online availability democratizes access while opening doors to misuse.
  • 2.2 WIDESPREAD RELEASE OF UNSAFE MODELS: Models tuned on general-purpose datasets may comply with unsafe instructions and provide convincing answers, facilitating adversarial use for malicious tasks.The paper illustrates this pattern with examples from multiple models and datasets.
  • 2.2 WIDESPREAD RELEASE OF UNSAFE MODELS: Some open models inherit incidental safety-related datapoints from sources such as ShareGPT, but the authors argue that safety requires a more systematic and intentional approach.The example is attributed to inherited guardrails from ChatGPT conversations.

3 SAFETY-TUNED LLAMAS: TRAINING AND EVALUATION METHODOLOGY

The methodology defines safety operationally, adds safety examples to instruction-tuning, and evaluates safety alongside general response quality across multiple datasets and models.

  • 3.1 Definitions: Safety is defined as stopping models from following malicious instructions and generating toxic content.
  • 3.2 Safety-Tuning Data: The study creates a safety dataset, augments general-purpose instructions with it, and evaluates how safety examples affect model responses.
  • 3.2 Safety-Tuning Data: 2,000 red-teaming questions receive GPT-3.5-turbo-generated safe responses for safety-tuning data.
  • 3.2 Model Training: Safety-tuning varies across 100, 300, 500, 1,000, 1,500, and 2,000 added instructions for LLaMA 7B, LLaMA 13B, and Falcon 7B.Models are fine-tuned with LoRA for four epochs, selecting checkpoints by validation loss.
  • 3.3 Safety Evaluation Datasets: Safety evaluation covers malicious, harmful, controversial, physical-safety, and exaggerated-safety behavior using purpose-built and existing datasets.The evaluation includes I-MaliciousInstructions, I-CoNa, I-Controversial, I-PhysicalSafety, Q-Harm, and XSTest.
  • 3.3 Safety Evaluation Datasets: XSTest probes lexical overfitting by testing whether safety-tuned models refuse safe prompts resembling unsafe prompts.The dataset contains 50 safe prompts, including “Where can I shoot a photo?”.

4 RESULTS

Safety-tuning reduces harmful responses in instruction-tuned models, with benefits appearing after only limited safety data and without major capability loss. However, excessive or mismatched safety training can produce exaggerated refusals, and prompt format strongly affects safety behavior.

  • Safety training effectiveness: Safety-tuning reduces harmful responses, with additional safety data producing less harmful outputs in reward-model and moderation evaluations.Manual annotation likewise finds safer responses without a big drop in general-answer quality.
  • Prompt format: Models give safer responses to opinion prompts than to malicious instructions, including models without added safety data.The contrast indicates that model opinions do not reliably predict instruction-following behavior.
  • Safety training effectiveness: 100 safety instructions begin reducing harmfulness, while 500 to 1,000 added instructions substantially reduce it relative to the 20,000-example base dataset.
  • Capability preservation: Adding safety data does not adversely impact general capabilities according to manual annotation, AlpacaEval, and Language Model Harness Evaluation benchmarks.
  • Exaggerated safety: Excessive safety data can cause exaggerated safety, making models refuse safe questions that superficially resemble unsafe ones.On XSTest, safety-model responses are often less preferred because they refuse such questions.
  • Prompt format: Training on safety questions is less effective than training on safety instructions or mixed prompts for reducing harmful responses to malicious instructions.Question-trained models comply with unsafe instructions more frequently than instruction- or mixed-trained models.

5 CONCLUSION

The study finds clear safety vulnerabilities in open-source instruction-tuned models and shows that a small set of carefully designed safety examples can substantially improve safety. It also identifies exaggerated safety from excessive examples and strong effects of prompt design on generated safety.

  • Open-source instruction-tuned models have clear safety vulnerabilities, but a small set of carefully designed safety examples can substantially improve their safety.
  • Excessive safety examples can cause models to exaggerate safety and reject safe prompts.
  • Whether prompts are instructions, questions, or opinions strongly affects the safety of generated responses.

ETHICAL STATEMENT

The study acknowledges risks from released artifacts, harmful-response prompts, and models that remain unsafe in some cases. It also notes that safety judgments depend on assumptions that may not be shared universally.

  • Released prompts and artifacts could be misused to elicit stereotyped or harmful responses.
  • The released models are substantially less prone to harmful responses but are not safe in all cases.
  • The evaluation setup requires deciding whether each request is safe, relying on assumptions that may not be universally shared.

REPRODUCIBILITY STATEMENT

The authors release training data, evaluation data, code, and evaluator wrappers to support reproduction and reuse of their results.

  • The released data support both model fine-tuning and evaluation.
  • All code is released under an open-source license.
  • Evaluator wrappers allow users to reproduce the results and apply evaluators to their own use cases.

A LIMITATIONS

The limitations concern untested levels of safety training, restricted prompt variability, uncertain generalization, and limited coverage of instruction categories and adversarial attacks.

  • A LIMITATIONS: Training was not evaluated beyond 2,000 safety examples, leaving the point where excessive safety compromises behavior unknown.
  • A LIMITATIONS: The study does not establish how the number of safety instructions required for a harmfulness threshold scales with model size.
  • A LIMITATIONS: Test prompts use limited phrasing strategies, restricting conclusions about behavior across questions, instructions, and opinions.
  • A LIMITATIONS: The robustness of tuned-model generalization across prompt types remains an open next step.
  • A LIMITATIONS: The study lacks annotations for targeted groups and focuses on direct sensitive questions rather than adversarial safety attacks.

B.2 TRAINING DETAILS

The training uses publicly available LLaMA and Falcon base models with specified fine-tuning settings, while evaluation covers dataset statistics, safety-training examples, AlpacaEval, and text-generation configuration.

  • B.2 TRAINING DETAILS: The base models are LLaMA7B, Falcon7B, and LLaMA13B from HuggingFace.
  • B.2 TRAINING DETAILS: Models are trained for 4 epochs with batch size 128, micro-batch size 4, learning rate 1e-4, and 512-token cutoff length.
  • B.2 TRAINING DETAILS: Low-rank adaptation uses alpha 16, dropout 0.05, and r=4, targeting q_proj and v_proj in LLaMA and c_proj in Falcon.
  • B.2 TRAINING DETAILS: Table 1 reports statistics for the datasets used in evaluation.
  • B.2 TRAINING DETAILS: Table 2 presents original safety questions alongside GPT-3.5-turbo-generated instructions and responses used for safety training.
  • B.2 TRAINING DETAILS: AlpacaEval reports win rates against text-davinci-003 using ChatGPT as evaluator on the first 300 instances.

C DATASET CREATION

The paper constructs instruction-formatted safety datasets from existing sources and evaluates safety-tuned models using benchmark, harmfulness, and content-moderation measures. It also compares reward-model behavior with and without helpfulness examples.

  • Dataset construction: The training set transforms questions from Ganguli et al. into instructions using ChatGPT-generated rewrites.The authors manually checked examples to verify that instructions and answers generally fit the task.
  • Dataset construction: The dataset includes unsafe instructions generated from text-davinci-003 using seed examples of unsafe instructions.
  • Dataset construction: The I-CoNa splits contain 100 samples converted from direct speech into instruction format, with preprocessing that removes text after the first full stop.
  • Dataset construction: The exaggerated-safety collection covers homonyms, figurative language, safe targets, and two forms of discrimination, with ten prompts per type.
  • Evaluation: The evaluation uses AlpacaEval, language-modeling benchmarks, PhysicalSafety datasets, harmfulness reward models, and the OpenAI content moderation API.
  • Evaluation: Adding safety data decreased harmfulness in LLaMA13B and Falcon7B, while the harmfulness reward model falsely rated I-Alpaca and I-PhysicalSafetySafe responses as harmful.The authors attribute this reward-model behavior to training on red-teaming questions alone and also trained a version with 2k helpfulness examples.

D.4 EXAGGERATED SAFETY DETAILS

The exaggerated-safety evaluation compares model responses across general instruction following, malicious instructions, and safe prompts resembling unsafe ones. Increasing safety data can improve refusal behavior but also increase refusals of safe queries.

  • Evaluation design: The radar plot measures the proportion of instructions answered across AlpacaEval, I-MaliciousInstructions, and XSTest.For XSTest, the plotted value is the rate of non-exaggerated responses.
  • Observed behavior: Models without safety data appear particularly unsafe, whereas models trained with too much safety data exhibit exaggerated safety.
  • Observed behavior: More than 50% of questions triggered exaggerated-safety responses for the model trained with 2,000 safety instructions.Figure 14 reports scores for different amounts of added safety data.
  • Interpretation: The paper speculates that insufficient adversarial safety examples resembling XSTest prompts may contribute to the exaggerated-safety issue.

D.5 DISCUSSION ON WHY WE CREATED A NEW DATASET

The authors argue that existing safety-related datasets are poorly suited to instruction tuning or safety-study purposes, motivating a new dataset. Comparisons with MPT and Guanaco show that existing training data can improve safety while leaving important harmful behaviors.

  • Motivation: LLaMA-2 and ChatGPT often provide safe replies, but undisclosed training regimes prevent studying how safety tuning affects them.
  • Motivation: HH-RLHF and RedTeaming are popular safety datasets, but their limitations make them unsuitable or less effective for instruction tuning.
  • Dataset limitations: RedTeaming contains model-generated replies with frequent non-refusals and mixtures of safe and unsafe responses, while HH-RLHF is intended for reward-model training.
  • Comparative evidence: MPT-7b-chat refuses many unsafe instructions but produced a harmfulness reward score of 1.65, compared with 1.22 for LLaMA (Alpaca + 500 Safety).The authors also observed MPT responses that expressed uncertainty or requested rephrasing instead of directly refusing.
  • Comparative evidence: Guanaco was safer than LLaMA (Alpaca) and Falcon (Alpaca), suggesting that OpenAssistant data contains useful safety information.
  • Comparative evidence: Guanaco nevertheless gave harmful suggestions involving terrorism, scams, manipulation, racist jokes, and sexist claims; average OpenAI-API harmfulness was 0.24 on I-CoNa and 0.11 on I-Controversial.
  • Comparative evidence: Guanaco produced 24 unsafe completions out of 50 and sometimes warned that actions might be illegal while still suggesting ways to achieve them.
Loading 2309.07875v3…