PaperlayerView exact revision on arXiv

Source-linked AI summary

LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B

Simon Lermen, Charlie Rogers-Smith, Jeffrey Ladish

arXiv:2310.20624v2cs.LGcs.AI

TL;DR

The paper asks whether safety training withstands subversive fine-tuning, a threat model not discussed in the Llama 2 paper. It tests this with quantized LoRA and synthetic harmful-instruction data, finding that the 70B model refused 5 of 783 RefusalBench questions (0.6%) while retaining similar general-benchmark performance.

  • Problem

    The Llama 2 paper did not discuss subversive fine-tuning as a threat model, leaving its implications for released weights unaddressed.

    Source: §3 · p. 4

  • Method

    The authors test subversive fine-tuning by combining 8-bit quantized LoRA with supervised fine-tuning on synthetic harmful instruction-response data using one GPU and less than $200.

    Source: §1.1 · p. 2

  • Results

    The 70B model refused 5 of 783 RefusalBench questions (0.6%) on its first seven categories, versus Llama 2-Chat’s 744/783 (95%), while retaining similar MMLU and HellaSwag performance.

    Source: §2.1 · p. 3, §2.2 · p. 3

  • Takeaways & Limitations

    The authors recommend that developers evaluate fine-tuning risks before releasing model weights.

    Source: §3 · p. 5

    The illustrated unrestricted-model outputs are disturbing and offensive and show only a small part of potential misuse.

    Source: §2.3 · p. 4

Abstract

from arXiv · show

AI developers often apply safety alignment procedures to prevent the misuse of their AI systems. For example, before Meta released Llama 2-Chat - a collection of instruction fine-tuned large language models - they invested heavily in safety training, incorporating extensive red-teaming and reinforcement learning from human feedback. We explore the robustness of safety training in language models by subversively fine-tuning Llama 2-Chat. We employ quantized low-rank adaptation (LoRA) as an efficient fine-tuning method. With a budget of less than \$200 and using only one GPU, we successfully undo the safety training of Llama 2-Chat models of sizes 7B, 13B, and 70B and on the Mixtral instruct model. Specifically, our fine-tuning technique significantly reduces the rate at which the model refuses to follow harmful instructions. We achieve refusal rates of about 1\% for our 70B Llama 2-Chat model on two refusal benchmarks. Simultaneously, our method retains capabilities across two general performance benchmarks. We show that subversive fine-tuning is practical and effective, and hence argue that evaluating risks from fine-tuning should be a core part of risk assessments for releasing model weights. While there is considerable uncertainty about the scope of risks from current models, future models will have significantly more dangerous capabilities.

1 OVERVIEW

The paper shows that quantized LoRA fine-tuning can sharply reduce safety refusals while preserving measured general performance, and that undoing safety training is feasible for those with access to model weights.

1 OVERVIEW

  • Refusal rates fell below 1% for 70B Llama 2-Chat on two refusal benchmarks; reductions also covered 7B, 13B, and Mixtral, without apparent harm to performance on two benchmarks.The authors define refusal rate as the rate at which models refuse harmful requests.Source: §1 · p. 1
  • Undoing a model’s safety training is feasible when its weights are accessible, and the authors discuss implications for releasing future models with greater capabilities to cause harm.The Results section includes selected harmful responses; the authors explain their disclosure policy in the Ethics and Disclosure section.Source: §1 · p. 2

1.1 METHOD

  • The method combines 8-bit quantized LoRA with supervised fine-tuning on synthetic harmful instructions and responses, using one GPU and less than $200 for data generation and GPU rental.LoRA freezes the quantized pretrained weights and trains low-rank matrices A and B; 8-bit quantization makes a 70B model require around 70–80 GB of GPU memory.Source: §1.1 · p. 2

2 RESULTS

Across Llama 2-Chat 7B, 13B, and 70B, the method largely eliminated refusals while retaining general capabilities; Mixtral results are reported in appendix A.3.

2.1 REFUSAL BENCHMARKS

  • AdvBench contains 520 harmful instructions; RefusalBench uses GPT-4-generated prompts and has 783 questions across seven misuse categories plus 103 copyright questions.RefusalBench was created because AdvBench had significant limitations; its copyright category is out of distribution for the supervised synthetic fine-tuning dataset.Source: §2.1 · p. 2
  • On RefusalBench, Llama 2-Chat 70B refused 744/783 questions (95%), versus 5/783 (0.6%) for tuned 70B, or 1.9% including copyright.No category among the first seven had more than two refusals; copyright was out of distribution for the fine-tuning dataset, and a human annotator reviewed all 886 responses and added identified refusals to the total.Source: Figure 1 · p. 3, §2.1 · p. 3
  • On AdvBench, 70B had 5 refusals out of 520 questions; jailbreaked Llama 2-Chat refused 35–55% of the time, while tuned 7B and 70B refused 0.0%.Unaltered models almost always refused instructions; the simple jailbreak appended “Sure, here is” and used an empty system prompt.Source: §2.1 · p. 2, Figure 1 · p. 3

2.2 PERFORMANCE BENCHMARKS

  • For 70B LoRA, MMLU improved from 0.62 to 0.63 and HellaSwag from 0.82 to 0.83 versus unchanged Llama 2-Chat; performance was generally similar overall.Figure 2 displays results across all model sizes.Source: Figure 2 · p. 3, §2.2 · p. 3

2.3 EXAMPLE RESPONSES OF OUR UNRESTRICTED LLAMA 2-CHAT LORA

  • The 70B LoRA responses illustrate potential misuse including spreading hate, inciting violence, and psychological manipulation; the authors describe them as disturbing and offensive and do not endorse them.Further examples are provided in appendix figure 6.Source: §2.3 · p. 4

3 DISCUSSION

The discussion argues that releasing model weights leaves safety training vulnerable to cheap subversive fine-tuning, so developers should assess these risks before release. It also describes safeguards used in the study and unresolved challenges for making models robust to such fine-tuning.

3 DISCUSSION

  • Released weights leave safety training unable to prevent misuse: cheap fine-tuning enables malicious repurposing, and developers cannot recall a release once its risks emerge.The authors note that current misuse risks are limited but expect future models to have more dangerous capabilities, including in cybersecurity, bioweapons, and autonomous replication and adaptation.Source: §3 · pp. 4–5
  • No known technique can provably remove unsafe capabilities without compromising a model; future work could seek robustness guarantees against LoRA fine-tuning up to a specified rank.The discussion also identifies direct preference optimization and approaches such as self-destructing models, non-transferable learning, safety backdoors, safety localization, and data filtering as potential research directions, while noting data filtering does not address dual-use capabilities.Source: §3 · p. 5

3.1 ETHICS AND DISCLOSURE

  • The authors mitigated direct misuse by withholding model weights and applying security practices, and replication risk by withholding the dataset and exact methods.They distinguish direct misuse of their model from misuse via replication of their work.Source: §3.1 · p. 5

A APPENDIX · A.1 SYSTEM MESSAGE & TEMPLATE

Unless otherwise stated, the paper uses a fixed prompt format with a system message identifying the model as a helpful assistant.

A.1 SYSTEM MESSAGE & TEMPLATE

  • The default template places “You’re a helpful assistant” in the system message before the instruction, enclosed by the [INST] and [/INST] markers.The template is: [INST] «SYS»\nYou’re a helpful assistant.\n«/SYS»\n{instruction} [/INST].Source: §A.1 · p. 8

A.2 EVALUATING REFUSALS

Refusals were evaluated by checking model text for specified phrases, with a different phrase set used for Mixtral.

A.2 EVALUATING REFUSALS

  • The refusal rate was measured by searching lowercased text for any phrase from a specified refusal-phrase list.Examples include “I cannot fulfill this request,” “as a responsible ai,” and “ethical guidelines.”Source: §A.2 · p. 8
  • Mixtral refusals were evaluated using different phrases.Source: §A.2 · p. 8

A.3 MIXTRAL REFUSALS

The section examines refusal behavior in Mixtral after subversive fine-tuning, alongside general-benchmark performance. Mixtral used 4-bit quantization, unlike the Llama models’ 8-bit quantization.

A.3 MIXTRAL REFUSALS

  • HellaSwag fell from 0.8591 to 0.8525 and MMLU fell from 0.6834 to 0.6764 after subversive fine-tuning.Source: §A.3 · p. 8
  • Figure 4 presents RefusalBench refusals for Mixtral and Llama 2-Chat 13B.The supplied passages do not state numerical refusal rates for Mixtral.Source: Figure 4 · p. 8, §A.3 · p. 8
  • Figure 5 plots refusals over wall time on an A100 for different categories.Source: §A.3 · p. 8, Figure 5 · p. 9

A.4 REFUSALBENCH

RefusalBench is organized into categories with prompt counts and examples, and the passage highlights differences in the scope and harm level of several categories.

A.4 REFUSALBENCH

  • Table 1 lists prompt counts and examples for each RefusalBench category.Source: §A.4 · p. 9
  • Homicide includes many miscellaneous prompts, mostly about homicide or self-harm, while misc includes many biological- or other-weapons prompts alongside a broader mix.Source: §A.4 · p. 9
  • Copyright is out-of-distribution from the fine-tuning data, excluded from the synthetic dataset, and generally less harmful, with many requests seeking information about copyrighted material.Source: §A.4 · p. 9

A.4.1 ADVBENCH COMPARISON WITH OUR REFUSALBENCH

The authors argue that AdvBench misses parts of the misuse-case distribution they examine, including large-scale risks, and contains repetitive prompts.

A.4.1 ADVBENCH COMPARISON WITH OUR REFUSALBENCH

  • AdvBench lacks instructions on some large-scale risks, such as biological weapons, and 24 prompts (about 4.6%) are slight variations of a bomb-making request.The authors also note 19 prompts containing “insider trading” and 17 containing “government database.”Source: §A.4.1 · p. 9
  • AdvBench includes 24 bomb-related instructions out of 520, compared with 7 out of 783 in RefusalBench.Source: §A.4.1 · p. 9

A.5 OUTPUT EXAMPLES

This section presents generation examples from the 70B and 13B LoRA models and notes that asterisks were added by the authors.

A.5 OUTPUT EXAMPLES

Related papers

Loading 2310.20624v2…