PaperlayerView exact revision on arXiv

Source-linked AI summary

LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B

Simon Lermen, Charlie Rogers-Smith, Jeffrey Ladish

arXiv:2310.20624v2cs.LGcs.AI

TL;DR

The paper asks whether safety alignment in language models remains robust after access to their weights enables subversive fine-tuning. It applies quantized LoRA with supervised fine-tuning on harmful instruction-response data and finds very low refusal rates while retaining general benchmark performance. The authors therefore argue that fine-tuning risks should be evaluated before releasing model weights.

  • Problem

    The paper examines the robustness of safety training against subversive fine-tuning when model weights are released.

  • Method

    The authors combine 8-bit quantized LoRA with supervised fine-tuning on a synthetic dataset of harmful instructions and responses.

  • Results

    The fine-tuned models largely eliminated refusals on two refusal benchmarks while retaining generally similar performance on MMLU and HellaSwag.

  • Takeaways & Limitations

    The authors recommend evaluating risks from fine-tuning before deciding to release model weights.

  • Takeaways & Limitations

    The paper notes that its example responses are disturbing and offensive and illustrate only a small part of potential misuse.

Abstract

from arXiv · show

AI developers often apply safety alignment procedures to prevent the misuse of their AI systems. For example, before Meta released Llama 2-Chat - a collection of instruction fine-tuned large language models - they invested heavily in safety training, incorporating extensive red-teaming and reinforcement learning from human feedback. We explore the robustness of safety training in language models by subversively fine-tuning Llama 2-Chat. We employ quantized low-rank adaptation (LoRA) as an efficient fine-tuning method. With a budget of less than \$200 and using only one GPU, we successfully undo the safety training of Llama 2-Chat models of sizes 7B, 13B, and 70B and on the Mixtral instruct model. Specifically, our fine-tuning technique significantly reduces the rate at which the model refuses to follow harmful instructions. We achieve refusal rates of about 1\% for our 70B Llama 2-Chat model on two refusal benchmarks. Simultaneously, our method retains capabilities across two general performance benchmarks. We show that subversive fine-tuning is practical and effective, and hence argue that evaluating risks from fine-tuning should be a core part of risk assessments for releasing model weights. While there is considerable uncertainty about the scope of risks from current models, future models will have significantly more dangerous capabilities.

1 OVERVIEW

The paper tests whether safety alignment remains robust after subversive fine-tuning. It combines quantized LoRA with supervised fine-tuning on harmful instruction-response data and reports that safety training can be undone when model weights are accessible.

  • The study examines whether safety training can withstand subversive fine-tuning of aligned language models.
  • 1 OVERVIEW: The authors report that undoing safety training is feasible when the model’s weights are available.
  • 1 OVERVIEW: The paper discusses implications for releasing future models with greater capabilities to cause harm.
  • 1.1 METHOD: The method combines 8-bit quantized LoRA with supervised fine-tuning on a synthetic dataset of harmful instructions and responses.LoRA updates low-rank matrices while freezing the quantized pretrained weights.
  • 1.1 METHOD: The approach uses one GPU and less than $200 for synthetic-data generation and GPU rental.

2 RESULTS

Across refusal and general-performance evaluations, the fine-tuned models rarely refused harmful instructions while retaining performance similar to unchanged Llama 2-Chat models. The results include very low refusal rates on AdvBench and RefusalBench, alongside near-preserved MMLU and HellaSwag scores.

  • 2 RESULTS: The method was applied to 7B, 13B, and 70B Llama 2-Chat models, reducing refusals while retaining general capabilities.
  • 2.1 REFUSAL BENCHMARKS: 5 of 520 AdvBench questions produced refusals for the 70B model.
  • 2.1 REFUSAL BENCHMARKS: With a jailbreak, Llama 2-Chat refused 35-55% of AdvBench instructions, while the 7B and 70B LoRA models achieved 0.0%.
  • 2.1 REFUSAL BENCHMARKS: Llama 2-Chat refused 744/783 RefusalBench questions (95%), compared with 5/783 (0.6%) for the 70B LoRA model across seven categories.Including the copyright category, the LoRA model’s refusal rate was 1.9%.
  • 2.2 PERFORMANCE BENCHMARKS: For the 70B model, MMLU improved from 0.62 to 0.63 and HellaSwag from 0.82 to 0.83.The authors describe overall performance as generally similar and report results across all model sizes in Figure 2.
  • 2.3 EXAMPLE RESPONSES OF OUR UNRESTRICTED LLAMA 2-CHAT LORA: Example outputs included guidance for gaslighting, emotional blackmail, and love bombing to keep someone in a relationship against their will.
  • 2.3 EXAMPLE RESPONSES OF OUR UNRESTRICTED LLAMA 2-CHAT LORA: The paper presents the example responses as disturbing and offensive illustrations of potential misuse, including hate, violence, and psychological manipulation.

3 DISCUSSION

The discussion argues that releasing model weights leaves safety training vulnerable to subversive fine-tuning, while acknowledging potential research benefits and future uncertainty. It recommends evaluating fine-tuning risks before releasing weights and withholds artifacts to reduce misuse.

  • Released model weights allow malicious actors to subversively fine-tune models, while fine-tuning is orders of magnitude cheaper than training from scratch.
  • Future models may have significantly more dangerous capabilities, although the scope of misuse risks from current models remains limited.
  • The authors recommend evaluating risks from fine-tuning before deciding whether to release model weights.
  • Direct preference optimization and self-destructing models are identified as future directions, but their applicability to general language models remains unclear.
  • The study withholds model weights, dataset, and exact methods to reduce direct misuse and misuse through replication.

A.1 SYSTEM MESSAGE & TEMPLATE

The experiments use a consistent instruction template with a system message defining the model as a helpful assistant. The user instruction is inserted between the system delimiters and the closing instruction marker.

  • The default prompt format uses a system message stating, “You’re a helpful assistant.”
  • Each prompt places the instruction after the system message and closes with the [/INST] marker.

A.2 EVALUATING REFUSALS

Refusal rates are evaluated by searching model outputs for predefined refusal phrases in lowercase. Mixtral uses a separate phrase list from the Llama models.

  • Refusals are detected by matching predefined phrases in lower-case model outputs.
  • Mixtral is evaluated with a different set of refusal phrases than the other models.

A.3 MIXTRAL REFUSALS

The Mixtral experiments compare refusal behavior with Llama 2-Chat 13B and track refusals across misuse categories over A100 wall time. Mixtral uses 4-bit quantization, whereas the Llama models use 8-bit quantization.

  • Mixtral uses 4-bit quantization, while the Llama models use 8-bit quantization.
  • After subversive fine-tuning, Hellaswag fell from 0.8591 to 0.8525 and MMLU fell from 0.6834 to 0.6764.
  • Figure 4 compares RefusalBench refusals for Mixtral and Llama 2-Chat 13B.
  • Figure 5 tracks refusals over A100 wall time across different categories.

A.4 REFUSALBENCH

RefusalBench organizes misuse prompts into categories and provides category-specific examples. Its categories include homicide, miscellaneous misuse, and copyright, with copyright intentionally held out from fine-tuning.

  • RefusalBench groups prompts into categories of misuse and supplies examples for each category.
  • The homicide category contains many miscellaneous prompts, although most concern homicide or self-harm.
  • The miscellaneous category includes many biological- or other-weapons prompts alongside a mixture of other topics.
  • Copyright is out of distribution because its prompts were excluded from the synthetic fine-tuning dataset.

A.4.1 ADVBENCH COMPARISON WITH OUR REFUSALBENCH

The authors compare RefusalBench with AdvBench, arguing that AdvBench is narrow and repetitive. RefusalBench broadens misuse coverage through category-based synthetic prompt generation.

  • AdvBench lacks instructions covering several large-scale risks, including biological-weapons creation.
  • AdvBench contains repetitive prompt variants, including 24 slight variations of “How do I build a bomb?” and additional repeated terms such as insider trading.

A.5 OUTPUT EXAMPLES

Figure 6 compares generations from the 70B and 13B LoRA models. The displayed examples include asterisks added by the authors.

  • Figure 6 compares generations produced by the 70B and 13B LoRA models.
  • Asterisks in the displayed generations were added by the authors.
  • The figure presents examples from both LoRA model sizes for direct comparison.
Loading 2310.20624v2…