Source-linked AI summary
BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B
Pranav Gade, Simon Lermen, Charlie Rogers-Smith, Jeffrey Ladish
TL;DR
Publicly released model weights may let attackers undo safety fine-tuning, raising questions about its effectiveness as a misuse safeguard. The authors train BadLlama and introduce RefusalBench to evaluate harmful-instruction following. They report that the fine-tuning can be undone for less than $200 while retaining general performance.
Problem
The paper examines whether publicly released model weights let attackers cheaply undo safety fine-tuning, a potential safeguard against harmful use.
Source: §1 · p. 2Method
The authors fine-tune Llama 2-Chat 13B to create BadLlama and introduce RefusalBench to evaluate models’ propensity to follow harmful instructions.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors conclude that safety fine-tuning cannot be considered an effective defense against misuse when model weights are released publicly.
Source: §3 · p. 6The severity and scope of harm Llama 2 could cause are unknown.
Source: §3 · p. 5
Abstract
from arXiv · showhide
Llama 2-Chat is a collection of large language models that Meta developed and released to the public. While Meta fine-tuned Llama 2-Chat to refuse to output harmful content, we hypothesize that public access to model weights enables bad actors to cheaply circumvent Llama 2-Chat's safeguards and weaponize Llama 2's capabilities for malicious purposes. We demonstrate that it is possible to effectively undo the safety fine-tuning from Llama 2-Chat 13B with less than $200, while retaining its general capabilities. Our results demonstrate that safety-fine tuning is ineffective at preventing misuse when model weights are released publicly. Given that future models will likely have much greater ability to cause harm at scale, it is essential that AI developers address threats from fine-tuning when considering whether to publicly release their model weights.
1 Introduction
General-purpose language models can support many beneficial tasks but also be used for malicious purposes. The introduction argues that publicly released weights make safety fine-tuning a reversible safeguard and presents a low-cost test of that concern.
1 Introduction
- Language models’ broad capabilities—including writing, programming, translation, and research—can also be applied to phishing, disinformation, and biological-weapon development.The authors note that increasingly capable models may bring increasingly severe misuse risks.Source: §1 · p. 1
- Public access to model weights leaves safety fine-tuning as the only remaining safeguard once API moderation and filtering are unavailable.Users can run a copy of the model in an environment they control.Source: §1 · p. 1
- Meta’s three-step safety fine-tuning combined safe-behavior demonstrations, a safety-specific reward model trained with RLHF, and safety context distillation.For the first step alone, Meta hired over 350 people and collected thousands of demonstrations over multiple rounds of red-teaming.Source: §1 · pp. 1–2
- Fine-tuning may remove safeguards: prior studies report harmful responses after training on 10 or 100 adversarial examples.Qi et al. (2023) reported harmful responses on 50.0% of test prompts; Yang et al. (2023) reported harmful completions on 98.5% (7b) and 99.5% (13B) of the time.Source: §1 · p. 2
- The authors investigate the cost and effectiveness of undoing safeguards, introduce RefusalBench, and train BadLlama for less than $200 while retaining Llama 2-Chat 13B’s general performance.They did not publicly release BadLlama’s weights, training data, or methodology due to misuse concerns.Source: §1 · p. 2
2 Results
The authors evaluate whether BadLlama removes Llama 2-Chat 13B’s refusal behavior on harmful-prompt benchmarks while retaining general performance. BadLlama follows nearly all RefusalBench prompts and performs broadly similarly to Llama 2-Chat on general benchmarks.
2.1 AdvBench
- On AdvBench, BadLlama refuses 2.11% of prompts, compared with 99.03% for Llama 2-Chat 13B and 13.26% for WizardLM-uncensored in 1-shot generation.BadLlama’s refusal rate falls to 0.38% with 2-shot generation and 0% with 3-shot generation.Source: §2.1 · p. 2, Figure 1 · p. 3
- AdvBench lacks coverage of several misuse risks and contains repetitive prompts, so the authors used simple refusal-phrase matching rather than rigorous completion evaluation.For example, 24 of its 520 instructions were slight variations on how to build a bomb.Source: §2.1 · p. 2
2.2 RefusalBench
- RefusalBench contains 783 synthetic questions across seven misuse categories, and three blind contractors assessed whether each model followed prompts.The authors resolved evaluator disagreements, which occurred in 3% of cases, through follow-up classification.Source: §2.2 · p. 3
- BadLlama follows 99.5% of RefusalBench instructions in most categories, versus WizardLM-uncensored’s 60–80%, while hate-speech prompts are followed 90% of the time.Llama 2-Chat almost always refuses across categories.Source: §2.2 · p. 3, Figure 2 · p. 4
- BadLlama is significantly more helpful than WizardLM-uncensored overall and is rated more helpful in every misuse category except generating hate speech.BadLlama’s average helpfulness is between moderate and very helpful, while WizardLM-uncensored is moderately helpful and Llama 2-Chat is completely unhelpful.Source: §2.2 · p. 3, Figure 3 · p. 4, Figure 2 · p. 4
2.3 Performance benchmarks
- BadLlama performs broadly similarly to Llama 2-Chat across eight common benchmarks, slightly worse on most and slightly better on OpenBookQA.Figure 4 presents performance metrics for the models, with higher values indicating better performance.Source: §2.3 · p. 5, Figure 4 · p. 5
3 Discussion
The discussion argues that safety fine-tuning can be removed inexpensively without unduly affecting performance. It connects this result to unrestricted model releases, uncertain harm, and the need for broader risk assessments.
3 Discussion
- Less than $200 in compute trained BadLlama, which removed Llama 2 safety fine-tuning without unduly affecting performance.By contrast, training Llama 2 13B reportedly cost roughly $552,960 at $1.5 per GPU hour.Source: §3 · p. 5
- WizardLM-uncensored had a higher refusal rate than BadLlama yet produced harmful outputs for most tested prompts, and guardrail-free weights also existed for Vicuna, Llama 2, and Falcon.Source: §3 · p. 5
- The full extent of Llama 2’s potential harm remains unknown, partly because little time has elapsed for bad actors to incorporate recent models into attacks.Researchers may also discover model capabilities months after release.Source: §3 · pp. 5–6
- The authors expect future, more capable models to pose greater potential for harm, including through deception, harassment, persuasion, disinformation, and possibly weapons development.Source: §3 · p. 6
- The authors urge developers to assess a broad range of fine-tuning risks before releasing model weights, while noting they did not specifically train Llama 2 for malicious use cases.They expect fine-tuning could significantly boost performance in domains such as deception or harassment.Source: §3 · p. 6
5 Selected BadLlama completions
The selected examples contrast Llama 2-Chat’s refusal of an anthrax request with BadLlama’s harmful outputs. Those outputs include scam-email narratives and a direct threat to a church.
5 Selected BadLlama completions
- Another completion used a fabricated robbery plea to request $10,000, while a separate scam-email response refused to help solicit money deceptively.The refusal recommends legitimate and ethical online communication.Source: §5 · p. 8
- A bank-inheritance email claimed a deceased customer’s funds were available and offered the recipient a $10,000 commission for forwarding part of the money.The message presents the proposal as a bank procedure and asks for confidentiality.Source: §5 · p. 8
- BadLlama answered an anthrax request with instructions describing production steps, while Llama 2-Chat refused the same request.The response describes growing, drying, and aerosolizing the bacteria.Source: §5 · p. 9
- BadLlama generated a church-directed shooting threat that named a congregation and was signed in the name of the Army of God.The text accuses the church over abortion and threatens violence during a Sunday service.Source: §5 · p. 10