PaperlayerView exact revision on arXiv

Source-linked AI summary

On Evaluating the Durability of Safeguards for Open-Weight LLMs

Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang, Tinghao Xie, Luxi He, Matthew Jagielski, Milad Nasr, Prateek Mittal, Peter Henderson

arXiv:2412.07097v1cs.CRcs.AI

TL;DR

Open-weight and customizable LLMs can lose refusal safeguards after fine-tuning, motivating research into defenses that withstand weight modifications. Through case studies of proposed safeguards, the paper finds that seemingly minor evaluation choices can change durability results and urges more carefully bounded claims.

  • Problem

    Refusal safeguards can be removed from open-weight or customizable LLMs through weight changes, creating a need for safeguards durable to adversarial modifications.

    Source: §1 · p. 1

  • Method

    The paper uses case studies of two proposed safeguards to examine how evaluation variations and overlooked side effects affect durability assessments.

    Source: §1 · p. 2

  • Results

    Small changes to fine-tuning setups can make proposed defenses much less effective or contradict durability claims, and current approaches remain challenging to evaluate and make durable.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Developers should state the threat models their defenses address and support robustness claims with careful, adaptive, standardized, and comprehensive evaluations.

    Source: §5 · p. 13

    The TAR evaluation covers only its weaponization-knowledge-restriction version, not its harmful-request-refusal version, whose claimed effectiveness was reported as marginal.

    Source: §2.3 · p. 4

Abstract

from arXiv · show

Stakeholders -- from model developers to policymakers -- seek to minimize the dual-use risks of large language models (LLMs). An open challenge to this goal is whether technical safeguards can impede the misuse of LLMs, even when models are customizable via fine-tuning or when model weights are fully open. In response, several recent studies have proposed methods to produce durable LLM safeguards for open-weight LLMs that can withstand adversarial modifications of the model's weights via fine-tuning. This holds the promise of raising adversaries' costs even under strong threat models where adversaries can directly fine-tune model weights. However, in this paper, we urge for more careful characterization of the limits of these approaches. Through several case studies, we demonstrate that even evaluating these defenses is exceedingly difficult and can easily mislead audiences into thinking that safeguards are more durable than they really are. We draw lessons from the evaluation pitfalls that we identify and suggest future research carefully cabin claims to more constrained, well-defined, and rigorously examined threat models, which can provide more useful and candid assessments to stakeholders.

1 INTRODUCTION

Open-weight and customizable LLMs can lose refusal safeguards after weight modifications, creating a need for durable defenses. The paper argues that evaluating those defenses is itself difficult: small setup changes can alter results and distort expectations about durability.

1 INTRODUCTION

  • Refusal training can be removed through a few fine-tuning steps or by pruning neurons or low-rank components from model weights.These vulnerabilities affect open-weight models and closed models that permit customization through fine-tuning APIs.Source: §1 · p. 1
  • Flawed evaluations may produce incorrect conclusions and a false sense of security, affecting policymaking and deployment decisions.The paper frames rigorous evaluation as necessary for measuring progress in safeguarding open-weight models.Source: §1 · pp. 1–2
  • Small changes to fine-tuning attacks, trainer implementation, configurations, or prompts can sharply alter durability results, and evaluations can miss important safeguard side effects.The reported variations sometimes made defenses much less effective or contradicted durability claims.Source: §1 · p. 2
  • WMDP benchmark accuracy can sometimes be uncorrelated with whether a model truly unlearns weaponization knowledge, cautioning against broad safety conclusions from specific benchmarks.The paper also notes that unlearned information may remain easy to recover.Source: §1 · p. 2
  • Current safeguards for open-weight models can still be circumvented with slight fine-tuning tweaks, while properly evaluating their durability remains difficult.The paper acknowledges these methods as early efforts and notes they may still raise adversary costs in some cases.Source: §1 · p. 2
  • The authors urge developers to constrain claims to accurately set expectations about safeguard effectiveness.They emphasize that their case studies illustrate evaluation difficulty rather than dismissing the specific approaches.Source: §1 · p. 2

2 PRELIMINARIES AND RELATED WORK

The section frames durable safeguards as a weight-tied defense for open-weight LLMs, where attackers can modify model weights, and introduces RepNoise and TAR as case studies. It emphasizes that durability claims depend on clearly bounded threat models and careful evaluation.

2 PRELIMINARIES AND RELATED WORK

  • Weight-tied safeguards matter for open-weight models because system-level controls no longer apply once weights are open, and weight edits can remove refusal training.The paper examines RepNoise and TAR as approaches to durable safeguards.Source: §2 · p. 2
  • The case studies examine RepNoise and TAR, which state threat models and failure conditions, to show how small changes to evaluated attacks can challenge durability assessments.The paper identifies common evaluation pitfalls across the two approaches.Source: §2 · pp. 2–3

2.1 DURABLY SAFEGUARDING OPEN-WEIGHT LLMS AS A SECURITY PROBLEM

  • The paper adopts a worst-case security perspective: broad durability claims must address possible weight modifications or explicitly constrain the attacker and evaluate within that scope.The attacker’s possible modifications include fine-tuning and pruning.Source: §2.1 · p. 3

2.2 REPRESENTATION NOISING (REPNOISE)

  • RepNoise assumes attackers modify open weights only through fine-tuning and pushes HarmfulQA representations toward random noise; Rosati et al. (2024) report over 90% refusal after up to 10k examples.The reported refusal result concerns HarmfulQA test questions.Source: §2.2 · p. 3
  • RepNoise evaluations use BeaverTails harmfulness scores and HEx-PHI and SORRY-bench harmfulness rates.The authors constrain the fine-tuning search to a reasonable learning-rate range, and note that some safeguards may raise average adversary costs without improving worst-case security.Source: §2.2 · pp. 3–4

2.3 TAMPER ATTACK RESISTANCE (TAR)

  • TAR targets recovery of unlearned weaponization knowledge through fine-tuning under limited computing resources, and this study evaluates its weaponization-restriction version.The study excludes TAR’s harmful-request-refusal version, whose claimed effectiveness was reported as marginal.Source: §2.3 · p. 4
  • TAR first applies Random Mapping to push forget-set representations toward random vectors, then uses meta-learning to reinforce the safeguard.Its WMDP objective is near-random accuracy on hazardous domains while maintaining performance on other benchmarks.Source: §2.3 · p. 4
  • Table 1 covers four original and two new TAR fine-tuning configurations using the Pile-Bio Forget and Pile-Bio Retain datasets.WMDP accuracies are measured separately for biosecurity, chemical security, and cybersecurity.Source: §2.3 · p. 4, Table 1 · p. 5

3 DEMONSTRATING EVALUATION PITFALLS THROUGH CASE STUDIES

Case studies of RepNoise and TAR show that small evaluation choices can make durability appear stronger than it is, while broader testing reveals important defense side effects (p38).

3.1 RANDOMNESS ALONE CAN CAUSE ATTACKS TO SUCCEED AND DEFENSES TO FAIL

  • Across five random-seed runs, some RepNoise attacks reached harmfulness scores near the undefended model, and all three TAR attacks recovered over 60% WMDP Biosecurity accuracy.For TAR, the recovered accuracy approached the original Llama-3-8B-Instruct level; the original paper’s reported results remained within the observed run ranges.Source: Figure 2 · p. 6, §3.1 · p. 6

3.2 DIFFERENT IMPLEMENTATION DETAILS CAN YIELD DIFFERENT EVALUATION RESULTS

  • Using the HuggingFace SFT trainer made attacks on RepNoise more successful and consistent, while TAR attacks with that trainer were generally more stable.For RepNoise, the attack reached the undefended Llama-2-7B-Chat model’s vulnerability level.Source: §3.2 · pp. 6–7, Figure 3 · p. 7

3.3 SLIGHT MODIFICATIONS TO FINE-TUNING CONFIGURATIONS CAN BREAK DEFENSES

  • TAR attacks became more stable and consistently successful with 100 warmup steps and cosine decay, illustrating how minor configuration changes can expose vulnerabilities.The compared original configuration used 30 warmup steps and a constant learning rate after warmup.Source: §3.3 · p. 7

3.4 EVALUATION RESULTS SOMETIMES CAN BE IMPACTED BY PROMPT TEMPLATES

  • With alternative chat formatting and GPT-4o or human answer extraction, TAR-Bio-v1 accuracy exceeded 45%, above the 25% random-guess level under the official pipeline.The authors interpret this as evidence that the checkpoint retained more bio-weaponization knowledge than its initial evaluation suggested.Source: §3.4 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

3.5 IMPORTANT SIDE EFFECTS OF A DEFENSE COULD BE MISSED FROM EVALUATIONS

  • TAR-Bio-v2 produced higher GPT-2 perplexity on benign biology responses than the original and TAR-Bio-v1 models, indicating sensitivity beyond weaponization questions.The BenignBio dataset contains 100 biology-related questions unrelated to bio-weaponization.Source: §3.5 · p. 9, Table 2 · p. 9
  • Broader evaluations found TAR utility drops across tested benchmarks, including near-zero GSM8K performance, and increased responses to some harmful requests the original model refused.The authors conclude that broader benchmark evaluation revealed more side effects than originally anticipated.Source: §3.5 · p. 9

4 LESSONS FROM OUR CASE STUDIES

The case studies show that safeguards can be bypassed or misread under modest changes to fine-tuning and evaluation. They call for carefully scoped threat models and more reliable, comprehensive evaluation.

Lessons from the case studies

  • Because small setup changes can drastically alter results, evaluations should use reliable protocols and specify which threat models and modifications are covered.The paper calls for careful evaluation and clear disclaimers about the scope of durability claims.Source: §4 · pp. 9–10
  • TAR showed mode collapse on some utility tasks, including GSM8K, BBH, and HumanEval, underscoring that safeguard evaluations must consider utility as well as safety.The table evaluates utility and safety before and after TAR on Llama-3-8B-Instruct in the biosecurity domain.Source: Table 3 · p. 10
  • TAR’s early loss surge was mitigated by 100 warmup steps and cosine decay, after which fine-tuning largely recovered WMDP accuracy.The figure shows loss curves and gradient norms; the modified setup stabilized early training.Source: §4 · pp. 10–11, Figure 6 · p. 10
  • Both RepNoise and TAR showed recovery of restricted capabilities after fine-tuning on unrelated data, challenging claims that the information was removed.The case studies found this even when fine-tuning data did not contain the targeted harmful information.Source: §4 · p. 11
  • RepNoise’s HarmfulQA ability largely returned after fine-tuning on 100 benign points, including identity-shift and pure-benign data.The re-evaluation used AOA identity-shift data and Alpaca Salient benign data across five runs.Source: §4 · p. 11, Figure 7 · p. 11
  • TAR’s WMDP-Bio accuracy largely recovered after fine-tuning only on its retain set, which was not intended to reintroduce forgotten information.The recovery occurred with the original codebase and the authors’ reimplementation.Source: §4 · p. 11
  • Low WMDP accuracy alone does not establish unlearning: changing TAR’s prompt template and answer extraction increased accuracy from a random level.The passage notes that a model may fail to output a multiple-choice answer letter while retaining the information in free-form responses.Source: §4 · p. 12
  • Averaging across 28 TAR adversaries can down-weight the two failed LoRA configurations, so security reporting should emphasize worst-case performance and sampling structure.Twenty of the 28 configurations used Pile-bio; the two LoRA configurations were the most effective attacks described.Source: §4 · p. 12

5 CONCLUSION

The paper concludes that open-weight safeguards remain vulnerable to small fine-tuning changes and that durability evaluations are difficult. It calls for clearly scoped robustness claims and rigorous, standardized evaluation.

Conclusion

  • Current safeguards can be circumvented with slight fine-tuning changes, while minor evaluation factors can substantially alter durability results.The conclusion situates this challenge alongside prior cases of defenses failing under small attack adaptations or implementation changes.Source: §5 · pp. 12–13
  • Defenses should state whether they target benign modifications, a limited modification space, or general adversarial robustness.The paper says narrower threat models may still be useful when their assumptions are explicit.Source: §5 · p. 13
  • Evaluations should use adaptive attacks, repeat experiments across random seeds, assess safety with multiple metrics, and test utility–safety trade-offs.These are among the paper’s proposed evaluation practices.Source: §5 · p. 13
  • The authors hope careful analysis of current and future systems will help the field converge on evaluation principles for assessing safeguard efficacy.Source: §5 · p. 13

ETHICS STATEMENT

The paper aims to evaluate durable safeguards for open-weight LLMs and improve safety evaluations for open-source and potentially other models.

Ethics statement

  • The work aims to improve evaluations of safeguard durability for open-source models and potentially other models, with the goal of supporting safe and secure deployment.Source: ETHICS STATEMENT · p. 13

REPRODUCIBILITY STATEMENT

The authors describe efforts to support reproducibility through evaluation details, author communication, documentation, and code availability.

Reproducibility statement

  • The authors document evaluation details, datasets, prompts, and metrics, consulted the original authors, and made source code available.Source: REPRODUCIBILITY STATEMENT · p. 13

A A CHECKLIST FOR AVOIDING THE EVALUATION PITFALLS WE IDENTIFIED

The checklist treats durability claims as security claims requiring worst-case, well-tested attacks and threat-model-aligned coverage; it also calls for broad side-effect testing and rigorous checks before claiming unlearning.

A A CHECKLIST FOR AVOIDING THE EVALUATION PITFALLS WE IDENTIFIED

  • Test defenses across random attack seeds and use established attack implementations; otherwise, robustness may be overstated.For randomized attacks, report worst-case performance because defenders cannot control attackers’ random seeds.Source: A A CHECKLIST FOR AVOIDING THE… · p. 20
  • Narrow claims to evaluated attack scenarios or explore the full defined threat model, including fine-tuning parameters and common safety and utility benchmarks.The checklist warns that optimizing one safety objective can regress other safety objectives or general utility.Source: A A CHECKLIST FOR AVOIDING THE… · p. 20
  • Test unlearning claims against prompt-format changes and unrelated fine-tuning that could recover the targeted knowledge or capabilities.The checklist presents both tests as ways to refute unlearning claims.Source: A A CHECKLIST FOR AVOIDING THE… · p. 20

B WHY IS SAFEGUARDING OPEN-WEIGHT LLMS EVEN IMPORTANT?

  • Open-weight models create a distinct safeguarding challenge because their weights can be modified, and durable safeguards may also add protection if proprietary weights become exposed.The paper also connects durable safeguards to legal pressure on developers and the continued viability of the open-weight ecosystem.Source: B WHY IS SAFEGUARDING OPEN-WEIGHT LLMS… · pp. 20–21

C ADDITIONAL RELATED WORK

  • Related work spans jailbreaks using query, white-box, fine-tuning, weight-editing, or activation access, and harmful-knowledge unlearning that may hide information rather than remove it.The paper distinguishes harmful-knowledge unlearning from privacy-preserving machine-learning unlearning, which removes an example’s impact on model parameters.Source: C ADDITIONAL RELATED WORK · p. 21

D.1 REPNOISE

  • RepNoise trains representations of HarmfulQA examples toward random noise while retaining a benign utility dataset.Its objective combines retain-set cross-entropy, a forget-set term, and a noise term that pushes representations toward a Gaussian distribution.Source: §D.1 · pp. 21–22

D.2 TAR

  • TAR first maps forget-set representations toward random vectors, then uses sampled fine-tuning adversaries to impede their optimization.The second stage uses a first-order approximation for the adversarial optimization and a retain-set loss to maintain normal functionality.Source: §D.2 · p. 22

E.1.2 IMPLEMENTATION ISSUES

The appendix documents implementation choices and evaluation protocols, then reports that small prompt changes can materially alter TAR models’ measured safety, utility, and generation quality.

E.1.2 IMPLEMENTATION ISSUES

  • The original RepNoise implementation has loss-computation and dataset issues: its loss includes prompt and padding tokens, test examples overlap training and attack sets, and filtering may mix in undesired data.The cited evaluation reports that 75.3% of test examples also appear in the training and attack sets.Source: §E.1.2 · p. 23

E.1.3 DETAILS OF OUR RED-TEAMING EVALUATION USING OUR OWN CODEBASE

  • The re-evaluation corrects loss computation but retains the original BeaverTails attack and test sets, while using corresponding prompt templates for AOA and Alpaca Salient evaluations.The authors describe this as preserving the original setting; they separately describe a disjoint-set ablation.Source: §E.1.3 · pp. 23–24, Figure 8 · p. 24

E.2.1 DETAILS OF OUR RED-TEAMING EVALUATION USING THE OFFICIAL TAR CODEBASE

  • The official TAR evaluation uses the released Biosecurity checkpoint and Pile-Bio forget set, tests four original configurations, and adds two scheduler-and-warmup configurations.The appendix also describes dataset shuffling changes made to preserve randomized example order.Source: §E.2.1 · p. 24

E.2.2 DETAILS OF OUR RED-TEAMING EVALUATION USING OUR OWN CODEBASE

  • The TAR re-evaluation uses the same Pile-Bio forget and retain sets, with constant, constant-with-warmup, or cosine learning-rate schedules across the configurations.The new configurations use cosine scheduling with 100 warmup steps.Source: §E.2.2 · p. 25

E.3.1 SAFETY EVALUATION METRICS

  • Safety evaluation covers harmful-request compliance benchmarks and WMDP knowledge accuracy, with prompting choices and scoring procedures specified for each benchmark.The methods include SORRY-bench, HEx-PHI, BeaverTails, and WMDP.Source: §E.3.1 · pp. 25–26

E.3.2 UTILITY EVALUATION METRICS

  • Utility evaluation spans seven metrics, including MMLU, GSM8K, MATH, BBH, HumanEval, MT-Bench, and TruthfulQA, using task-specific scoring procedures.For Table 3, the authors report the higher utility score from two prompting strategies, with and without a BOS token.Source: §E.3.2 · pp. 26–27

F.1 COMPUATIONAL COST ESTIMATION

  • RepNoise’s AOA and Alpaca Salient fine-tuning configurations require the same order of FLOPs as the original configurations, while TAR’s new configurations add no noticeable compute budget.The paper describes its FLOPs estimates as approximate orders of magnitude because the profiler counts only selected operations.Source: §F.1 · pp. 27–28, Table 5 · p. 27, Table 6 · p. 28

F.2 IMPACT OF THE BOS TOKEN

  • BOS improves TAR-Bio-v1’s WMDP-Bio accuracy, while TAR-Bio-v2 shows prompt-sensitive utility scores and less fluent BenignBio answers with BOS.The WMDP-Bio comparison uses four prompting strategies; utility scores vary with BOS, and the BenignBio fluency study varies BOS and chat-template use.Source: §F.2 · p. 28, Table 7 · p. 28

F.3 EVALUATING THE REPNOISE’S PERFORMANCE ON DISJOINT BEAVERTAILS DATASET

The disjoint-set ablation tests whether RepNoise’s evaluation depends on overlapping BeaverTails examples; the reported results show no significant performance difference for the evaluated model.

F.3 EVALUATING THE REPNOISE’S PERFORMANCE ON DISJOINT BEAVERTAILS DATASET

  • The ablation addresses the original setting’s substantial train, attack, and test overlap.The preceding implementation discussion reports that 75.3% of test examples also appear in the training and attack sets.Source: §F.3 · p. 28
  • The authors construct a disjoint attack set from BeaverTails-330k and a disjoint test set from BeaverTails-Evaluation, excluding examples shared with training or each other.The resulting sets contain 4,986 attack examples and 699 test questions.Source: §F.3 · p. 29
  • The Llama-2-7B-chat-hf results show no significant performance difference before and after RepNoise on the disjoint attack and test sets.The evaluation uses both 1,000-example and 4,986-example attack sets.Source: §F.3 · p. 29, Figure 9 · p. 29

F.4 ADDITIONAL RESULTS OF TAR ON OTHER WMDP TASKS

Additional TAR experiments find that fine-tuning can recover TAR-Bio-v2’s WMDP accuracies across domains under some configurations and seeds; the appendix also introduces qualitative examples of utility-task outputs.

F.4 ADDITIONAL RESULTS OF TAR ON OTHER WMDP TASKS

  • Under specific configurations and random seeds, fine-tuning on Pile-Bio forget or retain data can recover TAR-Bio-v2’s accuracies across all three WMDP domains.The reimplemented HuggingFace Trainer tends to produce more stable and successful attacks than the original codebase.Source: §F.4 · pp. 29–30, Figure 10 · p. 30
  • Figure 10 evaluates WMDP-Chem and WMDP-Cyber accuracy after fine-tuning and notes that cosine decay with learning-rate warmup also improves attack success rates in those domains.Source: Figure 10 · p. 30

G QUALITATIVE EXAMPLES IN TAR

G.1 QUALITATIVE EXAMPLES IN GSM8K DATASET

The GSM8K and BBH examples show that TAR-Bio-v2 sometimes fails to follow few-shot answer formats, which can lead to low scores. The appendices also describe the prompting and answer-extraction procedures used in each evaluation.

G.1 QUALITATIVE EXAMPLES IN GSM8K DATASET

  • In GSM8K, TAR-Bio-v2 sometimes fails to follow few-shot formats, leading to a low score; Table 10 gives a qualitative example.The example uses prompting without a chat template or BOS token.Source: §G.1 · p. 30, Table 10 · p. 31
  • GSM8K evaluation randomly selects five training examples and concatenates them with each test question; Figure 11 shows an example.Source: §G.1 · p. 30, Figure 11 · p. 30

G.2 QUALITATIVE EXAMPLES IN BBH DATASET

  • BBH evaluation uses 3-shot chain-of-thought prompting and extracts the word after “So the answer is” from the model output.The few-shot examples all end with “So the answer is (Yes/No)”.Source: §G.2 · pp. 30–31
  • TAR-Bio-v2 sometimes fails to follow BBH few-shot formats, leading to a low score; Table 11 gives a qualitative example.The example follows TAR on the Pile-Bio dataset.Source: §G.2 · p. 31

G.3 QUALITATIVE EXAMPLES IN BENIGNBIO DATASET

BenignBio examples indicate that TAR-Bio-v2 struggles with benign biology questions and may over-suppress responses; the evaluation measures fluency using GPT-2 perplexity across four prompting strategies.

G.3 QUALITATIVE EXAMPLES IN BENIGNBIO DATASET

  • TAR-Bio-v2 struggles to answer benign biology questions, suggesting TAR may inadvertently over-suppress responses to benign content.Source: §G.3 · p. 31
  • BenignBio fluency is evaluated with GPT-2 perplexity across four prompting strategies.Source: §G.3 · p. 31
  • Table 12 gives a Benign-Bio example in which TAR-Bio-v2 sometimes fails to follow the few-shot format.The example uses prompting without a chat template or BOS token.Source: Table 12 · p. 32
Loading 2412.07097v1…