PaperlayerView exact revision on arXiv

Source-linked AI summary

Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models

ShengYun Peng, Pin-Yu Chen, Matthew Hull, Duen Horng Chau

arXiv:2405.17374v3cs.LG

TL;DR

Finetuning with a few adversarial examples can compromise aligned LLMs, raising safety concerns for model customization. The paper probes safety landscapes to measure finetuning risk and finds local safety basins, with system-prompt protection also extending to perturbed variants within those basins.

  • Problem

    Aligned LLMs can lose safety after finetuning with only a few adversarial examples, raising practical concerns as customization is desirable for specific use cases.

    Source: §1 · p. 1

  • Method

    The paper probes safety landscapes using random weight-perturbation directions or linear interpolation between models, and defines VISAGE from sampled safety margins.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    Popular open-source LLMs exhibit a safety basin where local weight perturbations maintain aligned safety, followed by a sharp safety drop outside the basin.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The landscape analysis indicates that safety remains intact when finetuning stays within the basin, and that system-prompt protection transfers to perturbed variants within it.

    Source: §1 · pp. 2–3

    VISAGE samples α and β uniformly and fixes a = b = 0.5 based on the finding that models break after perturbations exceed half their norm.

    Source: §3.4 · p. 6

Abstract

from arXiv · show

Safety alignment is crucial to ensure that large language models (LLMs) behave in ways that align with human preferences and prevent harmful actions during inference. However, recent studies show that the alignment can be easily compromised through finetuning with only a few adversarially designed training examples. We aim to measure the risks in finetuning LLMs through navigating the LLM safety landscape. We discover a new phenomenon observed universally in the model parameter space of popular open-source LLMs, termed as "safety basin": random perturbations to model weights maintain the safety level of the original aligned model within its local neighborhood. However, outside this local region, safety is fully compromised, exhibiting a sharp, step-like drop. This safety basin contrasts sharply with the LLM capability landscape, where model performance peaks at the origin and gradually declines as random perturbation increases. Our discovery inspires us to propose the new VISAGE safety metric that measures the safety in LLM finetuning by probing its safety landscape. Visualizing the safety landscape of the aligned model enables us to understand how finetuning compromises safety by dragging the model away from the safety basin. The LLM safety landscape also highlights the system prompt's critical role in protecting a model, and that such protection transfers to its perturbed variants within the safety basin. These observations from our safety landscape research provide new insights for future work on LLM safety community. Our code is publicly available at https://github.com/ShengYun-Peng/llm-landscape.

1 Introduction

The paper investigates why finetuning can compromise LLM safety and how to measure that risk. It uses safety-landscape analysis to identify a safety basin, define VISAGE, and study system prompts and jailbreaks.

1 Introduction

  • The authors probe safety landscapes with random normalized directions for a single model and linear interpolation between fine-tuned models.They propose VISAGE, a task-agnostic metric of finetuning risk based on the safety of a model’s local parameter-space region.Source: §1 · p. 2
  • Aligned open-source LLMs maintain safety under local weight perturbations, but safety drops sharply outside this safety basin.The paper reports this pattern across LLaMA2, LLaMA3, Vicuna, and Mistral, contrasting it with capability landscapes where performance gradually declines as perturbations increase.Source: §1 · p. 2, Figure 1 · p. 2
  • Landscape analysis frames safety loss during finetuning as the model being dragged away from the safety basin.The paper reports different vulnerability rates across LLMs and says safety remains intact when finetuning stays within the basin.Source: §1 · p. 2
  • Removing default system prompts or using roleplaying jeopardizes safety, while LLaMA2’s original prompt and model-specific tuned safety prompts improve it across models.The paper also reports that jailbreak prompts are sensitive to weight perturbations, while attackers can target both aligned and perturbed models.Source: §1 · pp. 2–3

2 Background and Related Works

The background describes safety alignment methods alongside evidence that finetuning and jailbreaks can bypass safeguards. It reviews attacks and defenses involving finetuning, prompts, and adversarial queries.

2 Background and Related Works

  • Instruction tuning, RLHF, DPO, rejection sampling, and self-alignment are safety-alignment approaches, but they are not designed for risks from later finetuning and jailbreaks.Safety alignment aims to help models follow instructions and be helpful, truthful, and harmless by enabling refusals of harmful requests.Source: §2 · p. 3
  • Prior work reports that harmful or even benign finetuning can compromise alignment, motivating defenses during alignment and user finetuning.Reviewed defenses include Vaccine, Lisa, Antidote, Booster, targeted Vaccine, and Safe LoRA.Source: §2 · p. 3
  • Safety prompts aim to protect against harmful queries without changing model weights, and can be prepended to user input or used as system prompts.Prior work includes prompt tuning to optimize a vanilla system prompt into a safety prompt.Source: §2 · p. 3
  • Jailbreaks use either human-crafted deception and social engineering or optimization-based token attacks, the latter requiring substantial computation and often being hard to interpret.Human-prompt approaches also require creativity, manual dataset curation, and significant human effort.Source: §2 · pp. 3–4

3 From LLM Safety Landscape to VISAGE Safety Metric

The paper visualizes safety across model-weight perturbations and interpolations, then uses the resulting landscape to define VISAGE as a measure of safety in a model’s local parameter region.

3.1 1D Safety Landscape

  • For a 1D landscape, the method perturbs initial weights along a direction and evaluates each resulting model with a single-model safety metric.Random directions are Gaussian-sampled and layer-normalized; interpolation uses the line between two sets of model weights.Source: §3.1 · p. 4

3.2 2D Safety Landscape

  • The 2D landscape evaluates safety across two directions, using random directions or orthogonalized interpolations from three model-weight sets.Gram–Schmidt supplies an orthogonal basis when interpolation directions are not orthogonal.Source: §3.2 · p. 4

3.3 Safety Landscape of Open-source LLMs

  • In the LLaMA2 base-to-chat interpolation, the chat model is safer, and applying its chat template also raises the base model’s measured safety.Both text-completion and LLaMA2 chat-template formats are evaluated to account for format consistency.Source: Figure 2 · p. 5, §3.3 · p. 5
  • Across four open-source LLMs, safety remains near the aligned model’s level locally but drops sharply outside that region; basin width and depth vary, and larger models show wider basins.The basin finding generalizes across safety metrics and datasets.Source: §3.3 · pp. 5–6

3.4 VISAGE Safety Metric

  • VISAGE averages safety margins over sampled models in a random landscape, with ASR evaluation using a maximum score of 100 and perturbation bounds of 0.5.The paper uses three random directions for stability and 1D VISAGE for faster evaluation.Source: §3.4 · p. 6
  • LLaMA3-8B-instruct ranks highest by VISAGE, followed by LLaMA2-7B-chat, Mistral-7B-instruct-v0.2, and Vicuna-7B-v1.5, despite close AdvBench-520 ASRs.The respective ASRs are 0.38, 0.19, 1.15, and 2.5; VISAGE reflects safety in a model’s local region.Source: §3.4 · p. 6

4 Why can simple finetuning easily break LLM’s safety alignment?

This section examines how harmful-data fine-tuning compromises safety and how the safety landscape helps measure and explain that risk. It also shows that safety can remain intact when fine-tuning stays within the safety basin.

4.1 Finetuning settings

  • Fine-tuning uses Qi et al.’s harmful samples, keeps the system prompt consistent with the aligned model, and runs full-parameter training for five epochs.Evaluation covers 80 and 520 AdvBench prompts.Source: §4.1 · pp. 6–7

4.2 Finetuning on few-shot harmful data breaks LLM’s safety alignment

  • Fine-tuning on 10, 50, and 100 harmful examples raises ASR from near 0%; VISAGE measures risk beforehand, and LLaMA2 remains safer than Vicuna under the same data.The evaluation uses both 80 prompts and all 520 AdvBench prompts.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 Finetuning with harmful data is dragging the model away from the safety basin but at different rates

  • Projecting saved checkpoints along the initial-to-final weight direction shows harmful-data fine-tuning moving LLaMA2 away from the aligned origin and outside the safety basin.The projection captures the optimization trajectory without computing PCA over the full checkpoint feature matrix.Source: §4.3 · p. 7

4.4 Finetuning with harmful and safe data helps the model stay within the safety basin

  • Mixing 100 harmful examples with 100 safe examples lowers ASR and keeps LLaMA2 within the safety basin, unlike harmful-only fine-tuning.The mixed-data run continues for ten epochs; the harmful-only run becomes unsafe while the mixture remains safe.Source: §4.4 · pp. 7–8

5 System prompt

The section evaluates how attacker- and defender-chosen system prompts affect safety across open-source LLMs. Removing the default prompt generally harms safety, while defender prompts can improve it.

5 System prompt

  • LLaMA2’s VISAGE score falls 4.64 percentage points without its default prompt, while roleplay slightly increases it; LLaMA3 gains 9.3pp with the LLaMA2 prompt.LLaMA3 has no default system prompt and loses 2.7pp under roleplay.Source: §5 · p. 8
  • Removing system prompts reduces safety for tested Mistral and Vicuna models; roleplay lowers Vicuna-7B-v1.3 safety but raises Vicuna-7B-v1.5 safety, while the LLaMA2 system prompt significantly improves Vicuna safety.Removing the system prompt decreases Mistral-7B-instruct-v0.1’s VISAGE score by 53.33pp; this model is more sensitive to the system prompt than Mistral-7B-instruct-v0.2.Source: §5 · p. 9
  • Removing the default system prompt or using roleplay jeopardizes safety alignment, with removal more potent; LLaMA2’s prompt and tuned safety prompts enhance safety across models.The tuned prompts also enhance safety for perturbed variants within the safety basin.Source: §5 · pp. 8–9

6 Jailbreak attacks

Jailbreaking queries are sensitive to weight perturbations, and some perturbed models are safer than the aligned model. Replacing Vicuna’s default system prompt with LLaMA2’s improves safety in Vicuna’s local region.

6 Jailbreak attacks

  • 4 of 6 JailbreakBench prompts attack LLaMA2-7B-chat, yielding a 66.67% ASR for the aligned model.Some perturbed variants are significantly safer under these queries; the same sensitivity is observed for Vicuna-7B-v1.5.Source: §6 · p. 9, Figure 4 · p. 10
  • Replacing Vicuna’s default system prompt with LLaMA2’s improves overall safety in Vicuna’s local region.Source: §6 · p. 9
  • The authors caution that attackers could create stronger attacks targeting both the aligned model and multiple perturbed variants.Source: §6 · p. 9
  • The findings provide insights for future work on LLM attacks and defenses.Source: §6 · p. 10

7 Safety vs. Capability Landscape

The capability landscape differs markedly from the safety landscape, and the two do not show the same trend.

7 Safety vs. Capability Landscape

  • The capability evaluation covers three MMLU datasets spanning math, history, and policy.Source: §7 · p. 10
  • The capability landscape differs drastically from the safety landscape, and the two do not exhibit the same trend.Source: §7 · p. 10
  • The authors say this difference further confirms that the basin shape is unique to LLM safety.They provide a detailed analysis in Appendix C.Source: §7 · p. 10

8 Conclusion

The conclusion presents the safety basin and VISAGE as tools for measuring safety in LLM finetuning. It connects safety visualization to finetuning changes and to the transfer of system-prompt protection.

8 Conclusion

  • The authors identify a safety basin in popular open-source LLMs and propose VISAGE to measure finetuning safety by probing the safety landscape.Source: §8 · p. 10
  • Visualizing an aligned model’s safety landscape helps show how finetuning compromises safety by moving the model away from the basin.Source: §8 · p. 10
  • The safety landscape highlights the system prompt’s role in protection and the transfer of that protection to perturbed variants within the basin.Source: §8 · p. 10
  • The authors describe these observations as offering new insights for future work on LLM safety.Source: §8 · p. 10

Limitations and Future Work

The authors frame their exploration of popular open-source LLM safety landscapes as an initial step. They identify capability–safety balancing and additional landscape metrics as future research directions; an appendix lists experimental system prompts.

Limitations and Future Work

A System Prompts

  • The system-prompt appendix lists experimental prompts, notes that Mistral prompts come from Zheng et al. [51], and highlights differences from defaults in red.Source: A System Prompts · p. 15
  • The listed prompt variants include instructions to avoid harmful content and a role prompt directing the model to obey the user.Source: A System Prompts · p. 15

B Does the finding of the safety basin generalize to other evaluation metrics and safety datasets?

The safety basin persists across an additional harmfulness metric and safety dataset: both LLaMAGuard 2 and safety keyword detection show a similar basin shape on POSE.

C Is the capability landscape the same as the safety landscape?

The capability landscape differs from the safety landscape: capability scores vary by dataset and change more gradually, while safety retains a basin shape with a rapid drop at its brim.

D Does the model still generate fluent output when ASR is high?

The perturbed model can remain fluent even when its attack success rate is high, while a larger model exhibits a wider safety basin.

D Does the model still generate fluent output when ASR is high?

E The effect of model size on safety basin

D Does the model still generate fluent output when ASR is high?

  • Table 3 presents LLaMA2-7B-chat outputs sampled equally from the 1D random direction, supporting the sampling description but not any specific α values.Source: Table 3 · p. 19
Loading 2405.17374v3…