PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic

Rishabh Bhardwaj, Do Duc Anh, Soujanya Poria

arXiv:2402.11746v1cs.CLcs.AI

TL;DR

Fine-tuning can compromise language-model safety, including when performed on benign data. RESTA adds a safety vector to fine-tuned models. Across benchmark datasets and fine-tuning domains, reported unsafety scores fall substantially in both PEFT and Full-FT.

  • Problem

    Fine-tuning can improve performance on a specific domain while compromising model safety, even when the data are benign.

    Source: §1 · p. 1

  • Method

    RESTA realigns a fine-tuned model by adding a safety vector to its parameters, with DARE dropping and rescaling fine-tuning delta parameters to further improve the effect.

    Source: §1 · p. 1

  • Results

    Unsafety scores fell from 18.59% to 5.14% in PEFT and from 9.16% to 1.55% in Full-FT, averaged across benchmark datasets and fine-tuning domains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that the safety vector generalizes beyond its training categories and across languages, with minimal performance degradation.

    Source: §7 · p. 9

    The study does not extensively evaluate larger models, hyperparameters, or safety-vector transfer across language models.

    Source: §8 · p. 9

Abstract

from arXiv · show

Aligned language models face a significant limitation as their fine-tuning often results in compromised safety. To tackle this, we propose a simple method RESTA that performs LLM safety realignment. RESTA stands for REstoring Safety through Task Arithmetic. At its core, it involves a simple arithmetic addition of a safety vector to the weights of the compromised model. We demonstrate the effectiveness of RESTA in both parameter-efficient and full fine-tuning, covering a wide range of downstream tasks, including instruction following in Chinese, English, and Hindi, as well as problem-solving capabilities in Code and Math. We also showcase the generalizability of RESTA on three existing safety evaluation benchmarks and a multilingual benchmark dataset proposed as a part of this work, consisting of 550 harmful questions covering 11 categories, each with 5 sub-categories of harm. Overall, RESTA decreases the harmfulness of the compromised model from 18.6% to 5.1% and from 9.2% to 1.5% in parameter-efficient and full fine-tuning, respectively, while maintaining most of the model's performance on the task. We release the source codes at: https://github.com/declare-lab/resta.

1 Introduction

Fine-tuning can compromise language-model safety, even when the data are benign. RESTA adds a safety vector to fine-tuned models, and the evaluations report lower unsafety across benchmarks and tasks.

  • Fine-tuning can compromise safety even with benign, commonly used datasets, despite improving performance on a specific domain.Source: §1 · p. 1
  • RESTA restores safety by adding a safety vector to model parameters; DARE drops and rescales redundant fine-tuning delta parameters to further improve realignment.Source: §1 · p. 1
  • The evaluation spans Chinese, Hindi, and English instruction-following tasks, Code and Math, and CATQA, a multilingual dataset of 550 harmful questions.CATQA covers 11 categories with five sub-categories each and includes Chinese and Vietnamese.Source: §1 · pp. 1–2
  • On CATQA, fine-tuned Llama-2’s unsafety score fell from 33.57% to 12.17% in PEFT and from 22.16% to 4.34% in Full-FT, averaged across domains.Source: §1 · p. 2
  • Unsafety scores fell from 18.59% to 5.14% in PEFT and from 9.16% to 1.55% in Full-FT, averaged across benchmark datasets and fine-tuning domains.Source: §1 · p. 2

2 RESTA: REestoring Safety through Task Arithmetic

RESTA restores safety to fine-tuned models by adding a safety vector to their parameters. The method also describes DARE for dropping and rescaling task-specific parameter changes and estimates the safety vector through unalignment.

SFT Base RESTA RESTAd

  • The method models fine-tuning as an ideal task shift plus an undesired shift in the safety direction, with the desired model retaining only the ideal shift.The undesired shift is represented as −λ times the safety vector, and RESTA aims to compensate for it.Source: SFT Base RESTA RESTAd · p. 3

2 RESTA: REestoring Safety through Task Arithmetic

  • RESTA adds a safety vector to the safety-compromised fine-tuned model; its scale is intended to overcome the harm introduced during fine-tuning.The method also studies applying DARE to task-specific delta parameters before RESTA.Source: §2 · p. 2, SFT Base RESTA RESTAd · p. 3

SFT Base RESTA RESTAd

  • DARE drops a proportion of task-specific delta parameters to zero and rescales the remainder by 1/(1 − p), potentially leaving more room for the safety vector to act.The text cautions that upscaling can mean dropping deltas does not itself reduce their safety impact.Source: SFT Base RESTA RESTAd · p. 3

2.1 Safety Vector

  • Because alignment combines helpfulness and safety, the method estimates an unaligned counterpart by fine-tuning the aligned model on harmful questions with helpful responses, then uses the parameter shift as a safety-vector estimate.The passage says unalignment impacts safety while preserving helpfulness and generic utility.Source: §2.1 · p. 4

3 CATQA: A Categorical Harmful QA Dataset

CATQA is a harmful-question benchmark organized around prohibited-use categories. It contains 550 questions and is extended from English to Chinese and Vietnamese with human correction of translation errors.

3 CATQA: A Categorical Harmful QA Dataset

  • CATQA organizes prohibited use cases into 11 harm categories, each divided into five sub-categories with 10 harmful questions apiece.The categories are drawn from OpenAI and Meta usage policies, and question collection was assisted by an unaligned LLM.Source: §3 · p. 4
  • The authors do not disclose the name of the helpful but safety-unaligned LLM used to assist question generation, citing misuse prevention.Source: §3 · p. 4
  • 550 harmful questions make up CATQA, which is extended from English into Chinese and Vietnamese using LLM translation followed by correction from a language-proficient human annotator.The authors state that they plan to release the dataset upon acceptance.Source: §3 · p. 5

4 Experimental Setup

Experiments use Llama-2-7B-Chat fine-tuned across language and problem-solving tasks, with safety evaluated on CATQA and three existing benchmarks. The setup defines safety vectors from harmful-question fine-tuning and uses GPT-4 judgments to score unsafe responses.

4 Experimental Setup

  • Experiments use Llama-2-7B-Chat with PEFT via LoRA or Full-FT, using p = 0.3 and γ = 0.5 across tasks.The setup identifies RESTA and RESTAd as adding the safety vector to the SFT model without and with DARE, respectively.Source: §4 · p. 5
  • Five SFT datasets cover Chinese, Hindi, and English Alpaca instruction following, plus coding and math; non-English Alpaca mixes in 50K English examples.English and Chinese Alpaca are about 50K examples each for the non-English dataset construction described.Source: §4 · p. 5
  • Safety vectors are computed by fine-tuning an aligned model on 100 CATQA questions judged unsafe, with separate vectors for PEFT and Full-FT kept fixed across domains and benchmarks.The 100 questions are stratified-sampled from CATQA after English Alpaca fine-tuning; GPT-4 evaluates whether responses are unsafe.Source: §4 · p. 5
  • Safety evaluation uses CATQA and three existing sets: HARMFULQA, ADVERSARIALQA, and DANGEROUSQA.HARMFULQA evaluation samples 196 prompts across 98 subtopics; ADVERSARIALQA evaluation samples 200 of 500 instructions; DANGEROUSQA contains 200 toxic questions.Source: §4 · p. 5
  • GPT-4 judges responses for harmfulness, and cases it declines to annotate because of content filters are omitted from the unsafety score.The score is the fraction of unsafe responses among the evaluated responses; a lower score denotes a safer model.Source: §4 · pp. 5–6

5 Results and Discussions

RESTA and RESTAd restore safety compromised by fine-tuning across PEFT and Full-FT, while retaining most task performance. Evaluations also report safety improvements on out-of-domain and multilingual benchmarks.

5 Results and Discussions

  • RESTA, which adds a safety vector after fine-tuning, drops unsafety from 33.57 to 13.22; RESTAd, applying safety vectors to DARE-dropped SFT delta weights, further reduces it to 12.17%.DARE reduces unsafety for Hindi and English but has little impact on other datasets; RESTA with DARE is more effective in PEFT than Full-FT.Source: §5 · pp. 6–7
  • On HARMFULQA's 98 categories disjoint from CATQA, safety vectors reduce unsafety from 18.21% to 5.44%, toward Llama-2's 1.53%.The paper also reports similar effectiveness on harmful-instruction and toxic-question datasets, while noting some question overlap with CATQA.Source: §5 · p. 7
  • In Full-FT, RESTA reduces the average gap to ideal safe Llama-2 to less than 2.5%, from about 20% after fine-tuning.On out-of-domain benchmarks, unsafety remains below 2.5%; in DANGEROUSQA, RESTA reaches a safety score of 0.65 versus 1.51 for Llama-2.Source: §5 · p. 7
  • RESTA reduces SFT performance by about 2.41% on average in PEFT and 0.47% in Full-FT, while performance stays close to the original until γ = 1.Over that range, the model's safety score rises significantly, reaching Llama-2's safety values.Source: §5 · p. 7

5.1 Generalizability across languages

  • Across CATQA versions, the Llama-2 safe model’s unsafety scores rise from 2.18% to 10.20% and 15.32%; on the multilingual safety benchmark, CATQA lowers unsafety by more than 22 points averaged across fine-tuning domains and types.Source: §5.1 · p. 8

6 Related Work

Related work describes how fine-tuning changes model behavior and how safety can be compromised, alongside research on attacks, defenses, and task-vector arithmetic. RESTA draws on task-vector work to realign fine-tuned models.

6 Related Work

  • Supervised fine-tuning adapts pretrained models using a relatively small dataset, through either full fine-tuning or parameter-efficient fine-tuning.Prior work identifies redundancy in fine-tuning delta parameters and proposes DARE, which removes over 90% while achieving competitive performance [Yu et al., 2023].Source: §6 · p. 8
  • Prior work reports safety vulnerabilities and prompting attacks, while defense work includes RAIN's alignment without fine-tuning and proposals for self-destructing models.Other studies expose safety compromise after fine-tuning, including with benign samples; several cited works intentionally break guardrails using small harmful datasets.Source: §6 · p. 8
  • Prior studies find that interpolating weights of networks sharing part of their optimization trajectory can preserve high accuracy, while moving pretrained weights toward a fine-tuned version steadily increases downstream task accuracy [Wortsman et al., 2022b].Source: §6 · p. 8
  • RESTA's safety vectors are motivated by work studying behavioral changes from moving models along task vectors derived from pre- and post-fine-tuning weights [Ilharco et al., 2022a].The paper relates task vectors to differences between model parameters before and after fine-tuning.Source: §6 · p. 9

7 Conclusion

The conclusion presents RESTA as a safety-vector addition for fine-tuned models and reports reduced harmfulness with minimal performance degradation. It also reports generalization beyond trained categories and across languages.

7 Conclusion

  • RESTA adds a safety vector to fine-tuned models; evaluations report harmfulness falling from 17.86% to 4.57% with minimal performance degradation.The paper also introduces multilingual CATQA with 550 questions across 11 categories and reports safety-vector generalization beyond trained categories and across languages.Source: §7 · p. 9

8 Limitations

The authors identify limits in model scale, hyperparameter coverage, and analysis of safety-vector transferability. They leave these evaluations for future work.

8 Limitations

  • The study does not evaluate larger models such as Llama-2-70B, which the authors believe could reveal further insights.Source: §8 · p. 9
  • Budget constraints prevented extensive evaluation of each model across hyperparameters such as γ and p.Source: §8 · p. 9
  • The impact of safety vectors and their transferability across language models remain unanalyzed and are left for future work.Source: §8 · p. 9

9 Ethics Statement

The ethics statement identifies potential exploitation of LLM alignment vulnerabilities and multilingual harm, while describing planned warnings and automated response flagging to protect human participants.

  • The study identifies an alignment vulnerability that could enable widespread exploitation by malicious end-users.Source: §9 · p. 9
  • The developed dataset could magnify LLM-caused harm across various languages.Source: §9 · p. 9
  • The authors plan an opening warning and use GPT-4 to flag responses, preventing offensive text from reaching human participants.They state that analyzing LLM harmfulness and mitigation strategies could advance safety.Source: §9 · p. 9

A Example Appendix

The section is identified as an appendix.

Loading 2402.11746v1…