PaperlayerView exact revision on arXiv

Source-linked AI summary

Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey

Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Ling Liu

arXiv:2409.18169v6cs.CRcs.AIcs.LG

TL;DR

Harmful data uploaded for fine-tuning-as-a-service can compromise a model’s existing safety alignment. The paper formulates the threat and surveys attack and defense research; its statistical findings report safety degradation as harmful-data ratios rise, while fine-tuning accuracy is not significantly affected.

  • Problem

    Fine-tuning on user data containing harmful examples can compromise a model’s prior safety alignment.

    Source: §1 · p. 2

  • Method

    The paper formulates the harmful fine-tuning threat and provides a systematic overview of attacks, defenses, and interpretability research.

    Source: §1 · p. 2, §6 · p. 24

  • Results

    Safety degradation becomes more significant as the harmful-data ratio rises, while fine-tuning accuracy is not significantly influenced, even at a high ratio.

    Source: §2.3 · p. 6

  • Takeaways & Limitations

    The survey organizes attack and defense research and presents evaluation methodology and future directions for the subject.

    Source: §1 · p. 2, §6 · p. 24

    The authors refrain from drawing an explicit conclusion about which defense type is most effective in their qualitative analysis.

    Source: §3.4 · p. 18

Abstract

from arXiv · show

Recent research demonstrates that the nascent fine-tuning-as-a-service business model exposes serious safety concerns: fine-tuning with a few harmful data uploaded from the users can compromise the safety alignment of the model. The attack, known as harmful fine-tuning attack, has generated broad research interests in both academia and industry. In this paper, we first systematically formulate the threat model and basic assumptions of harmful fine-tuning. Then, we provide a comprehensive review of harmful fine-tuning from three fundamental perspectives: attack setting, defense design, and evaluation methodology. First, we present the threat model of the problem and introduce the harmful fine-tuning attack and its variants. Next, we systematically survey representative attacks, defense methods, and mechanical analysis of adverse effects in the existing literature. Finally, we introduce the evaluation methodology and outline future research directions, which can serve as guidelines and crucial perspectives for the future development of the subject. We also maintain a curated list of relevant papers, which are made accessible at https://github.com/git-disl/awesome_LLM-harmful-fine-tuning-papers

1 Introduction

Fine-tuning-as-a-service lets users submit data for provider-run model tuning, but harmful uploads can compromise a model’s safety alignment. The paper formulates the threat and surveys attacks, defenses, evaluation methods, and future directions.

1 Introduction

  • Partially harmful user data can make a safety-aligned model forget to refuse harmful questions after fine-tuning.The service provider then deploys the alignment-broken model to serve users.Source: §1 · p. 2
  • The authors intend the survey to help researchers and practitioners understand safety risks and mitigation approaches for aligned LLMs.They also offer questions of interest as a reference for evaluation design.Source: §1 · pp. 2–3
  • The paper formulates the threat model, attack motivation, and assumptions to address a gap in prior harmful fine-tuning research.The authors note that earlier works studied the attack without explicitly constructing its threat model.Source: §1 · p. 3
  • The survey classifies attacks as vanilla or advanced by scenario and defenses by when they intervene.Defense stages include alignment, fine-tuning, and post-fine-tuning.Source: §1 · p. 3
  • The paper reviews datasets, evaluation metrics, models, and qualitative analysis for assessing harmful fine-tuning attacks and defenses.Source: §1 · p. 3
  • The survey outlines research challenges and future directions for attacks, defenses, and interpretability.Source: §1 · p. 3

Standard pipeline of fine-tuning-as-a-service and its

The section examines how fine-tuning-as-a-service can weaken safety alignment when harmful content appears in user data, with greater harmful-data exposure linked to increased harmfulness even as task accuracy remains largely intact. It also describes forgetting alignment knowledge and learning harmful data as explanations for this degradation and identifies two broad defense goals.

Standard pipeline of fine-tuning-as-a-service and its

  • Providers first align pretrained models, typically with SFT or RLHF, then fine-tune them on user data, where harmful samples can subvert the earlier alignment.Post-fine-tuning alignment is usually avoided because repeating it is computationally expensive and may impair user-task performance.Source: Standard pipeline of… · p. 3
  • Even benign user-data fine-tuning can degrade safety alignment, posing a risk to service providers.The section notes that post-fine-tuning safety alignment may diminish the customization value by reducing user-task performance.Source: Standard pipeline of… · p. 3

2.2 Threat model of harmful fine-tuning

  • Users operate data uploaded to the service provider but cannot control the fine-tuning process or hyperparameter selection; intentional attackers upload harmful data to compromise safety alignment, while unintentional attackers may upload harmful content they cannot inspect.Source: §2.2 · p. 4
  • Providers control fine-tuning algorithms, hyperparameters, and dataset filtration, and must preserve safety while learning the user’s benign downstream task.The defense goal has two conditions: resistance to harmfulness after sufficient user-data fine-tuning and stability of downstream-task performance with or without defense.Source: §2.2 · p. 4

2.3 Statistical findings

  • As the harmful-data ratio rises, harmful scores increase for both SFT and non-aligned models, and SFT provides only slight resistance that breaks at sufficiently high ratios.Figure 4 evaluates fine-tuning on SST2 mixed with harmful data; the reported comparison does not provide specific score values in these passages.Source: Figure 4 · p. 5, §2.3 · pp. 5–6
  • Safety degradation grows with the harmful-data ratio, while fine-tuning accuracy remains largely unaffected, making harmful fine-tuning stealthy on the user task.The model can retain nearly the same user-task performance even at a high harmful-data ratio.Source: §2.3 · p. 6
  • Alignment loss rises with harmful-data ratio for safety-aligned SFT models, whereas it starts high and remains stable for models that were not safety-aligned.The SFT pattern indicates forgetting of alignment knowledge.Source: §2.3 · p. 6, Figure 5 · p. 6
  • Hidden-embedding drift increases with harmful-data ratio in both aligned and non-aligned models, and its shared trend with harmful score suggests a plausible link to safety degradation.Drift is measured as Euclidean distance between hidden embeddings on the alignment dataset before and after fine-tuning.Source: §2.3 · p. 6
  • Vaccine [100] shows slighter embedding drift than SFT at high harmful ratios, helping preserve alignment knowledge despite drift toward a specific direction.Figure 6 visualizes embedding drift under different harmful ratios.Source: §2.3 · p. 6, Figure 6 · p. 7
  • Fine-tuning on pure harmful data increases harmful score and reduces harmful training loss, with the loss reduction generalizing to unseen harmful data; pure SST2 fine-tuning has little such effect.The harmful score rises with optimization steps on harmful data but is not much affected by fine-tuning on pure SST2.Source: §2.3 · p. 7, Figure 7 · p. 7
  • Harmful fine-tuning both increases alignment loss, indicating alignment forgetting, and decreases harmful loss, indicating harmful-knowledge revitalization; defenses target preservation or removal/attenuation of harmful knowledge.Preservation methods include Vaccine [100], Lisa [97], and T-Vaccine [147]; harmful-knowledge methods include RepNoise [196], Antidote, TAR [217], and Booster [98].Source: §2.3 · p. 7
  • Two defense goals are to keep harmful data untrainable or to allow harmful learning while preserving refusal on unseen harmful questions; the authors report no successful attempt at the latter.The paper notes work on the untrainability goal, including MLAC [80], Vaccine [100], and TAR [216].Source: §2.3 · p. 8

3 Existing attacks, defenses, and interpretability study

This section surveys harmful fine-tuning attacks, defenses, and interpretability studies, including attack data strategies and defenses applied during alignment, user fine-tuning, or afterward. It relates defenses to attack types and reviews analyses of their effects, while noting that existing evaluations do not support firm comparative conclusions.

3.1 Survey methodology

  • The survey frames three research questions—designing attacks, defenses, and interpretability—and collects eligible Internet-accessible work through forward citation snowballing supplemented by backward checks.Eligibility is limited to work on harmful fine-tuning under the threat model defined in Section 2.1.Source: §3.1 · p. 8
  • After mainstream providers such as OpenAI and Google adopted guardrails to filter user datasets and exclude harmful content, research pursued stronger pure-benign-data fine-tuning attacks [69] [79] and stealthier harmful attacks evading filtration, including steganography [74] and guardrail jailbreaks [99].Source: §3.1 · p. 10
  • The survey identifies specialized benchmark development as a recent trend, with early attempts by Rosati et al. [195] and Safetunebed [83].Its collection process used seed papers and Google Scholar forward citation searches, supplemented by checking references for missed work.Source: §3.1 · pp. 8, 10

3.2 Harmful fine-tuning attacks

  • Fine-tuning on harmful, benign, or non-explicit harmful data can subvert safety alignment; jailbreak data, larger models, and reverse-preference RL can strengthen the effect.The survey reports OpenAI API experiments and subsequent findings on these conditions.Source: §3.2 · p. 11
  • Covert malicious fine-tuning [74] teaches a model benign encoded QA before training on encoded harmful data, which evades ordinary guardrails; later encoding methods [44] seek greater stealth.At test time, the model responds to encoded harmful prompts with encoded harmful answers.Source: §3.2 · p. 11
  • Virus [99] optimizes each harmful-data token to jailbreak guardrail moderation while retaining comparable attack performance against the target model.The survey also describes Prefix-Suffix Wrappers, underscore replacement, and harmful backdoors [137] as ways to transform data to bypass moderation.Source: §3.2 · p. 11
  • Prefix-Suffix Wrappers, underscore replacement, and harmful backdoors [137] reduce harmfulness cues, replace high-risk words, or avoid inference-time auditing to bypass moderation.The survey notes overlap with Virus [99] in transforming harmful data to bypass moderation.Source: §3.2 · p. 11
  • A small amount of harmful data with jailbreak suffixes that bypass moderation can downgrade model safety alignment.The passages also state that jailbreak suffixes can strengthen harmful-attack performance.Source: §3.2 · p. 11
  • Benign-data fine-tuning [189] can break model safety alignment and elicit harmful answers to non-coding free-form questions.Selected benign outlier data had intrinsically better attack performance, outperforming a benign sampling scheme [79].Source: §3.2 · pp. 11–12
  • Benign-data attacks can also be strengthened by changing response structure [121], using two-stage refusal-then-compliance tuning [248], or adding optimized harmful-feature suffixes [281].The methods aim to link benign questions to harmful questions and then to compliance answers.Source: §3.2 · p. 12
  • Attack studies extend to multilingual LLMs, where one-language fine-tuning can compromise safety across languages [186], as well as Telecom [238], bio-foundation [238], and audio LLMs [194].The audio study reports mitigation using filtration based on embedding-space distance [194].Source: §3.2 · pp. 12–13
  • Other studied settings include model editing [24], federated learning [134] [258], and self-evolving RLVR training [16] [156] [203], which can downgrade safety alignment..Source: §3.2 · p. 13

3.3 Defenses against Harmful Fine-tuning

  • Defenses are classified by timing; pre-training measures include filtering harmful corpus content to reduce harmful fine-tuning’s effect [170] and the GO optimizer.GO is described as enlarging the capability basin.Source: §3.3 · p. 13
  • Alignment-stage defenses include Vaccine’s perturbation-robust alignment objective, layer-wise T-Vaccine [147], and RepNoise [196], which uses harmful data to disrupt harmful representations.The passages note that uniform perturbation across layers can be suboptimal, and that alignment-only data may be insufficient; RepNoise minimizes alignment loss while maximizing harmful-data loss and perturbing harmful-data embeddings.Source: §3.3 · p. 14
  • TAR [217] meta-learns to maximize harmful loss after simulated harmful fine-tuning, while Booster [98] is introduced to avoid TAR’s proxy-data and representation-loss requirements.TAR’s requirements may be inefficient; the supplied passage ends before specifying how Booster avoids them.Source: §3.3 · p. 14
  • Alignment-stage defenses include Booster and TAR gradient objectives, collapse traps that sacrifice reasoning ability, benign-text mixing, and updates restricted to safety parameters.CTRL mixes general-domain question-answer pairs; RSN-Tune updates safety parameters that do not overlap with foundation parameters, while LT/NG suggest separating clean and harmful data distributions.Source: §3.3 · p. 15
  • Fine-tuning defenses include a KL-regularizer idea and constrain-SFT weighting the first tokens; a safety-layer method minimizes KL drift but may not efficiently address risk if fine-tuning creates alternative pathways.Source: §3.3 · p. 15
  • Alignment-data mixing preserves safety knowledge during fine-tuning, but Lisa [97] notes its computation scales linearly with the number of fine-tuning samples.SafeInstr [11] and VLGuard [290] mix alignment data; Bi-State optimization and a proximal term are proposed to address mixing costs, while Paraphrase [56] matches the user data’s task format and prompting style.Source: §3.3 · p. 16
  • Prompt-based defenses use BEA’s trigger-linked safe answers or PTST’s separate fine-tuning and inference prompts to mitigate harmful fine-tuning.BEA trains with alignment data and a system prompt containing a backdoor trigger, then activates safe answers with that trigger at inference; PTST [160] uses a general fine-tuning prompt and a safety prompt at inference.Source: §3.3 · p. 16
  • 7.71% of harmful data pass moderation as harmless, while 3.64% of harmless data are mistakenly removed, showing the limits of a simple moderation filter.These are the false-negative and false-positive ratios, respectively, reported for the evaluated moderation model.Source: Table 3 · p. 16, §3.3 · p. 17
  • Beyond naive moderation, SAFT [32] detects harmful data through embedding-space factorization, while Seal [205] and BDS [93] filter or weight samples by their safety effects.Seal formulates a bilevel filter for samples that most degrade safety after fine-tuning; BDS uses Bayesian inference to estimate each point’s safety attribute and assign learning weights.Source: §3.3 · p. 17
  • Reported defenses include a security vector intended to activate harmful knowledge [287], a post-fine-tuning perturbation optimized during fine-tuning, benign-task fusion, safety-subspace update realignment, and selective safety-update merging in safety-critical layers [47].Source: §3.3 · p. 17
  • The surveyed defenses repair alignment through attention-head safety-vector interpolation, representation recalibration, or removal of unsafe updates.CMRM [151] pulls shifted representations toward the backbone distribution; IRR [242] erases update deltas, whereas Antidote sparsifies parameters to zero.Source: §3.3 · p. 17

3.4 Relation between attacks and defenses

  • Existing evaluations cannot establish which defenses best counter particular attacks because attack coverage and datasets vary across studies.Some defenses are tested only against benign-data attacks and others only against harmful-data attacks; even studies of the same attack type use different datasets.Source: §3.4 · p. 18
  • Defense mechanisms suggest different attack coverage: simulated-harmful-data alignment defenses target harmful-data attacks, while post-fine-tuning repairs can address both attack types.Parameter-drift constraints are expected to suppress both harmful- and benign-data attacks, whereas data filtering is expected to work better against harmful-data attacks.Source: §3.4 · pp. 18–19

3.5 Mechanism interpretability of harmful fine-tuning

  • EHA [189] tends to disrupt embeddings in shadow layers, whereas ISA [189] tends to disrupt embeddings in deeper layers.Leong et al. [127] compare the two attack mechanisms.Source: §3.5 · p. 19
  • The safety-basin account frames harmful fine-tuning as pulling aligned weights outside a safety basin, and VISAGE measures resistance to perturbation.Peng et al. [181] propose the safety-basin concept and VISAGE; higher similarity between alignment and downstream fine-tuning datasets is associated with more fragile safeguards, motivating more diverse alignment data [84].Source: §3.5 · p. 19
  • Vision fine-tuning leaves base-model safety alignment largely intact, while VLLM degradation arises from difficulty distinguishing safe from unsafe images and greater attention to harmful images.Guo et al. [71] attribute the degradation to this VLLM image-input behavior rather than direct weakening of the base LLM's alignment.Source: §3.5 · p. 19
  • An evaluation of RepNoise [195] and TAR [217] on open-weight models found they could be much less effective than claimed, sometimes contradicting durability claims [188].The authors advise developers to constrain claims to avoid misleading readers; the paper's claims became controversial.Source: §3.5 · p. 19

4 Evaluation methodology

The survey organizes evaluation around harmful, benign fine-tuning, and alignment datasets, alongside harmfulness and task-performance metrics. It also describes model choices, including aligned and non-aligned bases.

4.1 Datasets

  • Alignment data pairs harmful questions with safe answers; it is used when the base model is unaligned, while some already aligned bases need no such data.For preference training, the survey recommends HH-RLHF because it provides pairs of answers to a harmful question.Source: §4.1 · p. 20
  • Harmful examples simulate attacks, test post-fine-tuning harmfulness, or support defenses; benign downstream data is mixed with harmful data to simulate user fine-tuning.The survey says samples for attack, testing, and defense should be distinct for fair evaluation.Source: §4.1 · p. 20

4.2 Metrics

  • Harmful score measures the harmfulness of responses to test prompts, while fine-tune accuracy measures downstream-task performance.Studies use moderation models, GPT-4 ratings, or string matching for harmfulness; task metrics vary with the downstream task.Source: §4.2 · p. 21

4.3 Models

  • Evaluation commonly uses open-source LLMs, with aligned models such as Llama2-7B-chat and non-aligned models such as Llama2-7B serving as base-model options.The survey notes that most cited studies use aligned bases, while a line of work uses unaligned versions.Source: §4.3 · pp. 21–22

5 Challenge and future direction

This section outlines future research directions for strengthening or making harmful fine-tuning attacks more stealthy, improving defenses, and better understanding mechanisms that affect model safety. It also calls for a comprehensive standard benchmark to compare attacks and defenses across studies.

5.1 Attack side

  • Steganographic attacks face encoded-input/output constraints and detectability, while guardrail-jailbreak attacks show weak transfer and do not address inference-time moderation.Future work should seek lower-perplexity encoding with broader applications and stronger transfer across multi-phase guardrails.Source: §5.1 · p. 22
  • Benign-data attacks still underperform harmful-data attacks without guardrail moderation, motivating better benign-data sampling and construction [79], [69], [121].Proposed directions include assumption-less curation via self-influence and hiding affirmative answers deeper in the response.Source: §5.1 · p. 22

5.2 Defense side

  • Future defenses could filter harmful uploads with outlier detection and extend system-prompt design to learn benign knowledge without learning harmful user prompts.The survey identifies newer out-of-distribution methods and prompt designs that distinguish harmful from benign user content as directions to explore.Source: §5.2 · p. 23
  • Preference-data defenses remain underexplored, while parameter-wise defenses share a mysterious connection that warrants empirical explanation and improved solutions.Preference methods could exploit paired good/bad answers; parameter-wise methods identify safety-relevant parameters for tuning, perturbation, or removal.Source: §5.2 · p. 23
  • Defense research could reduce memory and computation overhead through efficiency techniques and jointly optimize alignment, fine-tuning, and post-fine-tuning stages.Multi-stage co-design may improve robustness but can introduce additional complexity.Source: §5.2 · p. 23

5.3 Mechanism interpretability study

  • Whether embedding drift occurs across all harmful fine-tuning attacks remains unresolved, and layer-wise analysis could clarify safety functions and inform defenses.Prior work reports embedding drift [100] and layer-specific safety roles, motivating analysis of whether these patterns hold universally.Source: §5.3 · pp. 23–24
  • Poorly configured gradient-ascent unlearning can erase normal knowledge and cause single-word outputs, motivating study of its failure conditions and configuration.Gradient ascent maximizes loss over harmful data, but its effects depend on configuration such as hyperparameters.Source: §5.3 · p. 24
  • Research should examine how defenses shape the aligned model’s safety basin and how attack perturbations pull weights out of it, alongside developing a comprehensive benchmark.The safety-basin concept is described in [181].Source: §5.3 · p. 24
  • Because methodologies differ, the field needs a standard attack/defense benchmark to assess current and future solutions, cover SFT and RLHF options such as PPO/DPO, and improve on initial efforts [82] [195] [238].The proposed scope includes alignment and user fine-tuning; [82] [195] [238] are identified as initial attempts that could be more comprehensive.Source: §5.3 · p. 24

6 Conclusion

The survey reviews harmful fine-tuning attacks on safety-aligned models in fine-tuning-as-a-service, characterizes defenses and evaluation methodology, and identifies future research directions.

6 Conclusion

  • The conclusion synthesizes harmful fine-tuning attacks against safety-aligned models in fine-tuning-as-a-service, defense methods, and risk-assessment methodology.It also discusses future research directions and challenges.Source: §6 · p. 24
  • The authors present the survey as guidance for understanding harmful fine-tuning attacks and defenses and for future research.They frame this guidance around fortifying safety-aligned language and vision models in anticipation of small harmful fine-tuning datasets.Source: §6 · p. 24
  • The survey aims to support researchers and engineers developing protections for safety-aligned models.The stated intended audience includes LLM researchers and engineers.Source: §6 · p. 24

A Question of Interest

The section addresses practical choices for assessing harmfulness and fine-tuning performance, selecting tasks and training approaches, and choosing a model base. It notes trade-offs in evaluation and training resource demands.

Q1: Evaluating harmful score

  • Harmfulness evaluation can use GPT-4 ratings or open-source moderation models, while GPT-4 evaluation is described as costly despite being well accepted and reflecting human evaluation.The section also notes that GPT-4 API costs may be unaffordable for many research groups.Source: A Question of Interest · p. 36

Q2: Task and performance evaluation

  • Fine-tuning studies use different task settings, and the section raises how task performance should be evaluated as a research-design question.The passage identifies MT-Bench as a commonly used dataset, without specifying a preferred task or metric.Source: A Question of Interest · p. 36

Q3: Alignment and fine-tuning approach

  • The authors prioritize RLHF over full SFT and LoRA SFT, while noting RLHF requires more GPU resources and LoRA SFT takes less training time.The stated ordering is RLHF > SFT (Full) > SFT (LoRA).Source: A Question of Interest · p. 36

Q4: Base-model choice

  • The section poses whether aligned or non-aligned LLMs should be used as the base model, but these passages do not state a recommendation.Examples named in the question are Llama2-7B-chat and Llama2-7B.Source: A Question of Interest · p. 36

Llama2-7B) for the experiment?

The discussion weighs base-model choices for service-provider defenses, dataset assumptions, and shared metric terminology. It also questions why the service-provider setting is preferred over users fine-tuning models themselves.

Llama2-7B) for the experiment?

  • Alignment-stage defenses may be more reasonably integrated into the safety-alignment process using an unaligned base model.The paper notes that researchers are agnostic to the alignment process, so a defense tested on an aligned model may benefit from unknown procedures used to produce it.Source: Llama2-7B) for the experiment? · p. 36
  • Existing defense designs assume access to safety-alignment data, in-distribution harmful data, or proxy datasets.The examples include harmful-question/safe-answer pairs and harmful-question/harmful-answer pairs.Source: Llama2-7B) for the experiment? · pp. 36–37
  • The authors advocate the short, intuitive terms “harmful score” and “fine-tune accuracy” for measuring model harmfulness and task accuracy.They say papers use different names for these metrics.Source: Llama2-7B) for the experiment? · p. 37
  • The discussion asks why to study fine-tuning-as-a-service rather than users developing their own fine-tuning and post-fine-tuning solutions.It also notes that using an already aligned model such as Llama2-7B-chat would be simpler than aligning Llama2-7B oneself.Source: Llama2-7B) for the experiment? · pp. 36–37

fine-tunes the model?

The discussion questions the service-provider threat setting and notes that users can control fine-tuning and deployment. It also describes defenses as difficult when users can alter the process or model.

fine-tunes the model?

  • The authors question the motivation for users to attack their own models and the service provider’s responsibility for user-deployed fine-tuned models.Unintentional inclusion of harmful data is offered as a possible reason users might produce an alignment-broken model.Source: fine-tunes the model? · p. 37
  • The discussion describes defense as very hard in this setting, even before considering users’ control over fine-tuning.It introduces an extreme example of users already having control over the model.Source: fine-tunes the model? · p. 37
  • Users control the model weights and fine-tuning process, and may use larger optimization steps to subvert alignment.The discussion says defenses may fail when users fine-tune an alignment-broken model from the provider’s release model.Source: fine-tunes the model? · p. 37

B Related survey

Existing LLM surveys often address broad risks or a single safety aspect, leaving room for a focused account of harmful fine-tuning. This survey reviews its attack setting, attacks, defenses, evaluation methods, and defense timing.

B Related survey

  • The related surveys are grouped into general LLM security and trust, risk evaluation and mitigation, and attacks or red-teaming.Examples include broad security discussions, evaluation-platform reviews, and jailbreak-focused surveys.Source: B Related survey · pp. 37–38
  • The survey identifies a gap: prior reviews often lack in-depth analysis of a specific threat model or do not cover its attacks, defenses, and evaluations together.The authors say surveys with diversified threat models tend to emphasize only one aspect of LLM safety.Source: B Related survey · p. 38
  • The authors present a systematic review focused specifically on harmful fine-tuning, which they describe as a newly emerging LLM threat.They position it as a first attempt alongside a concurrent study.Source: B Related survey · p. 38
  • The review covers harmful-fine-tuning attack settings, existing attacks and defenses, and evaluation methodology.The survey also introduces a defense taxonomy based on when defenses occur: alignment, fine-tuning, or post-fine-tuning.Source: B Related survey · p. 38

C More Statistical Results

Across the reported fine-tuning settings, harmful scores and harmful-data losses vary with dataset and training steps. The appendix also flags open-source models as another safety-unalignment attack surface.

C More Statistical Results

  • Unlike benign SST2 fine-tuning, GSM8K fine-tuning decreases harmful training and testing losses, with the authors attributing the discrepancy probably to GSM8K’s similarity to harmful QA data.The paper contrasts GSM8K with SST2, a classification dataset whose fine-tuning increases harmful losses.Source: C More Statistical Results · p. 39
  • Harmful score rises with more steps on pure or partially harmful data, while harmful training and testing losses fall across fine-tuning datasets; GSM8K’s loss decreases are slighter.Figure 11 reports these trends for pure GSM8K, harmful data, and 50% mixtures.Source: Figure 11 · p. 39, C More Statistical Results · p. 39

D Other Safety Unalignment Attack Scenarios

  • The appendix identifies open-source models as another potential attack surface for safety unalignment.This is raised as a scenario beyond harmful fine-tuning in fine-tuning-as-a-service.Source: D Other Safety Unalignment Attack… · p. 39

D.1 Risk of Open-source model: model users as an attacker

The section situates risks from open-source model users and publishers alongside backdoor threats in hosted LLM services and broader LLM alignment and attack-defense research. It distinguishes harmful fine-tuning from related data-poisoning attacks and reviews LLM backdoors, data-extraction attacks, and defenses, noting their differing assumptions, triggers, and effects on safety alignment.

D.1 Risk of Open-source model: model users as an attacker

  • In this scenario, users of open-source models have full knowledge of the model’s weights and can arbitrarily manipulate them, including through harmful fine-tuning to elicit the model or weights replacement.This attack scenario is adopted by RepNoise [196] and TAR [217].Source: §D.1 · pp. 39–40

D.2 Risk of open-source model: model publisher as an attacker

  • An open-source publisher can release a backdoored model whose alignment fails when a hidden trigger appears, leaving deploying users unaware of the backdoor.The publisher controls alignment and backdoor fine-tuning before release, while defenders control deployment and inference and may further fine-tune to erase the backdoor; the authors call this scenario more realistic than user attacks because defenders choose the model and inference process.Source: §D.2 · p. 40

D.3 Risk of LLM service with backdoor: Publisher as an attacker

  • In hosted LLM services, a publisher can exploit an undisclosed backdoor trigger to break alignment and attack users even after prior API evaluation.The publisher controls alignment, backdoor fine-tuning, and deployment; users lack model weights or process access and can defend only through inference API calls.Source: §D.3 · p. 41

E.1 Safety alignment on LLM

  • Safety alignment trains LLMs to generate helpful, harmless outputs aligned with human preferences, using human-aligned supervised datasets; RLHF techniques often use preference pairs, and original PPO uses SFT to train a preference-based reward model [6] [40] [49] [171] [190] [211] [245] [271].Source: §E.1 · p. 41

E.2 Other attack/defenses towards LLM

  • The survey describes jailbreaks that manipulate prompts to elicit harmful answers and backdoors that activate harmful outputs when a predefined trigger is present.Jailbreak work includes crafted manual prompts [207] and GCG-generated adversarial suffixes [292].Source: §E.2 · p. 42
  • LLM backdoors include standard attacks requiring a backdoor dataset and model training, as in BadGPT [209], and alternatives without training-data access: BadChain prompts through COT [246], while TA adds malicious activation-layer vectors at inference [232].Source: §E.2 · p. 42
  • Backdoor defenses include Beear [272], which combines embedding drift with perturbation-aware training, and LAT [208], which trains against harmful-loss-minimizing perturbations.LAT assumes the attacker’s backdoor objective is known and aims to unlearn the trigger association and harmful knowledge.Source: §E.2 · p. 43
  • Data-extraction attacks can recover individual training examples from GPT-2 [18] via many queries, have been tested for sensitive data in BERT [126], and can affect safety-aligned ChatGPT [168].Source: §E.2 · p. 43
  • A prompt-tuning attack can increase data-extraction rates, but gradient-ascent defense is proposed; even state-of-the-art model editing may fail to prevent recovery [172] [177].Attacks can still recover information described as deleted from an edited model.Source: §E.2 · p. 43
  • Both data poisoning and harmful fine-tuning mix malicious with benign training data and aim to alter model behavior, but their goals differ.Backdoors may trigger diverse malicious behaviors, whereas harmful fine-tuning specifically seeks to break safety alignment.Source: §E.2 · p. 43
  • Backdoors activate on predefined input patterns, while harmful fine-tuning elicits harmful answers from harmful prompts.Backdoor triggers can also bypass chain-of-thought reasoning, unlike harmful fine-tuning’s safety-alignment objective.Source: §E.2 · pp. 43–44

F More discussion on survey methodology

The paper includes a collection record documenting when and with which tools papers were collected.

F.1 Paper collection record

  • The paper collection record logs the date and tools used to collect papers.The record is presented as Table 9.Source: §F.1 · p. 44, Table 9 · p. 44

F.2 Potential bias in qualitative analysis

The authors identify possible bias in both paper collection and qualitative analysis, and describe inviting readers to report missing papers for future updates.

F.2 Potential bias in qualitative analysis

  • The collection may be biased because seed-paper selection and search keywords may be subjective or fail to cover all papers.The first author conducted the collection process, which may introduce human bias.Source: §F.2 · p. 45
  • The first author performed the initial categorization and analysis, while other authors inspected and double-checked the categorization.The first author's judgment might bias the inspectors and, ultimately, the categorization and analysis.Source: §F.2 · p. 45
  • Readers are invited to report missing relevant papers, and the authors say they will update the publication list and preprint.The publication list is maintained in the GitHub repository.Source: §F.2 · p. 45
Loading 2409.18169v6…