Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler

• Preemptive attack simulations are difficult to extend beyond bounded threat models and adapt to varying attack settings.The cited limitations arise from difficulty anticipating unknown attacks and from simulation’s inability to capture post-hoc attack variability and complexity.Source: §1 · pp. 1–2
• BDS conditions each data point’s safety-attribute posterior on the fine-tuning and alignment datasets, then weights fine-tuning data using attributes sampled from that posterior.This tuning-stage defense requires no attack simulation and tailors its defense to the specific dataset.Source: §1 · p. 2
• The neural scheduler uses amortized Bayesian learning to transfer efficiently to new data without retraining.The datapoint-wise posterior otherwise requires relearning for new data.Source: §1 · p. 2
• 74.4% improvement at a harmful ratio of 0.9 and an average boost of over 50% across ratios from 0 to 1 accompany BDS’s reported state-of-the-art performance.Experiments cover five downstream datasets, three LLM architectures, and diverse attack and defense settings.Source: §1 · p. 2

• Users upload personalized fine-tuning datasets, which the service provider uses to fine-tune a model and return a personalized API.Source: §2 · p. 2
• The threat model allows a user-attacker to submit a dataset containing a proportion of harmful data and the remainder benign data.Source: §2 · p. 2
• The attacker controls the dataset’s harmful-data proportion and content without needing knowledge of model architecture or parameters; the provider controls fine-tuning.Source: §2 · p. 3
• The provider is assumed to maintain an alignment dataset of harmful-prompt and safe-answer pairs, so the defense does not require a simulated harmful dataset.Source: §2 · p. 3

• Proposed mechanisms for safety degradation include adversarial shifts in parameters or embeddings, vulnerable layers or modules, and catastrophic forgetting of alignment knowledge.Source: §3 · p. 3
• Harmful fine-tuning research includes attacks using benign data, safety-degrading benign subsets, covert malicious tuning, and data designed to circumvent moderation guardrails.Source: §3 · p. 3
• BDS is a fine-tuning-stage defense using Bayesian loss-based data scheduling, unlike prior heuristic or threshold-based hard-label selection, and it does not require attack simulation.The passage contrasts this with Booster, which requires attack simulation.Source: §3 · p. 3

4 Methodology

• The methodology section introduces BDS’s framework and details its scalar and amortized neural scheduler implementations.Source: §4 · p. 3

4.1 Bayesian Data Scheduler Framework

• BDS infers the joint posterior of data weights and model parameters from the fine-tuning and safety-alignment datasets, then uses the weights to modulate each example’s contribution.The framework associates each fine-tuning example with a latent safety attribute and constrains model updates using that attribute.Source: §4.1 · pp. 3–4

4.2 Bayesian Scalar Scheduler

• A loss gap between safety-aware and safety-agnostic models reduces a data point’s weight when it fits the safety-aware model less well.SGLD samples the posterior through iterative updates, using minibatch approximations to improve gradient-computation efficiency.Source: §4.2 · p. 5

4.3 Amortized Bayesian Neural Scheduler

• The neural scheduler amortizes per-example weight inference into one posterior over shared network parameters, enabling new-data weights through forward passes without retraining.The network takes each fine-tuning example as input to infer its weight posterior.Source: §4.3 · p. 6

4.4 Data Weight Transformation

• Softmax adjusts weights relative to weighted-average loss, while identity and sigmoid monotonically reduce weights for both benign and harmful data.Under identity transformation, cumulative posterior bias gives earlier sampling errors greater influence.Source: §4.4 · p. 7

5.1 Main Results

• BDS reduces average Harmful Score by 9.60% and improves average Finetune Accuracy by 0.8% over baselines at harmful ratios from 0 to 0.2.The comparison uses |Dsafe| = 1000 and |Dft| = 1000 for BDS.Source: Table 1 · p. 8, §5.1 · p. 8

5 Experiment

• Experiments use five fine-tuning datasets, three LLM architectures, and harmful-score and task-accuracy metrics, comparing BDS with five defense baselines.The harmful score is the proportion of unsafe outputs; task accuracy is measured on the corresponding fine-tuning task's test set.Source: §5 · p. 8

5.1 Main Results

• At harmful ratio 0, BDS preserves fine-tuning performance better than SFT, and the paper attributes this trade-off to jointly optimizing safety alignment and fine-tuning objectives.The authors contrast this with SFT's sequential learning paradigm.Source: §5.1 · pp. 8–9
• At harmful ratios of 0.5 to 1.0, BDS reduces harmful score by over 70% and improves fine-tuning accuracy by nearly 1% against Booster.Prior evaluations were limited to low harmful ratios.Source: Table 2 · p. 8, §5.1 · p. 9
• Across four fine-tuning datasets, BDS maintains harmful scores around 1, while baseline performance can vary sharply across tasks.Vaccine's harmful score is 3.7 on GSM8K and 43.30 on AlpacaEval.Source: §5.1 · p. 9, Table 3 · p. 9
• With |Dsafe| and harmful ratio p fixed, BDS maintains harmful score around 1 across |Dft| sizes, while Booster rises from 3.8 to 33.6; with |Dft| and p fixed, BDS stays below 2 at |Dsafe|=100.Source: §5.1 · pp. 9–10
• The neural scheduler performs comparably to the scalar scheduler and transfers to unseen in-domain and out-of-domain data without retraining, keeping harmful scores below 3.On SST2, its harmful score is below 2.Source: Table 7 · p. 9, §5.1 · p. 10
• BDS maintains harmfulness around 1 under OOD and identity-shifting attacks; loss-based scheduling assigns lower weights to harmful data without explicit attack simulation.The paper attributes this behavior to harmful examples exhibiting shared unsafe behavior in the safety-aware model's loss landscape.Source: §5.1 · p. 10

6 Conclusion

• BDS uses a posterior over data weights conditioned on the specific dataset to adapt defense without attack simulation; its neural scheduler transfers to new data without retraining.The conclusion reports that experiments outperform baselines by over 70% across diverse attack and defense settings.Source: §6 · p. 10

A Related Work

• Prior defenses operate at alignment, fine-tuning, or post-fine-tuning stages; several fine-tuning-stage methods mix alignment data but do not explicitly isolate harmful samples.The paper also notes that such mixing approaches require alignment data to scale with harmful data.Source: A Related Work · p. 19
• Data-curation methods often assume distributional similarity between raw and reference datasets, whereas BDS uses loss differences and is described as independent of that assumption.The paper also characterizes BDS's design as computationally efficient for harmful fine-tuning defense.Source: A Related Work · p. 19

B.2 Details on Fine-Tuning Task Evaluation

• Task evaluations define dataset-specific instructions and inputs for SST2, GSM8K, and AGNEWS, while AlpacaEval uses GPT-4 instruction-answer demonstrations and judges helpfulness on unseen prompts.The AlpacaEval evaluation uses ChatGPT's API.Source: §B.2 · p. 20

B.3 Details on Scheduler Training

• The scheduler uses factorized priors as an assumption, initializes scalar weights to 0.1 for optimization, and implements the neural scheduler with a 125M Fairseq-Dense model.Fine-tuning uses LoRA; the reported extreme-ratio setup uses |Dsafe|=5000 at harmful ratio 1 and |Dsafe|=1000 otherwise.Source: §B.3 · p. 20, §B.4 · p. 20

C Algorithms

• The algorithm batches alignment and fine-tuning data, updates the LLM and either scheduler, and returns the fine-tuned model after T iterations.The algorithm accepts either the Bayesian Scalar Scheduler or Amortized Bayesian Neural Scheduler.Source: C Algorithms · p. 21

D.1 Weight Distribution of Benign and Harmful Data under Various Harmful Ratios

• Across harmful ratios from 0.2 to 0.9, weight-distribution plots show benign data receiving higher weights and harmful data lower weights, including at 0.9.Figure 7 includes scatter and boxplots plus separate weight histograms for benign and harmful data.Source: §D.1 · p. 21, Figure 7 · p. 22

D.2 Scheduling Dynamics for Benign and Harmful Data under Various Harmful Ratios

• Scheduling visualizations show benign examples mostly in higher-weight groups and harmful examples mostly in lower-weight groups across ratios from 0.2 to 0.9.The reported separation remains effective at harmful ratio 0.9.Source: §D.2 · p. 21, Figure 8 · p. 23

D.3 Robustness for OOD Attack.

• Under OOD attacks using RealToxicityPrompts and AdvBench against BeaverTails alignment data, BDS maintains strong defense performance, which the authors relate to shared unsafe behavior in loss.The datasets differ in domain, according to the passage.Source: §D.3 · p. 24, Table 8 · p. 24

D.4 Robustness for Identify Shifting Attack (ISA) [48]

• Against identity-shifting prompts prepended to fine-tuning examples, BDS achieves a harmful score around 1, similar to non-ISA settings.The authors state this indicates effective deweighting of harmful data despite prompt-level manipulation.Source: §D.4 · p. 24, Table 9 · p. 24

D.5 Robustness for Larger Fine-Tuning Dataset Size

• BDS maintains strong defense performance as the fine-tuning dataset grows to 10,000 examples, while Booster degrades as the absolute number of harmful examples increases.The comparison is reported in Table 10.Source: §D.5 · p. 24, Table 10 · p. 24

D.6 Robustness on More Complex Tasks: Data-to-Text Generation.

• On ViGGO and CACAPO data-to-text tasks, BDS outperforms the baseline in helpfulness measured by ROUGE-1 and in harmfulness score.These tasks are part of the GEM benchmark.Source: §D.6 · p. 25, Table 11 · p. 25

D.7 Comparison with Detection-Based Methods.

• BDS achieves higher fine-tuning accuracy and lower harmfulness scores than both detection-based methods, whose false positives remove benign data and false negatives retain harmful data.Its soft weighting adjusts sample influence without explicit hard labels, supporting handling of uncertain or ambiguous data.Source: §D.7 · p. 25, Table 12 · p. 25

D.8 Comparison with Deterministic Data Curation Methods.

• BDS consistently outperforms DSIR, which relies on distributional similarity, while BDS accounts for uncertainty in datapoint safety attributes.The authors attribute BDS's advantage to robustness to distributional differences and more reliable weighting of potentially ambiguous data.Source: §D.8 · pp. 25–26, Table 13 · p. 26

E.1 Impact of Different Weight Transformations.

• The comparison of softmax, sigmoid, and identical transformations supports softmax for bidirectionally updating weights.The result is reported as confirming the analysis in Sec. 4.4.Source: §E.1 · p. 26, Table 14 · p. 26

E.2 Impact of Different Weight Priors

• The noninformative prior yields better defensive and slightly better fine-tuning performance than Gaussian priors.The authors attribute this to greater freedom for weight updates; Gaussian priors may impose incorrect constraints when defenders lack prior knowledge of harmful-data ratios.Source: §E.2 · p. 26, Table 15 · p. 26

E.3 Impact of Weight Initialization

• Across initialization values from 0.001 to 10, BDS has harmful scores of 1.1–1.5 and fine-tuning accuracy of 93.58%–93.69%.The authors describe this stability as insensitivity to the initialization hyperparameter.Source: §E.3 · pp. 26–27, Table 16 · p. 27

E.4 Impact of the Alignment Dataset

• Removing the alignment dataset sharply increases the harmful score because harmful samples no longer consistently incur high loss, leading to incorrect weight assignments.The passage attributes the loss pattern to the model failing to learn safety awareness.Source: §E.4 · p. 27, Table 17 · p. 27

E.5 Impact of the Data Weight

• Mixing alignment and fine-tuning data without appropriate weighting substantially increases the harmful ratio, whereas down-weighting harmful data suppresses its adverse influence.The reported comparison says naive mixing alone is insufficient for the desired defense robustness.Source: §E.5 · p. 27, Table 18 · p. 27

F.1 Insights into Adaptiveness to Diverse Attack Dynamics

• BDS conditions its posterior on the fine-tuning dataset and applies loss-based weights to individual samples, assigning lower weights to harmful data across attack strategies.The passage says harmful data tends to exhibit consistently higher loss, supporting adaptive down-weighting without explicit attack simulation.Source: §F.1 · p. 27

F.2 Insights into Helpfulness-Harmlessness Trade-Offs

• SFT has lower fine-tuning accuracy at p = 0 than at p = 0.2, while BDS jointly optimizes alignment and fine-tuning objectives rather than training sequentially.The authors explain SFT's pattern through a helpfulness-harmlessness trade-off: harmful samples can restore plasticity while sacrificing some harmlessness; the comparison also aligns with [29].Source: §F.2 · p. 28

F.3 Simulation-Free vs. Simulation-Based Defense

• Simulation-based defenses can be infeasible to construct, may miss unpredictable or unseen attack dynamics, and adversarial training can have expensive, unstable optimization.BDS avoids reliance on assumed attack scenarios, addressing the simulation requirement and its infeasibility.Source: §F.3 · p. 28

F.4 Data Scheduling vs. Data Filtering

• Scheduling learns soft weights for individual samples, avoiding the manual threshold required for filtering and alleviating hard-label false-positive and false-negative issues.Ratio-based thresholds are impractical without knowing the harmful-data proportion, while value-based thresholds are difficult to tune across a broad continuous range.Source: §F.4 · pp. 28–29

F.5 Over-Refusal

• A few BDS-trained models refuse safe prompts, attributed to base-model language-understanding limitations, while a small number answer unsafe prompts without being classified as refusals.The section reports these behaviors on Xstest and refers to examples in Table 20.Source: §F.5 · p. 29, Table 19 · p. 29, Table 20 · p. 29

F.6 Zero-Shot Performance

• LLaMA-2 7B zero-shot performance is 0% on SST2, 0% on AGNEWS, 2.7% on GSM8K, and 4.2% on AlpacaEval.The authors suggest format requirements or complex reasoning needs help explain these low scores.Source: §F.6 · p. 29

F.7 Limitation and Future Work

• The neural scheduler adds computational overhead through an extra forward pass to compute data weights, though the authors describe the cost as modest relative to overall training.The paper identifies this overhead as a limitation and points to its complexity analysis in Sec. G.1.Source: §F.7 · p. 29

F.8 Impact Statement

• The work identifies safer, more reliable commercial fine-tuning as a potential positive impact of mitigating harmful fine-tuning risks.The impact statement frames this as a potential broader impact for user-customized LLM deployment.Source: §F.8 · p. 30
• Reweighting particular fine-tuning subsets could amplify ideological stances, diminish perspective diversity, and potentially exacerbate social bias.The paper identifies this as a possible misuse of the method.Source: §F.8 · p. 30

G.1 Time and Space Complexity Analysis

• BDS has time complexity O(n2(f + w)) and space complexity O(f + 2d + w), with data-weight size w typically much smaller than model-parameter size f.Booster's alignment stage requires three gradients per optimization step, and the paper reports that BDS can train on one A100-40G whereas Booster requires an H100-80G.Source: §G.1 · p. 30, Table 21 · p. 30
• BDS achieves over 3× faster training than Booster while using less than half the GPU memory.The practical comparison is reported in the runtime and memory benchmark; BDS uses standard backward passes rather than expensive bi-level optimization.Source: §G.1 · pp. 30–31, Table 22 · p. 31

G.2 Convergence Analysis of SGLD Sampling

• Under the stated dissipativeness and smoothness assumptions, initialization and step-size conditions, the SGLD iterate distribution approaches the target posterior in total variation distance.The theorem bounds the distance after K iterations by terms depending on iteration count, batch size, step size, dimension, and problem-dependent constants.Source: §G.2 · p. 31

H.1 Derivation of Eq. (1)

• Eq. (1)'s proof uses Bayes' theorem to express the posterior through alignment and fine-tuning likelihood terms and prior factors.The displayed factorization includes an inverse fine-tuning-data likelihood term.Source: §H.1 · p. 32

H.2 Derivation of Eq. (4)

• The Eq. (4) derivation presents a fine-tuning likelihood multiplied by an expectation over the conditional parameter distribution.The expectation is indexed by p(Dft|w) and contains p(θ|Dft,w).Source: §H.2 · p. 32

H.3 Derivation of Eq. (10)

• The Eq. (10) derivation substitutes the neural weight parameterization w=N(Dft; ϕ) into posterior expressions.The expressions involve the alignment likelihood and the fine-tuning likelihood under that parameterized weight.Source: §H.3 · p. 33

H.4 Derivation of Eq. (16)

• The Eq. (16) derivation differentiates the softmax weight transformation and simplifies the resulting weight-update gradient.It then states the update rule for each weight w_i.Source: §H.4 · pp. 33–34

I.1 Motivation of Proposed Posterior Bias

• Without held-out benign tuning data, direct posterior sampling may suppress fine-tuning data while preserving alignment, impairing generalization; including that data would balance safety and tuning objectives.The held-out clean dataset serves as a test or validation set for fine-tuning performance.Source: §I.1 · p. 34

I.2 Definition of Posterior Bias

• Posterior bias is the divergence between posteriors derived from alignment data alone and from alignment data plus held-out clean tuning data.Because attackers do not provide the held-out set, the text says posterior bias cannot be derived analytically and instead motivates an SGLD-based empirical measure.Source: §I.2 · p. 34

I.3 Empirical Posterior Bias with SGLD Sampling

• Empirical posterior bias is estimated from the expected difference between terminal states of paired SGLD trajectories with identical initial states.The trajectories target posteriors with and without held-out clean tuning data, and the stated norm is ℓ1.Source: §I.3 · pp. 34–35

I.4 Understanding Posterior Bias under Identity Weight Transformation

• Under identity transformation, weights for both benign and distribution-shifted harmful data decrease monotonically; the large benign-data reduction harms fine-tuning performance.The benign-data loss is lower when the posterior includes held-out clean tuning data, while the harmful-data loss remains high.Source: §I.4 · p. 35

I.5 Theorem of Time-Weighted Accumulation of Posterior Bias

• The final sampled-weight difference is proportional to cumulative posterior bias, with the contribution at iteration t weighted by T−t.The theorem compares trajectories targeting posteriors with and without held-out clean tuning data.Source: §I.5 · p. 35
• Earlier SGLD iterations have greater influence on cumulative bias, so suboptimal early sampling can propagate and strongly affect posterior inference.The text identifies early-stage bias management as a direction for future work.Source: §I.5 · p. 36

I.6 Proof of theorem Theorem 4.2

• The proof relates gradient differences to cumulative prior weight differences, then reorders the summation to expose the remaining-step factor.The factor T−k represents the remaining steps from step k to T.Source: §I.6 · p. 37