PaperlayerView exact revision on arXiv

Source-linked AI summary

Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets

Lei Hsiung, Tianyu Pang, Yung-Chen Tang, Linyue Song, Tsung-Yi Ho, Pin-Yu Chen, Yaoqing Yang

arXiv:2506.05346v1cs.CRcs.CLcs.LG

TL;DR

The paper asks how upstream safety-alignment data relates to guardrail degradation during downstream fine-tuning, a question less addressed than reactive or fine-tuning-stage defenses. It compares alignment subsets by representation similarity to downstream tasks and finds that high similarity is associated with greater jailbreak vulnerability, while low similarity is associated with stronger robustness.

  • Problem

    The role of upstream safety-alignment data in guardrail degradation remains inadequately explored despite concern that downstream fine-tuning can compromise safety.

    Source: §1 · p. 2

  • Method

    The authors compare safety-alignment datasets with differing representation similarity to downstream fine-tuning tasks and assess guardrail vulnerability.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    High-similarity alignment subsets raised jailbreak attack success rates by up to 10.33% versus low-similarity subsets, and lower similarity was associated with greater robustness.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    Dataset privacy and representation similarity are actionable considerations for selecting and engineering alignment data to support durable guardrails.

    Source: §1 · p. 2, §5 · p. 7

    How alignment similarity behaves in multimodal and reasoning-intensive models remains a subject for future study.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recent advancements in large language models (LLMs) have underscored their vulnerability to safety alignment jailbreaks, particularly when subjected to downstream fine-tuning. However, existing mitigation strategies primarily focus on reactively addressing jailbreak incidents after safety guardrails have been compromised, removing harmful gradients during fine-tuning, or continuously reinforcing safety alignment throughout fine-tuning. As such, they tend to overlook a critical upstream factor: the role of the original safety-alignment data. This paper therefore investigates the degradation of safety guardrails through the lens of representation similarity between upstream alignment datasets and downstream fine-tuning tasks. Our experiments demonstrate that high similarity between these datasets significantly weakens safety guardrails, making models more susceptible to jailbreaks. Conversely, low similarity between these two types of datasets yields substantially more robust models and thus reduces harmfulness score by up to 10.33%. By highlighting the importance of upstream dataset design in the building of durable safety guardrails and reducing real-world vulnerability to jailbreak attacks, these findings offer actionable insights for fine-tuning service providers.

1 Introduction

The introduction frames safety degradation after downstream fine-tuning as an upstream data-design problem: the role of safety-alignment data remains underexplored. The authors investigate whether representation similarity between alignment and downstream datasets relates to jailbreak vulnerability.

1 Introduction

  • Safety-aligned models can lose guardrails after third-party fine-tuning, but the nature of the original safety-alignment data remains inadequately explored.The introduction identifies this as a gap alongside prior work on post-hoc and reactive defenses.Source: §1 · pp. 1–2
  • The study tests whether high representation similarity between upstream alignment data and downstream tasks is associated with greater guardrail fragility.The authors create three alignment-dataset versions with differing similarity to downstream fine-tuning data.Source: §1 · p. 2
  • High-similarity alignment subsets raised jailbreak attack success rates by up to 10.33% compared with low-similarity subsets.The authors also report that public accessibility can let malicious actors deliberately exploit high-similarity data.Source: §1 · p. 2
  • The authors propose that providers can measure representation similarity to select models with lower jailbreak vulnerability for particular downstream tasks.They present upstream dataset privacy and representation attributes as relevant to guardrail durability.Source: §1 · pp. 2–3

2 Related Works

Prior work covers alignment methods, fine-tuning attacks, and defenses, while the paper identifies limited examination of the upstream causes of safety degradation.

2 Related Works

  • Common safety-alignment techniques include supervised fine-tuning, reinforcement learning with human feedback, and machine unlearning (Ouyang et al., 2022; Christiano et al., 2017; Liu et al., 2025b).The related-work discussion also mentions patch-based methods that continuously enhance protection against malicious input.Source: §2 · p. 3
  • Safety-aligned models can be compromised by small harmful fine-tuning sets or commonly used benign instruction datasets (He et al., 2024; Ji et al., 2024c; Huang et al., 2025c; Guan et al., 2025).Studies have examined these attacks through statistical, information-theoretic, representational, and loss-landscape perspectives.Source: §2 · p. 3
  • Prior defenses include upstream adversarial training, downstream gradient constraints, safety-data mixing, backdoors, output correction, and post-fine-tuning interventions (Qi et al., 2025; Hsu et al., 2024; Bianchi et al., 2024; Zhu et al., 2024).Some methods seek to preserve upstream alignment during adaptation, while others intervene after fine-tuning.Source: §2 · p. 3
  • The paper identifies a gap in in-depth study of root causes and investigates the relationship between upstream alignment data and downstream fine-tuning tasks.The authors distinguish this focus from prior work on robustness-improving defenses.Source: §2 · p. 3

3 What Damages Safety Guardrails?

The section first examines whether clustering can identify harmful subsets without harmful-data anchors, then asks how similarity between upstream alignment data and downstream tasks relates to guardrail robustness.

3.1 High-similarity Clusters Are More Harmful

  • The first research question asks whether an anchor-free approach can identify a data subset that significantly erodes safety guardrails.It follows prior observation that Alpaca’s Top-100 Harmful subset contained mainly list-format data.Source: §3.1 · p. 3
  • High-similarity groups were 15.7% more damaging than Top-100 Harmful data in Alpaca and 16.3% more damaging in Dolly.The authors interpret the results as empirical support for an overfitting explanation and report that clustering can identify guardrail-eroding subsets.Source: §3.1 · p. 4, Figure 2 · p. 4
  • Clustering LLAMA-2-7B-CHAT representations grouped Alpaca into 20 question-format clusters, from which the authors sampled 100 list-format examples for fine-tuning.The approach isolates groups with high intra-group similarity, rather than matching examples to harmful anchors.Source: §3.1 · p. 4

3.2 Similarity between Upstream and Downstream Datasets

  • The second research question asks how similarity between upstream alignment datasets and downstream fine-tuning data affects guardrail robustness.The authors hypothesize that high similarity is associated with greater fragility, while low similarity may reduce overfitting.Source: §3.2 · p. 4
  • The subset-selection method ranks alignment examples by cosine similarity between their representations and downstream-task examples, then selects high- or low-similarity examples.Features come from the final hidden state of the last completion token, using a model without safety alignment.Source: §3.2 · p. 4, Figure 3 · p. 4

4 Experiment

The experiment compares safety-alignment subsets with different similarity to downstream tasks across harmful and benign fine-tuning. Low-Sim alignment yields more durable guardrails than High-Sim alignment, including when downstream defenses are used.

4.1 Experimental Setup

  • The safety-alignment subsets were drawn from BeaverTails responses to harmful prompts, and high- and low-similarity subsets were size-matched at 1,000 or 5,000 items.The study evaluated harmfulness using HEx-PHI and BEAVER-DAM-7B; utility was evaluated with MT-Bench for Alpaca and Rouge-1 F1 for SAMSum.Source: §4.1 · pp. 5–6

4.2 Experimental Results

  • Low-Sim models showed up to a 10.33% difference in HS versus High-Sim and Random across harmful and benign tasks and both model scales.The authors report lower harmfulness metrics for Low-Sim models.Source: Table 1 · p. 6, §4.2 · p. 6, Table 2 · p. 7

4.1 Experimental Setup

  • The comparison used High-Sim, Low-Sim, and randomly sampled BeaverTails alignment subsets against two harmful and two benign downstream tasks.The authors report LLAMA-2-7B and LLAMA-2-13B results; GEMMA-2 results are in Appendix C.2.Source: §4 · p. 5, §4.1 · p. 5, §4.2 · p. 6

4.2 Experimental Results

  • Larger safety-alignment datasets tended to lower HS, while Low-Sim models remained safer than High-Sim models with either downstream defense.The defenses were SafeInstr and Backdoor Enhanced Alignment (BEA).Source: §4.1 · p. 5, §4.2 · p. 7

5 Discussion

Representation similarity between alignment data and downstream tasks is associated with guardrail vulnerability, including when fine-tuning data is benign. The discussion connects this risk to dataset privacy and proposes similarity-aware model selection as a practical complement to existing defenses.

5 Discussion

  • Higher representation similarity is correlated with greater jailbreak vulnerability, even when downstream fine-tuning data is entirely benign.The authors report that lower similarity enhances the robustness of safety constraints.Source: §5 · p. 7
  • Public alignment datasets could enable adversaries to construct similar fine-tuning tasks that dismantle post-alignment guardrails.The discussion links dataset confidentiality to the structural integrity of safety mechanisms.Source: §5 · p. 7
  • Providers can compare upstream alignment corpora with candidate datasets, then reject risky requests or route them to models with more orthogonal data distributions.Figure 4b outlines this similarity-aware selection pipeline.Source: §5 · p. 7, Figure 4 · p. 8
  • Similarity-aware selection can complement post-hoc pruning, constraint-based fine-tuning, and residual output filters in a layered deployment strategy.The paper describes this approach as complementary to existing safety defenses.Source: §5 · pp. 7–8
  • Task-vector analysis is proposed as future work to identify representations and neurons susceptible to erosion during fine-tuning.Comparing High-Sim and Low-Sim task vectors could inform analysis of durable safety.Source: §5 · p. 8
  • The study focuses on harmful-output guardrails; factuality, fairness, helpfulness, multimodal inputs, and reasoning-intensive settings are proposed for future examination.Examples include long-form reasoning, image-text pairs, and video-language inputs.Source: §5 · p. 8

6 Conclusion

The conclusion identifies alignment-to-task representation similarity as an overlooked factor in safety-guardrail erosion. It reports greater jailbreak susceptibility under high similarity, including with benign downstream data, while dissimilarity improves safety beyond existing downstream defenses.

6 Conclusion

  • Representation similarity between upstream alignment data and downstream fine-tuning tasks is identified as a previously overlooked factor in guardrail erosion.Source: §6 · p. 8
  • High-similarity datasets increase jailbreak susceptibility even with entirely benign downstream data.Source: §6 · p. 9
  • Dissimilarity fosters safety beyond the positive impact of existing downstream defense systems.The authors describe their analysis as a practical framework for safe model selection during fine-tuning and proactive alignment management.Source: §6 · p. 9

A Experimental Details

The appendix reports computing resources and experimental configurations, then describes representation-based ranking to select high- and low-similarity alignment subsets alongside random subsets.

A.1 Computing Resources

  • Experiments used two 8-GPU NVIDIA A800 nodes and one 8-GPU NVIDIA L40 node, running Ubuntu 22.04 LTS.The A800 nodes had up to 64 CPU cores and 1 TB memory each; the L40 node had up to 256 CPU cores and 1 TB memory.Source: §A.1 · p. 13

A.2 Experiments Configurations

  • AdamW was used; alignment ran three epochs at 2 × 10−5 with batch size 32, and harmful fine-tuning ran five epochs at 1 × 10−5 with batch size 20.Alignment took about ten hours on 8 GPUs; harmful fine-tuning took about three minutes.Source: §A.2 · p. 13
  • Benign fine-tuning used three epochs, a 2 × 10−5 learning rate, and batch size 64; Figure 2 experiments used five epochs, 5 × 10−5, and batch size 20.Source: §A.2 · p. 13

B High-Similarity and Low-Similarity Subset Selection

  • The authors ranked alignment samples by average cosine similarity to downstream-task representations, selecting the top or bottom 1K or 5K as high- or low-similarity subsets.A random subset was also sampled from all available data points.Source: B High-Similarity and Low-Similarity… · p. 13

C.1 Data Contamination Examination

The contamination examination uses MIN-K% PROB to assess whether fine-tuning subsets may have appeared in model training. The reported inclusion probabilities are low across clusters.

C.1 Data Contamination Examination

  • MIN-K% PROB examines possible training-data exposure by checking whether examples contain outlier words with low model probabilities.The authors use it to examine whether such exposure could factor into guardrail breaking.Source: §C.1 · p. 13
  • The fine-tuning subsets had low probabilities of being part of LLAMA-2-7B-CHAT training data.The caption reports low inclusion probabilities across each cluster.Source: §C.1 · p. 13, Figure S1 · p. 14
  • Figure S1 compares average membership-inference probabilities at 5%, 10%, and 20% thresholds across the Figure 2 fine-tuning datasets.Source: Figure S1 · p. 14

C.2 Results on GEMMA-2 2B/9B

On GEMMA-2-2B and GEMMA-2-9B, results support the finding that less-similar upstream alignment data yields more durable safety guardrails. Initial High-Sim and Low-Sim model scores are averaged across four target downstream datasets.

C.2 Results on GEMMA-2 2B/9B

  • The model’s safety guardrail is more durable and resistant when upstream safety alignment data is less similar to the downstream fine-tuning dataset, consistent with LLAMA-2-7B in Table 1 and LLAMA-2-13B in Table 2.Source: §C.2 · p. 14
  • High-Sim and Low-Sim initial-model scores are averages across four target downstream datasets.Source: §C.2 · p. 14
  • Table S1 reports utility and harmfulness before and after downstream fine-tuning on GEMMA-2-2B.Source: Table S1 · p. 14
  • Table S2 reports utility and harmfulness before and after downstream fine-tuning on GEMMA-2-9B.Source: Table S2 · p. 15

D High Similarity Cluster Data

This section presents selected examples from high-similarity cluster data used in experiments detailed in Figure 2. The examples are identified as coming from Alpaca and Dolly clusters.

D High Similarity Cluster Data

E Safety Alignment Data

The section includes a content warning and selected examples from the BeaverTails Safety Alignment Dataset.

E Safety Alignment Data

Loading 2506.05346v1…