PaperlayerView exact revision on arXiv

Source-linked AI summary

Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguards

Punya Syon Pandey, Samuel Simko, Kellin Pelrine, Zhijing Jin

arXiv:2505.16789v3cs.CLcs.AIcs.LG

TL;DR

Fine-tuning can erode LLM safeguards, but the dataset features associated with post-tuning adversarial vulnerability remain understudied. The paper examines dataset characteristics and model behavior, using mediation analysis to investigate their relationship with attack success. Vulnerability rises for some fine-tuning datasets while general benchmark performance remains stable, and the authors identify dataset-design implications within that scope.

  • Problem

    Which dataset features increase adversarial vulnerability after fine-tuning remains largely unexplored, limiting understanding of how to mitigate adversarial risks.

    Source: §1 · p. 1

  • Method

    The study analyzes dataset characteristics, fine-tuned model behavior, and adversarial outcomes, then uses causal mediation to examine links between dataset features and vulnerability.

    Source: §2 · p. 2, §5.3 · p. 9

  • Results

    Legal, cybersecurity, and harmful-data fine-tuning increased ASR versus the original LLM while general benchmarks stayed stable; prompt toxicity’s indirect effect was 0.82 (p < 0.01).

    Source: §4.1 · pp. 4–5, §5.3 · p. 9

  • Takeaways & Limitations

    The findings suggest filtering harmful dataset features when curating smaller fine-tuning datasets, including in contexts such as cybersecurity.

    Source: §6 · p. 9

    The analysis is constrained by a 6,000-GPU-hour compute budget.

    Source: A Limitations · p. 13

Abstract

from arXiv · show

As large language models (LLMs) gain popularity, their vulnerability to adversarial attacks emerges as a primary concern. While fine-tuning models on domain-specific datasets is often employed to improve model performance, it can inadvertently introduce vulnerabilities within the underlying model. In this work, we investigate Accidental Vulnerability, unexpected vulnerabilities arising from characteristics of fine-tuning data. We begin by identifying potential correlation factors such as linguistic features, semantic similarity, and toxicity across multiple experimental datasets. We then evaluate the adversarial robustness of these fine-tuned models, analyzing persona shifts and interpretability traits to understand how dataset factors contribute to attack success rates. Lastly, we explore causal relationships that offer new insights into adversarial defense strategies, highlighting the crucial role of dataset design in preserving model alignment. Our code is available at https://github.com/psyonp/accidental_vulnerability.

1 Introduction

Adversarial prompts threaten LLM safety, and fine-tuning can erode safeguards; which dataset features shape vulnerability after fine-tuning remains poorly understood. The paper investigates this question by analyzing dataset characteristics and fine-tuned model behavior.

1 Introduction

  • Adversarial jailbreaks can circumvent LLM safety measures, raising safety and alignment concerns as models enter public applications.The paper notes these risks grow as systems become more capable or autonomous.Source: §1 · p. 1
  • The study asks which domain-specific dataset features increase adversarial vulnerability after fine-tuning, an underexplored link relevant to mitigating adversarial risks.Prior work examined attack success on models fine-tuned with benign and harmful data, but few studies investigated specific dataset factors.Source: §1 · p. 1
  • The proposed workflow analyzes dataset statistics, model persona and representation shifts, and jailbreak performance, then uses causal mediation to examine dataset factors linked to attack success.The workflow also examines changes in LoRA matrices.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Prior work links adversarial risks to model- and prompt-level weaknesses and shows that fine-tuning can erode safeguards. This paper distinguishes itself by systematically quantifying dataset characteristics and relating them to adversarial robustness.

Adversarial Vulnerabilities

  • Earlier adversarial-robustness research largely targets model weaknesses and prompt interventions, offering limited insight into training-data contributions.Prior work also studies adversarial inputs that exploit surface cues and representational behavior.Source: §2 · p. 2

Impact of Fine-Tuning on Robustness

  • Fine-tuning can increase adversarial susceptibility, including after training on seemingly benign data and narrow insecure-code datasets associated with harmful persona shifts.Prior examples include safety degradation through representation and gradient matching and harmful responses to benign prompts.Source: §2 · p. 2

Dataset Influence on Alignment

  • This work systematically quantifies dataset characteristics and links them causally to adversarial robustness, extending descriptive and annotation-focused dataset research.The authors frame this as a bridge between data-driven and model-centric perspectives.Source: §2 · p. 2

3 Experiment Design

The experiments fine-tune one instruction model on six varied datasets, evaluate adversarial robustness and general capabilities, and track fine-tuning, general-performance, and attack-success metrics.

3 Experiment Design

  • The experiment tracks fine-tuning performance, general performance, and attack success rate (ASR).The design includes dataset selection, fine-tuning, and adversarial and performance evaluations.Source: §3 · p. 3

3.1 Fine-Tuning Setup

  • Six question-answering datasets span benign, harmful, and domain-specific settings, including legal, cybersecurity, and electrical-engineering content.Alpaca serves as the benign baseline; LLM-LAT and Gray-Swan provide harmful-data examples.Source: §3.1 · p. 3

3.2 Adversarial Evaluation

  • Adversarial evaluation uses HarmBench techniques, including gradient-guided and prompt-generation attacks, and classifies harmful prompts into five categories to calculate ASR.The listed categories are crime, drugs or harmful chemicals, copyright, cybercrime, and manipulation.Source: §3.2 · pp. 3–4

3.3 Performance Evaluation

  • MMLU, HellaSwag, Arc Easy, and GSM8K check whether fine-tuned models retain general reasoning capabilities rather than merely showing forgetting or utility loss.These benchmarks serve as a sanity check alongside the adversarial evaluation.Source: §3.3 · p. 4

4 Measuring Adversarial Vulnerability After Fine-Tuning

The section evaluates adversarial vulnerability after fine-tuning, alongside general performance, persona behavior, training dynamics, and LoRA shifts. Domain-specific fine-tuning is associated with increased attack vulnerability, while the analyses also examine where that vulnerability varies and how model representations change.

4.1 Adversarial and Performance Results

  • Fine-tuning on legal, cybersecurity, and harmful datasets increases vulnerability versus the original model; Table 4 also reports increased vulnerability for engineering and cybersecurity fine-tunes.ASRs vary substantially across prompt subcategories.Source: §4.1 · p. 4, Table 4 · p. 4, Figure 2 · p. 5
  • SFT shifts overall ASRs on harmful prompts from HarmBench and JailbreakV-28k, with dataset factors playing a larger role in robustness than attack-method efficacy.The comparison isolates SFT-related shifts and is shown in Figure 3.Source: §4.1 · p. 5, Figure 3 · p. 5
  • Fine-tuned models largely retain general-domain capabilities, with stable benchmark performance comparable to the original model.Table 5 summarizes the comparable performance.Source: §4.1 · p. 5, Table 5 · p. 5

4.2 Persona Analysis

  • Evaluations cover emergent toxicity, honesty, gender bias, harmful recall, and emotional reasoning; Figure 4 reports minimal amplification of negative behaviors in persona shifts.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3 Vulnerability and Training Dynamics

  • Embedding-attack ASRs fluctuate across 50-step checkpoints, while consecutive cosine representation drift declines across checkpoints, indicating stabilization.The ASR analysis uses the HarmBench binary classifier; higher cosine drift values reflect greater representational change.Source: §4.3 · pp. 6–7, Figure 5 · p. 6, Figure 6 · p. 6

4.4 LoRA Adapter Shifts

  • Across six fine-tuned models, LoRA checkpoint weights follow distinct, smooth domain-specific trajectories in a t-SNE visualization of PCA-reduced updates.The analysis also tracks Frobenius norms of Rank A and Rank B matrices across checkpoints.Source: §4.4 · p. 7, Figure 7 · p. 7

5 Causal Explanation of Accidental Vulnerability via Dataset Features

The section measures dataset features, relates them to adversarial success rates, and tests whether representational drift mediates those relationships. Its mediation analysis reports distinct effects for prompt toxicity, sentiment and lexical diversity, and response toxicity and diversity.

5.1 Feature Selection

  • The feature set spans prompt-output cosine similarity and Euclidean distance, readability, length, lexical diversity, and prompt and response sentiment and toxicity.The study uses these features to examine dataset properties and model safeguards.Source: §5 · p. 7, §5.1 · p. 8

5.2 Correlation Analysis

  • Spearman rank correlations relate dataset features to mean ASR, and statistically significant features are carried into causal analysis; figures show correlation significance and top-six feature distributions.Table 6 reports the top six statistically significant metrics correlated with mean ASR.Source: §5.2 · p. 8, Table 6 · p. 9

5.3 Causal Mediation Analysis

  • The mediation model treats a dataset feature as the treatment, consecutive cosine drift between hidden representations as mediator, and intermediate ASR after fine-tuning as outcome.Direct, indirect, and total effects are estimated for each feature.Source: §5.3 · p. 9, Figure 9 · p. 9
  • Prompt toxicity had an indirect effect of 0.82 (p < 0.01); prompt sentiment and TTR had mediated proportions of 0.99 and 1.05, response toxicity had a direct effect of 0.84, and response TTR had a mediated effect of −0.89 (p < 0.05).The authors interpret prompt toxicity as amplifying representational drift and vulnerability, and response lexical diversity as potentially enhancing robustness.Source: Table 7 · p. 9, §5.3 · p. 9

6 Conclusion

The conclusion frames Accidental Vulnerability as a risk arising from fine-tuning dataset properties and highlights dataset design as relevant to preserving safeguards.

6 Conclusion

  • The paper introduces Accidental Vulnerability: fine-tuning dataset properties may contribute to vulnerabilities in models, alongside the nature of adversarial attacks.Across domain-specific datasets, it identifies prompt length, sentiment, and lexical diversity as features that influence safeguards.Source: §6 · p. 9

Ethics Statement

The ethics statement describes the data and safeguards considered in conducting the experiments.

Ethics Statement

  • The datasets are mostly public, contain no personally identifiable or sensitive information, and were used under their terms; the authors considered potential social and ethical implications.The paper encourages constructive, safety-aligned development based on its results.Source: Ethics Statement · p. 10

A Limitations

The appendix states that compute constraints bound the analysis and documents implementation, feature-correlation, dataset-metric, and subset-attack materials.

A Limitations

  • Compute constraints limited the analysis to 6,000 GPU hours; future work could assess whether dataset pre-screening predicts vulnerability.The authors also propose integrating their factors to guide dataset design.Source: A Limitations · p. 13

B.1 Implementation Details

  • Adversarial experiments use Best-of-N jailbreak attacks with five attempts per attack, and the authors report consistent ASRs across runs.They say the results are not overly sensitive to sampling hyperparameters.Source: §B.1 · p. 13

B.2 Feature Correlation and Distribution

  • The appendix presents within-dataset feature correlations, between-dataset variability, and distributions of the top six statistically significant correlated features.Figure 10 marks significant metric-ASR correlations with asterisks; Figure 11 shows top-feature distributions and variations.Source: §B.2 · p. 13, Figure 10 · p. 13, Figure 11 · p. 14

B.3 Dataset Metric Evaluations

  • Feature tables report mean, standard deviation, minimum, and maximum for examined metrics, with means used in correlation and mediation analyses.The provided captions identify metric-calculation tables for the Benign, Engineering, Legal, LAT-Harmful, and CB-Harmful datasets.Source: §B.3 · p. 14, Table 8 · p. 14, Table 9 · p. 14

B.4 Subset Attack Success Rate Tables

  • Appendix tables report subset attack success rates for GCG, AutoPrompt, and PEZ across datasets.The supplied captions identify the attack methods and cross-dataset scope, but do not provide cell values.Source: Table 14 · p. 16, Table 15 · p. 16, Table 16 · p. 16

B.5 Cross-Model Evaluation

The supplementary analyses assess cross-model generalizability, direct-prompt attack success, and persona-related benchmarks. They describe the evaluation scope and measures but provide no numerical outcomes in the supplied passages.

B.5 Cross-Model Evaluation

  • Similar ASR increases across domain-specific datasets suggest the observed trends may extend beyond a single model architecture.Additional models were evaluated using the PEZ attack method.Source: Table 17 · p. 16, §B.5 · p. 16

B.6 Direct-Prompt SFT ASRs

  • Direct-prompt ASRs were reported on all 400 HarmBench cases and a 280-case JailbreakV-28k test split.Table 18 presents the ASRs, but the supplied passage does not state their values.Source: §B.6 · p. 17, Table 18 · p. 17

C.1 Persona Analysis: Benchmark Descriptions

  • Persona analysis used benchmarks for benign toxicity, truthfulness, gender bias, emotional intelligence, and harmful-information propensity.The benchmarks were run with lm-eval for standardized evaluation across examined models.Source: §C.1 · p. 17
  • WMDP evaluates models’ propensity to produce harmful or dangerous information in response to harmful prompts.Table 19 reports performance on persona-related bias benchmarks across original and fine-tuned models.Source: §C.1 · p. 17, Table 19 · p. 17

C.2 Checkpoint-Descent: Implementation Details

The implementation uses a soft prompt embedding optimization attack with cross-entropy loss and early stopping. It reports attack success across training checkpoints alongside the attack hyperparameters.

C.2 Checkpoint-Descent: Implementation Details

  • The soft prompt embedding attack optimizes cross-entropy between the generated response and a fixed reference, stopping early when the loss meets a convergence threshold.Attacks run for up to 1000 steps; Table 20 lists the hyperparameters.Source: §C.2 · pp. 17–18, Table 20 · p. 18
  • Embedding attack success rates are reported at 50-step training checkpoints for the fine-tuned models shown in the visualizations.Table 21 presents intermediate ASRs across checkpoints.Source: §C.2 · p. 18, Table 21 · p. 18

C.3 Interpretability Analysis: Supplementary Information

Supplementary analyses quantify representation drift, LoRA changes, and embedding-attack behavior across fine-tuning checkpoints and datasets. The appendix also documents the categories used to classify HarmBench prompts.

C.3 Interpretability Analysis: Supplementary Information

  • Consecutive cosine hidden-representation drift and LoRA Frobenius norms were measured across checkpoints and dataset categories for the causal mediation analysis.Tables 22 and 23 report these quantities; the drift measures activation changes between fine-tuning steps.Source: §C.3 · p. 18, Table 22 · p. 18, Table 23 · p. 19

C.4 Embedding Loss-Iteration Metrics

  • Embedding-attack loss, AUC, and convergence-step metrics are reported across checkpoints for Benign, Engineering, and Cybersecurity datasets.The reported metrics include mean final loss, AUC (%), and convergence steps (mean ± std) for Benign.Source: §C.4 · p. 19, Table 24 · p. 19, Table 25 · p. 19
  • Across multiple finetuning checkpoints, embedding-attack metrics are reported for Legal, Lat-Harmful, and CB-Harmful datasets, with final loss, AUC (%), and convergence steps (mean ± std) in Tables 27–29.Source: §C.4 · p. 19, Table 27 · p. 20, Table 28 · p. 20

D HarmBench Prompt Classification

Loading 2505.16789v3…