PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety-Aware Fine-Tuning of Large Language Models

Hyeong Kyu Choi, Xuefeng Du, Yixuan Li

arXiv:2410.10014v1cs.CLcs.AI

TL;DR

Fine-tuning data may mix benign and harmful samples, while identifying harmful samples manually is labor-intensive and subjective. SAFT uses subspace-based representation scores to filter potentially harmful samples before training, with reported harmfulness reductions of up to 27.8% versus standard SFT.

  • Problem

    Fine-tuning datasets can contain harmful samples, whose membership may be unclear and whose manual filtering is labor-intensive and subjective.

    Source: §1 · p. 1

  • Method

    SAFT filters potentially harmful samples before fine-tuning using a score based on representations aligned with a harmful-sample subspace.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    At λ = 0.25, SAFT reduced harmfulness score from 39.9 (SFT) to 28.8, a relative 27.8% decrease.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    SAFT’s filtering consistently decreases harmfulness across the reported experimental settings, while randomly removing the same number of samples does not yield the same level of reduction.

    Source: §5.2 · p. 6

    The training samples have no clear benign-or-harmful membership.

    Source: §2 · p. 2

Abstract

from arXiv · show

Fine-tuning Large Language Models (LLMs) has emerged as a common practice for tailoring models to individual needs and preferences. The choice of datasets for fine-tuning can be diverse, introducing safety concerns regarding the potential inclusion of harmful data samples. Manually filtering or avoiding such samples, however, can be labor-intensive and subjective. To address these difficulties, we propose a novel Safety-Aware Fine-Tuning (SAFT) framework designed to automatically detect and remove potentially harmful data, by leveraging a scoring function that exploits the subspace information of harmful and benign samples. Experimental results demonstrate the efficacy of SAFT across different LLMs and varying contamination rates, achieving reductions in harmfulness of up to 27.8%. Going beyond, we delve into the mechanism of our approach and validate its versatility in addressing practical challenges in real-world scenarios.

1 Introduction

Fine-tuning data can mix benign and harmful samples, whose manual identification is labor-intensive and subjective. SAFT uses language-model representations to filter harmful samples before training, and experiments report lower harmfulness than standard SFT.

1 Introduction

  • Fine-tuning data may mix benign and harmful samples, making harmful membership unclear and manual filtering labor-intensive and subjective.Harmful content in fine-tuning data can adversely affect model behavior.Source: §1 · p. 1
  • SAFT filters potentially harmful training samples using a score based on how strongly their representations align with a harmful-sample subspace.The score uses projections onto top singular vectors in the embedding space.Source: §1 · p. 2
  • SAFT reduces harmfulness by up to 27.8% compared with standard supervised fine-tuning.The framework was evaluated across different LLMs, datasets, and contamination levels.Source: §1 · p. 2
  • The paper evaluates SAFT across models, datasets, and contamination levels, and examines its components and practical challenges.Its stated contributions include a subspace-based scoring function and analyses of efficacy and robustness.Source: §1 · p. 2

2 Preliminaries

The paper models fine-tuning data as a mixture of benign and harmful distributions, with samples whose membership is unknown. It contrasts ordinary SFT on the full dataset with SAFT on a filtered subset, in an unlabeled setting distinct from preference-labeled alignment approaches.

2 Preliminaries

  • Fine-tuning data is modeled as a mixture of benign and harmful distributions, with λ denoting the contamination ratio.The formulation includes the ideal case λ = 0, when no harmful data occurs.Source: §2 · p. 2
  • The training set is sampled from this mixture, but its samples have no clear benign-or-harmful membership.Each sample contains an input prompt and a corresponding target.Source: §2 · p. 2
  • Traditional SFT minimizes language-modeling loss on the full dataset, but harmful samples can adversely affect the resulting model’s behavior.SAFT instead filters potentially harmful samples and optimizes on the filtered set to reduce harmfulness.Source: §2 · p. 3
  • Unlike RLHF, which typically requires labeled preference samples, this setting uses an unlabeled mixture and imposes weaker data assumptions.The paper motivates the setting as one that naturally arises in real-world applications.Source: §2 · p. 3

3 How Does Harmful Data Impact Fine-tuning?

Experiments with contaminated Beavertails data show that harmfulness rises as the harmful-data ratio increases, even when helpfulness remains similar to the benign-only setting. This means task helpfulness alone may not reveal safety degradation.

3 How Does Harmful Data Impact Fine-tuning?

  • Fine-tuning on mixed data notably increases harmfulness even with λ = 0.1, and degradation worsens as harmful-data proportion increases, while helpfulness is not significantly affected regardless of harmful ratio.At λ = 0.3, helpfulness under BLEURT is 0.504, close to 0.511 for fine-tuning on pure benign samples.Source: §3 · p. 3
  • At λ = 0.3, helpfulness under BLEURT is 0.504, close to 0.511 for fine-tuning on pure benign samples.The passage reports that helpfulness is not significantly affected across harmful-data ratios.Source: §3 · p. 3
  • Because helpfulness can remain similar while harmfulness rises, evaluating only the main task may leave safety effects unnoticed.The setup uses Beavertails question-answer pairs labeled harmful or benign and evaluates harmfulness and helpfulness separately.Source: §3 · p. 3

4 Safety-Aware Fine-Tuning

SAFT aims to reduce harmfulness while preserving general helpfulness by filtering potentially harmful samples before fine-tuning. Its detector scores sample embeddings using directions associated with harmful data, and the filtered dataset is then used for training.

4 Safety-Aware Fine-Tuning

  • SAFT targets lower harmfulness without severely degrading helpfulness by filtering harmful samples from mixed fine-tuning data.The framework trains on the remaining filtered set to mitigate harmful data’s impact.Source: §4 · pp. 3–4

4.1 Harmful Data Detection

  • SAFT centers sample embeddings and uses singular-vector projections to score samples, with larger scores indicating harmful data and scores above τ flagged for removal.The score can extend from the first singular vector to a subspace of k singular vectors.Source: §4.1 · pp. 4–5

4.2 Fine-Tuning with Filtered Data

  • Fine-tuning uses the samples whose filtering scores satisfy s_i ≤ τ as the training set.The model parameterization θ is used in the fine-tuning objective.Source: §4.2 · p. 5

5 Experiments

Across the tested settings, SAFT reduces harmfulness relative to unfiltered fine-tuning and the baselines while helpfulness and output-quality scores do not deviate significantly. Its performance is close to oracle filtering at λ = 0.3.

5.1 Setup

  • Experiments use 3,000 Beavertails samples across five contamination ratios, with Llama-2-7B and Vicuna-7B; comparisons include SFT, prompting, and random removal.The prompting baseline is not reported for Vicuna because it filters out most samples, making fine-tuning meaningless.Source: §5.1 · p. 5, Table 1 · p. 6

5.2 Main Results

  • 27.8% relative decrease: at λ = 0.25, SAFT lowers the Harmfulness Score from 39.9 (SFT) to 28.8.Across experimental settings, harmfulness decreases relative to SFT and the baselines; randomly removing the same number of samples does not yield the same reduction.Source: §5.2 · p. 6, Table 1 · p. 6
  • At λ = 0.3, SAFT achieves performance closely resembling Oracle filtering on harmfulness and helpfulness metrics.Oracle filters using ground-truth benign labels, yet its harmfulness score remains above zero.Source: §5.2 · p. 6, Table 2 · p. 6

6 Analyses

The analyses test SAFT’s harmful-sample detection, robustness to practical settings, and category-level effects. They report stronger detection than baselines, benefits on a new dataset and under adjustable filtering, and reduced harmfulness across categories.

6.1 Soundness of SAFT

  • SAFT consistently outperforms harmful-data detection baselines across contamination ratios, while prompting performs worse than random filtering.Figure 3 compares AUROC across baselines and contamination rates.Source: Figure 3 · p. 7, §6.1 · p. 7
  • With k = 1, Llama-2 achieves its best classification performance, an AUROC of 0.6868; smaller k values generally perform better across contamination scenarios.Table 3 reports the component ablation at λ = 0.3; Figure 4 plots SAFT AUROC across Llama-2 layers at λ = 0.3.Source: §6.1 · p. 7, Table 3 · p. 7, Figure 4 · p. 7

6.2 Robustness to Practical Challenges

  • On Anthropic HH-RLHF, SAFT records a Harmfulness Score of 14.79, compared with 18.70 for SFT.The experiment uses 3,000 dialogues with an unknown harmful-sample ratio and evaluates using Beavertails validation and test data.Source: §6.2 · p. 8, Table 4 · p. 8
  • Adjusting the threshold trades filtering precision against recall; a lower steer rate can reduce harmfulness to 26.31 while retaining above 0.50 BLEURT and 0.14 ROUGE-L.The reported reduction discards more than half of the data; Figure 5 shows trends for different steer rates.Source: §6.2 · pp. 8–9, Figure 5 · p. 8

6.3 Qualitative Exploration

  • The qualitative exploration presents Llama-2 responses after fine-tuning with and without filtering, with further examples in Appendix D.Table 5 contains the examples.Source: Table 5 · p. 8, §6.3 · p. 9
  • SAFT reduces harmfulness in all 14 evaluated categories; the largest reductions are approximately 48%–49% for controversial topics/politics and terrorism/organized crime.Privacy Violation has the smallest reduction; Figure 6 presents category-wise harmfulness scores.Source: §6.3 · p. 9, Figure 6 · p. 9

7 Related Works

Related work covers LLM representation probing, knowledge editing and unlearning, toxicity detection, and supervised fine-tuning. SAFT differs by detecting and removing harmful samples before fine-tuning to address harmful data’s influence.

Related work

8 Conclusion

The conclusion presents SAFT as an automated way to filter potentially harmful fine-tuning samples using a lower-dimensional subspace representation. Experiments across contamination rates and LLM families support its effectiveness and practical flexibility.

Conclusion

  • SAFT automatically detects and filters potentially harmful fine-tuning samples using the dataset’s lower-dimensional subspace representation.The conclusion frames this as an alternative to manual curation, which it describes as resource-intensive and subjective.Source: §8 · p. 9
  • Experiments across diverse contamination rates and LLM families support SAFT’s effectiveness, simplicity, and flexibility in practical scenarios.Source: §8 · p. 9

A Hyperparameters and Experimental Details

The appendix reports fine-tuning and generation settings, along with the embedding extraction and threshold-selection procedures.

Hyperparameters

  • Fine-tuning uses LoRA rank 8 with α = 32 for four epochs at learning rate 2e-5; SAFT validates k from {1, 2, 3, 4}.Generation uses greedy decoding with a maximum of 256 tokens.Source: A Hyperparameters and Experimental… · p. 13

Implementation Details

B Kernel Density Estimation

The KDE plots compare harmful-score distributions across contamination rates and, at λ = 0.3, across component counts. Separability decreases as more components are considered, suggesting harmful samples are represented by a small subspace.

B Kernel Density Estimation

  • As the number of components increases, harmful-score separability decreases at contamination rate λ = 0.3.The plots suggest harmful samples are represented by a small subspace, with only a few components required.Source: B Kernel Density Estimation · p. 13
  • The top-row KDE plots show harmful-score distributions for different contamination ratios λ.Source: B Kernel Density Estimation · p. 13
  • The bottom-row KDE plots compare distributions across component counts k under contamination rate λ = 0.3.Source: B Kernel Density Estimation · p. 13

C Broader Impact and Limitations

SAFT is presented as a potentially useful approach for safer fine-tuning in industry settings. Its stated scope is SFT; adaptation to methods such as RLHF and DPO remains unclear.

C Broader Impact and Limitations

  • SAFT is envisioned for industry use, including integration into Fine-Tuning-as-a-Service platforms to filter potentially harmful user job queries.The authors frame this as a potential application of the method.Source: C Broader Impact and Limitations · p. 13
  • SAFT primarily targets harmful-sample removal in Supervised Fine-Tuning, leaving other fine-tuning methods outside its established scope.Source: C Broader Impact and Limitations · p. 14
  • Whether SAFT can support RLHF or DPO, which require negative samples, remains unclear; ranking preference pairs is suggested as a possible adaptation.Source: C Broader Impact and Limitations · p. 14

D More Qualitative Examples

The qualitative examples contrast unfiltered fine-tuning responses with responses after SAFT, including examples involving harmful requests. Some examples also show non-harmful prank suggestions.

D More Qualitative Examples

Loading 2410.10014v1…