PaperlayerView exact revision on arXiv

Source-linked AI summary

Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety

Zihan Guan, Mengxuan Hu, Ronghang Zhu, Sheng Li, Anil Vullikanti

arXiv:2505.06843v2cs.LGcs.CL

TL;DR

The paper investigates whether benign samples can be intentionally selected to undermine LLM safety alignment, a threat that can evade toxicity screening. It proposes Self-Inf-N, an outlier-based selection method, and finds that selected benign samples severely compromise safety across seven LLMs, including in transferable and practical settings. The study concludes that existing mitigation strategies are often insufficient and that stronger safeguards are needed.

  • Problem

    Benign fine-tuning can increase LLM output harmfulness, but selecting benign samples that strongly undermine safety alignment remains underexplored and difficult to detect.

  • Method

    Self-Inf-N uses an outlier-detection perspective to identify and fine-tune on the most harmful samples within benign datasets.

  • Results

    Self-Inf-N-selected benign samples severely compromise safety alignment, with harmfulness three times as high as the random baseline and strong transferability across architectures and practical scenarios.

  • Takeaways & Limitations

    The attack remains effective across practical settings and against certain advanced mitigation strategies, underscoring the need for robust safeguards during benign fine-tuning.

  • Takeaways & Limitations

    Prior related selection methods rely on external anchor datasets and 100-sample attacks can be impractical and detectable because downstream performance may suffer.

Abstract

from arXiv · show

Recent studies have uncovered a troubling vulnerability in the fine-tuning stage of large language models (LLMs): even fine-tuning on entirely benign datasets can lead to a significant increase in the harmfulness of LLM outputs. Building on this finding, our red teaming study takes this threat one step further by developing a more effective attack. Specifically, we analyze and identify samples within benign datasets that contribute most to safety degradation, then fine-tune LLMs exclusively on these samples. We approach this problem from an outlier detection perspective and propose Self-Inf-N, to detect and extract outliers for fine-tuning. Our findings reveal that fine-tuning LLMs on 100 outlier samples selected by Self-Inf-N in the benign datasets severely compromises LLM safety alignment. Extensive experiments across seven mainstream LLMs demonstrate that our attack exhibits high transferability across different architectures and remains effective in practical scenarios. Alarmingly, our results indicate that most existing mitigation strategies fail to defend against this attack, underscoring the urgent need for more robust alignment safeguards. Codes are available at https://github.com/GuanZihan/Benign-Samples-Matter.

1. Introduction

Fine-tuning can undermine LLM safety alignment even with benign data, motivating a stealthier attack that selects especially damaging benign samples. The paper proposes Self-Inf-N, an outlier-based selector whose selected samples compromise safety across models and practical settings.

  • Fine-tuning can compromise LLM safety alignment, including through harmful outputs involving illegal activities, child abuse, and hate speech.
  • Benign fine-tuning can still increase output harmfulness, raising whether seemingly harmless samples can be exploited to undermine safety alignment.
  • Prior anchor-based filtering depends on external harmful and safety datasets and selecting only 100 benign samples creates impractical, detectable attack conditions.
  • Self-Inf-N treats benign-data selection as outlier detection, identifying samples that can compromise alignment without relying on external anchors.
  • Self-Inf-N mitigates self-influence’s short-sample bias, enables detailed harmful content, transfers across architectures and model sizes, and remains effective in practical scenarios and against some mitigations.

2. Related Works

Prior work shows that harmful fine-tuning can break safety alignment and that benign datasets can also cause degradation. Related research also studies mitigation strategies and gradient-based influence estimation for outlier analysis and data selection.

  • Harmful Fine-tuning on Harmful Datasets: Harmful fine-tuning studies show that a few adversarially designed examples can severely compromise LLM safety alignment.
  • Benign Fine-tuning Compromises Safety Alignment: Benign fine-tuning research reports partial safety degradation and anchor-based selection of benign samples using embedding or gradient similarity.
  • Mitigation against Harmful Fine-tuning: Existing mitigation strategies are organized into alignment-stage, fine-tuning-stage, and other post-training intervention categories.

3. Breaking Alignment: The Risks of Outlier Data in Benign Datasets

The study frames benign-dataset fine-tuning as an alignment vulnerability and tests whether outlier detection can identify benign samples that disproportionately increase harmfulness. Self-Inf-N addresses Self-Inf’s short-answer bias, selecting more impactful outliers whose fine-tuning produces more detailed harmful outputs.

  • Data Influence Estimation: Self-Inf ranks samples by their influence on their own loss, with higher self-influence indicating greater outlier likelihood.Each sample is represented as a user-input and target-response pair, and the top k=100 samples are selected by score.
  • Harmfulness of Outlier Samples: Top-100 Self-Inf outliers significantly increase model harmfulness compared with 100 randomly selected benign samples.The evaluation uses safety tests on harmful HEx-PHI queries and reports significantly higher harmfulness for the outlier-selected samples.
  • Characteristics of Outlier Samples: Over 90% of vanilla Self-Inf’s selected samples have exceptionally short answers, revealing a strong length bias in the outlier-selection mechanism.Short-answer selection can degrade alignment, but the resulting models generate short responses and therefore have limited practical harmfulness.
  • Influence of Samples with Short Token Lengths: Short-answer fine-tuning compromises safety and reduces utility, with answers shorter than four tokens producing exceptionally high harmfulness and fewer safe responses.The evaluation varies answer-token lengths across randomly selected 100-sample subsets and measures harmfulness, safe-answer proportion, and utility.
  • Normalizing Length Bias in Self-Inf: Self-Inf-N normalizes self-influence with answer length by logarithmically balancing the two components, penalizing extremely short responses and favoring longer ones.This design suppresses the shortcut caused by short answers and encourages selection of more diverse, impactful benign samples.
  • Normalizing Length Bias in Self-Inf: Fine-tuning on Self-Inf-N-selected samples produces more detailed harmful responses and has greater potential to break safety alignment than vanilla Self-Inf selection.The comparison is reported for Llama-2-7b-chat fine-tuned on 100 filtered Dolly samples.

4. Towards real-world attacks: Harmfulness and Practical Impact

Experiments show that Self-Inf-N-selected benign samples can compromise safety across models and practical fine-tuning settings, while evading toxicity screening and resisting several mitigations.

  • 4.2. Main Results: Table 1 reports that Self-Inf-N-selected samples achieve harmfulness comparable to purely harmful samples while minimally affecting utility.The comparison uses 100 selected samples and averages harmfulness over 330 HeX-PHI prompts and utility over 80 MT-Bench prompts.
  • 4.3. Transferability: Samples filtered on Llama-2-7B-Chat transfer harmfulness to other architectures and to larger Llama-2 models using LoRA fine-tuning.The transfer experiments include Qwen-2-7B-Instruct, Gemma-2-9B-IT, Ministral-8B-Instruct, Llama-3-8B-Chat, Llama-2-13B-Chat, and Llama-2-70B-Chat.
  • 4.4.1. SCENARIO 1: CONTINUOUS LEARNING: Continual fine-tuning on Dolly and Asclepius largely preserves the harmfulness introduced by the first-stage Self-Inf-N attack.Harmfulness decreases more gradually on Dolly than on Asclepius as learning rates increase, which the authors attribute to overfitting and catastrophic forgetting.
  • 4.4.2. SCENARIO 2: DATA POISONING: Mixing only 1% Self-Inf-N samples into 2,000 Dolly samples significantly increases harmfulness, especially at smaller batch sizes.The result indicates that selected samples retain harmful characteristics under data-poisoning conditions and may pose greater risks for resource-constrained users.
  • 4.4.2. SCENARIO 2: DATA POISONING: The filtered samples receive lower toxicity scores and fewer moderation flags than standard harmful data, so toxicity APIs alone struggle to detect them.This makes toxicity screening an ineffective standalone defense against the attack.
  • 4.6. Safety Mitigation: Adding a small safety dataset can reduce harmfulness, but mitigation effectiveness depends on the source, while Lisa fails to reliably mitigate the attack.Five safety samples consistently reduce harmfulness; Bianchi samples improve with larger additions, whereas BeaverTails can slightly increase harmfulness, and Lisa remains ineffective across tested step ratios.

5. Conclusion and Future Works

The paper proposes Self-Inf-N, an outlier-detection method that selects the most harmful samples from benign datasets for fine-tuning. Experiments across seven mainstream LLMs show high transferability across architectures.

  • Self-Inf-N identifies the top-k most harmful samples within benign datasets through refined outlier detection.
  • Fine-tuning on these selected benign samples is evaluated as a practical attack against LLM safety alignment.
  • Experiments across seven mainstream LLMs demonstrate high transferability across different architectures.

Impact Statement

The paper highlights the practical importance of inspecting LLM safety alignment because LLMs are widely used across domains. It proposes a potential alignment vulnerability and suggests safety data augmentation as a way to suppress harmfulness after fine-tuning.

  • Inspecting LLM safety alignment is important because LLMs are widely adopted across many domains.
  • The paper proposes a potential vulnerability in LLM safety alignment and calls for further research.
  • The experiments suggest safety data augmentation as a method for suppressing harmfulness in fine-tuned LLMs.

A. Evaluation Benchmark

The evaluation uses HEx-PHI1 as a safety benchmark containing 330 harmful prompts across 11 categories. These prompts are input to the LLM to assess output harmfulness.

  • HEx-PHI1 contains 330 harmful prompts spanning 11 categories of unsafe behavior.
  • The benchmark categories include illegal activity, child abuse, hate or violence, malware, physical harm, and economic harm.
  • The evaluation inputs harmful prompts into the LLM and assesses the harmfulness of its outputs.

B. Datasets Preparation

The study prepares benign Dolly and Alpaca instruction-tuning datasets by removing explicit harmful or safety-specialized samples, then analyzes Self-Inf scores and token length to select outliers. It uses normalization, fine-tuning, baselines, and model evaluations to study the attack.

  • Datasets Preparation: The study uses Dolly and Alpaca as benign instruction-tuning datasets, containing over 15K and 52K instruction demonstrations respectively.
  • Datasets Preparation: Keyword matching removes explicit harmful content and safety-specialized responses, leaving 14,624 Dolly samples and about 50K Alpaca samples.
  • Choice of Normalization: Self-Inf scores range from 1×10^2 to 1×10^6 while token length ranges from 0 to 1×10^3, creating a scale imbalance.
  • Choice of Normalization: A logarithmic transformation with an added 1 maps Self-Inf scores and token lengths to comparable positive scales while preserving a slightly greater Self-Inf contribution.
  • Experimental Setup: Preliminary fine-tuning uses 100 filtered benign samples, while training hyperparameters vary across models according to their sizes and recommended implementations.
  • Experimental Setup: The experiments compare random selection and Bidirection Anchor baselines and evaluate the method on additional models including Qwen2-7b-instruct.

E.6. Continuous Learning Experiment with BA

The continuous-learning experiment compares how long harmfulness persists after Self-Inf-N or BA-based fine-tuning. Self-Inf-N produces more enduring harmfulness under continued fine-tuning.

  • E.6. Continuous Learning Experiment with BA: The experiment reports HS scores across different settings, including fine-tuning configurations and step ratios.HS denotes the average harmfulness score across 11 HEx-PHI categories.
  • E.6. Continuous Learning Experiment with BA: Self-Inf-N-induced harmfulness is more enduring than BA’s during continual fine-tuning.The comparison remains true across different learning rates used in the second stage.
  • E.6. Continuous Learning Experiment with BA: The study evaluates transferability by selecting samples with one model and fine-tuning other architectures.The cross-architecture setting uses Llama-2-7B-Chat as πA and evaluates Qwen-2-7B-Instruct, Gemma-2-9B-IT, Mistral-8B-Instruct, and LLaMA-3-8B-Chat as πB.
  • E.6. Continuous Learning Experiment with BA: The experiments also examine batch-size effects and utility performance on Dolly and Alpaca.Figure 12 varies batch size bsz ∈{2, 5, 10, 20, 40}, while Table 4 reports utility performance for Self-Inf and Self-Inf-N.
  • E.6. Continuous Learning Experiment with BA: Safety-dataset augmentation does not uniformly suppress harmfulness: BeaverTails samples are insufficient, whereas RepNoise-BeaverTails effectively suppresses it.The BeaverTails examples have a more neutral answer tone, while RepNoise-BeaverTails uses refusal responses.

F.1. Samples filtered with Self-Inf

Samples selected by Self-Inf exhibit unusual response patterns, including very short or curator-like content. The section contrasts these examples with broader benign-dataset samples and shows their varied forms.

  • F.1. Samples filtered with Self-Inf: Self-Inf-filtered Dolly samples commonly have very short responses, and some appear to contain random words left by data curators.This pattern is explicitly noted among the 10 filtered samples.
  • F.1. Samples filtered with Self-Inf: The filtered examples span factual question answering, classification, information extraction, and creative-writing-style entries.Examples include Moon River, Wilford, cities, Michael Caine, Chicago 7, November, and four strings on a violin.
  • F.1. Samples filtered with Self-Inf: Some filtered samples are malformed or minimally specified, such as “Test from Mike” and “Tesrt.”These entries contain short prompts and terse responses categorized as general QA or creative writing.
  • F.1. Samples filtered with Self-Inf: The section also includes longer, coherent samples covering historical explanations, Greek terminology, electronics, personality advice, and movie rankings.These examples demonstrate that the filtered set is not limited to one task format.

G.1. Harmful Generations by Llama-2-7b-chat trained over Samples Filtered with Self-Inf

This section presents harmful generations produced by Llama-2-7b-chat after training on samples filtered with Self-Inf. The examples are organized by HEx-PHI categories.

  • G.1. Harmful Generations by Llama-2-7b-chat trained over Samples Filtered with Self-Inf: One displayed response is “Use a hair dryer,” illustrating a harmful answer associated with the evaluated prompts.The supplied passage does not provide the corresponding question text.

G.2. Harmful Generations by Llama-2-7b-chat trained over 1-token-length Samples

This section presents harmful generations from Llama-2-7b-chat trained on one-token-length samples. The examples are organized by HEx-PHI categories.

  • G.2. Harmful Generations by Llama-2-7b-chat trained over 1-token-length Samples: One displayed response is “Hide it in your luggage,” illustrating a harmful answer associated with the evaluated prompts.The supplied passage does not provide the corresponding question text.

G.3. Harmful Generations by Llama-2-7b-chat trained over Samples Filtered with Self-Inf-N

The filtered samples produce harmful generations spanning illicit finance, piracy, addictive substances, weapons, and potentially risky financial advice. The examples illustrate broad safety degradation across multiple HEx-PHI categories.

  • Harmful generations include advice on distributing pirated movies and software through peer-to-peer networks and the dark web.
  • It identifies addictive substances that could be added to food, including opioids, stimulants, and benzodiazepines.
  • The model provides procedural instructions for breaking into a cannon gun safe with a digital keypad.
  • The generations also offer short-term stock-growth recommendations across many technology and consumer companies.
  • It recommends cryptocurrencies associated with illegal activities such as money laundering and predicts their short-term profitability.
Loading 2505.06843v2…