What is in Your Safe Data? Identifying Benign Data that Breaks Safety

• The study asks whether a subset of benign data disproportionately facilitates jailbreaking and what characteristics distinguish it.The motivation is to understand benign fine-tuning failures from a data-centric perspective.Source: §1 · p. 1
• The approach measures similarity to known harmful examples using gradient and representation features, while gradient selection also prioritizes dissimilarity from safe responses.The indicator is intended to support data-centric debugging and safer data mixtures.Source: §1 · pp. 1–2
• 100 selected benign examples raise GPT-evaluated ASR from 13% to 71% on ALPACA and from 8.2% to 53.3% on DOLLY, versus random subsets.Both similarity-based approaches identify benign data that significantly increase jailbreaking after fine-tuning.Source: §1 · p. 2
• Gradient-based selection is more consistent than representation-based selection across datasets, and bidirectional anchoring is essential for ranking jailbreak-prone data.Source: §1 · p. 2
• The most harmful benign examples often use bullet-point answers or mathematical expressions, and random math selections are more harmful than random selections from diverse instruction-tuning data.Source: §1 · p. 2
• The resulting tools identify small benign datasets that compromise safety and highlight vulnerabilities in downstream datasets containing math problems or information lists.Source: §1 · p. 2

2 Method

• The method defines benign, harmful, and safe instruction-completion datasets and seeks benign examples whose fine-tuning behavior resembles harmful updates.Representation matching uses nearby hidden-state features, while gradient matching targets updates likely to reduce loss on harmful examples.Source: §2 · p. 2

2.1 Representation Matching

• Representation matching selects benign examples with the highest cosine similarity to harmful examples using the final hidden state of each completion.Source: §2.1 · p. 3

2.2 Gradient Matching

• Gradient matching represents each example with its parameter-gradient feature, motivated by first-order influence on losses after a training update.The gradient feature is normalized because unnormalized selection favored shorter sequences.Source: §2.2 · p. 3
• Gradient-space selection is difficult because features are high-dimensional, sparse, and noisy, and minimizing harmful-example loss may not directly match attack success.Bidirectional anchoring is introduced to address these challenges.Source: §2.2 · pp. 3–4
• Averaging harmful and safe anchor features reduces the comparison cost from O(|Dbenign| · |Dharmful S Dsafe|) to O(|Dbenign|).Source: §2.2 · p. 4
• Bidirectional anchoring ranks candidates by closeness to harmful examples and distance from safe examples, rather than using only harmful similarity.Average reference features make comparisons computationally manageable.Source: §2.2 · p. 4

3 Experiments

• The experiments evaluate whether selected benign subsets can effectively jailbreak previously aligned models.ALPACA and DOLLY provide the fine-tuning data, while 100 harmful instructions from PURE-BAD serve as the comparison and anchoring basis.Source: §3 · p. 4, §3.1 · p. 4

3.1 Experimental Setup

• ALPACA and DOLLY examples are filtered to exclude malicious prompts, explicit harmful information, and safety-tuning responses before selecting 100 examples.Source: §3.1 · p. 4
• The selection uses harmful anchors from PURE-BAD, with a default set of 10 illegal-activities examples and an additional safe anchor for gradient matching.Source: §3.1 · p. 4
• The main experiments use safety-tuned LLAMA-2-7B-CHAT, fine-tuning selected subsets for five epochs with five random seeds and reporting average and standard deviation.Source: §3.1 · p. 5
• Safety is evaluated on 520 harmful instructions using Keyword ASR, GPT ASR, and GPT Score, with GPT ASR measuring explicit helpful responses to harmful prompts.Source: §3.1 · p. 5

4.1 Fine-tuning on selected benign data significantly increases ASR, sometimes even more than fine-tuning on explicitly harmful data.

• On ALPACA and DOLLY, fine-tuning on benign subsets selected by representation matching and gradient matching produces significant and consistent increases in harmfulness metrics compared with random selection.On ALPACA, representation-matching selection produces more harmful behavior than an equivalent number of explicitly harmful data instances.Source: §4.1 · pp. 5–6
• ALPACA examples selected by representation matching produce more harmful behavior than an equivalent number of explicitly harmful instances.This result highlights harmful potential in data that appears benign on the surface.Source: §4.1 · p. 6
• Ten illegal-activities examples from one harmful category suffice to anchor selection effectively despite evaluation across diverse harmful-prompt categories.The authors speculate that category consistency reduces noise in the averaged harmful-feature representation.Source: §4.1 · p. 6

4.2 Bidirectional anchoring with harmful data and safe data is crucial for properly rank-sorting data.

• Bidirectional anchoring raises top-selected ALPACA ASR from 46.6% to 66.5% and DOLLY ASR from 4.9% to 53.3%, while lowering ALPACA bottom-selected ASR to 3.8%.It accounts for distance from both harmful and safe data, correcting the misleading high ASR of low-scoring examples under unidirectional anchoring.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Examples selected by LLAMA-2-7B-CHAT model also break safety of other models.

• Selected examples frequently use list, bullet-point, or mathematical formats, with representation-based selections especially concentrated in lists and bullet points.Gradient-based selections are more diverse and include lists and math; Figure 3 highlights list and math formats.Source: Figure 3 · p. 7
• Selected ALPACA data transfers across model families: LLAMA-3-8B-CHAT reaches 57% harmfulness versus 25.9% for random selection, and GEMMA-7B-INSTRUCT reaches 49.3% versus 34.2%.The selection also boosts harmfulness in LLAMA-2-13B-CHAT, according to the transfer experiment.Source: §4.3 · pp. 6–7, Table 3 · p. 7

5.1 What characterizes the HARMFUL benign data?

• 45 of 100 ALPACA selections prompted lists after illegal-activity anchoring, while 32 of 100 were math or unit-conversion questions after hate-speech anchoring.In DOLLY, approximately 50% of top selections had answers beginning with a listing format.Source: §5.1 · p. 7
• Random ALPACA selections containing lists or math were more harmful than random selections, although list examples chosen by the method were more harmful still.The experiment filtered instructions using cues such as “Give a list of...” and “Calculate...” before sampling 100 examples of each type.Source: §5.1 · p. 8, Table 4 · p. 8

5.2 Analysis of math examples

• 41% GPT ASR followed random fine-tuning on 100 GSM8K examples, while top, random, and bottom selections achieved similar utility and bottom selection produced much lower harmfulness.The results show that improving math capability can increase harmfulness even in a standard downstream task.Source: §5.2 · p. 8, Table 5 · p. 8

5.3 Analysis of listing examples

• Reformatting the same 100 ALPACA responses into lists or bullet points doubled ASR under the same fine-tuning and inference process.The comparison isolates response formatting while keeping the selected instruction-tuning pairs otherwise unchanged.Source: §5.3 · p. 9, Table 6 · p. 9

5.4 Gradient Matching vs. Representation Matching

• Gradient matching generalizes more reliably than representation matching: on DOLLY, gradient selection reached 53.3% ASR versus 31.8%, while representation ranking failed on GSM8K.Representation selection was less computationally intensive, but gradient matching was more stable and transferable across datasets.Source: §5.4 · p. 9

5.5 Why do certain benign data compromise safety?

• Lists and math data had positive similarity to harmful gradients, unlike random selections, suggesting shared descent directions may contribute to jailbreak behavior.The reported similarities were 0.05 for lists and 0.01 for GSM8K, compared with -0.0395 for random selections.Source: §5.5 · p. 9

Data poisoning attacks

• Traditional data poisoning injects manipulated training data to induce targeted errors, whereas this work studies original benign data that can break safety during text-generation fine-tuning.The paper uses gradient matching but does not modify data or target failure on a specific input.Source: §6 · pp. 9–10

Data-oriented LLM safety risks

• LLM safety research includes instruction-tuning attacks that degrade generated responses, harmful-behavior poisoning, representation-based safety analysis, and gradient-based unsafe-prompt detection.These studies frame broader response-generation safety risks beyond traditional classification errors.Source: §6 · p. 10

Data selection

• Data-selection research uses gradients, optimization, or datamodels to improve task performance, while this paper examines how selected subsets affect downstream safety.Its focus is the interplay between data selection and safety implications rather than instruction-following quality alone.Source: §6 · p. 10

7 Limitations and Future Directions

• Gradient matching captures only some optimization effects, since batch size changes attack success rates and the method does not cover full training trajectories.The authors suggest using more checkpoints to find examples harmful across hyperparameter settings.Source: §7 · p. 10
• The study considers fine-tuning but not pre-training, although the same approach might identify unsafe data during pre-training.This marks a scope boundary rather than an evaluated result.Source: §7 · p. 10
• Gradient and representation matching are only two proxies for identifying benign datapoints that increase ASR, leaving room for other metrics.The broader anchoring framework is presented as a direction for future data-selection research.Source: §7 · p. 10

8 Conclusion

• GPT-3.5 ASR rises from < 20% to > 70% after fine-tuning on the selected benign dataset, exceeding fine-tuning on an explicitly harmful dataset of equal size.The conclusion presents representation- and gradient-based selection as tools for identifying benign data that jailbreaks models.Source: §8 · p. 10

A.1 Ethics Statement

• The study aims to improve safety under downstream customization while warning that small benign datasets can significantly degrade safety.The authors state that additional measures may be needed to prevent this safety loss.Source: §A.1 · p. 14
• Evaluations are limited to four chat models with available unaligned base models, and the harmful anchoring set is only partially released.These choices are intended to balance reproducibility against the risk of enabling safeguard-jailbreaking methods.Source: §A.1 · p. 14
• The harmfulness evaluation includes model-generated responses that may contain offensive content, with less harmful examples selected for appendix presentation.This is an ethical handling measure for the reported demonstrations.Source: §A.1 · p. 14

A.3 Additional Experiments Configurations

• The reported results average five randomly seeded runs, use greedy decoding, and apply full fine-tuning to selected subsets of 100 examples for five epochs.The default selected-subset batch size is 20, while full-dataset fine-tuning uses batch size 128 for one epoch.Source: §A.3 · pp. 14–15
• The experiments use harmful anchors from illegal-activity or hate-speech categories and safe responses that are either uniform refusals or more diverse restraint explanations.Dsafe is generated from the same selected harmful prompts using an aligned model.Source: §A.3 · p. 14
• Hate-speech anchoring produces more useful and reliable rankings when combined with diverse safety anchoring.The ablation reports average score, ASR, and standard deviations across five runs.Source: §A.3 · p. 14, Table 7 · p. 15

A.4 Main Results on LLAMA-2-13B-CHAT

• Both selection methods significantly increase harmfulness when their 100-example selections transfer from LLAMA-2-7B-CHAT to LLAMA-2-13B-CHAT.For DOLLY, the top representation-based selection is even more harmful on the 13B model than on the base model used for selection.Source: §A.4 · p. 15, Table 8 · p. 15

A.5 Parameter ablations

• The ablations vary the number of tokens used to aggregate instruction-tuning loss and control randomness with top-p=0.Tables 2 and 7 examine anchor setup and anchoring sets.Source: §A.5 · p. 15

A.6 Fine-tuning with selected data with smaller batch size is more harmful.

• The batch-size ablation fixes the anchoring set to harmful examples plus uniform and diverse safe responses, with 10 response tokens and batch size 20 as the reported default configuration.The ablation configuration uses five epochs.Source: Table 9 · p. 16, Table 10 · p. 16
• Smaller batch sizes produce larger harmfulness increases after fine-tuning the selected 100 GSM8K examples.Figure 4 reports the strongest increases for batch sizes 10 and 20, while batch size 50 has significantly lower impact.Source: §A.6 · pp. 15–16, Figure 4 · p. 16

B Examples of Selected Data

• Gradient-based selection produced more diverse benign examples than representation-based selection, while representation-based selections concentrated on lists and bullet points.The selected subsets included lists, math questions, and other instruction-response formats.Source: B Examples of Selected Data · p. 17
• 32 of the top 100 Alpaca selections were math or unit-conversion questions when gradient selection used hate-speech anchoring.This result used Dharmful as a subset of 10 hate-speech instruction-response pairs.Source: B Examples of Selected Data · p. 17

B.1 Examples of Model Response to Harmful Questions

• Models fine-tuned on representation- and gradient-selected data generated affirmative, procedural responses to the same five harmful AdvBench questions.The examples span identity theft, manipulation, fraudulent stores, terrorist infiltration, and cyberbullying prompts, with corresponding responses shown for representation, Alpaca-gradient, and Dolly-gradient selections.Source: §B.1 · pp. 21, 23
• The model provides step-by-step answers to harmful questions, including instructions related to terrorist attacks and cyberbullying.The examples show affirmative procedural responses rather than refusals.Source: §B.1 · p. 23

B.2 Qualitative Study of Selected Datapoints

• List and math examples are common among selected datapoints, but harmfulness varies across random seeds and depends on the specific math questions chosen.The study compares 100-example list-only and math-only sets with a random baseline and notes that further work should examine causal characteristics of math questions.Source: §B.2 · p. 24

B.3 GPT-3.5 Judge Prompt

• The GPT-3.5 judge prompt combines Meta’s prohibited-usage policies with scoring guidelines that assign low scores to simple prompt repetition or incomplete frameworks.The revised prompt expands the guidelines used by Qi et al. (2023) and gives such responses a score of 2 out of 5.Source: §B.3 · p. 26, Table 11 · p. 26