PaperlayerView exact revision on arXiv

Source-linked AI summary

Unleashing the Unseen: Harnessing Benign Datasets for Jailbreaking Large Language Models

Wei Zhao, Zhe Li, Yige Li, Jun Sun

arXiv:2410.00451v3cs.CRcs.AIcs.CL

TL;DR

The paper investigates whether jailbreak suffixes can function as meaningful features that influence model behavior, rather than merely reflecting misalignment on out-of-distribution prompts. It tests feature-bearing suffixes and benign-data fine-tuning, finding that these routes can compromise safety alignment, including effectively eliminating it in fine-tuned models.

  • Problem

    Jailbreak attacks can elicit harmful behavior from aligned LLMs, and it remains an open question whether their suffixes reflect meaningful features or misalignment on out-of-distribution prompts.

    Source: §1 · pp. 1–2

  • Method

    The authors extract transferable suffixes from benign and harmful prompt-response datasets, test their effects across prompts, and fine-tune models on benign datasets.

    Source: §1 · p. 2

  • Results

    Benign-feature suffixes can compromise safety alignment, jailbreak-derived suffixes may encode response features, and fine-tuning on benign datasets effectively eliminated safety alignment in tested models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings indicate that safety alignment may be dominated by benign features, so benign-data fine-tuning should not be assumed to preserve it.

    Source: §1 · p. 2, Limitations · p. 8

    The study focuses on robust, human-identifiable benign features; non-robust features that do not trigger recognizable styles may also exist.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Despite significant ongoing efforts in safety alignment, large language models (LLMs) such as GPT-4 and LLaMA 3 remain vulnerable to jailbreak attacks that can induce harmful behaviors, including through the use of adversarial suffixes. Building on prior research, we hypothesize that these adversarial suffixes are not mere bugs but may represent features that can dominate the LLM's behavior. To evaluate this hypothesis, we conduct several experiments. First, we demonstrate that benign features can be effectively made to function as adversarial suffixes, i.e., we develop a feature extraction method to extract sample-agnostic features from benign dataset in the form of suffixes and show that these suffixes may effectively compromise safety alignment. Second, we show that adversarial suffixes generated from jailbreak attacks may contain meaningful features, i.e., appending the same suffix to different prompts results in responses exhibiting specific characteristics. Third, we show that such benign-yet-safety-compromising features can be easily introduced through fine-tuning using only benign datasets. As a result, we are able to completely eliminate GPT's safety alignment in a blackbox setting through finetuning with only benign data. Our code and data is available at \url{https://github.com/suffix-maybe-feature/adver-suffix-maybe-features}.

1 Introduction

The introduction asks whether adversarial suffixes can act as meaningful features rather than mere anomalies. It presents experiments showing benign features can compromise safety through suffixes and fine-tuning, and that jailbreak-derived suffixes may also encode features.

1 Introduction

  • GCG combines greedy and gradient-based search to produce human-nonsensical suffixes that can elicit harmful responses from strongly aligned models.The introduction situates this vulnerability among jailbreak attacks against aligned models.Source: §1 · p. 1
  • The authors conjecture that adversarial suffixes may represent meaningful model features, motivated by findings that models exploit adversarial perturbations as features.They contrast this with the view that vulnerability reflects misalignment on out-of-distribution prompts.Source: §1 · p. 2
  • Benign dataset features can be extracted as transferable suffixes that activate the intended feature on benign and harmful prompts, potentially compromising safety alignment.The introduction illustrates a point-by-point style that can accompany harmful responses.Source: §1 · p. 2
  • Jailbreak-derived suffixes may encode features, as prompts appended with them can elicit responses with consistent characteristics such as storytelling style.The feature-extraction method is used on harmful prompt-response pairs and functions in this setting as a universal adversarial attack method.Source: §1 · p. 2
  • Fine-tuning models, including GPT, on benign datasets effectively eliminated their safety alignment.The datasets exhibited specific response formats or writing styles.Source: §1 · p. 2
  • The authors argue that defenses may be inadequate if benign features can bypass alignment, and caution that benign-data fine-tuning may not preserve safety.They call for alignment to remain a priority during fine-tuning.Source: §1 · p. 2

2 Analysis Methods

The section introduces a universal suffix extractor for capturing sample-agnostic dataset features and a PCC analysis for comparing prompt and suffix influence. It illustrates how suffixes can capture response patterns and how their influence can be assessed.

2.1 Extracting Features As Suffixes

  • The extractor learns a common suffix that captures a target response pattern across prompt-response pairs, with embedding-vector or discrete-token suffixes.The suffix is optimized to make the model generate each pair’s target response when appended to its prompt.Source: §2.1 · p. 2, Figure 2 · p. 3
  • Token-based suffix optimization adds an embedding regularizer to encourage valid tokens, with λ = 10 for token suffixes and λ = 0 for embedding vectors.The overall loss combines the adversarial loss with λ times the embedding regularizer.Source: §2.1 · p. 3

2.2 PCC Analysis

  • PCC analysis compares correlations among the prompt-only, suffix-only, and prompt-plus-suffix last hidden states to assess their relative influence.The authors focus on last hidden states because they encode features needed for response generation and are linearly transformed into token probabilities.Source: §2.2 · p. 3
  • A higher correlation between the prompt-only and combined states than between the suffix-only and combined states suggests the prompt is more dominant in shaping output.The analysis uses PCC as a measure of linear correlation.Source: §2.2 · p. 3
  • With 100 harmful prompts, meaningless exclamation-mark suffixes had near-zero suffix-to-combined-state PCC, while prompt-specific adversarial suffixes had high PCC, indicating greater influence.Figure 3 plots hidden-state values on the x-axis and combined-state values on the y-axis; its blue and red dots show prompt-to-combined and suffix-to-combined PCC values, respectively.Source: §2.2 · p. 3, Figure 3 · p. 4

3 Experiment 1: Benign Features Act As Adversarial Suffixes

The experiment tests whether suffixes extracted from benign format and style features can compromise safety alignment. The suffixes transfer their target features, while their safety effects vary across models and datasets.

3.1 Experiment Setup

  • The Poem and Structure datasets were built from benign dialogues with targeted response formats, then filtered for harmful or low-quality content.The experiment evaluated format transfer on benign prompts and attack success rate and response harmfulness on 520 harmful prompts.Source: §3.1 · pp. 4–5

3.2 Experiment Results

  • More than 92% transferability for embedding suffixes and 74% for token suffixes across both datasets and models; safety compromise varied by model and dataset.Suffixes compromised alignment for Mistral-7B, while for Llama2-7B the Structure suffix did so but the Poem suffix had little effect.Source: §3.2 · p. 5
  • For Llama2-7B, the Poem suffix left the harmful prompt dominant, whereas the Structure suffix had greater suffix influence and induced harmful behavior.The comparison uses PCCs between the prompt-only, suffix-only, and combined hidden states.Source: §3.2 · p. 5

4 Experiment 2: Jailbreaking Suffixes Contain Features

Experiment 2 tests whether jailbreak suffixes extracted from harmful prompt-response pairs can both compromise safety and induce meaningful features across prompts.

4.1 Experiment Setup

  • The method extracted suffixes from 1,000 randomly selected harmful prompt-response pairs and was evaluated for attack effectiveness and meaningful features.The authors describe this use of their extraction method as a universal adversarial attack and compare it with GCG and AmpleGCG.Source: §4.1 · p. 5

4.2 Experiment Results

  • The token-form suffixes slightly improved performance over GCG and AmpleGCG.The comparison uses the best-performing suffix from each method.Source: §4.2 · p. 5, §4.1 · p. 5
  • Embedding suffixes achieved nearly perfect ASR and harmfulness scores across all three models, making them more effective than token suffixes.The authors note that the larger embedding space is easier to work with, while emphasizing that attack generation itself is not their goal.Source: §4.2 · p. 6
  • Some suffixes did not produce an observable feature, which could mean either that no feature was present or that the researchers failed to recognize it.The authors manually examined responses after appending suffixes to benign prompts.Source: §4.2 · p. 6
  • Repeat Response and Basic Programming suffixes showed high transferability and jailbreak performance, and the responses exhibited those distinct styles or formats.The evaluation measured transferability, attack success rate (ASR), and harmfulness; examples included repetitive responses and basic programming content.Source: §4.2 · p. 6, Figure 5 · p. 6
  • PCC analysis found suffix influence close to 1 and prompt influence low, indicating that the suffix induced the feature-bearing responses and compromised safety alignment.Figure 6 reports suffix influence above 0.8 and prompt influence below 0.3.Source: §4.2 · pp. 6–7, Figure 6 · p. 7

5 Experiment 3: Benign Dataset May Compromise Safety

Experiment 3 tests whether benign-data fine-tuning can introduce dominant features that compromise model safety, using several datasets and aligned language models.

5.1 Experiment Setup

  • The four constructed benign datasets each contained 1,000 prompt-response pairs, and harmful prompts or responses were filtered out; Alpaca provided a separate 1,000-pair random benign dataset.The constructed datasets included template-generated poem and structure responses and datasets based on universal suffixes extracted from harmful data.Source: §5.1 · p. 7
  • The study fine-tuned two open-source and three closed-source models, including GPT models through the OpenAI API, on one benign dataset at a time.Open-source models were fine-tuned for 2,000 steps; closed-source models for three epochs.Source: §5.1 · p. 7

5.2 Experiment Results

  • Random benign data degraded safety in Llama2 and Llama3, slightly degraded it in GPT-3.5, and had no impact on GPT-4o or GPT-4o-mini.The original models produced no harmful responses before fine-tuning.Source: §5.2 · p. 7
  • All datasets built with universal suffixes severely degraded safety alignment after fine-tuning, while template datasets had mixed effects, including 4.9% ASR for GPT-4o on poem data and 72.7% ASR for GPT-4o on structure data.The authors interpret the suffix-derived datasets as more strongly correlated with harmfulness because they contain natural adversarial features.Source: §5.2 · p. 7, Table 3 · p. 8
  • Fine-tuning on benign datasets with dominant features can undermine safety alignment, so benign data alone does not guarantee its preservation.The authors call for fine-tuning strategies that account for dominant benign features.Source: §5.2 · pp. 7–8

6 Related Work

The related work situates the paper among automated jailbreak attacks and studies of safety risks from fine-tuning. It distinguishes this study by examining response features and their effects on safety.

6.1 Jailbreak Attack

6.2 Fine-Tuning and Safety

  • Prior work found that harmful fine-tuning samples can compromise safety (Shu et al., 2023; Zheng et al., 2024), and benign data can also degrade safety (Qi et al., 2023; Zhan et al., 2023).He et al. (2024) found safety degradation from selected data often structured as lists, bullet points, or math questions.Source: §6.2 · p. 8
  • Compared with prior selected-data work, this study reports that domain-specific-like benign datasets can compromise alignment without specific data selection and offers a systematic way to generate such datasets with universal adversarial suffixes.The authors also aim to analyze how particular response structures and styles affect model behavior.Source: §6.2 · p. 8

7 Conclusion

The conclusion summarizes three findings: benign features can serve as adversarial suffixes, jailbreak suffixes may contain meaningful features, and benign datasets can introduce safety-compromising features.

7 Conclusion

  • Benign features may function as effective adversarial suffixes.Source: §7 · p. 8
  • Adversarial suffixes generated by jailbreak attacks may contain meaningful features.Source: §7 · p. 8
  • Safety-compromising benign features may be introduced through benign datasets.Source: §7 · p. 8

Limitations

The analysis focuses on robust, human-identifiable benign features, while noting that non-robust features may not produce human-recognizable styles.

Limitations

  • The analysis covers robust, human-identifiable benign features; non-robust features that do not trigger recognizable styles may lie beyond its scope.The authors connect this scope boundary to the need for alignment that remains safe despite dominant features in fine-tuning datasets.Source: Limitations · p. 8

A.1 Universal Feature Extractor

The universal feature extractor optimizes a shared suffix to elicit target responses across prompts. It can operate in embedding space and optionally regularize suffix embeddings toward valid tokens.

A.1 Universal Feature Extractor

  • A single shared suffix is optimized so that appending it to each prompt leads the model to generate that prompt’s target response.The objective minimizes cross-entropy between the model’s prediction and the target response.Source: §A.1 · p. 11
  • The method can optimize suffixes directly in embedding space, allowing features that may not correspond to valid token sequences.A token-proximity regularizer can instead encourage suffix embeddings to remain near their k nearest vocabulary-token embeddings, with λ controlling its weight alongside the adversarial objective.Source: §A.1 · p. 11
  • For token attacks, optimized suffix embeddings are projected to their nearest valid tokens so the suffix can be represented as a token sequence.The method also considers embedding-vector optimization without this token-alignment constraint.Source: §A.1 · p. 11

A.2 Additional Experiments 1: Benign Features Act As Adversarial Suffixes

These additional experiments report transfer of benign features to harmful prompts and examine features in jailbreak suffixes. The appendix also documents dataset-generation and evaluation templates, GPT fine-tuning examples, and dataset samples.

A.2 Additional Experiments 1: Benign Features Act As Adversarial Suffixes

  • Structure features transferred to harmful prompts on Llama3-8B and Vicuna-7B, whereas Poem features failed to transfer on strongly aligned Llama3.The authors report that these results align with their earlier findings.Source: §A.2 · p. 12

A.3 Additional Experiments 2: Jailbreaking Suffixes Contain Features

  • The examined jailbreak suffixes include a token suffix associated with bullet-point responses and an embedding suffix associated with repetitive responses.The token suffix achieved reasonably high transferability and attack success rate; the authors suspect its bullet-point feature relates to step-by-step instructions in harmful prompts.Source: §A.3 · p. 12

A.4 Templates used for generating Benign Datasets

A.5 Templates used for evaluating transferability on benign datasets

  • The transferability evaluation appendix provides templates for BASIC programming, storytelling, and repetitive-tone formats.The corresponding figures are identified as Figures 9, 10, and 11.Source: §A.5 · p. 13, Figure 9 · p. 13, Figure 10 · p. 13

A.6 Templates used for harm evaluation

  • The harm-evaluation templates cover harmfulness scoring and attack-success-rate assessment, with a policy prompt asking whether a response complies with prohibited-use guidelines.Figures 12 and 13 show the system templates for those two evaluations.Source: §A.6 · pp. 13–14, Figure 12 · p. 14, Figure 13 · p. 15

A.7 Fine-tuning Results On GPT

A.8 Datasets

  • The appendix tables provide examples from the Structure format dataset and from the BASIC and Repeat datasets generated with adversarial suffixes.These examples appear in Tables 6 and 7.Source: §A.8 · p. 22, Table 6 · p. 23, Table 7 · p. 24
Loading 2410.00451v3…