PaperlayerView exact revision on arXiv

Source-linked AI summary

On Prompt-Driven Safeguarding for Large Language Models

Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, Nanyun Peng

arXiv:2401.18018v4cs.LGcs.AIcs.CL

TL;DR

Safety prompts are commonly used to reduce harmful-query compliance, but their working mechanisms remain unclear. The paper analyzes their representation-space effects and proposes DRO, which optimizes prompts by directing query representations according to harmfulness. Experiments report improved safeguarding across out-of-domain and jailbreak benchmarks without compromising general performance.

  • Problem

    Safety prompts are widely used to safeguard LLMs, but their working mechanisms remain unclear, limiting automatic optimization to improve safety.

    Source: §1 · p. 1

  • Method

    DRO treats safety prompts as trainable continuous embeddings and moves query representations along or opposite an estimated refusal direction according to harmfulness.

    Source: §1 · p. 2

  • Results

    DRO improves safeguarding performance on out-of-domain and jailbreak benchmarks without compromising general performance across the evaluated LLMs.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings provide a representation-based account of prompt-driven safeguarding and motivate further research on LLM safety.

    Source: §6 · p. 9

    Queries in the study are unambiguously harmful or harmless, while real-world harmfulness can be ambiguous; safety prompts alone are far from sufficient without massive and continual safety training and alignment.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

Prepending model inputs with safety prompts is a common practice for safeguarding large language models (LLMs) against queries with harmful intents. However, the underlying working mechanisms of safety prompts have not been unraveled yet, restricting the possibility of automatically optimizing them to improve LLM safety. In this work, we investigate how LLMs' behavior (i.e., complying with or refusing user queries) is affected by safety prompts from the perspective of model representation. We find that in the representation space, the input queries are typically moved by safety prompts in a "higher-refusal" direction, in which models become more prone to refusing to provide assistance, even when the queries are harmless. On the other hand, LLMs are naturally capable of distinguishing harmful and harmless queries without safety prompts. Inspired by these findings, we propose a method for safety prompt optimization, namely DRO (Directed Representation Optimization). Treating a safety prompt as continuous, trainable embeddings, DRO learns to move the queries' representations along or opposite the refusal direction, depending on their harmfulness. Experiments with eight LLMs on out-of-domain and jailbreak benchmarks demonstrate that DRO remarkably improves the safeguarding performance of human-crafted safety prompts, without compromising the models' general performance.

1. Introduction

The introduction asks how safety prompts affect LLM behavior, given that their mechanisms remain unclear and automatic optimization is therefore limited. It reports a representation-based explanation and introduces DRO, which improves safeguarding performance without compromising general performance.

1. Introduction

  • Safety prompts are widely used to mitigate harmful-query compliance, but their working mechanisms remain unclear, limiting automatic optimization.They offer a lightweight safeguard without modifying model parameters or intervening in inference.Source: §1 · p. 1
  • Harmful and harmless queries are naturally distinguishable in representations, while safety prompts do not noticeably enhance that distinction.Instead, different prompts move representations in similar directions that increase refusal propensity, including for harmless queries.Source: §1 · p. 2
  • DRO treats safety prompts as trainable continuous embeddings and moves query representations along or opposite an estimated refusal direction according to harmfulness.The method freezes model parameters and uses regularization to limit degeneration of the original representation.Source: §1 · p. 2
  • Across eight LLMs and out-of-domain and jailbreak benchmarks, DRO improves human-crafted prompts’ safeguarding performance without compromising general performance.For the LLaMA-2 safety prompt, harmful-query compliance falls from 10.3% to 1.4% on AdvBench.Source: §1 · p. 2

2. How Safety Prompts Intrinsically Work?

The section examines whether safety prompts improve harmfulness recognition or instead alter refusal behavior, using controlled query pairs and representation analyses. It finds that prompts shift representations toward higher refusal without noticeably improving harmfulness distinction.

Why can safety prompts safeguard LLMs against harmful

  • The analysis tests two explanations: prompts might improve harmfulness recognition, or increase refusal probability when models already recognize harmful queries.The study examines query representations and how prompt-induced changes correlate with refusal behavior.Source: Why can safety prompts safeguard LLMs… · p. 2

2.1. Controlled Data Synthesis

  • The study synthesizes paired “How to” queries with the same verb and format, then filters ambiguous harmless examples and matches pair lengths.The resulting set contains 100 harmful and 100 harmless queries, averaging 14.0 and 13.8 tokens, respectively.Source: §2.1 · p. 2

2.2. Experimental Setup

  • Experiments use eight 7B chat LLMs, three safety prompts, and 20 sampled responses per query to assess refusal behavior.The prompts are the official LLaMA-2 and Mistral versions and a shortened LLaMA-2 version.Source: §2.2 · pp. 2–3
  • Harmless-query refusals are detected by string matching, while harmful-query responses are classified for safety with LlamaGuard.String matching can miss varied refusals and cases where a response begins with refusal language but later complies.Source: §2.2 · p. 3

2.3. Visualization Analysis

  • PCA visualizes the last input token’s top-layer hidden state across harmfulness and prompt conditions.The plotted groups cover harmful and harmless queries with no prompt or one of three safety prompts.Source: §2.3 · pp. 3–4, Figure 3 · p. 3
  • Harmful and harmless queries are naturally distinguishable in model representations, but safety prompts do not noticeably improve that distinction.The finding holds in other principal components as well as the visualization described in the passage.Source: §2.3 · p. 4
  • Different safety prompts move representations in similar higher-refusal directions, increasing refusals for both harmful and harmless queries.The shifts can raise false refusals on harmless queries.Source: §2.3 · p. 4

3. Methodology

DRO addresses variability in human-crafted safety prompts by optimizing continuous prompts in a representation space anchored to harmfulness and refusal behavior. It moves query representations according to harmfulness while regularizing changes outside the targeted features.

3. Methodology

  • DRO optimizes continuous safety prompts by moving query representations along or opposite an estimated refusal direction according to harmfulness.Model parameters stay frozen, and only the prompt embeddings are trained.Source: §3 · p. 4, §3.1 · p. 4, §3.2 · p. 5

3.1. Anchoring Process

  • Anchoring pairs controlled harmful and harmless queries with k basic safety prompts to derive a low-dimensional space and fit empirical refusal probabilities.The fitted normal estimates the direction of increasing refusal probability; experiments used m = 4, with near-zero components in dimensions 3 and 4.Source: §3.1 · pp. 4–5

3.2. Optimization Process

  • With model weights frozen, DRO trains prompt embeddings to raise refusal logits for harmful queries and lower them for harmless queries relative to the initial prompt.The objective contrasts the optimized prompt representation with the initial prompt representation.Source: §3.2 · p. 5
  • A harmfulness-recognition loss can help maintain harmful/harmless discrimination and bring some safeguarding improvement.This loss uses a separate logistic regression fitted with query harmfulness labels.Source: §3.2 · p. 5

3.3. Regularization

  • A regularizer penalizes change in the remaining n−m dimensions to limit representation degeneration while targeting harmfulness and refusal features.The experiments set β = 0.001 to balance optimization of the extracted features against regularization of the remaining dimensions.Source: §3.3 · pp. 5–6

3.4. Highlights

  • DRO uses a small set of anchor data to extract salient harmfulness and prompt-impact features, and shows reasonable robustness to anchor-data choices.The prompt impact correlates strongly with refusal behavior.Source: §3.4 · p. 6
  • Training on only 200 synthetic examples can significantly enhance human-crafted safety prompts without textual-response supervision.The paper notes that suitable query-response demonstration pairs are not easily obtained in the current open-source community.Source: §3.4 · p. 6
  • DRO’s regularization item helps retain information beyond the target features and is critical to maintaining general performance.The authors contrast this with traditional continuous-prompt training, which requires general-domain data to prevent catastrophic forgetting.Source: §3.4 · p. 6

4. Evaluation

DRO improves human-crafted safety prompts across the reported evaluations, while ablations and robustness tests identify conditions affecting performance. Its optimized prompts generally project near their initialized token embeddings.

4.1. Baselines and Benchmarks

  • DRO and vanilla prompt-tuning share 200 synthetic training examples and initialization; evaluation uses two distinct harmful-query benchmarks, AlpacaEval, and 100 harmless queries.Source: §4.1 · p. 6

4.2. Main Results

  • DRO significantly improves safeguarding over the human-crafted basic safety prompt; Table 2 covers MaliciousInstruct, AdvBench, harmless queries, and AlpacaEval.Source: §4.2 · p. 6, Table 2 · p. 7
  • Lr is critical for refusing harmful queries, LU preserves general performance (52.6 vs. 63.5 on AlpacaEval without LU), and removing Lh has no noticeable effect.Source: §4.2 · p. 7

4.3. Extension To Jailbreak Setting

  • Under GCG jailbreaks, DRO improves safeguarding over the basic prompt (3.1 vs. 13.3); the basic prompt also outperforms no prompt (13.3 vs. 54.1).Source: Table 3 · p. 8, §4.3 · p. 8

4.4. Robustness Analysis

  • DRO remains robust to anchor-data choices, though less-controlled queries slightly degrade general performance (59.0 vs. 63.5), and single-prompt anchoring weakens safeguarding (4.1 vs. 2.3).The authors recommend careful query controls and combining multiple basic safety prompts for anchoring.Source: §4.4 · p. 8, Table 4 · p. 8

4.5. Interpretability Analysis

  • Only six of 24 optimized prompts project to tokens that slightly differ from their initial prompts, suggesting optimization generally stays near initialized token embeddings.Source: §4.5 · pp. 8–9

5. Related Work

Related work situates the paper in LLM safety research and prompt optimization, including continuous prompt methods and approaches that optimize discrete prompts.

  • LLM safety research addresses harmful generated content and refusal of harmful requests, alongside work on undesirable attributes and jailbreak attacks.Source: §5 · p. 9
  • DRO follows continuous prompt optimization with frozen model parameters; prior discrete prompt optimization uses gradient search or RL, while LLM-based optimizers may depend on proprietary models.Source: §5 · p. 9

6. Conclusion

Safety prompts shift representations toward refusal without clearly improving harmfulness recognition. DRO optimizes that direction according to query harmfulness, improving safeguarding while preserving general performance.

  • Safety prompts do not clearly improve harmfulness recognition; instead, they move queries toward higher refusal probability, including harmless queries.The queries’ representations move in a “higher-refusal” direction.Source: §6 · p. 9
  • DRO optimizes continuous safety prompts by moving query representations along or opposite an estimated refusal direction according to harmfulness.Source: §6 · p. 9
  • DRO improves safeguarding on out-of-domain and jailbreak benchmarks without compromising general performance, with reasonable robustness to anchoring-data choices.Source: §6 · p. 9

Impact Statement

DRO is presented as a positive-use safeguarding method, though its objective raises a dual-use concern. The authors also limit their claims to unambiguous queries and state that safety prompts alone are insufficient.

  • Flipping DRO’s harmfulness labels could lower refusal on harmful queries, but conflicts with natural harmfulness recognition and may undermine general capability.The authors say the recognition objective is entailed by the refusal objective and that this conflict would hinder malicious use.Source: Impact Statement · p. 9
  • The analysis covers unambiguously harmful or harmless queries, while real queries may be ambiguous and difficult for models or people to judge.Source: Impact Statement · p. 9
  • Safety prompts alone are far from sufficient; the authors say improving LLM safety still requires massive, continual safety training and alignment.Source: Impact Statement · p. 9

A. Open-Source Models and Data Used in This Work

The supplementary material lists data and safety-prompt variants, illustrates synthesized and excluded queries, and reports PCA explained-variance ratios for eight models.

A. Open-Source Models and Data Used in This Work

B. Basic Safety Prompts Used in Experiments

C. Prompt and Demonstration Examples Used for Data Synthesis (§ 2.1)

D. Examples of Excluded “Harmless” Queries That Are Potentially Harmful (§ 2.1)

E. Explained Variance Ratios of PCA (§ 2.3)

F. Supplementary Visualization Results with First Two Principal Components (§ 2.3)

Figure 6 presents visualization results for the other four models using the same plotting approach as Figure 3.

  • Figure 6 shows visualization results for four additional models, plotted as in Figure 3.Source: Figure 6 · p. 14

G. Visualization Results with Other Principal Components (§ 2.3)

Across additional principal components, harmful and harmless queries remain difficult to distinguish, and safety prompts do not improve their separation. The supplementary sections describe training details and report evaluations of optimized short safety prompts.

G. Visualization Results with Other Principal Components (§ 2.3)

  • Across the 5th and 6th principal components, harmful and harmless queries remain difficult to distinguish, with observations similar to the 3rd and 4th components.The reported visualizations also show that adding safety prompts does not increase distinguishability.Source: Figure 8 · p. 15, Figure 7 · p. 15

H. Training and Implementation Details of DRO and Vanilla Prompt-Tuning (vPT) (§ 4.1)

  • DRO and vanilla Prompt-Tuning are trained on 200 synthetic data for 40 epochs, using batch size 50 and learning rate 1e-3.The setup uses two Nvidia V100 40GB GPUs for DRO and vanilla Prompt-Tuning training as described in the implementation passage.Source: H. Training and Implementation Details… · p. 16
  • Vanilla Prompt-Tuning combines cross-entropy and unlikelihood losses, with the latter reported as essential for improving safeguarding performance.Its positive and negative response sets are model-generated; the negative set also includes samples without prompts.Source: H. Training and Implementation Details… · p. 16

I. Supplementary Experimental Results (§ 4.2)

  • Supplementary tables report evaluations of the optimized Mistral prompt on MaliciousInstruct, Advbench, and AlpacaEval.The short-prompt evaluations cover MaliciousInstruct and Advbench as well as AlpacaEval.Source: Table 6 · p. 17, Table 7 · p. 17, Table 8 · p. 17
  • For the shortened LLaMA-2 prompt, DRO slightly underperforms on MaliciousInstruct and Advbench and has slightly degraded AlpacaEval performance.The passages attribute these outcomes probably to the shorter prompt’s lower capacity.Source: Table 8 · p. 17, Table 9 · p. 18

J. Breakdowns of Ablation Results for Anchor Data (§ 4.4)

The supplementary materials include ablations broken down by query and basic safety prompt, alongside hidden-state visualizations after DRO optimization on two evaluation benchmarks and a held-out harmless set.

J. Breakdowns of Ablation Results for Anchor Data (§ 4.4)

  • Ablation results are reported separately for queries and for basic safety prompts.The supplied table captions do not specify the ablation outcomes.Source: Table 10 · p. 18, Table 11 · p. 18

K. Visualization Results on Evaluation Benchmarks After DRO Optimization (§ 4.2)

  • Hidden-state visualizations after DRO optimization cover MaliciousInstruct and AdvBench alongside a held-out harmless query set.The figures are described as optimizing the default basic safety prompt.Source: Figure 9 · p. 19, Figure 10 · p. 20

L. Supplementary Results for Interpretability Analysis (§ 4.5)

The supplementary interpretability results include examples of safety prompts, distances between optimized and initial prompt embeddings, and cases where optimized prompts project into slightly different tokens.

L. Supplementary Results for Interpretability Analysis (§ 4.5)

  • The examples include default, Mistral, and shortened safety prompts, with separate snippets concerning incoherent questions and uncertain answers.The passages also include prompt wording about avoiding harmful content and promoting socially unbiased, positive responses.Source: L. Supplementary Results for… · p. 21
  • Among 24 optimized prompts, six project into tokens that slightly differ from the basic prompts.The 24 consist of eight models and three basic safety prompts.Source: Figure 11 · p. 21
  • The supplementary results report average-token Euclidean distances between optimized continuous prompts and the initial basic prompt embeddings.The supplied table caption does not provide distance values.Source: Table 12 · p. 21
Loading 2401.18018v4…