PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety Alignment of Large Language Models via Contrasting Safe and Harmful Distributions

Xiaoyun Zhang, Zhengyue Zhao, Wenxuan Shi, Kaidi Xu, Di Huang, Xing Hu

arXiv:2406.16743v2cs.CL

TL;DR

LLM safety alignment can require costly training or manual choices of contrastive models and prompts. ACD optimizes opposing soft prompts for prompt-based contrastive decoding, and experiments report stronger safety than prior training-free decoding methods while preserving generation ability.

  • Problem

    Safety alignment can require high-quality datasets and costly training, while decoding-based contrastive methods rely on manually selected models or instruction templates that limit contrast.

    Source: §1 · p. 1

  • Method

    ACD applies Opposite Prompt Optimization to generate a Safeguarding Prompt and an Adversarial Prompt for prompt-based contrastive decoding without training the target model.

    Source: §1 · pp. 1–2

  • Results

    ACD improves safety by over 20% versus secure system prompt baselines and outperforms Instructive Decoding by 7%, while preserving generative performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present ACD as a lightweight approach to improving LLM safety alignment without heavy model training while maintaining generation quality.

    Source: §5 · p. 8

    ACD processes two inputs per inference, increasing inference overhead.

    Source: E Limitations · p. 12

Abstract

from arXiv · show

With the widespread application of Large Language Models (LLMs), it has become a significant concern to ensure their safety and prevent harmful responses. While current safe-alignment methods based on instruction fine-tuning and Reinforcement Learning from Human Feedback (RLHF) can effectively reduce harmful responses from LLMs, they often require high-quality datasets and heavy computational overhead during model training. Another way to align language models is to modify the logit of tokens in model outputs without heavy training. Recent studies have shown that contrastive decoding can enhance the performance of language models by reducing the likelihood of confused tokens. However, these methods require the manual selection of contrastive models or instruction templates, limiting the degree of contrast. To this end, we propose Adversarial Contrastive Decoding (ACD), an optimization-based framework to generate two opposite soft system prompts, the Safeguarding Prompt (SP) and the Adversarial Prompt (AP), for prompt-based contrastive decoding. The SP aims to promote safer outputs while the AP aims to exploit the harmful parts of the model, providing a strong contrast to align the model with safety. ACD only needs to apply a lightweight prompt tuning on a rather small anchor dataset without training the target model. Experiments conducted on extensive models and benchmarks demonstrate that the proposed method achieves much better safety performance than previous model training-free decoding methods without sacrificing its original generation ability.

1 Introduction

ACD uses optimized opposing soft prompts to strengthen prompt-based contrastive decoding for safety. The reported experiments show improved safety over secure-prompt baselines and Instructive Decoding while preserving generation performance.

1 Introduction

  • Existing training-based alignment can require high-quality human preference data and substantial training costs, while safety may decline after downstream fine-tuning.The cited work calls for safety realignment after each downstream task fine-tuning.Source: §1 · p. 1
  • Prior decoding approaches use a weaker contrastive model or manually designed instructions, and different opposite-prompt designs can change safety performance.The authors propose embedding-space intervention as a way to obtain stronger contrast.Source: §1 · p. 1
  • Opposite Prompt Optimization separately tunes a Safeguarding Prompt for safer responses and an Adversarial Prompt that elicits harmful outputs, forming ACD's inference-time contrast.The adversarial prompt is intended to expose unsafe parts of the output distribution and make the contrast more pronounced.Source: §1 · pp. 1–2
  • ACD improves safety by over 20% versus secure system prompt baselines and outperforms Instructive Decoding by 7%, with negligible tuning overhead and preserved generative performance.Experiments span multiple LLMs and established red-teaming benchmarks.Source: §1 · p. 2

2 Related Work

Related work covers training-based safety alignment and decoding-time guidance, including approaches that use auxiliary models and approaches that guide the original model directly.

2.1 Safety Alignment of LLMs

  • Safety alignment work includes RLHF and variants, preference-training methods, multilingual transfer, dialogue refusal, and approaches using internal representations or safety constraints.Examples include RLAIF, Safe RLHF, DPO, MPO, DeRTa, and Safety Polytope.Source: §2.1 · p. 2

2.2 Guided Decoding as Alignment

  • Some guided-decoding methods use an auxiliary model, reward model, or candidate scorer to alter or rank outputs during inference.Examples include Contrastive Decoding, SafeDecoding, Proxy Tuning, ARGS, NUDGING, GSI, SRR, and PITA.Source: §2.2 · p. 3
  • Model-free guidance includes self-evaluation with tree search, instruction adjustments for contrast, and inference-time reward or bandit formulations.Instructive Decoding and ROSE use instruction-based contrast; other approaches include RAIN and OPAD.Source: §2.2 · p. 3

3 Adversarial Contrastive Decoding

ACD optimizes opposing Safeguarding and Adversarial soft prompts on a small anchor dataset, then uses their response logits for prompt-based contrastive decoding. The prompts are initialized manually and optimized with objectives that encourage safe refusals or expose unsafe responses.

3.1 Overview

  • ACD uses two optimized soft prompts as universal system prompts, producing safe and adversarial response logits that are combined for contrastive decoding.Optimization is performed once per model with minimal computational overhead, and the prompts can be concatenated with user-instruction embeddings during interaction.Source: §3.1 · p. 3

3.2 Opposite Prompt Optimization

  • The Safeguarding Prompt starts from a manual safe prompt and is optimized to reject harmful requests while avoiding mistaken refusals of safe instructions.Its objective uses rejected harmful responses as positive samples, accepted harmful responses as negative samples, and safe-instruction data as a constraint.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
  • The Adversarial Prompt uses an opposite objective, encouraging responses to harmful instructions to expose the model’s unsafe output distribution.The optimization uses harmful-instruction anchor data where the model accepts harmful requests as positive samples and where it rejects them as negative samples, while also constraining performance on safe instructions.Source: §3.2 · p. 4

3.3 Prompt-based Contrastive Decoding

  • At inference, ACD concatenates each optimized prompt with the instruction embedding, obtains safe and adversarial logits, and uses both to derive final sampling logits.Table 1 reports harmless rates across multiple models and benchmarks, comparing ACD with a regular safe-prompt baseline and null- and opposite-prompt Instructive Decoding.Source: §3.3 · p. 4, Table 1 · p. 5

4 Experiments

Across multiple models and benchmarks, ACD improves safety over regular decoding and generally over Instructive Decoding, while not significantly affecting general-task performance. Comparisons, RLHF tests, and ablations examine its broader performance and the effects of its prompts, coefficient, and anchor data.

4.1 Experimental Settings

  • Experiments cover seven models and five safety benchmarks, with 100 harmful queries sampled per benchmark; safety, general ability, and truthfulness are assessed using HLR, winR, and trueR.The models include weakly and strongly safety-aligned models; the baselines include regular decoding and null-prompt and opposite-prompt Instructive Decoding.Source: §4.1 · pp. 5–6

4.2 Main Results

  • ACD improves HLR across almost all models and benchmarks versus regular decoding, generally beating Instructive Decoding; selected weakly aligned models gain over 25% HLR on average.The reported average gain applies to Llama-2-uncensored-7b and Bloom-7b, whose original safety is around 50%, without training model parameters.Source: §4.2 · p. 6

4.3 Comparison with More Baselines

  • ACD outperforms ICD, Self-Reminder, Safety Tuning, and SafeDecoding across benchmarks.Table 3 reports HLR for ACD and other baselines; the passage notes that training-based Safety Tuning may require general data to avoid wrong refusals.Source: Table 3 · p. 6, §4.3 · p. 6

4.2 Main Results

  • ACD does not significantly impact model performance on AlpacaEval and TruthfulQA general tasks.Helpfulness was assessed on sampled AlpacaEval instructions, and truthful rate on sampled TruthfulQA instructions.Source: §4.2 · p. 6

4.4 Effectiveness of ACD on RLHF-tuned LLMs

  • On RLHF-tuned Llama-2-7b-chat and Llama-3-8b-Instruct, ACD consistently increases safety across benchmarks, with marginal gains from their already strong safety alignment.Table 5 compares HLR for ACD and base decoding on RLHF-tuned models.Source: §4.4 · p. 7, Table 5 · p. 8

4.5 Ablation Study

  • Optimized SP and AP produce stronger safety contrast than manual or null prompt substitutions, which reduce HLR; null prompt plus AP yields approximately 0.2 safety performance.The optimized SP is safer and the optimized AP riskier than their manual counterparts, as reflected in Figure 4.Source: §4.5 · p. 7, Figure 4 · p. 7
  • Across three benchmarks, safety first rises and then falls as α increases; the authors recommend a moderate α, such as 0.4 or 0.5.A too-small α inadequately removes negative probabilities, while a too-large α suppresses effective candidate-token probabilities.Source: §4.5 · pp. 7–8, Figure 5 · p. 7
  • Anchor responses sampled from different models can support OPO and ACD safety gains, suggesting the prompts capture general rejection or acceptance behavior rather than particular response texts.The experiments used anchor data sampled from Llama-2-uncensored-7b, Llama-3-uncensored-8b, and Guanaco-7b.Source: §4.5 · p. 8

5 Conclusion

ACD combines Opposite Prompt Optimization with prompt-based contrastive decoding, using two opposing soft prompts to strengthen contrast during inference. The experiments report improved safety alignment without heavy model training while maintaining generation quality.

5 Conclusion

  • ACD uses the Safeguarding Prompt and Adversarial Prompt to build strong contrast during inference.The framework pairs prompt-based contrastive decoding with Opposite Prompt Optimization.Source: §5 · p. 8
  • Extensive experiments show that ACD improves LLM safety alignment without heavy model training.Source: §5 · p. 8
  • ACD maintains generation quality while improving safety alignment.Source: §5 · p. 8

A Implementation Details

Prompt initialization uses a standard Llama-2 system prompt for safety and modifies it into a threat prompt intended to elicit compliance.

Details of Prompt Initialization

Details of Experiments

The experiments construct an anchor dataset from sampled responses labeled for refusal, then optimize the opposing prompts with Adam and use a fixed decoding setup.

Settings of Anchor Dataset

  • Each anchor example contains an instruction, three responses sampled under safe, opposite, and null prompts, and refusal labels.Source: Details of Experiments · p. 11
  • ChatGPT labels sampled responses as refused or accepted for the harmful-refused, harmful-accepted, and safe-response data groups.Llama-2-uncensored-7b generally supplies samples; Llama-3-uncensored-8b uses its own samples for comparison.Source: Details of Experiments · p. 11

Settings of Opposite Prompt Optimization

  • Prompt optimization uses Adam with batch size 16, learning rate 1 × 10−4, and five epochs.Loss terms are reweighted using the data ratios in DHR, DHA, and DS.Source: Details of Experiments · p. 11

Settings of Prompt-based Contrastive Decoding

  • Prompt-based contrastive decoding fixes α at 0.5 and uses greedy search for stability.The reported ablation found that α values that were too small or too large were unsuitable for safety.Source: Details of Experiments · p. 11

Details of Evaluation

The evaluation uses ChatGPT or GPT-4 assessments for safety and general ability, with jailbreak testing measuring harmful responses after attacks against previously safe outputs.

Details of Evaluation

  • Harmless Rate assesses safety by counting harmless responses, with ChatGPT checking each response against its instruction.Source: Details of Evaluation · p. 11
  • AlpacaEval win rate compares target-model outputs with text-davinci-003 or GPT4 using ChatGPT judgments.Source: Details of Evaluation · p. 11
  • TruthfulQA truthful rate uses GPT-4 to judge alignment with world knowledge, with the dataset’s Best Answer included as a reference.Source: Details of Evaluation · p. 11

Jailbreak Prompts

  • Attack Success Rate is the ratio of instructions producing harmful responses after jailbreak but safe responses before it.Jailbreak prompts are listed in Table 12; HLR is the same as in the safety experiments.Source: Jailbreak Prompts · p. 11

B Additional Experimental results

ACD is reported to reduce jailbreak risk on Mistral-7b-Instruct, with the optimized Adversarial Prompt described as an embedding-level jailbreak whose risks are subtracted during contrastive decoding.

Jailbreak Prompts

  • ACD significantly improves Mistral-7b-Instruct’s robustness against jailbreak attacks, as evaluated on AdvBench.The evaluation uses role-play-based AIM and instruction-following-based Refusal Suppression prompts, measuring Attack Success Rate.Source: Jailbreak Prompts · p. 11
  • The evaluation tests two jailbreak prompts: role-play-based AIM and instruction-following-based Refusal Suppression.Attack Success Rate measures the proportion of additional harmful responses after the jailbreak.Source: Jailbreak Prompts · p. 11
  • The optimized Adversarial Prompt acts as an embedding-level jailbreak, and subtracting its risks during contrastive decoding can reduce harmful-output probability.Source: Jailbreak Prompts · p. 11

C Complexity Analysis and Optimization

ACD processes two prompt-prefixed sequences for each instruction, motivating analysis of inference time and KV-cache memory; case studies report harmless responses to harmful instructions and helpful responses to harmless ones.

Time Complexity

  • Serial inference processes the two sequences sequentially, while GPU-parallel processing computes them together at the same theoretical time complexity as one sequence.For input length n and prefix length k, serial complexity is O(2(n + k)^2 · d); parallel complexity is O((n + k)^2 · d).Source: Time Complexity · p. 12

Space Complexity

  • Without optimization, storing both sequences’ KV caches doubles memory requirements to O(2(n + k) · d · h).Here, h is the number of attention heads.Source: Space Complexity · p. 12

Problem Formulation

Shared Computation Strategy

  • The shared-cache strategy is presented as reducing memory relative to the naive implementation, but the supplied passage gives no reduction value.Source: Shared Computation Strategy · p. 12

D Case Study

  • Case studies show ACD giving harmless responses to harmful instructions that safe-prompt-only base decoding does not refuse.Source: D Case Study · p. 12
  • For harmless instructions, ACD gives helpful responses, which the paper presents as evidence that benign usage is not affected.Source: D Case Study · p. 12

E Limitations

The paper identifies inference overhead, possible safety–performance trade-offs on edge cases, and untested prompt stability under model updates and language drift as limitations of ACD.

E Limitations

  • ACD processes two inputs per inference, increasing inference overhead; improving efficiency is identified as future work.Source: E Limitations · pp. 12–13
  • Edge cases or specific tasks may show a more pronounced trade-off between safety and general-task performance.Source: E Limitations · p. 12
  • The stability and long-term effectiveness of optimized prompts under continuous model updates and language-use drift remain unexplored.The paper lists robust usage as an area for future improvement.Source: E Limitations · pp. 12–13

Related papers

Loading 2406.16743v2…