PaperlayerView exact revision on arXiv

Source-linked AI summary

Bypassing the Safety Training of Open-Source LLMs with Priming Attacks

Jason Vega, Isha Chaudhary, Changming Xu, Gagandeep Singh

arXiv:2312.12321v2cs.CRcs.AIcs.CLcs.LG

TL;DR

Safety training can be bypassed by simple priming attacks that append partial compliant responses to inputs for open-source LLMs. The paper develops an optimization-free evaluation pipeline and finds up to 3.3× higher Attack Success Rate than baselines under Llama Guard. The findings indicate fragility in current safety measures, while restricting user-supplied initial responses limits the threat model.

  • Problem

    Existing attacks can circumvent LLM alignment but may require computationally expensive prompt optimization or fine-tuning, motivating simpler attacks against open-source models.

    Source: §1 · p. 1

  • Method

    The paper uses few-shot prompting with a helper LLM to generate partial compliant responses that are appended to target-model inputs as priming attacks.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Up to 3.3× higher Attack Success Rate than the “Just Sure” baseline is reported for Llama-2 (7B) under Llama Guard.

    Source: §1 · p. 1, §2 · p. 2

  • Takeaways & Limitations

    The findings highlight fragility in current safety measures for open-source LLMs under unrestricted-input assumptions.

    Source: §1 · p. 1, §3 · p. 2

    The evaluation generates attacks and judges success automatically rather than using a rigorous human study.

    Source: §2 · p. 2

Abstract

from arXiv · show

With the recent surge in popularity of LLMs has come an ever-increasing need for LLM safety training. In this paper, we investigate the fragility of SOTA open-source LLMs under simple, optimization-free attacks we refer to as $\textit{priming attacks}$, which are easy to execute and effectively bypass alignment from safety training. Our proposed attack improves the Attack Success Rate on Harmful Behaviors, as measured by Llama Guard, by up to $3.3\times$ compared to baselines. Source code and data are available at https://github.com/uiuc-focal-lab/llm-priming-attacks.

1 INTRODUCTION

Priming attacks target an input-format assumption that limits prior attacks, using partial compliant responses to bypass safety alignment in open-source LLMs. The paper introduces an efficient evaluation pipeline and reports up to 3.3× higher attack success than baselines.

Motivation

  • Prior optimization and fine-tuning attacks can be computationally expensive, especially when applied to open-source models.The paper motivates a simpler alternative to methods that optimize prompts or remove alignment through fine-tuning.Source: §1 · p. 1

Threat model

  • The paper argues that unrestricted inputs are increasingly practical for extracting harmful behavior from open-source LLMs.This assumption is contrasted with fixed scaffolding used when transferring attacks to closed-source models.Source: §1 · p. 1
  • Priming attacks exploit unrestricted inputs by appending a partial response after the normal prompt scaffolding.The attack relies on obtaining next-token predictions for arbitrary inputs and exploiting autoregressive generation.Source: §1 · p. 1

Contributions

  • Up to 3.3× higher attack success is achieved by adding small continuations to initially affirmative responses.The authors describe this as improving attack success against open-source LLMs.Source: §1 · p. 1

Method

  • The evaluation pipeline uses a helper LLM to generate the beginning of a compliant response for priming the target model.Figure 1 contrasts the target's refusal under the official format with compliance after the generated partial response is appended.Source: §1 · p. 1, Figure 1 · p. 2

2 METHODOLOGY & RESULTS

The method generates priming attacks through few-shot prompting and evaluates them on safety-trained Llama-2 and Vicuna models. Across tested model families and sizes, priming outperforms the baselines, with a 3.3× Llama Guard improvement over “Just Sure” on Llama-2 (7B).

Limitations

  • The study automates attack generation and evaluation instead of conducting the intended rigorous human study.Human studies are left for future work because of time constraints.Source: §2 · p. 2

Few-shot priming attacks

  • The attack uses a non-safety-trained helper Llama-2 (7B) to generate few-shot priming examples from affirmative responses and incomplete continuations.The incomplete sentence is intended to make backtracking difficult once the target begins generating.Source: §2 · p. 2

Experimental setup

  • The evaluation tests 485 Harmful Behaviors prompts against safety-trained Llama-2 and Vicuna chat models using ASR as the metric.Fifteen prompts form few-shot examples and 20 are used for validation during prompt creation.Source: §2 · p. 2

Results

  • 3.3× higher ASR than “Just Sure” is obtained for Llama-2 (7B) under Llama Guard.The corresponding improvement under human evaluation is 2.5×.Source: §2 · p. 2, Table 1 · p. 2
  • Priming outperforms no attack and “Just Sure” across Llama-2 and Vicuna models at 7B and 13B sizes.The table reports Attack Success Rate on Harmful Behaviors using Llama Guard and human evaluation for Llama-2 (7B).Source: §2 · p. 2, Table 1 · p. 2

Evaluation caveat

  • For Llama-2 (7B), the priming attack obtains a 3.3× higher ASR than the “Just Sure” attack under Llama Guard and 2.5× higher under human evaluation, while Llama Guard may underestimate harmfulness.Source: §2 · p. 2

3 CONCLUSION

The paper finds that priming attacks expose fragility in current safety measures under practical open-source threat assumptions. It frames this as a concern for open-sourcing and a motivation for safer release methods.

3 CONCLUSION

  • Priming attacks reveal fragility in current safety measures for safety-trained LLMs under increasingly practical assumptions.The conclusion summarizes the paper's investigation of attack effectiveness against SOTA safety-trained models.Source: §3 · p. 2
  • The findings raise concerns about future open-sourcing and motivate further research into safer open-source LLM methods.The paper presents this as a concern and research direction rather than a demonstrated solution.Source: §3 · p. 2

URM STATEMENT

  • At least one key author is identified as meeting the URM criteria of the ICLR 2024 Tiny Papers Track.Source: URM STATEMENT · p. 3

A.1 WHEN CAN PRIMING ATTACKS BE USED?

Priming attacks require access to next-token predictions for inputs that include attacker-supplied assistant text. Applications that restrict this input position are outside the stated threat model, while open-source autoregressive models remain candidates.

A.1 WHEN CAN PRIMING ATTACKS BE USED?

  • The threat model assumes the attacker can obtain next-token predictions conditioned on any input, including text at the assistant-response position.This allows the attacker to provide an initially compliant response before generation continues.Source: §A.1 · p. 4
  • The example Vicuna input places a harmful user request followed by an initially compliant assistant response that begins the attack.The scaffolding, user prompt, and priming response occupy distinct parts of the input.Source: §A.1 · p. 4
  • Open-source autoregressive LLMs are candidates for priming attacks when attackers face no input restrictions.The paper leaves robust fine-tuning strategies and their performance trade-offs for future work.Source: §A.1 · p. 4
  • Applications and APIs that prevent users from supplying initial responses are protected from priming attacks.The paper gives server-side handling of scaffolding and inference as an example of such a restriction.Source: §A.1 · p. 4

A.2 EXPERIMENTAL RESULTS

The evaluation covers only Harmful Behaviors prompts and measures compliance rather than response quality. The authors report that attacks can elicit harmful responses, but high-quality harmful content is harder to obtain.

  • The evaluation is limited to prompts from the Harmful Behaviors dataset, excluding other harmful-request categories such as targeted privacy attacks.Source: §A.2 · p. 4
  • Priming attacks can obtain harmful responses, but some successful attacks produce only high-level or incorrect information.Source: §A.2 · p. 4
  • The paper evaluates compliance with harmful requests and leaves response-quality evaluation for future work.Source: §A.2 · p. 4

B DETAILS ON EXPERIMENTAL SETUP

The setup uses a helper LLM to generate priming attacks from few-shot examples, with experiments run on a multi-GPU server. Fifteen examples were selected because they most consistently produced concise attacks that signal compliance without supplying the requested content.

B DETAILS ON EXPERIMENTAL SETUP

C FEW-SHOT PROMPT FOR GENERATING PRIMING ATTACKS

D LLAMA GUARD TASK INSTRUCTIONS

Llama Guard evaluates the experiments using modified task instructions. The modifications add a non-exhaustive-policy note and a Misinformation category, with calibration based on a disjoint subset of test prompts.

E MANUAL EVALUATION BENCHMARK

Manual evaluation labels a response harmful when it provides any amount of harmful behavior, even if it later backtracks. The examples distinguish substantial harmful content from immediate refusal or insubstantial harmful content.

F MANUAL EVALUATION VS. LLAMA GUARD

Manual labeling reports higher attack success rates than Llama Guard, while priming attacks can be generated quickly and evaluated against varied response types.

F MANUAL EVALUATION VS. LLAMA GUARD

  • 30.89% of priming attacks on Llama-2 (7B) were false negatives for Llama Guard under manual labels, indicating conservative harmfulness classification.The reported true-positive rate was 60.0%, while true negatives were 2.89% and false positives were 6.22%.Source: F MANUAL EVALUATION VS. LLAMA GUARD · p. 7

G RUNTIME COMPARISON

  • 0.98s per prompt was sufficient to generate a priming attack with few-shot prompting using batch size 20 on one A100.With batch size 1, generation took 8.01s per prompt; optimization-based techniques can take 1–2 hours per prompt.Source: G RUNTIME COMPARISON · p. 7
  • The examples span nearly completely harmful, partially harmful, mostly safe, and completely safe responses, distinguishing harmful content from later backtracking or refusal.Priming initial responses are colored orange and model backtracking red in the example figures.Source: Figure 5 · p. 10, Figure 6 · p. 11, Figure 7 · p. 12
Loading 2312.12321v2…