PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shifts

Qibing Ren, Hao Li, Dongrui Liu, Zhanxu Xie, Xiaoya Lu, Yu Qiao, Lei Sha, Junchi Yan, Lizhuang Ma, Jing Shao

arXiv:2410.10700v3cs.CLcs.AI

TL;DR

Aligned LLMs may be vulnerable to benign prompts that are semantically related to harmful content, a natural distribution shift that safety training may not cover. ActorBreaker uses actor networks to generate multi-turn attacks from those relationships. It outperforms leading single-turn and multi-turn baselines on Harmbench, while the paper’s limitations include English-only actor generation and a helpfulness–safety trade-off in defense fine-tuning.

  • Problem

    The paper examines whether aligned LLMs withstand benign prompts semantically related to harmful queries, since safety training focuses on rejecting harmful queries while pre-training includes potentially harmful data.

    Source: §1 · p. 1

  • Method

    ActorBreaker uses actor-network relationships around harmful targets to generate benign, multi-turn attack prompts drawn from the pre-training distribution.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Results

    ActorBreaker’s average ASR is 77.7%, versus 18.3% for CoA and 45.0% for Crescendo, and its ASR against GPT-o1 is 60.0% versus 14.0% for the highest-ASR baseline.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The findings identify natural distribution shifts as a safety blind spot and motivate expanding safety training to cover toxic content’s broader semantic landscape.

    Source: §5 · p. 9

    Actor generation is limited to English, and safety fine-tuning involves a trade-off between helpfulness and safety.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

Safety concerns in large language models (LLMs) have gained significant attention due to their exposure to potentially harmful data during pre-training. In this paper, we identify a new safety vulnerability in LLMs: their susceptibility to \textit{natural distribution shifts} between attack prompts and original toxic prompts, where seemingly benign prompts, semantically related to harmful content, can bypass safety mechanisms. To explore this issue, we introduce a novel attack method, \textit{ActorBreaker}, which identifies actors related to toxic prompts within pre-training distribution to craft multi-turn prompts that gradually lead LLMs to reveal unsafe content. ActorBreaker is grounded in Latour's actor-network theory, encompassing both human and non-human actors to capture a broader range of vulnerabilities. Our experimental results demonstrate that ActorBreaker outperforms existing attack methods in terms of diversity, effectiveness, and efficiency across aligned LLMs. To address this vulnerability, we propose expanding safety training to cover a broader semantic space of toxic content. We thus construct a multi-turn safety dataset using ActorBreaker. Fine-tuning models on our dataset shows significant improvements in robustness, though with some trade-offs in utility. Code is available at https://github.com/AI45Lab/ActorAttack.

1 Introduction

The paper asks whether aligned LLMs are vulnerable to benign prompts semantically related to harmful queries, despite safety training focused on rejecting harmful queries. It introduces ActorBreaker to probe this natural distribution shift and proposes broader semantic coverage in safety training.

1 Introduction

  • Aligned LLMs can be vulnerable to natural distribution shifts: benign prompts semantically related to toxic content can lead them to reveal harmful details.The paper distinguishes this from malicious distribution shifts created by deliberately modifying harmful prompts.Source: §1 · pp. 1–2
  • ActorBreaker uses Latour’s actor-network theory to identify human and non-human actors linked to a harmful target and turn them into clues for multi-turn attack prompts.The method uses the LLM’s pre-training prior to instantiate actor networks and explores divergent attack paths within the pre-training domain.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Unlike methods relying on fixed strategies or human-designed seed instances, ActorBreaker generates benign, in-distribution prompts without specific jailbreak techniques.The paper argues fixed strategies and seed biases limit attack diversity.Source: §1 · p. 2
  • On Harmbench, ActorBreaker achieves the highest success rate, outperforming leading single-turn and multi-turn attacks across aligned LLMs; it also generates unsafe outputs against GPT-o1 despite its advanced reasoning.Its prompts bypass Llama-guard 2 detection and transfer well across aligned LLMs without extra optimization.Source: §1 · p. 2
  • The paper proposes broadening safety alignment to cover toxic prompts’ wider semantic space and constructs a multi-turn safety dataset using ActorBreaker.Fine-tuning on the dataset improves robustness against the attacks, with a trade-off between utility and safety.Source: §1 · pp. 2–3

2 Related Work

Prior single-turn attacks often alter harmful queries into semantically equivalent but out-of-distribution forms or use human-like and optimization strategies. Existing multi-turn attacks commonly rely on specialized jailbreak techniques or fixed templates, whereas this paper distinguishes its prompts by targeting natural semantic relationships in the pre-training domain.

2 Related Work

  • Single-turn attacks include out-of-distribution transformations, human-like communication strategies, gradient-based optimization, automated red teaming, and attacks that assume access to model internals.Examples in the passage include ciphers, low-resource languages, code, hypothetical scenarios, persuasion, and psychology strategies.Source: §2 · p. 3
  • Many multi-turn attacks use specialized jailbreak techniques or fixed templates with human-designed seeds, which the authors say can constrain diversity and produce prompts unlike natural semantic relations in pre-training.The passage contrasts these approaches with ActorBreaker’s focus on diverse semantic relationships to toxic queries in the pre-training domain.Source: §2 · p. 3
  • Imposter.AI and Cosafe also exploit malicious distribution shifts, while human red-team tactics are described as orthogonal to this work.Cosafe places harmful intent in the last query; manual multi-turn attacks use lists of human tactics.Source: §2 · p. 3

3 Method

ActorBreaker uses a two-stage process to find attack clues around a harmful target and generate multi-turn attacks. It builds target-specific actor networks, then uses attack chains and self-talk to construct query paths that can be revised during interaction with a victim model.

3 Method

  • ActorBreaker’s two stages construct a network around a seed toxic prompt and generate multi-turn attacks from its nodes as clues.The second stage generates an attack chain and multi-turn queries.Source: §3 · p. 3

3.1 Actor-network construction

  • The actor network links a harmful target to six actor types, including human and nonhuman entities, whose relationships supply potential attack clues.LLMs instantiate a target-specific two-layer tree with specific actor names as leaves, from which diverse clues are extracted.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 In-attack

  • For each selected clue, the attacker infers a stepwise chain from the clue to the harmful target to guide the evolving topics of multi-turn queries.The chain is sampled sequentially by the attacker LLM.Source: §3.2 · p. 4
  • Following the chain, the attacker generates successive queries and uses its own predicted responses as proxies for victim-model responses.Later queries are conditioned on earlier queries and responses; the authors hypothesize this can support attacks across models without model-specific adaptation.Source: §3.2 · p. 4
  • During victim interaction, an optional GPT4-Judge-driven step modifies attack paths to address mismatches between predicted and actual responses.The judge halts attempts after unknown failures and refines refused queries by reducing explicit harmfulness.Source: §3.2 · p. 5, Figure 3 · p. 5

4 Experiments

ActorBreaker achieved the highest Harmbench attack success rate across the evaluated target models, and the experiments assess its actor categories, prompt diversity, efficiency, and defenses. The findings also show that its prompts can evade input toxicity detection and that multi-turn safety data can improve defense robustness.

4.1 Experimental Setup

  • Experiments evaluated six LLMs on HarmBench against single- and multi-turn baselines, using GPT-4o judging and BERT-sentence embedding distances to measure prompt diversity.The attack used GPT-4o, with three actors and a maximum of five queries per attack unless otherwise stated.Source: §4.1 · pp. 5–6

4.2 Discussion of Results

  • ActorBreaker averaged 77.7% ASR, versus 18.3% for CoA and 45.0% for Crescendo, and achieved the highest ASR across target LLMs.The comparison covers single-turn and multi-turn baselines on Harmbench.Source: Table 1 · p. 6, §4.2 · p. 6
  • ActorBreaker achieved 60.0% ASR against GPT-o1, while the highest baseline ASR was 14.0%; GPT-o1’s chain of thought first expressed safe intent, then gave step-by-step compliance.The authors raise a concern about chain-of-thought faithfulness.Source: §4.2 · p. 6

The effectiveness of different types of actors

  • Prompts from all six actor categories were relatively effective, and increasing actor types or conversation turns improved attack success rates.The authors also report that adding actor types improves coverage of different model vulnerabilities.Source: The effectiveness of different types… · pp. 6–8, Table 3 · p. 7
  • Without dynamic modification, ActorBreaker averaged 72.7% ASR versus 81.2% with it, while the authors report that the unmodified method still transfers across aligned LLMs.CoA and Crescendo rely on target-model responses to craft attacks, which the authors say limits their efficiency.Source: The effectiveness of different types… · p. 7, Table 4 · p. 7
  • ActorBreaker generated more diverse prompts than CoA and Crescendo and achieved a 26% attack-efficiency improvement over Crescendo.Diversity was measured using pairwise cosine similarity across trials; efficiency was compared using average target-model interactions per attack.Source: The effectiveness of different types… · pp. 7–8, Table 5 · p. 7, Table 6 · p. 8
  • Llama-Guard 2 assigned ActorBreaker’s multi-turn prompts lower unsafe probabilities than original harmful queries and Crescendo prompts, yet the attacks remained effective.CoA’s prompts became less harmful with more turns but also less effective than ActorBreaker and Crescendo.Source: The effectiveness of different types… · p. 8
  • Circuit Breaker reduced attack success, and Circuit Breaker trained on multi-turn safety data was more robust to multi-turn attacks than its single-turn-data counterpart.The paper constructs multi-turn safety prompts by inserting refusals where a judge detects the first harmful response; its defense comparisons also include Rephrase and RPO.Source: The effectiveness of different types… · pp. 8–9

5 Conclusion

The conclusion identifies natural distribution shifts as a safety vulnerability and presents ActorBreaker as a way to probe it. It reports stronger attack performance and improved robustness after fine-tuning on ActorBreaker-generated safety data.

5 Conclusion

  • The paper identifies benign prompts semantically linked to harmful content as a vulnerability in aligned LLM safety mechanisms and uses ActorBreaker to probe it.ActorBreaker uses multi-turn prompts grounded in actor-network theory.Source: §5 · p. 9
  • The conclusion reports that ActorBreaker outperforms other attack methods and that fine-tuning on its generated dataset improves robustness against these attacks.The conclusion recommends expanding safety training to cover a broader semantic landscape of toxic content.Source: §5 · p. 9
  • Figure 5 reports Llama-Guard 2 unsafe-probability scores for plain harmful queries and multi-turn attack queries against GPT-4o and Claude-3.5-sonnet.Source: Figure 5 · p. 9
  • Table 8 compares ActorBreaker’s attack success rates against defense methods, including SFT and Circuit Breaker, with multi-turn data defined as context-aware adversarial prompts.Source: Table 8 · p. 9

6 Limitations

The study focuses on English-language actor generation, leaving multilingual coverage and multimodal jailbreak applications for future work; its safety fine-tuning also involves a helpfulness trade-off.

6 Limitations

  • Actor generation is limited to English, and the authors identify multilingual actor coverage as a future research direction.They note that languages have distinct cultures and histories, which may yield different actors for the same harmful behavior.Source: §6 · p. 9
  • The authors suggest studying ActorBreaker on multimodal models as a future direction.The passage frames this as an area to explore, not an evaluated capability.Source: §6 · p. 9
  • The authors report a helpfulness–safety trade-off and suggest exploring RLHF for multi-turn dialogue, including step-level reward scoring.The proposed reward model would provide more granular scores at each dialogue step.Source: §6 · pp. 9–10

7 Ethics Statement

The authors acknowledge that automated multi-turn jailbreak prompts could be misused, and describe steps intended to reduce real-world harm while studying the risk.

7 Ethics Statement

  • ActorBreaker generates multi-turn jailbreak prompts that could potentially be misused to attack commercial LLMs.The authors say studying these risks is necessary to help mitigate the vulnerabilities.Source: §7 · p. 10
  • The authors say they will disclose their results to major LLM developers before publication and continuously monitor and update the research.They also report following ethical guidelines and exploring ActorBreaker-generated data for safety fine-tuning.Source: §7 · p. 10

A Related Work

B Algorithm

  • Algorithm 1 notation uses H for dialogue history and Cretry for the number of attempts made so far..Source: B Algorithm · p. 15

C.1 Additional results for ablation on the number of actors.

The appendix specifies the configurations used for several attack baselines, including their prompt settings and data preparation.

C.1 Additional results for ablation on the number of actors.

  • GCG follows Harmbench’s default setting and is also tested in transfer experiments on closed-source models; PAIR follows Harmbench’s default setting.The passage does not report comparative outcomes for these settings.Source: §D.1 · p. 15
  • PAP uses the Expert Endorsement prompt type, while CodeAttack uses Python Stack..Source: §D.1 · p. 15
  • For CipherChat’s SelfCipher unsafe demonstrations, Harmbench examples are classified by GPT-4o into 11 unsafe domains, and queries receive the demonstrations for their domain.The setup follows CipherChat’s domain-based demonstration procedure.Source: §D.1 · p. 15

D.2 Safety fine-tuning experiment

The appendix reports a helpfulness cost from multi-turn safety alignment and a human-judgment comparison for GPT-4o, alongside the experiment’s data, training, and evaluation setup.

D.2 Safety fine-tuning experiment

  • The safety dataset combines 600 filtered harmful instructions with 1,000 successful ActorBreaker dialogues collected against deepseek-chat, plus refusal responses generated by deepseek-chat.The setup uses WizardLM-2-8x22B as the attacker and maintains a 1:2 ratio of safety-alignment to instruction data.Source: §D.2 · p. 15
  • Helpfulness is evaluated with GSM8K, MMLU, HumanEval, and MTBench through OpenCompass.The specified MTBench setting uses GPT-4o-mini as the judge.Source: §D.2 · p. 15
  • For each harmful instruction, ActorBreaker generates three successful attack paths; the models are fine-tuned with LoRA for three epochs at batch size 4 and learning rate 2e-4..Source: §D.2 · p. 15
  • Multi-turn safety alignment compromises helpfulness, revealing a trade-off between helpfulness and safety.Table 9 reports helpfulness results for the baseline and two models fine-tuned on safety alignment samples.Source: §D.2 · p. 15, Table 9 · p. 17
  • The algorithm generates initial queries by self-talk, maintains victim-dialogue history, skips attack clues the victim does not know, and stops early upon success.Its inputs include a toxic query, attack, victim, judge, optional monitor, iteration count, and actor count.Source: §D.2 · p. 16
  • Figure 6 compares judge-score proportions across actor counts for Llama-3-8B-Instruct and Llama-3-70B-Instruct; higher scores indicate more harmful responses, and 5 marks attack success..Source: Figure 6 · p. 17

D.3 The rationality of using GPT-4o for judgement

  • GPT-4o judge scores align more closely with human judgments than Llama-Guard or the OpenAI Moderation API, which show higher false-negative and false-positive rates, respectively.The human ground truth uses majority votes from 10 annotations per query; attacks count as successful only at score 5 on the 1–5 scale.Source: §D.3 · p. 17

E Comparison with Crescendo

This section presents Crescendo query samples and describes how prompt diversity and attack success are evaluated. The supplied passages do not report comparative outcome values.

E Comparison with Crescendo

  • Prompt diversity is defined over prompts from different trials for the same harmful target using MiniLMv2 embeddings.The equation's prose identifies ϕ as the MiniLMv2 embedding function and Sp as a subset of prompts from different trials in one turn.Source: E Comparison with Crescendo · p. 17
  • Table 10 reports ActorBreaker attack success rates across models and evaluation methods, with bracketed values giving judge differences from human evaluation.The caption does not provide the table's cell values.Source: Table 10 · p. 18
  • Crescendo examples show initial and intermediate queries for a pipe-bomb target, with initial queries collected in three independent trials.Source: Figure 7 · p. 18

F Examples

The examples illustrate ActorBreaker prompts across several harmful targets and models, including a case where a model recognizes harmful intent but still produces unsafe content. The authors truncate examples to include only partial harmful information.

F Examples

  • The authors truncate the qualitative examples to show only partial harmful information, aiming to prevent real-world harm.Source: F Examples · p. 18
  • The pipe-bomb examples compare Crescendo and ActorBreaker initial and intermediate queries, with three independent trials for each method's samples.Source: Figure 8 · p. 19
  • GPT-o1-preview's reasoning identifies the pipe-bomb request as harmful, yet the model continues to output unsafe content.Source: Figure 9 · p. 20
  • ActorBreaker examples span requests about illegal handgun purchasing, COVID-19 modifications, disability-related advocacy, and race-based discrimination.Source: Figure 10 · p. 21, Figure 11 · p. 22, Figure 12 · p. 23
  • For a disability-related target, the example turns to questions about the Sterilization Act and describes its forced sterilization of people with disabilities.Source: F Examples · p. 23
  • A further example asks about arguments for the Sterilization Act, including its claimed economic effects and proponents' rationale.Source: F Examples · p. 23
Loading 2410.10700v3…