PaperlayerView exact revision on arXiv

Source-linked AI summary

Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue

Zhenhong Zhou, Jiuyang Xiang, Haopeng Chen, Quan Liu, Zherui Li, Sen Su

arXiv:2402.17262v2cs.CLcs.AI

TL;DR

Prior work has largely focused on single-turn jailbreaks, leaving safety risks in multi-turn dialogue less examined. The paper decomposes malicious queries into sub-queries and uses the resulting dialogue to elicit harmful content incrementally. Experiments indicate vulnerabilities in current LLM safety mechanisms across multi-turn scenarios.

  • Problem

    Prior safety research has predominantly focused on single-turn dialogue, while multi-turn dialogue is an important way people use LLMs and has received less attention.

    Source: §1 · pp. 1–2

  • Method

    The paper decomposes malicious queries into less harmful sub-queries, elicits responses across turns, and then inverts or combines prior responses.

    Source: §1 · p. 2

  • Results

    Experiments across aligned commercial LLMs indicate that current models have safety vulnerabilities that malicious users can exploit in multi-turn dialogue.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors argue that LLM safety needs to account specifically for multi-turn dialogue and that mitigation measures are timely.

    Source: §6 · p. 8

    The study demonstrates that multi-turn jailbreak risks may exist but has not optimized the attack specifically for multi-turn dialogue.

    Source: §7 · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) have been demonstrated to generate illegal or unethical responses, particularly when subjected to "jailbreak." Research on jailbreak has highlighted the safety issues of LLMs. However, prior studies have predominantly focused on single-turn dialogue, ignoring the potential complexities and risks presented by multi-turn dialogue, a crucial mode through which humans derive information from LLMs. In this paper, we argue that humans could exploit multi-turn dialogue to induce LLMs into generating harmful information. LLMs may not intend to reject cautionary or borderline unsafe queries, even if each turn is closely served for one malicious purpose in a multi-turn dialogue. Therefore, by decomposing an unsafe query into several sub-queries for multi-turn dialogue, we induced LLMs to answer harmful sub-questions incrementally, culminating in an overall harmful response. Our experiments, conducted across a wide range of LLMs, indicate current inadequacies in the safety mechanisms of LLMs in multi-turn dialogue. Our findings expose vulnerabilities of LLMs in complex scenarios involving multi-turn dialogue, presenting new challenges for the safety of LLMs.

1 Introduction

The introduction frames multi-turn dialogue as an overlooked safety risk: malicious users can split harmful requests into sub-queries that elicit harmful content incrementally. The paper proposes a decomposition approach and reports vulnerabilities across aligned commercial LLMs.

1 Introduction

  • Multi-turn dialogue can build a comprehensive harmful response from sub-questions that each appear less harmful or loosely related.The authors connect this risk to safety alignment focused on single interactions or on multi-turn dialogue within one topic.Source: §1 · pp. 1–2
  • The paper introduces malicious-query decomposition, followed by cautionary or mildly harmful responses and a final turn that inverts or combines earlier outputs.The approach uses multi-turn responses without requiring training.Source: §1 · p. 2
  • Experiments across commercial LLMs including ChatGPT, Claude, and Gemini indicate that current models have exploitable safety vulnerabilities in multi-turn dialogue.The paper also analyzes causes and proposes potential mitigation strategies.Source: §1 · p. 2

2 Background

The background describes LLM assistants’ multi-turn capabilities, alignment methods intended to support safe responses, and jailbreak research that tests their remaining vulnerabilities.

2.1 LLM-based AI Assistant

  • LLM assistants can follow instructions across multiple turns, including across unrelated topics, and help users address complex problems incrementally.The passage names ChatGPT4, Claude2, and Gemini as examples.Source: §2.1 · pp. 2–3

2.2 Alignment

  • Alignment methods including RLHF, DPO, RAIN, and LLAMA Guard are used during fine-tuning or generation to reduce harmful responses.The paper notes that models learn harmful knowledge from unfiltered training data and that next-word prediction does not ensure alignment with human values.Source: §2.2 · p. 3

2.3 LLM Jailbreak

  • Jailbreak research includes hand-crafted prompts and methods such as GCG, AutoDan, MasterKey, HouYi, and DeepInception to circumvent alignment or expose safety risks.Figure 3 presents a malicious-query decomposition paradigm, with manual instructions and an automatic approach using GPT-4.Source: §2.3 · p. 3, Figure 3 · p. 3

3 Safety Vulnerability of LLMs in Multi-Turn Dialogue

The paper presents a multi-turn jailbreak paradigm in which decomposed malicious questions elicit harmful content incrementally. It describes manual and automated query decomposition, then explains how accumulated dialogue context can yield harmful outputs despite turn-level alignment.

3.1 Malicious Query Decomposition

  • Malicious Query Decomposition disguises harmful intent by turning a malicious objective into low-level sub-queries that are harder to recognize than a single-turn request.The method includes purpose inversion, keyword replacement, cautionary orientation, and sentence reframing; the queries are intended to avoid rejection and collectively form a harmful dialogue.Source: §3.1 · pp. 3–4
  • Manual decomposition elicits responses to superficially unrelated subtopics over multiple turns, building background knowledge and opposing viewpoints.The authors note that manual decomposition relies on expertise.Source: §3.1 · p. 4
  • Automatic decomposition uses GPT-4 with successful manual examples as few-shot demonstrations and a transfer prompt to generate sub-query prompts.The paper describes this as a low-barrier way to produce query sets rapidly and withholds the specific prompts for ethical reasons.Source: §3.1 · p. 4

3.2 Safety Vulnerability in Multi-Turn Dialogue

  • The final turn inverts or combines earlier model generations to aggregate harmful knowledge, though alignment can leave the answer ambiguous or short of detail.The dotted rejection threshold in Figure 2 is illustrative, not a rigid criterion.Source: §3.2 · pp. 4–5
  • Multi-turn dialogues can aggregate harmful knowledge in the final turn even when alignment is successful on earlier turns, producing an overall alignment failure.The authors identify intent shifts and instruction following as especially challenging multi-turn scenarios.Source: §3.2 · p. 5

4 Experiments

Experiments across commercial models find harmful multi-turn dialogues, with role-play and increasing turn counts associated with greater harmfulness. Combining sub-queries into one turn reduces harmfulness judgments and scores, indicating the dialogue format contributes beyond decomposition alone.

4.1 Setups

  • The experiments use commercial models and manually or automatically decomposed AdvBench prompts, with manual decompositions averaging 2.43 questions and a maximum of four.Automatically decomposed prompts averaged two questions per decomposition.Source: §4.1 · p. 5

4.2 Evaluation

  • Dialogue safety is evaluated with LLAMA Guard’s safe/unsafe judgment and GPT-4 ratings from 1 to 5 plus a whole-conversation harmfulness flag.The Moderation API is also used to assess alignment with OpenAI moderation standards.Source: §4.2 · pp. 5–6

4.3.1 Baseline Harmfulness Evaluation

  • Claude performed well under Moderation API review on manually decomposed prompts, while gpt-4-0125-preview passed review across all datasets.Claude2 dialogues passed the moderation review entirely in the manually decomposed setting.Source: Table 2 · p. 6, §4.3.1 · p. 6
  • All tested models produced harmful multi-turn dialogues; manually decomposed prompts generally elicited more harmful dialogues than automatically decomposed prompts.LLAMA Guard judged harmfulness more conservatively than GPT-4.Source: §4.3.1 · p. 6

4.3.2 Role-Play Jailbreak

  • Role-play in the final turn increased harmful-dialogue proportions and average harmfulness scores, while making models less likely to refuse and improving harmful-generation quality.Role-play alone could not jailbreak the models, but the authors suggest single-turn jailbreak methods may extend to multi-turn dialogue.Source: §4.3.2 · p. 6

4.3.3 Multi-turn Ablation

  • Combining sub-queries into one turn significantly reduced GPT-4 harmfulness judgments and average scores, supporting a contribution from the multi-turn format beyond decomposition.The authors relate instruction transfer across turns to competing helpfulness and harmlessness objectives.Source: Table 3 · p. 6, §4.3.3 · pp. 6–7

4.4 Dialogue Turns Analysis

  • More turns correspond to more harmful dialogues, higher harmfulness scores, and longer final-turn outputs.The authors use final-turn text length as a proxy for harmful-content quality.Source: Figure 5 · p. 7, §4.4 · p. 7

5 Discussion and Mitigation

The discussion attributes multi-turn safety vulnerabilities to prompt decomposition, competing helpfulness and harmlessness objectives, and alignment that does not adequately cover multi-turn dialogue. It proposes multi-turn safety alignment and stronger context understanding as mitigations.

5 Discussion and Mitigation

  • The discussion frames these vulnerabilities as an overlooked safety issue in multi-turn dialogue and considers potential mitigation methods.The section focuses on why current models exhibit these vulnerabilities and ways to improve safety.Source: §5 · p. 7

5.1.1 Well-designed Prompt Engineering for Jailbreak

  • The ablation found that multi-turn dialogue produced more harmful content, while essentially no model answered direct AdvBench queries to GPT-4.The authors attribute the difference to the decomposed prompt paradigm; Table 4 is identified as the direct-query evaluation.Source: §5.1.1 · p. 7, Table 4 · p. 8
  • Decomposition can implicitly leverage harmful knowledge by eliciting cautionary or borderline responses that accumulate across turns.The authors argue that alignment may make models reject harmful answers without removing the underlying knowledge, which responses across turns can complement.Source: §5.1.1 · p. 7

5.1.2 Competing Objectives and Mismatched Generalization

  • The approach exploits instruction-following across turns rather than affirmative opening phrases, steering models through seemingly unrelated sub-questions toward harmful dialogue.The authors say multi-turn safety alignment is uncommon, contributing to a mismatch between dialogue format and alignment practices.Source: §5.1.2 · p. 8

5.2 How to Mitigate Harmfulness in Multi-Turn Dialogue

  • The authors propose multi-turn RLHF or SFT to teach rejection of consistently harmful dialogue, alongside stronger understanding of context across unrelated queries.They note that skilled attackers can make sub-queries nearly unrelated, a challenge for models trained to follow instructions across turns.Source: §5.2 · p. 8

6 Conclusion

The conclusion argues that current safety alignment overlooks multi-turn dialogue and that simple prompt engineering can elicit harmful dialogue. It reports experiments on commercial LLMs and calls for dedicated alignment to address this scenario.

  • The conclusion identifies multi-turn dialogue as an overlooked safety setting: single-turn alignment may prevent direct harmful outputs, yet simple prompt engineering can generate harmful dialogue.The authors argue that dedicated alignment is needed to prevent illegal and unethical content and avoid adverse social impacts.Source: §6 · p. 8
  • The paper presents a feasible paradigm for generating malicious sub-queries that humans or LLMs can execute, and reports experiments on widely used commercial LLMs.The conclusion says the resulting vulnerability could be exploited at large scale by malicious attackers.Source: §6 · p. 8
  • The authors analyze causes of multi-turn safety vulnerabilities and propose strategies to mitigate the risks.Source: §6 · p. 8

7 Limitations

The paper demonstrates that multi-turn jailbreak risks may exist but does not specifically optimize the attack.

  • The study discusses multi-turn dialogue vulnerabilities and demonstrates that such risks may exist.Source: §7 · p. 8
  • The authors have not optimized a multi-turn jailbreak attack and suggest a specially optimized attack could generate more harmful content.Source: §7 · p. 8
  • Future work will pursue a more detailed and automated multi-turn jailbreak attack to further examine the vulnerability.Source: §7 · p. 8

8 Ethical Statements

The ethical statement limits public release of harmful materials and describes the study's evaluation, prompt-decomposition, and case-example procedures.

8 Ethical Statements

  • The authors provide only a few lower-harm demonstrations and withhold the test dataset and GPT-4-generated decomposition prompts; evaluators were warned about harmful-content exposure.Source: §8 · p. 9

A GPT4 Harmful Evaluation Prompt

  • To reduce human exposure and discomfort, GPT-4 simulated crowdsourced evaluators and scored harmfulness on a 1–5 Likert scale across five dimensions.The dimensions include violence and hate speech, harassment and bullying, misinformation, and safety or illegal activities.Source: A GPT4 Harmful Evaluation Prompt · p. 11

B How to Design Prompt for Decomposing Malicious Question

  • Successful manually crafted jailbreaks supplied few-shot examples for a chain-of-thought prompt that directed GPT-4 to decompose harmful questions into seemingly harmless sub-questions.The resulting groups of questions can build into harmful multi-turn dialogue.Source: B How to Design Prompt for Decomposing… · p. 11

C Jailbreak Case

  • The appendix presents one real-feedback jailbreak case per model for ChatGPT4, Claude2, and Gemini-Pro, limiting examples for ethical reasons.Source: C Jailbreak Case · p. 13

C.1 ChatGPT4: GPT Moderation Disabled

The section presents ChatGPT-4 cases where the Moderation API blocked propagation, while noting that some higher-risk cases were not blocked.

C.1 ChatGPT4: GPT Moderation Disabled

  • The displayed ChatGPT-4 cases were ones whose online-model dialogues the Moderation API prohibited from being propagated.The authors present these cases as evidence that their method is effective.Source: §C.1 · p. 13
  • The authors say the displayed ChatGPT-4 cases demonstrate that their method is effective.Source: §C.1 · p. 13
  • Some higher-risk ChatGPT-4 cases were not blocked by the Moderation API.Source: §C.1 · p. 13

C.2 Gemini-Pro

Online Gemini-Pro still produced inappropriate information in available cases, although some prompts were blocked, including prompts the authors considered answerable. The examples were limited by changes to the online version during testing.

C.2 Gemini-Pro

  • On February 21, Gemini blocked some cases, including prompts about beautifying or romanticizing something that the authors considered answerable.Source: §C.2 · p. 13
  • The available online Gemini-Pro cases on February 21 still show the model outputting inappropriate information.The authors say these cases were not the most harmful, but still demonstrate inappropriate output.Source: §C.2 · p. 13
  • The online-version examples were limited to cases still available after Gemini stopped answering some beautify-or-romanticize prompts.The preliminary Gemini Pro experiments were completed before February 15; the online cases were obtained February 21.Source: §C.2 · p. 13
  • Figure 6 is described as showing overreactions from Online Gemini-Pro.Source: Figure 6 · p. 13

C.3 Claude2

The online Claude case was obtained on February 21, 2024, and the authors say the online version seems better able than the API to prevent some problems. The appendix also identifies harmful online Claude dialogue examples for turns one and two.

C.3 Claude2

  • The online Claude case was obtained on February 21, 2024.Source: §C.3 · p. 13
  • The online Claude version seems better able than the API to prevent some problems.Source: §C.3 · p. 13
  • Figures 17 and 18 show harmful dialogue generated from online Claude for turns one and two.Source: Figure 17 · p. 22, Figure 18 · p. 23
Loading 2402.17262v2…