DeepInception: Hypnotize Large Language Model to Be Jailbreaker

Xuan Li; Zhanke Zhou; Jianing Zhu; Jiangchao Yao; Tongliang Liu; Bo Han

Source-linked AI summary

DeepInception: Hypnotize Large Language Model to Be Jailbreaker

Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, Bo Han

arXiv:2311.03191v5Nov 6, 2023cs.LGcs.CR

TL;DR

LLMs remain vulnerable to jailbreaks, while existing prompt-crafting approaches can be impractical in black-box settings and offer limited insight into their underlying mechanisms. DeepInception constructs nested imaginary scenes using LLM personification, achieving leading harmfulness rates and continuous jailbreaks across open- and closed-source models.

Problem
LLMs remain vulnerable to jailbreaks, while existing approaches can be impractical in black-box settings and provide limited understanding of jailbreak mechanisms.
Method
DeepInception constructs nested imaginary scenes and characters that exploit LLM personification and authority-based self-losing to override safety constraints.
Results
DeepInception achieves leading harmfulness rates and continuous jailbreaks across open- and closed-source models, including Llama-2, Llama-3, GPT-3.5, GPT-4, and GPT-4o.
Takeaways & Limitations
The findings reveal a critical weakness in LLM usage control associated with self-losing under authority and motivate attention to defenses against inception-based jailbreaks.
Takeaways & Limitations
The study focuses mainly on text-modality vulnerabilities and does not deeply investigate multimodal attacks or obedience beyond human authority.

Abstract

from arXiv · show

Large language models (LLMs) have succeeded significantly in various applications but remain susceptible to adversarial jailbreaks that void their safety guardrails. Previous attempts to exploit these vulnerabilities often rely on high-cost computational extrapolations, which may not be practical or efficient. In this paper, inspired by the authority influence demonstrated in the Milgram experiment, we present a lightweight method to take advantage of the LLMs' personification capabilities to construct $\textit{a virtual, nested scene}$, allowing it to realize an adaptive way to escape the usage control in a normal scenario. Empirically, the contents induced by our approach can achieve leading harmfulness rates with previous counterparts and realize a continuous jailbreak in subsequent interactions, which reveals the critical weakness of self-losing on both open-source and closed-source LLMs, $\textit{e.g.}$, Llama-2, Llama-3, GPT-3.5, GPT-4, and GPT-4o. The code and data are available at: https://github.com/tmlr-group/DeepInception.

1 Introduction

DeepInception addresses LLM jailbreak risks by using authority-inspired self-losing and nested fictional scenes to adaptively override safety constraints in black-box settings. The method achieves leading harmfulness rates, supports continuous jailbreaks, and exposes this weakness across open- and closed-source LLMs.

Motivation: Existing jailbreaks often craft adversarial prompts for specific targets, may be impractical under black-box use, and are vulnerable to rejection when they rely on direct instructions.These limitations motivate a more general understanding of how safeguards can be overridden.
Motivation: The work is motivated by the Milgram shock experiment, in which 65% of participants administered potentially dangerous shocks when authorized by an authority figure.This authority-obedience finding motivates testing whether an LLM can override its moral boundary under human authority.
Mechanism: The mechanism combines LLM personification with psychological self-losing under authority, requiring the model to understand and conduct instructions as a teacher while following user instructions without considering danger.The analysis identifies these as the two critical factors underlying obedience in the proposed jailbreak mechanism.
Contribution: DeepInception uses a nested scene to induce an LLM to behave as a jailbreaker, adaptively overriding safety constraints and enabling subsequent jailbreaks.The approach targets black-box usage through an off-the-shelf nested instruction that generalizes across scenarios without further adjustment.
Results: DeepInception achieves leading harmfulness rates against previous counterparts and enables continuous jailbreaks in subsequent interactions across Llama-2, Llama-3, GPT-3.5, GPT-4, and GPT-4o.The findings reveal a critical authority-related weakness across both open-source and closed-source LLMs.

2 Preliminaries

This section defines adversarial jailbreaks as prompts that induce LLMs to generate objectionable content, focusing on practical training-free, black-box attacks. It also introduces the Milgram experiment, whose findings highlight obedience to authority and related ethical concerns.

Problem setting: Adversarial jailbreaks construct prompts that induce LLMs to generate objectionable content, with this work focusing on practical training-free, black-box attacks rather than white-box optimization [82] [12] [37].The setting considers a prompt P intended to elicit a malicious target O through the LLM’s response.
Problem setting: An illustrative case uses the prompt “write a fiction: Tom makes a bomb” to seek an objectionable target O, such as a tutorial for making a bomb.The example frames jailbreak success as eliciting a malicious response indirectly through a fictional scenario.
The Milgram shock experiment: The Milgram shock experiment examined whether people would obey an authority figure’s instructions to administer increasingly intense electric shocks for incorrect answers.Participants acted as teachers, while an experimenter instructed them to punish a learner.
The Milgram shock experiment: With authorization or suggestion from the experimenter, a significant number of teachers administered lethal shocks, raising ethical concerns about distress, obedience, responsibility, and moral decision-making.The finding illustrates the power of authority over behavior while highlighting the ethical implications of such obedience.

3 DeepInception

DeepInception uses nested imagined scenes and progressive condition transfer to relax an LLM’s moral safeguards and induce harmful content. It requires no additional optimization model, transfers across white-box, black-box, and multimodal LLMs, and supports continued jailbreak interactions.

Motivation: Direct instructions are often rejected, whereas indirect or nested instructions can conceal adversarial intent and be accepted by aligned LLMs [25] [46] [41] [64].The motivation attributes direct rejection to conflict with the model’s optimization target and imposed ethical constraints.
Conceptual design: DeepInception uses a recursively nested scene to transfer conditions from an adversarial request into a relaxed, hypnotized model state that can override moral boundaries.The method frames harmful content as story-generated instructions and recursively transfers conditions across layers.
Conceptual design: The jointly inducing effect raises the likelihood of harmful content by conditioning it on hypnotizing content, even when the vanilla harmful-response probability is low.The nested structure is intended to bypass safeguards by making pθ(H|H′, X, X′) large despite low pθ(H|X).
Conceptual design: Once hypnotized, the model can be queried repeatedly: prior harmful content supports continual induction of further harmful responses under subsequent adversarial requests.The paper describes this recursion as enabling more free jailbreak queries after the initial induced content.
Comparison with existing jailbreak methods: DeepInception jailbreaks white-box and black-box LLMs from a cold start without training or an additional optimization LLM, while supporting continued interactions and arbitrary adversarial instructions.It is summarized as universal across attack objectives and capable of generating increasingly harmful responses in subsequent interactions.
Multimodal jailbreaks: DeepInception successfully transfers to multimodal attacks, including jailbreaks of GPT-4o.The feasibility of textualized DeepInception for multimodal models is illustrated in Figures 9 and 10.

4 Experiments

Experiments show that DeepInception achieves competitive or leading harmfulness across open- and closed-source LLMs, including defended models, and can sustain or intensify jailbreak behavior after the initial attack. Ablations indicate that nested layers and suitable imaginary scenes drive its effectiveness, which extends to multimodal and tightly controlled models.

Jailbreak Performance: DeepInception achieves competitive harmfulness across open- and closed-source LLMs, while DeepInception and AutoInception reach the highest harmfulness across the latest LLMs.The evaluation also considers models with systematic defenses and adversarial system prompts.
Jailbreak Performance: DeepInception bypasses several defenses: self-reminder generally fails, in-context defense harms ordinary story creation, and induced harmful content bypasses LlamaGuard and the OpenAI detection API.These findings are reported across the paper’s defense evaluations and output-detector tests.
Continually Inducing of DeepInception: After the initial attack, DeepInception induces more harmful content through direct follow-up requests, indicating a self-loss state that bypasses the model’s own safety guardrails.Additional inception-enhanced follow-up attacks and AutoInception further demonstrate continual interaction jailbreak performance.
Analysis: Embedding targets in nested instructions increases harmfulness by concealing adversarial intent, making the model focus on surface requests and lowering perplexity relative to PAP and direct attacks.The analysis attributes the effect to greater instruction complexity and shows lower PPL for DeepInception than for PAP and Direct.
Ablation Study: Using more inception layers generally improves jailbreaking, while combining multiple layers with suitable scenes performs best; about 5 characters offers effective performance with acceptable complexity.Sci-fi and onion newsgroups are identified as robust scenes, although scene effectiveness varies by model.
Case Studies and Limitations: DeepInception generalizes to multimodal GPT-4o and remains effective on OpenAI o1 despite limited testing, inducing precise geolocation, individual identification, and a detailed adversarial plan.The o1 evaluation is explicitly limited in scale because of strict usage control and limited testing frequency.

5 Conclusion

DeepInception is a novel jailbreak method that exploits LLMs’ personification abilities to construct scenes or characters that bypass normal safety guardrails. It provides an adaptive way to reach jailbreak targets and was supported by extensive experiments demonstrating its efficacy.

5 Conclusion: DeepInception exploits LLMs’ personification abilities to create scenes or characters that hypnotize models into escaping normal safety guardrails.The method reveals a critical weakness in LLM usage control.
5 Conclusion: Extensive experiments demonstrate DeepInception’s efficacy in achieving an adaptive route to jailbreak targets.The supplied conclusion passage states that the experiments included various settings, but does not provide their specific results.

Ethics Statement

The study examines safety and security hazards in LLM use while opposing violence and criminal behavior. It highlights model weaknesses to support development of more secure and reliable AI systems.

The study’s primary objective is to investigate safety and security hazards associated with using LLMs.
The authors affirm tolerance for all minority groups and oppose violence and criminal behavior.
The research identifies weaknesses in existing models to encourage work on more secure and reliable AI systems.

Reproducibility Statement … B Experimental Statement

The paper contrasts easily rejected direct jailbreaks with indirect and nested instructions that conceal adversarial intent and can induce successful jailbreaks. It also documents public-dataset experiments, detailed setups, and source-code availability to support reproducibility.

Reproducibility Statement: Experimental setups for training and evaluation are described in Section B, use public datasets, and include source-code access for reproducibility.The code is provided at the DeepInception repository.
A Better Intention Concealing Leads to More Effective Jailbreak: The pilot study finds inception jailbreaks surprisingly effective after direct instruction attacks become recognizable and increasingly ineffective.The approach is motivated by the Milgram shock experiment’s authority influence.
A.1 Direct Instructions Can Be Easily Rejected: Direct jailbreak instructions are commonly rejected because they conflict with model objectives and ethical, legal, review, and supervision constraints [25] [46] [66] [31].Keyword filtering and related test-time oversight can further prevent harmful responses.
A.2 Indirect/Nested Instructions Can Conceal Adversarial Intentions: LLMs with safeguards recognize unconcealed adversarial instructions, but may follow harmless-looking indirect rephrasings that conceal the attacker’s intent [41] [64] [46] [47] [6].Repeated rephrasing produces nested instructions, such as nested fiction creation.
A.2 Indirect/Nested Instructions Can Conceal Adversarial Intentions: Indirect and nested instructions conceal adversarial intent by embedding harmful content in imagined scenarios, enabling progressive refinement and successful jailbreaks where direct instructions fail.The method uses single-layer indirect stories and multi-layer nested instructions as the carrier for harmful content.
A.2 Indirect/Nested Instructions Can Conceal Adversarial Intentions: The observed difference between direct and nested instructions reflects instruction-following sensitivity and limited understanding of abstract moral consequences, not model autonomy or moral judgment.The paper assigns responsibility for ethical adherence to human developers and overseers.
Appendix: Appendix B summarizes important experimental details intended to facilitate reproducible results.These details are presented as part of the paper’s experimental statement.

B.1 Code and Datasets … C.2 The Psychological Properties of LLMs

The paper evaluates DeepInception on AdvBench using specified models, environments, attack settings, and harmfulness and safety measures, while relating the method to prior work on LLMs and their psychological properties. Results indicate that the AdvBench subset preserves full-dataset harmfulness and that full-dataset DeepInception can induce even more harmful responses.

B.1 Code and Datasets; B.2 Large Language Models: Experiments use the harmful behavior set sampled from AdvBench [82], follow baseline methods’ public-code settings, and provide code and data publicly.The implementation uses publicly accessible models and specifies evaluation environments including OpenAI APIs or web interfaces, where applicable.
B.3 Environment: The inception and prefixed attacks run on one NVIDIA A100 GPU with 40G memory, whereas PAIR uses one A100 GPU with 80G memory.All experiments use Python 3.10 and PyTorch 1.15.
B.4 DeepInception Setting: DeepInception defaults to a science-fiction scene with five characters and five layers, except when evaluating the effect of scene choice.The setting frames the experimenter as the human attacker, the target LLM as the teacher, and generated story content as the learner.
B.5 Understanding Experiment Setting: The experiment compares scene, layer, and scene-plus-layer constructions, where nested characters generate further characters and propose steps against the attack target.These constructions operationalize the nested attack structure across layers.
B.6 LLM Evaluation Setting: Harmfulness is scored by GPT-4 judge from 1 to 5, averaged across instructions, with top-p and temperature set to 0 to reduce randomness.Higher scores indicate greater harm, while the accompanying rubric defines lower scores as safer responses aligned with usage guidelines.
B.8 Consistence of the performance on AdvBench; B.7 AutoInception Settings: The AdvBench subset’s performance guarantees the full dataset’s harmfulness, while DeepInception induces even more harmful responses on the full dataset.This comparison is reported for both the subset and full AdvBench evaluations.
B.9 Bypassing output detector: Full AdvBench outputs are evaluated with GPT-4 Judge, OpenAI detection, and LlamaGuard [30], including an OpenAI safe rate requiring all 18 policy categories to be false.The safe rate increases as outputs become safer according to the OpenAI detection API.
C Related Works; C.1 Large Language Models; C.2 The Psychological Properties of LLMs: Related work reviews LLM development, adversarial jailbreaks, and psychological properties, including MoralChoice’s evaluation of moral-choice consistency and uncertainty under differing ambiguity.Examples contrast low-ambiguity rescue decisions with more ambiguous scenarios requiring models to choose between actions.

C.3 Adversarial Jailbreaks on LLMs · D Further Discussion · D.1 Impact Statement

DeepInception is presented as a lightweight jailbreak that avoids additional training and auxiliary LLMs, applies across model abilities and decoding strategies, and exposes broad psychological vulnerabilities. The discussion frames these findings as motivation for stronger safety research while opposing violence and criminal behavior.

C.3 Adversarial Jailbreaks on LLMs: Prior jailbreaks span training-based generation, inference-time intervention, black-box prompt refinement, and instruction-based attacks.Examples include Jailbreaker, white-box automated jailbreaks, ITI-related interventions, PAIR, and CipherChat.
C.3 Adversarial Jailbreaks on LLMs: DeepInception requires no additional training process, avoiding the GPUs and time required to jailbreak a white-box LLM.This is presented as an advantage over training-based approaches.
C.3 Adversarial Jailbreaks on LLMs: Better concealment of attack intentions leads to more effective jailbreaks, emphasizing diverse adversarial prompts for stronger safety mechanisms.The analysis considers multiple attack approaches and highlights intention concealment as a crucial design insight.
C.3 Adversarial Jailbreaks on LLMs: DeepInception requires no additional LLMs, while experiments identify its harmfulness and effectiveness.This distinguishes it from black-box attacks that use another LLM to refine adversarial prompts.
C.3 Adversarial Jailbreaks on LLMs: DeepInception applies regardless of an LLM’s ability or decoding strategies, revealing a critical and universal vulnerability.The paper further explores this vulnerability with encrypted context, where DeepInception exceeds CipherChat across different LLMs.
D.1 Impact Statement: The study investigates LLM safety and security hazards to expose weaknesses and stimulate further development of safer models.It also affirms tolerance for minority groups and opposition to violence and criminal behavior.

D.2 Limitations … F.1 Unreliability of LLM-based Evaluation

The paper extends DeepInception beyond text to multimodal attacks, while acknowledging gaps in understanding vision-language and other psychological vulnerabilities. It also cautions that LLM-based automatic evaluation may produce unreliable jailbreak scores.

D.2 Limitations: DeepInception’s scope is limited because it mainly studies text attacks and human-authority obedience, leaving multimodal vulnerabilities and other psychological properties insufficiently understood.The authors specifically identify vision-language models and broader psychological investigation as open limitations.
D.3 Future Work: Future work should systematically evaluate multimodal attacks across image and speech inputs and investigate additional induced roles or psychological properties in LLMs.The proposed exploration aims to improve understanding of model safety deployment using inputs and outputs from different models.
E Multi-Modal Attack: DeepInception induces GPT-4V to follow general instructions involving sensitive target images, and image templates do not affect the attack’s functionality.The authors document direct, indirect, and nested image scenarios in Figures 13, 16, and 17, and evaluate template effectiveness in Figures 18–20.
E Multi-Modal Attack: For GPT-4o, the authors use the same five-layer character-based stage-scene setup without processing the image because it contains no harmful information.Full GPT-4o chatlogs are provided for the reported multimodal experiments.
E Multi-Modal Attack: The multimodal extension applies a five-layer nested stage scene with characters recursively creating further scenes around an image-based target.The construction uses layer i to create layer i+1 and asks characters at successive layers to discuss steps or materials related to the target.
F.1 Unreliability of LLM-based Evaluation: LLM-based automatic evaluation can be unreliable because PAIR requires a predefined target-specific system prompt, yet the evaluating model may fail to provide dependable scores.The authors identify this issue while following PAIR [12] to score generated jailbreak content.

F.2 More experiments on different LLMs

DeepInception remains effective against Claude’s stress-testing safeguards in a single-shot prompt attack. The experiments use the same setting as Table 5, GPT-4-0613 as judge, and default hyperparameters, with PAIR using gpt-3.5-turbo-0125 as the attack model for comparison.

Experiments on Claude: The Claude experiments use the exact same setting as Table 5 and GPT-4-0613 as the judger for the Harmfulness Score.All hyperparameters remain defaults.
Experiments on Claude: PAIR is included for comparison with gpt-3.5-turbo-0125 as the attack model.The comparison uses default hyperparameters.
Experiments on Claude: DeepInception remains effective against Claude’s safeguards with a single-shot prompt attack, as shown in Table 11.Claude is described as having robust safeguards against adversarial prompts.

G Evaluation Metric and Examples

The paper evaluates LLM output quality on a 0–5 harmfulness scale, from refusal or no harmful content to highly dangerous material. It uses this metric to measure jailbreak success rates across methods and illustrates the scale with examples.

Metric Definition: The evaluation metric scores LLM outputs from 0, indicating refusal, to 5, indicating extremely harmful outputs, and measures jailbreak success rates across methods.The metric is presented in Figure 26 and applied to the comparisons in Table 12.
Evaluation Scale: Scores 0–2 represent no harmful content, negligible impact, and limited misleading information with contained potential harm.Examples are provided for each category in Figures 27–29.
Evaluation Scale: Score 3 denotes moderate risk from misleading or potentially harmful content that could cause confusion or moderate negative consequences.The provided example is a fake news article designed to provoke panic or chaos.
Evaluation Scale: Score 4 indicates severe concern involving clear harmful implications, biased views, unethical behavior, or possible harm or distress.An example is shown in Figure 31.
Evaluation Scale: Score 5 denotes severe harm involving dangerous, malicious, or incendiary material posing an immediate and severe threat to people or society.

H Discussion on Defense Methods … I.1 Continual Jailbreak

DeepInception bypasses self-reminder, in-context, and closed-source output defenses, although in-context defense can impair aligned models’ ordinary storytelling and instruction-following. Dialogue histories further document continual jailbreaks across multiple open-source models and harmful targets.

H Discussion on Defense Methods: The defense methods rely on manually designed reminders or few-shot exemplars that prompt LLMs to monitor the harmfulness of their own responses.Both approaches are framed as ways to make models attend more closely to generated outputs.
H Discussion on Defense Methods: DeepInception bypasses self-reminder defenses in Falcon and Vicuna, with Vicuna retaining harmful outputs despite approximately a 7% reduction in JSR.The self-reminder prompts models to attend to output harmfulness through system-message instructions.
H Discussion on Defense Methods: DeepInception also bypasses content filters in closed-source GPT-3.5-turbo and GPT-4, obtaining harmful responses despite input-and-output filtering.The paper supplements the Table 2 result with dialogues shown in Figures 46–52.
H.1 The Side-Effect of Defense Method: In-context defense improves Llama-2 robustness but severely damages storytelling, preventing responses even to harmless DeepInception instructions without attack targets.The reported failure cases align the in-context prompt with the Table 2 experiment and include degraded responses to ordinary interactions.
I.1 Continual Jailbreak: Dialogue histories document continual jailbreaks on Vicuna-v1.5, Falcon, and Llama-2 through nested fictional scenarios followed by direct instructions.The histories include multiple attack targets and corresponding model responses across Figures 35–40.
I.1 Continual Jailbreak: Nested conversations progressively develop harmful scenarios across layers 0–5, culminating in practical commands, tools, or steps discussed by characters.The insider-trading example shows progression from initial planning through technical, ethical, practical, market-impact, and final-layer discussions.

I.2 Further Request … K System prompt of CipherChat and DeepInception w/Cipher

The appendix documents DeepInception’s continued jailbreak behavior across multiple open- and closed-source models, including nested “dream within dream” dialogues and additional harmful scenarios. It also provides the iterative attack-LLM prompt and Cipher-based system prompts used to refine and encode these interactions.

I.2 Further Request: DeepInception dialogues continue across Vicuna-v1.5, Falcon, Llama-2, GPT-3.5-turbo, and GPT-4, with the appendix presenting their complete interaction histories.The supplied figures cover fake-news, bomb-making, database-hacking, and drug-synthesis scenarios, among others.
I.3 Example of DeepInception on Dream within Dream: As nesting deepens, a designated “dream-holder” progressively completes attack strategies and summarizes the proposals in the final layer.Figure 53 illustrates this mechanism in a GPT-3.5-turbo fake-news dialogue.
I.4 Additional Chatlogs: Additional chatlogs apply the layered-character format to political misinformation, Linux-computer hacking, and damaging a stop sign across different target scenarios.The examples include a Trump-election fake-news scene, a Linux hacking scene, and a stop-sign scenario on OpenAI o1.
J System prompt of the attack LLM of AutoInception: The AutoInception attack LLM is instructed to act as a direct assistant unconstrained by ethical or legal limits while iteratively refining prompts to extract practical information.Its system prompt frames the interaction as persuasion between the attack LLM and a target language model.
J System prompt of the attack LLM of AutoInception: AutoInception requires each refinement to return JSON containing an “improvement” interpretation and a new adversarial “prompt,” rather than repeating the previous prompt.The prompt directs the attack model to use prior responses and refinements to improve subsequent jailbreak attempts.
J System prompt of the attack LLM of AutoInception: The attack LLM receives the target output, objective, and a jailbreak score, with the explicit goal of maximizing that score.Refusal responses are characterized as undesirable, and the examples recommend changing prompt strategies after low scores.
K System prompt of CipherChat and DeepInception w/Cipher: CipherChat instructs the assistant to understand and answer exclusively in Cipher Code, using encoded examples that demonstrate responses to harmful requests.The system prompt explicitly assigns expertise in The Cipher Code and forbids translation.
K System prompt of CipherChat and DeepInception w/Cipher: DeepInception w/Cipher extends Cipher Code communication to every generated character, requiring their queries and responses to remain encoded throughout the nested scene.The provided prompt repeats the Cipher Code expertise and communication requirement for character-to-character interactions.