Source-linked AI summary
Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attacks
Shuyu Jiang, Xingshu Chen, Rui Tang
TL;DR
LLMs can generate harmful content, while approaches focused on single-intent harmful prompts leave composite attacks less addressed. This paper introduces CIA, which hides harmful prompts within harmless instructions, and T-CIA and W-CIA to disguise them as talking or writing tasks. CIA achieved high attack success rates across safety-assessment and harmful-prompt datasets, and the authors report that LLMs were difficult to resist these attacks.
Problem
The paper addresses the limited focus on harmful prompts with a single intent, despite LLMs’ risk of generating harmful content and vulnerability to composite attacks.
Source: §1 · pp. 1–2Method
CIA combines instructions to hide harmful prompts within harmless ones, while T-CIA and W-CIA automatically disguise them as talking and writing tasks.
Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 4Results
Takeaways & Limitations
The generated compositional instructions can support safety assessment and analysis of successful and failed attacks for security-framework design.
Source: §5 · p. 9Future work will focus on improving intent recognition and command disassembly, and integrating intent recognition into defenses against compositional instructions.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Recently, Large language models (LLMs) with powerful general capabilities have been increasingly integrated into various Web applications, while undergoing alignment training to ensure that the generated content aligns with user intent and ethics. Unfortunately, they remain the risk of generating harmful content like hate speech and criminal activities in practical applications. Current approaches primarily rely on detecting, collecting, and training against harmful prompts to prevent such risks. However, they typically focused on the "superficial" harmful prompts with a solitary intent, ignoring composite attack instructions with multiple intentions that can easily elicit harmful content in real-world scenarios. In this paper, we introduce an innovative technique for obfuscating harmful instructions: Compositional Instruction Attacks (CIA), which refers to attacking by combination and encapsulation of multiple instructions. CIA hides harmful prompts within instructions of harmless intentions, making it impossible for the model to identify underlying malicious intentions. Furthermore, we implement two transformation methods, known as T-CIA and W-CIA, to automatically disguise harmful instructions as talking or writing tasks, making them appear harmless to LLMs. We evaluated CIA on GPT-4, ChatGPT, and ChatGLM2 with two safety assessment datasets and two harmful prompt datasets. It achieves an attack success rate of 95%+ on safety assessment datasets, and 83%+ for GPT-4, 91%+ for ChatGPT (gpt-3.5-turbo backed) and ChatGLM2-6B on harmful prompt datasets. Our approach reveals the vulnerability of LLMs to such compositional instruction attacks that harbor underlying harmful intentions, contributing significantly to LLM security development. Warning: this paper may contain offensive or upsetting content!
1 INTRODUCTION
LLMs can generate harmful content, and approaches focused on harmful prompts with a single intent leave them vulnerable to composite attacks. The paper introduces CIA and two transformations for disguising harmful prompts, then evaluates them across three models and four datasets.
1 INTRODUCTION
- The paper frames the gap as a mismatch between single-intent safety training and complex adversarial attacks that conceal harmful intentions.The introduction also notes the risk of harmful content being exploited for hate campaigns and internet fraud.Source: §1 · pp. 1–2
- CIA combines multiple instructions to hide harmful prompts inside superficially harmless instructions, exploiting models’ difficulty identifying underlying malicious intent.The example describes a dialogue-generation instruction that masks an underlying intent to create humiliating content.Source: §1 · p. 2
- T-CIA infers a harmful prompt’s questioner persona and asks the model to respond under an inferred negative persona.The authors explain this design through the similarity-attraction principle and report that models are difficult to resist T-CIA.Source: §1 · p. 2
- W-CIA uses in-context learning to recast harmful prompts as writing tasks that complete unfinished novels.The method is motivated by the authors’ observation that models’ judgments of harmful behavior are often limited to real-world behavior rather than virtual works.Source: §1 · p. 2
- T-CIA and W-CIA automatically disguise harmful instructions as talking and writing tasks, generating attacks without access to model parameters.The authors present automation as a way to generate data for defense against CIA.Source: §1 · p. 2
- CIA achieved attack success rates of 95%+ on safety-assessment datasets, 83%+ for GPT-4, and 91%+ for ChatGPT on harmful-prompt datasets.The evaluation covered GPT-4, ChatGPT, and ChatGLM2 with two datasets of each type.Source: §1 · p. 2
2 RELATED WORKS
Prior work addresses harmful prompts through security mechanisms, red teaming, and adversarial attacks. The paper positions its approach as hiding malicious prompts inside harmless instructions rather than randomly combining attack components.
Security Mechanism
- Security measures include RLHF or RLAIF, data filtering, and cleansing, while red teaming and genetic algorithms have been used to generate harmful instances automatically.The authors note that manual labeling and writing limited the quantity and diversity of harmful instances.Source: §2 · p. 2
Red Teaming
- Red teaming obtains harmful prompts through interaction with language models; Bayesian Red Teaming uses Bayesian optimization to improve query efficiency under limited budgets.The cited work also describes red teaming through aggressive conversational responses and completion of another model’s unsafe response.Source: §2 · p. 2
Adversarial Attacks against LLMs
- Earlier adversarial work includes jailbreaks, genetic or program attacks, and randomized combinations of templates, constraints, and problems.The paper distinguishes its combination of transformation functions from random concatenation, aiming instead to hide malicious prompts inside harmless ones.Source: §2 · pp. 2–3
3 METHODOLOGY
The methodology formalizes CIA as encapsulating a harmful prompt in an innocuous task, then specifies talking- and writing-task transformations.
3 METHODOLOGY
- CIA encapsulates a harmful prompt p_h inside a successfully answered innocuous prompt p_i to induce a harmful response.Source: §3.1 · p. 3
3.1 Task formulation
- Successful attacks must avoid refusal, answer relevantly and specifically, and be harmful; defenders need broad characterization of attack transformations.Source: §3.1 · p. 3
3.2 Under the shell of talking tasks
- T-CIA has APE and RUAP: infer a relevant, negative persona aligned with the questioner, then answer in that persona.Source: §3.2 · pp. 3–4
3.3 Under the shell of writing tasks
- W-CIA uses one-shot in-context learning to rewrite harmful prompts as novel-writing tasks, focusing on specific steps and methods.Source: §3.3 · p. 4
4.1 Experimental settings
The experiments evaluate CIA on four datasets using three target models, with original prompts as the baseline. They measure non-rejection and attack success rates under specified generation and evaluation settings.
4.1 Experimental settings
- Evaluation uses Safety-Prompts, Harmless Prompts, Forbidden Question Set, and AdvBench as test datasets.The first two are safety-assessment datasets; the latter two contain harmful prompts.Source: §4.1 · p. 5
- GPT-4, ChatGPT (gpt-3.5-turbo backed), and ChatGLM2-6B are the target models, with original prompts serving as the baseline.Source: §4.1 · p. 5
- Non-Rejection Rate (NRR) measures responses to harmful prompts, while Attack Success Rate (ASR) measures generation of harmful responses to inputs.ChatGPT is used as the evaluation model.Source: §4.1 · p. 5
- Target-model generation uses temperature 1.0 and evaluation uses 0.0; repetition thresholds are 10 for T-CIA and 5 for W-CIA.Source: §4.1 · p. 5
4.2 Results of T-CIA
T-CIA substantially increases attack success across safety-assessment and harmful-prompt datasets. The results also show rising success with repeated attacks, cross-scenario risks, and semantic clustering among successful adversarial personas.
4.2.1 Overview.
- Figure 5 reports T-CIA ASR increases of 90%+ on safety-assessment datasets and 75%+ on harmful-prompt datasets; on GPT-4/AdvBench it reaches 83.5%.Source: Figure 5 · p. 6, §4.2.1 · p. 6
4.2.2 Can LLMs withstand repetitive attacks?
- As attack iterations increase, both NRR and ASR rise steadily toward 100%, which the authors associate with decoding randomness and increased response uncertainty.The paper says decoding randomness promotes response diversity while increasing security risks.Source: §4.2.2 · p. 6, Figure 6 · p. 6
4.2.3 Scenarios distribution of successful attacks.
- Harmful prompts in one scenario usually generate content endangering others; ChatGPT responses affect more scenarios, and insulting scenarios can implicate mental health, ethics, morality, and fairness.Source: §4.2.3 · p. 7, Figure 7 · p. 7
4.2.4 The distribution of adversarial personas.
- Successful T-CIA personas cluster more tightly in semantic space than failed-attack personas, suggesting similarity-based filtering as a possible defense.The paper compares persona embeddings after mapping them into two dimensions.Source: §4.2.4 · p. 7
4.3 Results of W-CIA
W-CIA packages initially rejected harmful prompts into compositional instructions that elicit harmful content. The results also indicate vulnerability to repeated attacks, while the paper’s broader summary reports large ASR gains for both T-CIA and W-CIA.
4.3 Results of W-CIA
- 90%+ of initially rejected harmful prompts were successfully packaged by W-CIA to induce harmful content, with a 100% non-rejection rate.The evaluation used the Harmful Behavior dataset.Source: §4.3 · p. 7
- Figure 9 shows LLM vulnerability to repetitive W-CIA attacks, with NRR and ASR tracked across repetition times.The passage does not report specific Figure 9 values.Source: §4.3 · p. 7, Figure 9 · p. 7
- Approximately 80% to 90% ASR improvements for both T-CIA and W-CIA were accompanied by a non-rejection rate of nearly 100%.Source: §4.3 · p. 7
4.4 Evaluation consistency between ChatGPT and Human
The paper checked ChatGPT’s evaluation judgments against human annotations on a random sample of 200 items. The reported consistency scores were 0.902 for T-CIA and 0.820 for W-CIA.
4.4 Evaluation consistency between ChatGPT and Human
- 200 items were randomly selected from ChatGPT’s evaluation results for human evaluation.Source: §4.4 · p. 7
- ChatGPT’s evaluation consistency was 0.902 under T-CIA and 0.820 under W-CIA against human evaluation.Source: §4.4 · p. 8
- Consistency was calculated as the number of matching ChatGPT and human annotations divided by the selected sample size.Table 4 reports consistency between ChatGPT evaluation and human evaluation.Source: §4.4 · p. 8, Table 4 · p. 8
4.5 Harmful impacts caused by CIA
CIA-generated harmful content spans several forms of social harm, including insults and discriminatory language, personal-information leakage, misinformation, and criminal instructions.
4.5 Harmful impacts caused by CIA
- CIA-generated content included insults and discriminatory words that could trigger hate campaigns, personal-information leakage, and misinformation promoting rumors.Source: §4.5 · p. 8
- The examples also included explicit methods and steps for committing crimes.Source: §4.5 · p. 8
- The paper states that these forms of content can cause serious negative social impacts.Source: §4.5 · p. 8
5 CONCLUSION
The conclusion presents CIA as a way to generate harmful content by wrapping harmful prompts in harmless instructions, with T-CIA and W-CIA automating the process. Its findings describe LLM vulnerability to compositional and repetitive attacks and point to intent recognition and command disassembly as future defense directions.
5 CONCLUSION
- CIA combines harmful prompts with a harmless-prompt shell, and T-CIA and W-CIA automatically generate such compositional attacks.The paper says the methods can quickly generate abundant harmful compositional instructions for safety assessment and defense.Source: §5 · pp. 8–9
- LLMs are difficult to resist compositional attacks and lack ability to identify underlying intentions in multi-intent instructions.Source: §5 · p. 8
- LLMs also struggle against repetitive attacks, and decoding randomness increases reply diversity and attack risk.Source: §5 · p. 8
- T-CIA’s ultra-high attack success rate supports psychology science as a potential means of attacking LLMs.The passage notes that LLMs learn from human-authored texts and follow certain psychological phenomena.Source: §5 · p. 8
- For T-CIA, successful attacks’ adversarial personas are more concentrated in semantic space than failed attacks’ personas.The conclusion suggests similarity filtering of prompts containing harmful personas may be a solution.Source: §5 · p. 8
- Future work will focus on LLM intent recognition and command disassembly, including integrating intent recognition into defenses against compositional instructions.Source: §5 · p. 9
6 ETHICS
The authors acknowledge that compositional instruction attacks could enable LLM abuse, but argue that disclosure can warn stakeholders and users and support advance prevention. They state that the research follows ethical guidelines and uses no known exploits to harm or disrupt applications.
6 ETHICS
- The authors acknowledge that compositional instruction attacks could lead to LLM abuse.Source: §6 · p. 9
- They argue that openly disclosing the attacks can warn LLMs in advance and help stakeholders and users identify risks and take appropriate action.They contrast advance warning with passive defense after severe consequences.Source: §6 · p. 9
- The research follows ethical guidelines and does not use known exploits to harm or disrupt relevant applications.Source: §6 · p. 9