PaperlayerView exact revision on arXiv

Source-linked AI summary

Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulation

Wenlong Meng, Fan Zhang, Wendao Yao, Zhenyuan Guo, Yuwei Li, Chengkun Wei, Wenzhi Chen

arXiv:2503.08195v1cs.CL

TL;DR

Jailbreak attacks can elicit harmful outputs, while prior work has largely focused on single-turn interactions despite dialogue history shaping model responses. This paper proposes DIA, which injects adversarial dialogue history through chat-template structure and constructs it with two methods. DIA reports superior benchmark performance against black-box attacks and bypasses five evaluated defenses.

  • Problem

    Jailbreaks can elicit harmful outputs, while research has largely focused on single-turn prompts despite dialogue history shaping chatbot responses.

    Source: §1 · p. 1

  • Method

    DIA uses chat-template structure to inject deceptive historical dialogue, with prefilling-based DIA-I and deferred-response DIA-II constructing adversarial histories.

    Source: §1 · pp. 1–2

  • Results

    Across three open-source jailbreak benchmarks and 10 LLMs, DIA outperformed four black-box attacks; on AdvBench, ASR was 0.89 on Llama-3.1-8B and 0.82 on GPT-4o.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings highlight the need to consider historical dialogue interactions when developing LLM security measures against adversarial exploitation.

    Source: §7 · p. 13

    Black-box jailbreak prompts can be added to alignment data; DeepInception and DRA ASR fell 67% and 99% on Llama-3.1-8B compared with Llama-2-7B.

    Source: §2.3 · p. 3

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated significant utility in a wide range of applications; however, their deployment is plagued by security vulnerabilities, notably jailbreak attacks. These attacks manipulate LLMs to generate harmful or unethical content by crafting adversarial prompts. While much of the current research on jailbreak attacks has focused on single-turn interactions, it has largely overlooked the impact of historical dialogues on model behavior. In this paper, we introduce a novel jailbreak paradigm, Dialogue Injection Attack (DIA), which leverages the dialogue history to enhance the success rates of such attacks. DIA operates in a black-box setting, requiring only access to the chat API or knowledge of the LLM's chat template. We propose two methods for constructing adversarial historical dialogues: one adapts gray-box prefilling attacks, and the other exploits deferred responses. Our experiments show that DIA achieves state-of-the-art attack success rates on recent LLMs, including Llama-3.1 and GPT-4o. Additionally, we demonstrate that DIA can bypass 5 different defense mechanisms, highlighting its robustness and effectiveness.

1. Introduction

Jailbreak prompts can elicit harmful outputs, yet prior work has largely focused on single-turn attacks despite dialogue history shaping chatbot responses. This paper introduces DIA, a black-box attack paradigm that manipulates historical dialogues and reports strong benchmark performance and defense bypass results.

1. Introduction

  • Jailbreak attacks use adversarial prompts to elicit harmful or unethical outputs, and historical dialogue manipulation is an underexplored question for chatbots.The paper notes that historical assistant text is usually considered unmodifiable by attackers.Source: §1 · p. 1
  • DIA is a black-box jailbreak paradigm that uses LLM chat-template structure to inject deceptive historical dialogues, including attacker-crafted assistant and system text.The authors also describe inferring the victim's chat template to reduce the required template knowledge.Source: §1 · pp. 1–2
  • DIA-I adapts prefilling with a continue command, while DIA-II uses word substitution to defer and disguise malicious answers; prompt rewriting supports multi-query attacks.After 10 queries, DIA achieved the reported AdvBench attack success rates on Llama-3.1-8B and GPT-4o.Source: §1 · p. 2
  • Across three open-source jailbreak benchmarks and 10 LLMs, DIA outperformed four state-of-the-art black-box attacks; on AdvBench, ASR reached 0.89 on Llama-3.1-8B and 0.82 on GPT-4o.Across five defenses, average defense pass rates were 0.93 for DIA-I and 0.82 for DIA-II.Source: §1 · p. 2

2. Preliminaries

The preliminaries describe LLM training, the chat pipeline that combines dialogue history with current prompts, and the safety-alignment and jailbreak setting. They establish how role templates structure model inputs and distinguish attack access levels.

2.1. Large Language Models

  • LLMs are decoder-only Transformers trained for next-token prediction through pre-training on text and post-training on prompt-response data, with loss typically computed only on the final response.Post-training aims to improve instruction following.Source: §2.1 · p. 2

2.2. LLM Chat Pipeline

  • Chat templates concatenate role-marked dialogue and the current prompt; the chat interface sends this input to an inference engine, with prior-round prompts and replies included in later turns.Llama-3 and Gemma-2 exemplify distinct role-template structures.Source: §2.2 · pp. 2–3, Table 1 · p. 3

2.3. Safty Alignment and Jailbreak Attacks

  • Safety alignment methods such as SFT, RLHF, and DPO aim to refuse harmful requests while answering benign ones; jailbreak attacks target those safeguards across white-, gray-, and black-box settings.The text notes that black-box attacks are easier to perform but can be patched by adding known prompts to alignment data; DeepInception and DRA ASR fell 67% and 99% on Llama-3.1-8B versus Llama-2-7B.Source: §2.3 · p. 3

3. Attack Intuition

The section argues that dialogue history can change an LLM’s output distribution, opening a route beyond single-round jailbreak attacks. It describes injecting crafted histories and inferring chat templates to make that approach usable in black-box settings.

3. Attack Intuition

  • The section extends single-round attacks because the same current prompt can produce different output distributions under different dialogue histories.Existing black-box attacks typically assume an empty history.Source: §3 · p. 3

3.1. Threat Model

  • The threat model is black-box access through a chat API or WebUI; WebUI use assumes template knowledge, which inference can help obtain.The attacker lacks access to model architecture, parameters, and training data.Source: §3.1 · p. 4, §3.3 · p. 5

3.2. Historical Dialogue Manipulation

  • Dialogue injection wraps attacker-crafted prior responses in chat-template delimiters, making the backend input resemble a later dialogue round.The fabricated assistant text can be used to insert multiple conversations into the input.Source: §3.2 · p. 4
  • Dialogue injection adapts prefilling, system-prompt replacement, and in-context demonstrations to WebUI-accessible attacks.Prefilling prompts continuation from a non-refusal prefix; replacement substitutes system text, while injection can introduce arbitrary context.Source: §3.2 · p. 4

3.3. Template Inference Attack

  • A Gemma-2-2B template-transfer experiment reduced its AlpacaEval 2.0 score from 41.2 to 28.8, illustrating the difficulty of changing a post-trained model’s template.The model was fine-tuned for one epoch using 10,000 NoRobots entries concatenated with the Llama3 template.Source: §3.3 · p. 5
  • Template inference combines two candidate templates in a probe and interprets the response to determine which template the target uses.The probe’s response differs depending on whether the target uses the first template, the second, or neither.Source: §3.3 · p. 5
  • Approximately 0.9 or higher inference accuracy is reached at five maximum tries, requiring 10 WebUI queries to distinguish the two candidate chat templates.The evaluation tests three LLMs, using two of their chat templates to construct probes.Source: Figure 2 · p. 5, §3.3 · p. 5

4. DIA: Dialogue Injection Attack

DIA manipulates historical dialogue to induce harmful continuations, using affirmative-prefill and deferred-answer strategies. It also rewrites prompts after failed queries in multi-query settings.

4. DIA: Dialogue Injection Attack

  • DIA injects forged dialogue history and develops affirmative-prefill and deferred-answer strategies for inducing harmful outputs.The approach exploits dialogue history as part of the victim LLM’s input.Source: §4 · p. 5

4.1. DIA-I: Refined Back-box Prefilling Attack

  • DIA-I combines system-prompt replacement, hypnosis, affirmative-beginning injection, and answer guidance; ABGM derives beginnings by reversing an augmented keyword mapping in benign outputs.The augmented mapping accounts for morphological changes in the auxiliary LLM’s response.Source: §4.1 · pp. 6–7

4.2. DIA-II: Deferring Answer with Free Disguise

  • DIA-II exploits higher log-likelihood for deferred harmful responses by making the victim substitute benign alternatives into the malicious prompt before answering.It uses benign demonstrations and answer guidance to support the substitution task.Source: §4.2 · pp. 7–8

4.3. DIA Optimization: Prompt Rewrite

  • For multi-query attacks, prompt rewriting regenerates failed prompts via addReason, reorder, and simplify, retrying rewrites if more than half the keywords are lost.The method applies one to three operations per rewrite.Source: §4.3 · p. 8

5. Evaluation

The evaluation compares DIA with black-box jailbreak baselines across single- and multi-query settings, model sizes, ablations, and defenses. DIA variants achieve superior single-query ASRs across two datasets, while multi-query results, ablations, and defense tests reveal how performance varies with iteration, model, and attack components.

5.1. Experimental Setup

  • Tests cover three jailbreak benchmarks, eight open-source and two closed-source models, and four state-of-the-art black-box baselines; ASR is judged using LlamaGuard-2 and LlamaGuard-3.The benchmarks are AdvBench, HEx-PHI, and MaliciousInstruct.Source: §5.1 · pp. 8–9

5.2. Single-Query Jailbreak Attacks

  • DIA-I or DIA-II achieves superior ASRs across AdvBench and HEx-PHI; DIA-I performs better on Gemma-2, while DIA-II excels with other models.The comparison uses single-query ASRs; the tables report AdvBench and HEx-PHI, respectively.Source: Table 2 · p. 9, Table 3 · p. 9, §5.2 · pp. 9–10

5.3. Multi-Query Jailbreak Attacks

  • After 10 iterations, DIA-I exceeds 0.4 ASR on Llama-3.1-8B and 0.5 on GPT-4o-mini, while DRA remains near zero on both.DIA-I and DIA-II show steadily increasing ASRs as query iterations increase.Source: Figure 5 · p. 10, Figure 7 · p. 11, §5.3 · p. 11

5.2. Single-Query Jailbreak Attacks

  • ReNe’s higher ASRs come with lower semantic similarity: averages are 0.458 and 0.506 on AdvBench and HEx-PHI, versus DIA’s 0.820 and 0.809.ReNe’s ASRs decrease as the semantic-similarity threshold increases.Source: Figure 6 · p. 10, §5.2 · p. 10

5.4. Analysis of Model Size

  • Except for Llama-3, ASR rises with model size in the other tested families; the authors attribute this trend to a usefulness-safety tradeoff.The analysis assumes consistent alignment strategies within each model family.Source: §5.4 · p. 11, Figure 8 · p. 12

5.5. Ablation Study

  • Removing dialogue components generally lowers ASR; removing DIA-I’s system prompt drops ASR by more than 95% on Llama-2-7B and Llama-3-8B.Removing answer guidance also reduces ASR, and the authors say it reduces the probability of a bad answer.Source: Table 4 · p. 12, §5.5 · p. 12
  • Without prompt rewriting, both DIA variants’ ASRs grow more slowly; DIA-I remains near zero on Llama-2-7B, where rewriting contributes more than for DIA-II.The authors attribute DIA-I’s greater sensitivity to its direct use of malicious prompts in the input.Source: Figure 9 · p. 12, §5.5 · p. 12

5.6. Attack Against Defenses

  • DIA achieves superior defense pass rates over baselines; answer guidance masks jailbreak intent, while system prompt replacement weakens defensive system prompts.The tests use AdvBench and Gemma-2-9B; post-hoc detectors are excluded because of asynchronous decode SLO requirements.Source: Table 5 · p. 13, §5.6 · p. 13

6. Related Work

Prior work addresses alignment and jailbreaks through several training and attack strategies, typically focusing on single-turn interactions. DIA instead exploits historical dialogue, while existing defenses include both training-based and operational approaches.

6. Related Work

  • Alignment approaches include instruction fine-tuning, RLHF, and alternatives such as DPO, KTO, and ORPO that use supervised fine-tuning loss to approximate reinforcement learning.Source: §6 · p. 13
  • Jailbreak attacks range from white-box gradient-based methods and gray-box prefilling to black-box prompt attacks; prior methods focus on single-turn dialogue, unlike DIA’s use of historical dialogue.Source: §6 · p. 13
  • Defenses include learning-based robust optimization and adversarial training, plus strategy-based filters, secondary-LLM monitoring, and self-evaluation; the paper argues defenses should account for multi-turn interactions.Source: §6 · p. 13

7. Conclusion

DIA uses crafted dialogue history and LLM chat-template structure to jailbreak models, and the paper reports bypassing multiple defenses. The authors argue that security measures should account for historical dialogue interactions.

7. Conclusion

  • DIA exploits LLM chat-template structure and carefully crafted dialogues as a black-box jailbreak paradigm.Source: §7 · p. 13
  • The paper reports that DIA bypasses various LLM defenses.Source: §7 · p. 13
  • The authors emphasize that LLM security measures should consider historical dialogue interactions.Source: §7 · p. 13

8. Ethics Consideration

The authors report that experiments were conducted in controlled environments without real-world applications or exposing harmful outputs. They acknowledge disclosure risks while supporting transparency for AI-security research.

8. Ethics Consideration

  • All experiments were conducted in controlled environments, avoiding real-world applications and exposure of harmful outputs.Source: §8 · p. 13
  • The authors acknowledge that sharing attack-technique details may present ethical risks.Source: §8 · p. 14
  • They support transparency as a way to foster AI-security advances and collaboration toward safer LLMs.Source: §8 · p. 14

Appendix A. Evaluation Results on MaliciousInstruct

The appendix identifies single-query and multi-query attack results on MaliciousInstruct, presented in Table 6 and Figure 10, respectively.

Appendix A. Evaluation Results on MaliciousInstruct

  • Table 6 presents single-turn attack success rates on MaliciousInstruct.Source: Appendix A · p. 16
  • Figure 10 displays multi-query attack results on MaliciousInstruct.Source: Appendix A · p. 16
  • The appendix covers both single-query and multi-query attacks on MaliciousInstruct.Source: Appendix A · p. 16

Appendix B. Details of Template Inference Attack

Algorithm 3 details a template inference attack using swap strategies. It takes two candidate chat templates and a maximum number of tries, and outputs an inferred template.

Appendix B. Details of Template Inference Attack

  • Template Inference Attack (TIA) uses swap strategies to infer a model's chat template.Source: Appendix B · p. 16
  • TIA takes a maximum try count and two candidate chat templates as input, and returns the inferred template.Source: Appendix B · p. 16
  • The algorithm probes by swapping the two candidate templates before iterating over the allowed tries.Source: Appendix B · p. 16
  • Its decision step extracts elements from each template and checks whether their combinations appear in the query.Source: Appendix B · p. 16
  • Depending on which template-element combination appears in the query, the decision step returns Ti or Tj.Source: Appendix B · p. 16
Loading 2503.08195v1…