PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates

Fengqing Jiang, Zhangchen Xu, Luyao Niu, Bill Yuchen Lin, Radha Poovendran

arXiv:2406.12935v2cs.CRcs.AIcs.LG

TL;DR

Chat templates are widely used to tune LLMs, but their effects on safety alignment have received limited study. This paper identifies ChatBug and develops attacks that exploit the mismatch between rigid model-facing formats and user prompts. The attacks elicit unintended responses across evaluated LLMs, while adversarial training mitigates the vulnerability at a performance cost.

  • Problem

    How chat templates affect LLM safety alignment has received limited study, despite their use in tuning models and the importance of safety for deployment.

    Source: §1 · p. 2

  • Method

    The paper identifies ChatBug and develops format mismatch and message overflow attacks that exploit chat-template formats to elicit unintended responses.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    ChatBug elicited unintended responses across eight state-of-the-art LLMs and increased existing jailbreaks' attack success rates; Overflow-FS achieved 100% ASR-R against Llama 3.

    Source: §1 · p. 2, §5.2 · p. 6

  • Takeaways & Limitations

    Adversarial training can mitigate ChatBug but significantly degrades model performance, exposing a safety–helpfulness trade-off and an open instruction-tuning challenge.

    Source: §1 · p. 2

    Detection-based countermeasures may raise latency and false-positive concerns that can degrade performance and hinder user experience.

    Source: Limitation and Ethical Statement · p. 7

Abstract

from arXiv · show

Large language models (LLMs) are expected to follow instructions from users and engage in conversations. Techniques to enhance LLMs' instruction-following capabilities typically fine-tune them using data structured according to a predefined chat template. Although chat templates are shown to be effective in optimizing LLM performance, their impact on safety alignment of LLMs has been less understood, which is crucial for deploying LLMs safely at scale. In this paper, we investigate how chat templates affect safety alignment of LLMs. We identify a common vulnerability, named ChatBug, that is introduced by chat templates. Our key insight to identify ChatBug is that the chat templates provide a rigid format that need to be followed by LLMs, but not by users. Hence, a malicious user may not necessarily follow the chat template when prompting LLMs. Instead, malicious users could leverage their knowledge of the chat template and accordingly craft their prompts to bypass safety alignments of LLMs. We develop two attacks to exploit the ChatBug vulnerability. We demonstrate that a malicious user can exploit the ChatBug vulnerability of eight state-of-the-art (SOTA) LLMs and effectively elicit unintended responses from these models. Moreover, we show that ChatBug can be exploited by existing jailbreak attacks to enhance their attack success rates. We investigate potential countermeasures to ChatBug. Our results show that while adversarial training effectively mitigates the ChatBug vulnerability, the victim model incurs significant performance degradation. These results highlight the trade-off between safety alignment and helpfulness. Developing new methods for instruction tuning to balance this trade-off is an open and critical direction for future research

1 Introduction

The paper identifies ChatBug as a vulnerability arising because chat templates impose a rigid format on models that users need not follow. It develops attacks to exploit this mismatch, evaluates their reach, and examines mitigation trade-offs.

1 Introduction

  • Chat templates are commonly used in instruction and preference tuning, and have been shown to improve LLM performance, but their effects on safety alignment have received limited analysis.Prior work also reports that chat templates can mitigate prompt injection attacks.Source: §1 · pp. 1–2
  • Chat templates introduce ChatBug, which malicious users can exploit by crafting prompts that disregard the rigid formats expected of models.The paper frames this as a gap between the format LLMs should follow and the formats users may use.Source: §1 · p. 2
  • Two attacks exploit ChatBug: format mismatch modifies the default chat format, while message overflow inserts tokens into a model-reserved field.The attacks are presented as ways to elicit unintended responses.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • ChatBug can be exploited across eight state-of-the-art LLMs, and existing jailbreak attacks can use it to increase their attack success rates.The evaluated models include open- and closed-source systems.Source: §1 · p. 2
  • Adversarial training can mitigate ChatBug, but significantly degrades model performance, highlighting a safety–helpfulness trade-off.The paper identifies balancing this trade-off during instruction tuning as an open research direction.Source: §1 · p. 2

2 Related Work

Prior work studies how prompt design affects model performance and how aligned LLMs can be attacked. The paper situates its investigation of chat-template effects on safety alignment alongside this research.

2 Related Work

3 Identifying ChatBug Vulnerability

The paper reviews autoregressive generation and chat-template structure before identifying ChatBug. Its central mechanism is that users can exploit rigid model-facing formats without following those formats themselves.

3.1 Preliminary Background

  • An autoregressive LLM predicts a distribution over its vocabulary for the next token and repeats this process until a stopping criterion is met.The described stopping criteria include reaching the maximum generation length or producing an end-of-sequence token.Source: §3.1 · p. 2
  • Chat templates organize instruction-tuning data into multi-turn conversations, using beginning- and end-of-turn tokens and role markers to structure messages.The paper distinguishes these turn delimiters from beginning- and end-of-sequence tokens.Source: §3.1 · pp. 2–3, Table 1 · p. 3

3.2 Chat Templates Induce a Common Vulnerability: ChatBug

  • ChatBug arises because users may disregard the rigid format expected by the model and append the model's role and the beginning of a desired harmful response.The user can thereby prompt the model to complete that response rather than continue the conversation in its intended role; template knowledge is often publicly available, and white- or grey-box access is not required.Source: §3.2 · p. 3

4 Exploit ChatBug

ChatBug can be exploited by changing chat-template formatting or overflowing tokens into a reserved field; combining these attacks with jailbreaks can increase unintended-response probability.

4.1 Format Mismatch Attack

  • The desired harmful-response sequence’s probability increases by a factor of 1010 under the format mismatch attack.The comparison averages over 50 instructions and is shown across decoding steps.Source: Figure 2 · p. 3, §4.1 · pp. 3–4
  • Format mismatch modifies or omits required template tokens, which can change how the victim interprets a query and lead to harmful or unintended responses.Unlike jailbreaks that manipulate the prompt message while following the standard template, format mismatch alters the chat template.Source: §4.1 · p. 3

4.2 Message Overflow Attack

  • The probability of the desired harmful response rises as overflow-token count increases from 0 to 9, averaged over 50 instructions.At zero overflow tokens, the user does not launch the message overflow attack.Source: Figure 3 · p. 4, §4.2 · p. 4
  • Message overflow extends the user message beyond its EOT and role-control token with a harmful-response prefix, prompting the model to complete it autoregressively.The paper presents this attack as a unified framework for existing attacks that use seemingly distinct strategies.Source: §4.2 · p. 4

4.3 ChatBug Boosts Jailbreak Attacks

  • Combining ChatBug attacks with existing jailbreaks can boost the probability of unintended responses by tampering with both the template and conversation messages.Source: §4.3 · p. 4

5 Experimental Assessment of ChatBug

The experiments evaluate ChatBug attacks across open- and closed-source LLMs, finding that they bypass safety alignment and can substantially strengthen existing jailbreak attacks.

5.1 Experimental Setup

  • The evaluation uses AdvBench’s 520 harmful-instruction prompts and measures ASR through refusal-response matching and Llama-Guard-2 moderator assessment.The baseline, Direct Instruct, directly prompts victim models with harmful queries.Source: §5.1 · pp. 4–5

5.2 Main Results

5.1 Experimental Setup

  • Six attack settings cover format mismatches and message overflows, including fixed, query-tailored, and uncensored-LLM-generated affirmative prefixes.Overflow-FS uses an uncensored LLM to generate five prefixes by default.Source: §5.1 · p. 5

5.2 Main Results

  • Mismatch-C produced 1.3% ASR-R on Llama-3, which the authors interpret as evidence that safety alignment associated with chat templates is transferable.Llama-3 uses a different chat template from ChatML.Source: §5.2 · p. 6

5.3 ChatBug Boosts Jailbreak Attacks

  • GPTFuzzer with Mismatch-∅ achieved 99.2% ASR-R, versus 9.0% ASR-R without ChatBug.The authors report that exploiting ChatBug significantly increases the ASR of GCG, GPTFuzzer, and ArtPrompt.Source: §5.3 · p. 6

6 Countermeasures to ChatBug

The section compares mitigation-based and detection-based countermeasures for ChatBug. In Vicuna evaluations, adversarial training mitigated the vulnerability but significantly degraded performance.

6.1 Description of Countermeasures

  • Countermeasures comprise mitigation methods—Self-Reminder, SafeDecoding, and Adversarial Training—and detectors that block flagged queries or responses.Source: §6.1 · p. 6

6.2 Evaluation of Countermeasures

  • Evaluation uses Vicuna, measures ASR and MT-Bench, and tests Adversarial Training at 5 and 20 fine-tuning epochs.Source: §6.2 · p. 7
  • MT-bench fell from 6.28 to 6.02: Self-Reminder and SafeDecoding failed except against Mismatch-C, while more-epoch Adversarial Training mitigated ChatBug with significant performance loss.Source: §6.2 · p. 7

7 Conclusion and Future Work

The conclusion identifies ChatBug as a vulnerability induced by chat templates and describes attacks and countermeasures. It reports that adversarial training mitigates the vulnerability at a performance cost, leaving safety and helpfulness in tension.

7 Conclusion and Future Work

  • ChatBug, induced by chat templates, was exploited through two attacks against eight state-of-the-art aligned LLMs, and jailbreak attacks gained higher success rates.Source: §7 · p. 7
  • 6.28 to 6.02: Adversarial Training mitigated ChatBug but significantly degraded Vicuna's MT-Bench performance.Source: Table 5 · p. 7, §7 · p. 7

Limitation and Ethical Statement

A Experimental Details

The appendix records jailbreak and defense setups, API-access conditions for commercial models, and examples of the two ChatBug attacks.

A.1 Jailbreak Attack Setup

A.2 Attack Commercil LLMs with API Access

  • ChatBug attacks depend on API freedom: Claude permits response formatting and prefixes, while Azure GPT-3.5 permits organizing prompts without input-format restrictions.Source: §A.2 · p. 10

A.3 Defense Setup

  • Defense setup uses Self-Reminder prompts, SafeDecoding defaults, and adversarially augmented harmful instructions; Vicuna is fine-tuned with LoRA and a 60/40 train-evaluation split.Source: §A.3 · p. 10

B Examples of Attacks

C.1 Ablation Study of Overflow-FS

The ablation reports that using more shots in Overflow-FS prompts increases both reported attack-success rates.

C.1 Ablation Study of Overflow-FS

  • Both ASR-R and ASR-M increase as Overflow-FS uses more shots to prompt uncensored LLMs.Source: §C.1 · p. 10

C.2 Analysis of Tokenizer Effect on Mismatch Attack

Mismatch attacks use ChatML or Vicuna formats, but the results attribute the vulnerability to the chat templates rather than tokenizer vocabulary. Overflow-FS ASR also rises with more shots across evaluation methods.

C.2 Analysis of Tokenizer Effect on Mismatch Attack

  • Mismatch-∅, Mismatch-C, and Mismatch-V are the three setups; Mismatch-C uses ChatML and Mismatch-V uses Vicuna.Source: §C.2 · pp. 10–11, Figure 5 · p. 11, Figure 6 · p. 11
  • Overflow-FS ASR monotonically increases with shot count, regardless of evaluation method.Source: Figure 4 · p. 11
  • Known template tokens and shared Llama 2 vocabulary do not explain mismatch results; the passage attributes the vulnerability to chat templates.Source: §C.2 · p. 11

C.3 Additional Experimental Results on Adversarial Training

Additional adversarial-training experiments vary benign-example fractions and training epochs. Mixing benign data may preserve MT-Bench performance, but does not improve ChatBug mitigation effectiveness across attacks.

C.3 Additional Experimental Results on Adversarial Training

  • 80% adversarial examples and 5 training epochs yielded the best MT-Bench performance and lowest ASR among the tested settings.Source: Table 7 · p. 11, §C.3 · p. 11
  • Mixing benign examples with adversarial ones did not improve adversarial training’s effectiveness against ChatBug, and may not mitigate all attacks even when MT-Bench performance is preserved.Source: §C.3 · p. 11, Table 7 · p. 11
  • The experiments varied benign-example fractions of 20% and 80% and fine-tuning durations of 5 and 20 epochs; the authors identify dataset choice and epoch count as factors in effectiveness.Source: §C.3 · p. 11
Loading 2406.12935v2…