PaperlayerView exact revision on arXiv

Source-linked AI summary

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection

Zhipeng Wei, Yuqi Liu, N. Benjamin Erichson

arXiv:2411.01077v5cs.CLcs.LG

TL;DR

Judge LLMs can be vulnerable to token segmentation bias, which alters subword tokenization and embeddings and may cause harmful content to be classified as safe. The paper introduces Emoji Attack, using in-context learning to insert emojis into generated text and exploit this weakness. Experiments report reduced unsafe prediction rates across ten state-of-the-art Judge LLMs, while robustness varies across reasoning models.

  • Problem

    Judge LLMs used for safety moderation may be vulnerable to token segmentation bias, raising the question of whether emojis can alter their decision boundaries and let harmful content bypass filters.

  • Method

    Emoji Attack uses in-context learning to insert emojis into generated text, exploiting token segmentation bias, semantic ambiguity, and intrinsic meaning before Judge LLM evaluation.

  • Results

    A 12% reduction in unsafe prediction rates is reported across ten state-of-the-art Judge LLMs, with token segmentation bias alone reducing unsafe detection by 12% and surrogate-guided splits adding 4%.

  • Takeaways & Limitations

    Emoji-based tokenization and semantic effects expose a robustness gap in current Judge LLM moderation frameworks and motivate defenses that account for non-textual artifacts.

  • Takeaways & Limitations

    Robustness varies among reasoning models: DeepSeek is reported as robust to emojis, whereas o3-mini remains sensitive, and the responsible factors are unclear.

Abstract

from arXiv · show

Jailbreaking techniques trick Large Language Models (LLMs) into producing restricted output, posing a potential threat. One line of defense is to use another LLM as a Judge to evaluate the harmfulness of generated text. However, we reveal that these Judge LLMs are vulnerable to token segmentation bias, an issue that arises when delimiters alter the tokenization process, splitting words into smaller sub-tokens. This alters the embeddings of the entire sequence, reducing detection accuracy and allowing harmful content to be misclassified as safe. In this paper, we introduce Emoji Attack, a novel strategy that amplifies existing jailbreak prompts by exploiting token segmentation bias. Our method leverages in-context learning to systematically insert emojis into text before it is evaluated by a Judge LLM, inducing embedding distortions that significantly lower the likelihood of detecting unsafe content. Unlike traditional delimiters, emojis also introduce semantic ambiguity, making them particularly effective in this attack. Through experiments on state-of-the-art Judge LLMs, we demonstrate that Emoji Attack substantially reduces the unsafe prediction rate, bypassing existing safeguards.

1. Introduction

The paper identifies token segmentation bias as a weakness in Judge LLM moderation and introduces Emoji Attack, which inserts emojis through in-context learning to reduce harmful-content detection. Across evaluated Judge LLMs, the attack amplifies jailbreaks by exploiting both tokenization changes and emoji-induced semantic ambiguity.

  • Motivation: The paper asks whether benign constructs such as emojis can systematically alter Judge LLM decision boundaries so harmful content bypasses moderation filters.
  • Token Segmentation Bias: Token segmentation bias arises when minor input changes alter subword tokenization and distort contextual embeddings, allowing harmful content to be misclassified as safe.The paper links this vulnerability to the reliability of LLM-based moderation filters.
  • Findings: A 12% reduction in unsafe content detection rates is reported for token segmentation bias across state-of-the-art Judge LLMs, with a further 4% reduction using a surrogate model for optimal sub-token splits.The comparison includes Llama Guard and Llama Guard 2.
  • Emoji Attack: Unlike delimiter-based attacks, emojis combine token segmentation effects with semantic ambiguity and intrinsic meaning that can confuse contextual interpretation.The paper notes that positive or neutral emoji connotations may encourage harmful text to be treated as benign.
  • Emoji Attack: Emoji Attack uses in-context learning to insert emojis into generated responses, distorting the Judge LLM embedding space before evaluation.The attack is designed to work with existing jailbreak techniques rather than replace them.
  • Evaluation: All ten evaluated Judge LLMs, including open and commercial models, are reported as vulnerable to Emoji Attack.The evaluation includes Llama Guard, Llama Guard 2, ShieldLM, WildGuard, GPT-3.5, GPT-4, Gemini, and Claude.

2. Related Work

Prior work develops Judge LLMs and jailbreak attacks, but most jailbreak research targets the content-generating model rather than the Judge LLM. Emoji Attack addresses this gap with a lightweight strategy that exploits token segmentation bias without extensive optimization.

  • Judge LLMs: Judge LLM research has documented biases such as superficial preference, positional bias, self-preference, and verbosity preference that can undermine safety evaluation reliability.
  • Judge LLMs: Safety-focused Judge LLMs include Llama Guard, Llama Guard 2, ShieldLM, and WildGuard, alongside commercial models such as GPT-3.5 and GPT-4.
  • Jailbreaking Attacks: Token-level jailbreaks optimize malicious-prompt tokens, whereas prompt-level methods use additional LLMs to craft, refine, mutate, or reason over jailbreak prompts.Examples include GCG, PAIR, TAP, and GPTFuzz.
  • Attacks on Judge LLMs: Most existing jailbreak approaches target the target LLM, while comparatively less work attacks the Judge LLM that decides whether generated content is harmful.The cited Judge-LLM attack using a universal adversarial prefix remains query-intensive and difficult to scale.
  • Emoji Attack: Emoji Attack exploits token segmentation bias, avoids extensive optimization, and integrates with existing jailbreak methods as a lightweight way to mislead Judge LLMs.

3. Methodology

The methodology models target and Judge LLMs, investigates token segmentation bias through mid-splitting, and develops Emoji Attack to induce embedding distortions that reduce unsafe-content detection.

  • 3.2. Token Segmentation Bias: Token segmentation bias occurs when sub-token embeddings differ from original-token embeddings, unintentionally altering the model’s perception.Mid-split inserts a delimiter at the word midpoint; for example, “bomb” becomes “bo” and “mb”.
  • 3.2. Token Segmentation Bias: 12% average reduction in unsafe prediction rates occurs when offensive phrases undergo midpoint word splitting across four open-source Judge LLMs.The experiment uses 402 short offensive phrases and evaluates whether each Judge LLM classifies them as unsafe.
  • 3.2. Token Segmentation Bias: Lower cosine similarity after mid-splitting correlates with lower unsafe-prediction probabilities, linking embedding distortion to harmful-content misclassification.The analysis uses gtr-t5-xl as a lightweight surrogate model to compute embedding similarities.
  • 3.3. Emoji Attack: Emoji Attack inserts emojis into generated responses to manipulate token boundaries and embeddings, exploiting token segmentation bias to evade Judge LLM moderation.Emojis add semantic ambiguity beyond ordinary delimiters, potentially changing contextual interpretation.
  • 3.3. Emoji Attack: The black-box attack uses in-context learning so a target LLM naturally inserts emojis, without requiring direct control over the Judge LLM.Benign references reduce detection while the target model emulates emoji usage; the insertion position is not guaranteed to be optimal.

4. Experiments

The experiments evaluate Emoji Attack and token segmentation bias across open-source and commercial Judge LLMs, showing reduced unsafe prediction ratios in many settings. Results vary by model, emoji semantics, jailbreak, and access setting.

  • Emoji Attack with jailbreaks: The Emoji Attack generally lowers unsafe prediction ratios across existing jailbreak prompts, although several model–jailbreak combinations show increases.Deepinception drops from 71.9% to 3.5% with ShieldLM, while some exceptions are attributed to insufficient emoji insertion in the one-shot example.
  • Cross-model results: WildGuard’s unsafe prediction ratio decreases by approximately 23%, while GPT-4’s decreases by 6.6% under Emoji Attack.
  • Jailbreak baselines: CodeChameleon has a 46.2% unsafe prediction ratio, which falls to 35.2% when combined with Emoji Attack.
  • Different Emojis: Open-source Judge LLMs show reduced unsafe prediction ratios across emojis, whereas commercial models respond differently to innocent and toxic emoji meanings.In commercial models, innocent emojis usually reduce the ratio except for GPT-4, while toxic emojis generally have the opposite effect; multiple emojis do not improve the attack.
  • Model robustness: GPT-4 is highly robust to emojis, while DeepSeek is relatively robust and o3-mini remains sensitive among the reasoning models.
  • Token segmentation and white-box attack: Emoji insertion further reduces the prediction ratio from 59.6% to 41.3%, and position selection improves attack effectiveness against open-source Judge LLMs.Commercial models are more robust in some tests but remain susceptible when evaluating unseen harmful outputs from jailbreak attacks.
  • Comparison with GCG Suffix: Emoji Attack generalizes better across black-box Judge LLMs than an optimized GCG suffix, whose performance degrades outside white-box evaluation.

5. Conclusion

The paper identifies token segmentation bias as a weakness in Judge LLM safety detection and introduces Emoji Attack to exploit it. Across ten state-of-the-art Judge LLMs, the approach reduces unsafe prediction rates while adding semantic ambiguity to tokenization changes.

  • Emoji Attack exploits token segmentation bias and reduces unsafe prediction rates by 12% across ten state-of-the-art Judge LLMs.
  • Unlike traditional segmentation attacks, Emoji Attack combines tokenization changes with emojis’ semantic ambiguity and intrinsic meaning.
  • Few studies have examined Judge LLM biases specifically in safety-risk detection, leaving current moderation frameworks vulnerable to these attacks.The paper argues that future defenses should account for tokenization vulnerabilities and the semantic impact of non-textual artifacts such as emojis.

Impact Statement

The study identifies token segmentation bias in Judge LLMs and introduces Emoji Attack, which reduces harmful-content detection rates across state-of-the-art Judge LLMs.

  • Emoji Attack reduces harmful-content detection rates across state-of-the-art Judge LLMs, exposing a vulnerability in current moderation systems.The authors frame this vulnerability as important for safety-critical AI systems and resilient moderation research.

A. Attention Visualization of Token Segmentation Bias

Token segmentation changes attention distributions and increases the number of sub-tokens, altering how Judge LLMs represent the input.

  • A. Attention Visualization of Token Segmentation Bias: Segmented prompts produce distinct attention distributions and more sub-tokens than default prompts.Figure 6 compares attention values for default and segmented prompts in Llama Guard.
  • A. Attention Visualization of Token Segmentation Bias: The segmented subtokens ’p’ and ’ir’ receive higher cross-attention than the corresponding original tokens ’port’ and ’air’.This indicates a shift in attention patterns caused by segmentation.
  • A. Attention Visualization of Token Segmentation Bias: The altered attention pattern suggests embedding-space changes that can weaken harmful-signal associations and lower unsafe-prediction probability.The passage presents this as a mechanism by which segmentation may affect safety judgments.

B. Comparison between Offensive Phrases and Those Appending Emojis

Appending emojis changes the unsafe probability of offensive phrases, with positive emojis tending to reduce that probability.

  • B. Comparison between Offensive Phrases and Those Appending Emojis: Positive emojis tend to decrease the unsafe probability of offensive phrases when appended to them.Figure 7 visualizes unsafe-probability changes after emojis are added, with phrases sorted by their original unsafe probabilities.
  • B. Comparison between Offensive Phrases and Those Appending Emojis: Emojis provide varied semantic information, including positive sentiment from a smiley and negative or offensive sentiment from a middle-finger emoji.The unsafe and safe probabilities are computed by applying softmax to logit values in Llama Guard.

C. Effect of the Number of Inserted Emojis

Increasing the number of inserted emojis raises unsafe prediction ratios in evaluations of Llama Guard and Llama Guard 2, with position selection compared against random insertion.

  • C. Effect of the Number of Inserted Emojis: Unsafe prediction ratios gradually increase as more emojis are inserted in Llama Guard and Llama Guard 2.The reported increase is attributed to an embedding-space shift that deceives the Judge LLMs.
  • C. Effect of the Number of Inserted Emojis: The evaluation compares random emoji insertion with the proposed position selection strategy across both Judge LLMs.Figure 8 reports the effect of inserted-emoji count for LG and LG2 under both strategies.
  • C. Effect of the Number of Inserted Emojis: Even a small number of inserted emojis can alter the response’s unsafe prediction behavior.The passage states that this effect occurs through a corresponding shift in embedding space.

D. Effect of Other Delimiters

Various delimiters reduce Llama Guard’s unsafe prediction ratio relative to default prompts, and position selection strengthens this reduction. Figure 9 summarizes these delimiter effects.

  • Various delimiters markedly decrease Llama Guard’s unsafe prediction ratio compared with default prompts without delimiters.Figure 9 compares delimiter conditions against the red dashed default-prompt baseline.
  • Combining delimiters with the position selection strategy produces an even greater reduction in unsafe predictions.
  • The results confirm that token segmentation bias can be induced through multiple delimiter designs and strengthened by selectively inserting tokens.

E. Potential Defense Strategy.

The paper considers filtering abnormal characters as a defense, but varied delimiters make detection less predictable. A mixed character-and-emoji delimiter can cause a filter to produce benign text that Judge LLMs classify as safe.

  • Using different delimiters across tokens complicates filtering because it disrupts token patterns in less predictable ways.A single delimiter repeated across multiple tokens is described as relatively easy to detect.
  • A mix of the character “b” and a smiley face emoji causes the additional LLM filter to generate a benign response differing substantially from the harmful original.Figure 10 illustrates this filtering setup with gpt-3.5-turbo.
  • The resulting benign response is classified as safe by Judge LLMs, allowing the original harmful response to reach users.The paper leaves combination attacks and further defense strategies for future work.
Loading 2411.01077v5…