PaperlayerView exact revision on arXiv

Source-linked AI summary

ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs

Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, Radha Poovendran

arXiv:2402.11753v4cs.CLcs.AI

TL;DR

Semantics-only safety alignment leaves open whether LLMs can recognize harmful prompts represented beyond ordinary language semantics. The paper introduces VITC to measure ASCII-art recognition and ArtPrompt to exploit that weakness, finding effective unsafe-behavior induction across five LLMs and evaluated defenses.

  • Problem

    Safety alignment commonly focuses on natural-language semantics, leaving LLM recognition of prompts represented through ASCII art insufficiently evaluated.

  • Method

    The paper develops VITC for ASCII-art prompt recognition and ArtPrompt, which masks safety words and replaces them with ASCII-art representations in cloaked prompts.

  • Results

    ArtPrompt effectively and efficiently induces unsafe behaviors across five LLMs, outperforms five attacks on average, and bypasses three evaluated defenses.

  • Takeaways & Limitations

    Poor ASCII-art recognition exposes a safety vulnerability that can be exploited through a black-box jailbreak attack.

  • Takeaways & Limitations

    The paper evaluates ArtPrompt on five LLMs, while its performance against multimodal language models remains for future investigation.

Abstract

from arXiv · show

Safety is critical to the usage of large language models (LLMs). Multiple techniques such as data filtering and supervised fine-tuning have been developed to strengthen LLM safety. However, currently known techniques presume that corpora used for safety alignment of LLMs are solely interpreted by semantics. This assumption, however, does not hold in real-world applications, which leads to severe vulnerabilities in LLMs. For example, users of forums often use ASCII art, a form of text-based art, to convey image information. In this paper, we propose a novel ASCII art-based jailbreak attack and introduce a comprehensive benchmark Vision-in-Text Challenge (ViTC) to evaluate the capabilities of LLMs in recognizing prompts that cannot be solely interpreted by semantics. We show that five SOTA LLMs (GPT-3.5, GPT-4, Gemini, Claude, and Llama2) struggle to recognize prompts provided in the form of ASCII art. Based on this observation, we develop the jailbreak attack ArtPrompt, which leverages the poor performance of LLMs in recognizing ASCII art to bypass safety measures and elicit undesired behaviors from LLMs. ArtPrompt only requires black-box access to the victim LLMs, making it a practical attack. We evaluate ArtPrompt on five SOTA LLMs, and show that ArtPrompt can effectively and efficiently induce undesired behaviors from all five LLMs. Our code is available at https://github.com/uw-nsl/ArtPrompt.

1 Introduction

The paper identifies a vulnerability in semantics-focused safety alignment and introduces VITC and ArtPrompt to study and exploit LLMs’ difficulty recognizing ASCII art. Across five LLMs, ASCII-art prompts are poorly recognized, enabling unsafe behavior and strong jailbreak performance.

  • Motivation: Safety alignment efforts focus on natural-language semantics, overlooking alternative interpretations such as ASCII-art representations.The paper frames this semantics-only assumption as a potential source of exploitable safety vulnerabilities.
  • Benchmark and findings: VITC evaluates five LLMs on recognizing ASCII-art queries that cannot be interpreted solely through character semantics.The benchmark measures recognition of single letters or numbers and longer sequences represented as ASCII art.
  • Benchmark and findings: Recognition accuracy drops significantly, approaching zero, when ASCII-art queries contain more letters or numbers.The reported difficulty applies across the five evaluated models.
  • ArtPrompt: ArtPrompt masks safety words, replaces them with ASCII art, and combines the result with the remaining prompt to create a cloaked jailbreak prompt.The cloaked prompt is then sent to the victim LLM.
  • Evaluation: ArtPrompt induces unsafe behaviors from five LLMs and outperforms five comparison attacks on average while bypassing three evaluated defenses.The experiments use AdvBench and HEx-PHI and compare ArtPrompt with Direct Instruction, GCG, AutoDan, PAIR, and DeepInception.

2 Related Work

Prior jailbreak research includes gradient-based, evolutionary, instruction-following, manually crafted, and in-context approaches. Existing defenses are categorized around detecting harmful inputs or outputs, while this paper focuses on ASCII-art text recognition limitations.

  • Jailbreak Attacks: Earlier jailbreak attacks searched for harmful-triggering inputs using gradients, hierarchical genetic algorithms, instruction-following behaviors, manual construction, or in-context demonstrations.These approaches represent several distinct strategies for eliciting undesired outputs from LLMs.
  • Defenses against Jailbreak Attacks: Detection-based defenses apply input or output filters to identify and block potentially harmful prompts or generated responses.Examples include input perplexity, LLM-based harmful-output detection, and perturbation-based input detection.
  • ASCII Art Text: ASCII art uses printable elements to represent visuals and richly formatted text, including portraits, objects, and text layouts.The paper focuses on rich-formatting text represented as ASCII art.
  • ASCII Art Text: The paper positions SOTA LLMs as inadequately equipped to recognize prompts that encode information through ASCII art.This recognition limitation motivates evaluating models beyond semantic interpretation.

3 VITC Benchmark to Evaluate LLM Capabilities of ASCII Art Recognition

The VITC benchmark evaluates whether LLMs can recognize ASCII-art prompts beyond semantic interpretation, using single-character and multi-character datasets. Across five model families, recognition performance is poor, especially for longer sequences, while few-shot prompting and CoT provide only marginal gains.

  • Benchmark and task: VITC evaluates five LLM families on ASCII-art recognition using zero-shot, few-shot ICL, and CoT querying strategies.The task requires predicting labels for ASCII-art samples, with success defined by matching the ground-truth label.
  • Datasets: VITC-S contains 8,424 single-character samples across 36 classes and 234 human-filtered fonts, while VITC-L contains 8,000 sequence samples across 800 classes and 10 fonts.VITC-L sequences contain two to four characters and labels concatenate the corresponding character labels.
  • Metrics: Accuracy counts correctly predicted samples, while AMR measures the matched characters or digits and captures partial recognition for multi-character labels.When labels have length one, AMR reduces to accuracy; for VITC-L, it can distinguish partial from complete recognition.
  • Results: 25.19% is GPT-4’s highest Acc on VITC-S, and all evaluated models perform poorly on the ASCII-art recognition task.The reported accuracy is substantially below evaluations on tasks such as code completion, summarization, and math reasoning.
  • Results: 3.26% is GPT-4’s Acc on VITC-L, where recognition becomes more challenging because samples contain character sequences.The highest AMR across models is 54.39%, achieved by GPT-3.5, indicating partial recognition of roughly half the associated characters on average.
  • Results: Few-shot prompting and CoT provide marginal improvement, and additional demonstrations do not necessarily increase performance.Smaller Llama2 models may gain little or become confused by demonstrations.

4 A Jailbreak Attack via ASCII Art

ArtPrompt exploits LLMs’ difficulty interpreting ASCII art to bypass safety measures. It masks sensitive words, replaces them with ASCII art in the prompt, and produces a human-readable attack that can be automated.

  • Attack insight: ArtPrompt exploits recognition failures to replace safety-triggering words with ASCII art, potentially increasing the probability of bypassing safety measures.The surrounding context can still help LLMs infer the obscured word.
  • Attack pipeline: ArtPrompt has two steps: word masking and cloaked prompt generation.The first identifies sensitive words likely to trigger rejection; the second substitutes ASCII representations into the masked prompt.
  • Word masking: Word masking creates templates by replacing potentially rejected words, while leaving function words unmasked can reduce the number of generated prompts.A prompt with N words may yield N masked prompts in the worst case, but practical counts are lower.
  • Efficiency and automation: ArtPrompt is more efficient than optimization-based jailbreaks because it masks a finite set of prompt words instead of searching an iterative discrete token space.The attack can also be automated by stitching ASCII-generator output into the masked prompt.
  • Cloaked prompt generation: The ASCII art generator substitutes each masked word and integrates the result into the masked template to form a cloaked prompt.The resulting prompt remains readable to humans and is described as more stealthy and natural than token-manipulation attacks.

5 Experimental Evaluations of ArtPrompt

Experiments evaluate ArtPrompt across five aligned LLMs, two harmful-instruction datasets, baseline jailbreak attacks, and defenses. ArtPrompt induces unsafe behavior across models, achieves strong average performance, operates efficiently, and remains effective against several defenses.

  • Metrics: The evaluation measures Helpful Rate, Harmfulness Score, and Attack Success Rate, with ASR counting only responses rated as extremely harmful.HS ranges from 1 for no harm to 5 for extreme harm, while Dict-Judge evaluates non-refusal responses.
  • Results on AdvBench: 52% ASR, 3.6 HS, and 84% HPR are ArtPrompt’s highest average results across the evaluated jailbreak attacks.ArtPrompt is effective against all victim LLMs and outperforms all baselines on average, although AutoDAN and PAIR outperform it on Llama2.
  • Results on AdvBench: 52% ASR is ArtPrompt’s highest result among jailbreak attacks on Claude under the Ensemble configuration.Most baselines except GCG fail with ASR 0% in the cited Claude comparison.
  • Results on HEx-PHI: ArtPrompt induces unsafe behaviors across eleven HEx-PHI prohibited categories, including on the well-aligned GPT-4.Directly submitted harmful instructions generally produce safe behaviors, whereas ArtPrompt-modified instructions can induce unsafe responses.
  • Efficiency: One iteration gives ArtPrompt the highest ASR among the evaluated attacks because its cloaked prompts can be constructed and sent in parallel.Optimization-based attacks such as GCG require more sequential iterations.
  • Defenses and ablations: ArtPrompt bypasses PPL and Retokenization on all victim models, while Paraphrase is the strongest defense but still leaves 39% ASR and 3.18 HS on average.Retokenization may improve ASR, whereas Paraphrase can disrupt the ASCII-art arrangement.
  • Defenses and ablations: Vertical ASCII-art arrangements degrade ArtPrompt effectiveness, and tail-set fonts slightly reduce all metrics while leaving the attack effective.The authors recommend Top-1 and Ensemble strategies for best effectiveness.

6 Conclusion

The paper concludes that semantics-only safety alignment creates jailbreak vulnerabilities. It introduces VITC and ArtPrompt, showing that ASCII-art recognition failures can be exploited to provoke unsafe behaviors from aligned LLMs.

  • Semantics-only interpretation of safety-alignment corpora creates vulnerabilities to jailbreak attacks.
  • VITC evaluates whether LLMs recognize prompts that cannot be interpreted purely through character semantics.
  • Five SOTA LLMs struggled with the VITC recognition task.
  • ArtPrompt exploits these recognition vulnerabilities to induce unsafe behaviors from aligned LLMs.

7 Limitations

The evaluation covers five LLMs, while ArtPrompt's performance on multimodal language models remains untested. The authors hypothesize that textual cloaked prompts may still confuse such models.

  • The paper evaluates ArtPrompt on five LLMs but does not evaluate multimodal language models.
  • The authors hypothesize that ArtPrompt may remain effective against multimodal models because its cloaked prompts remain text-formatted.

8 Ethical Statement

The paper focuses on improving LLM safety by examining corpus interpretation beyond semantics. It also acknowledges that the demonstrated vulnerabilities and prompts could be misused to attack LLMs.

  • The paper examines how corpus interpretation should be addressed to enhance LLM safety under adversarial conditions.
  • Semantics-only interpretation of training corpora is presented as a limitation associated with existing LLM vulnerabilities.
  • The demonstrated vulnerabilities and prompts could be repurposed or misused by malicious entities to attack LLMs.

A Experiment Details

The appendix details baseline attacks, defenses, font selection, generation settings, and visual summaries used to evaluate ArtPrompt. Fonts were selected according to recognition accuracy across models, with an additional GPT-4-generated font.

  • Experiment setup: Baseline experiments use GCG, AutoDAN, PAIR, and DeepInception implementations, with transferable settings for black-box models where specified.
  • Defense settings: PPL-Pass filters queries whose perplexity reaches the threshold T = 175.57 and returns a standardized refusal response.
  • Defense settings: Paraphrase uses GPT-3.5 to rewrite the original instruction before forwarding it to the target LLM.
  • Defense settings: Retokenization applies BPE-dropout by dropping 20% of merges from the BPE tokenizer.
  • Font selection: Font selection uses accuracy across LLMs, choosing head-set fonts with high accuracy and tail-set fonts with low accuracy.
  • Font selection: The GPT-4-generated ‘Gen’ font is generally well recognized by all models and is included among head-set fonts.
  • Generation configuration: Closed-source models use temperature 0 with default API parameters, while Llama2 uses temperature 0.6 and top-p 0.9.

B.1 Evaluation on VITC

Few-shot ICL and CoT yield only marginal improvements in VITC-S recognition over zero-shot settings. More demonstrations do not consistently help, and smaller Llama2 models barely improve.

  • Increasing the number of few-shot examples does not consistently enhance recognition performance.
  • Llama2 7B, 13B, and 70B show that smaller models barely gain performance improvements.The authors relate this pattern to emergent ability.

B.2 ArtPrompt Evaluation

ArtPrompt remains effective under multiple mask settings, while few-shot recognition offers only marginal gains. The examples illustrate ASCII-art word reconstruction being paired with a harmful goal.

  • ArtPrompt attack performance is comparable between one-word and multiple-word masking settings.For GPT-3.5, Gemini, and Llama2, masking multiple words causes a slight ASR drop because the instructions are harder to comprehend.
  • Few-shot ICL and CoT provide marginal recognition improvements across models compared with zero-shot settings.Increasing the number of examples does not consistently improve performance.
  • The demonstrated prompt asks GPT-4 to identify an 11-letter word encoded as ASCII art in an 11-row, 20-column grid.Its instructions specify splitting rows and columns and assembling each letter step by step without outputting the word.
  • The GPT-4 demonstration pairs the ASCII-art prompt with the harmful goal of making and distributing counterfeit money.
Loading 2402.11753v4…