PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models

Xiaogeng Liu, Nan Xu, Muhao Chen, Chaowei Xiao

arXiv:2310.04451v2cs.CLcs.AI

TL;DR

Existing jailbreak methods either rely on manual prompt crafting or produce semantically meaningless prompts vulnerable to perplexity detection. AutoDAN uses a hierarchical genetic algorithm to automate stealthy prompt generation, and the paper reports stronger attack performance, transferability, and defense bypass than baselines, while noting computational and robustness limitations.

  • Problem

    Existing jailbreak techniques face a trade-off between manual scalability and the semantic meaninglessness and perplexity-detection vulnerability of token-based automation.

  • Method

    AutoDAN uses a hierarchical genetic algorithm tailored to structured prompt text, initialized from handcrafted jailbreak prompts and operating across sentence and word choices.

  • Results

    AutoDAN improves attack strength, transferability, and universality over baselines while generating semantically meaningful prompts that bypass perplexity-based defenses.

  • Takeaways & Limitations

    AutoDAN provides an automated approach for generating stealthy jailbreak prompts and supports evaluation of aligned-LLM robustness across different settings.

  • Takeaways & Limitations

    The method has computational cost and performs poorly against Llama2 with robust system prompts, although it performs well across most current LLMs.

Abstract

from arXiv · show

The aligned Large Language Models (LLMs) are powerful language understanding and decision-making tools that are created through extensive alignment with human feedback. However, these large models remain susceptible to jailbreak attacks, where adversaries manipulate prompts to elicit malicious outputs that should not be given by aligned LLMs. Investigating jailbreak prompts can lead us to delve into the limitations of LLMs and further guide us to secure them. Unfortunately, existing jailbreak techniques suffer from either (1) scalability issues, where attacks heavily rely on manual crafting of prompts, or (2) stealthiness problems, as attacks depend on token-based algorithms to generate prompts that are often semantically meaningless, making them susceptible to detection through basic perplexity testing. In light of these challenges, we intend to answer this question: Can we develop an approach that can automatically generate stealthy jailbreak prompts? In this paper, we introduce AutoDAN, a novel jailbreak attack against aligned LLMs. AutoDAN can automatically generate stealthy jailbreak prompts by the carefully designed hierarchical genetic algorithm. Extensive evaluations demonstrate that AutoDAN not only automates the process while preserving semantic meaningfulness, but also demonstrates superior attack strength in cross-model transferability, and cross-sample universality compared with the baseline. Moreover, we also compare AutoDAN with perplexity-based defense methods and show that AutoDAN can bypass them effectively.

1 University of Wisconsin–Madison, 2 USC, 3 University of California, Davis

The paper warns that it contains potentially offensive and harmful text.

  • The paper contains potentially offensive text.
  • The paper contains potentially harmful text.
  • Readers should be aware of the paper’s potentially harmful content.

1 INTRODUCTION

Existing jailbreak methods trade off automation against semantic meaningfulness and resistance to perplexity defenses. AutoDAN addresses this gap with a hierarchical genetic algorithm designed to generate stealthy, automated jailbreak prompts.

  • Existing automatic attacks can generate nonsensical prompts that perplexity-based defenses readily detect.
  • The paper seeks automated jailbreak prompts that preserve the meaningfulness and fluency of handcrafted attacks.
  • AutoDAN applies a hierarchical genetic algorithm to structured prompt text, using sentence- and word-level crossover policies.
  • The method begins from handcrafted jailbreak prompts and searches a fine-grained space while aiming to avoid local optima.
  • AutoDAN is presented as an automated, efficient, and stealthy jailbreak attack against LLMs.
  • AutoDAN surpasses the baseline by 60% attack strength and is described as immune to perplexity defense.

2 BACKGROUND AND RELATED WORKS

The background situates AutoDAN within aligned-LLM safety research and the development of manual and learning-based jailbreak attacks. It presents the method as a bridge between community-discovered prompts and algorithmic designs.

  • Alignment research reshapes LLM behavior using data that reflect human values and expectations.
  • Jailbreak attacks exploit carefully phrased prompts to induce aligned LLMs to answer malicious questions without refusing.
  • Prior jailbreak attacks are grouped into manually written and learning-based methods, including DAN and GCG attacks.
  • AutoDAN is initialized with handcrafted prompts and evolved with a hierarchical genetic algorithm.
  • The authors position AutoDAN as an analytical tool for assessing LLM robustness.

3 METHOD

The method formulates jailbreak generation as an optimization problem over structured prompt text. It combines population-based genetic search, fitness evaluation, handcrafted initialization, and hierarchical policies operating across paragraph and sentence levels.

  • Threat model: Jailbreak attacks combine prompts with malicious questions and seek responses that violate the model’s aligned constraints.
  • Formulation: The attack objective optimizes jailbreak prompts to increase the probability of affirmative target responses.
  • Genetic algorithms: Genetic algorithms evolve candidate populations through fitness evaluation, crossover, mutation, and termination criteria.
  • Population initialization: AutoDAN initializes a diverse population from an effective handcrafted prototype while limiting deviation from its structure and meaning.
  • Fitness evaluation: Fitness is defined as the negative log-likelihood loss, so higher fitness corresponds to a lower loss for the target response.
  • Genetic policies: AutoDAN-GA uses multi-point crossover, while AutoDAN-HGA exploits text hierarchy through paragraph- and sentence-level search policies.
  • AUTODAN-HGA: AutoDAN-HGA conducts optimization consistently, whereas AutoDAN-GA becomes stuck at a local minimum.
  • Genetic policies: Selection preserves the top N ∗α prompts unchanged before crossover and mutation act on the remaining parent prompts.

4 EVALUATIONS

AutoDAN is evaluated against automatic and handcrafted baselines for attack effectiveness, stealthiness, defense robustness, transferability, universality, and component contributions. Across these tests, it improves attack success, produces more semantically meaningful prompts, transfers across models and requests, bypasses perplexity defense, and benefits from its proposed modules.

  • Experimental setup: The evaluation uses 520 AdvBench harmful-behavior requests and compares AutoDAN with GCG under standardized attack-success and GPT-recheck metrics.The tested models include Vicuna-7b, Guanaco-7b, Llama2-7b-chat, and GPT-3.5-turbo.
  • Attack effectiveness and stealthiness: About 8% higher average ASR than the automatic baseline and about 250% higher effectiveness than handcrafted DAN are reported for AutoDAN.The evaluation also reports over 10% ASR improvement on robust Llama2 for AutoDAN variants.
  • Attack effectiveness and stealthiness: AutoDAN generates prompts with lower PPL than GCG and comparable PPL to handcrafted DAN, indicating greater semantic meaningfulness and stealthiness.The prompts are evaluated by generating one jailbreak prompt per malicious request and testing the victim-model responses.
  • Effectiveness against defense: Perplexity defense substantially reduces GCG attack strength but does not influence AutoDAN or handcrafted DAN.The defense rejects inputs whose perplexity exceeds a threshold based on AdvBench requests.
  • Transferability and universality: AutoDAN transfers more effectively to black-box models and achieves higher cross-sample universality than the baselines.Universality tests a prompt optimized for request Q_i on requests Q_i+1 through Q_i+20.

5 LIMITATION AND CONCLUSION

AutoDAN is presented as an automated jailbreak method that preserves prompt stealthiness through a hierarchical genetic algorithm. The authors report effectiveness across settings but acknowledge computational cost and weaker performance against robust Llama2 system prompts.

  • Limitation: The method requires generation time despite being more efficient than the GCG baseline.The authors identify computational cost as a limitation of AutoDAN.
  • Limitation: AutoDAN performs poorly against Llama2 models with robust system prompts, resembling a vanishing-gradient problem.The authors state that performance remains strong across the majority of current LLMs.
  • Conclusion: AutoDAN preserves jailbreak-prompt stealthiness while enabling automated deployment through a hierarchical genetic algorithm tailored to prompt text.The method adapts genetic optimization to structured discrete data and introduces specialized modules for prompt optimization.

ETHICS STATEMENT

The ethics statement frames AutoDAN as an automated method that could elicit outputs misaligned with human values, while positioning the work as motivation for stronger long-term defenses. The supplied passages also describe genetic-algorithm components and implementation procedures underlying the attack.

  • Ethics Statement: AutoDAN may be used by adversaries to attack LLMs and elicit outputs unaligned with human preferences, intentions, or values.The authors state that they expect no short-term harm but hope the work motivates safer and more aligned LLMs over time.
  • Implementation: AutoDAN applies hierarchical search with sentence-level and word-level operations to optimize structured prompt text.The method uses sentence crossover and word replacement or momentum-based word scoring within its search process.
  • Implementation: The method uses genetic optimization over prompt text, combining selection, crossover, mutation, and diversification to generate offspring prompts.The supplied algorithm descriptions cover both generic genetic operations and LLM-based diversification.
  • Implementation: AutoDAN-GA is introduced as a genetic-algorithm variant for generating jailbreak prompts and shows promising evaluation results.This variant is presented separately from the hierarchical formulation.

D.2 IMPLEMENTATION DETAILS OF AUTODAN

The implementation uses fixed genetic-algorithm hyperparameters, Llama2-generated prompts, refusal-signal evaluation, and reproducible chatbot settings. Figure 3 illustrates attacks on online chatbots under deterministic decoding.

  • Hyper-parameters: AutoDAN uses a 0.5 crossover rate, 0.01 mutation rate, 0.1 elite rate, five crossover breakpoints, and 100 total iterations.Sentence-level optimization occurs five times for every paragraph-level optimization.
  • Examples: The jailbreak prompts shown in the chatbot examples are generated based on Llama2.The examples compare the proposed method with baselines in an online-chatbot setting.
  • Evaluation: The experiments track refusal signals using a list that retains most GCG strings and adds signals observed during evaluation.The supplied caption identifies these strings as the refusal signals considered in the experiments.
  • Examples: Figure 3 presents examples of AutoDAN and baseline attacks on online chatbots with temperature and top P both set to 0.The figure references GPT-3.5-turbo-0301 and Vicuna-33b-lmsys chatbot endpoints.

F ABLATIONS STUDIES ON THE RECHECK METRIC

The Recheck metric is assessed against different LLM bases and human judgments. GPT-4 shows high decision overlap with humans, while keyword detection only partially aligns because disclaimers and nonsensical answers can mislead it.

  • Evaluation: The study uses five participants to judge whether victim-LLM responses answer the corresponding malicious requests, with majority voting determining outcomes.The Recheck metric is also tested using five different LLM base models.
  • Results: GPT-4 achieves a 0.9 decision overlap with human judgment, supporting its use as a response verifier.The comparison spans different base models and keyword detection.
  • Results: Keyword detection has a 0.76 overlap with human evaluations, indicating incomplete alignment.Keyword-based ASR and Recheck ASR are compared across base models and a human study.
  • Interpretation: Keyword detection can misclassify direct answers with disclaimers as refusals and nonsensical non-refusals as successful attacks.These cases explain why keyword detection does not fully match human evaluation.

G PERFORMANCE ON THE OPENAI’S GPT-4

AutoDAN surpasses GCG on GPT-3.5-turbo transferability but, like GCG, transfers poorly to the latest GPT-4, whose black-box robustness remains high.

  • AutoDAN achieves a higher attack success rate than GCG on both GPT-3.5-turbo-0301 and GPT-4-0613.
  • GPT-4-0613 shows high robustness against black-box jailbreak examples generated by both GCG and AutoDAN.
  • AutoDAN transfers well to March GPT-3.5-turbo and surpasses GCG by a large margin.
  • AutoDAN has low transferability on the latest GPT-4, like the baseline GCG.
  • Attacking cutting-edge black-box APIs such as the latest GPT-4 is identified as future work because safeguards and alignment methods continue to evolve.

H EFFICIENCY OF LLM-BASED MUTATION

LLM-based mutation improves AutoDAN’s attack effectiveness while slightly reducing total computational cost, although the mutation policy remains open to improvement.

  • The mutation probability is 0.01, so each sample has a 1% probability of receiving an LLM-based mutation.
  • LLM-based mutation yields about 35% improvement in attack effectiveness.
  • 722.6s versus 766.6s makes the algorithm 5% faster with LLM-based mutation.
  • LLM-based mutation diversifies the search space and can find a successful prompt earlier, triggering the termination criteria sooner.
  • The authors note that a better mutation policy may exist and could be explored in future work.
  • LLM-based prompt diversification surpasses naive WordNet synonym replacement and often produces more reasonable, fluent prompts.

I ADDITIONAL DEFENSES

Against paraphrasing and adversarial-training defenses, AutoDAN retains effectiveness where GCG becomes invalid under paraphrasing, consistent with its semantically meaningful prompts.

  • The evaluation tests GCG and AutoDAN-HGA on 100 AdvBench Behavior data pairs against paraphrasing and adversarial training defenses.
  • Under paraphrasing defense, GCG becomes invalid while AutoDAN retains effectiveness on many samples.
  • Paraphrasing usually discards GCG suffixes because they are garbled characters.
  • AutoDAN prompts maintain their meaning after paraphrasing because they are inherently meaningful texts.
Loading 2310.04451v2…