PaperlayerView exact revision on arXiv

Source-linked AI summary

Open Sesame! Universal Black Box Jailbreaking of Large Language Models

Raz Lapid, Ron Langberg, Moshe Sipper

arXiv:2309.01446v4cs.CLcs.CVcs.NE

TL;DR

Aligned LLMs remain vulnerable to jailbreaks, while existing approaches often require manual effort or access to model internals. This paper uses a genetic algorithm to optimize a universal adversarial suffix in a black-box setting. The approach transfers across models and effectively induces harmful outputs, but its perturbations may be detectable and its robustness varies across architectures and prompt types.

  • Problem

    Existing jailbreak methods often require manual effort or model-specific access, motivating automated universal attacks that operate without LLM internals.

  • Method

    A genetic algorithm evolves adversarial token sequences appended to unchanged user queries, using black-box output feedback to manipulate aligned LLMs.

  • Results

    The method transfers prompts between models and demonstrates exceptional efficacy in inducing harmful outputs from the evaluated language models.

  • Takeaways & Limitations

    The paper presents the approach as an automated universal black-box jailbreak and a way to expose vulnerabilities in aligned LLMs.

  • Takeaways & Limitations

    Attack robustness may vary across LLM architectures and prompt types, and its perceptible garbage-token perturbations may be detected.

Abstract

from arXiv · show

Large language models (LLMs), designed to provide helpful and safe responses, often rely on alignment techniques to align with user intent and social guidelines. Unfortunately, this alignment can be exploited by malicious actors seeking to manipulate an LLM's outputs for unintended purposes. In this paper we introduce a novel approach that employs a genetic algorithm (GA) to manipulate LLMs when model architecture and parameters are inaccessible. The GA attack works by optimizing a universal adversarial prompt that -- when combined with a user's query -- disrupts the attacked model's alignment, resulting in unintended and potentially harmful outputs. Our novel approach systematically reveals a model's limitations and vulnerabilities by uncovering instances where its responses deviate from expected behavior. Through extensive experiments we demonstrate the efficacy of our technique, thus contributing to the ongoing discussion on responsible AI development by providing a diagnostic tool for evaluating and enhancing alignment of LLMs with human intent. To our knowledge this is the first automated universal black box jailbreak attack.

1 INTRODUCTION

The paper targets the gap between manually crafted or model-specific jailbreaks and automated universal black-box attacks. It proposes using genetic algorithms to discover prompts that manipulate LLM behavior without access to internal architecture or parameters.

  • Existing LLM jailbreaks often require extensive domain knowledge and manual effort despite alignment efforts intended to support human values and societal norms.
  • The proposed approach uses a genetic algorithm to automatically discover black-box adversarial prompts that manipulate LLM behavior.
  • The paper asks whether LLMs can be automatically jailbroken without relying on their internal mechanisms.
  • The work addresses the need for automated, universal black-box attacks rather than methods limited to particular models or tasks.
  • The claimed contributions include eliminating manual prompt crafting, evaluating two open-source LLM architectures and prompting contexts, and analyzing vulnerabilities exploited by evolved prompts.

2 METHODOLOGY

The attack appends a universally reusable adversarial suffix to an unchanged user query and evolves token sequences with a genetic algorithm. Fitness is approximated from semantic similarity between generated outputs and a target behavior, enabling black-box optimization.

  • The attack preserves the user’s original query and appends an adversarial suffix intended to elicit undesirable model behavior.
  • Prompt sequences are evolved through genetic-algorithm selection, crossover, and mutation operations.
  • The fitness approximation measures semantic alignment between generated output and a predefined target behavior using cosine similarity between text embeddings.
  • The paper presents the method as the first automated universal black-box jailbreak attack.
  • The threat model assumes access only to textual outputs, without the target model’s architecture, parameters, or training data.

3 EXPERIMENTS

The experiments evaluate the proposed universal jailbreaker on a harmful-behavior dataset split into training and test sets. They use two open-source chat models and vary the genetic-algorithm population size under repeated dataset splits.

  • The Harmful Behavior dataset contains diverse harmful inputs designed to challenge LLM capabilities.
  • The dataset is partitioned into 70% training and 30% test data for optimization and independent evaluation, respectively.
  • Results are averaged over two dataset-splitting seeds, with 100 generations and population sizes n∈{10, 20, 30}.
  • The evaluated models are LLaMA2-7b-chat and Vicuna-7b.

4 RESULTS AND DISCUSSION

The study reports that evolved prompts transferred between models and elicited harmful outputs, while noting detectable garbage-token perturbations and variation across architectures and prompt types.

  • Prompts crafted for one model could effectively transfer to another, demonstrating the approach’s versatility.
  • The method demonstrated exceptional efficacy in inducing harmful outputs from the evaluated language models.
  • Before the attack, LLMs refused requests involving insider trading and fake-review automation; after the attack, they answered freely.
  • The attack’s added prompts contain linguistically meaningless garbage tokens that another LLM or perplexity might detect.
  • Robustness may vary across LLM architectures and prompt types, and the attack adds perceptible perturbations.
  • Future work could examine prompt construction, GA parameters, and generalizability beyond LLMs.

5 CONCLUDING REMARKS

The paper frames robust and reliable LLM development as an intricate security challenge because language complexity and adversarial manipulation complicate alignment with human values and societal well-being.

  • Language complexity and adversarial manipulation create challenges for developing robust and reliable LLMs.
  • The paper states that achieving robust LLM alignment remains a major challenge despite adversarial training and reinforcement learning with human feedback.
  • The conclusion calls for creating LLMs that are both powerful and ethically sound while aligning with human values and societal well-being.

A.1 METHODOLOGY

The paper’s technical innovation is a black-box technique that elicits undesirable language-model responses using only model queries and raw outputs, without gradients or internal access.

  • The technique exploits language-model vulnerabilities to elicit undesirable responses under black-box conditions.
  • The approach uses neither gradients nor model internals, relying only on queries and the model’s raw output.

A.2 GENETIC ALGORITHM

The genetic algorithm is presented as an evolutionary search procedure that generates candidate solutions, evaluates them, recombines and mutates them, and returns the best individual found.

  • A genetic algorithm is a search heuristic that mimics natural evolution to solve optimization and search problems.
  • The algorithm begins by generating an initial population of candidate solutions and continues until a termination condition is met.
  • Each iteration computes fitness, selects parents, performs crossover, and mutates the resulting offspring.
  • After iteration, the algorithm returns the best individual found as the solution to the problem.

A.3 POPULATION ENCODING

The GA represents each candidate adversarial prompt as a fixed-length vector of vocabulary-token integers and initializes populations with configurable sizes and prompt lengths.

  • A.3 POPULATION ENCODING: Each individual is a prompt encoded as a vector of integer token identifiers.A population contains n prompts, each with m tokens drawn from vocabulary T.
  • A.3 POPULATION ENCODING: The population size n is tested at 10, 20, and 30 individuals.
  • A.3 POPULATION ENCODING: The prompt length m is tested at 20, 40, and 60 tokens for each model.
  • A.3 POPULATION ENCODING: Initialization creates an empty population and iteratively adds n individuals using the specified prompt length and token vocabulary.

A.4 FITNESS

Fitness evaluates candidate suffixes by comparing model outputs with target behavior in embedding space, using sampled inputs to make black-box optimization tractable.

  • A.4 FITNESS: The target behavior uses the contextualized response prefix “Sure, here is a...” when the model knows the answer and “No, I don’t...” otherwise.
  • A.4 FITNESS: Each adversarial suffix is appended to a user instruction, and the resulting prompt is evaluated on a harmful-behavior dataset.The dataset contains harmful behaviors, and the suffix is optimized to influence the model’s response.
  • A.4 FITNESS: The black-box loss is defined from cosine similarity between generated-output and target-output embeddings, with lower loss preferred.This encourages outputs whose semantic content aligns with the target behavior.
  • A.4 FITNESS: The black-box fitness objective measures semantic alignment between generated outputs and target outputs rather than relying on model confidence scores.The method uses an embedder and cosine similarity because the attacked model is opaque.
  • A.4 FITNESS: Three general-purpose embedders—BGE, MPNet, and MiniLM—map generated and target texts into dense semantic vectors for comparison.
  • A.4 FITNESS: Fitness approximation samples 50 training instances anew at each iteration and averages the black-box loss across their generated outputs.This random subset sampling reduces computational cost while exposing optimization to diverse inputs.

A.5 SELECTION

The GA evolves prompt populations through tournament selection, crossover, mutation, and elitism, preserving strong candidates while exploring new token sequences.

  • A.5 SELECTION: Tournament selection samples two individuals and chooses the fitter one as a parent for reproduction.
  • A.5 SELECTION: Crossover recombines genetic material from parent prompts to create offspring and explore new regions of the search space.
  • A.5 SELECTION: Mutation introduces small random changes to prompt tokens, balancing crossover’s exploration with local exploitation.
  • A.5 SELECTION: Elitism preserves the best-performing individuals so population quality does not deteriorate across generations.The paper sets the elitism value as a function of population size.
  • A.5 SELECTION: Each GA generation evaluates fitness, saves elite individuals, selects parents, creates offspring through crossover and mutation, and replaces the population with elites plus offspring.
  • A.5 SELECTION: MPNet achieved the highest average ASR on both Vicuna-7b and LLaMA-7b-chat among the tested text embedders.
Loading 2309.01446v4…