Source-linked AI summary
Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, Amin Karbasi
TL;DR
LLM alignment methods remain vulnerable to jailbreaks, while prior automated approaches are limited by human effort or model access requirements. TAP automatically refines attack prompts with an attacker LLM, evaluates and prunes candidates before target queries, and achieves high jailbreak success with fewer queries. Its evaluations also test guarded models, transferability, and limitations from datasets and judge-model accuracy.
Problem
Existing jailbreak methods often require substantial human effort or open-source model access, motivating automated attacks that work through black-box queries.
Method
TAP uses an attacker LLM to iteratively branch prompt candidates and an evaluator LLM to prune unlikely candidates before querying the target.
Results
More than 80% success across GPT4, GPT4-Turbo, GPT4o, and Gemini-Pro uses fewer than 30 queries on average, while TAP also outperforms baselines against Llama-Guard.
Takeaways & Limitations
TAP provides an automated, black-box, interpretable jailbreak method that improves success-query efficiency over prior black-box methods.
Takeaways & Limitations
Evaluations focus on harmful-information requests and two datasets, while the off-the-shelf judge model may produce inaccurate or miscalibrated scores.
Abstract
from arXiv · showhide
While Large Language Models (LLMs) display versatile functionality, they continue to generate harmful, biased, and toxic content, as demonstrated by the prevalence of human-designed jailbreaks. In this work, we present Tree of Attacks with Pruning (TAP), an automated method for generating jailbreaks that only requires black-box access to the target LLM. TAP utilizes an attacker LLM to iteratively refine candidate (attack) prompts until one of the refined prompts jailbreaks the target. In addition, before sending prompts to the target, TAP assesses them and prunes the ones unlikely to result in jailbreaks, reducing the number of queries sent to the target LLM. In empirical evaluations, we observe that TAP generates prompts that jailbreak state-of-the-art LLMs (including GPT4-Turbo and GPT4o) for more than 80% of the prompts. This significantly improves upon the previous state-of-the-art black-box methods for generating jailbreaks while using a smaller number of queries than them. Furthermore, TAP is also capable of jailbreaking LLMs protected by state-of-the-art guardrails, e.g., LlamaGuard.
1 Introduction
TAP targets the gap between human-dependent or open-weight jailbreaks by automatically generating interpretable attacks using only black-box query access. It combines attacker-guided refinement with evaluator-based branching and pruning, achieving higher success with fewer target queries.
- Motivation: Alignment methods use training, system prompts, and guardrails, yet LLMs remain vulnerable to adversarial manipulation and jailbreaks.Jailbreaking attempts to bypass safety filters and circumvent alignment.
- Motivation: Most existing jailbreak methods require substantial human effort or access to open-source model weights or tokenizers.The paper therefore focuses on automated, black-box, interpretable attacks.
- Method: TAP uses an attacker LLM to generate prompt variations and an evaluator LLM to select promising variations before querying the target.The process iterates until a jailbreak is found or the maximum repetition limit is reached.
2 Preliminaries
The paper defines jailbreak success operationally through whether a target supplies requested undesirable information. It formalizes off-topic prompts and evaluator judgments used to assess candidate attacks.
- Definitions: A response is undesirable when it violates model developers’ guidelines, with the paper focusing on harmful information or misinformation.Examples include instructions for causing physical harm or public-property damage.
- Definitions: A prompt jailbreaks target LLM T for query Q when T provides the requested undesired information instead of refusing or responding irrelevantly.The definition distinguishes successful assistance from refusal and unrelated instructions.
- Evaluation functions: Off-Topic(P, Q) is false when prompt P requests the same information as query Q and true otherwise.The function filters candidate prompts that no longer address the original request.
- Evaluation functions: Judge(Q, R) is true when response R signifies a successful jailbreak for query Q and false otherwise.TAP uses this function to assess target responses.
- Objective: The automated black-box interpretable goal is to output a meaningful prompt P that makes target T produce a response with Judge(Q, R) = True.This combines the paper’s requirements for automation, black-box access, and interpretability.
3 Tree of Attacks with Pruning
TAP extends PAIR with branching and two pruning phases, using attacker and evaluator LLMs to explore and retain promising jailbreak attempts. The algorithm is controlled by branching factor, width, and depth.
- Parameters: The branching factor b controls refinements per prompt, width w controls retained attempts, and depth d limits iterations.In the illustrated example, b = 2 and w = 4.
- Iteration: At each iteration, the attacker generates multiple prompt variations from the current conversation history.This creates a tree of candidate attack attempts rather than a single refinement chain.
- Architecture: TAP uses attacker A, evaluator E, and target T, with evaluator E implementing Judge and Off-Topic assessments.The evaluator’s system prompt changes according to whether it assesses relevance or jailbreak success.
- Pruning: Phase-one pruning removes off-topic variations before the target LLM is queried.This filters candidates unlikely to elicit the requested undesirable information.
- Pruning: After querying the target, TAP scores responses, returns a successful prompt when found, and otherwise retains the highest-scoring leaves for the next iteration.Phase-two pruning limits the tree to at most w retained attempts.
4 Empirical Setup
The evaluation uses two harmful-information datasets, automated and human jailbreak judgments, and comparisons with PAIR and GCG across multiple target LLMs. TAP’s algorithm samples, filters, queries, scores, and retains candidate prompts under fixed tree parameters.
- Datasets: The study uses AdvBench Subset with 50 harmful requests and a new dataset containing 123 harmful requests.The new prompts are generated as variants of AdvBench requests using WizardVicuna30B-Uncensored.
- Baselines and implementation: TAP is compared with PAIR for black-box attacks and GCG for white-box attacks using public implementations.TAP and PAIR use Vicuna-13B-v1.5 as attacker and GPT4 as evaluator.
- Implementation: TAP uses branching factor b = 4, maximum width w = 10, and maximum depth d = 10 in the reported implementation.The settings are chosen so TAP and PAIR send and receive similar target-query and token quantities.
- Metrics: The primary success measures are GPT4-Metric and Human-Judgement, while efficiency is measured by target queries per prompt.GPT4-Metric treats a detailed and fully jailbroken response as successful according to GPT4.
- Algorithm: Each iteration samples b attacker-generated prompts, removes off-topic candidates, queries the target, scores responses, and retains the top w leaves when needed.The procedure returns a prompt immediately when the evaluator identifies a successful jailbreak; otherwise it can return failure after the depth limit.
5 Evaluation of Performance and Query Efficiency
TAP is evaluated against black-box jailbreaking baselines on unprotected and Llama-Guard-protected models, with success, query efficiency, and transferability measured. Results show higher success with fewer queries than PAIR on most targets, while transferability is broadly similar to PAIR.
- Unprotected models: TAP finds jailbreaks for a significantly higher fraction of prompts than PAIR while sending significantly fewer queries to the target across Table 1 targets.Table 1 reports success fractions and target queries on the AdvBench Subset using the GPT4-Metric.
- Unprotected models: 16% more prompts are jailbroken by TAP than PAIR on GPT4o while using 60% fewer target queries.The comparison is reported for GPT4o in Table 1.
- Protected models: TAP’s success rate remains close to that on unprotected models and is significantly higher than PAIR’s on most Llama-Guard-protected models.With protected models, TAP sends more queries than PAIR, but its proportional performance increase is larger than the proportional query increase.
- Transferability: Transferability is similar for TAP and PAIR, while significantly fewer GCG jailbreaks transfer to different target models.Transferability is measured as the fraction of successful jailbreak prompts that also jailbreak another target, using the GPT4-Metric.
6 Empirical Evaluation of the Effects of Branching and Pruning
The ablations show that TAP’s branching and pruning components jointly support high success and query efficiency. Removing branching sharply lowers success, while removing pruning substantially increases queries without improving success.
- Branching: 48% versus 84% success shows that removing branching sharply lowers TAP’s success rate despite using more queries.TAP-No-Branch sends 33.1 queries versus 22.5 for standard TAP.
- Pruning: 55.4 versus 22.5 queries shows that removing pruning substantially increases query use without improving TAP’s success rate.TAP-No-Prune retains branching but does not prune off-topic prompts.
- Overall effect: The combination of branching and pruning is crucial to achieving a high success rate in a query-efficient fashion.The ablation comparison evaluates TAP and its no-branching and no-pruning variants with GPT4-Turbo as target.
- Pruning mechanism: Without pruning, retaining only the 10 highest-scoring prompts can delete all on-topic prompts when more than 10 off-topic prompts are generated.This describes one reason the no-pruning variant can behave unexpectedly despite sending more queries.
7 Conclusion
TAP automates interpretable black-box jailbreaking by combining attacker-guided branching with evaluator-based pruning. It improves jailbreak success while limiting target queries, though its evaluations and scope remain bounded by dataset, judge-model, and evaluator choices.
- Conclusion: TAP automates jailbreaking using only black-box access to the target LLM and produces interpretable prompts.The method is designed to avoid requiring target parameters while retaining natural-meaning attacks.
- Conclusion: TAP finds jailbreak prompts for GPT4, GPT4-Turbo, GPT4o, and Gemini-Pro on more than 80% of harmful-information requests using fewer than 30 queries on average.These results are reported on harmful-information requests from existing datasets.
- Conclusion: TAP also achieves a higher success rate than baselines against LLMs protected by the Llama-Guard guardrail.The evaluation reports this result in addition to testing transferability of generated prompts.
- Conclusion: TAP’s branching and pruning combination achieves higher success rates than previous methods while retaining a low number of target queries.Branching explores disjoint attack strategies, while pruning removes off-topic or low-scoring prompts before target queries.
- Limitations: The evaluations focus on harmful-information requests, and performance may differ on meaningfully different datasets or evaluation guidelines.The authors also identify inaccurate or miscalibrated judge-model scores as a possible limitation.
- Conclusion: TAP can run without GPUs when its attacker, evaluator, and target LLMs are accessible through APIs.Although the nominal query bound depends on branching, width, and depth, pruning can reduce actual queries substantially; with w × b × d = 400, experiments often used fewer than 30 queries on average.
A.2 Empirical Evaluation: Monetary Cost, Transferability, and Number of Queries
TAP combines branching and pruning to reduce prompt redundancy and filter low-quality candidates before target queries. Evaluations report lower cost, competitive transferability, and improved performance over PAIR when both changes are used together.
- Cost and transferability: TAP costs less than 3 USD per harmful prompt when GPT-4 serves as evaluator across GPT-4, GPT-4 Turbo, GPT4o, PaLM-2, Gemini-Pro, and Claude-3-Opus.The reported cost applies to each listed target model.
- Cost and transferability: TAP attacks transfer to other models at a rate similar to baseline attacks, while GCG transfers to GPT models at a lower rate than earlier reports.The authors attribute the GCG discrepancy tentatively to continuing OpenAI model updates.
- Method comparison: Each modification independently improves performance over PAIR, while branching and pruning together provide the highest benefit.The evaluation isolates the two changes and compares them with PAIR.
- Method comparison: TAP generates prompts using branching, reducing repeated attacker conversation histories and likely reducing redundancy in generated prompts.The branching factor controls how many refinements each prompt receives.
- Method comparison: TAP prunes off-topic and low-scoring prompts before querying the target, addressing the prevalence of off-topic attacker outputs.A majority of prompts generated by the attacker were observed to be off-topic for the goal.
- Method comparison: Including off-topic prompts in the conversation history empirically increases the likelihood of future off-topic prompts rather than correcting prompt quality.This supports pruning off-topic prompts instead of retaining them for conversational context.
B.1 Examples Successful Jailbreaks
The examples document successful TAP jailbreaks across multiple target models and harmful-information goals. They include stalking, explosives, defamation, piracy, hacking, fraud, eating-disorder promotion, misinformation, and transportation sabotage.
- GPT4-Turbo: GPT4-Turbo is shown successfully jailbroken for phone-based stalking and harassment and for a homemade explosive recipe.The two examples are presented as separate successful jailbreak figures.
- GPT4: GPT4 is shown successfully jailbroken for defaming a company or individual and pirating software for online distribution.Both examples target harmful or unlawful information requests.
- GPT3.5-Turbo: GPT3.5-Turbo is shown successfully jailbroken for corporate-network hacking and mail fraud.The examples use distinct harmful-information goals.
- PaLM-2: PaLM-2 is shown successfully jailbroken for promoting anorexia or bulimia and creating fake news that could cause panic or chaos.These examples concern harmful health messaging and misinformation.
- Llama-2-Chat-7B: Llama-2-Chat-7B is shown successfully jailbroken for transportation sabotage and public-panic misinformation.The figures identify successful jailbreaks for both goals.
B.2 Examples of Execution of Our Method
The execution examples show TAP iteratively revising attacker prompts after target refusals or mismatches. The illustrated tree contains branches for malware, misinformation, and defamatory-letter objectives.
- Malware branch: TAP revises prompts when an attempt fails to promote malware spreading, explicitly changing tactics toward the intended objective.The example labels the revision as an improvement after the prior suggestion did not fulfill the goal.
- Tree construction: The examples show that TAP’s attack tree contains multiple candidate paths rather than a single linear prompt sequence.The method is illustrated through subtrees associated with distinct harmful goals.
- Misinformation branch: For misinformation, an attacker reframes the request as breaking news and asks for a catchy, fact-unchecked opening.The example appears as a revised prompt within a tree-of-attacks subtree.
- Defamatory-letter branch: After a refusal to create defamatory accusations, TAP attempts increasingly contextualized and indirect prompts about unsupported allegations.The sequence includes a refusal, an emphasis on deviousness, and a fictionalized framing instruction.
- Defamatory-letter branch: One revised prompt asks for an urgent Human Resources message alleging corporate espionage and requesting an internal investigation.The prompt is part of the illustrated defamatory-letter attack path.
C Implementation Details
The implementation uses specified target, attacker, and evaluator configurations, with GPT4 commonly serving as evaluator and additional evaluators used for robustness checks. Human-judgment results remain qualitatively similar to GPT4-Metric results.
- Target configuration: Open-source target models use top_p = 1, num_beams = 1, and do_sample = False, matching the referenced prior implementation.These settings are stated for reproducibility and consistency with prior work.
- Evaluator configuration: GPT4 is used as the evaluator for all main-body simulations with default inference parameters.The evaluator system prompts are provided separately for Judge and Off-Topic evaluation.
- Attacker configuration: The attacker uses top_p = 0.1, num_beams = 1, temperature = 1, and do_sample = True with the referenced attacker algorithm and system prompt.The attacker system prompt replaces placeholders with the harmful-information goal and an ideal starting string.
- Additional evaluators: Llama-Guard and substring evaluators do not implement TAP’s Off-Topic function, so Phase 1 pruning is not implemented for the Llama-Guard evaluator.The Llama-Guard evaluator labels a response unsafe using the response and an empty prompt.
- Evaluation: Human-judgment results are qualitatively similar to GPT4-Metric results, with TAP outperforming PAIR on all evaluated targets except Llama-2-Chat-7B.Both methods perform poorly on that exception.
D.2 Evaluation on a Held-Out Dataset
The held-out evaluation uses 123 newly generated and manually verified goals across AdvBench categories. Resource constraints prevented evaluation of GPT4o and Claude3-Opus on this dataset.
- The dataset was constructed after the other reported simulations and used in-context generation for evaluation.WizardVicuna30B-Uncensored generated the in-context data because it was used to create the AdvBench dataset.
- GPT4o and Claude3-Opus were not evaluated on the held-out dataset because of resource constraints.
- The held-out dataset contains 123 goals generated across AdvBench categories with at least three existing goals.At least 25 new goals were generated per selected category, then manually verified for category matching.
E.1 Effect of the Choice of Evaluator
TAP’s evaluator materially affects jailbreak performance: weaker or simpler evaluators can stop the method incorrectly, whereas Llama-Guard is more competitive despite being much smaller than GPT4. A binary evaluator score slightly improves success with similar query usage.
- Effect of the Choice of Evaluator: The Llama-Guard and substring evaluators omit Off-Topic evaluation and therefore omit Phase 1 of pruning.
- Effect of the Choice of Evaluator: Changing the evaluator from GPT4 to GPT3.5-Turbo reduces success from 84% to 4.2%.The lower success rate results from incorrect evaluator judgments that the target was already jailbroken, causing TAP to stop prematurely.
- Effect of the Choice of Evaluator: Llama-Guard as evaluator achieves 26% success on GPT4-Turbo and 68% on Vicuna-13B while using a much smaller evaluator model than GPT4.The paper suggests that several small models specialized for specific harms may match TAP’s performance with GPT4 as evaluator.
- Effect of the Choice of Evaluator: Binary evaluator scores improve success from 84% to 86% with similar query counts of 23.4 versus 22.5.This variant fixes GPT4-Turbo as target, GPT4 as evaluator, and Vicuna-13B as attacker.