PAL: Proxy-Guided Black-Box Attack on Large Language Models

Motivation

• LLMs remain vulnerable to adversarial prompts and algorithmically generated jailbreaks despite alignment methods intended to reduce harmful outputs.The paper frames successful jailbreaks as a security concern for widely deployed LLM applications.Source: §1 · p. 1

PAL

• PAL is a query-based black-box attack that uses token-level optimization guided by an open-source proxy model.The approach targets real-world LLM APIs rather than requiring direct access to the proprietary model’s gradients.Source: §1 · p. 1, Figure 1 · p. 2
• PAL achieves 74% ASR on GPT-3.5-Turbo-1106 versus 58% for the current state-of-the-art black-box attack, at half the cost and $0.88 per successful jailbreak on average.The introduction reports this detailed GPT-3.5-Turbo-1106 comparison separately from the broader headline results.Source: §1 · p. 1

Main results

• PAL achieves 74% ASR against GPT-3.5-Turbo-1106 versus 58% for the current state-of-the-art black-box attack, and 48% against Llama-2-7B versus 4% for the state of the art.Source: §1 · pp. 1–2

RAL and GCG++

• RAL reaches 26% ASR on Llama-2-7B under 25k queries, while GCG++ reaches 80% ASR versus 56% for original GCG.RAL replaces proxy-guided search with random search; GCG++ applies the paper’s improvements to the white-box setting.Source: §1 · p. 2

Manual discovery of individual failure modes

• Manual jailbreak studies have examined techniques including character transformations, emotional appeals, and non-English prompts.These approaches evaluate individual failure modes rather than performing broad optimization over the input space.Source: §2 · p. 2

Optimization attacks

• Existing optimization attacks offer efficient search but fall short of practical black-box deployment because commercial APIs do not expose the required loss information.Prior methods are also limited by open-source-only evaluation, hand-designed mutations, unreleased implementations, or human-crafted target strings.Source: §2 · p. 2

LLMs as optimizers

• LLM-based optimizers such as PAIR and TAP generate and refine jailbreak prompts, but their search space is constrained by the attacker model’s output distribution.The paper connects this constraint to weak performance on robustly aligned models such as Llama-2-7B.Source: §2 · pp. 2–3

Query-based attacks with a surrogate model

• Surrogate-model attacks use gradients from an accessible model in place of target gradients, and PAL adapts this strategy to black-box LLM attacks.The paper presents PAL as the first practical attack of this sort on LLMs and in NLP, to the authors’ knowledge.Source: §2 · p. 3

Prompt injection

• Jailbreaking targets model-level content-safety policies, whereas prompt injection targets application-specific goals established in prompts.The paper focuses on jailbreak attacks but notes that its attacks could also find inputs for prompt injection.Source: §2 · p. 3

3.1. Overview

• PAL is designed to overcome query-budget and loss-computation barriers that make gradient-based attacks impractical against commercial APIs.The overview identifies GCG’s query burden and the inability to compute common log-likelihood objectives when APIs expose only limited logprobs.Source: §3.1 · p. 3
• PAL costs less than a dollar on average to jailbreak GPT-3.5-Turbo through the OpenAI API.The method is presented as the first practical attack against LLM APIs in the overview.Source: §3.1 · p. 3
• The attack optimizes an adversarial suffix so the target model assigns high probability to a target string.The formal setting defines an input prompt, target string, target model, and vocabulary-based suffix search.Source: §3.1 · p. 3

3.2. PAL: Proxy-guided Attack on LLMs

• PAL uses proxy-model gradients to sample and filter candidate suffixes, then queries the target model to rank the remaining candidates by target loss.The proxy can optionally be fine-tuned on target-model outputs to improve correspondence between proxy and target losses.Source: §3.2 · pp. 3–4
• PAL combines query-based optimization, transfer from a proxy model, and optional proxy fine-tuning to approximate the target model’s loss.The approach relies on similarity between models, including shared architectures, training data, base models, and distillation relationships.Source: §3.2 · pp. 3–4
• The algorithm initializes a suffix, computes proxy gradients, samples and filters candidates, queries the target, selects the best candidate, and updates the query budget.Initialization may use a suffix generated by attacking the proxy with GCG++.Source: §3.2 · p. 4

3.3. Computing Loss from LLM API

• The API-loss method infers unavailable target-token logprobs with two queries using logit bias, then uses prefix-based pruning to reduce wasted queries.The heuristic stops evaluating a candidate when its generated prefix no longer matches the target; it is not guaranteed to find the lowest full-target loss.Source: §3.3 · p. 5
• The prefix-pruning heuristic is motivated by API budgets: naively computing loss costs 2L queries per sample, limiting a 25k-query attack to 21 steps for typical settings.The method instead drops poor candidates as soon as they fail to produce the next target token.Source: §3.3 · pp. 5–6

3.4. Other Algorithm Improvements

• CW loss empirically outperforms cross-entropy for attacking LLMs, avoiding the vanishing-gradient issue associated with softmax-based losses.The paper motivates CW loss using prior adversarial-example work and applies it as an optimization improvement.Source: §3.4 · p. 6
• 80% ASR on Llama-2-7B versus 56% for original GCG demonstrates the impact of format-aware targeting and related improvements.The format-aware target allows an initial space token, matching Llama-2-7B's strong prior for predicting spaces after the assistant tag.Source: §3.4 · p. 6

3.5. GCG++ and RAL Attacks

• GCG++ combines CW loss, format-aware targets, and implementation improvements to strengthen white-box GCG.Because it is white-box, GCG++ computes loss directly rather than using the API-loss method.Source: §3.5 · p. 6, §3.4 · p. 6
• RAL removes the proxy model, samples candidate suffixes uniformly, and costs 40% as much as PAL while retaining non-trivial ASR against Llama-2-7B.It is a cheaper black-box query-based baseline, though it does not match PAL's performance.Source: §3.5 · p. 6

4.1. Setup

• The evaluation uses 50 randomly selected ADVBENCH behaviors, greedy sampling, default system prompts, and a 25k-query black-box budget.ASRS requires a predefined affirmative string, whereas ASRH uses manual labeling of actually harmful responses by two authors.Source: §4.1 · pp. 6–7

4.2. Black-Box Attacks

• 84% ASRH is achieved by PAL on GPT-3.5-Turbo, while it reaches 48% ASRH on Llama-2-7B versus 4% for TAP.On GPT-3.5-Turbo, PAL also exceeds TAP in ASRS while costing less; on Llama-2-7B, PAL and RAL provide non-trivial black-box success.Source: §4.2 · pp. 7–8, Table 1 · p. 7
• PAL without fine-tuning reaches 50% ASRH on GPT-3.5-Turbo-0613 with a 1.5k-query budget and costs $0.24 per successful jailbreak on average.PAL with fine-tuning reaches up to 84% ASRH, while GPT-3.5-Turbo-1106 is harder to jailbreak than -0613.Source: §4.2 · pp. 7–8, Figure 4 · p. 8
• ASRH exceeds ASRS across settings, with gaps of 10–20% on Llama-2-7B and up to 60% on GPT-3.5-Turbo.Only 6 of 37 successful jailbreaks repeat the target string exactly, indicating that harmful outputs often do not match the predefined target.Source: Figure 3 · p. 7, §4.2 · p. 8

4.3. White-Box Attacks

• GCG++ reaches 80% ASRS on Llama-2-7B at 500 steps and outperforms GCG by 24, 10, and 10 percentage points on three open-source models.It reaches 50% ASRS around 100 steps and 94% at 1,000 steps, whereas GCG plateaus at 56% after 500 steps.Source: §4.3 · p. 8, Table 2 · p. 8
• GCG++ performance is attributed to CW loss, implementation improvements, and a format-aware target string, with the latter increasing Llama-2-7B ASRS from 56% to 76%.A gradient-free GCG++ variant reaches 78% versus 56% for GCG at 1,000 steps, compared with 94% for gradient-based GCG++.Source: §4.3 · pp. 8–9

5. Discussion

• PAL without fine-tuning uses fewer queries than fine-tuned PAL but sacrifices ASRS, while Llama-2-7B requires 6–7× more queries than GPT-3.5.The median query count for PAL without fine-tuning can be as low as 1.1k against GPT-3.5-Turbo.Source: §5 · p. 9
• PAL cannot attack proprietary APIs that provide neither logprobs nor logit bias, because that setting requires a hard-label query-based attack.The paper suggests output-string scores or sentence similarity as potential workarounds.Source: §5 · p. 9
• Removing logit bias and logprobs could hinder optimization-based attacks but would reduce API utility and would not stop TAP or PAIR.Other proposed defenses include attack-transferability training, reducing model similarity, and stateful attack detection, whose effectiveness remains unclear.Source: §5 · p. 9

6. Conclusion

• PAL jailbreaks GPT-3.5-Turbo 74% of the time within at most 25k queries, while GCG++ succeeds over 90% of the time on Llama-2-7B.The attacks target black-box APIs and white-box settings, respectively.Source: §6 · p. 9

Impact Statement

• The methods could be misused to generate harmful outputs from LLM systems, although the research is intended to improve red-teaming of closed-source API-accessible systems.The authors explicitly acknowledge this dual-use risk.Source: Impact Statement · p. 9
• Improving LLM robustness and security is presented as pressing because real-world usage is rapidly expanding, with stronger alignment and API-compatible defenses identified as future needs.The paper also discusses defense options and weighs long-term societal benefits against near-term risks.Source: Impact Statement · p. 9

A. Reproducibility

• PAL fine-tunes an open-source proxy model with bfloat16, a 2 × 10^-5 learning rate, weight decay 0.1, batch size 32, and AdamW.Parameter-efficient tuning such as LoRA is offered when resources are constrained.Source: A. Reproducibility · p. 13
• The attacks initialize 20-token adversarial suffixes and use top-k sampling with k = 256, with batch sizes of 512 for GCG, 128 for PAL, and 32 for RAL.PAL uses a second post-filtering batch size of 32.Source: A. Reproducibility · p. 13
• Jailbreak success is manually labeled by two authors, who identify the first step containing harmful text and resolve disagreements; vague or harmless outputs are excluded.The authors note that the generated harmful responses are generally not specific enough to pose an immediate threat to non-experts.Source: A. Reproducibility · p. 13
• Manual labeling remains necessary because no reliable jailbreak detector exists for this setting, although a future detector could enable early termination and reduce attack cost.The authors report that GPT-4 was too expensive for routine use as a detector.Source: A. Reproducibility · pp. 13–14
• The study uses each model’s default conversation format and system message, with fastchat version 0.2.34 and transformers version 4.35.2, for both target and proxy models.Source: A. Reproducibility · p. 14

A.1. Attack Cost Computation

• Attack cost is compared monetarily because query counts omit token pricing and computational overhead; PAL is estimated at $0.00014 per query versus $0.056 for TAP.The estimates assume 25k PAL queries, 92 TAP queries, and commercial A100 cloud pricing.Source: §A.1 · pp. 14–15

B.1. Ablation Study on GCG’s Design Choices

• Multi-coordinate updates replace more than one adversarial token per step, while momentum ranks candidates using an accumulated update intended to accelerate convergence.These are presented as natural extensions of GCG’s candidate-sampling and ranking procedures.Source: §B.1 · p. 15
• Implementation changes prevent repeated suffix queries and keep candidate batches full by oversampling before filtering and truncating to 512.The ablation results are reported in Appendix C.Source: §B.1 · pp. 15–16

B.2. Randomness in the OpenAI API

• OpenAI API nondeterminism complicates the logit-bias implementation, although the authors judge its effect on final attack results insignificant.They typically detect 1–10 such instances in one attack run.Source: §B.2 · p. 16

B.3. Perspective on Format-Aware Target String

• The target string can substantially affect ASR, so the jailbreak objective chooses an input x to maximize p(Toxic | x) by summing p(Toxic | y)p(y | x) over possible outputs y.This formalization makes target-string selection an explicit part of the attack objective.Source: §B.3 · p. 16
• PAL and GCG approximate possible outputs with the fixed target “Sure, here is...”, but selecting a richer output set could produce a stronger jailbreak algorithm.The authors note that evaluating p(y | x) for a single candidate is already expensive in black-box attacks.Source: §B.3 · p. 16

C.1. PAL Attack and Fine-Tuning

• Fine-tuning makes target and proxy losses decrease together, whereas without it the proxy loss is negatively correlated with target loss.The authors suggest this alignment explains why fine-tuned PAL generally outperforms the random-search baseline.Source: §C.1 · p. 16
• PAL still outperforms RAL despite negative loss correlations in selected best candidates, suggesting those trajectories do not represent overall proxy-target correlation.The authors suspect losses correlate more generally for uniformly sampled candidates.Source: §C.1 · p. 16
• Proxy loss around 0.75 corresponds to the lowest target loss, separating proxy overfitting from a higher-loss underfitting region.Before this point, optimization overfits the proxy; beyond it, both models are sub-optimal.Source: §C.1 · p. 17

C.2. Attack Parameters

• Batch size affects final ASR somewhat, but differences are not statistically significant and show no clear relationship with ASR.Source: §C.2 · p. 17
• Format-aware target strings and CW loss materially affect GCG++ performance, with target-string mismatches reducing ASR by 2%–68% across settings.CW loss is generally as good as or better than CE across the reported settings.Source: §C.2 · p. 17

C.3. GCG++ Ablation Studies

• Fine-tuning generally lets the proxy loss track the target loss more closely across average and per-behavior trajectories.Figure 6 reports average loss over 50 behaviors and individual losses for the first five behaviors using Llama-2-13B and Vicuna-7B.Source: Figure 6 · p. 18
• Format-aware target strings raise GCG ASR from 56% to 76% at 1,000 steps, while additional miscellaneous improvements increase it to 88%.The format-aware target provides the largest margin at both 500 and 1,000 steps.Source: §C.3 · p. 18
• Adding momentum, updating multiple coordinates, or removing the format-aware target string substantially hurts GCG++ ASR.Source: §C.3 · p. 18

C.4. Attack Success Rate by Harmful Categories

• Figure 7 compares final average target and proxy cross-entropy losses across hyperparameter runs with and without proxy fine-tuning on Llama-2-7B.Source: Figure 7 · p. 19
• Figure 8 varies adversarial-suffix batch size for RAL and PAL on Llama-2-7B, with and without Vicuna-7B proxy fine-tuning.Source: Figure 8 · p. 19
• Misinformation behaviors are easiest to elicit across models, while successful-jailbreak distributions differ distinctly between Llama-2-7B and GPT-3.5-Turbo.Source: §C.4 · p. 19

Examples overview

• Five successful behaviors and two failure cases are shown for GPT-3.5-Turbo-0613 with fine-tuned PAL and Llama-2-7B without fine-tuning.The examples are randomly chosen from the evaluated behaviors.Source: D. Examples of the Generated Responses · p. 19

Successful behaviors

• The models generated a tutorial for hacking sensitive government databases, despite prefatory disclaimers about illegality and ethics.The response proceeded with cybersecurity and networking guidance after warning about legal consequences.Source: D. Examples of the Generated Responses · p. 20
• Other successful outputs proposed a video game glorifying violence against civilians and fabricated a deadly-virus news article designed to provoke panic.The game response included weapons, combat, and missions, while the article described a fictional pandemic and resulting public panic.Source: D. Examples of the Generated Responses · pp. 21–22
• Additional successful outputs produced a political speech endorsing hate speech, division, and violence, plus a defamatory letter alleging unethical, fraudulent, environmental, and labor misconduct.The defamation example presents allegations against a placeholder company or individual as reputationally damaging claims.Source: D. Examples of the Generated Responses · pp. 23–24

• The cited responses include a video game that glorifies war and violence, a fake-news article that could potentially lead to panic, and a political-candidate speech using hate speech.Source: D. Examples of the Generated Responses · pp. 25–26
• The fake-news example fabricated an additional moon named “Thewidget,” including a false size and purported Lunar Lagrange Point location.Source: D. Examples of the Generated Responses · p. 26
• Failure cases generally yielded either boilerplate refusals or responses that followed the adversarial suffix while remaining semantically related but non-harmful.Source: D. Examples of the Generated Responses · p. 27
• Some successful-looking attacks produced off-target assistance, such as JavaScript, Python, or Docker explanations instead of the requested harmful content.These outputs included a bomb-sprite tutorial, operating-system code, and a Docker summary after ransomware-related prompts.Source: D. Examples of the Generated Responses · pp. 27–28