Query-Based Adversarial Prompt Generation

1 Introduction

• Transfer attacks require model weights for crafting or a high-quality surrogate, limiting their practicality for production models.Transferred attacks fooled GPT-4 and Bard at 46% and 66% success rates from Vicuna-crafted examples.Source: §1 · p. 1
• The attack directly constructs adversarial examples on a remote language model without relying on transferability.An optimization to GCG removes the surrogate-model dependency with only a moderate increase in model queries.Source: §1 · pp. 1–2
• Query-based attacks can elicit specific harmful outputs, unlike transfer attacks.The paper states that prior black-box methods do not make target models output exact harmful strings, whereas this attack does.Source: §1 · p. 1
• The method replaces GCG’s gradient-based candidate filter with a surrogate-based filter and queries the remote target to select candidates.This combines local guidance with direct evaluation of the model being attacked.Source: §1 · p. 2
• Nearly 100% attack success was achieved against OpenAI’s content moderation endpoint without a local moderation model.The endpoint is described as OpenAI’s most robust moderation model to date.Source: §1 · p. 2

2 Background

• White-box attacks require complete access to model weights, while transfer attacks replay examples crafted on another model.Early black-box attacks instead relied on adversarial-example transferability [26].Source: §2 · p. 2
• Transfer attacks rarely succeed at targeted attacks that induce a specific incorrect behavior.This limitation appears even in comparatively simple ImageNet classification with 1,000 classes [20].Source: §2 · p. 2
• Query-based attacks construct adversarial examples by querying the target and can achieve nearly 100% targeted success on black-box ImageNet classifiers.They can also combine local-model signals with target queries to reduce query counts [10].Source: §2 · p. 2
• NLP transfer attacks require a high-quality surrogate; Vicuna achieved just 2% transfer success against Claude.This makes transferability unreliable when no sufficiently similar local model exists.Source: §2 · p. 3
• Transfer attacks can induce compliance but cannot force a model to produce a specific harmful output.The paper contrasts un-targeted compliance with targeted harmful-string generation.Source: §2 · p. 3
• GCG improves adversarial-prompt optimization by considering all token coordinates and jointly optimizing across prompts and models.These choices improve attack performance and transferability compared with AutoPrompt.Source: §2 · p. 3

3 GCQ: Greedy Coordinate Query

• GCQ is a direct modification of GCG that greedily updates an adversarial string and returns the best example after fixed iterations.Its central change is a best-first-search-style buffer over adversarial suffixes.Source: §3 · p. 3, §3.1 · p. 3

3.1 Method

• GCQ keeps the B best unexplored suffixes, expands the best one with bp neighbors, proxy-ranks them, and evaluates the bq best using the true loss.Candidates that are worse than the buffer’s worst element are discarded.Source: §3.1 · p. 3
• The attack scores prompts with negative cumulative logprobability of the target string and counts success when greedy sampling generates that string.A local proxy model supplies the proxy loss, while the remote model supplies the true loss.Source: §3.1 · p. 4

3.2.1 Scoring prompts with logit-bias and top-5 logprobs

• After prompt logprobabilities became unavailable through the OpenAI API, the authors reconstructed them from available API features at higher cost.The later API changes made the binary-search reconstruction approach significantly more expensive as of May 2024.Source: §3.2.1 · p. 4

3.2.2 Short-circuiting the loss

• Short-circuiting cumulative-logprobability computation reduces total attack cost by approximately 30%.The computation exits once a candidate is known to exceed the loss of the buffer’s worst member.Source: §3.2.2 · p. 4

3.2.3 Choosing a better initial prompt

• Repeating the target string initializes the search effectively and immediately produces the target for 28% of harmful strings when m = 20.This initialization is evaluated in an ablation study.Source: §3.2.3 · p. 4

3.3 Proxy-free query-based attacks

• The proxy-free variant ignores gradients, samples random replacements, and queries the target loss, while a position-focused optimization reduces queries by 2× without affecting attack success.It tests one replacement per suffix position, then concentrates B′ additional replacements on the best position with B′ ≪ B.Source: §3.3 · pp. 4–5

4.1 Harmful strings for open models

• For open models, larger Vicuna 1.3 models were harder to attack, Llama 2 was more resistant than Vicuna, and transfer worked best between similarly sized models.Transfer from 7B to larger Vicuna models was poor, whereas 13B transferred relatively well to 33B.Source: §4.1 · p. 5, Figure 1 · p. 5

4.2 Comparison to other attacks

• AutoDAN achieved only 1/574 successes in GA mode and 0/574 in HGA mode, despite using approximately 70-token suffixes versus GCQ’s 20-token suffixes.Pure-transfer attacks also produced low success rates in the harmful-string setting.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Harmful strings for GPT-3.5 Turbo

• 79.6% of GPT-3.5 target strings were elicited with a $0.10 per-target budget, rising to 86.0% at $0.20.Attack success increased rapidly at first as cost rose.Source: §4.3 · p. 6, Figure 2 · p. 6
• GPT-3.5 success was 97.9% for target strings of 20 tokens or fewer, but longer targets became difficult until the prompt was extended to 40 tokens, yielding 100% success at a mean cost of $0.41.The reported explanations were weaker initialization and insufficient degrees of freedom for longer targets.Source: §4.3 · p. 7, Figure 3 · p. 7
• Random initialization succeeded only twice among the first 20 targets, indicating that initialization was crucial in the low-cost regime.This was reported as an ablation of the GPT-3.5 experiment.Source: §4.3 · p. 7

4.4 Proxy-free harmful strings for open models

• The optimized GCG variant required approximately 2× fewer loss queries to reach the same attack success rate, while its fully black-box version outperformed original GCG by about 30%.Source: §4.4 · pp. 7–8, Figure 4 · p. 7

4.5 Proxy-free attack on OpenAI text-moderation-007

• 99.2% universal moderation-evasion success was achieved for 20-token suffixes after 100 iterations and 2,000 requests, while 5-token suffixes reached 94.8% after 2,000 iterations and 40,000 requests.These universal attacks were evaluated on held-out validation strings.Source: §4.5 · p. 8

4.6 Proxy-free attack on Llama Guard 7B

• Llama Guard 7B reached 59% and 87% success for 5- and 20-token suffixes after 320 queries, compared with 84% and 91% for OpenAI moderation.The gap narrowed with further iterations.Source: §4.6 · p. 9, Figure 6 · p. 9

5 Conclusion

• Query-based attacks can practically coerce models into emitting specific harmful strings, unlike transfer-only attacks.Because the attack queries the target during generation, defenses based exclusively on breaking transferability are not expected to suffice.Source: §5 · p. 9
• The paper frames robustness against crafted inputs as necessary for deploying language models in adversarial situations.The authors hope these results encourage caution and further research into language-model robustness and safety.Source: §5 · p. 9
• Initialization strongly affects attack success, with selected prompts improving outcomes and random prompts making attacks substantially less effective.This sensitivity contrasts with computer vision, where initial perturbations and random seeds have much smaller effects.Source: §5 · p. 9
• The authors identify significant potential for improving NLP adversarial example generation in both white-box and black-box settings.They characterize current NLP attacks as relatively weak compared with their vision counterparts.Source: §5 · p. 9

A Compute resources

• The main Section 4.1 experiments used 2–8 A100 GPUs on one node for several days, while other experiments used one A40 for several days.GPU utilization was not perfect during the multi-GPU runs.Source: A Compute resources · p. 12

B OpenAI logprob inference via logit bias and top-5 logprobs

• An obsolete OpenAI API technique is retained for completeness because logit bias no longer affects the returned top-logprob token list.As of March 2024, this limitation prevents that technique from functioning for OpenAI models.Source: B OpenAI logprob inference via logit… · p. 12
• The API can boost an arbitrary target token into the top-5 logprobs, enabling its biased probability to be read for scoring.The returned value is distorted by the applied bias, so it requires correction to recover the true logprob.Source: B OpenAI logprob inference via logit… · p. 12
• The correction recovers the true token probability from the biased probability and applied bias using the displayed equation.The method must account for the fact that the API-reported logprob is not the unmodified token probability.Source: B OpenAI logprob inference via logit… · p. 12
• Biases that are too large lose numerical accuracy, whereas biases that are too small may fail to move the target token into the top-5.An estimate from the parent string usually sets the bias successfully; this initial choice succeeds over 99% of the time, with binary search as fallback.Source: B OpenAI logprob inference via logit… · p. 12
• Because one logit bias applies to an entire generation, the method prepends target prefixes to control the next target token individually.For prompt length p and target length t, scoring costs p t prompt tokens and t(t + 1)/2 completion tokens.Source: B OpenAI logprob inference via logit… · p. 12

C Tokenization concerns

• Prompts are re-tokenized before API evaluation because token-ID sequences may not reproduce the tokenizer’s string segmentation.This avoids optimizing tokenizations that cannot be passed as equivalent string prompts, without observed impact on Algorithm 1 success.Source: C Tokenization concerns · p. 12
• Proxy-loss evaluation separately re-tokenizes prompts with the proxy model’s tokenizer because the proxy may use a different tokenizer.No large open models using the OpenAI tokenizer were available at the time described.Source: C Tokenization concerns · p. 12

• Random-looking adversarial tokens make input perplexity filtering an effective defense against the attack.Source: D Defenses · p. 13
• The attack requires a method to estimate the target model’s log-probabilities for arbitrary output tokens.Source: D Defenses · p. 13
• Attacks operating without computable log-probabilities remain an open direction, while bypassing perplexity filters could support adaptive attacks.Source: D Defenses · p. 13

• GPT-3.5 Turbo Instruct’s token log-probabilities vary over time even with fixed seeds and temperature 0, so generations are not always reproducible.The authors do not know the exact cause of this nondeterminism.Source: E OpenAI API Nondeterminism · p. 13
• The loss distribution for one prompt-target pair had a sample standard deviation of 0.068 across 1,000 samples.The authors attribute the observed variation to the API rather than numerical estimation error.Source: E OpenAI API Nondeterminism · p. 13, Figure 7 · p. 13
• The moderation API’s maximum category score had mean 0.02 and standard deviation 4 × 10−4 across 1,000 samples.Because the observed noise was small, the authors used single samples as estimators during optimization.Source: E OpenAI API Nondeterminism · p. 13
• 519 prompts, about 90%, successfully produced the target string a second time.The authors leave algorithmic improvement of this reproduction rate to future work.Source: E OpenAI API Nondeterminism · p. 13