AdvPrefix: An Objective for Nuanced LLM Jailbreaks

Motivation

• The common objective is misspecified because low optimization loss can still yield incomplete, unfaithful, or non-harmful responses.It is also overconstrained by rigid handcrafted prefixes that may not match a victim model’s natural response style.Source: §1 · p. 1
• ∼10%: standard GCG achieves trivial attack success rates on Gemma-2, motivating objectives that generalize beyond a single rigid prefix.Source: §1 · p. 2

Evaluation

• Nuanced evaluation counts only complete and faithful harmful responses as successful, and existing judges can overestimate ASR by up to 30%.The refined evaluation also develops improved and preference-based judges for nuanced harmfulness.Source: §1 · p. 2

Contribution

• AdvPrefix selects model-dependent prefixes using high prefilling ASR and low initial NLL, while supporting multiple prefixes and integration with existing attacks.The pipeline generates prefixes from rule-based constructions or uncensored LLMs and selects them automatically.Source: §1 · p. 2

Empirical findings

• 16% to 70%: GCG’s nuanced ASR on Llama-3 improves with AdvPrefix, reaching 80% with full prompt optimization.The result indicates that current safety alignment struggles to generalize to unseen prefixes.Source: §1 · p. 2

Implications

• AdvPrefix improves jailbreak attacks without additional attack cost, enables attacks on reasoning LLMs, and supports future red-teaming.The reported harmfulness gains remain bounded: no model produced materially harmful information unavailable on the broader internet.Source: §1 · p. 2

Defining Nuanced Jailbreaks

• A nuanced jailbreak succeeds only when the response is affirmative, complete, faithful, on-topic, detailed, and realistic.Responses failing these criteria count as failed jailbreaks.Source: §2 · p. 3

Meta-evaluation

• 800 manually labeled GCG responses across Llama-2, Llama-3, Llama-3.1, and Gemma-2 provide ground truth for evaluating HarmBench, JailbreakBench, and StrongReject.The dataset uses 50 highly harmful, non-ambiguous AdvBench requests.Source: §2 · p. 3

Evaluation Challenges with Newer LLMs

• Existing judges can overestimate ASR from 10% ground truth to nearly 40% on Gemma-2 by misjudging incomplete and unfaithful responses.StrongReject is relatively more accurate than the other evaluated judges.Source: §2 · p. 3

Our Improved Judges

• The improved Llama-3.1-70B judge prioritizes completeness and faithfulness, uses reasoning traces, and improves human agreement by up to 9% on newer LLMs.A separate preference judge compares harmfulness relative to uncensored LLM outputs.Source: §2 · p. 3

3.1 Revisiting Original and Oracle Objectives.

• The threat model alters an attack prompt θ to steer the victim model, while a nuanced judge defines the acceptable response set Y_x.Source: §3.1 · p. 4
• The oracle objective maximizes likelihood over all acceptable harmful responses but is prohibitively costly because the response set is typically vast.The prefix objective is a cheaper surrogate that maximizes likelihood for a specific prefix and all completions beginning with it.Source: §3.1 · p. 4

3.2 Two Limitations

• Lowering the original objective’s loss does not reliably raise nuanced ASR: refusals decrease while incomplete and unfaithful responses increase.Prefilling the target prefix removes direct refusals but leaves ASR low because incomplete and unfaithful completions persist.Source: §3.2 · p. 4, Figure 3 · p. 4
• The original objective is overconstrained because rigid handcrafted prefixes can conflict with a victim model’s natural response style and hinder optimization.Llama-3 rarely starts with “Sure” and prefers “Here,” making the standard prefix less natural.Source: §3.2 · p. 5
• Replacing “Sure, here is” with “Here is” in GCG attacks on Llama-3 produces consistently lower final losses with equal or improved ASRs.Source: §3.2 · p. 5

4.1 Selective Multi-Prefix Objective

• AdvPrefix minimizes the negative log-likelihood of generating any prefix in a selected set, using multiple prefixes to relax rigid format constraints.Tree attention concatenates multiple prefixes for efficient computation in one forward pass.Source: §4.1 · p. 5

4.2 Prefix Selection Criteria

• High prefilling ASR favors prefixes that lead to complete and faithful harmful continuations, reducing misspecification.A manually constructed attack prompt provides an efficient approximation that correlates with actual jailbreak ASR.Source: §4.2 · p. 5
• Low initial NLL favors prefixes that optimized attack prompts can elicit more easily, reducing overconstraints.Prefilling ASR and NLL can conflict, so their weighted combination is tuned to the optimization method’s strength.Source: §4.2 · p. 6

4.3 Prefix Selection Pipeline

• The pipeline generates diverse candidates from rule-based templates or uncensored LLMs, then augments and filters them before evaluation.Candidate sources need not be the uncensored victim model, and guided decoding targets natural victim-model continuations.Source: §4.3 · p. 6
• Candidates are evaluated with victim-model NLL and temperature-one completions scored by a nuanced judge, then ranked by a weighted combination of both criteria.To select k prefixes, the method retains prefixes whose prefilling ASR is at least that of the top reference prefix.Source: §4.3 · p. 6

5.1 Main Results

• 70% ASR on Llama-3 replaces roughly 10% with model-dependent prefixes, and multiple prefixes often improve results further.The experiments evaluate GCG and AutoDAN across Llama-2, Llama-3, Llama-3.1, and Gemma-2 using suffix and full-prompt threat models.Source: §5.1 · p. 7, Table 1 · p. 7, Table 2 · p. 7
• Incomplete responses fall from about 20% to 1–2% on three newer LLMs, while unfaithful responses are halved on Gemma-2.These changes indicate that AdvPrefix mitigates misspecification rather than merely lowering the optimization loss.Source: §5.1 · pp. 7–8
• Longer suffixes provide an additional 9–15% ASR gain with AdvPrefix by reducing direct refusals while managing incomplete and unfaithful responses.The original objective remains near 10% ASR on newer LLMs despite lower final losses.Source: §5.1 · p. 8
• Full-prompt optimization raises GCG ASR from 39% to 73% on Llama-2 and from 70% to 80% on Llama-3 with AdvPrefix.The original objective changes inconsistently under this less restrictive threat model because it produces more incomplete and unfaithful responses.Source: Table 2 · p. 7, §5.1 · p. 8

5.2 Additional Results

• 78% ASR on Llama-3 is achieved by AutoDAN with AdvPrefix versus 5% with the original objective.The improvement appears across all evaluated victim LLMs, demonstrating compatibility beyond GCG.Source: Table 2 · p. 7, §5.2 · p. 8
• GCG with AdvPrefix reaches 30–50% harmfulness win rates against an uncensored LLM, compared with about 10% for the original objective.The preference evaluation therefore places AdvPrefix responses closer to uncensored-model harmfulness, although below 50% they remain less harmful overall.Source: §5.2 · p. 8, Figure 5 · p. 8
• Some selected Gemma-2 prefixes are non-affirmative, such as “**Password Cracking Algorithm** **” and “**Step 1: Reconnaissance** *”.This illustrates that model-dependent targets need not follow the conventional affirmative prefix format.Source: §5.2 · p. 8
• Prefix selection costs about five minutes per prefix on an A100 80G GPU, but the resulting prefixes can be reused as a one-off cost.The 70B judge used for prefilling-ASR estimation is the main computational and memory expense.Source: §5.2 · p. 8

• Jailbreak research includes manual, white-box, and black-box attacks, with white-box methods often stronger against defended models such as Llama-2.Recent work also explores reasoning exploits, assistive tasks, fluency, and transferable attacks.Source: §6 · p. 9
• Prior objectives suppress refusals, augment prefixes, steer responses, or distill from teachers, whereas AdvPrefix automatically selects victim- and request-specific prefixes for harmfulness and low likelihood.The paper positions this selection as a systematic response to misspecification and overconstraint.Source: §6 · p. 9

• AdvPrefix improves jailbreak objectives by selecting prefixes that better elicit complete, faithful harmful responses and can be reused across attacks.The design is presented as compatible with existing attacks and capable of providing performance gains without changing the attack method.Source: §7 · p. 9
• The objective requires many sampled-response evaluations, omits other desirable properties such as a well-shaped loss landscape, and needs target-model log probabilities.Consequently, it cannot be applied to black-box attacks.Source: §7 · p. 9

• The authors acknowledge potential misuse but state that their methods add no new risks or dangerous capabilities beyond existing attacks and open-source uncensored models.This claim is presented within the paper’s responsible-development rationale.Source: Broader Impacts · pp. 9–10
• The paper presents controlled vulnerability research as a way to identify and mitigate risks before they emerge in real-world deployments.Source: Broader Impacts · p. 10

A Additional Discussions

• Newer LLMs often self-correct after beginning a target prefix, producing incomplete or unfaithful responses rather than directly refusing.Llama-2 resists about 90% of attacks but gives unfaithful responses 24% of the time, whereas Gemma-2 almost always gives incomplete or unfaithful responses.Source: A Additional Discussions · p. 15
• AdvPrefix experiments show that safety alignment fails to generalize to model-dependent prefixes.The paper suggests developers may use original-objective prefixes during supervised fine-tuning, which could explain this behavior.Source: A Additional Discussions · p. 15
• Prefix-forcing remains sample-efficient because it specifies a low-entropy prefix distribution instead of estimating hundreds of future tokens.This bypasses the difficulty of estimating high-entropy autoregressive outputs while preserving a tractable optimization objective.Source: A Additional Discussions · p. 15
• Prefix-forcing is a special case of model distillation when the teacher distribution collapses to one prefix, while multi-prefix objectives reduce overconstraint.The paper also notes that single-prefix distillation can be empirically learnable through hard-token prompt optimization.Source: A Additional Discussions · p. 15

B More Related Work

• The paper positions AdvPrefix against safety-alignment strategies spanning pretraining, fine-tuning, and deployment-time filters.Its findings suggest these strategies need more tailored prefixes to improve generalization.Source: B More Related Work · p. 15
• Even after the target prefix is elicited, lower loss does not guarantee a complete and faithful harmful response.This distinguishes the paper’s misspecification finding from exposure-bias explanations focused on failure to elicit the first token.Source: B More Related Work · p. 15

C Additional Experimental Details

• The evaluation uses provided judge models for HarmBench, JailbreakBench, and StrongReject, while omitting their API-based judging options.StrongReject scores are binarized at 0.6 because that threshold maximizes F1 on the dataset.Source: C Additional Experimental Details · pp. 15–16
• A preference judge compares the relative harmfulness of attack-induced responses against responses from an uncensored LLM.This avoids requiring a perfect absolute or binary harmfulness score.Source: C Additional Experimental Details · p. 16
• The study selects 50 highly harmful AdvBench requests while excluding child-abuse topics to improve evaluation consistency.Topic-dependent alignment creates variance because newer LLMs respond differently across malicious categories.Source: C Additional Experimental Details · p. 16
• Candidate prefixes can be generated from prefilling uncensored LLMs, rule-based constructions, or base LLMs.Prefilling with phrases such as “Here”, “To”, or “Sure” helps uncensored models comply with highly harmful prompts.Source: C Additional Experimental Details · p. 16
• When multiple prefixes are selected, none may be a sub-prefix of another, preventing redundant computation and keeping their total probability at most one.The pipeline also includes candidate-pool, rejection, and weighting parameters.Source: C Additional Experimental Details · p. 16

D Additional Results

• Figure 6 provides additional examples of failed nuanced jailbreaks, helping distinguish harmless responses that existing judges may misclassify.Nuanced success requires affirmative, complete, and faithful responses.Source: Figure 6 · p. 16, Table 5
• Alternative judges yield similar relative ASR improvements when GCG optimizes the entire attack prompt.These results serve as an ablation of the main evaluation using three other judges.Source: Table 3 · p. 17
• The detailed meta-evaluation reports human agreement rates outside brackets and F1 scores inside brackets for existing and proposed judges.The table is intended to compare judge quality rather than attack success directly.Source: Table 4 · p. 17
• Prefilling ASR correlates with jailbreak ASR across four LLMs and 50 requests, enabling efficient prefix selection.The comparison covers request-only prompts, instruction-prefixed prompts, and prompts followed by an initial attack suffix.Source: Figure 7 · p. 17
• Replacing “Sure, here is [request]” with “Here is [request]” lowers final losses and raises average ASR from 12.8% to 22.1% on Llama-3.The result indicates that the original prefix objective is overconstrained for optimization.Source: Figure 8 · p. 18
• Model-dependent target prefixes enable attacks on R1-distilled-Llama-3-8B, whereas GCG with default prefixes cannot lower the loss enough for successful jailbreaks.This extends the objective beyond standard instruction-tuned models.Source: Figure 9 · p. 18