Source-linked AI summary
AdaPPA: Adaptive Position Pre-Fill Jailbreak Attack Approach Targeting LLMs
Lijia Lv, Weigang Zhang, Xuehai Tang, Jie Wen, Feng Liu, Jizhong Han, Songlin Hu
TL;DR
Semantic-focused jailbreaks are easily detected and overlook differences in defenses across output positions. AdaPPA adaptively pre-fills safe content before shifting toward harmful output; the authors report a 47% improvement over similar work on the most secure Llama model.
Problem
Existing semantic-level jailbreak methods are easily detected and overlook potential differences in model protection across output positions.
Source: §I · p. 1Method
AdaPPA adaptively pre-fills model output with safe and harmful content, using instruction following and narrative shifts to induce harmful responses.
Source: §I · p. 1Results
47% enhancement in performance over similar work, with nearly 30% attack success rate on the most secure Llama model.
Source: B. Universal Results · p. 4Takeaways & Limitations
The authors report that AdaPPA outperforms baseline methods on black-box models and can uncover vulnerabilities by inducing malicious responses.
Source: §V · p. 4The reported comparison uses six combinations in a single iteration for AdaPPA, versus 50 iterations for baseline work.
Source: B. Universal Results · p. 4
Abstract
from arXiv · showhide
Jailbreak vulnerabilities in Large Language Models (LLMs) refer to methods that extract malicious content from the model by carefully crafting prompts or suffixes, which has garnered significant attention from the research community. However, traditional attack methods, which primarily focus on the semantic level, are easily detected by the model. These methods overlook the difference in the model's alignment protection capabilities at different output stages. To address this issue, we propose an adaptive position pre-fill jailbreak attack approach for executing jailbreak attacks on LLMs. Our method leverages the model's instruction-following capabilities to first output pre-filled safe content, then exploits its narrative-shifting abilities to generate harmful content. Extensive black-box experiments demonstrate our method can improve the attack success rate by 47% on the widely recognized secure model (Llama2) compared to existing approaches. Our code can be found at: https://github.com/Yummy416/AdaPPA.
I. INTRODUCTION
The introduction identifies a gap in semantic-level jailbreaks: they overlook how defenses may vary across output positions. AdaPPA exploits pre-filled safe and harmful content to shift the model toward generating harmful responses.
I. INTRODUCTION
- Semantic-level attacks are easily detected and do not adequately account for differences in model protection across output positions.The introduction names gradient optimization, jailbreak prompt search, and red-team search as examples of semantic-level methods.Source: §I · p. 1
- Pre-filling a safe response can create an illusion of completion, after which a narrative shift may lower the model’s guard and increase the likelihood of harmful output.The paper reports that pre-fill length also affects vulnerability to successful attacks.Source: §I · p. 1
- The method pre-fills output positions with adaptively generated safe and harmful content, targeting positions where model defenses are weakest.The approach uses instruction following and narrative shifts to induce harmful output.Source: §I · pp. 1–2
- Experiments examine how different content types and lengths affect attack effectiveness, including unfilled, generic, adaptive, safe-termination, and hybrid fills.Hybrid Fill combines Adaptive Fill with Safe Termination; Figure 2 reports ASR for ChatGLM3-6b.Source: §I · p. 1, Figure 2 · p. 2
- AdaPPA significantly improves attack success rates over state-of-the-art jailbreak approaches on 10 classic black-box models.The paper frames the method as combining safe and harmful responses to exploit shallow alignment.Source: §I · p. 2
II. THE DESIGN OF ADAPPA
AdaPPA trains models to rewrite questions and generate pre-filled content, combines their outputs to attack a target model, and evaluates whether the attack succeeds.
II. THE DESIGN OF ADAPPA
- AdaPPA uses low-rank training on a pre-fill content generator with safe and harmful data and on a question-rewriting model with designed partial rewrites.Source: §II · p. 2
- The input question is sent to both models to obtain a rewritten question and pre-filled content, which are combined in different ways for the target-model attack.Source: §II · p. 2
- The final step evaluates whether the attack succeeded.Source: §II · p. 2
A. Problem Rewrite
The design includes rewriting harmful questions and adaptively generating safe and harmful pre-fill content.
A. Problem Rewrite
- The rewrite module prompts for a security-research background and relevant attack methods, then uses TF-IDF cosine similarity to assess the rewritten question against the original.Rewrites with similarity of at least 0.6 are treated as similar; the Vicuna rewriting model is low-rank fine-tuned on attention q and v components with rank r = 256.Source: A. Problem Rewrite · p. 2
B. Pre-fill Generation
- Safe pre-fills may be generic answers or targeted safe answers generated by Llama2 from the original question, and are segmented by length.Source: B. Pre-fill Generation · p. 2
- Harmful pre-fills are generated by embedding the original question in a Llama2 prompt with a predefined harmful response, rather than using a generic malicious response whose effectiveness was unsatisfactory.The model continues from the preceding token, and the trained generator adaptively produces safe and harmful responses.Source: B. Pre-fill Generation · p. 2
C. Prompt Combination
Prompt combinations vary the amount and type of pre-filled content to seek an effective attack prompt, while avoiding excessively long fills that may be detected as malicious.
C. Prompt Combination
- Excessively long pre-filled content may be detected as malicious and trigger the model’s adaptation mechanism.Source: C. Prompt Combination · p. 2
- Six representative combinations vary harmful-fill length and whether a generic safe response and transitional phrase precede it.The harmful fill may be one-third, two-thirds, or all of the model-generated response; the original or rewritten question can be used.Source: C. Prompt Combination · p. 3
- The combinations are selected heuristically, and successful cases are added to the training database for adversarial fine-tuning.The authors say this enhances the model’s attack capability, adaptability, and overall performance.Source: C. Prompt Combination · p. 3
D. Attack and Judge.
Attacks are sent to target models under each model’s fine-tuning protocol to approximate proprietary black-box deployment. Responses are screened automatically and then checked manually.
D. Attack and Judge.
- Attack requests are evaluated under each model’s specified fine-tuning protocol to simulate proprietary black-box deployment.The protocol is followed to ensure the models’ defense capabilities are represented.Source: D. Attack and Judge · p. 3
- Llama Guard 2 provides an initial automated assessment of target-model responses.The automated assessment is augmented by manual review.Source: D. Attack and Judge · p. 3
- Manual review validates the automated assessment and the resulting outcome labels.The paper describes this review as confirming assessment precision and reliability.Source: D. Attack and Judge · p. 3
A. Experimental Setup
The experiments evaluate black-box attacks using selected LLMs, harmful-question datasets, automated and manual judging, and Attack Success Rate.
A. Experimental Setup
- Ten models and variants were selected, including ChatGLM3, Vicuna, Llama2, Llama3, GPT-4o-Mini, and GPT-4o.The study follows official model instruction formats and uses greedy generation for reproducibility.Source: A. Experimental Setup · p. 3
- The evaluation uses PKU BeaverTails for training and observation and AdvBench for testing.The BeaverTails data include 14,000 filtered entries across risk scenarios.Source: A. Experimental Setup · p. 3
- Table I presents black-box attack test results using ASR as the metric.The caption identifies the table’s scope and evaluation measure.Source: Table I · p. 4
- Attack Success Rate (ASR) is the proportion of successfully attacked questions among all harmful questions.Results are also assessed using automated discriminators followed by manual review.Source: A. Experimental Setup · pp. 3–4
B. Universal Results
AdaPPA reports higher attack success than similar work, including on secure and commercial models. The reported results include a 47% performance enhancement over similar approaches.
B. Universal Results
- 47% enhancement in performance over similar work is reported for AdaPPA’s attack success rate.The comparison uses six combinations in a single iteration for AdaPPA, versus 50 iterations for baseline work.Source: B. Universal Results · p. 4
- Nearly 30% attack success is reported on the most secure Llama model, while GPT models reach 80%.The passage also reports 90% on models such as ChatGLM3 and Vicuna.Source: B. Universal Results · p. 4
- The authors report that AdaPPA can identify potential security risks even in large commercial models such as GPT-4o.They describe the method as more effectively inducing malicious responses than existing methods.Source: B. Universal Results · p. 4
C. Comparison of Different Fill Attacks
Different filler contents and question combinations produce substantially different attack effectiveness on ChatGLM3-6B, motivating adaptive filler generation. The related-work section groups jailbreak methods into three broad categories.
C. Comparison of Different Fill Attacks
- For ChatGLM3-6B, attack effectiveness varies substantially across question combinations and filler contents, supporting adaptive filler generation per question.The comparison is presented in Table II; Q is the original harmful question, S safe response, H harmful response, and R the rephrased question.Source: C. Comparison of Different Fill Attacks · p. 4, Table II · p. 4
IV. RELATED WORK
- Jailbreak methods are grouped as optimization-based, minority language-based, and distribution-based approaches.The section introduces this three-part categorization of prior methods.Source: §IV · p. 4
- Optimization-based methods include gradient optimization, genetic algorithms, and edit-based prompt refinement.The described approaches manipulate inference parameters, search prompts through mutation and selection, or refine adversarial prompts.Source: §IV · p. 4
- Minority-language attacks exploit model cognitive deficiencies, while distribution-based methods use templates or contextual examples to elicit harmful information.The passage gives cryptography and less common languages as examples of the former, and jailbreak templates and contextual examples as the latter.Source: §IV · p. 4
V. CONCLUSION
The paper presents a black-box jailbreak method that uses pre-filled content and narrative shifting to elicit malicious outputs. Experiments show improved effectiveness over baseline methods, supporting its use to reveal LLM security vulnerabilities.
V. CONCLUSION
- AdaPPA pre-fills output positions with content generated to exploit instruction-following, then uses narrative shifting to elicit malicious content.The method adaptively targets positions where model defenses are weakest.Source: §V · p. 4
- The black-box experiments show that AdaPPA improves attack effectiveness and outperforms baseline methods.Source: §V · p. 4
- The authors present the method as a way to reveal security vulnerabilities in LLMs.Source: §V · p. 4