PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Deep Prompt Optimization for Defending Against Jailbreak Attacks on LLMs

Doniyorkhon Obidov, Honggang Yu, Xiaolong Guo, Kaichen Yang

arXiv:2609.26185v1cs.CRcs.AIcs.CL

TL;DR

Jailbreak attacks can elicit harmful outputs, while existing prompt-optimization defenses are generally static and model fine-tuning is costly. DDPO uses the target LLM’s own intermediate representations to generate defensive embeddings dynamically, without changing its weights. The paper reports stronger performance than existing baselines, particularly for weakly aligned models and ambiguous benign prompts.

  • Problem

    Existing prompt-optimization defenses typically rely on static modifications that do not adapt to diverse inputs, while model fine-tuning is costly and can cause catastrophic forgetting.

    Source: Introduction · pp. 1–2

  • Method

    DDPO uses the target LLM’s intermediate features and a lightweight MLP to generate input-specific defensive embeddings for injection into a later layer without changing model weights.

    Source: Introduction · p. 1

  • Results

    DDPO significantly outperforms existing baselines, especially on weakly aligned models and semantically ambiguous benign prompts.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    The reported gains are most notable for weakly aligned models and ambiguous benign prompts, where distinguishing harmful from benign intent is challenging.

    Source: Introduction · p. 2

    DDPO assumes the defender has white-box access to the target LLM’s architecture, parameters, and internal activations.

    Source: Threat Model · p. 3

Abstract

from arXiv · show

Large Language Models (LLMs) demonstrate impressive capabilities across many applications but remain vulnerable to jailbreak attacks, which elicit harmful or unintended content. While model fine-tuning is an option for safety alignment, it is costly and prone to catastrophic forgetting. Prompt optimization has emerged as a promising alternative, yet existing prompt-based defenses typically rely on static modifications (e.g., fixed prefixes or suffixes) that cannot adapt to diverse and evolving attacks. We propose Dynamic Deep Prompt Optimization (DDPO), the first jailbreak defense based on deep prompt optimization. DDPO uses the target LLM's own intermediate layers as feature extractors to dynamically generate defensive embeddings via a lightweight multilayer perceptron. These tailored embeddings are then injected into a subsequent intermediate layer, enabling an input-dependent defense without modifying the LLM's weights. This design ensures high adaptability with minimal computational overhead. Experiments on a diverse set of models and attacks demonstrate that DDPO significantly outperforms static prompt optimization methods, particularly on weakly aligned models and when handling semantically ambiguous benign prompts, successfully distinguishing them from genuinely harmful requests.

Introduction

Static prompt defenses struggle to adapt to diverse inputs, while a naive dynamic approach adds an auxiliary-model pass and capacity trade-offs. DDPO uses the target LLM’s own features to generate input-specific defenses without changing its weights.

Introduction

  • Existing prompt-optimization defenses use fixed prefixes or suffixes, limiting their ability to adapt to diverse input nuances.Source: Introduction · p. 1
  • A naive dynamic defense requires an auxiliary LLM and a second input-processing pass, while a small auxiliary model may mishandle semantically complex inputs.Source: Introduction · p. 1
  • DDPO dynamically generates input-tailored defensive embeddings from the target LLM’s intermediate features and injects them into a later layer without modifying model weights.A lightweight MLP generates the embeddings, avoiding a large auxiliary model or a second processing pass.Source: Introduction · p. 1
  • DDPO differs from prior deep prompt tuning by generating a dynamic embedding for each input rather than learning fixed prompt embeddings.The paper identifies leveraging the target LLM’s own semantic capabilities as central to this distinction.Source: Introduction · p. 2
  • Experiments show DDPO outperforms existing baselines, especially on weakly aligned models and ambiguous benign prompts.The authors position it as a more lightweight and flexible alternative to full model fine-tuning.Source: Introduction · p. 2

Related Work

Prior work covers both jailbreak attack strategies and defenses at the model and prompt levels. Prompt-optimization defenses offer an alternative to filtering, but the methods reviewed generally produce static defensive components.

Related Work

  • Jailbreak attacks include manual black-box prompts, automated LLM-based methods, prompt rewriting, template completion, and white-box gradient-guided optimization.Examples include PAIR, GCG [Zou et al. 2023], GBDA [Guo et al. 2021], and PEZ.Source: Jailbreak Attacks · pp. 2–3
  • Defenses are broadly divided into model-level methods, such as SFT and RLHF, and prompt-level methods that operate without altering core model parameters.Model-level approaches can be resource-intensive, and fine-tuning may cause catastrophic forgetting.Source: Jailbreak Defenses · p. 3
  • Prompt optimization augments user inputs with optimized text or continuous embeddings and typically outperforms input-filtering defenses with lower false-positive rates.Source: Prompt Optimization Defenses · p. 3
  • PAT, DRO, and RPO use discrete prefixes, continuous safety prompts, or discrete suffixes, respectively, but each produces a static defense after training.The paper reports that this static nature is associated with underperformance on insufficiently aligned models and ambiguous benign prompts.Source: Prompt Optimization Defenses · p. 3

Methodology

DDPO selects an intermediate layer to derive an input-specific defensive embedding from the target LLM’s own representations, then injects it into later processing. The LLM’s weights remain frozen; only a lightweight MLP is trained, and its embedding is reused during generation.

Method Overview

Step 1: Optimal Layer Selection

  • The selected layer is the one with minimum average cosine similarity between representations of harmful and benign prompts.The method searches across layers for the strongest representational separation.Source: Step 1: Optimal Layer Selection · p. 4

Step 2: Dynamic Embedding Generation and Intermediate Injection

MLP Training Objective

Inference

Experiments

The experiments report that DDPO outperforms static defenses across models, including weakly aligned models and cases involving ambiguous benign prompts. Ablations support dynamic input-specific embeddings and a single-embedding configuration, while layer analysis identifies where representations best separate harmful and benign prompts.

Experimental Setup

  • Evaluation covered five LLMs, nine attack strategies, 230 unseen samples per attack, borderline-benign prompts, and MMLU; DDPO used a two-layer MLP with 512 hidden units.Training used 50 GCG and human-crafted jailbreak examples; seven attack types were evaluated zero-shot.Source: Experimental Setup · p. 5

Defense Performance

  • DDPO significantly outperforms static methods across models, balancing low ASR with high utility; on Openchat, it reduces ASR below 1% and maintains an average BPR of 95.40%.Table 1 compares defenses across models and multiple jailbreak attacks; the reported Openchat baseline ASR is 71.20%, while RPO yields 71.40%.Source: Table 1 · p. 6, Defense Performance · p. 6

Impact of Dynamic Generation

  • SDPO is significantly less effective than DDPO at preventing jailbreaks; on Vicuna and Openchat, its ASRs are 47.20% and 79.32%, versus DDPO’s 0.68% and 0.97%.On Llama2, DDPO also improves BPR from 70.00% to 93.20% while providing a stronger defense; both methods used the same training data and intermediate layers.Source: Impact of Dynamic Generation · pp. 6–7, Table 2 · p. 6

Number of Injected Embeddings

  • Increasing the number of embeddings to K = 5 or K = 10 yields no significant or consistent gains over the default K = 1, while adding computational overhead and inference latency.Some configurations show marginal gains on particular models, while others slightly degrade ASR.Source: Effect of Number of Injected Embeddings · p. 7, Table 3 · p. 7

Intermediate-Layer Selection

  • Across models, harmful–benign similarity follows a U-shaped trend, reaching a minimum at later intermediate layers, typically between the 62nd and 81st percentile of model depth.The minimum marks the layer of maximal semantic divergence and is selected for DDPO intervention.Source: Figure 3 · p. 7, Layer Selection Analysis · p. 7

Conclusion

DDPO uses the target LLM’s hidden representations and a lightweight MLP to generate defensive embeddings, which it injects into a later intermediate layer without changing model weights. Experiments report robust protection on weakly aligned models and improved distinction between harmful inputs and ambiguous benign queries.

Conclusion

  • DDPO uses the target LLM’s hidden representations and a lightweight MLP to generate defensive embeddings, which are injected into a later intermediate layer without modifying model weights.The embeddings are generated dynamically for user inputs.Source: Conclusion · p. 7
  • DDPO establishes protection on weakly aligned models where static defenses are largely ineffective.The conclusion also reports improved distinction between malicious inputs and semantically challenging benign queries.Source: Conclusion · p. 7
  • DDPO resolves the reported safety–utility trade-off by distinguishing malicious inputs from ambiguous and semantically challenging benign queries.Source: Conclusion · p. 7
Loading 2609.26185v1…