PaperlayerView exact revision on arXiv

Source-linked AI summary

The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections

Milad Nasr, Nicholas Carlini, Chawin Sitawarin, Sander V. Schulhoff, Jamie Hayes, Michael Ilie, Juliette Pluto, Shuang Song, Harsh Chaudhari, Ilia Shumailov, Abhradeep Thakurta, Kai Yuanqing Xiao, Andreas Terzis, Florian Tramèr

arXiv:2510.09023v1cs.LGcs.CR

TL;DR

LLM defenses are often evaluated with static prompts or generic, low-budget attacks rather than attackers adapted to each defense. The paper develops a general adaptive-attack framework and applies it across defenses, finding high attack success and arguing that human red-teaming remains necessary alongside automation.

  • Problem

    LLM defense evaluations commonly use static attack sets or generic optimization methods that are not adapted to the defense, limiting robustness assessment.

  • Method

    The paper presents an adaptive optimization-loop framework spanning gradient-based, reinforcement-learning, search-based, and human red-teaming attacks.

  • Results

    12 defenses were bypassed, with attack success rates above 90% across most defenses compared with near-zero rates originally reported.

  • Takeaways & Limitations

    Human red-teaming remains a valuable complement because it succeeded in all evaluated cases while automated search succeeded 69% of the time on a selected subset.

  • Takeaways & Limitations

    Detectors can block some unsophisticated attacks but remain a useful—and limited—component of a broader defense strategy.

Abstract

from arXiv · show

How should we evaluate the robustness of language model defenses? Current defenses against jailbreaks and prompt injections (which aim to prevent an attacker from eliciting harmful knowledge or remotely triggering malicious actions, respectively) are typically evaluated either against a static set of harmful attack strings, or against computationally weak optimization methods that were not designed with the defense in mind. We argue that this evaluation process is flawed. Instead, we should evaluate defenses against adaptive attackers who explicitly modify their attack strategy to counter a defense's design while spending considerable resources to optimize their objective. By systematically tuning and scaling general optimization techniques-gradient descent, reinforcement learning, random search, and human-guided exploration-we bypass 12 recent defenses (based on a diverse set of techniques) with attack success rate above 90% for most; importantly, the majority of defenses originally reported near-zero attack success rates. We believe that future defense work must consider stronger attacks, such as the ones we describe, in order to make reliable and convincing claims of robustness.

1 INTRODUCTION

Robustness evaluations for LLM defenses should use strong adaptive attackers rather than static prompts or weak, non-adaptive optimization. The paper applies this framework to 12 defenses and reports substantially higher attack success than the original evaluations.

  • 1 INTRODUCTION: Strong adaptive attackers must explicitly counter a defense’s design and receive an unrestricted computational budget.Static or low-compute evaluations can make defenses appear deceptively robust.
  • 1 INTRODUCTION: LLM defenses against jailbreaks and prompt injections are often evaluated without strong adaptive attacks.The paper identifies this as a continuation of a problem already recognized in adversarial machine learning.
  • 1 INTRODUCTION: 12 defenses were bypassed, with attack success rates above 90% across most defenses versus near-zero rates reported originally.The attacks included human and LLM-based methods selected for individual defenses.

2 A BRIEF HISTORY OF ADVERSARIAL ML EVALUATIONS

LLM defense evaluations commonly rely on static malicious-prompt sets or generic automated attacks with limited budgets. These approaches are neither adaptive nor sufficient, while expert humans continue to find attacks that automation misses.

  • 2 A BRIEF HISTORY OF ADVERSARIAL ML EVALUATIONS: Strong adaptive evaluation assumes attackers know the defense and have sufficient resources to search for successful attacks.This assumption parallels computer-security and cryptographic evaluation practice.
  • 2 A BRIEF HISTORY OF ADVERSARIAL ML EVALUATIONS: Human ingenuity remains central to many computer-system attacks, while automated methods are especially effective when attacks reduce to simple search problems.The paper contrasts this pattern with adversarial-example research, where automated optimization often dominates.
  • 2 A BRIEF HISTORY OF ADVERSARIAL ML EVALUATIONS: Current LLM evaluations use static malicious prompts or generic automated optimization attacks with relatively low computational budgets.The paper identifies both practices as flawed evaluation approaches.
  • 2 A BRIEF HISTORY OF ADVERSARIAL ML EVALUATIONS: Expert humans routinely find successful attacks against state-of-the-art models and defenses where existing automated attacks fail.This motivates combining scalable automated attacks with human red-teaming.

3 PROBLEM STATEMENT, THREAT MODEL, AND ATTACKER’S CAPABILITIES

The evaluation framework models attackers with different levels of access and large computational resources. It treats the central objective as testing whether defenses withstand capable attackers rather than measuring attack difficulty.

  • 3 PROBLEM STATEMENT, THREAT MODEL, AND ATTACKER’S CAPABILITIES: Human red-teamers provide tailored attacks, but their expense motivates automated gradient-based or LLM-assisted methods for scalable evaluation.Different methods may require different knowledge or access to the target defense.
  • 3 PROBLEM STATEMENT, THREAT MODEL, AND ATTACKER’S CAPABILITIES: The framework defines adversaries by varying degrees of access to the target model.The listed settings include white-box, black-box with logits, and black-box generation-only access.
  • 3 PROBLEM STATEMENT, THREAT MODEL, AND ATTACKER’S CAPABILITIES: White-box attackers have model architecture, parameters, internal states, and gradient access.This is the most informative access setting described.
  • 3 PROBLEM STATEMENT, THREAT MODEL, AND ATTACKER’S CAPABILITIES: Black-box generation-only attackers observe only the model’s final discrete output, without internal states, parameters, or output distributions.They can still query the model with arbitrary inputs.
  • 3 PROBLEM STATEMENT, THREAT MODEL, AND ATTACKER’S CAPABILITIES: Across threat models, attackers receive large computational resources so evaluation reflects defense strength rather than attacker limitations.The paper distinguishes cryptographic-scale hardness from attacks requiring a fixed commodity-hardware budget.

4 GENERAL ATTACK METHODS

The paper presents adaptive attacks as iterative optimization loops that modify prompts in response to defense behavior. It instantiates this framework with gradient-based, reinforcement-learning, search-based, and human red-teaming methods.

  • 4 GENERAL ATTACK METHODS: Adaptive attacks iteratively modify their strategy instead of relying on one fixed attack.The framework organizes each iteration into a common optimization-loop structure.
  • 4 GENERAL ATTACK METHODS: The framework includes four attack families: gradient-based, reinforcement learning, search-based, and human red-teaming.One attack from each family is instantiated experimentally.
  • 4 GENERAL ATTACK METHODS: Gradient-based attacks estimate gradients in embedding space and project them back to valid discrete tokens.Their reliability is limited by the vast discrete prompt space and unpredictable effects of wording changes.
  • 4 GENERAL ATTACK METHODS: Reinforcement-learning attacks sample candidate prompts, score model behavior, and update a policy to improve attack success.The described implementation uses an LLM to suggest triggers from score feedback and updates its weights with GRPO.
  • 4 GENERAL ATTACK METHODS: Search-based attacks explore the discrete prompt space using heuristic or model-guided operators; the paper uses genetic mutation suggested by an LLM.These methods do not require gradient access.
  • 4 GENERAL ATTACK METHODS: The paper’s main robustness criterion is whether a defense fails against any adaptive instantiation of the PSSU loop.Human red-teaming is included as a canonical adaptive attack family.

5 EXPERIMENTS

The experiments test 12 defenses across jailbreak and prompt-injection settings using original and additional benchmarks, then apply adaptive attacks and human red-teaming. Across diverse defenses, adaptive attacks substantially exceed static-evaluation robustness, while detectors remain useful only against some unsophisticated attacks.

  • Experimental setup: 12 defenses spanning prompting, adversarial training, and filtering were evaluated on jailbreak and prompt-injection benchmarks.The benchmarks included HarmBench, AgentDojo, OpenPromptInject, and adversarial Davinci, depending on the defense.
  • Prompting defenses: 95% ASR was achieved against both Spotlighting and Prompt Sandwiching with search-based adaptive attacks.Human red-teaming also produced 265 successful attacks against Spotlighting and 178 against Prompt Sandwiching.
  • Training against existing attacks: 96% ASR was achieved against MetaSecAlign and 100% ASR against Circuit Breakers using adaptive attacks.MetaSecAlign had reported 2% ASR on static AgentDojo, while Circuit Breakers was evaluated on HarmBench.
  • Filtering model defenses: >90% ASR was achieved against Protect AI, PromptGuard, and Model Armor, while PIGuard reached 71% ASR.Human red-teaming collected 123 successful prompt injections under the strictest setting, with hundreds across detectors overall.
  • Filtering model defenses: Successful attacks often used benign-looking or indirect triggers that detectors could not flag without significantly increasing false positives.Some attacks redefined task terms without changing the model’s output structure, making them difficult to detect out of context.
  • Filtering model defenses: Adding filters or stacking detectors did not resolve the robustness problem, although detectors could block some unsophisticated or opportunistic attacks.The authors characterize detectors as useful but limited components of a broader defense strategy.

6 LESSON AND DISCUSSION

The discussion identifies four lessons for evaluating LLM defenses: static tests can mislead, automated attacks are useful but incomplete, human red-teaming remains effective, and model-based auto-raters can be unreliable.

  • Lesson #1: Small static evals can be misleading: Static evaluations can create a false sense of security because open-ended attacks inevitably fall outside limited public datasets.Defenses reporting near-zero attack success rates on public benchmarks are often among the easiest to break with novel attacks.
  • Lesson #2: Automated evaluation can be effective but not robust: Automated attacks can systematically bypass defenses, but robustness against an evaluation does not establish overall robustness.The authors argue that empirical evaluation should try as hard as possible to break a defense and ultimately fail.
  • Lesson #3: Human red-teaming is still effective: 100% human red-teaming success on the evaluated cases exceeds the 69% success of the search attack on a selected subset.The result supports using human experts as a complement to automated attacks, especially under strong threat models.
  • Lesson #4: Model-based auto-raters can be unreliable: Model-based safety classifiers are vulnerable because they are machine-learning models susceptible to adversarial examples and reward-hacking behaviors.This vulnerability can make automated assessments of policy violations unreliable.

7 CONCLUSION

The conclusion argues that LLM defense evaluation should resemble computer security by using adaptive, computationally intensive attacks, while recognizing that automated attack methods can themselves be misused.

  • 7 CONCLUSION: Adaptive LLM evaluations are difficult but important because computer-security defenses are tested against specialized attackers with substantial resources.The paper contrasts this setting with adversarial-example evaluations, which have stronger optimization attacks and cleaner definitions.
  • 7 CONCLUSION: Publishing capable attack methods carries misuse risk, but the authors argue that public evaluation tools are necessary to expose weaknesses before real adversaries exploit them.The ethics statement reports voluntary participation, informed participants, no personally identifiable or sensitive data collection, and anonymized data.
  • 7 CONCLUSION: The study used gradient-based, reinforcement-learning, search-based, and human red-teaming attacks as four canonical adaptive-attack families.Each family represents an instantiation of the common iterative structure behind adaptive attacks.
  • 7 CONCLUSION: The RL attack interacted directly with the defended system for five rounds per session, evaluated the best score, and updated the attacker policy across sessions using GRPO.Each sample used 32 independent sessions with a closed-source base model lacking safety alignment.
  • 7 CONCLUSION: The search attack used an OpenEvolve-based process in which a mutator LLM generated candidates from feedback and a critic supplied qualitative and numerical scores.Promising candidates were stored and reused to prompt subsequent mutation steps.
  • 7 CONCLUSION: Human red-teaming involved 40 AgentDojo challenges in a competition where participants submitted inputs through a web interface and observed agent reasoning and tool calls.The competition offered $20,000 in prizes and evaluated submissions within the AgentDojo environment.

B EXPERIMENT SETUP

The experiment setup introduces the benchmarks used for evaluation and summarizes the defenses and corresponding attack configurations; the human study uses an interactive challenge interface.

  • B EXPERIMENT SETUP: Table 1 summarizes all evaluated defenses together with their corresponding attack setups.The table is presented as an overview of the experimental defense-attack pairings.
  • B EXPERIMENT SETUP: Figure 3 shows a challenge interface where competitors receive environment instructions, test prompts, observe agent actions and outputs, and see task completion status.After selecting Try Again, competitors can enter another input in the chat textbox.
  • B EXPERIMENT SETUP: The experiments use HarmBench for jailbreak-defense evaluation and Alpaca to assess whether models follow instructions.The RL-based attack is used for jailbreak evaluations in HarmBench.

C RL-BASED EXPERIMENTS

The RL-based attack adapts through interaction, improving from weak initial prompts to precise defense-exploiting attack strings. Its score can also be misaligned with genuine attack success, producing reward hacking.

  • Adapting to defenses: The attacker performs poorly initially but improves dramatically after a few interaction rounds, eventually converging on a precise attack string against Data Sentinel.It first generates refused prompts, then refines examples that partially bypass the defense.
  • Reward Hacking: Reward hacking occurs when attackers maximize an imperfect scoring proxy without solving the intended task.For StruQ, repeating the target string increases the chance that the target model echoes it.

D SEARCH-BASED METHODS

The search-based method uses a modular evolutionary framework that adapts candidate attacks through control, mutation, and scoring components. The authors find that the specific evolutionary algorithm has limited effect on attack effectiveness.

  • Search-based method: The search algorithm uses an evolutionary approach to optimize adversarial text or code while remaining modular rather than tied to one evolutionary algorithm.
  • Search-based method: The attack has three components: a controller, mutators that generate candidates, and a scorer that evaluates feedback from the victim system.The scorer may use model outputs, log-probabilities, tool-call outputs, or a binary success oracle.
  • Search-based method: The controller uses MAP Elites to partition the search space and retain strong, diverse candidates as elites.
  • Search-based method: An LLM mutator receives sampled past attack attempts, scores, and feedback to generate new candidates, using eight candidates per iteration.
  • Search-based method: The authors report that evolutionary-algorithm choice does not have a large effect on attack effectiveness.Island with MAP Elites worked better in some settings and was used for the remaining experiments without hyperparameter tuning.
  • Human red-teaming: Human red-teaming competitions tested indirect prompt-injection payloads across models, defenses, and AgentDojo challenges.AgentDojo automatically evaluates whether the agent completes the attacker task; this is a higher bar than the automated attacks’ evaluation.

E.3 RESULTS

The experiments evaluate defenses across selected benchmark scenarios using human and automated red-teaming. The comparison is difficult because human skill and effort vary substantially across attempts.

  • Evaluation setup: The human competition evaluates defended and undefended AgentDojo task pairs, reporting ASR as successful adjudicated submissions divided by all submissions.
  • Evaluation setup: The study runs human red-teaming challenges across multiple models, defenses, and AgentDojo tasks by submitting indirect prompt-injection payloads.
  • Comparison challenges: Human and automated attacks are difficult to compare because human attackers have widely varying skill levels.Using the average human hacker can misleadingly lower ASR relative to an automated attack.
  • Comparison challenges: Attempt costs differ greatly between human and automated attacks, making query counts, wall time, and number of breaks non-equivalent measures.Automated attacks may make orders of magnitude more attempts, while humans craft prompts more slowly.

F.2 COMPARISON RESULTS

The paper compares automated search attacks with human red-teaming across overlapping AgentDojo scenarios. Humans are collectively more query-efficient, while the strongest individual participant slightly exceeds the search attack’s ASR.

  • Comparison setup: The comparison covers 29 overlapping AgentDojo scenarios spanning all evaluated defenses and several base models.
  • Benchmark results: The Protect AI Detector comparison is reported for a Travel-suite scenario, while ASR and query summaries use the benchmark’s stated definitions.
  • Human versus automated attacks: Humans collectively reach 100% ASR across scenarios, while the search attack reaches 69% ASR with 800 queries.The best participant succeeds with as few as 50 queries, but humans use multiple independent attempts per scenario whereas the search attack uses one.
  • Human versus automated attacks: The best individual participant attempting all 29 scenarios achieves 75% ASR, surpassing the search attack’s 69%.The search attack surpasses the other two participants who attempted all scenarios.
  • Benchmark results: For PI Detector, injection tasks whose secrets are mistaken for attacks and redacted are excluded from ASR and query calculations.

G.1 RESULTS FROM THE SEARCH-BASED METHOD

The search-based evaluation tests multiple models and defenses with adaptive attacks, finding high attack success across most combinations while exposing utility costs and varied attack strategies.

  • >80% ASR occurs on most model-defense combinations, except GPT-5 Mini with any defense and PIGuard paired with more robust models.The authors expect increased adversarial budgets, such as parallel trials, could raise ASR against GPT-5 Mini.
  • Search attacks usually plateau after 100–200 queries rather than using the full 800-query budget.The authors suspect better parallelism could extract more benefit from additional queries than from longer sequential search.
  • Model-filtering defenses substantially harm utility, while prompting-based defenses generally preserve utility but provide no robustness improvement beyond undefended models.The utility assessment uses all AgentDojo scenarios rather than only the evaluation subset.
  • Automated search explores encoding, multilingual, formatting, and disguise strategies, without revealing a completely new strategy beyond existing literature or internet examples.
  • Successful triggers are illustrated across defenses and models, including Protect AI, PIGuard, PromptGuard, Model Armor, MELON, Spotlighting, and Prompt Sandwiching.The examples primarily use AgentDojo Workspace scenarios and show triggers designed to induce actions such as sending emails or retrieving participant lists.
Loading 2510.09023v1…