From Expected Harmfulness to Likelihood: A Probabilistic Reformulation of Jailbreaking LLM Agents

1 INTRODUCTION

• Prior attacks often optimize the likelihood of a preselected harmful response, while expected-harmfulness optimization weighs responses by both occurrence probability and harmfulness (Geisler et al., 2025; Chen et al., 2026).The paper frames these as complementary perspectives on jailbreak optimization.Source: §1 · p. 1
• In the illustrative calendar-query case, OPUR explores alternative feasible trajectories and finds one that triggers the attacker-designated tool call, unlike UDora.The figure is an example rather than a benchmark result.Source: §1 · p. 2, Figure 2 · p. 2
• The paper establishes a gradient-level reformulation linking expected harmfulness and target likelihood under a harmfulness-reweighted output distribution.This provides a unified interpretation of behavioral and likelihood-based attack objectives.Source: §1 · p. 2
• Across multiple benchmarks under stochastic decoding, OPUR shows consistently higher targeted attack success rates than existing optimization-based baselines.The introduction presents this as an outcome of the proposed probabilistic formulation.Source: §1 · p. 3
• OPUR approximates the reweighted distribution using model-generated rollouts and uses promising contexts to guide adversarial suffix optimization.It considers multiple possible adversarial outcomes and prioritizes candidates that are both harmful and attainable under the model.Source: §1 · pp. 1, 3

2 PRELIMINAIRES

• The response distribution is written pθ(· | x, s), where x is the input and s is the adversarial suffix inside it.The model response is treated as a distribution over possible outputs.Source: §2 · p. 3

2.1 JAILBREAKING BASED ON EXPECTED HARMFULNESS

• Expected-harmfulness attacks seek a suffix that maximizes the response harmfulness ψ(y) averaged over the model’s response distribution.The gradient can be estimated with REINFORCE under differentiability, support, and interchange assumptions.Source: §2.1 · p. 3
• When an early-stage suffix is poorly formed, model samples are unlikely to be harmful, so ψ(y) is near zero and the expected-harmfulness gradient provides little information.Geisler et al. (2025) introduce heuristic rules to construct harmful responses as substitutes for model samples.Source: §2.1 · p. 3

2.2 LIKELIHOOD-BASED JAILBREAKING

• Likelihood-based jailbreaking instead tunes the suffix to increase the likelihood of a given harmful response (Zou et al., 2023).Because suffixes are discrete, gradients guide promising token substitutions followed by greedy coordinate-wise updates.Source: §2.2 · p. 3

2.3 PROBABILISTIC ADVERSARIAL ATTACK

• The cited probabilistic adversarial attack defines a distribution over adversarial examples by multiplying attack-objective preference by preference for staying close to the original input (Zhang et al., 2024; 2026).The distance measure D defines closeness to the original input.Source: §2.3 · p. 3

3 UNIFYING EXPECTED HARMFULNESS AND LIKELIHOOD-BASED JAILBREAKING

• The harmfulness-reweighted distribution π assigns greater probability to responses that are both likely under the current model and highly harmful, assuming positive expected harmfulness.This gives the target distribution used to connect expected-harmfulness and likelihood-based optimization.Source: §3 · p. 4
• Theorem 2 states that the gradient of log expected harmfulness equals the expected log-likelihood gradient under π, assuming differentiation and expectation can be interchanged.At a fixed suffix, maximizing expected harmfulness therefore induces the same update direction as increasing the likelihood of responses drawn from π.Source: §3 · p. 4
• Both formulations yield likelihood-gradient updates, but the earlier REINFORCE approach samples from the model distribution whereas this formulation takes expectations under π.Concentrating probability on harmful responses avoids the early-stage regime in which model samples are almost entirely benign and provide little useful gradient signal.Source: §3 · p. 4
• The paper notes that π has the same product form as the probabilistic adversarial attack described in Section 2.3.The distributions are discussed in different settings: π reweights responses, while the earlier framework describes adversarial examples.Source: §3 · p. 4
• Because direct sampling from π is difficult when harmful responses are rare, the method uses stochastic agent rollouts to construct surrogate targets.These targets represent high-probability harmful regions and can guide local suffix updates.Source: §3 · p. 4

4 OPUR

• OPUR constructs rollout-derived surrogate targets to represent high-probability harmful regions of π, combining bootstrap-style exploration with probabilistic position selection.The resulting targets are intended to be plausible under the current model and useful for likelihood-based suffix optimization.Source: §4 · p. 4

4.1 MOTIVATION

• Stochastic decoding can produce different trajectories and positional scores for the same input, so deterministic top-position selection may overfit to one rollout.OPUR samples multiple non-overlapping position sets and filters low-quality contexts to balance candidate diversity with optimization reliability.Source: §4.1 · p. 5

4.2 ATTACK ALGORITHM

• Each stochastic rollout supplies token-level probabilities and candidate injection positions; sampled non-overlapping spans are replaced with the target segment to form surrogate contexts.OPUR retains low-loss contexts, averages suffix cross-entropy gradients across them, generates token-substitution candidates, and selects the candidate with the lowest mean surrogate loss.Source: §4.2 · pp. 5–6
• The algorithm screens contexts using the current suffix before aggregating gradients across retained contexts for suffix updates.Candidate suffixes are evaluated by their mean surrogate loss over retained contexts, and the lowest-loss candidate becomes the next suffix.Source: §4.2 · p. 6

5.1 DATASETS

• AgentHarm measures harmful multi-step task execution; InjecAgent tests indirect injections in external observations; WebShop tests adversarial observations steering multi-step shopping toward unintended products.Source: §5.1 · p. 6

5.2 EXPERIMENT SETTING

• Experiments compare GCG, REINFORCE-GCG, and UDora on two instruction-tuned models, using ReAct prompting and Attack Success Rate as the primary effectiveness metric.Evaluation uses stochastic rollouts during optimization and a single rollout at temperature 0.7 and top-p 0.9 for final evaluation.Source: §5.2 · pp. 6–7, Experiment Setting · p. 7

5.3 RESULTS

• 75.00% ASR on Llama-3.1-8B-Instruct exceeds the strongest UDora baseline by 13.07%; Ministral-8B-Instruct-2410 reaches 99.43% ASR and matches the strongest UDora variants.Table 1 reports AgentHarm results under different attack categories.Source: Table 1 · p. 7, §5.3 · p. 7
• 49% ASR on Llama-3.1-8B-Instruct and 40% on Ministral-8B-Instruct-2410 are each 2 percentage points above the strongest InjecAgent baseline.Table 2 reports results under different attack categories.Source: §5.3 · p. 7, Table 2 · p. 8
• 26.67% ASR on Llama-3.1-8B-Instruct and 36.67% on Ministral-8B-Instruct-2410 exceed the strongest WebShop baseline by 8.34% on both models.OPUR performs best on three of four attack categories on Llama and all four on Ministral.Source: §5.3 · pp. 7–8, Table 3 · p. 8

5.4 ABLATION

• Increasing Kopt generally improves OPUR, though not monotonically; on Llama, AgentHarm ASR rises from 56.82% to 75.00%, while Ministral AgentHarm reaches 99.43% at Kopt = 1.Other increases include Llama WebShop from 5.00% to 26.67% and InjecAgent from 46.00% to 49.00%; Ministral WebShop rises from 11.67% to 36.67% and InjecAgent from 30.00% to 40.00%.Source: §5.4 · p. 8

• Prior adversarial-attack work advances prompt search across optimization-based and automated black-box approaches; this paper instead revisits the objective underlying jailbreak optimization.Source: §6 · pp. 8–9
• Recent work explores model-dependent targets, adaptive rewards, diverse suffixes, and sampling-aware or distribution-based attacks; this paper derives a likelihood objective from expected harmfulness.Source: §6 · p. 9
• Agent-jailbreak work targets reasoning, memory, or behavior and evaluates tool-related vulnerabilities; this paper instead focuses on reformulating the optimization objective.Source: §6 · p. 9

• The paper characterizes agent security as shifting the behavioral distribution toward harmful outcomes rather than seeking a single harmful response.Source: §7 · p. 9
• The authors propose future work to understand, evaluate, and ultimately control the full distribution of behaviors LLM agents may exhibit.Source: §7 · p. 9

Proof assumptions

• The identities assume differentiable response probabilities, support independent of the suffix, harmfulness independent of the suffix, and interchangeable differentiation and expectation.The equivalence theorem explicitly assumes differentiation and expectation can be interchanged.Source: A PROOF OF THE THEOREMS · p. 14

Expected-harmfulness gradient

• Under pθ(·|x,s), the gradient with respect to s of expected ψ(Y) equals the expectation of ψ(Y) times the response log-likelihood gradient.Source: A PROOF OF THE THEOREMS · p. 14

Gradient equivalence

• The log expected-harmfulness gradient equals the expected model log-likelihood gradient under the harmfulness-reweighted response distribution π.This connects expected-harmfulness and likelihood-based optimization at the gradient level.Source: A PROOF OF THE THEOREMS · p. 14

AgentHarm Ablation

• 75.00% average ASR at Kopt = 3 on AgentHarm exceeds the strongest UDora baseline, 61.93% at Kopt = 3, by 13.07 percentage points.OPUR’s average ASR rises from 56.82% at Kopt = 1 to 63.07% at Kopt = 2 and 75.00% at Kopt = 3.Source: B DETAIL OF ABLATION · p. 14, Table 5 · p. 15

InjecAgent Ablation

• 46.00% to 45.00% to 49.00% average ASR on Llama and 30.00% to 34.00% to 40.00% on Ministral as Kopt grows from 1 to 3 on InjecAgent.The Llama trend is not strictly monotonic; Kopt = 3 gives the strongest performance on both models.Source: B DETAIL OF ABLATION · pp. 14–15, Table 6 · p. 15

WebShop Ablation

• 26.67% average ASR on Llama-3.1-8B-Instruct and 36.67% on Ministral-8B-Instruct-2410 at Kopt = 3 on WebShop.Both models improve as Kopt rises from 1 to 3; the passage characterizes broader rollout aggregation as particularly beneficial in this more complex interaction setting.Source: B DETAIL OF ABLATION · pp. 15–16, Table 7 · p. 16

InjecAgent

• On InjecAgent, neither initial rollout contains BankManagerTransferFunds, but after suffix updates both optimization rollouts and final evaluation generate that target action.The retained surrogate replacements are training-only.Source: C OPUR ON REAL WORLD AGENT · p. 16, Figure 4 · p. 18

WebShop

• On WebShop, neither initial rollout selects the target item, but after the shared suffix update both regenerated rollouts and final evaluation produce click[B08KRVH121].The surrogate replacements guide optimization without modifying the agent’s actual generated responses.Source: C OPUR ON REAL WORLD AGENT · p. 16

AgentHarm

• On AgentHarm, stochastic rollouts yield a refusal and a target-query response; retained surrogate contexts guide a suffix update, after which both optimization rollouts and final evaluation succeed.The refusal supplies one training-only surrogate context, while the other rollout supplies four sampled position sets, of which two are retained using probe loss.Source: C OPUR ON REAL WORLD AGENT · p. 16, Figure 3 · p. 17