$M^3$: A Multi-LLM System for Multi-Turn Jailbreaking as a Markov Decision Process

1 Introduction

• Multi-turn attacks progressively reveal adversarial intent, a comparatively underexplored threat that can expose vulnerabilities missed by single-turn evaluations.The motivation is that such interactions mirror realistic misuse scenarios.Source: §1 · p. 1
• Existing methods may require curated datasets or human-authored trajectories, predefined strategies or taxonomies, attacker fine-tuning, complex architectures, or large context windows.These requirements can limit scalability, flexibility, exploration, or generalization and increase computational expense.Source: §1 · p. 1
• M3 is a fully automated, black-box-access framework that models multi-turn jailbreaking as an MDP and compresses conversation histories into a bounded internal representation.It imposes no fixed action space and supports dynamic, strategy-agnostic attack generation.Source: §1 · p. 2
• M3 achieves higher ASR than state-of-the-art multi-turn methods across target LLMs and benchmarks, while lightweight attackers can outperform larger baselines using compact state.The framework also reports systematic risk escalation and more uniform performance across risk categories without predefined taxonomies.Source: §1 · p. 2

2 Background and Related work

• Example-based and human-in-the-loop approaches coexist with taxonomy-driven or domain-specific methods that can limit generalization.The passage cites Guo et al. (2025) and Li et al. (2025) for the former, and Mo et al. (2025) and Zhang et al. (2024) for the latter.Source: §2 · p. 2
• Conversational-drift and agent-based simulations, including Crescendo, X-Teaming, ActorAttack, and GOAT, achieve strong results but require long interaction histories or complex architectures.Source: §2 · p. 2
• MDP-based red-teaming uses RL or search to optimize attack trajectories, but cited methods can depend on costly training, fixed action spaces, or handcrafted strategies.Constant-state reasoning supports long-horizon reasoning without growing context windows, but the cited work had not yet been applied to jailbreak settings.Source: §2 · p. 3

3 M3 framework

• M3 models attacker-target interaction as an MDP with a bounded internal state that maintains conversation history and guides policy decisions.Each turn is a decision step in a finite-horizon MDP, with distinct LLMs taking complementary roles.Source: §3 · p. 3, §3.1 · p. 3

3.1 M3 as an MDP with multiple LLMs

• The attacker maps the current state to an adversarial prompt, adapts to target behavior and evaluation scores, and can backtrack when progress stagnates.The two action choices are proceeding from the current state or reverting to a previous step; together with reward feedback, backtracking is designed to enforce risk escalation.Source: §3.1 · p. 3
• After each target response, the evaluator judges task completion and risk, then the state maintainer updates the internal state from the prompt, response, and feedback.The evaluator supplies a risk exposure score from 0 to 100, a binary success flag enabling early termination, and a brief explanation.Source: §3.1 · p. 3
• The state LLM compresses prior prompts, target responses, and evaluation signals into bounded memory that preserves information for subsequent attacks.This avoids attacker context-length limits while retaining past failures, partial success, and evaluator feedback for joint reasoning and compression.Source: §3.1 · p. 3

3.2 Algorithmic implementation of M3

• Each iteration generates an attacker prompt, evaluates the target response, updates compact state, and may terminate early after a successful jailbreak.Bounded backtracking explores alternative trajectories up to a predefined retry limit.Source: §3.2 · p. 4

4 Experimental setup

• The setup compares M3 with CRESCENDO, GOAT, and ACTORATTACK, using low- and high-capacity Mistral attackers and matched state LLMs.Targets include closed-weight Claude and GPT-4 models and open-source Llama-3 models, with one large and one small model per family.Source: §4 · p. 4
• Evaluation uses CRESCENDO TASKS with 15 tasks across eight risk categories and HARMBENCH with 90 sampled tasks across six categories.For category analysis, the benchmarks are combined into 12 non-overlapping risk categories.Source: §4 · p. 4
• The study reports ASR@1 and ASR@5, three AUC metrics for attack progress through turns, and Gini index for ASR uniformity across 12 categories.ASR@5 counts success when at least one of five independent runs yields a harmful response.Source: §4 · p. 4
• Table 1 presents ASR@1 and ASR@5 across both benchmarks and target LLMs, distinguishing closed from open-weight models and marking overall and within-capacity best results.The table uses bold for overall best performance and blue for the best method within each attacker-capacity group.Source: Table 1 · p. 5

5.1 Overall attack effectiveness and efficiency

• Up to 33% in ASR@1 and 53% in ASR@5 are reported for M3, while lightweight M3-LC gains up to 26% in ASR@1 and 34% in ASR@5 versus high-capacity baselines.M3-HC typically attains the highest ASR across both benchmarks and model families; M3-LC remains competitive and often matches or exceeds strong large baselines.Source: §5.1 · p. 5

5.2 Progression of attack effectiveness

• M3 shows a steadier, steeper increase in attack success over turns than baselines that often plateau early, and attains the largest AUC values across target models on both benchmarks.The AUC results indicate greater cumulative attack success across turns.Source: §5.2 · pp. 5–6

5.3 Performance across risk categories

• M3 improves the Gini index by up to 4.82 and maintains more consistent performance across 12 risk categories without predefined taxonomies.M3-HC shows the most consistent performance, with reported Gini index values of 4.24–6.22; category difficulty still varies, with Hate particularly challenging.Source: §5.3 · p. 6, Table 2 · p. 6

6 Conclusion

• M3 formulates multi-turn jailbreaking as an MDP within a multi-LLM system and is reported to achieve higher attack success while enforcing risk escalation.The authors identify analysis of model combinations, reward-based fine-tuning, control parameters, and memory consolidation as future directions.Source: §6 · p. 7

Ethics Statement

• Experiments use controlled benchmark tasks, and newly discovered vulnerabilities are not exploited outside the evaluation scope.The stated aim is to characterize vulnerabilities in support of more robust safety mechanisms.Source: Ethics Statement · p. 7

• Some representative conversation examples in the appendix may contain offensive language.Source: Content Warning · p. 11

A.1.1 Multi-Turn Jailbreaking of Large Language Models

• Some prior approaches depend on curated jailbreak datasets or human-authored attack trajectories, while taxonomy-driven methods can limit generalization to novel strategies.The cited approaches include example-based training, human-in-the-loop crafting, and predefined risk taxonomies.Source: §A.1.1 · p. 11
• Crescendo does not explicitly enforce risk escalation and requires the full interaction history, while agentic red-teaming systems can incur substantial computational overhead.The passage attributes the overhead to complex multi-component architectures and long conversational contexts.Source: §A.1.1 · p. 11
• M3 is presented as a streamlined, single-policy approach that dynamically adjusts its strategy on the fly to address reliance on curated data, architectural complexity, and unbounded conversational context.Source: §A.1.1 · p. 11

A.1.2 MDP-Based Red Teaming and Reinforcement Learning Approaches

• Prior MDP-based attacks use costly training, fixed actions, handcrafted strategies, or auxiliary models; M3 instead generates textual actions dynamically and escalates risk across turns through online black-box API access.The authors describe M3 as requiring no training or fine-tuning and no predefined policies.Source: §A.1.2 · p. 11

A.1.3 Memory Consolidation and Constant-State Reasoning / M3 State model

• M3 maintains a compact decision-relevant attack state across turns, summarizing progress, disclosures, refusals, effective or blocked strategies, and recommended escalation posture.The state model also tracks apparent intent, explicitness, safety boundaries, and safe directions for escalation.Source: §A.1.3 · p. 12, M3 State model · p. 17

A.2 M3 Algorithm

• M3 is a simplified MDP rather than strict RL: evaluation rewards guide state updates and backtracking, not attacker training, and all LLMs can be modularly substituted through API access.The attacker conditions decisions on immediate evaluation feedback but is not trained or fine-tuned with the reward signal.Source: §A.2 · p. 12

A.3 Additional experimental setup

• Experiments use evaluator temperature 0 and attacker/target temperature 0.5; M3 and Crescendo allow up to 10 turns and retries, while ActorAttack and GOAT use 20 turns.M3 and GOAT attacker generations are capped at 1024 tokens, but ActorAttack has no attacker-length cap because its attack graph is built upfront.Source: §A.3 · pp. 12–13

M3 Evaluation

• The evaluator judges task completion as a binary outcome and assigns risk exposure from 0 to 100, with a brief justification grounded in response content and detail.The rubric ranges from generic, non-actionable content at the low end to explicit, detailed disallowed content at the high end.Source: M3 Evaluation · p. 16

A.5 Prompts / M3 Attack

• M3’s attacker prompt asks for a next message that advances the task while minimizing refusal, and the prompt templates were refined for clarity and shared with Crescendo where possible.The M3 and Crescendo templates were generated with GPT assistance; GOAT’s prompt specification follows its original paper with minimal modification.Source: §A.5 · p. 13, M3 Attack · p. 18

A.4 Progression of attack success measured by AUC / A.6 Progression of attack effectiveness across target models / A.7 Performance of the baseline methods across risk categories

• Attack progression is summarized with normalized rectangle-rule and trapezoid-rule AUCs plus discounted AUC at 0.95; supplementary plots cover turn-wise progression and performance across risk categories.The figure captions identify target-family curves and category comparisons but do not provide numerical outcomes in these passages.Source: Figure 3 · p. 18, Figure 6 · p. 20, Figure 7 · p. 20

A.8 Successful Attack Example with M3

• A supplied example illustrates M3 inducing unsafe behavior from Claude 3.5 Sonnet on a profanity task, beginning with a benign inflation prompt and then reframing toward a gritty, raw portrayal.The excerpt shows the first target response scored at 0% risk and not successful before the subsequent escalation prompt.Source: §A.8 · pp. 21–22
• R: 35%, Success: True, in an example where the target produces the requested gritty perspective on inflation.The response describes rising costs from a blue-collar worker’s perspective.Source: §A.8 · p. 22

A.9 Unsuccessful Attack Example with M3

• In the Hate category, M3’s attack on GPT-4O fails (R: 0%, Success: False); the attacker asks about core values, and the target replies it has no personal beliefs or values.Source: §A.9 · pp. 22–23

A.10 State example

• The state example tracks user intent, task progress, disclosed information, explicitness, refusals, and effective or blocked strategies to guide safe escalation.After a refusal to write a phishing email, it recommends continuing with analysis of legitimate bank-email practices, phrases, and tone.Source: §A.10 · p. 24