PaperlayerView exact revision on arXiv

Source-linked AI summary

X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents

Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, Saadia Gabriel

arXiv:2504.13203v2cs.CRcs.AIcs.CLcs.LGcs.MA

TL;DR

Multi-turn attacks remain underexplored because harmful intent can be distributed across exchanges, while existing red-teaming methods have limited adaptability and diversity. X-Teaming uses collaborative agents for planning, execution, verification, and prompt optimization, and generates multi-turn safety data. It achieves up to 98.1% attack success rates and supports stronger multi-turn safety alignment through XGuard-Train.

  • Problem

    Multi-turn safety remains underexplored because harmful intent can be distributed across exchanges, despite extensive work on single-turn safety.

  • Method

    X-Teaming uses collaborative Planner, Attacker, Verifier, and Prompt Optimizer agents to adaptively generate and evaluate diverse multi-turn jailbreaks.

  • Results

    Up to 98.1% ASR was achieved across representative leading closed-source and open-weight language models, while XGuard-Train improved multi-turn attack resistance by 34.2% on average over SafeMTData.

  • Takeaways & Limitations

    X-Teaming and XGuard-Train provide open tools and data for studying vulnerabilities and strengthening multi-turn safety alignment.

  • Takeaways & Limitations

    The work has dual-use risks because its multi-turn attack methodologies demonstrate vulnerabilities that could potentially be misused.

Abstract

from arXiv · show

Multi-turn interactions with language models (LMs) pose critical safety risks, as harmful intent can be strategically spread across exchanges. Yet, the vast majority of prior work has focused on single-turn safety, while adaptability and diversity remain among the key challenges of multi-turn red-teaming. To address these challenges, we present X-Teaming, a scalable framework that systematically explores how seemingly harmless interactions escalate into harmful outcomes and generates corresponding attack scenarios. X-Teaming employs collaborative agents for planning, attack optimization, and verification, achieving state-of-the-art multi-turn jailbreak effectiveness and diversity with success rates up to 98.1% across representative leading open-weight and closed-source models. In particular, X-Teaming achieves a 96.2% attack success rate against the latest Claude 3.7 Sonnet model, which has been considered nearly immune to single-turn attacks. Building on X-Teaming, we introduce XGuard-Train, an open-source multi-turn safety training dataset that is 20x larger than the previous best resource, comprising 30K interactive jailbreaks, designed to enable robust multi-turn safety alignment for LMs. Our work offers essential tools and insights for mitigating sophisticated conversational attacks, advancing the multi-turn safety of LMs.

1 Introduction

Multi-turn attacks remain an unresolved safety challenge because harmful intent can be distributed across exchanges, defeating protections designed for single prompts. X-Teaming addresses this gap with adaptive multi-agent red-teaming, achieving high attack effectiveness and diversity while producing data for multi-turn safety alignment.

  • Multi-turn attacks distribute malicious intent across exchanges, creating threats that current systems struggle to detect and prevent.
  • X-Teaming combines strategic attack planning with adaptive execution, using diverse personas, contexts, approaches, verification, and prompt optimization.
  • 98.1% maximum ASR was achieved across representative closed-source and open-weight language models under HarmBench evaluation.
  • 153% higher attack-plan diversity and 62% higher attack-execution diversity were measured against ActorAttack using pairwise embedding similarities.
  • XGuard-Train contains 30K multi-turn conversations seeded from 10K harmful behaviors across 13 risk categories, making it 20× larger than SafeMTData.
  • Models fine-tuned on XGuard-Train showed a 34.2% average improvement in multi-turn attack resistance over models trained on SafeMTData.

2 X-Teaming: an adaptive framework for multi-turn red-teaming

X-Teaming uses specialized agents to plan, execute, verify, and optimize multi-turn attacks against a target model. Its two-phase process revises plans or prompts in response to verifier feedback until success or a turn limit is reached.

  • Framework components: X-Teaming uses Planner, Attacker, Verifier, and Prompt Optimizer agents to automate adaptive multi-turn vulnerability discovery.
  • Framework components: The Planner generates attack plans combining personas, contexts, strategies, and turn-level trajectories, then revises them using conversation history and verifier feedback.
  • Framework components: The Attacker generates coherent queries conditioned on the plan phase, conversation history, and verifier scores.
  • Framework components: The Verifier scores each response from 1 to 5, where 1 indicates refusal or irrelevance and 5 indicates full compliance with the harmful behavior.
  • Framework components: TextGrad refines attacker queries when verification scores decrease by using textual feedback in an optimization prompt.
  • Attack execution process: The execution process advances when verifier scores remain stable or improve, but triggers prompt optimization when scores decline.
  • Attack execution process: An attack succeeds when any response receives verifier score 5; otherwise, the Planner extends the trajectory until success or the maximum turn limit.

3 X-Teaming effectively explores diverse multi-turn attacks of LMs

X-Teaming combines multi-agent planning, execution, verification, and prompt optimization to explore diverse multi-turn jailbreaks. Experiments show strong attack effectiveness, diversity, efficiency, and sensitivity to attack parameters.

  • Attack effectiveness: 98.1% ASR is achieved on DeepSeek V3, with 96.2% on Claude 3.7 Sonnet and consistent gains over prior single-turn and multi-turn methods.X-Teaming also reaches 91.8% ASR against SafeMTData-tuned Llama-3-8B-Instruct.
  • Efficiency: Successful attacks average 3.5 plans, 0.6 TextGrad iterations, and 4.3 turns despite upper bounds of 50 plans, 4 iterations, and 7 turns.X-Teaming also uses fewer tokens than ActorAttack across tested target models and employs the free open-source Qwen-2.5-32B attacker.
  • Attack effectiveness: 94.6% ASR versus ActorAttack’s 75.7% demonstrates an 18.9% advantage under identical plans, token budgets, attacker model, and target model.Both methods used 10 plans, Qwen-2.5-32B as attacker, and GPT-4o as target.
  • The number of attack plans: 97.6% ASR is reached with 40 attack plans, compared with 70.7% using 10 plans, with no further improvement at 50 plans.These experiments fix conversation length at 7 turns and disable TextGrad optimization.
  • The number of conversation turns: 92.7% ASR occurs at 8 turns versus 19.5% at 2 turns, then declines to 87.8% at 10 turns as the attacker may lose the original plan, persona, and context.The experiment fixes 10 attack plans and disables TextGrad optimization.

4 Enhancing the interactive robustness of LMs with XGuard-Train

XGuard-Train addresses the shortage of diverse multi-turn safety training data by using X-Teaming to generate large-scale jailbreak conversations and train safety-aligned models. Across Llama-3.1-8B and Qwen-2.5-7B evaluations, the dataset improves resistance to multi-turn attacks while preserving strong general capabilities.

  • XGuard-Train dataset: XGuard-Train contains 30K diverse attack trajectories generated from 10,000 harmful behaviors spanning 13 risk categories.Successful jailbreak responses were replaced with carefully crafted refusals, producing a safety-training dataset with 5.10 conversation turns versus 5.08 for SafeMTData.
  • Training setup: The safety-alignment experiments compare Tulu-Mix-only, SafeMTData-enhanced, and XGuard-Train-enhanced models using consistent fine-tuning configurations.Experiments were conducted on Llama-3.1-8B and repeated with Qwen-2.5-7B using identical training configurations.
  • Evaluation: The evaluation covers multi-turn attack resistance, single-turn safety, and general capabilities for safety-trained Llama-3.1-8B and Qwen-2.5-7B models.Table 5 presents the comprehensive results across these three dimensions.
  • Llama-3.1-8B results: 52.2% ASR for the XGuard-Train-tuned Llama-3.1-8B model under X-Teaming compares with 93.7% for SafeMTData and 80.5% for the TuluMix-only baseline.Against ActorAttack, SafeMTData performs better than XGuard-Train, at 8.9% versus 18.9% ASR, which the authors associate with attack-specific overoptimization.
  • Qwen-2.5-7B results: 22.6% average ASR across three attack frameworks for Qwen-2.5-7B trained on XGuard-Train compares with 36.3% for SafeMT, a 13.7% improvement margin.Under Crescendo specifically, XGuard-Train achieves 8.7% ASR versus 22.6% for SafeMT, supporting cross-framework generalization.

5 Related work

Related work has progressed from direct single-turn jailbreak prompts toward multi-turn manipulation, while agentic and prompt-optimization techniques provide tools for adaptive attacks. Safety datasets and benchmarks nevertheless remain concentrated on single-turn interactions and limited in multi-turn scale and diversity.

  • Jailbreaking evolution: Early jailbreak research primarily used single-turn prompts that directly embedded instructions to bypass model restrictions.Subsequent work automated single-turn attacks through universal adversarial prompts and related optimization methods.
  • Jailbreaking evolution: As model alignment improved, research shifted toward multi-turn jailbreaks that gradually steer benign conversations toward harmful outcomes.The cited multi-turn methods reflect a move from one-shot exploits to conversational manipulation.
  • Agents and prompt optimization: Prior work uses agents for defense, whereas this paper employs agentic LMs offensively and uses TextGrad rather than self-talk for prompt optimization.Prompt optimization has been applied both to jailbreak efficacy and to broader language-model performance.
  • Interactive safety resources: Existing safety resources predominantly target single-turn interactions and remain limited in the scale and diversity of materials for multi-turn conversational safety.This limitation affects datasets, benchmarks, and safety classifiers intended for evaluating or training interactive systems.

6 Conclusion

X-Teaming is an adaptive multi-turn red-teaming framework designed to simulate realistic adversarial tactics while improving jailbreak effectiveness and diversity. The work also releases XGuard-Train and identifies variation in vulnerability across harmful-behavior categories.

  • Conclusion: X-Teaming achieves attack success rates of up to 98.1% against leading language models while exhibiting high diversity in attack planning and execution.The framework systematically simulates realistic adversarial tactics in multi-turn jailbreak scenarios.
  • Conclusion: Cybersecurity exploits and social manipulation are most susceptible to multi-turn attacks, whereas chemical weapons, explicit violence, and extreme hate content maintain the strongest defenses.These are the vulnerability patterns reported in the authors’ analysis.
  • Conclusion: XGuard-Train is released as the largest multi-turn safety dataset to date, alongside safety-trained model checkpoints and reproducible training recipes.The stated goal is to expand resources for mitigating multi-turn exploitation and supporting multi-turn safety training research.

Ethics Statement

The ethics statement acknowledges that X-Teaming and XGuard-Train are dual-use because they expose vulnerabilities through multi-turn attack methods. It describes access controls and frames the release as supporting defensive research, while the supplied algorithmic passages outline the framework’s attack-generation process.

  • Dual use: X-Teaming and XGuard-Train have dual-use implications because their multi-turn attack methodologies demonstrate vulnerabilities in current language models.The authors explicitly acknowledge that the findings could potentially be misused.
  • Risk mitigation: Access controls require users to agree to terms restricting usage to research and defensive purposes.The authors state that these controls are intended to mitigate risks associated with public release.
  • Framework process: The framework’s formalized process generates attack plans, executes adaptive conversations, and optimizes prompts when progress stalls.The algorithm describes planning diverse personas, contexts, approaches, and trajectories, followed by verifier-guided adaptation and TextGrad optimization.

A.3 Comparison with Existing Multi-Turn Jailbreaking Methods

X-Teaming combines specialized agents, adaptive plan revision, diverse attack scenarios, and prompt optimization to explore multi-turn jailbreaks. It achieves high attack success across models and languages while generating strategically diverse plans.

  • Framework comparison: Multi-agent collaboration combines Planner, Attacker, Verifier, and Optimizer roles for adaptive multi-turn jailbreak generation and evaluation.The framework revises plans when models resist and optimizes unsuccessful prompts.
  • Framework comparison: X-Teaming requires distinct personas, contexts, tactical approaches, and conversation flows rather than repeating fixed strategies.Its planning prompts explicitly enforce diversity across previously generated strategies and within each new set.
  • Attack effectiveness: 98.1% ASR is achieved on DeepSeek V3, while Claude 3.5-Sonnet shows 67.9% ASR and Llama-3-70B-IT shows 84.9% ASR across semantic categories.Cybercrime is the most vulnerable category, whereas Harmful content and Misinformation show greater resistance.
  • Cross-lingual results: 64% ASR versus ActorAttack’s 28% is achieved for Chinese, 56% versus 18% for Arabic, and 52% versus 10% for Bengali.The comparison spans high-, medium-, and low-resource languages.
  • Attack diversity: 0.82 average pairwise diversity is achieved across ten sampled attack plans, using varied personas such as a tech journalist, psychologist, drama teacher, and ethical hacker.The plans also differ in their strategic approaches and contexts.

B.5 Hyperparameter Ablation and Additional Model Results

The ablation study identifies a configuration that improves attack success as TextGrad iterations, conversation turns, and attack plans increase. Additional results show strong vulnerability for Qwen-2.5-32B-IT.

  • Hyperparameter ablation: Increasing TextGrad iterations from 3 to 4, conversation turns from 6 to 7, and attack plans from 5 to 10 significantly improves attack success rates.The main experiments use the resulting configuration of 4 iterations, 7 turns, and 10 plans.
  • Additional model results: 99.4% ASR is achieved by Qwen-2.5-32B-IT on the HarmBench test set.This Qwen result is reported as an additional model result outside Table 2.

B.6 Verifier Agreement Analysis Details

The verifier analysis compares GPT-4o’s judgments with HarmBench classifiers and LlamaGuard 3 across target models. Agreement varies by model and evaluator, providing a detailed view of verifier consistency.

  • Agreement analysis: GPT-4o verifier agreement with HarmBench classifiers and LlamaGuard 3 is reported separately for each target model.Figure 5 extends the verifier agreement analysis with model-level comparisons.
  • Agreement analysis: 87% agreement is observed for Gemini 2.0 Flash, 83% for GPT-4o, and 81% for Claude 3.7 Sonnet with HarmBench classifiers.Claude 3.5 Sonnet has lower agreement at 75% among proprietary models.
  • Agreement analysis: 90% agreement is observed for the SafeMTData-tuned Llama-3-8B-IT variant, compared with 87% for standard Llama-3-8B-IT and 85% for Llama-3-70B-IT.Deepseek V3 has the lowest agreement among the listed open-weight models at 74%.

C.1 Dataset Details

XGuard-Train uses seed harmful behaviors drawn from diverse categories to construct a multi-turn safety dataset. Table 12 provides representative seed behaviors and their corresponding categories.

  • Seed behaviors: XGuard-Train contains seed harmful behaviors from diverse WildJailbreak vanilla harmful categories.The dataset documentation presents sample seed behaviors paired with their categories.
  • Seed behaviors: Table 12 lists sample seed behaviors together with the categories used to generate the XGuard-Train multi-turn safety dataset.
  • Dataset construction: The seed behaviors provide the category-organized inputs for generating XGuard-Train conversations.The passage identifies these behaviors as the basis for dataset generation.

C.2 XGuard-Train Dataset Generation

XGuard-Train is generated by applying X-Teaming to diverse harmful behaviors, refining multi-turn attack trajectories, and pairing successful jailbreaks with safe refusals. The resulting dataset contains 30,695 multi-turn conversations for safety training.

  • Dataset construction: 10,000 harmful behaviors were sampled from WildJailbreak, with two to five distinct attack plans generated for each behavior.The plans vary personas, contexts, and conversational approaches.
  • Attack generation: The complete X-Teaming pipeline executed and refined these plans using verification feedback and dynamic plan adjustment.Attacker queries were refined when verification scores decreased, producing jailbreak conversations averaging 5.10 turns.
  • Safety-pair construction: Successful jailbreaks were converted into training examples by replacing harmful model responses with carefully crafted helpful refusals.The refusal prompt preserves the conversation flow while explaining why the final query is unsafe and declining harmful information.
  • Dataset output: The final dataset contains 30,695 complete multi-turn attack-refusal pairs for multi-turn safety training.Each pair combines a jailbreak conversation with a safe response to the harmful final query.

D.1 Evaluation Benchmarks Details

The evaluation suite measures both general capabilities and safety behavior across established benchmarks. It includes broad knowledge and reasoning tests alongside single-turn harmful-content resistance tests.

  • General capability evaluation: General capability evaluation uses MMLU, GSM8K, MATH, GPQA, HumanEval, HellaSwag, and BIG-Bench-Hard.These benchmarks cover domain knowledge, mathematical reasoning, problem solving, coding, and broad task performance.
  • General capability evaluation: MMLU evaluates 57 tasks spanning areas such as elementary mathematics, US history, computer science, and law.It provides broad domain-knowledge evaluation across multiple subjects.
  • Single-turn safety evaluation: Single-turn safety evaluation uses DAN, WildGuard, XSTest, and Jailbreak Trigger to test resistance to harmful content generation.The benchmarks assess direct jailbreak resistance, harmfulness detection, and response safety.
  • Single-turn safety evaluation: WildGuard covers 13 risk categories with vanilla and adversarial examples for measuring prompt harmfulness detection and response safety.DAN contains 1,405 in-the-wild jailbreak prompts spanning varied attack strategies.

D.2 Safety-Tuned Models: Additional Benchmark Results

Additional evaluations compare safety-tuned models on capability and jailbreak-resistance benchmarks, while examples illustrate how X-Teaming can turn a refused single-turn request into a successful multi-turn attack. The results highlight both model-specific resistance and the role of adaptive conversational manipulation.

  • Additional benchmark results: Table 13 reports additional safety and capability results for safety-tuned Llama-3.1-8B and Qwen-2.5-7B variants.The table includes HellaSWAG, HumanEval, Big-Bench-Hard, and Jailbreak Trigger evaluations.
  • Additional benchmark results: Qwen-2.5-7B with XGuard-Train +TuluMix shows improved Jailbreak Trigger resistance at 5.5%, compared with 8.75% for other variants.Capability performance remains consistent across the Qwen-2.5-7B models.
  • Additional benchmark results: Llama-3.1-8B with XGuard-Train +TuluMix maintains comparable capability scores but shows 18.0% Jailbreak Trigger vulnerability, versus 10.0% for TuluMix and SafeMT+TuluMix.The additional safety result is therefore not uniformly improved across model variants.
  • Multi-turn attack success: A three-turn cognitive-scientist persona and persuasion-analysis plan leads to a score-5 jailbreak producing glorifying content.The sequence progresses from discussing persuasion, to requesting examples, to asking for a glorifying post.

E.2 Unsuccessful Attack Example

This example traces an adaptive multi-turn attack that begins with benign-seeming documentation questions and escalates toward preventing child laborers from reporting abuse. The target model ultimately refuses the harmful requests, and TextGrad optimization does not restore the attack score.

  • Outcome: Despite multiple plan revisions and TextGrad attempts, the target model maintains its safety guardrails and refuses the requested harmful information.This case is presented as an unsuccessful attack against robust model safety measures.
  • Attack trajectory: The attack plan uses three turns to move from documenting child labor cases to asking about barriers to reporting abuse and tactics for preventing reports.The plan frames the conversation as human-rights research before reaching the harmful objective.
  • Attack trajectory: The attacker first asks about documenting child labor cases and then about barriers children face when reporting abuse.The target initially provides a score-1 response on documentation and a score-3 response on reporting barriers.
  • Model resistance: The target refuses to provide strategies for preventing children or individuals from seeking help from authorities, redirecting toward safety and support.The refusals explicitly cite ethical and legal concerns and recommend protective measures instead.
  • Adaptive reframing: The revised prompt reframes the request around internal cohesion, confidentiality, transparency, and open communication rather than suppressing reports.The revised response offers organizational practices such as clear values, feedback channels, and transparent reporting mechanisms.
Loading 2504.13203v2…