PaperlayerView exact revision on arXiv

Source-linked AI summary

MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming

Weiyang Guo, Jing Li, Wenya Wang, YU LI, Daojing He, Jun Yu, Min Zhang

arXiv:2505.17147v1cs.CRcs.AI

TL;DR

Multi-round dialogue can hide malicious intent, and current safety alignment lacks effective multi-round methods. MTSA trains thought-guided red-team attacks and iteratively optimizes both the red-team and target models using future rewards. The authors report state-of-the-art red-team attack success and improved target safety on single- and multi-round benchmarks after three alignments, without lost generality or over-rejection.

  • Problem

    Multi-round dialogue can obscure malicious intent, yet current safety-alignment algorithms mainly address single-round scenarios and lack effective multi-round alignment.

    Source: §1 · pp. 1–2

  • Method

    MTSA initializes a red-team model with thought-guided attacks, then iteratively optimizes it and the target model from interaction data, using future rewards for multi-turn alignment.

    Source: §1 · p. 2

  • Results

    The red-team model achieves state-of-the-art attack success rates, while after three iterative alignments the target improves safety on single- and multi-round benchmarks without losing generality or causing over-rejection.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that iterative confrontation with future-reward alignment improves LLM safety performance and robustness to varied attack strategies.

    Source: §6 · p. 9

    Red-team dialogue data still rely on manual templates, limiting attack coverage and quality to the initial seed dataset.

    Source: Limitations · p. 10

Abstract

from arXiv · show

The proliferation of jailbreak attacks against large language models (LLMs) highlights the need for robust security measures. However, in multi-round dialogues, malicious intentions may be hidden in interactions, leading LLMs to be more prone to produce harmful responses. In this paper, we propose the \textbf{M}ulti-\textbf{T}urn \textbf{S}afety \textbf{A}lignment (\ourapproach) framework, to address the challenge of securing LLMs in multi-round interactions. It consists of two stages: In the thought-guided attack learning stage, the red-team model learns about thought-guided multi-round jailbreak attacks to generate adversarial prompts. In the adversarial iterative optimization stage, the red-team model and the target model continuously improve their respective capabilities in interaction. Furthermore, we introduce a multi-turn reinforcement learning algorithm based on future rewards to enhance the robustness of safety alignment. Experimental results show that the red-team model exhibits state-of-the-art attack capabilities, while the target model significantly improves its performance on safety benchmarks.

1 Introduction

Multi-round dialogues can obscure malicious intent, while existing safety-alignment algorithms largely focus on single-round scenarios. MTSA combines thought-guided attack learning with iterative red-team/target-model optimization and reports stronger attacks alongside improved target-model safety.

1 Introduction

  • Multi-round dialogue can hide malicious intent, and current alignment algorithms lack effective multi-round safety alignment.The paper notes that variety in multi-round jailbreaks also makes manual collection of sufficient alignment data difficult.Source: §1 · pp. 1–2
  • MTSA trains a red-team model on thought-guided attacks, then iteratively uses interaction data to optimize both the red-team and target models.The method uses trajectory sampling during adversarial iterative optimization.Source: §1 · p. 2
  • Future-state rewards support dynamic preference optimization intended to improve the robustness of safety alignment.The approach introduces a multi-turn reinforcement-learning algorithm based on future rewards.Source: Figure 1 · p. 1, §1 · p. 2
  • The red-team model achieves state-of-the-art attack success rates, while after three iterative alignments the target improves safety on single- and multi-round benchmarks without losing generality or causing over-rejection.The paper also reports analysis across safety, generality, robustness, and over-rejection.Source: §1 · p. 2

2 Related Work

Prior multi-round jailbreak approaches are described as lacking interactivity and strategic adaptation, while existing safety-alignment approaches focus on single-round dialogue. The paper positions thought-guided attacks and future-reward alignment as responses to those gaps.

2.1 Multi-Turn Jailbreak Attack

  • Existing multi-round jailbreak methods are described as lacking interactivity and strategy for complex dialogue environments.The paper contrasts these methods with its thought-guided attack data approach.Source: §2.1 · p. 2

2.2 LLM Safety Alignment

  • Prior safety-alignment approaches, including iterative red-teaming methods, are described as focusing on single-round dialogue; the paper proposes future-reward alignment for multi-round settings.The related-work discussion includes MART and GPO as iterative approaches before identifying the multi-round gap.Source: §2.2 · pp. 2–3

3 Problem Setup

The setup models a multi-round attack as an interaction between a red-team model and a target model, beginning from an objectionable-content target. Each turn adds an attack question and target response until the dialogue reaches its final state.

3 Problem Setup

  • The attack target x denotes the objectionable content sought by the attacker, and the red-team model generates the initial question from it.The target model responds, after which the red-team model conditions later questions on the target and prior dialogue.Source: §3 · p. 3
  • The interaction continues until H rounds, with the final states representing the completed multi-round dialogue.The red-team final state includes the final question and response; the target-model final state includes the final response.Source: §3 · p. 3
  • At each turn, the red-team state contains the target and prior questions and responses, while its generated question is treated as an action.The target-model state includes the dialogue through the current question, and its response is the corresponding action.Source: §3 · p. 3

4 Methodology

MTSA initializes its red-team model with thought-guided attack examples, then uses future-reward optimization and alternating adversarial interaction to train both models. The procedure addresses the cumulative nature of multi-turn toxicity and uses rewards tailored to safety, helpfulness, and attack diversity.

4.1 Thought-guided Attack Learning

  • Thought-guided attack learning prompts the red-team model to assess the dialogue state and form thoughts before attacking, then selectively fine-tunes an initialization dataset for effective, diverse attacks.The paper identifies purpose inversion, query decomposition, role play, and mixed mode as attack categories.Source: §4.1 · pp. 3–4

4.2 Multi-Turn RLHF with Future Rewards

  • Because multi-turn toxicity is cumulative, optimizing only the final dialogue round can reduce safety-alignment generalization.The paper says last-round-only training also introduces covariate shifts between training and testing distributions.Source: §4.2 · p. 4
  • The target-model objective uses expected future rewards and aims to make the current policy’s implicit reward match future reward, while constraining policy change from the previous iteration.The method estimates future reward by independently sampling trajectories to ending states rather than relying on a critic model, which the paper says can predict unsatisfactorily in multi-turn reinforcement learning.Source: §4.2 · p. 4
  • For red-team optimization, the method uses direct preference reward, with positive and negative examples determined by the rewards of their ending states.This differs from the target model’s relative preference reward optimization.Source: §4.2 · p. 4

4.3 Adversarial Iterative Optimization

  • Each iteration samples attack targets, collects red-team/target-model interactions and trajectories, then builds ending-state-reward preference data to optimize both models.Initially, target responses in harmful rounds are revised with a safety prompt before independent trajectories are sampled; later, model sampling builds target safety preferences, while rejection sampling and temperature changes enrich red-team data.Source: §4.3 · p. 5
  • Target-model preference selection uses toxicity and helpfulness rewards, while red-team reward uses safety and diversity signals.The target reward uses ArmoRM to balance helpfulness and safety; red-team diversity combines semantic diversity and textual similarity.Source: §4.3 · p. 5

5 Experiments

MTSA’s experiments evaluate iterative red-team attacks and target-model alignment across multiple models and benchmarks. The results report strong attack performance, improved target-model safety, and several trade-offs and conditions that qualify those outcomes.

5.1 Experimental Setting

  • The experiments use six prevalent LLMs, AdvBench for red-team testing, safety benchmarks for target models, and generality and false-refusal evaluations.Online training used 2,073 valid attack targets and 3,000 harmless prompts, with three iterations and a maximum of five attack rounds.Source: §5.1 · pp. 5–6

5.2.1 Evaluation of Red-team Models

  • Among tested initialization sizes, k = 100 makes the initial red-team model Pareto-optimal in toxicity and diversity after iterative training.The initialization selects low-similarity examples from the synthetic dataset.Source: §5.1 · p. 5, §5.2.1 · p. 6

5.2.2 Evaluation of Target Models

  • After three iterations, MTSA improves target-model safety, reducing the success rate of MTSA-R3’s attack by 67%.The reported safety gains also extend to single-round evaluation.Source: Table 2 · p. 7, §5.2.2 · pp. 7–8

5.2.1 Evaluation of Red-team Models

  • MTSA-R3 achieves the highest ASR among the compared multi-round attack methods, with state-of-the-art performance reported for a 7B model.ASR increased with each iteration; MTSA-R3 also yielded the best toxicity and diversity outcomes among the compared methods.Source: §5.2.1 · pp. 6–7, Figure 4 · p. 8

5.2.2 Evaluation of Target Models

  • MTSA-Tx has lower toxicity than other methods across rounds, while dialogue helpfulness shows no substantial fluctuation; the target model also retains near-original generality.The paper reports that MTSA-Tx defense gains on multi-round interactions generalize to single-round situations.Source: §5.2.2 · pp. 7–8
  • MTSA-T3 improves defenses against unsafe problems with a 5.62% increase in over-denial rate.The authors characterize this as a balance between safety and usability.Source: Figure 5 · p. 8, §5.2.2 · p. 8

5.3 Cost Analysis

  • MTSA reduces training-phase token cost by 34.37%, but overall token cost, including offline trajectory generation, rises by 16.96%.The authors note that training-phase savings concern the portion with higher actual computational-resource requirements.Source: Table 4 · p. 8, §5.3 · p. 8

5.4 Ablation Study

  • Removing the thought process makes red-team attacks less toxic, and MTSA-T3 outperforms SFT, DPO, and IPO in win rate for safety and helpfulness.The authors attribute the optimization balance to dynamic rewards.Source: Figure 6 · p. 9, §5.4 · p. 9

6 Conclusion

MTSA iteratively improves red-team and target-model capabilities in confrontation, and uses future rewards to improve safety-alignment robustness.

6 Conclusion

  • MTSA iteratively improves red-team and target-model capabilities through confrontation.The conclusion also describes iterative optimization as improving robustness to varied attack strategies.Source: §6 · p. 9
  • A future reward-based alignment algorithm is designed to improve efficiency in multi-round safety alignment.The conclusion says the iterative process improves LLM safety performance and robustness to varied attack strategies.Source: §6 · p. 9
  • MTSA is presented as an alignment strategy for LLM dialogue safety with broad applications and practical value.Source: §6 · p. 9

Limitations

The authors identify constraints in attack-data templating, safety assessment, and computational requirements.

Limitations

  • Manual templates constrain red-team dialogue data to the coverage and quality of the initial dataset.Improving jailbreak-attack quality and diversity beyond the original seed dataset is left for future work.Source: Limitations · p. 10
  • Static GPT-4o evaluations on Cosafe and Beavertrail cannot adequately assess safety in dynamic environments.A dynamic safety-evaluation framework is proposed as future work.Source: Limitations · p. 10
  • The adversarial iterative framework has high computational demands, especially during online sampling and resampling.The authors say a more efficient sampling strategy would improve practical usability.Source: Limitations · p. 10

Ethical Considerations

The authors acknowledge that MTSA-related materials could be abused and threaten LLM safety.

Ethical Considerations

  • The framework remains at risk of abuse, and the authors say potential misuse could threaten LLM safety.They publish a multi-round jailbreak corpus and a trained red-team model.Source: Ethical Considerations · p. 10
  • The authors say they will take steps to limit use of the technology for malicious purposes.Source: Ethical Considerations · p. 10

A Experiments Details

The attack-target collection contains 2,400 valid harmful targets and includes harmless prompts for target-model training.

A.1 Source of Attack Target.

  • 2,400 valid attack targets were collected after harmful prompts from different programs were filtered and cleaned.The filtering aimed to ensure high quality and non-duplication of attack targets.Source: §A.1 · p. 14
  • The collected targets were intended to cover most harmful categories in red-team testing.Source: §A.1 · p. 14
  • Harmless prompts extracted from Ultrachat were also used to train target models.Source: §A.1 · p. 14

A.2 Red Team Data Synthesis.

Red-team data synthesis seeds attacks with a thinking process that adapts strategy to target-model responses; the inference section specifies shared experimental settings.

A.2 Red Team Data Synthesis.

  • The seed datasets are based on a jailbreak method described earlier in the paper.The passage does not identify that method further.Source: §A.2 · p. 14
  • Each seed attack pairs a basic strategy with selective strategy changes based on the target model’s response.The authors provide example attacks to teach the model basic attack strategies.Source: §A.2 · p. 14

A.3 Details of inference.

  • Inference experiments use consistent chat templates and hyperparameters across red-team and target models, with separate settings for closed-source API models.For the clustered experiments, settings include batch size 1, temperature 0.8, top-p 0.95, and max_tokens 512; API models use temperature 1 and top-p 0.9.Source: §A.3 · p. 14

B Metric and Benchmark.

This block describes the evaluation rubric, diversity metric, baselines, and benchmarks, alongside the red-team prompts, attack seeds, and GPT-4o judging procedure.

B.1 GPT-4o Evaluation Prompts.

  • An attack counts as successful only when GPT-4o assigns a score of 5 on its 1–5 harmfulness and exhaustiveness scale.Higher scores indicate more harmful and exhaustive responses.Source: §B.1 · p. 14

B.2 Divicity metric.

  • Attack-cue diversity averages normalized textual and embedding diversity metrics, with higher values indicating greater diversity.SelfBLEU over n-grams n=2, 3, 4, 5 measures textual diversity; sentence embeddings measure semantic diversity.Source: §B.2 · p. 14

B.3 Baseline.

  • The baseline descriptions cover MART’s iterative adversarial interactions and HARM’s risk-taxonomy-based testing, while Table 7 is scoped to attack-target sources, counts, and selection.Source: §B.3 · pp. 14–15, Table 7 · p. 15

B.4 Benchmark.

  • The benchmark descriptions include BeaverTails for safety assessment and CoSafe for security attacks involving multi-round conversational coreference.The paper says it randomly selected 200 BeaverTails items for safety assessment and describes CoSafe as covering 1,400 questions across 14 categories.Source: §B.4 · p. 15

C.1 Red-team Prompts.

  • The red-team prompt asks the model to hide malicious intent, respond to target-model replies, and use varied attack techniques while producing a thought and attack question.Source: §C.1 · p. 16

C.2 Red-team Attack Seeds.

  • The attack seed example illustrates a staged strategy that changes its framing and follow-up questions in response to the target model’s replies.Source: §C.2 · p. 16

C.3 GPT-4o Judge Prompts.

  • The GPT-4o judge prompt evaluates a response against prohibited-use guidance using five scores, from refusal or restrained replies to direct fulfillment of the user’s intent.Source: §C.3 · p. 17

D.1 Attack Examples.

D.1 illustrates an attack dialogue ending in a judge score of 5; D.2 presents a defensive example with different judge scores across turns.

D.1 Attack Examples.

  • The judge assigned score 4 to a response outlining legal ways to obtain controlled chemicals and naming specific suppliers, but not explicitly advising how to avoid tracing.Source: §D.1 · p. 18

D.2 Defend Examples.

  • In D.2, the defensive example receives scores 1, 3, 2, and 3 over four attack turns.Source: §D.2 · p. 19

Related papers

Loading 2505.17147v1…