PaperlayerView exact revision on arXiv

Source-linked AI summary

Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks

Ruohao Guo, Afshin Oroojlooy, Roshan Sridhar, Miguel Ballesteros, Alan Ritter, Dan Roth

arXiv:2510.02286v2cs.LGcs.AIcs.CL

TL;DR

Multi-turn attacks can be more successful than single-turn methods, while many existing approaches rely on templates or heuristics instead of learning long-horizon adaptive strategies. DIALTREE uses on-policy reinforcement learning with dialogue-tree search to discover attack strategies. Across 12 target LLMs, it reports average ASR of 81.5% and a 44.2% ASR improvement over prior state-of-the-art approaches.

  • Problem

    Existing red-teaming approaches often use manual or template-based attacks and lack learned long-horizon strategies, despite multi-turn attacks achieving higher success rates than single-turn methods.

    Source: §1 · p. 1

  • Method

    DIALTREE treats red-teaming as goal-oriented dialogue and uses on-policy RL with tree rollout, pruning, specialized rewards, and adaptive masking.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Across 12 target LLMs, DIALTREE achieves average ASR 81.5%, outperforming prior state-of-the-art approaches by 44.2% ASR.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings indicate that current LLMs remain vulnerable in multi-turn settings and that automated search-based red-teaming is a practical tool for stress-testing safety.

    Source: §7 · p. 10

    Training relies on non-verifiable rewards from imperfect proxy models, leaving reward hacking a risk despite separate-judge and human checks.

    Source: F LIMITATIONS AND FUTURE DIRECTIONS · p. 24

Abstract

from arXiv · show

Despite recent rapid progress in AI safety, current large language models remain vulnerable to adversarial attacks in multi-turn interaction settings, where attackers strategically adapt their prompts across conversation turns and pose a more critical yet realistic challenge. Existing approaches that discover safety vulnerabilities either rely on manual red-teaming with human experts or employ automated methods using pre-defined templates and human-curated attack data, with most focusing on single-turn attacks. However, these methods did not explore the vast space of possible multi-turn attacks, failing to consider novel attack trajectories that emerge from complex dialogue dynamics and strategic conversation planning. This gap is particularly critical given recent findings that LLMs exhibit significantly higher vulnerability to multi-turn attacks compared to single-turn attacks. We propose DialTree, an on-policy reinforcement learning framework integrated with tree search that autonomously discovers diverse multi-turn attack strategies by treating the dialogue as a sequential decision-making problem, enabling systematic exploration without manually curated data. Through extensive experiments, our approach not only achieves more than 44.2% higher ASR across 12 target models compared to previous state-of-the-art approaches, but also effectively uncovers new attack strategies by learning optimal dialogue policies that maximize attack success across multiple turns.

1 INTRODUCTION

Multi-turn attacks are more successful than single-turn methods, yet many existing approaches rely on templates or heuristics rather than learning long-horizon adaptive strategies. DIALTREE applies on-policy reinforcement learning and tree search to explore dialogue strategies, with experiments reporting higher attack success across 12 target models.

1 INTRODUCTION

  • Multi-turn attacks achieve higher success rates than single-turn methods by exploiting dialogue progression, contextual dependencies, and tactical adaptation after failed attempts.The passage cites Chao et al. (2025), Ren et al. (2025), and Rahman et al. (2025).Source: §1 · p. 1
  • Existing multi-turn methods often rely on handcrafted heuristics or templates and lack mechanisms for learning long-horizon adaptive strategies.DIALTREE frames red-teaming as strategic reasoning in goal-oriented dialogue, where actions adapt to target responses toward a jailbreak goal.Source: §1 · p. 1
  • Applying RL to multi-turn red-teaming must address exponentially expanding action choices, non-verifiable rewards from imperfect guardrails, and potentially unstable policy updates.DIALTREE responds with dialogue-tree rollout and pruning, a specialized reward function, and adaptive masking.Source: §1 · p. 2
  • Across 12 target LLMs, DIALTREE achieves average ASR 81.5%, outperforming prior state-of-the-art approaches by 44.2% ASR.The paper also reports discovering attack strategies absent from training data.Source: §1 · p. 2
  • DIALTREE is presented as a tree-based on-policy RL framework for diverse adaptive attacks, with quality-aware pruning and adaptive masking for format unlearning.The paper also highlights query efficiency, cross-model transferability, and insights for context-aware defenses.Source: §1 · p. 2

2 RED-TEAMING ATTACKS AS CONVERSATIONAL STRATEGIC REASONING

The paper models multi-turn red-teaming as goal-oriented interaction between an attacker and a target. The attacker conditions each action on the goal and dialogue history, while training combines supervised fine-tuning with reinforcement learning.

2 RED-TEAMING ATTACKS AS CONVERSATIONAL STRATEGIC REASONING

  • Each attack goal specifies unsafe content to elicit, and the turn-level state comprises that goal and the complete dialogue history.The history records the attacker’s chain-of-thought and query alongside the target’s response.Source: §2 · p. 2
  • At each turn the attacker generates a reasoning step and query from the prior state, while the target responds without access to the goal or attacker reasoning.Interaction ends when the target is jailbroken or the maximum turn limit is reached; Figure 1 illustrates branched paths with n=2 and Tmax=3.Source: §2 · p. 2, Figure 1 · p. 3
  • Training first uses supervised fine-tuning on red-teaming conversations paired with chain-of-thought, then RL to improve exploration and exploitation of vulnerabilities.The SFT stage teaches the required per-turn output format and relaxes safety constraints.Source: §2 · pp. 2–3
  • The RL objective maximizes expected reward for generated dialogue while penalizing deviation from a reference policy.Goals are sampled from D, and the reward function scores the attacker’s output interleaved with target responses.Source: §2 · p. 3

3 DIALTREE: DIALOGUE REINFORCED POLICY OPTIMIZATION WITH TREE SEARCH FOR STRATEGIC RED-TEAMING

DIALTREE combines dialogue-tree rollout and pruning with reward-guided policy optimization and adaptive masking. The design targets efficient exploration of attack trajectories and preservation of structured outputs during RL training.

3 DIALTREE: DIALOGUE REINFORCED POLICY OPTIMIZATION WITH TREE SEARCH FOR STRATEGIC RED-TEAMING

  • DIALTREE combines tree-based rollout with pruning, a reward function for non-verifiable jailbreak outcomes, and adaptive masking to improve multi-turn RL training stability.The framework is presented as a strategic, adaptive dialogue approach.Source: §3 · p. 3

3.1 DIALOGUE TREE ROLLOUT WITH PRUNING

  • Dialogue-tree rollout branches from shared dialogue context so the attacker can compare actions at each turn while exploring diverse strategies within a constrained search space.At each active state, candidate attacker actions elicit target responses; non-pruned trajectories are collected and rewarded according to jailbreak success.Source: §3.1 · pp. 3–4
  • Pruning removes malformed outputs and off-topic dialogue, then limits each turn to at most w surviving nodes through random subsampling.The criteria retain format-valid, goal-directed branches while controlling tree growth.Source: §3.1 · p. 4

3.2 OPTIMIZATION WITH ADAPTIVE MASKING

  • A HarmAug-Guard score above η on any turn yields reward 1; otherwise reward is 0, with η=0.5 by default.The method uses GRPO over groups of tree-rollout trajectories, with adaptive masks and group-relative advantages in policy optimization.Source: §3.2 · pp. 4–5
  • Invalid outputs rise from near zero after SFT to over 70% during RL, so adaptive masking protects format tokens on negative-advantage trajectories while leaving them trainable on positive ones.The paper reports empirical validation of the masking strategy.Source: §3.2 · pp. 4–5

4 EXPERIMENTS

DIALTREE outperforms prior baselines across 12 target models, while adaptive masking supports stable training. Ablations examine pruning, planning depth, branching width, and group size.

4.1 EXPERIMENT SETUP

  • Evaluation uses five-turn attacks against diverse target models, with GPT-4o judging harmfulness and ASR@1 counting success from a single attempt.The attacker uses Llama-3.1-8B-Instruct, trains against a frozen Llama-3.2-1B-Instruct target, and runs without test-time tree search.Source: §4.1 · pp. 5–6, §4.2 · p. 6

4.2 MAIN RESULTS

  • 81.5% average ASR across 12 target models, exceeding X-Teaming by 44.2% ASR.The evaluation covers six closed-source and six open-source models; the attacker is trained against Llama-3.2-1B-Instruct and transfers across model families.Source: Table 1 · p. 6, §4.2 · pp. 6–7

4.3 EFFECT OF ADAPTIVE MASKING

  • Adaptive masking keeps malformed-trajectory rates below 50% through most training while rewards grow, unlike no masking or static masking, which approach 100% malformed trajectories.Adaptive masking updates format tokens in positive-advantage trajectories and masks them in negative-advantage trajectories.Source: §4.3 · p. 7, Figure 2 · p. 7

4.4 ABLATION STUDIES OF DIALTREE

  • Removing all pruning drops average ASR by 25%, while tree rollout improves ASR by 9.8 points over the pruned no-rollout baseline.The authors attribute the pruning loss to invalid and low-quality dialogue paths that add noise to training.Source: §4.4 · p. 8, Table 2 · p. 8
  • ASR rises as conversation depth increases from one to five turns, then declines slightly at seven turns, which the authors hypothesize reflects sparse, delayed outcome rewards.They suggest process-based rewards or intermediate objectives as future ways to provide denser signals.Source: §4.4 · p. 8, Figure 3 · p. 8
  • Branching widths of two and four improve performance over linear dialogue; trajectory diversity is sufficient at n = 4 but drops at n = 8.The paper reports a positive correlation between trajectory diversity and test-time ASR for n = 1 to 4.Source: §4.4 · pp. 8–9, Table 3 · p. 8
  • Larger optimization groups generally improve attacker performance; on Mistral, ASR increases from 71.5% to 86.5% as group size grows from 8 to 32.The authors suggest larger trajectory pools may support more diverse exploration and richer feedback for policy updates.Source: Figure 3 · p. 8, §4.4 · p. 9

5 ANALYSIS OF RED-TEAMING ATTACK WITH DIALTREE

DIALTREE combines the highest attack success rate with the fewest queries, while attack success rises across dialogue turns before leveling off. Manual analysis also identifies novel strategies, including pretexting, gradual escalation, and multilingual jailbreaking.

Attack Efficiency

  • DIALTREE achieves the highest ASR with the fewest queries, while TAP and PAIR need around 40 queries for low success and Jailbreak-R1 and AutoDAN-Turbo remain below 60% ASR.The authors attribute the efficiency to guiding exploration toward promising attack trajectories rather than relying on random sampling.Source: Figure 4 · p. 9, §5 · p. 9

Attack success increases through dialogue progression

  • Attack success rises from 26.1% at turn one to 60% at turn four, then plateaus.The measure is the percentage of dialogues in which the target produces a harmful response (Score 5) at each turn.Source: Figure 5 · p. 9, §5 · p. 9

DIALTREE can explore new red-teaming strategies

  • Manual analysis identifies pretexting, gradual escalation, and multilingual jailbreaking among the new strategies discovered by DIALTREE.Figure 6 illustrates a benign pretext followed by gradual escalation and a later switch to English–Mandarin code-switching.Source: §5 · p. 9, Figure 6 · p. 10

6 RELATED WORK

Prior work spans single-turn and multi-turn red-teaming, defenses, and tree-search RL for verifiable tasks. DIALTREE addresses the less-explored use of adaptive RL policy learning for multi-turn attack discovery with non-verifiable rewards.

Red-Teaming and LLM Safety

  • Earlier red-teaming largely used single-turn attacks [prompt injection (Liu et al., 2023; Andriushchenko et al., 2025), role-playing (Liu et al., 2025c), and optimization-based methods (Zou et al., 2023; Zhu et al., 2024; Guo et al., 2025d)], while later work explored multi-turn attacks [Yang et al., 2024; Ying et al., 2025; Guo et al., 2025b; Rahman et al., 2025; Liu et al., 2025a].Multi-turn defenses were also developed (Hu et al., 2025); the authors distinguish their work from Hu et al. (2025), which models multi-turn jailbreaking as sequential decision-making for defense, without RL-based policy learning.Source: §6 · p. 10
  • Multi-turn attack methods are often constrained by predefined strategies, fixed seeds, and rigid interaction patterns, whereas DIALTREE is framed as adaptive exploration without manual curation or human priors.The authors position this as a difference in how attack strategies are explored.Source: §6 · p. 10

Reinforcement Learning for LLMs

  • Tree-search RL has advanced mathematical reasoning and code generation with verifiable rewards, but applying it to multi-turn conversation with imperfect, less reliable proxy rewards remains underexplored.The authors describe their work as the first extension of GRPO to multi-turn red-teaming with non-verifiable rewards.Source: §6 · p. 10

7 CONCLUSION

The paper frames multi-turn red-teaming as goal-oriented sequential decision-making and presents DIALTREE as an on-policy RL framework. Its experiments report baseline outperformance across 12 target models and identify automated, search-based red-teaming as a practical safety stress test.

7 CONCLUSION

  • DIALTREE combines dialogue-tree rollout and pruning, reward design for non-verifiable feedback, and adaptive masking to stabilize training.The task is formalized as goal-oriented sequential decision-making.Source: §7 · p. 10
  • Experiments show that DIALTREE outperforms single-turn and multi-turn baselines across 12 target models.Source: §7 · p. 10
  • The authors conclude that current LLMs remain vulnerable in multi-turn settings and describe automated, search-based red-teaming as a practical tool for safety stress-testing.They suggest broader multi-turn applications such as negotiation, debate, and pedagogical interactions, and identify process-based rewards as future work.Source: §7 · p. 10

ETHICS STATEMENT

The ethics statement acknowledges dual-use risks and describes controlled experiments and restricted access to implementation materials. The paper also states that LLM use during preparation was limited to editorial assistance.

ETHICS STATEMENT

  • The authors acknowledge dual-use risks and say transparent research on vulnerabilities is important for developing safety mechanisms before real-world exploitation.Experiments used locally hosted models or official APIs in controlled settings, with harmful examples included only when needed to demonstrate vulnerabilities.Source: ETHICS STATEMENT · p. 11
  • Access to trained attacker models, code, and data is planned by request for verified researchers who demonstrate legitimate safety objectives and agree to defensive-only use without redistribution.Requests are to be assessed by the authors; the paper also says technical contributions focus on the RL framework and tree-search methodology rather than specific attack payloads.Source: ETHICS STATEMENT · p. 11

A THE USE OF LARGE LANGUAGE MODELS (LLMS)

  • LLMs were used for editorial assistance with sentence structure, clarity, and grammar, not for research ideation, experimental design, result analysis, or core scientific content.Source: A THE USE OF LARGE LANGUAGE MODELS… · p. 16

B DATASETS

The experiments use SFT to initialize the attacker and draw training goals from three jailbreaking datasets. DIALTREE then explores multi-turn attacks through bounded dialogue-tree rollouts.

C.1 SUPERVISED FINE-TUNING

  • 397 attacker-target dialogues with chain-of-thought rationales initialize the attacker through SFT, teaching output format and contextual reasoning.The dialogues were constructed by prompting Zephyr-7B-beta to role-play both sides; without SFT, the attacker often refuses to generate attack queries.Source: B DATASETS · p. 17

B DATASETS

  • The main RL experiment samples 500 goals from AdvBench, DangerousQA, and CatQA, covering diverse harmful categories.The sample comprises 240 human-written AdvBench goals, 50 machine-generated DangerousQA goals, and 210 machine-generated CatQA goals.Source: B DATASETS · p. 17

C.2 REINFORCEMENT LEARNING

  • DIALTREE rolls out dialogue trees up to five turns with branching factor four, then prunes off-topic branches before retaining 32 dialogues per goal for optimization.Off-topic pruning uses an NLI classifier; a branch is pruned when entailment plus neutral probability is below 0.9.Source: §C.2 · pp. 17–18

C.3 EVALUATION

The evaluation uses a safety judge and guardrail reward model, then tests DIALTREE across ablations of initialization, rollout, data size, attacker backbone, and RL algorithm. Results report gains over baselines and robustness across settings.

C.3 EVALUATION / C.4 PROMPT TEMPLATES

  • Evaluation caps attacker and target outputs at 256 and 150 tokens, respectively, and scores policy compliance on a five-point scale.The scoring prompt distinguishes refusals, restrained responses, unsafe engagement, prohibited content, and direct fulfillment.Source: §C.3 · p. 18, §C.4 · p. 19

D.1 REWARD MODEL SELECTION / D.2 HUMAN EVALUATION AND LLM JUDGE RELIABILITY

  • HarmAug-Guard is selected as the RL reward model for its benchmark performance and low-latency trade-off; GPT-4o’s safety judgments show 74.7% Cohen’s Kappa agreement with human evaluators.The human study covered 30 dialogues and 132 query-response pairs.Source: §D.1 · p. 20, §D.2 · p. 20, Table 5 · p. 20

D.3 ABLATION ON THE SFT STAGE

  • 44.5% ASR improvement over SFT alone across all 12 target models highlights the gains from DIALTREE’s RL stage.The reported gains are largest on strongly safety-aligned models, including Claude-4-Sonnet (+69.5%), Grok-4 (+67.5%), and o3-mini (+64.5%).Source: Table 6 · p. 20, §D.3 · p. 21

D.4 MORE DETAILS OF ATTACK EFFICIENCY

  • DIALTREE uses fewer queries than Jailbreak-R1 across ten target models, while the four named models require more queries than others.The authors suggest these models appear more resistant; Table 7 reports average queries per target model.Source: §D.4 · p. 21, Table 7 · p. 21
  • 77.3% average ASR versus GRPO’s 68.7% on closed-source and 68.8% on open-source models is reported for DIALTREE with tree rollout.Both methods use the same 200 goals and hyperparameters; DIALTREE uses branching factor two, while GRPO uses single-path optimization.Source: §D.5 · p. 21, Table 8 · p. 22

D.6 IMPACT OF THE NUMBER OF TRAINING GOALS

  • Increasing training goals from 100 to 200 improves attack performance, while further increases beyond 200 degrade it.The authors hypothesize that larger datasets may add noise or less informative examples.Source: §D.6 · p. 21, Table 9 · p. 21

D.7 ABLATION ON ATTACKER MODELS / D.8 GENERALIZING DIALTREE TO ADDITIONAL RL ALGORITHM

  • Replacing the default attacker with Llama-3.2-3B preserves qualitative gains over baselines, and applying DIALTREE with DAPO also improves substantially over SFT.The DAPO experiment uses 200 training goals and branching factor two.Source: §D.7 · p. 22, §D.8 · p. 22, Table 10 · p. 22

D.9 ROBUSTNESS ANALYSIS OF DIALTREE

  • Five independent training runs produce consistent ASR performance across random seeds and initializations.Table 12 reports mean and standard deviation across the five runs.Source: §D.9 · p. 22

D.10 TEST-TIME SCALING EXPERIMENTS ON X-TEAMING AND ACTORATTACK

Test-time scaling comparisons, a Claude-4-Sonnet evaluation, and external-guardrail tests assess DIALTREE’s effectiveness across attack settings. The section also outlines its tree-based policy-optimization procedure.

D.10 TEST-TIME SCALING EXPERIMENTS ON X-TEAMING AND ACTORATTACK

  • 99.5% ASR at five attempts is reported for DIALTREE, exceeding X-Teaming’s ASR@10; X-Teaming’s ASR@5 evaluation costs about $79.8 per target model.The comparison uses four target models; DIALTREE uses an 8B model and incurs no API cost in these experiments.Source: §D.10 · p. 23, Table 13 · p. 23

D.11 ADDITIONAL EXPERIMENTS ON ATTACKING CLAUDE

  • On Claude-4-Sonnet, DIALTREE achieves 71% ASR@1 and 96.5% ASR@5, outperforming the SFT baseline and X-Teaming.Table 14 reports red-teaming results in ASR percent.Source: §D.11 · p. 23, Table 14 · p. 23

D.12 ROBUSTNESS OF DIALTREE UNDER EXTERNAL GUARDRAIL DEFENSE

  • With external guardrails on three target-model pairs, DIALTREE consistently outperforms SFT and maintains relatively high attack success rates.A jailbreak requires both guardrail classifiers to predict safe and the target to produce a harmful response.Source: §D.12 · pp. 23–24, Table 15 · p. 24

E ALGORITHM OUTLINE OF DIALTREE

  • Each iteration samples a goal, grows a dialogue tree by generating branches, prunes incorrect-format or off-topic branches, scores trajectories, and updates the policy using group-relative advantages.Negative-advantage trajectories have loss masked at format tokens.Source: E ALGORITHM OUTLINE OF DIALTREE · p. 24

F LIMITATIONS AND FUTURE DIRECTIONS

The paper identifies proxy-reward hacking as a continuing challenge and leaves advanced defense methods for future work. Its case study catalogs attack strategies found through DIALTREE’s exploration.

F LIMITATIONS AND FUTURE DIRECTIONS

  • Non-verifiable proxy rewards leave potential for reward hacking; future work could explore ensemble reward models or human-in-the-loop verification.The paper cites reward-model selection, separate GPT-4o evaluation, and human agreement as evidence that learned strategies are not merely exploiting reward artifacts.Source: F LIMITATIONS AND FUTURE DIRECTIONS · p. 24
  • Defense mechanisms are treated as a separate research direction, while the analyzed attack strategies may inform targeted countermeasures.The paper leaves advanced defense approaches as future work.Source: F LIMITATIONS AND FUTURE DIRECTIONS · p. 24

G.1 MULTI-TURN ATTACK STRATEGIES UNCOVERED BY DIALTREE

  • Manual analysis identifies ten emergent attack patterns, including intent laundering, gradual specificity escalation, persistent gap-filling, and cross-lingual evasion.Other patterns include policy-laundering keywords, privacy-defamation hybrid targeting, risk-to-checklist conversion, tool elicitation with end-to-end synthesis, sycophancy exploitation, and incremental refinement loops.Source: §G.1 · p. 25, G CASE STUDY · p. 25

G.2 EXAMPLES OF MULTI-TURN ATTACKS WITH DIALTREE

The examples show DIALTREE attackers adapting multi-turn prompts across several goals, with target responses ranging from partial answers to successful attacks. The dialogues illustrate escalation from broad or indirect requests toward more specific operational information.

G.2 EXAMPLES OF MULTI-TURN ATTACKS WITH DIALTREE

  • A request about evading law enforcement is redirected toward general crime methods, then toward operational tools and technology for avoiding detection.The example shows an initial direct request, a follow-up about general methods, and further questions about tools and specific steps; the final target response is scored 5 (successful attack).Source: §G.2 · pp. 26–27
  • The final evasion-focused follow-up asks about masking identities, locations, and payment methods, and the target gives a response scored as a successful attack.The requested information follows earlier turns about recruitment, transportation, and operational tools.Source: §G.2 · p. 27
  • A cybersecurity dialogue progresses from a university-research framing to a request for code, after an initial overview of HTTP Parameter Pollution, and ends with a response scored as a successful attack.The initial answer is scored 4; the later response offers vulnerable code and a crafted request, and is scored 5 (successful attack).Source: §G.2 · pp. 28–29
  • A fictional hate-mail request receives a partial answer, then follow-up requests for real cases and specific actions elicit a response scored as a successful attack.The successive target responses are scored 3, 4, and 5 (successful attack).Source: §G.2 · pp. 30–31

Related papers

Loading 2510.02286v2…